LeRobot Worldwide Hackathon 🌍
My experiences and thoughts from the 2-day event
Last weekend (14th-15th June 2025), I joined the LeRobot Worldwide Hackathon organized by HuggingFace. With 100+ local events globally and more than 3,000 registered participants, it was the largest hackathon I've ever experienced. This event was a celebration of technology and creativity, where diverse teams worked tirelessly to solve real-world problems and create innovative demonstrations in embodied AI and open-source robotics.
https://huggingface.co/spaces/LeRobot-worldwide-hackathon/worldwide-map
I participated in the Eindhoven edition, hosted by NLRobotics at the High Tech Campus. Our group of approximately 15 participants brought a diverse range of skills and experiences. The hackathon focused on the open-source SO100 and SO101 robot arms from The Robot Studio. And we ended up with more robots than participants (a few of which were provided by yours truly; I even ended up selling a pair at the event), ensuring ample resources for experimentation and innovation. In addition to the SO100/SO101s, we also had access to bigger robot arms - an AgileX PiPER and a UFactory xArm 6, a Unitree dog, and a Unitree humanoid. None of us used the Unitree robots, so it ended in the hands of a bunch of kids (kids of the host and participants), who seemed to have a great time driving them around.
https://twitter.com/AdilZtn/status/1933818070017220691
Day 1
We launched into the hackathon by tuning into HuggingFace’s live stream, which laid out the various tracks we could dive into. After that, we jumped into a quick brainstorming session to spark ideas for what we could create over the next couple of days. We collaborated for a few hours to get everyone’s robots up and running, reaching the point where we could teleoperate the follower arm using the leader arm. With that foundation set, we split into our own teams and dove headfirst into hacking.
The Project
I had partnered with a recent TU Delft graduate whom I had met on LeRobot’s Discord server, and we decided to focus on fine-tuning SmolVLA (an open-source VLA model trained extensively on SO100 and SO101 datasets collected by the LeRobot community) for our application: stacking 3D-printed cubes on top of each other. Luckily, we already found a stacking dataset collected by LeRobot to demonstrate the SmolVLA model, and we wanted to train some more episodes ourselves, combine datasets, and see what the model could do. Things didn’t go according to plan as you’ll soon see.
https://twitter.com/kamathsblog/status/1933965312912957622
The first day went swimmingly, as we were able to get our teleoperation setup running with one wrist camera and one external depth camera (which we were using only for RGB images) and were able to record two different datasets with 25 episodes each, where we stacked 3D printed cubes on top of each other. We left for home that evening, intending to train LeRobot’s SmolVLA model during the night. This is when things started going wrong. While we did get $20 in GPU credits on HuggingFace, their hub went down that evening, clearly due to the spike in usage from the hackathon. I decided to pay for Google Colab Pro to access an NVidia A100 GPU for training the model, but that ended up crashing in the middle of the night as well. Even my teammate’s attempts were unsuccessful.
Meanwhile, I decided to try something else - from the start of the hackathon, I wanted to build something for the hardware improvements track, since I had an inkling that ~36 hours was too less for training a model reliably. I had a couple of OAK-D cameras from Luxonis (which use the DepthAI framework) and wanted to integrate them into LeRobot. The LeRobot framework already provides support for OpenCV and RealSense cameras, so it made sense to add DepthAI as a new camera type. So, armed with my experience with OAK-D and DepthAI, and my handy Cursor Pro subscription, I got to work sometime after midnight on Sunday.
https://twitter.com/kamathsblog/status/1934035624949911928
The application I ended up building adds a new camera class to LeRobot, which can be used by simply using the camera type depthai instead of opencv or realsense from any of the LeRobot scripts. Additionally, using existing arguments, the depth image can be enabled and displayed using Rerun. Finally, when using the dataset recording script (record.py), my addition now also enables depth images to be recorded in a dataset, and can also be uploaded to HuggingFace. Earlier that day, my teammate and I realized that we were registered as separate teams, so we had two submissions instead of one. I decided to make this DepthAI integration my submission, and let my teammate submit the stacking demo as his.
https://twitter.com/kamathsblog/status/1934747239194243366
If you want to try it out yourself, my fork of the LeRobot repository has all the relevant code. I need to create a PR for this after some cleanup Looks like someone already created a PR with DepthAI integration, but without the ability to record depth images into the datasets, which I hope to add when I get the chance. You can also find the dataset with the depth stream (alongside the RGB streams from the OAK-D and the wrist camera) here:
Day 2
Day 2 started after about 4 hours of sleep, mostly by going through all the Discord messages from the teams that had worked through the night, one team working from the venue itself. While our team’s training sessions had crashed overnight, we now had some very clear documentation on how to use our HuggingFace credits. So, we decided to do that and try to train SmolVLA with our 25-episode dataset.
The Side Quest
While my teammate tried to figure that out, I cleaned up my DepthAI integration code (by fixing the dataset uploading feature, for example), and decided to play around with Mujoco. Mujoco, or MuJoCo (which stands for Multi-Joint dynamics with Contact), is an open-source physics engine. I was working with the SO100 and SO101 URDFs a few weeks ago and realized that the SO101 folder had some XML files, which I then realized were for Mujoco. I tried them with the Mujoco viewer app and already had the idea of controlling a simulated SO101 follower with a real leader robot. And I had the free time to do it now rather than leave it for another weekend.
My idea was to essentially duplicate the teleoperate.py script, and instead of sending the leader joint states to the follower arm, we send it to the simulated follower using the Mujoco Python API and with the help of Cursor. And it worked exactly like I wanted it to! You can try it out yourself here; I also made a PR to the LeRobot repository.
https://twitter.com/kamathsblog/status/1934224881693913176
Since my tweet has blown up (well, in comparison to my regular posts), I must make some things clear, especially about the use of AI in robotics. I mention that it took less than 30 minutes - this only includes the time it took me to get AI to write code. I had already planned the idea weeks ago, and I also spent nearly an hour before and after preparing some stuff for AI to work on, and then reviewing the code. Both of these steps (planning, providing context, and reviewing) are very important when it comes to AI-generated code, especially when it comes to robotics. Now, in my case, the robot being controlled was in simulation, but if it were a real robot, the risk would have been higher. If you are new to robotics or hardware in general, I’d recommend reading this article about safety in robotics from Six Degrees of Robotics.
https://sixdegreesofrobotics.substack.com/p/robot-safety-for-not-dummies
Other Submissions
Besides our Depthai Integration and Stacking using SmolVLA submissions, our local hackathon ended up with three more submissions. One team implemented several home-based scenarios using ACT, and one team trained ACT to place a roll of tape in a holder. Unfortunately, I don’t know their team numbers, so I cannot find their videos on the LeRobot Hackathon submissions space on HuggingFace. But here’s the video of the last team, which they shared on Twitter. They used the Gemini Live API and a web portal to interact with a SO100 and get it (or at least try) to pick up objects. It was hilarious to listen to their sassy robot talk back to them while we worked on our submissions at the next table.
https://twitter.com/Rens_van_Dijk/status/1934361496928297207
As for the other submissions from the rest of the global participants, please check out the LeRobot Global Hackathon page on HuggingFace, where you can find the list of the winners and their demos (top 30 teams), a list of datasets recorded for and during the hackathon, and models that some of the teams also uploaded to the page.
Final Thoughts
The hackathon was certainly a celebration, but a lot of it was happening online. The Discord was buzzing constantly with questions (and answers), tips, memes, and photos/videos of successes and failures, and both the community and the organizers were actively involved. The Eindhoven event was a bit smaller than I would have hoped for. It was very well organized, the space was well equipped, and the atmosphere was enthusiastic and collaborative. The weather was great too! However, looking at some of the other events around the world, I would have liked to see local robotics companies and the university involved. Hopefully next time..
I had a great time, though. I made some new friends, learned some new things, and returned inspired with new ideas. I’ll be honest, after going through all the submissions, I was more impressed by the hardware improvements people have made to the platform than by the AI implementations. One team (and I believe the winning team) implemented a really cool rail system, while another designed a way to connect multiple arms in a modular, centipede-like creature. And then there were tons of gripper improvements. From a chopstick holder to a suction cup. I am sure someone has made a parallel gripper, so that I can replace the default configuration (with one fixed and one moving jaw) on my SO100. I am not sure if there were other teams besides me that added a new sensor, but I haven’t gone through every submission yet. I hope someone added Lidar support. And ROS too. I’m yet to see a complete ROS implementation with ros2_control, MoveIt, and Gazebo, and I hope someone was able to work on this during the hackathon.
As for the AI part, I wish we had gone with the ACT instead of SmolVLA. It seems easier to train, and it worked well for two teams at the venue and many others globally, but SmolVLA did not work well for us. The submissions relating to imitation/reinforcement learning were quite impressive, with applications in all sorts of scenarios. I was just a little more excited about all the hardware improvements. But I also cannot wait to see how HuggingFace / LeRobot and their community make use of all the data that was collected during this event. Surely the best AI stuff is yet to come…
What’s Next
I have some plans for the SO100/SO101 arms that I have. I have one full SO-101 follower and one SO-100 follower with no gripper. Since I do not have an extra motor available, I would like to replace the gripper with a non-actuated end-effector. I am thinking of adding a depth camera and fixed attachments like a spoon or a fork. Or maybe I can simply add a suction cup. I still need to decide on this. Luckily, the hackathon demos are a treasure chest of ideas.
On the software side, I want to integrate ROS into LeRobot. I’ve seen some examples where people have attempted to convert the entire LeRobot framework to ROS, which I think is a bad idea. I want to keep it simple and focus on creating an interface where hardware-related data (motor joint states, camera images/depth) from LeRobot can be converted to ROS messages and vice versa. This allows the LeRobot framework to work independently, and also with ROS packages and tools if necessary.
Meanwhile, for the coming weekend, I’ve got plans to try out Mat Sadowski’s amazing work on visualizing LeRobot using Foxglove. To be honest, I’ve been following his work on LinkedIn since before the hackathon, and it’s what inspired me to work with Mujoco in the first place.
My goal eventually is to work with AI for high-level thinking, like the Gemini demo I shared above. I also want to use AI to do tasks like identifying objects to interact with and calculating their 3D poses, rather than controlling the entire robot. I think classic control systems are more than enough (and probably more reliable) to control the movement of the robot as compared to an end-to-end AI black box. And since using ROS packages saves me a lot of time in this case, creating that ROS interface is quite crucial for me.
Speaking of Gemini, Google just launched their Gemini Robotics On-Device model, which can run on local devices (I read somewhere that it was tested with a Raspberry Pi 4, but I cannot find a reference; it’s amazing if it is true). I haven’t tried it out or read a lot about it, but I’m keen to see how it is going to be used by the community and to try it out myself at some point in time.
https://deepmind.google/discover/blog/gemini-robotics-on-device-brings-ai-to-local-robotic-devices/
I also realized that while I’ll be visiting Singapore in a few months for ROSCon, I haven’t actually used ROS since the last ROSCon. So, I might as well refamiliarize myself with ROS and its newest features (like the new Kilted Kaiju release and all the cool new things it offers) before my trip. From last year’s experience, I’ve started enjoying including ROSCon as a part of my holiday, so I’m glad that I’m doing it again this year. I’ve already booked an 8-day trip to Singapore and Malaysia and am slowly planning my itinerary. I cannot wait! If you are also attending ROSCon, or are around Singapore or Kuala Lumpur in the last week of October, let me know!
