Making/breaking

Out in the Cold: Recalcitrant Robots

Designing, developing, and maintaining hardware presents unique challenges. At Dusty Robotics, we develop robot-powered tools for the modern construction workforce. Here's a story of fixing finicky robots.

A Dusty Robotics FieldPrinter.

In December 2021, we started getting reports from the field about unpredictable behavior when our robots were turned on. Our engineering and customer success team investigated all such reports to try to discover the root cause. Every time a report came in from the field, we'd immediately troubleshoot, bringing the robot back in-house to try to replicate the failure.

Dusty Robotics FieldPrinters on a job site.

We couldn't reproduce it, but we did come up with a theory for why it was happening. When the coupler that connected the motor to the wheel came loose, the wheel wouldn't turn when it should and the robot would drive unpredictably.

The coupler that connects the motor to the wheel.

We redesigned the coupler system so that it wouldn't slip under normal operation, and rolled out the fix to all the robots. With many in customers' hands, it took a while to cycle through them all. We also developed a field fix that involved taking the robot apart and tightening the coupler. The next time a customer called in with this problem, we had them apply the fix. And it worked! Problem solved.

Fast-forward to November 2022. We started getting reports again of robots behaving erratically in the field. Again, we brought the robots back in-house, and again we couldn't reproduce the behavior. It must be a new problem, we thought. After all, we had fixed the couplers and hadn't had any coupler-related issues for the past nine months. Our team started systematically debugging all other possible causes. We assigned an intern to try to reproduce the problem. Since it seemed to happen first thing in the morning, maybe it was related to the power-up sequence. The intern turned a robot off and on hundreds of times, but it never showed the problem.

The kitchen became a morgue, with robot cadavers spread out on operating tables.

Then someone had the brilliant idea to put a robot in the fridge. We pulled it out the next morning, and… it exhibited the same problem. For 10 minutes. Then it stopped. Could the problem have something to do with temperature?

Ten minutes didn't give us a lot of time to debug before the problem went away. So we filled the fridge with robots and took them out one by one to experiment on. The kitchen became a morgue, with robot cadavers spread out on operating tables. Some of the experiments involved measuring what was happening inside the bot while it was cold. At one point I found an oscilloscope outside the fridge, measuring the vitals of the patient inside.

To debug the problem, several robots were placed in the fridge.

Finally, we discovered the problem. One of the off-the-shelf components we use behaved out of spec at certain temperatures, generating a noisy signal. We reverse engineered the component and found that removing two resistors fixed it. Problem solved, for real this time.

It turns out that last year's coupler problems had the same root cause. While people were opening up the robot and tightening the coupler, the robot warmed up. By the time the robot was reassembled, the problem had gone away. It had nothing to do with couplers.

Monitoring the robots' vital signs while in the fridge.

By the time we had rolled out the coupler "fix" to all robots, the weather had warmed up enough across the country that the issue didn't reoccur. We thought we had fixed it, when actually spring had fixed it.

There are hundreds of components in the simplest robot, and each one can have unpredictable failure modes like this one—that's why hardware is hard. Kudos to Team Dusty for solving hard problems like this one.

  Author

Tessa Lau is the CEO of Dusty Robotics. Her passion is creating businesses around technology that gives people superpowers. A generalist who also enjoys diving deep to solve thorny systems issues, Lau has expertise in robotics, computer science, AI, machine learning, and data analytics. She has a Ph.D. in computer science from the University of Washington. [email protected]

Copyright held by author

The Digital Library is published by the Association for Computing Machinery. Copyright © 2023 ACM, Inc.