Driving a robot with one forearm

(AJ) Hi, I'm AJ, and with me are Jose, Kiran and Xinyi. In the next five minutes we'll show you how we drive the Jackal with one forearm and a Myo armband, what the muscle and motion signals taught us, and the results from the sim. Here's the plan.

Five parts. How a driver controls the robot. What the signals told us, which is why our design looks the way it does. How a new driver calibrates, and what keeps the robot safe. What broke while we tested. And the results. First, what we built.

Here's what we built. One forearm and the Myo armband drive all three courses. Over 51 official runs we finished every single one, with no missed gates and no collisions on the clean courses. Why should you care? A new driver records their own calibration in about two minutes, and it kept working under the judges' slow, lossy link. So how do you drive it?

The arm is the joystick. Raise your forearm and the robot stops. Lower it to level and it drives, and the lower you go toward level, the faster. Tip it down past level and it backs up. Roll your forearm like a doorknob to steer. And if something goes wrong, make a firm fist and hold it for under half a second: that's the brake. Your hand stays relaxed the whole time, so it isn't tiring. Why not just use muscle gestures? We tried.

We started with five muscle gestures and a classifier. On still poses it was right 99% of the time, on takes it had never seen. But hold a fist with your forearm rolled, which is exactly what you do while steering, and it read "forward" only 38% of the time. The rest came out as right, reverse or left. Rolling your arm moves the muscles under the electrodes. So we kept the muscle signal for the brake only, and steer with the band's tilt sensor, which rolling can't fool.

(AJ) We also asked whether deep learning would read the muscles better. We trained a temporal convolutional network on the raw signal, with gain and noise augmentation, and tested it on takes and drivers it never saw. On each person's own two-and-a-half-minute calibration, the simple linear model still won: 85% of poses against 68%. On a driver it had never seen, both were near 32%, against seventeen percent for guessing. With this little data per person, the simple model is the right choice, so the controller keeps it for the brake.

Every driver records their own calibration: six poses, each held for two, three and five seconds, about 2 minutes in all, with prompts. At the start of each drive they hold their arm straight for two seconds, so straight is straight for them. Then they raise the arm to arm it. Underneath, the robot stops whenever it's unsure: if the signal drops, if the arm leaves its calibrated range, or if the band has moved, in which case it refuses to start and asks for a new calibration.

(Kiran) To get past one driver, we made setup a single command. It connects the Myo, records your calibration, opens the sim and drives. Four of us recorded, and the data is public. The bars show how much stronger each person's fist is than their relaxed hand, which is what the brake detects. My first recording came out below one: the fist didn't register, so I recorded again. Now the tool shows that check after every calibration.

(Xinyi) I added a gate assist. The lidar finds the two posts of the next gate. If you are already aimed within twenty degrees of it, it adds a small correction toward the centre, never more than 0.3 radians per second, so you stay in control. In 6 test approaches from off the line, it got through the first gate 2 times against 0 without it, with no collisions either way. When the approach wasn't aimed at the gate, it did nothing.

(Jose) Four things broke. First, the brake fired on its own 38% of the time, because just holding your arm out uses the same forearm muscles. Now it needs a fist pattern held for 0.4 seconds, and it dropped to zero. Second, right turns were weak: with the arm out you only roll about thirty degrees right, so we tune each side from the driver's own logs. Third, after a battery recharge the band went back on differently and the robot reversed 58% of the time. Now it checks the pose at every start and refuses if the band moved. And today the Myo vanished: once unplugged it fell asleep and no laptop could find it. Wearing it and moving while the tool keeps scanning brought it back, and that's in our troubleshooting guide.

Here are the official scores, the medians over repeated runs so it isn't one lucky lap: 20.7 seconds on L1, 26.6 on the slalom, 16.3 on the corridor and dock, with no collisions. The last bar is L1 under the judges' degraded link, 300 milliseconds of lag and ten percent packet loss: about 21.7 seconds, barely slower than clean. That's the robustness we were after. Speaking of lag, we tested one more idea.

With 300 milliseconds of lag, drivers overcorrect, so we added lag compensation: it measures the delay, softens steering, and checks obstacles from where the robot will be when each command lands. To test it fairly we ran it blind: 8 sessions, the controller secretly switching it on or off, so the driver couldn't know. With it on, the medians were a little better and there were no collisions, against 2 without. But with this few runs, the difference isn't significant. We kept it on because it did no harm.

(AJ) What we'd do next. Four drivers isn't enough for a neural network, so we'd pretrain on a public Myo dataset of thirty-six people, which has the relaxed hand and the fist the brake needs, then fine-tune on each calibration. The lag result needs more sessions before we'd claim it helps. And we capped speed at one metre per second while tuning, half what the sim allows, so going faster means the obstacle assist has to look further ahead first.

That's our controller: one forearm, no squeezing, every course finished. The code and the dataset are public on GitHub. Thank you, and we're happy to take questions.

Backup slides for questions.

Backup. The blind test in full: medians of each session's two runs, compared with an exact two-sided permutation test.

Backup. What each signal does: the tilt sensor gives speed, steering and reverse; the muscle signal only gives the brake.