Broke my arm - a robotics story

Cartesian motion,
Many maps and many paths.
Joints still take the wheel

Even about a year out after grad school, it's hard to not think about manipulation in the context of robotics. It's especially harder when it's your arm that's busted and you have to deal with it on a day-to-day basis. Over the last 3 months, I've slowly transitioned from having a totally useless left (secondary) arm to a somewhat decently-capable left arm. For me at least, that's led to a lot of interesting insights into the strengths and limitations of dexterous, single-handed manipulation vs simple, bi-manual manipulation, as well as some of the biomechanical contributors to hand functionality.

The many stages of recovery####

My arm went through a couple of notable phases during recovery:

1. Totally useless - I couldn't even move it without pain.
2. Non-prehensile stub - With a cast post-surgery, I could sort of poke at stuff with the end of my cast.
3. Multiple precision non-prehensile stubs - At some point I could move my index and middle fingers. Shoe-tying unlocked!
4. Weak pinch-grasping - Even in a cast, I eventually recovered some thumb mobility, allowing me to lightly pinch small objects. A lot of thumb numbness would still give me trouble for several more weeks.
5. Weak pinch-grasping with a gimpy wrist - After getting the cast off, I gained some extra degrees of motion, but my wrist had next to no mobility or strength.
6. Weak power-grasping with a gimpy wrist - My hand definitely recovered faster than my wrist, and I often resorted to wearing a wrist brace whenever I needed to get shit done at work.
7. Limited wrist and a numb thumb - Full sensation in my thumb and range of motion in my wrist were the last to return.
8. Not-so-weak hand on a limited wrist - And here we are, possibly as good as it gets. I'm told at rehab that there's not much more can be done. I just have to work at improving my grip strength and regaining my last bit of wrist rotation.

1. Robots are totally handicapped, at 1####

Puns always intended

I would argue that the bulk of the current state of the art in robotics manipulation research is done on systems with a single manipulator. For most practical intents and purposes, controlling the pose of the manipulator itself in all six degrees of freedom is pretty much a solved problem under quasi-static assumptions. The focus of my research and that of many others (those poor souls) was to figure out ways to make the end-effector more effective (I don't like the word dexterous). The golden standard was always the human hand, and while I don't think I was at all close to succeeding, I'm not sure it would've mattered if I or anyone else had, because it totally sucks to just have a single manipulator, no matter how useful the end-effector.

Even with a single fully-functioning arm and hand remaining, a lot of manipulation tasks became much more haphazard and random, if not impossible. Aside from having my payload capability cut in half, I had trouble bracing or even partially constraining objects if my primary arm was already in use. We're not always interacting with rigid, fixed features in the environment, or even single-piece objects that are entirely rigid, for that matter. With just a single hand, I felt limited to either simply grasping small items or manipulating parts of larger assemblies that could resist the forces I was imparting on them. Even if I was using a tool, I might have been able to grasp it securely, but with respect to whatever I was using the tool for, I was still dependent on either a sufficiently large mass or environmental constraints for aid. For example, scrubbing my sink didn't change, scrubbing a cast-iron pot was fairly straightforward, scrubbing a bowl was problematic, and forget about cleaning a glass properly.

A secondary manipulator and end-effector is more than an optional aid. For all sorts of manipulation tasks, I need to secure some object in a variety of ways, not just one. In cooking prep, if I'm cutting vegetables or meat, I can't just apply knife to food. A pan of stir-fry doesn't obediently sit on the stove when you're mixing in ingredients, and surprisingly, when you twist only the lid, the jar moves with it. Who would've thunk it. Also, it's not just about securing the object in some arbitrary way. Certain parts of the object geometry may need to remain free and exposed, and in some cases, it may even be beneficial to give the object some limited degree of freedom. With just one manipulator, all of this becomes far more difficult, and for all the oft-cited advantages of human hand dexterity, it didn't exactly help me make up for the lack of a second arm, half-functioning or otherwise.

Bags are the worst. They should be considered robotic research kryptonite from here on out. They're compliant and floppy, so you need distributed points of contact to constrain it, but they're also not as heavy as towels, so they don't even reconfigure reliably with respect to gravity. The material's also often slick and difficult to even pinch. It seems
like the best you can typically do is secure the bag at a few selective locations and use them as control points. One manipulator means one control point, which means I often found myself just uselessly waving bags around, not knowing what quite to do next.

To me, this really highlights again just how far behind the robotics research is relative to human capability. Labs are still struggling with reliably securing a grasp on objects. There's hardly any work done on cooperative manipulation, not even exchanging objects between grasps. By the end of my PhD, it felt like the robotics community hadn't even started scratching the surface of even re-grasping or even optimally securing (fully or selectively) a tool for a given task. Yet, here I am, with at least one fully functioning hand, arguably more capable than any robotic hand currently in development, struggling to execute the day-to-day tasks I was used to doing.

That's not to say there weren't alternative strategies to compensate. For a smaller object, I could use my pinky and ring finger to at least partially secure it against my palm and then manipulating it with my index and middle fingers. Actually, when certain elements were in close proximity, I could sometimes use my single hand as two independent pinching-pairs. This gives a bit more credibility to end-effectors with redundant jaws/fingers that can form alternative or multiple force closures. In addition, as much as I hate to admit it, I guess it also makes some recent projects on wearable, supernumerary robotic limbs all the more reasonable. As it would become more apparent to me later on, nonprehensile manipulation plus a reliable surface equals pretty sufficient closure conditions.

Clamping stuff in my mouth or between my thighs or under my chin also was commonplace in the weeks immediately before and after surgery. Given the mobility of the human arm, it seemed that many tasks requiring bimanual manipulation doesn't necessarily require two hands and two arms. A single arm/hand combo along with an offset fixturing device of some sort could be just as good. Relative motion between two end-effectors doesn't mean that both have to move. Even for a task like scrubbing a plate or pan, the sponge or brush could be the one statically mounted, allowing for more flexibility in how the dish is handled.

2. Non-prehensile: Where to hinder progress is progress itself####

Even with my fingers mostly immobile post-surgery, it was a huge boon to have some use of my left arm again. I could use it to either help support heavier objects or partially constrain objects on a table, mostly to prevent them from sliding around. Personally, I don't like the research fascination with force or form closure. They make for nice theoretical statements on how objects can be fully constrained, but I think they also don't apply to the vast majority of basic manipulation strategies used in day-to-day tasks.

I liken manipulation to applying some set of elastic bands or bounds on an object's pose. This could be through direct contact or indirect contact through a tool. Strategies can leverage environmental features or not, and objects may only be partially constrained, easily dislodged and perturbed by external forces in certain directions. I don't think it's reasonable to insist on only a complete set of rigid constraints fixturing an object in place. There should be an expectation of mobility in certain degrees of freedom for any planned manipulation primitive.

That said, an arm in a cast serves effectively as a mobile, unidirectional constraint that I could move relative to my good arm/hand so that I could apply forces in directions I couldn't as easily previously. I could also pin objects against a table surface, either securely by taking advantage of friction or selectively to allow for a degree of rotation.

3. Precision non-prehensile: Boldly going where no cast has gone before####

It took a few days post-surgery for me to make contact with my fingers without too much pain. Turns out, when you dislocate and fully break the radius and ulna bones, all the muscles and soft tissue, which were previously held in tension by these bones, settle in different ways, and surgery has to re-tension them properly, leading to a lot of tearing and re-tearing. Even now, close to 4 months post-injury, I can still feel scar tissue and stiffness in my forearm.

Even immobilized fingers gave me some improved capabilities in terms of handling smaller items. The precision of my arm motion didn't change, but force application still depends on the end-effector geometry, so I could localize constraint forces better through my fingers than my forearm as a whole. Having direct tactile feedback was helpful as well, since I now had a better sense of an object's mobility before I interacted with it, instead of waiting to see how the object behaved after the fact.

4. Somehow more useless than Zoidberg

I got abduction and adduction of my fingers back before any meaningful flexion capability, so I could effectively use my index and middle fingers like a pair of floppy chopsticks. Maybe the muscles for such lateral grasps are more localized to the hand and not as connected to the more comprehensive network of muscles in the forearm. This was what first enabled me to tie my shoelaces post-surgery. I suppose it would've been possible to pin the shoelace against my shoe in some way, but it definitely helped to be able to maneuver two shoelaces independently relative to the shoe. I guess in some cases, bimanual manipulation requires coordinated motion relative to some preferable object pose, as opposed to just relative motion between end-effectors. Despite the ability to sort of pinch smaller items and hold them, I would have preferred better flexion to hook at least one of my fingers around objects. Maybe that speaks more to how much more natural flexion feels in comparison to lateral grasps.

5. It's all in the wrist####

Two weeks post-surgery, I got my cast off and was told to use and move my wrist as much as possible to encourage recovery. Instead, I promptly went out and bought a wrist brace because of how painful it was. It's pretty incredible to think about how the wrist facilitates so many muscles crossing between the forearm and hand while also being capable of so much motion. It's also pretty incredible how I've managed to severely mess up both my wrists while leading a ridiculously mundane life (but I digress).

Not many robotic manipulators implement a wrist like the human wrist in such a compact manner. While many are capable of pitch/roll/yaw about a common center, the mechanism itself can be fairly bulky to allow for direct drive instead of a more complicated transmission. Many manipulators, especially those on EOD platforms, make do with a limited or static wrist. The wrist also seems to be a common limiting factor to a manipulator's maximum payload capability, and to be fair, when tasks are as repetitive and structured as they are in industry, perhaps a wrist isn't always necessary.

In all my research and review, I've yet to find a single commercially-available arm that can transmit forces/torques from the rest of the system through the wrist to the hand. Restricted to space-limited modular add-ons, it seems that conventional manipulator systems, especially mobile ones, will continue to sorely lack hand capability unless either someone invents magical actuators or wrist design is improved. When I first got my cast removed, I noticed that I could feel myself moving my fingers all the way to past my elbows. Sure, the intricacy of an anthropomorphic muscle network would greatly complicate control, as current robotic arms and hands generally assign an independent actuator per degree of freedom, but it seems that the compactness of the human hand is an overlooked requirement for dexterity in support of physical mobility and controllable degrees of freedom.

Adjustable, projected centers of rotation in parallel mechanisms is interesting to me for the reasons above. By offsetting the actuated mechanisms away from the center of rotation, you should get some power gains by coupling or using larger actuators. It probably seems weird to incur costs in both complexity and repeatability (due to complexity) for compactness, but maybe we should stop considering manipulator performance in a generic vacuum, irrespective of the actual use case. Maybe a non-anthropomorphic design approach is the right way to get closer to anthropomorphic capability.

6. Oh look, someone's trying to compensate for something

Due to wrist weakness and pain, I often resorted to using a wrist brace whenever I would be assembling anything at work or doing anything else remotely physical. I mostly wanted to keep my wrist in its neutral and passive pose as much as possible. With my elbows by my side, this would leave my thumbs pointed upwards, and it turned out to be a rather awkward pose for most tasks. Luckily (or unluckily as I would find out later), I could generally use my shoulder and elbow to re-orient my hand as necessary for most tasks.

Wrist restrictions were most notable whenever I'd type. As much of a keyboard enthusiast as I am, I used to chuckle at designs that tilted keyboards up at a near 90 degree angle, but maybe there's something after all to that ergonomic idea. To tilt my hand appropriately for a standard keyboard, I had to swing my elbow outwards like a chicken wing. In doing this over the course of a week or so, I of course ended up injuring my shoulder, which stay sore for the next month. Typical. I eventually caved and printed a tilted stand for my split keyboard so that I could type with my wrist in a more natural pose.

Having to move my elbow so much not only made motions more tiring, but also meant that I needed clearance around my arm for all that extra motion. Despite having all my fingers free, some constrained motions weren't possible, since I physically couldn't get my hand in the right orientation. Sometimes, using my fingers' lateral surfaces make certain tasks manageable, though they certainly felt more challenging and unnatural. I don't know if this is due to unfamiliarity in moving my fingers in that manner, or an actual functional deficiency. This sort of flies in the face of my proposal above to consider actuating mechanisms distal to the point of rotation and interaction.

7. Again, with feeling (please?)

I was expecting poor wrist mobility and weak grip strength from my last wrist surgery, but this time, my thumb was also numb for about a full month. I regained full sensation in the back of my thumb pretty quickly, but I'd get a tingling sensation on the front of my thumb whenever I'd try to grab something, almost like I had been lying on it, and blood was just beginning to flow back through. Even with full thumb mobility, I had a lot of trouble with pinch/precision grasps on smaller objects, which would often slip out of my hands.

I'm not sure if this was due to not knowing how to modulate grasp forces due to the lack of sensation, or some involuntary physical reflex in response to the unfamiliar tingling feeling. It's interesting to me just how much the thumb is used for tactile exploration and localization. My other fingers should (presumably) feel just fine, but I felt dependent on using my thumb as I was accustomed. Maybe this is due to how the thumb is used as a bracing surface, or the relatively larger surface area used to contact objects relative to other fingers. Or maybe I would've had the same difficulty if my thumb was perfectly fine, and it was one of my other fingers lacking sensation.

Assuming 2 primary contact points/pads, is it necessary to have tactile feedback on both surfaces? For any 2 opposing fingers in a precision grasp, how much redundancy is there in functionality? Alternatively, how good are we as humans in controlling one appendage relative to the sensations in another? If I were to numb feeling in a finger altogether, could I move it appropriately with respect to the reaction force felt on an opposing finger? In robotics, I guess we have the flexibility to freely dictate how input from sensors dictate/influence output, but I now wonder how much low-level coupling there is our hand functionality due to either physiology or just familiarity from daily use.

8. Back to the boundaries of normal

Nowadays, in getting back my remaining range of motion and grip strength, I'm starting to figure out what tasks require a full range of motion. Up until last week, I still couldn't rotate my hand such that my palm pointed straight up, so I had a lot of difficulty cupping my left hand and using it to hold or cradle objects. I discovered that limitation for the first time when I tried pouring some bolts/fasteners into my left hand and instead watched them slide off my hand and fall onto the ground. That was a sad day. I experience a similar limitation whenever I'm handling pots/pans and pouring things out of them. It's tough rotating the handle while supporting the changing weight of whatever's in the pot. While my wrist is perfectly fine holding the load in its neutral position, I start struggling as I approach the limits of my wrist's current workspace.

According to the measurements taken at my last rehab appointment, my supination/pronation range of motion is only off by about 5 degrees relative to normal, and yet I'm still very aware of my lack of mobility on a daily basis. It's interesting how much we use the full extent of our wrist mobility in daily tasks, but I wonder if we're leveraging our physical limitations like a mechanism would utilize a hardstop, or if we'd actually like to utilize an extended range of motion and simply can't. Continuous rotation and a relatively consistent force/torque output are a few advantages that a robotic mechanism have over human joints, but I think it's rare that we ever see designs explicitly leverage that extra capability in improving upon anthropomorphism.

tldr; Round doorknobs defeat both killer robots and velociraptors