National Robotics Initiative Project

Overview

The use of robotics and automation as assistive technology is becoming increasingly prevalent in applications ranging from industrial manufacturing to healthcare. In the particular case where physical contact is required between a human operator and a haptic device (e.g. force feedback haptic devices), a coupled dynamic system is established. Such a bilateral interaction can lead to instability, with serious potential consequences (safety, performance), specifically during contact with stiff environments (e.g. operating a powered drill) or the presence of a time delay. The operator will often attempt to control an encountered oscillation by stiffening their arm, leading to a stiffer system with greater instability. Robot co-worker controllers do not typically account for this issue in their current designs. The goals of the current project are to study the association between neuromuscular adaptations and system performance limits, develop methods to classify and predict operator physical and cognitive states based on physiological measures, and demonstrate the efficacy of the methods for various human-robot collaborative tasks.

Human Arm Stiffness & EMG-based adaptive haptic control

Preliminary Concept.

The level of endpoint stiffness is not directly measurable in typical haptic control situations. Previous estimation methods have either employed offline techniques or have shown a significant amount of error and variance in the online cases. On the operator side, human arm stiffness is changed through modulation of co-activity across muscles crossing joints. Electromyogram (EMG) signals have frequently been used to record muscle activity. Previous studies have shown strong association between muscle co-contraction and corresponding joint stiffness/impedance. A feasibility study was performed in collaboration with General Motors to develop a one-degree-of-freedom robotic device. Using gain scheduling approach based on classifying operator arm stiffness from EMG signals, the study showed improved stability and performance (25% reduction in average time) for 10 healthy participants performing pick-and-place tasks. The choice of muscle pairs is validated using an integrated model that can characterize the coupling between the human musculoskeletal system and robot dynamics. However, large variability and randomness have been observed in muscle stiffness, even in a single subject. In addition, due to the bilateral nature of the human-robot interaction, EMG signals should not be used as “reference” commands. Achieving increased stability and performance requires treatment of operator’s dynamics as a stochastic system with random varying stiffness parameter. Also, given the observed change in operator’s reaction as a result of change of stiffness compensation state, a “proactive”, rather than “reactive” gain-scheduling approach is preferred. The current project addresses this issue through probabilistic human state estimation.

Intent Inference through EMG and Eye Movement Measurements.

Hidden Markov Model.

EMG measurement may provide limited input, because they are contemporary with the human limb movement. Indications of state transitions that precede physical motion would improve adaptive capability in a bilateral human-robot interaction. Eye movement has shown promise in characterizing state transitions that precede limb motion. For instance, information required to perform a movement is sought by the human eye a second before it occurs. Hence, incorporating metrics such as gaze, muscle activity and end-point force would indicate how a person is attempting to move a haptic robot. Since intent is not directly measurable or observable, a Bayesian approach (e.g. Hidden Markov Models or HMMs) that estimates intent based on measurements known to correlate with intention is preferred. Deterministic classifiers (e.g. Support Vector Machines) are also being explored.

Eye Tracking

Eye Tracking.

Eye-tracking studies have revealed that saccades are restricted to objects involved in an activity, and that the eyes operate serially with a duration equivalent to the duration of the activity. When studying changes in human locomotion, directional changes were preceded by an eye saccade in the direction of the new destination. Because different types of tasks require different visual information, it may be possible to pinpoint not only the onset of a state transition, but also the state that is likely being transitioned to using the pattern of fixations.

Current activity recognition approaches tend to focus on what activity is happening now, and are not generally predictive gaze-based HRI does not incorporate gaze “intelligently”
– Does not use saccade/fixation classification
– Does not utilize its predictive power for manipulation tasks
Our research objective here is to develop an algorithm which uses gaze data, in conjunction with scene data, to dynamically update a haptic controller by predicting the intentions of the human operator. There exists a fairly well understood relationship between gaze and hand coordination in slow-paced pick-and-place style tasks with advanced off-the-shelf hardware solutions. In addition, useful gaze features include:
– Look-ahead fixations (>200 ms), which occur on targets 1-3 seconds before onset of motion
– Look-ahead saccades (<100 ms), which occur on targets 0.5-1 second before onset of motion

Adaptability of Human Stiffness Control with Neuromuscular Practice.

Recent EMG studies have shown that neural activation of agonist and antagonist muscles is characterized by distinct oscillatory components during co-activation (simultaneous activation of muscles with opposite functions): in-phase and out-of-phase components.  The in-phase components between agonist and antagonist muscles appear to be the extension of the central descending “common drive” that has been demonstrated between synergist muscles.  The out-of-phase component appears to be produced through the neural mechanisms including reciprocal Ia inhibition.  Since simultaneous activation of agonist and antagonist muscles produces opposing force/torque about a joint, the in-phase components should increase joint stiffness while the out-of-phase component would destabilize position control in the stiffened human-robot dynamics during co-activation.  The frequency of position fluctuations in human-robot interaction usually falls into the range of the out-of-phase component.

It is expected that attenuation of the out-of-phase component during co-activation due to motor practice reduces muscle force oscillations and thereby minimizes the destabilization of the human-robot dynamics with increased stiffness.  A motor practice is performed for minimizing out-of-phase frequency components in EMG.

We investigated the modulations of neural strategy that accompany the Improvement of co-contraction control due to practice. Using visual feedback of Biceps and Triceps EMG signals to improve co-contraction. Three groups, 20 subjects each, were tested before and after different intervention practices.

It is a common practice in the neuro-muscular community to pre-process and full wave rectify EMG signal, however, one need to be careful how it affects signal. Above showing coherence between Biceps and Triceps muscles during co-contraction that is washed away through rectification.

Grand average signals during co-contraction test (left column = 2 target pairs) and co-contraction practice (right column) averaged across subjects group. Top rows show time series data (top: Biceps, bottom: Triceps). Rows 3&4 in the figure below show amplitude coherence between Biceps and Triceps before and after practice, left column; first and second half of practice sessions on the right. Rows 5&6 show phase coherence before and after practice, left column; first and second half of practice sessions on the right. ​Note how the phase of reference muscle change each cycle from leading/rising(blue) to lagging/falling(red) at frequencies below 2Hz.