Researchers at Cornell University have created a machine-learning model for generating an appropriate robot response based on evaluating human activities. For these researchers, the secret to creating a technological “glass ball” for robots to anticipate our actions is a conditional random field (CRF) model and Kinect real-time technology typically used in video games to trace motions. Capabilities such as these will open this technology up to a range of applications spanning from the restaurant industry to manufacturing lines.

The idea of creating a model that allows robots to consistently and successfully respond to our actions stemmed from the notion that robots unable to anticipate and react to humans could be viewed as impractical in human-robot interactions. As machine hardware and software continues to advance, this model could be the next critical step for preparing robots to better integrate with natural human behavior.

Previous research has been successful in enabling a robot to see human activities and label them, but have not found success in using that labeling system to anticipate the future. For example, if a robot sees a person move his hand to a drinking glass, there are predictable places that the glass might end up such as a mouth, the sink or a different location on the table. If a robot is able to anticipate these movements, then the robot would avoid pouring soda into a cup that is about to be placed in the sink for washing. In order for a robot to anticipate actions, the robots and the researchers needed to understand these various possibilities, called object affordances.

An object affordance is a collection of action possibilities provided by the environment, including the identification of an object (glass), and the functionalities of an object (drink, move, etc.). Understanding these object affordances are critical to the machine-learning model of robot responses and anticipation. In every human activity (drinking, eating, cleaning, etc.), there is a hierarchical structure composed of a sequence of sub-activities. Like in the previous example, a glass is used in the drinking activity which is composed of the sub-activities reach, move and drink. Therefore, a robot can reason about and anticipate the future by observing the sub-activities performed in the past. Modeling this problem is extremely intricate as the future is infinite and anything can happen.

But this is exactly what Dr. Hema Koppula and her team set out to do.

Following the evaluation of 120 every day human activities for their object affordances, the researchers sought to overcome the complexity of object affordance by predicting how future sub-activities are physically represented by the object and the human.

Using the CRF model and Kinect real-time technology, they used the position of an object with respect to the human and the environment (glass up to human mouth vs. glass on a stable surface) and the object’s motion trajectories to represent probabilistic outcomes of future activities. Combining all this information essentially creates a machine-learning model for robots that anticipates future activities as well, or nearly as well, as humans and use this to perform look-ahead planning of its reactive responses. The image below represents the steps that the robots take to inform their reactive response using this CRF model.



This robot response model can be used in various settings such as in hospitals where robots monitor patients’ vitals, or in factories where robots work a manufacturing line. It can also be applied to semi-autonomous cars where the vehicle could alert the driver of a potentially dangerous move based on predicting what other agents around it will do.

“Humans use the art of anticipation in every interaction, every movement and every thought and feeling without realizing it. If humans couldn’t anticipate, we would be in many more car accidents, frustrate or embarrass those we interact with and be unable to manage our time,” said Koppula. “Our research enables robots to mimic our ability to do this, bringing them one step closer to being safer, more efficient, and more lifelike.”

While the future of advanced robotics is bright, its application uses for improving everyday human life from here is all but predictable.

Read more about robot response at IEEE Xplore.