The SecondHands project

The SecondHands project was born in 2015 to develop a robot that could proactively assist technicians with industrial automation maintenance tasks. This ‘cobot’, named ARMAR-6, would effectively provide a second pair of hands – the inspiration for the project name.

The goals were for the cobot to proactively assist with physical and unskilled practical tasks such as holding or lifting heavy machinery, guard insertion or removal, and reaching for and passing objects like maintenance tools. This would create a safer working environment for technicians and free up their time to focus on the more skilled aspects of their role.

A revolutionary robot

ARMAR-6 is a humanoid robot with two compliant 8 degree-of-freedom arms, under-actuated hands, holonomic platform, a head with five cameras, and a functional control architecture for integration of sensorimotor skills, and learning and reasoning abilities. 

Several things make ARMAR-6 special. By combining visual and haptic sensing with model-based and data-driven machine learning approaches, ARMAR-6 physically interacts with people in a way that feels natural to the person it interacts with. 

Advancements in natural language processing mean ARMAR-6 is able to verbally interact more naturally with people, listening and responding to commands appropriately. It can recognise human activities and intentions, evaluating when help is required and proactively offering suitable assistance.

These cognitive and speech capabilities, which exceed the state-of-the-art, make it much more interactive, in a natural way – a prerequisite for a truly interactive and collaborative cobot.

Task understanding for proactively offering help

For ARMAR-6 to be truly beneficial as a cobot in an industrial setting it needed the capacity to autonomously recognise its collaboration partner’s need for help, and offer assistance proactively.

With safety always a consideration, due to the close proximity of the robot and technician, we needed advancements in existing state-dependent dynamical system approaches to provide solutions for compliant interaction with humans. This required the development of new algorithms for gathering knowledge - from activities and action sequence recognition, to tools segmentation and context classification. 

New algorithms help ARMAR-6 build geometric and semantic 3D maps of the environment even when there are multiple independently moving objects. This dynamic scene understanding teaches robots how to see in 3D without the requirement of fully supervised 3D training data - a huge step forward in Machine Learning. Dynamic AI capabilities allow ARMAR-6 to react appropriately in unforeseen situations.

Reactive motion planning means ARMAR-6 can respond appropriately to human or environmental factors, such as offering a technician a tool they’ve turned to pick up, or avoiding collisions with a moving object. Novel architectures for help recognition enable fast interaction and assistance via anticipation and forecasting techniques, which allows the cobot to recognise and offer help in situations that were not anticipated during the original programming.

Building trust through natural language 

Cobots must complement the existing ways we work for us to trust and adopt them in our workplaces and everyday lives. They must be able to respond in human-suited timescales and in ways that are meaningful for us. 

‍

ARMAR-6’s speech interface is based solely on all-neural models: all-neural automatic speech recognition, all-neural dialog modelling and all-neural speech synthesis. Commands like ‘Pass me that screwdriver’ were tested by Ocado technicians inside our warehouses to ensure ARMAR-6 could listen and respond naturally in a working environment.

Communicating in natural language is crucial to usability and acceptance; the breakthroughs made in natural language interfaces and task understanding will contribute to the emergence of true collaboration between humans and cobots, and for creating robots with embodied intelligence.

Testing in the Ocado warehouse

Testing performance in everyday scenarios is key for the development of reliable technology. Maintenance technicians at Ocado, and in other industries, have many physical and mental challenges on a daily basis. Our highly automated warehouses contain rapidly moving objects, heavy and high value equipment, and audio interference such as echos and background noise

‍

These dynamic conditions provided the perfect test environment to ensure the robotics, language processing algorithms and obstacle avoidance capabilities of ARMAR-6 were robust. In the warehouse context, this includes holding heavy machinery components like diverter panels, picking the right tool from within a box and passing it over, and much more.

The feedback from technicians who took part in the real-world trials was unanimously positive. They were impressed by the robot's understanding and ability to communicate naturally with them; the ability to follow commands like “Pass me that screwdriver – the big one, not the small one!”  

Usually, technicians work alone and have to manage heavy objects by themselves, but they commented on how natural the interaction with the robot felt when sharing the load of a panel from the underside of a diverter.

Beyond the warehouse

SecondHands has demonstrated how robots can amplify the benefits of human expertise. Outside of industrial settings like our warehouses, robots with sophisticated manipulation, learning and adaptation abilities will be key for solving the bigger challenges our society faces. 

Healthcare and assisted living, hazardous environments like oil and gas, or disaster response are all scenarios that could benefit from the technologies that enable ARMAR-6 to communicate and interact with humans, like natural language comprehension, soft manipulation and 3D spatial awareness. Here at Ocado Technology, we’ll continue to build on our learnings, looking towards a future where we’ll see these breakthroughs applied in real-world industrial environments.

The Horizon 2020 SecondHands Consortium

SecondHands was made possible thanks to Horizon 2020: the biggest EU research and innovation programme ever. 

Research is often undertaken in isolation – but we worked with leading scientists and roboticists from a range of disciplines in academia. This ensured the SecondHands project would be able to push the boundaries of our research and make breakthroughs in several areas of AI and Robotics that far exceed what each could have accomplished alone.

The SecondHands Consortium included:

• Ecole Polytechnique Fédérale de Lausanne (EPFL)
• Karlsruhe Institute of Technology (KIT)
• Ocado Technology
• Sapienza, University of Rome
• University College London  (UCL)
• Read more about the SecondHands Consortium here.

Change your world with us

Across Ocado Technology, we have a diverse, rich mix of teams and expertise working  to solve complex problems. Learn more about our full range of opportunities here.