Intelligent Glasses Replace Construction And Operating Instructions

Professor Dr. Thomas Schack, Dr. Kai Essig and Dr. Matthias Schröder further developed the intelligent glasses at the Cluster of Excellence Cognitive Interaction Technology (Citec) at Bielefeld University. With “computer vision”, the system can independently recognize objects and action steps and, based on this, transfer the appropriate action support directly to the display of the glasses.

Picture gallery

“People with cognitive impairments are quickly overwhelmed when building a birdhouse. With the smart glasses you can build a bird house step by step and are guided according to your needs,”says Dr. Kai Essig, who helped to develop the Adamaas system. "By measuring the mental representation structures of the users beforehand, the glasses can be individually adjusted to the needs of the test subjects."

Personal support

The test subjects can conveniently carry out the measurements in advance on a tablet by assigning different images to specific work steps. For example, if a person has difficulty mounting the roof on the bird feeder, the system knows that and can provide more extensive assistance in this step. In addition, the Adamaas system checks the heart rate and shows soothing information on the display even before the person is overwhelmed.

So far, the Adamaas researchers have used the system to support, for example, the operation of a modern coffee machine. The advantage of this was that the coffee machine used (project partner Miele) was able to directly identify the current condition and pass it on to the glasses. “The difficulty with building a bird feeder is that the individual components and steps must be recognized directly by the glasses. What is self-evident for humans presents a technical system with great challenges,”says Dr. Matthias Schröder, who programmed the "computer vision" of the Adamaas system. "Using a camera, the glasses can not only recognize where each object is located, but also which component is currently being picked up by the user or viewed with the eyes."

Additional information about the project

In addition to Prowerk and the Breipohls Hof senior center, the v. Bodelschwinghschen Stiftungen Bethel is the family company Hettich Partner in the Adamaas project. When assembling a cabinet drawer with drive technology, an instruction manual is often not very helpful. Trainees or employees are ideally supported by the displayed steps in the glasses display. "It would be conceivable, for example, that employees at Hettich use the glasses during assembly to quickly learn how to put the drawers together," says Schack. "The Adamaas system is also configured for Work 4.0 by supporting assembly tasks through augmented reality."

The scientists are currently carrying out Adamaas studies with subjects from Hettich, Prowerk and the Breipohls Hof senior center. Studies with employees at Hettich are also planned in the future. In this way, they can respond even better to the individual needs of users and explore further research options. In the future, the Adamaas system will do without an additional camera and will only support users via the camera integrated in the glasses.

Another partner in the Adamaas project is the eye tracking specialist SensoMotoric Instruments (SMI), who develop and provide the hardware for the glasses. Velamed GmbH Medical Technology and Biomechanical Concepts develops a system for resource-efficient and robust recording and evaluation of vital parameters. Various faculties of Bielefeld University also work with external partners. In addition to Professor Dr. Helge Ritter (AG Neuroinformatics), Professor Dr. Thomas Schack (AG Neurocognition and Movement), Dr.-Ing. Sven Wachsmuth (central laboratory) and Dr. Kai Essig (AG Neurocognition and Movement) from Citec is involved in the Faculty of Psychology and Sports Science, as well as the Faculty of Technology and the Bielefeld 2000plus network.The Faculty of Economics supports ADAMAAS with the Institute for Technological Innovation, Market Development and Entrepreneurship (iTIME) in the development of a marketing and sales concept.

Action support via overlays

The system can then give instructions depending on the situation and directly display personal action support via on-screen displays. "With the computer vision, the Adamaas system can support almost every construction process," says Professor Dr. Thomas Schack, head of the Adamaas project. “This means that test subjects no longer have to deal with operating instructions and laboriously carry out the work steps with one hand. You can work directly with both hands and the steps one to one are shown. This is also exciting for the industry.”(Mz)