AR and VR in Manufacturing: Being There

Smart Communication

AR and VR in Manufacturing: Being There

Virtual reality is not exactly new – VR goggles have been around since the ’90s – but now virtual reality is poised to arrive on the shop floor, along with augmented reality, both of which promise to revolutionize how things are made.

by Rainer Claassen

In 1985 Mercedes-Benz introduced the world’s first virtual driving simulator in Berlin. A whole car could be placed on a moving platform – and projectors turned the surrounding walls into a computer-generated landscape. With this device, the research and development (R&D) department was able to analyze a driver’s behavior in difficult or dangerous situations, without the risk of anyone getting hurt. Back then, this called for a seven-figure investment, but today, 30 years later, kids are exploring virtual environments, wearing simple VR sets that cost less than their smartphones. Entertainment has been one of the driving forces behind the development of the technology, but it’s very likely to bring big changes in the world of industry. Virtual reality (VR) and augmented reality (AR) are similar in many ways but distinct in both their underlying technologies and the way they are used. A VR computer generates a 3D image of a desired setting which can then be experienced via a screen, a smartphone display, or VR goggles – a headset containing monitor screens that provides a truly immersive, “real-life” experience. The headsets are usually equipped with gyro sensors that precisely catch the user’s every motion, allowing the displayed image to adapt to the wearer’s movement. This creates the illusion of actually “being there.”

AR and VR in Manufacturing : Mercedes Benz Simulation

Early days 1985: The virtual driving simulator at Daimler-Benz cost a small fortune

More sophisticated headsets have two dedicated displays mounted inside, one for each eye. The unit is attached to a workstation that provides two high resolution video images displayed at a rate of 90 frames per second. These systems – like Oculus Rift by a spin-off from Facebook, or HTC’s Vive – also come with handheld controllers to allow interaction and surround-sound headphones to make the illusion even more realistic.

Augmented reality, on the other hand, doesn’t cut users off from the world around them. Instead, by superimposing visuals and text over a camera image of a real scene, additional information is supplied which is intended to enrich the viewer’s perception of reality. Smartphones are one way of doing this. When the phone’s camera is pointed at something, the AR app recognizes the image and displays relevant bits of information or superimposed images into the display.

Spending on AR technology is predicted to hit $60bn by the year 2020

Dedicated devices like Microsoft’s HoloLens are more sophisticated and allow more thorough integration. It looks like a normal pair of spectacles. Objects or textual elements are displayed in the viewing field of the user – for example a calendar from a computer could be displayed and would look like it was hanging on an empty wall. By tracking the user’s movements, the virtual overlays always seem to stay tethered to the correct place. In combination with handheld controllers, interactions with virtual parts of the scene are possible. Though this technology is just beginning to spread, the outlook is fantastic, with estimates predicting spending on AR technology will hit €60bn by the year 2020.

A learning process

R&D is the typical use case for both technologies but AR and VR are being adopted increasingly in production environments. Ford Motor Company uses touchscreen devices to train its assembly line workers when new processes are introduced. A model of a work space on the assembly line is displayed on the screen. In the first step, the actions the worker has to do are demonstrated. As the learning process advances, the demo becomes more sophisticated in four steps. In the last one, the user has to choose all the tools correctly and use them in the right way. This system allows workers to learn new skills without disrupting work at the assembly line – and without the need to withdraw an experienced worker.

AR and VR in Manufacturing  - Ford Plant

The right choice: Workers at Ford can learn without disrupting the assembly line

VR can also be useful before production even begins. It takes architects to fully understand blueprint plans of structures, and it takes engineers to read details of a construction plan. With VR technology it is now possible to take the data from the two components and put them together to build a 3D model of a finished construction site before the first brick has been laid.

Now that it no longer takes expert training to understand and judge these models, the CEO of a corporation can wander through the virtual building and get a vivid impression of the current status of the work just as clearly as a worker who lives with the construction every day. Mistakes in planning that would not have been apparent before the building was completed can now become glaringly obvious much earlier. To make this possible, it’s necessary to “translate” construction data from different sources into formats suitable for visualization – a task that is not always easy but, as the demand is growing, solutions are developing. Engineering software developer and consultancy Salt & Pepper, located in Osnabrück, Germany, specializes in data transfer and visualization.

AR and VR Manufacturing: Timo Seggelmann and Fabian Schlarmann

VR pioneers: Timo Seggelmann and Fabian Scharmann of Salt & Pepper convert CAD data into virtual reality with the help of their Forestage tool

CAD and 3D models can be transformed through its Forestage tool for virtual reality displays on HTC Vive, Oculus Rift, or Google Cardboard. This can be done in seconds and the software is more or less intuitive, say Timo Seggelmann and Fabian Scharmann of Salt & Pepper. Clients include companies like Claas, a producer of farm machinery, and carmaker Daimler.

When using tools like Vive users are not limited to just watching the virtual surroundings – they can also interact with them. Forestage is able to import metadata from construction plans – so moveable parts can actually be “picked up” in the virtual world with a handheld controller as the user moves within the virtual surroundings to view things from different angles.

Augmented workplace

On the shop floor, scenarios for augmented reality are somewhat different. Service companies like Munichbased Reflekt focus mostly on themes such as service and maintenance operations and training. Similar to the earlier examples, Reflekt’s tools can also be used with construction data. One of the typical use cases for AR lies in maintenance where technicians can be guided through inspection procedures, and benefit from features like screenshots, displays of relevant measurements and values, as well as automated creation of inspection reports. Recurring repair procedures can be simplified and the frequency of errors reduced drastically when an AR guide is used instead of traditional printed descriptions.

AR and VR in Manufacturing: Bergolin

Gloves on: Paintshop workers at Bergolin can leave their protective gloves on when operating a virtual computer terminal

Similar to the example from Ford, training the workforce can also be made a lot more efficient with AR. Italian mixed reality company Inglobe Technologies has developed a post installation checking solution for the Huawei SUN2000-25KTL inverter – a piece of equipment employed in large industrial solar power applications. Technicians can follow step-by-step instructions with augmented reality graphic elements and animations to perform assembly and wiring operations, quickly and easily locating the access points on the equipment and performing technical assignments. During the process, the technician is aided in choosing the most suitable tools for executing individual tasks.

AR technology has other benefits. In the production facilities of Bergolin, a medium-sized paint manufacturer in Northern Germany, the HoloLens has become an important part of the production work. Production staff are required to wear heavy chemical resistant gloves and had to remove them before entering feedback into the company’s terminals, which are located in specific places. With the introduction of HoloLens, the facility is now equipped with virtual mobile terminals and 3D touch points, which can be virtually displayed anywhere in the room and can even move along with the user. Simple hand gestures allow the recording of all job data in real time, so the entered information is available immediately – and fewer errors occur when entering the data. Not only is the cost of providing the interface low, additional savings have been made by Bergolin because the old terminals’ buttons, switches, and dials are no longer needed.

AR and VR Manufacturing: DHL Vision Picking

Look, no hands: Hands-free pickup at DHL can actually be fun

AR completely changes the way user interfaces work. Virtual control panels can be superimposed directly on a machine, which can then be operated by hand gestures and voice commands. Workers wearing smart glasses are able to walk along a line of factory machines, see their performance parameters, and adjust each machine without physically touching it. Maintenance can also be done with the help of remote experts, who can almost literally see things through the eyes of the operator.

AR and VR Manufacturing: Huawei UPS2000 Augmented Reality installation

Mixed reality: At Huawei, VR assists in assembly and wiring of solar power inverters

At its engine plant in Győr, Hungary, Audi has done some AR testing in the area for assembly of original parts, where engines are built by hand in accordance with customers’ specifications. The production process takes several hours and some shelves in the engine assembly area have up to 200 compartments for small parts, many of which look very similar.

AR and VR Manufacturing: Audi TechDay-Smart-Factory Assisted-Reality

Step by step: Workers at Audi’s Györ plant in Hungary are shown how to assemble engines by hand

With a plant pass and a QR code, assembly workers check into their workplace and individual assembly orders are loaded from a central server into the Google Glass spectacles each worker wears. The individual steps of the assembly process for the given engine version are shown over the worker’s right eye in a picture and text presentation. Although Google stalled the development of Glass as a gadget for private use, this project shows how AR can be of great benefit in production facilities.

No more science fiction

As in the Salt & Pepper VR application mentioned earlier, AR can similarly help in planning production facilities. A virtualized model of a construction machine can be positioned on the ground, created using actual planning data. Engineers and workers can walk around it, or even go inside, to gain a full appreciation of the sight lines and ergonomics of the design at full scale in its intended setting.

Workers can adjust each machine without physically touching it

Does all this sound like science fiction or something restricted to big players with huge budgets? Companies, like CMC Engineers in Hülben near Stuttgart, are starting to address SMEs who are doing specialized engineering work, such as CMC’s ViewR VR software. The SMEs offer complete solutions that can be operated by clients without long training processes – and the investment does not have to be in the five-digit area.

While shop floors are just beginning to make use of AR, larger steps have already been made in logistics firms. DHL Supply Chain, a contract logistics specialist within the Deutsche Post DHL Group, has completed global AR pilots for its Vision Picking system and is now expanding its use to warehouses around the globe.

VR and AR headsets that are available in the market

AR and VR Manufacturing: VR Set Glasses

Though both of them are most popular among gamers, Oculus Rift (1) and HTC Vive (2) VR headsets can also be used in industrial settings.
One of the first AR products for the public was Google Glass (3), which was removed from the market in 2015. In summer 2017, X Company in California started promoting Glass Enterprise Edition especially for industrial and business use. Probably the best known AR headset is Microsoft’s HoloLens (4), but there are competitors like Meta’s Meta 2 (5). As an alternative to transparent glasses are headsets with integrated cameras, like HP Windows Mixed Reality (6).

The smart glasses used provide visual displays of order-picking instructions along with information on where items are located and where they need to be placed on a cart. This frees the pickers from having to collect and carry paper instructions and allows them to work more efficiently and comfortably. The trials have shown an average improvement in productivity of 15%, along with higher accuracy rates. The user-friendly and intuitive solution has also halved on boarding and training times.

DHL’s employees have been enthusiastic about being able to use state-of-the-art technology. They are happy with the comfortable smart glasses and now actually enjoy the process of hands-free picking. VR and AR environments seem bound to make production more efficient and transparent while reducing mistakes – but they also make work a lot more fun.


  1. Richele Hrobsky says:

    Would like information or possible onsite visit to discuss VR and AR for automotive plant

  2. Pingback: Bionics and IoT: Man Machines - SMART INDUSTRY

  3. Pingback: VR Camera Insta360: Just Like Being there - SMART INDUSTRY

Leave a Reply

Your email address will not be published. Required fields are marked *