From virtual worlds to the augmented human

The industry of the future is expected to undergo a revolution in the way humans interact with increasingly more intelligent machines. In the coming years, the industry faces the challenge of adjusting to and taking advantage of significant change drivers that are already very noticeable. Based on various digital technologies, the world we live in today is already highly interconnected, with humans, intelligent systems, machines, and devices acting together, trying to evolve into new forms of collective intelligence. With the development of cyber-physical systems, the Internet of Things (IoT), and sensor networks, we now see a greater interpenetration between the physical (real world) and the virtual world. Today's human being finds increasingly autonomous and intelligent environments and productive systems that involve a considerable diversity of technologies. ● Transformação da força de trabalho, promovendo o desenvolvimento de novas competências no ser humano, que lhe permitam gerir o trabalho digitalmente com o suporte dos sistemas ciber-físicos;

This trend brings new challenges, mainly concerning the relationship between human beings and these industrial environments, with an increasing amount of data and information available that has to be understood, increasing the complexity of management and decision making. The revolution in the way humans interact with machines translates into solving several challenges, of which we can highlight the following:

• Transformation of the workforce, promoting the development of new skills in humans that enable them to manage work digitally with the support of cyber-physical systems;

• Development of human-centred industrial systems, enabling greater action in all industrial processes and providing them with more adequate resources to extend their decision-making and action capabilities;

• Promotion of more significant, more efficient, and more effective collaboration of the human being with intelligent machines, systems, and robots, also enabling the increase of the added value of final products and services.

Given these challenges, immersive systems technology has been developing to increase the human beings' ability to intervene in this new work environment through technologies such as Virtual Reality (VR) or Augmented Reality (AR).

Paul Milgram's seminal article, published in 1994[1] but still prevailing today, defines a continuum between Reality and Virtuality, where solutions like VR and AR are mapped (figure 1). Therefore, this continuum represents different combinations of real and virtual elements within what Milgram refers to as Mixed Reality (MR) and is currently encompassed in the broader concept of Extended Reality (XR).

On the right side of the reality-virtuality continuum come the virtual environments, where the user is fully immersed in a world that is not interconnected with the real context in which the user is - immersive systems. In the virtual environment of Fig. 2, developed within the scope of an industrial project of INESC TEC, we observe a training situation in which a maintenance technician practices a procedure. The user is completely isolated from the real world by the equipment s/he uses, feeling "transported" to a virtual environment, where s/he can operate a set of tools to repair electrical equipment. Immersion and the sense of presence that can result from it are two very relevant characteristics of a VR system.

On the left side of the continuum comes augmented reality, in which virtual three-dimensional objects are positioned in an integrated way with real objects. Here arises a strong connection of the user with the context in which s/he is inserted or operates. As an example, Fig. 3 shows a paper map being augmented with virtual information such as routes and green areas. The tablet is, in this case, used as a "magnifying glass"...

Considering these examples, what solutions could we consider as something intermediate in the reality-virtuality continuum? The intermediate solutions of Mixed Reality combine virtual environments augmented by real objects present in the actual environment in which the user is inserted. The real room walls and the furniture can be integrated into the virtual environment, allowing the user to touch the wall or lean against a table to perform a specific task. Fig. 4 illustrates stepping on a step of a virtual walkway. Although the user is only a few millimetres off the ground, on a wooden board, his brain feels present in a virtual environment hundreds of metres high, providing a physiological sensation of vertigo.

Research in the area of immersive systems is several decades old, from Morton Heilig's "Sensorama" in the 50s to Ivan Sutherland's "Sword of Damocles" in the 60s. However, only in the last decade has the technology and its production reached levels of maturity that allow, on the one hand, the generation of real-time content, with quality and response times feasible to create a good sense of presence and immersion, and, on the other hand, produce the devices at prices that facilitate wider adoption. We can find several systems on the market today with different device configurations. One of the most appealing is the "Head-Mounted Display" (HMD). It includes a viewing and listening device to be placed on the user's head and a pair of controls to be used in the hands - called "wands" (among other auxiliary devices).

These configurations accommodate:

(a) Stereoscopic vision - the user has a sense of depth, as each eye receives the image from a slightly different viewpoint, as happens in a real situation;

(b) Synchronization between head movements and the virtual viewpoint/audience - the view received follows the head movement, and sounds are reproduced according to the orientation relative to the sound sources;

(d) Position of the user in space (in some configurations) - the user has the sensation of moving in virtual space when doing so in the real space where s/he is located.

In some cases, users may report a feeling of nausea when using these types of devices, which mainly occurs when the synchronization between actual movements and what is visualized is not well achieved. Motion synchronization is one of the challenges shared with augmented reality systems. Additionally, the integration - also referred to as registration - of virtual elements into the real elements of the scene being augmented is another challenge of AR. For example, a virtual box seems to be correctly placed on a real table, maintaining visual coherence even when the user or the device moves.

INESC TEC has been developing training scenarios in the scope of Industry 4.0. One of the applications that have been gaining interest is the use of "digital twins": real machines and systems in operation are virtually replicated and augmented with real-time information about their status and task planning. Additionally, the institute has been working on interactive 360º videos[3], which are an alternative to virtual replicas when it comes to capturing and visualizing reality in an immersive way. Also, it has been possible to test future industrial layouts in virtual environments at the planning and management level and evaluate their effectiveness and cognitive load, focusing the new industrial systems on the human being. And in maintenance tasks or even in jobs on an assembly line, augmented reality allows more efficient performance, through augmented information on the machine itself, or remote collaboration with experts, in a mixed reality environment to solve more complex problems.

Through immersive technologies, humans can increase their ability to intervene in the industrial environment more effectively and with less cognitive overload. The future industry will most likely be the stage for the emergence of an augmented human being, integrating immersively into industrial systems that are more intelligent and autonomous. References

1. Milgram, P., & Kishino, F. (1994). A taxonomy of mixed reality visual displays. IEICE TRANSACTIONS on Information and Systems, 77(12), 1321-1329.

https://massive.inesctec.pt/

https://av360.inesctec.pt

From virtual worlds to the augmented human

António Coelho

Augusto Sousa

Rui Rodrigues

Filipa Ramalho