Multimodality and Immersion

Future Technologies & Media (FTM)

Andy Weeger

Neu-Ulm University of Applied Sciences

April 15, 2024

Revision

Characteristics of emerging technologies

Rotolo, Hicks, and Martin (2015) outlines five attributes that classify emerging technologies and differentiate them from other technologies:

1. Radical novelty
   
2. Relatively fast growth
   
3. Coherence
   
4. Prominent impact
   
5. Uncertainty and ambiguity
   

Stages of emergence

Pre-emergence, emergence, and post-emergence: attributes and ’stylised’trends. (Rotolo, Hicks, and Martin 2015, 1833)

The steam enginge

During the late 18th century and the early 19th century, the steam engine was considered an emerging technology.

Why?

• Radical novelty: Steam enginges represented a significant departure from previous methods of power generation (e.g., such as human or animal labor).
• Relatively fast growth: Once the potential of steam power was realized, the adoption and diffusion of steam engines spread rapidly across various industries and regions.
• Coherence: The development of the steam engine was part of a coherent technological trajectory.
• Prominent impact: The steam engine had a profound impact on society, economy, and culture (i.e., the Industrial Revolution)
• Uncertainty and ambiguity: In its early stages of development, the steam engine faced uncertainty and ambiguity regarding its reliability, efficiency, and practical applications.

Artificial Intelligence

Tech classified as AI definitely qualify as emergent technologies.

• Radical novelty: While the underlying concepts of AI have been around for decades, its current applications are truly groundbreaking.
• Relatively fast growth: New advancements and discoveries are happening at an incredible pace, pushing the boundaries of what’s possible.
• Coherence: AI research isn’t happening in isolation. It draws on various disciplines like computer science, mathematics, linguistics, and neuroscience.
• Prominent impact: AI is transforming industries and shaping the future in a significant way. Its influence is felt in healthcare, finance, entertainment, and countless other sectors.
• Uncertainty and ambiguity: The full potential and limitations of AI are still being explored as well as the manifold ethical concerns.

Technological characteristics

What are distinct characteristics of emerging technologies in the digital age?

Hypothesis 1

Emerging information technologies enable multimodal and immersive systems.

Multimodality

Discussion

Which senses have you used when you have interacted with (information) technology?

Give examples.

How the
computer
sees us.

For traditional computers, humans are reduced to an eye and a finger.
Courtesy Dan O’Sullivan and Tom Igoe.

Multimodality — definition

Multimodality

Using more than one mode of communication to create meaning.

Multimodality emphasizes the importance of multiple modes (e.g., verbal, visual, spatial) to form overall understanding of a message.

Adami (2016)

Multimedia — definition

Multimedia

Using more than one media to express meaning.

Multimedia emphasizes the medium or technology used to present information.
It refers to content that uses two or more media formats.

Here’s an analogy: Think of multimedia like a toolbox with different tools (text, images, audio, etc.). Multimodal is about how you use those tools together to build something meaningful.

Adami (2016)

Distinction

Every multimodal project is multimedia
as it uses multiple methods.

Not every multimedia project is multimodal
as it might just throw different media together without considering how they work together.

Reading

Image from unsplash

Writing

Image from unsplash

Texting

Image from unsplash

Multimodal interactions
are natural

Usage of modalities in current computer systems

Advantages of multimodal systems

Improved accuracy and robustness, enhanced bandwidth, flexibility and user preference, naturalness and ease of use, redundancy and error correction, accessibility, support for complex tasks.

Improved accuracy and robustness

Multimodal systems can enhance accuracy and robustness by integrating multiple modalities, such as speech, gesture, and touch, which can compensate for limitations in individual modalities.

Enhanced bandwidth

By utilizing multiple modalities simultaneously, multimodal systems can provide users with a higher bandwidth of interaction, allowing for more efficient communication and interaction.

Flexibility and user preference

Multimodal systems offer users the flexibility to choose the most convenient modality for interaction based on their preferences, abilities, and the context of the task.

Naturalness and ease of use

Integrating multiple modalities can make interaction with technology more natural and intuitive, mimicking the way humans naturally communicate and interact with each other.

Redundancy and error correction

Multiple modalities provide redundancy in input, allowing for error detection and correction. For example, if one modality fails or is misunderstood, other modalities can provide additional context or clarification.

Accessibility

Multimodal systems can improve accessibility for users with diverse abilities and preferences by offering multiple modes of interaction that cater to different needs and capabilities.

Support for complex tasks

Combining multiple modalities can support complex tasks that require different types of input or involve multiple steps, allowing users to interact with technology in a more efficient and effective manner.

Oviatt (1999)

Exercise

Find examples for the main modalities.

Think of visual, auditory, and haptic modalities for input (i.e., goes into the computer) and output (computer creates as output to the user) each for both data (i.e., the contents of the interaction) and control (i.e., information that is needed to make the interaction happen).

Take 10 minutes to find examples.

Fundamental problems

Multimodal fusion

The integration of communication modalities in interactive systems (Input)

and

Multimodal fission

The re-partitioning of information among several communication modalities (Output)

Human-machine interaction loop

A representation of multimodal man machine interaction loop based on Dumas, Lalanne, and Oviatt (2009, 8)

 

 

 

 

 

 

Data level

e.g., combining 2 webcam video streams, multiple perspectives

Feature level

e.g., combining extracted features such as speech and lip movements

Decision level

e.g., combining interpretations such as gestures and sentiment of speech

Exercise

Find examples for multimodal technologies.

Take 10 minutes for your research. Prepare to present your findings.

Example: multimodal AI

Immersiveness

Homework

Form small groups and synthesize your findings from reading Suh and Prophet (2018) by findings answers to following questions:

1. How can immersive technology be defined?
2. What are examples of immersive technology provided in the paper?
3. What technological features are key to enhancing user experience in immersive environments?
4. How does immersiveness relate to multimodality?
5. What fields of application are revealed in the paper?

Take 20 minutes to synthesize your findings and to create a short presentation.

Immersion

Immersion refers to the state of being deeply engaged, absorbed, or submerged in an environment, either physically or mentally.

Immersion implies that the consciousness of the immersed person is detached from their physical self. Immersiveness is the quality or degree of being immersive.

Suh and Prophet (2018), Lee, Chung, and Lee (2013)

Immersion in games

From engagement (e.g.: invest time, effort , and attention in the game) to
engrossment (e.g. invest a high level of emotion in the game) to
total immersion (e.g.fully present and empathize with the game) (Sunyaev 2020).

Image from unsplash

Immersive technology

Immersive technology blurs the line between the physical, virtual, and simulated worlds, thereby creating a sense of immersion.

Technology has different abilities to create a sense of presence and engagement in the user.

Lee, Chung, and Lee (2013), Handa, Aul, and Bajaj (2012)

Mechanisms

The feeling of immersion is created by temporarily altering a person’s sense of presence by tricking their cognitive and perceptual systems into believing they are in a place other than their actual physical location.

Milgram and Kishino (1994)

Virtuality continuum

Simplified representation of a “virtuality continuum” based on Milgram and Kishino (1994)

 

 

Stimuly

Important stimuli that determine the immersiveness of environments created by technology are

Visual stimuly, auditory stimuly, tactile stimuli, olfactory stimuly, and interactive stimuly.

In order to create these, technology needs visual display with hihg representational fidelity and media, auditory and haptic interfaces and the capability to track movements.

Summary based on the findings on sensory and perceptual stimuly by Suh and Prophet (2018).

Augmented reality (AR)

Augmented reality refers to the combination of a real scene viewed by a user with a virtual scene that augments the scene with additional information.

AR technology superimposes virtual objects onto a live view of physical environments, helping users visualize how these objects fit into their physical world.

Milgram and Kishino (1994)

Hardware components

AR technologies require following hardware components:

1. Devices for displaying the virtual objects
2. Input devices to detect users’ interaction with the real and virtual object
3. Sensors to capture users’ positions and movements
4. Computing devices that combine the input data and create the virtual overlays

Advancements in mobile computing and telecommunications infrastructures (will) increase the mobility and, thus, usability of these devices significantly.

Sunyaev (2020)

Exercise

What are examples of AR technologies?

Briefly introduce the example and name application fields and benefits.

Virtual reality (AR)

Virtual Reality refers to technology that generates an interactive virtual environment that is designed to simulate a real life experience.

VR technology reveals different levels of immersion ranging from non-immersive VR (e.g., second life) to immersive VR (e.g., PlayStation VR2).

Suh and Prophet (2018), Milgram and Kishino (1994)

Exercise

Search for immersive technologies used in the healthcare context.

Take 15 minutes to research and portray a technology as well as to discuss the reasons for the prominent impact of immersive technologies in the healthcare context. Prepare yourself to present your findings.

Q&A

Literature

Adami, Elisabetta. 2016. “Introducing Multimodality.” The Oxford Handbook of Language and Society, 451–72.

Dumas, Bruno, Denis Lalanne, and Sharon Oviatt. 2009. “Multimodal Interfaces: A Survey of Principles, Models and Frameworks.” In Human Machine Interaction: Research Results of the Mmi Program, 3–26. Springer.

Handa, Mandeep, Er Gagandeep Aul, and Shelja Bajaj. 2012. “Immersive Technology–Uses, Challenges and Opportunities.” International Journal of Computing & Business Research 6 (2): 1–11.

Lee, Hyuck-Gi, Sungwon Chung, and Won-Hee Lee. 2013. “Presence in Virtual Golf Simulators: The Effects of Presence on Perceived Enjoyment, Perceived Value, and Behavioral Intention.” New Media & Society 15 (6): 930–46.

Milgram, Paul, and Fumio Kishino. 1994. “A Taxonomy of Mixed Reality Visual Displays.” IEICE TRANSACTIONS on Information and Systems 77 (12): 1321–29.

Oviatt, Sharon. 1999. “Ten Myths of Multimodal Interaction.” Communications of the ACM 42 (11): 74–81.

Rotolo, Daniele, Diana Hicks, and Ben R Martin. 2015. “What Is an Emerging Technology?” Research Policy 44 (10): 1827–43.

Suh, Ayoung, and Jane Prophet. 2018. “The State of Immersive Technology Research: A Literature Analysis.” Computers in Human Behavior 86: 77–90.

Sunyaev, Ali. 2020. Internet Computing: Principles of Distributed Systems and Emerging Internet-Based Technologies. 1st ed. Springer Nature Switzerland AG.