Multimodality and Immersion

Emergent Technologies & Media (ETM)

Andy Weeger

Neu-Ulm University of Applied Sciences

September 23, 2025

Recap

The steam enginge

During the late 18th century and the early 19th century, the steam engine was considered an emerging technology.

Why?

• Radical novelty: Steam enginges represented a significant departure from previous methods of power generation (e.g., such as human or animal labor).
• Relatively fast growth: Once the potential of steam power was realized, the adoption and diffusion of steam engines spread rapidly across various industries and regions.
• Coherence: The development of the steam engine was part of a coherent technological trajectory.
• Prominent impact: The steam engine had a profound impact on society, economy, and culture (i.e., the Industrial Revolution)
• Uncertainty and ambiguity: In its early stages of development, the steam engine faced uncertainty and ambiguity regarding its reliability, efficiency, and practical applications.

Characteristics of emerging technologies

Rotolo et al. (2015) outlines five attributes that classify emerging technologies and differentiate them from other technologies:

1. Radical novelty
   
2. Relatively fast growth
   
3. Coherence
   
4. Prominent impact
   
5. Uncertainty and ambiguity
   

Stages of emergence

Pre-emergence, emergence, and post-emergence: attributes and ’stylised’trends. (Rotolo et al., 2015, p. 1833)

AI — towards a definition

‘AI system’ means a machine-based systems designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment and that, for explicit or implicit objectives, infers, from the input it received, how to generate output such as content, predictions, recommendations, or decisions, that can influence physical or virtual environment. (European Commission, 2024)

Systems that perceive, learn, think and act human-like.

AI an emergent technology

Many systems currently discussed under the umbrella-term AI definitely qualify as emergent technologies.

• Radical novelty: While the underlying concepts of AI have been around for decades, its current applications are truly groundbreaking.
• Relatively fast growth: New advancements and discoveries are happening at an incredible pace, pushing the boundaries of what’s possible.
• Coherence: AI research isn’t happening in isolation. It draws on various disciplines like computer science, mathematics, linguistics, and neuroscience.
• Prominent impact: AI is transforming industries and shaping the future in a significant way. Its influence is felt in healthcare, finance, entertainment, and countless other sectors.
• Uncertainty and ambiguity: The full potential and limitations of AI are still being explored as well as the manifold ethical concerns.

Discussion

What are distinct characteristics of emerging digital technologies?

Hypothesis 1

Emerging digital technologies enable multimodal and immersive experiences.

Multimodality

Discussion

Which senses have you used when you have interacted with (information) technology?

Give examples.

How the
computer
sees us.

For traditional computers, humans are reduced to an eye and a finger.
Courtesy Dan O’Sullivan and Tom Igoe.

Multimodality — definition

Multimodality

Using more than one mode of communication to create meaning simultaneously.

Multimodality emphasizes the importance of multiple modes (e.g., visual, linguistics, audio, spatial, gestural) to form overall understanding of a message.

Multimodality is a theoretical concept about communication and meaning-making.

Adami (2016)

Multimedia — definition

Multimedia

Using more than one media formats in the presentation of content.

Multimedia emphasizes the technical format and delivery systems used to present information.
A multimedia presentation might include slides with text, embedded videos, and audio narration.

Multimedia is a practical term about content delivery and technology.

Here’s an analogy: Think of multimedia like a toolbox with different tools (text, images, audio, etc.). Multimodal is about how you use those tools together to build something meaningful.

Adami (2016)

Distinction

Every multimodal system is a multimedia system
as it uses multiple methods for content delivery.

Not every multimedia system is multimodal
as it might just throw different media together without considering how they work together.

Reading

Image from unsplash

Writing

Image from unsplash

Texting

Image from unsplash

Multimodal interactions
are natural

Usage of modalities in current computer systems

Usage of modalities in current computer systems

Advantages of multimodal systems

Improved accuracy and robustness, enhanced bandwidth, flexibility and user preference, naturalness and ease of use, redundancy and error correction, accessibility, support for complex tasks.

Improved accuracy and robustness

Multimodal systems can enhance accuracy and robustness by integrating multiple modalities, such as speech, gesture, and touch, which can compensate for limitations in individual modalities. Example: medical diagnosis assistant (verbal description, medical imaging, history, sensor data)

Enhanced bandwidth

By utilizing multiple modalities simultaneously, multimodal systems can provide users with a higher bandwidth of interaction, allowing for more efficient communication and interaction. Example: autonomous vehicle navigation (LIDAR, Radar, GPS, Camera Vision)

Flexibility and user preference

Multimodal systems offer users the flexibility to choose the most convenient modality for interaction based on their preferences, abilities, and the context of the task. Example: smart home assistant (voice, touch, gesture, mobile app)

Naturalness and ease of use

Integrating multiple modalities can make interaction with technology more natural and intuitive, mimicking the way humans naturally communicate and interact with each other. Example: language learning application

Redundancy and error correction

Multiple modalities provide redundancy in input, allowing for error detection and correction. For example, if one modality fails or is misunderstood, other modalities can provide additional context or clarification. Example: speach recognition system (audio, lip reading, facial expression)

Accessibility

Multimodal systems can improve accessibility for users with diverse abilities and preferences by offering multiple modes of interaction that cater to different needs and capabilities.

Support for complex tasks

Combining multiple modalities can support complex tasks that require different types of input or involve multiple steps, allowing users to interact with technology in a more efficient and effective manner. Example: surgical robotic system (hand movement, 3D medical imaging, patien vital signs, AR)

Oviatt (1999)

Fundamental problems

Multimodal fusion

The integration of communication modalities in interactive systems (Input)

and

Multimodal fission

The re-partitioning of information among several communication modalities (Output)

Human-machine interaction loop

A representation of multimodal man machine interaction loop based on Dumas et al. (2009, p. 8)

 

 

 

 

 

 

Data level

e.g., combining 2 webcam video streams, multiple perspectives

Feature level

e.g., combining extracted features such as speech and lip movements

Decision level

e.g., combining interpretations such as gestures and sentiment of speech

Multimodal AI in action

Example: Multimodal AI

Example: The Future of Windows

Source: The Verge, assessed October 18, 8:43 am

Exercise

Modality examples

Identify specific examples of multimodal systems. Analyze which modalities are used for input (data and control) as well as output (data and control).

Inputs and outputs per modality:

Visual Modalities

Data Input

• Reading text from documents
• Image recognition (identifying objects, faces, scenes)
• Gesture recognition
• Tracking eye movements
• Scanning QR codes or barcodes

Control Input

• Mouse pointer movement
• Cursor selection
• Touch screen gestures
• Blinking to select
• Gaze-based interface navigation

Data Output

• Displaying text
• Rendering images and graphics
• Video playback
• Data visualizations
• Augmented reality overlays

Control Output

• Highlighting selected items
• Progress bars
• Color-coded status indicators
• Cursor changes
• Visual feedback animations

Auditory Modalities

Data Input

• Speech recognition
• Voice commands
• Musical input
• Environmental sound detection
• Language translation

Control Input

• Volume control via voice
• Selecting options through audio cues
• Pitch and tone-based commands
• Whisper or silent speech recognition
• Audio-based authentication

Data Output

• Text-to-speech conversion
• Music playback
• Sound effects
• Audiobooks
• Language pronunciation guidance

Control Output

• Confirmation beeps
• Error sounds
• Navigation audio cues
• Volume and pitch changes
• Audio feedback for interactions

Haptic Modalities

Data Input

• Touch screen interactions
• Pressure-sensitive input
• Gesture recognition
• Handwriting recognition
• Tactile sensing for object identification

Control Input

• Swipe gestures
• Pinch-to-zoom
• Pressure-based selection
• Vibration patterns for input
• Tactile input for accessibility

Data Output

• Vibration patterns representing information
• Texture simulation
• Force feedback in games
• Medical training simulations
• Braille displays

Control Output

• Confirmation vibrations
• Error alerts
• Navigation guidance
• Intensity-based feedback
• Accessibility cues for visually impaired users

Homework

Read Suh & Prophet (2018) and figure out what immersive technologies are and how this concept relates to multimodality.

Recap

AI as emergent tech

Many systems currently discussed under the umbrella-term AI qualify as emergent technologies.

What is AI and why can it be considered as emergent tech?

‘AI system’ means a machine-based systems designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment and that, for explicit or implicit objectives, infers, from the input it received, how to generate output such as content, predictions, recommendations, or decisions, that can influence physical or virtual environment. (European Commission, 2024)

Systems that perceive, learn, think and act human-like.

Hypothesis 1

Emerging digital technologies enable multimodal and immersive experiences.

Multimodality

Multimodality

Using more than one mode of communication (e.g., visual, linguistics, audio, spatial or gestural) to create meaning simultaneously.

The main challenges in creating multimodal systems are:

Multimodal fusion and fission

That is, the integration of communication modalities in interactive systems (input fusion) and the re-partitioning of information among several communication modalities (output fission).

Immersiveness

Immersion in games

Engagement is the investment of time, effort and attention in the game.
Engrossment is the investment of a high level of emotion in the game.
Total immersion means being fully present and empathising with the game.

See Sunyaev (2020); image from unsplash

Exercise

Take 10 minutes and recap your findings from reading Suh & Prophet (2018) along following questions:

1. How can immersive technology be defined?
2. What are examples of immersive technology provided in the paper?
3. What technological features are key to enhancing user experience in immersive environments?
4. How does immersiveness relate to multimodality?
5. What fields of application are revealed in the paper?

Immersion

Immersion refers to the state of being deeply engaged, absorbed, or submerged in an environment, either physically or mentally.

Immersion implies that the consciousness of the immersed person is detached from their physical self. Immersiveness is the quality or degree of being immersive.

Suh & Prophet (2018), Lee et al. (2013)

Discussion

Have you ever felt detached from your physical self? What caused that feeling?

Immersive technology

Immersive technology blurs the line between the physical, virtual, and simulated worlds, thereby creating a sense of immersion.

Technology has different abilities to create a sense of presence and engagement in the user.

Lee et al. (2013), Handa et al. (2012)

Immersive?

Design challenge

Redesign the weather experience
to be more immersive.

Group with your neighbor and ideate on a more immersive weather experience.

Guiding questions:

• What modalities will you add?
• How do they work together (multimodal fusion)?
• What stimuli create the feeling of immersion??
• How are the stimuli created?
• What limitation do you hit with current technologies?

Mechanisms

The feeling of immersion is created by temporarily altering a person’s sense of presence by tricking their cognitive and perceptual systems into believing they are in a place other than their actual physical location.

Milgram & Kishino (1994)

Virtuality continuum

Simplified representation of a “virtuality continuum” based on Milgram & Kishino (1994)

 

 

The virtuality continuum (also called the reality-virtuality continuum) is a concept that places all possible variations of real and virtual environments on a single continuous scale. Here’s how it’s organized:

• At one end of the continuum is the Real Environment - our actual physical world that we perceive directly through our senses without any technological mediation.
• At the opposite end is the Virtual Environment (or Virtual Reality) - a completely synthetic digital environment that replaces the real world with computer-generated content.
• Between these two extremes lies Mixed Reality, which includes:
  • Augmented Reality (AR) - closer to the real environment, where virtual objects are overlaid onto the real world. The primary world remains the physical environment, but it’s enhanced with digital elements (like Pokémon GO or AR navigation apps).
  • Augmented Virtuality (AV) - closer to the virtual environment, where elements from the real world are brought into a predominantly virtual space.

The key insight from Milgram and Kishino’s taxonomy is that rather than seeing these as discrete categories, they represent points along a spectrum where the proportion of real to virtual content gradually shifts. This continuum helps us understand and classify different mixed reality technologies based on how much they blend or replace our physical reality.

Extent of world knowledge (comprehensiveness of the world model)

This refers to how much the system “knows” about the environment it’s representing. On the reality end of the continuum, there’s no need for the system to have knowledge about the world since it’s the actual physical world. As you move toward virtuality, the system requires increasingly comprehensive models of what it’s displaying. A fully virtual environment needs complete knowledge about every object, surface, and behavior in the simulated world.

Reproduction fidelity

This describes the quality and realism of the displayed imagery. At the reality end, reproduction fidelity is perfect because you’re seeing the actual world. As you move toward virtual environments, the challenge becomes creating images that approach the visual quality of direct viewing. Early VR had low fidelity (primitive graphics), while modern systems strive for photorealism. This factor addresses questions like: How realistic do the virtual elements look? How well do they blend with real objects?

Extent of presence metaphor (immersiveness)

This relates to the degree to which the user feels present within the displayed scene. It includes factors like:

• Field of view (how much of your visual field is covered)
• Stereo viewing capabilities (3D depth perception)
• Head tracking and other sensory feedback
• The extent to which the user feels immersed rather than just observing

Stimuli

Visual Stimuli

• The extent to which virtual objects obscure real objects and vice versa
• Whether the display is monoscopic or stereoscopic (providing depth perception)
• The realism and fidelity of the rendered images
• Field of view considerations
• Registration accuracy (how well virtual content aligns with the real world)

Auditory Stimuli

• Real environments have natural, unmediated sound
• Augmented reality might overlay virtual sounds onto real-world audio
• Virtual environments must synthesize all auditory information

Tactile Stimuli

• Real environments provide natural tactile feedback
• Mixed reality might offer limited haptic feedback for virtual objects
• Advanced virtual environments aim to simulate tactile sensations

Olfactory Stimuli

Smell is one of the more challenging senses to reproduce in mixed and virtual reality contexts, though it would follow the same conceptual continuum.

Interactive Stimuli

How users engage with and manipulate the environment:

• In real environments, interaction is direct and natural
• In augmented reality, users might interact with virtual objects using real-world movements
• In fully virtual environments, all interactions must be captured and translated into the virtual space

Important stimuli that determine the immersiveness of environments created by technology are

Visual stimuli, auditory stimuli, tactile stimuli, olfactory stimuli, and interactive stimuli.

In order to create these, technology needs visual displays with high representational fidelity, auditory and haptic interfaces, olfactory delivery systems (e.g., scent devices), and the capability to track movements and enable real-time interaction.

Summary based on the findings on sensory and perceptual stimuli by Suh & Prophet (2018).

Augmented reality (AR)

Augmented reality refers to the combination of a real scene viewed by a user with a virtual scene that augments the scene with additional information.

AR technology superimposes virtual objects onto a live view of physical environments, helping users visualize how these objects fit into their physical world.

Milgram & Kishino (1994)

Discussion

Immersion refers to the state of being deeply engaged, absorbed, or submerged in an environment, either physically or mentally.

How is immersion reflected in augmented reality?

Immersion in augmented reality (AR) manifests quite differently than in fully virtual environments, creating what’s often called a “blended” or “mixed” immersion experience. Here’s how this plays out:

Sensory integration

AR achieves immersion by seamlessly layering digital content onto the real world. When done well, virtual objects appear anchored to physical spaces—shadows fall correctly, digital items occlude behind real objects, and elements respond to real-world lighting. This sensory coherence makes your brain accept the digital additions as part of your environment rather than separate overlays.

Presence through context

Unlike VR’s total replacement of reality, AR creates immersion through contextual relevance. Digital information appears exactly where and when you need it—navigation arrows on actual streets, furniture visualized in your real room, or maintenance instructions overlaid on the specific machine you’re repairing. This contextual anchoring keeps you engaged because the digital content directly enhances what you’re already doing.

Physical embodiment AR maintains your connection to the physical world, which paradoxically can deepen immersion for certain tasks. You use natural movements—walking around objects, reaching out to manipulate virtual items, looking around corners. Your real body remains the interface, making interactions feel more intuitive than controller-based systems.

The immersion spectrum

AR exists on a spectrum. Simple smartphone AR (like Pokémon GO) offers limited immersion—you’re clearly looking at a screen. Advanced headsets with wide fields of view, spatial audio, and hand tracking create much deeper immersion by occupying more of your sensory bandwidth while still preserving real-world awareness.

The unique quality of AR immersion is that it’s additive rather than replacing—you’re deeply engaged with an enhanced version of reality rather than transported elsewhere entirely.

Hardware components

AR technologies require following hardware components:

1. Devices for multi-sensory output: visual displays, audio systems, haptic actuators, and (potentially) olfactory generators
2. Input devices to detect users’ interactions with real and virtual objects
3. Sensors to capture users’ positions and movements
4. Computing devices that combine input data and create the virtual overlays

Advancements in mobile computing and telecommunications infrastructures (will) increase the mobility and, thus, usability of these devices significantly.

Extends on Sunyaev (2020)

Exercise

What are examples of AR technologies?

Briefly introduce one example and name application fields and benefits.

Virtual reality (VR)

Virtual Reality refers to technology that generates an interactive virtual environment that is designed to simulate a real life experience.

VR technology reveals different levels of immersion ranging from non-immersive VR (e.g., second life) to immersive VR (e.g., PlayStation VR2).

Suh & Prophet (2018), Milgram & Kishino (1994)

Exercise

Search for immersive technologies used in the healthcare context.

Take 15 minutes and …

• research and portray a technology based on the characteristics discussed here;
• discuss the reasons for the prominent impact of immersive technologies in healthcare;
• and prepare yourself to present your findings.

Q&A

Literature

Adami, E. (2016). Introducing multimodality. The Oxford Handbook of Language and Society, 451–472.

Dumas, B., Lalanne, D., & Oviatt, S. (2009). Multimodal interfaces: A survey of principles, models and frameworks. In Human machine interaction: Research results of the mmi program (pp. 3–26). Springer.

European Commission. (2024). Artificial intelligence in the european commission — a strategic vision to foster the development and use of lawful, safe and trustworthy artificial intelligence systems in the european commission. C(2024) 380.

Handa, M., Aul, E. G., & Bajaj, S. (2012). Immersive technology–uses, challenges and opportunities. International Journal of Computing & Business Research, 6(2), 1–11.

Lee, H.-G., Chung, S., & Lee, W.-H. (2013). Presence in virtual golf simulators: The effects of presence on perceived enjoyment, perceived value, and behavioral intention. New Media & Society, 15(6), 930–946.

Milgram, P., & Kishino, F. (1994). A taxonomy of mixed reality visual displays. IEICE TRANSACTIONS on Information and Systems, 77(12), 1321–1329.

Oviatt, S. (1999). Ten myths of multimodal interaction. Communications of the ACM, 42(11), 74–81.

Rotolo, D., Hicks, D., & Martin, B. R. (2015). What is an emerging technology? Research Policy, 44(10), 1827–1843.

Suh, A., & Prophet, J. (2018). The state of immersive technology research: A literature analysis. Computers in Human Behavior, 86, 77–90.

Sunyaev, A. (2020). Internet computing: Principles of distributed systems and emerging internet-based technologies (1st ed.). Springer Nature Switzerland AG.