Intelligence and Affection

Emergent Technologies & Media (ETM)

Andy Weeger

Neu-Ulm University of Applied Sciences

September 23, 2025

Revision

Hypothesis 1

Emerging information technologies enable multimodal and immersive systems.

Multimodality

Multimodality refers to the use of multiple modes of communication to to create meaning.

Multimodality implies that the use of several means of communication contributes to a better overall understanding of a message.

Adami (2016)

Immersion

Immersion refers to the state of being deeply engaged, absorbed, or submerged in an environment, either physically or mentally.

Immersion implies that the consciousness of the immersed person is detached from their physical self. Immersiveness is the quality or degree of being immersive.

Suh & Prophet (2018), Lee et al. (2013)

Interdependency

Stimuli that determine the immersiveness of environments created by technology are multimodal.

Visual, auditory, tactile, olfactory, and interactive.

Hypothesis 2

Emerging information technologies enable intelligent and affective systems.

Intelligence

Discussion

What do we mean
by intelligence?

Provide a description that outlines what intelligence could mean.

Human intelligence

Human intelligence “covers the capacity to learn, reason, and adaptively perform effective actions within an environment, based on existing knowledge. This allows humans to adapt to changing environments and act towards achieving their goals.” Dellermann et al. (2019, p. 632)

Sternberg et al. (1985) proposes three distinctive dimensions:

• Componential intelligence
  the ability to take apart problems and being able to see solutions not often seen
• Experiential intelligence
  the ability to learn and adapt through evolutionary experience
• Contextual intelligence
  the ability to create an ideal fit between the environment by adaptation, shaping, and selection

Artificial intelligence

‘AI system’ means a machine-based systems designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment and that, for explicit or implicit objectives, infers, from the input it received, how to generate output such as content, predictions, recommendations, or decisions, that can influence physical or virtual environment (European Commission, 2024).

Based on this definition, three main properties of intelligent agents can be distinguished:

Capacity to work in a complex environment¹, cognitive abilities², and complex behavior³.

Complex environments

Agents

An agent is anything that can be viewed as perceiving its environment through sensors and acting upon that environment through actuators.

Agents and environments

Necessary components to interact with complex environments

Example

Example for an intelligent systems

Task environment

When designing an intelligent system, the task environment (i.e., the problem) must be specified as fully as possible, including

The performance measure
the task environment,
the actuators,
and the sensors

Russel & Norvig (2022) call the task environment PEAS.

Example of an PEAS description

Task environment of a taxi driver agent

• P: Safe, fast, legal, comfortable, maximize profits, minimize impact on other road users
• E: Roads, other road users, police, pedestrians, customers, weather
• A: Steering, accelerator, brake, signal horn, display, speech
• S: Cameras, radar, speedometer, GPS, engine, sensors, accelerometer, microphones, touchscreen

Source: Russel & Norvig (2022, p. 61)

Properties

Task environments can be categorized along following dimensions (Russel & Norvig, 2022, pp. 62–64):

• Fully observable ⇠⇢ partially observable
• Single agent ⇠⇢ multi-agent
• Deterministic ⇠⇢ nondeterministic
• Episodic ⇠⇢ sequential
• Static ⇠⇢ dynamic
• Discrete ⇠⇢ continuous
• Known ⇠⇢ unknown

Explanations

• If an agent’s sensors give it access to the full state of the environment at any point in time, then we say that the task environment is fully observable (e.g., image analysis).
• When multiple agents intend to maximize a performance measure that depends on the behavior of other agents, we say the environment is multi-agent (e.g., chess).
• When the environment is completely determined by the current state and the actions performed by the agent(s), it is called a deterministic environment (e.g., crossword puzzle). When a model of the environment explicitly uses probabilities, it is called a stochastic environment (e.g., poker).
• If an agent’s experience is divided into atomic episodes in which the agent receives a perception and then performs a single action, and if the next episode does not depend on the actions performed in the previous episodes, then we say that the task environment is episodic (e.g., image analysis).
• If the environment changes while an agent is deliberating, then the environment is dynamic (e.g., taxi driving).
• If the environment has a finite number of different states, we speak of discrete environments (e.g., chess).
• If the outcomes (or outcome probabilities) for all actions are given, then the environment is known (e.g., solitaire card game).

Source: Russel & Norvig (2022), p.62-64

The hardest case is partially observable, multi-agent, nondeterministic, sequential, dynamic, and continuous.

Exercise

Describe the task environment of a chess player and an autonomous car.

10:00

Chess player:

• static
• discrete
• fully-observable
• deterministic
• sequential
• known

Autonomous car

• dynamic
• continuos
• partial-observable
• stochastic
• sequential
• known

Autonomous vs. advisor system

Types of intelligent systems in terms of their interaction with the environment (Molina, 2020)

 

Cognitive abilities

Definition

Cognitive abilities—or “thinking skills”—are mental capacities that enable us to acquire knowledge, process information, and solve problems. They involve processing mental information through (Carroll, 1993):

perception, attention, memory, reasoning, language, and executive functions

Implementation in AI systems

Primary cognitive abilities of intelligent systems

 

 

 

 

Perception refers to AI systems’ ability to gather and interpret data from their environment through sensors, cameras, microphones, or data inputs

Deliberation refers to the computational processes of analyzing information, drawing conclusions, and making decisions.

Interaction refers to the communication capabilities with AI systems, including natural language processing and generation.

Action control refers to the mechanisms that translate decisions into actions or outputs in the environment.

Discussion

What are the basic cognitive abilities of a chatbot and how they are technically implemented?

07:00

Deliberation and reactive behavior

Different types of behavior require different “thinking systems” based on Molina (2020)

Multiagent systems

Meta cognitive abilities by means of multiagent systems based on Molina (2020)

Metacognition is “thinking about thinking” — the ability to reflect on, monitor, and regulate one’s own cognitive processes.

It involves two main components:

• Metacognitive knowledge: Understanding and awareness of your own cognitive abilities, strategies, and limitations. This includes knowing what you know and don’t know, understanding how you learn best, and recognizing the demands of different tasks.
• Metacognitive regulation: The active control and monitoring of cognitive processes during learning or problem-solving. This includes planning how to approach a task, monitoring your comprehension or progress, evaluating your performance, and adjusting strategies when needed.

In traditional (single-agent) systems, metacognition is about self-monitoring: “Do I know enough to solve this problem?” But in multiagent systems, metacognition becomes distributed and “social”.

This architecture supports metacognitive abilities because agents can reflect on their own knowledge, recognize the limitations of their partial view, communicate to fill knowledge gaps, and reason about what other agents might know or perceive. The distributed nature of the system requires agents to be aware of their own cognitive boundaries.

Example scenario: Imagine robots exploring a building. Each sees only their room (partial environment). A metacognitive agent recognizes “I can’t see the exit from here” (knowledge limitation), reasons “Agent 3 near the entrance likely knows” (distributed knowledge model), and initiates communication (regulatory action).

Exercise

Find two real-world examples of multi-agent systems (one digital, one physical)

For each system, document:

• What are the individual agents?
• How do they jointly perceive and deliberate?
• How do agents interact/coordinate?

10:00

Complex behavior

Properties

Complex behavior refers to the observable patterns of actions and responses that an AI system exhibits (i.e., what the system does), particularly sophisticated, adaptive, or emergent conduct that goes beyond simple stimulus-response patterns.

To realize complex behavior, the components of an intelligent system (i.e., what the system can do cognitively) must have the following properties (to some extent):

Autonomy, rationality,
learning, and introspection

Rationality

A rational agent is one that does the right thing—it is goal directed.

For each possible percept sequence, a rational agent should select an action that is expected to maximize its performance measure, given the evidence provided by the percept sequence and whatever built-in knowledge the agent has. Russel & Norvig (2022, p. 58)

What is rational at any given time depends on four things:

• The performance measure that defines the criterion of success
• The agent’s prior knowledge of the environment
• The actions that the agent can performance
• The agent’s percept sequence to date

It can be quite hard to formulate a performance measure correctly, however:

If we use, to achieve our purposes, a mechanical agency with those operation we cannot interfere once we have started it […] we had better be quite sure that the purpose built into the machine is the purpose which we really desire Wiener (1960, p. 1358)

Discussion

Under which circumstances does a chatbot act rational?

Under following circumstances, the vacuum cleaning agent is rational:

• The performance measure of the vacuum cleaner might award one point for each clean square at each time step, over a “lifetime” of 1,000 time steps (to prevent the cleaner to oscillate needlessly back and forth).
• The “geography” of the environment is known a priori but the dirt distribution and the initial location of the agent are not. Clean squares stay clean and sucking cleans the current square. The Right and Left actions move the agent one square except when this would take the agent outside the environment in which case the agent remains where it is.
• The only available action is Right, Left, and Suck.
• The agent correctly perceives its location and whether that location contains dirt.

For details such as tabulated agent functions please see Russel & Norvig (2022).

Rationality and perfection

Rationality != perfection

• Rationality maximizes expected performance
• Perfection maximizes actual performance
• Perfection requires omniscience
• Rational choice depends only on the percept sequence to date

As the environment is usually not completely known a priori and completely predictable (or stable), information gathering and learning are important parts of rationality (Russel & Norvig, 2022, p. 59).

Example: The vacuum cleaner needs to explore an initially unknown environment (i.e., exploration) to maximize its expected performance. In addition, a vacuum cleaner that learns to predict where and when additional dirt will appear will do better than one that does not.

Rationality and cognitive abilities

Rational decisions affect different cognitive abilities (Molina, 2020)

 

 

 

 

Learning

Learning agents are those that can improve their behavior through diligent study of past experiences and predictions of the future Russel & Norvig (2022, p. 668)

A learning agent

• uses so-called machine learning (ML), if it is a computer;
• improves performance based on experience (i.e., observations of the world);
• is required when the designer lacks omniscience (i.e., in unknown environments) and/or
• has no idea how to program a solution themselves (e.g., recognizing faces)

Learning types

Supervised learning
Involves learning a function from examples ➞ test and training data

Unsupervised learning
The agent has to learn patterns in the input ➞ identification of categories or classifications

Reinforcement learning
The agent must learn from punishments or rewards ➞ learning by trial and error

Learning and cognitive abilities

Adaptation through learning can affect differnt cognitive abilities (Molina, 2020)

 

 

 

 

Introspection

Introspection refers to the capabilitiy to analyze one’s cognitive abilities.

The system uses an observable model of its own abilities. 
This model is used to simulate self-awareness processes.

Introspection allows the system …

• … to judge its own actions and, thus, provides learning opportunities
  (e.g., analyzing past outputs ot identify errors or biases) and
• … to generate explanations and, thus, to justify decisions to the user
  (e.g., explainable AI — showing how a systems arrives at a solution)

Summary

The properties of an intelligent system are

Capacity to work in a
complex environment
Interaction with the environment and other agents

Cognitive abilities
Perception, action control, deliberation, and interaction

Complex behavior
Acting autonomously, rationally, adaptation through learning, and introspection

Homework

Select an intelligent system and analyse it using the properties outlined here.

Note: This is an excellent task to prepare for the exam.

Read Dellermann et al. (2019).

Affections

Affective computing

Computing that relates to, arises from or deliberately influences emotion.

Picard (2000)

Objectives

Assigning systems “the human-like capabilities of observation, interpretation and generation of affect features⁴” (Tao & Tan, 2005, p. 981)

The goal is to simulate empathy — that affective systems can interpret the emotional states of humans and adapt their behavior to them, giving an appropriate response for those emotions (i.e., emotion aware systems).

Properties

Emotion recognition
Interpreting the emotional states of humans

Emotion expresssion
Ability to simulate human affects (e.g. ‘emotional modality’)

Adequate response to emotion
Linking emotion recognition and expression e.g., to reinforce the meaning of messages

Emotional signals

Facial expression, posture, speech, force or rhythm of key stroke, temperature change (e.g., hand on mouse) can signify changes in user’s emotional state.

These can be detected and interpreted by an affective system.

Affective systems can use some of these to simulate emoptions.

Basic emotions

Ekman et al. (1987) categorized emotions into 6 groups:

Fear, surprise, disgust, anger, happiness, and sadness

All of these can facially expressed.

Examples

• Facial expression analysis
  Using computer vision and machine learning to analyze facial expressions and determine the emotional state of a person.
• Voice analysis
  Analyzing the tone, pitch, and other characteristics of a person’s voice to determine their emotional state.
• Physiological sensing
  Using wearable devices to monitor physiological signals such as heart rate, skin conductance, and body temperature to detect emotional responses.
• Emotion simulation
  Developing systems that can generate emotional responses, such as a virtual assistant that can express empathy or a chatbot that can adapt its tone based on the user’s emotional state.

Homework

Search for real-life use cases for affective computing.

Prepare to present the use case and the technologies that enable it.
Relate them to the basic properties of affective systems.

Argue why affective computing is effective in this use case.

Hybrid intelligence

Exercise

Form small groups and synthesize your findings from reading Dellermann et al. (2019) by findings answers to following questions:

1. How can hybrid intelligence be defined?
2. What are main characteristics of hybrid intelligence?
3. What are complementary strengths of humans and machines?
4. What implications does that concept have for practice?

Take 7 minutes to discuss your findings with your neigbour.

Concept

The idea is to combine the complementary capabilities of humans and computers to augment each other.

Dellermann et al. (2019)

Complementary strengths

Human intelligence

Flexibility & transfer

Empathy & creativity

Eventualities

Common sense

Intuition

Artificial intelligence

Pattern recognition

Probabilistic

Consistency

Speed & efficiency

Analysis

Humans are flexible, creative, empathic, and can adapt to various settings. This allows, for instance, human domain experts to deal with so called ‘‘broken-leg’’ predictions that deviate from the currently known probability distribution. However, they are restricted by bound rationality that prevents them from aggregating information perfectly and drawing conclusions from that. On the other hand, machines are particularly good at solving repetitive tasks that require fast processing of huge amounts of data, at recognizing complex patterns, or weighing multiple factors following consistent rules of probability theory (Dellermann et al., 2019).

However, there is also a technology-centric perspective that assumes that true intelligence can ultimately only be found in well-developed and mature (general) AI systems. Humans are biologically limited in their information processing and reasoning abilities and exhibit many types of cognitive biases, while computers offer virtually infinite possibilities to develop rational intelligence at human levels and beyond (Peeters et al., 2021).

Model of human cognition

Kahneman (2011) proposed a two-system model of human cognition, which he called System 1 and System 2.

System 1 is an intuitive, automatic, and fast mode of thinking that operates outside of our conscious awareness. It is responsible for generating impressions, making quick judgments, and executing routine tasks with minimal effort.

System 2, on the other hand, is a more analytical, controlled, and deliberate mode of thinking that requires conscious effort and attention. It is responsible for problem-solving, critical thinking, and decision-making.

While System 1 operates quickly and automatically, it can be prone to biases and errors, particularly in complex or unfamiliar situations. System 2, though slower and more effortful, can help us avoid these biases and make more accurate decisions.

Definition

Hybrid intelligence is defined as the ability to achieve complex goals by combining human and artificial intelligence, thereby reaching superior results to those each of them could have accomplished separately, and continuously improve by learning from each other. Dellermann et al. (2019, p. 640)

Main characteristics of hybrid intelligence are:

• Collectively
  Tasks are performed collectively and activities are conditionally dependent
• Superior results
  Neither AI nor humans could have achieved the outcome without the other
• Continuous learning
  All components of the socio-technical system learn from each other through experience

Visualization

Distribution of roles in hybrid intelligence (Dellermann et al., 2019, p. 640)

Implications

According to Peeters et al. (2021) following conclusions can be drawn:

• Intelligence should not be studied at the level of individual humans or AI-machines, but at the group level of humans and AI-machines working together.
• Increasing the intelligence of a system should be achieved by increasing the quality of the interaction between its constituents rather than the intelligence of the constituents themselves.
• Both human as well as artificial intelligence can be regarded as very shallow when considered in isolation.
• No AI is an island (it’s about the system of systems).

Examples

Robots in de klas

A team consisting of a remedial teacher, an educational therapist, and a Nao robot collaborate to support a child with learning difficulties. The robot provides expertise and advice while also helping the child stay focused and engaged.

Spawn

The musician Holly Herndon created “Spawn,” an AI system that generates unique music different from her usual style. By using Spawn as a tool, Holly is able to avoid creating music that repeats her previous works but to to expand the possibilities of their music.

GitHub Copilot

In collaborative coding, Copilot can engage in back-and-forth dialogue about software design decisions, propose implementations, and explain reasoning about technical approaches - moving beyond simply generating code.

What examples do come to your mind?

Q&A

Literature

Adami, E. (2016). Introducing multimodality. The Oxford Handbook of Language and Society, 451–472.

Carroll, J. B. (1993). Human cognitive abilities: A survey of factor-analytic studies. Cambridge university press.

Dellermann, D., Ebel, P., Söllner, M., & Leimeister, J. M. (2019). Hybrid intelligence. Business & Information Systems Engineering, 61, 637–643.

Ekman, P., Friesen, W. V., O’sullivan, M., Chan, A., Diacoyanni-Tarlatzis, I., Heider, K., Krause, R., LeCompte, W. A., Pitcairn, T., Ricci-Bitti, P. E., et al. (1987). Universals and cultural differences in the judgments of facial expressions of emotion. Journal of Personality and Social Psychology, 53(4), 712.

European Commission. (2024). Artificial intelligence in the european commission — a strategic vision to foster the development and use of lawful, safe and trustworthy artificial intelligence systems in the european commission. C(2024) 380.

Kahneman, D. (2011). Thinking, fast and slow. macmillan.

Lee, H.-G., Chung, S., & Lee, W.-H. (2013). Presence in virtual golf simulators: The effects of presence on perceived enjoyment, perceived value, and behavioral intention. New Media & Society, 15(6), 930–946.

Molina, M. (2020). Intelligent systems. Master Course (Lecture Slides).

Peeters, M. M., Diggelen, J. van, Van Den Bosch, K., Bronkhorst, A., Neerincx, M. A., Schraagen, J. M., & Raaijmakers, S. (2021). Hybrid collective intelligence in a human–AI society. AI & Society, 36, 217–238.

Picard, R. W. (2000). Affective computing. MIT press.

Russel, S., & Norvig, P. (2022). Artificial intelligence: A modern approach. Pearson Education.

Sternberg, R. J. et al. (1985). Beyond IQ: A triarchic theory of human intelligence. CUP Archive.

Suh, A., & Prophet, J. (2018). The state of immersive technology research: A literature analysis. Computers in Human Behavior, 86, 77–90.

Tao, J., & Tan, T. (2005). Affective computing: A review. International Conference on Affective Computing and Intelligent Interaction, 981–995.

Wiener, N. (1960). Some moral and technical consequences of automation. Science, 131(3410), 1355–1358.

Footnotes

1. The capacity to work in a complex environment is described as agency

2. Cognitive abilities are, for instance, perception and language

3. Behavior refers the observable patterns of actions and responses that an AI system exhibits. Autonomy, adaptiveness, goal-directedness, emergence, and context-sensitivity makes it complex

4. ”Affect” is basically a synonym for emotion.