Intelligence and Affection

Future Technologies & Media (FTM)

Andy Weeger

Neu-Ulm University of Applied Sciences

February 21, 2025

Revision

Hypothesis 1

Emerging information technologies enable multimodal and immersive systems.

Multimodality

Multimodality refers to the use of multiple modes of communication to to create meaning.

Multimodality implies that the use of several means of communication contributes to a better overall understanding of a message.

Adami (2016)

Immersion

Immersion refers to the state of being deeply engaged, absorbed, or submerged in an environment, either physically or mentally.

Immersion implies that the consciousness of the immersed person is detached from their physical self. Immersiveness is the quality or degree of being immersive.

Suh and Prophet (2018), Lee, Chung, and Lee (2013)

Interdependency

Stimuli that determine the immersiveness of environments created by technology are multimodal.

Visual, auditory, tactile, olfactory, and interactive.

Hypothesis 2

Emerging information technologies enable intelligent and affective systems.

Intelligence

Discussion

What do we mean
by intelligence?

Provide a description that outlines what intelligence could mean.
Take 5 minutes to reflect in small groups.

Human intelligence

Human intelligence “covers the capacity to learn, reason, and adaptively perform effective actions within an environment, based on existing knowledge. This allows humans to adapt to changing environments and act towards achieving their goals.” Dellermann et al. (2019, 632)

Sternberg et al. (1985) proposes three distinctive dimensions:

• Componential intelligence
  the ability to take apart problems and being able to see solutions not often seen
• Experiential intelligence
  the ability to learn and adapt through evolutionary experience
• Contextual intelligence
  the ability to create an ideal fit between the environment by adaptation, shaping, and selection

Artificial intelligence

‘AI system’ means a machine-based systems designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment and that, for explicit or implicit objectives, infers, from the input it received, how to generate output such as content, predictions, recommendations, or decisions, that can influence physical or virtual environment (European Commission 2024).

Based on this definition, three main properties of intelligent agents can be distinguished:

Capacity to work in a complex environment¹, cognitive abilities², and complex behavior³.

Working in complex environments

Agents and environments

Necessary components to interact with complex environments

An agent is anything that can be viewed as perceiving its environment through sensors and acting upon that environment through actuators.

Example

Example for an intelligent systems

Task environment

When designing an intelligent system, the task environment (i.e., the problem) must be specified as fully as possible, including

Thep performance measure
the environment,
the actuators,
and the sensors

Russel and Norvig (2022) call the task environment PEAS.

Example of an PEAS description

Task environment of a taxi driver agent

• P: Safe, fast, legal, comfortable, maximize profits, minimize impact on other road users
• E: Roads, other road users, police, pedestrians, customers, weather
• A: Steering, accelerator, brake, signal horn, display, speech
• S: Cameras, radar, speedometer, GPS, engine, sensors, accelerometer, microphones, touchscreen

Source: Russel and Norvig (2022, 61)

Properties

Task environments can be categorized along following dimensions (Russel and Norvig 2022, 62–64):

• Fully observable ⇠⇢ partially observable
• Single agent ⇠⇢ multi-agent
• Deterministic ⇠⇢ nondeterministic
• Episodic ⇠⇢ sequential
• Static ⇠⇢ dynamic
• Discrete ⇠⇢ continuous
• Known ⇠⇢ unknown

Explanations

• If an agent’s sensors give it access to the full state of the environment at any point in time, then we say that the task environment is fully observable (e.g., image analysis).
• When multiple agents intend to maximize a performance measure that depends on the behavior of other agents, we say the environment is multi-agent (e.g., chess).
• When the environment is completely determined by the current state and the actions performed by the agent(s), it is called a deterministic environment (e.g., crossword puzzle). When a model of the environment explicitly uses probabilities, it is called a stochastic environment (e.g., poker).
• If an agent’s experience is divided into atomic episodes in which the agent receives a perception and then performs a single action, and if the next episode does not depend on the actions performed in the previous episodes, then we say that the task environment is episodic (e.g., image analysis).
• If the environment changes while an agent is deliberating, then the environment is dynamic (e.g., taxi driving).
• If the environment has a finite number of different states, we speak of discrete environments (e.g., chess).
• If the outcomes (or outcome probabilities) for all actions are given, then the environment is known (e.g., solitaire card game).

Source: Russel and Norvig (2022), p.62-64

The hardest case is partially observable, multi-agent, nondeterministic, sequential, dynamic, and continuous.

Exercise

Describe the task environment of a chess player and a autonomous car.

Form small groups, take 15 minutes for this exercise and prepare yourself to present your findings.

Chess player:

• static
• discrete
• fully-observable
• deterministic
• sequential
• known

Autonomous car

• dynamic
• continuos
• partial-observable
• stochastic
• sequential
• known

Autonomous vs. advisor system

Types of intelligent systems in terms of their interaction with the environment (Molina 2020)

 

Cognitive abilities

Processing mental information

A cognitive ability is an ability that requires to process mental information (Carroll 1993).

Processing mental information requires attention, memory, perception, reasoning, and executive functions.

Molina (2020) discusses four types of cognitive abilities that AI sytems need to implement:

perception⁴,
deliberation⁵,
interaction⁶,
action control⁷

Primary cognitive abilities

Primary cognitive abilities of intelligent systems based on Molina (2020)

 

 

 

 

Exercise

Describe the basic cognitive abilities of an autonomous car and a chatbot and how they are technically implemented.

Form small groups, take 15 minutes for this exercise.
Prepare yourself to present your findings.

Deliberation and reactive behavior

Different types of behavior require different “thinking systems” based on Molina (2020)

Multiagent systems

Meta cognitive abilities by means of multiagent systems based on Molina (2020)

Exercise

Find two real-world examples of multi-agent systems (one digital, one physical)

For each system, document:

• What are the individual agents?
• What goals/objectives do they have?
• How do agents communicate or interact?
• What emergent behaviors arise from these interactions?
• What mechanisms handle conflicts or coordination?

Form small groups, take 15 minutes for this exercise.
Prepare yourself to present your findings.

Complex behavior

Properties

To realize complex behavior, the components of an intelligent system (i.e., perception, deliberation, action control and interaction) must have following properties (to some extent):

Rationality
Learning
Introspection

Rationality

A rational agent is one that does the right thing.

For each possible percept sequence, a rational agent should select an action that is expected to maximize its performance measure, given the evidence provided by the percept sequence and whatever built-in knowledge the agent has (Russel and Norvig 2022, 58).

What is rational at any given time depends on four things:

• The performance measure that defines the criterion of success
• The agent’s prior knowledge of the environment
• The actions that the agent can performance
• The agent’s percept sequence to date

It can be quite hard to formulate a performance measure correctly, however:

If we use, to achieve our purposes, a mechanical agency with those operation we cannot interfere once we have started it […] we had better be quite sure that the purpose built into the machine is the purpose which we really desire (Wiener 1960, 1358)

Exercise

Under which circumstances does a vacuum cleaning agent act rational?

Under following circumstances, the vacuum cleaning agent is rational:

• The performance measure of the vacuum cleaner might award one point for each clean square at each time step, over a “lifetime” of 1,000 time steps (to prevent the cleaner to oscillate needlessly back and forth).
• The “geography” of the environment is known a priori but the dirt distribution and the initial location of the agent are not. Clean squares stay clean and sucking cleans the current square. The Right and Left actions move the agent one square except when this would take the agent outside the environment in which case the agent remains where it is.
• The only available action is Right, Left, and Suck.
• The agent correctly perceives its location and whether that location contains dirt.

For details such as tabulated agent functions please see Russel and Norvig (2022).

Rationality and perfection

Rationality != perfection

• Rationality maximizes expected performance
• Perfection maximizes actual performance
• Perfection requires omniscience
• Rational choice depends only on the percept sequence to date

As the environment is usually not completely known a priori and completely predictable (or stable), information gathering and learning are important parts of rationality (Russel and Norvig 2022, 59).

Example: The vacuum cleaner needs to explore an initially unknown environment (i.e., exploration) to maximize its expected performance. In addition, a vacuum cleaner that learns to predict where and when additional dirt will appear will do better than one that does not.

Rationality and cognitive abilities

Rational decisions affect different cognitive abilities (Molina 2020)

 

 

 

 

Learning

Learning agents are those that can improve their behavior through diligent study of past experiences and predictions of the future Russel and Norvig (2022, 668)

A learning agent

• uses so-called machine learning (ML), if it is a computer;
• improves performance based on experience (i.e., observations of the world);
• is required when the designer lacks omniscience (i.e., in unknown environments) and/or
• has no idea how to program a solution themselves (e.g., recognizing faces)

Learning types

Supervised learning
Involves learning a function from examples ➞ test and training data

Unsupervised learning
The agent has to learn patterns in the input ➞ identification of categories or classifications

Reinforcement learning
The agent must learn from punishments or rewards ➞ learning by trial and error

Learning and cognitive abilities

Adaptation through learning can affect differnt cognitive abilities (Molina 2020)

 

 

 

 

Introspection

Introspection refers to the capabilitiy to analyze one’s cognitive abilities.

The system uses an observable model of its own abilities. 
This model is used to simulate self-awareness processes.

Introspection allows the system …

• … to judge its own actions and, thus, provides learning opportunities
  (e.g., analyzing past outputs ot identify errors or biases) and
• … to generate explanations and, thus, to justify decisions to the user
  (e.g., explainable AI — showing how a systems arrives at a solution)

Summary

The properties of an intelligent system are

Capacity to work in a
complex environment
Interaction with the environment and other agents

Cognitive abilities
Perception, action control, deliberation, and interaction

Complex behavior
Acting rationally, adaptation through learning, and introspection

Exercise

Select an intelligent system and analyse it using the properties outlined here.

Form small group and take 20 minutes to work on a comprehensive analysis regarding the environment, its cognitive abilities as well as the set of complex behaviors.
Prepare a short presentation and prepare yourself to perform it.

Affections

Affective computing

Computing that relates to, arises from or deliberately influences emotion.

Picard (2000)

Objectives

Assigning systems “the human-like capabilities of observation, interpretation and generation of affect features⁸” (Tao and Tan 2005, 981)

The goal is to simulate empathy — that affective systems can interpret the emotional states of humans and adapt their behavior to them, giving an appropriate response for those emotions (i.e., emotion aware systems).

Properties

Emotion recognition
Interpreting the emotional states of humans

Emotion expresssion
Ability to simulate human affects (e.g. ‘emotional modality’)

Adequate response to emotion
Linking emotion recognition and expression e.g., to reinforce the meaning of messages

Emotional signals

Facial expression, posture, speech, force or rhythm of key stroke, temperature change (e.g., hand on mouse) can signify changes in user’s emotional state.

These can be detected and interpreted by an affective system.

Affective systems can use some of these to simulate emoptions.

Basic emotions

Ekman et al. (1987) categorized emotions into 6 groups:

Fear, surprise, disgust, anger, happiness, and sadness

All of these can facially expressed.

Examples

• Facial expression analysis
  Using computer vision and machine learning to analyze facial expressions and determine the emotional state of a person.
• Voice analysis
  Analyzing the tone, pitch, and other characteristics of a person’s voice to determine their emotional state.
• Physiological sensing
  Using wearable devices to monitor physiological signals such as heart rate, skin conductance, and body temperature to detect emotional responses.
• Emotion simulation
  Developing systems that can generate emotional responses, such as a virtual assistant that can express empathy or a chatbot that can adapt its tone based on the user’s emotional state.

Exercise

Search for real-life use cases for affective computing.

Form small group and take 10 minutes for your research.

Prepare a short presentation of a use case and the technologies that enable it.
Relate them to the basic properties of affective systems.

Argue why affective computing is effective in this use case.

Hybrid intelligence

Homework

Form small groups and synthesize your findings from reading Dellermann et al. (2019) by findings answers to following questions:

1. How can hybrid intelligence be defined?
2. What are main characteristics of hybrid intelligence?
3. What are complementary strengths of humans and machines?
4. What implications does that concept have for practice?

Take 7 minutes to discuss your findings with your neigbour.

Concept

The idea is to combine the complementary capabilities of humans and computers to augment each other.

Dellermann et al. (2019)

Complementary strengths

Human intelligence

Flexibility & transfer

Empathy & creativity

Eventualities

Common sense

Intuition

Artificial intelligence

Pattern recognition

Probabilistic

Consistency

Speed & efficiency

Analysis

Humans are flexible, creative, empathic, and can adapt to various settings. This allows, for instance, human domain experts to deal with so called ‘‘broken-leg’’ predictions that deviate from the currently known probability distribution. However, they are restricted by bound rationality that prevents them from aggregating information perfectly and drawing conclusions from that. On the other hand, machines are particularly good at solving repetitive tasks that require fast processing of huge amounts of data, at recognizing complex patterns, or weighing multiple factors following consistent rules of probability theory (Dellermann et al. 2019).

However, there is also a technology-centric perspective that assumes that true intelligence can ultimately only be found in well-developed and mature (general) AI systems. Humans are biologically limited in their information processing and reasoning abilities and exhibit many types of cognitive biases, while computers offer virtually infinite possibilities to develop rational intelligence at human levels and beyond (Peeters et al. 2021).

Model of human cognition

Kahneman (2011) proposed a two-system model of human cognition, which he called System 1 and System 2.

System 1 is an intuitive, automatic, and fast mode of thinking that operates outside of our conscious awareness. It is responsible for generating impressions, making quick judgments, and executing routine tasks with minimal effort.

System 2, on the other hand, is a more analytical, controlled, and deliberate mode of thinking that requires conscious effort and attention. It is responsible for problem-solving, critical thinking, and decision-making.

While System 1 operates quickly and automatically, it can be prone to biases and errors, particularly in complex or unfamiliar situations. System 2, though slower and more effortful, can help us avoid these biases and make more accurate decisions.

Definition

Hybrid intelligence is defined as the ability to achieve complex goals by combining human and artificial intelligence, thereby reaching superior results to those each of them could have accomplished separately, and continuously improve by learning from each other. Dellermann et al. (2019, 640)

Main characteristics of hybrid intelligence are:

• Collectively
  Tasks are performed collectively and activities are conditionally dependent
• Superior results
  Neither AI nor humans could have achieved the outcome without the other
• Continuous learning
  All components of the socio-technical system learn from each other through experience

Visualization

Distribution of roles in hybrid intelligence (Dellermann et al. 2019, 640)

Implications

According to Peeters et al. (2021) following conclusions can be drawn:

• Intelligence should not be studied at the level of individual humans or AI-machines, but at the group level of humans and AI-machines working together.
• Increasing the intelligence of a system should be achieved by increasing the quality of the interaction between its constituents rather than the intelligence of the constituents themselves.
• Both human as well as artificial intelligence can be regarded as very shallow when considered in isolation.
• No AI is an island (it’s about the system of systems).

Examples

Robots in de klas

A team consisting of a remedial teacher, an educational therapist, and a Nao robot collaborate to support a child with learning difficulties. The robot provides expertise and advice while also helping the child stay focused and engaged.

Spawn

The musician Holly Herndon created “Spawn,” an AI system that generates unique music different from her usual style. By using Spawn as a tool, Holly is able to avoid creating music that repeats her previous works but to to expand the possibilities of their music.

GitHub Copilot

In collaborative coding, Copilot can engage in back-and-forth dialogue about software design decisions, propose implementations, and explain reasoning about technical approaches - moving beyond simply generating code.

What examples do come to your mind?

Q&A

Literature

Adami, Elisabetta. 2016. “Introducing Multimodality.” The Oxford Handbook of Language and Society, 451–72.

Carroll, John Bissell. 1993. Human Cognitive Abilities: A Survey of Factor-Analytic Studies. 1. Cambridge university press.

Dellermann, Dominik, Philipp Ebel, Matthias Söllner, and Jan Marco Leimeister. 2019. “Hybrid Intelligence.” Business & Information Systems Engineering 61: 637–43.

Ekman, Paul, Wallace V Friesen, Maureen O’sullivan, Anthony Chan, Irene Diacoyanni-Tarlatzis, Karl Heider, Rainer Krause, et al. 1987. “Universals and Cultural Differences in the Judgments of Facial Expressions of Emotion.” Journal of Personality and Social Psychology 53 (4): 712.

European Commission. 2024. “Artificial Intelligence in the European Commission — a Strategic Vision to Foster the Development and Use of Lawful, Safe and Trustworthy Artificial Intelligence Systems in the European Commission” C(2024) 380.

Kahneman, Daniel. 2011. Thinking, Fast and Slow. macmillan.

Lee, Hyuck-Gi, Sungwon Chung, and Won-Hee Lee. 2013. “Presence in Virtual Golf Simulators: The Effects of Presence on Perceived Enjoyment, Perceived Value, and Behavioral Intention.” New Media & Society 15 (6): 930–46.

Molina, Martin. 2020. “Intelligent Systems.” Master Course (Lecture Slides).

Peeters, Marieke MM, Jurriaan van Diggelen, Karel Van Den Bosch, Adelbert Bronkhorst, Mark A Neerincx, Jan Maarten Schraagen, and Stephan Raaijmakers. 2021. “Hybrid Collective Intelligence in a Human–AI Society.” AI & Society 36: 217–38.

Picard, Rosalind W. 2000. Affective Computing. MIT press.

Russel, Stuart, and Peter Norvig. 2022. Artificial Intelligence: A Modern Approach. Harlow: Pearson Education.

Sternberg, Robert J et al. 1985. Beyond IQ: A Triarchic Theory of Human Intelligence. CUP Archive.

Suh, Ayoung, and Jane Prophet. 2018. “The State of Immersive Technology Research: A Literature Analysis.” Computers in Human Behavior 86: 77–90.

Tao, Jianhua, and Tieniu Tan. 2005. “Affective Computing: A Review.” In International Conference on Affective Computing and Intelligent Interaction, 981–95. Springer.

Wiener, Norbert. 1960. “Some Moral and Technical Consequences of Automation.” Science 131 (3410): 1355–58.

Footnotes

1. The capacity to work in a complex environment is described as agency

2. Cognitive abilities are, for instance, perception and language

3. Intelligent behavior is reflected, for instance, by rationality and learning

4. Perception refers to AI systems’ ability to gather and interpret data from their environment through sensors, cameras, microphones, or data inputs

5. Deliberation refers to the computational processes of analyzing information, drawing conclusions, and making decisions.

6. Interaction refers to the communication capabilities with AI systems, including natural language processing and generation.

7. Action control refers to the mechanisms that translate decisions into actions or outputs in the environment.

8. ”Affect” is basically a synonym for emotion.