Agentic AI

Business Value Creation with IT/AI (BVC)

Andy Weeger

Neu-Ulm University of Applied Sciences

March 16, 2026

Learning objectives

After completing this unit, you will be able to:

1. Explain human, collective, and artificial intelligence and their complementary strengths.
2. Describe the evolution from rule-based agents to agentic AI systems and their key characteristics.
3. Analyse how agentic AI creates and conditions business value in organisations.
4. Design and evaluate human-AI collaboration considering governance, explainability, and accountability requirements.
5. Reflect on the ethical and organisational implications of deploying autonomous AI systems.

Intelligence

Discussion

What is intelligence?

Definition

Intelligence is the ability to accomplish complex goals, learn, reason, and adaptively perform effective actions within an environment. Gottfredson (1997)

Or more concisely: think and act — humanly and/or rationally.

This definition is deliberately broad — it covers humans, groups, machines, and hybrid systems alike. Ask students: “Is this definition technology-neutral?” (Yes — which is exactly why it is useful here.) Emphasise the four verbs: accomplish, learn, reason, act. All four apply to AI systems in different degrees, and tracking which degrees is what this session is about.

Human intelligence

Human intelligence “covers the capacity to learn, reason, and adaptively perform effective actions within an environment, based on existing knowledge. This allows humans to adapt to changing environments and act towards achieving their goals.” Dellermann et al. (2019, p. 632)

Sternberg et al. (1985) proposes three distinctive dimensions:

• Componential (analytica) intelligence — the ability to break down complex information and apply logical processes to find the most efficient solution
• Experiential (creative) intelligence — the ability to synthesize prior knowledge to navigate novel situations and automate new tasks
• Contextual (practical) intelligence — the ability to read environmental demands and adapt your behavior (or the environment) to achieve success

Sternberg’s triarchic theory is useful here not as an end in itself, but as a way to highlight what humans do well that AI has historically struggled with: contextual intelligence (reading implicit social cues, adapting to ambiguous environments) and experiential intelligence (generalising from small numbers of novel experiences). Componential intelligence — analytical problem-solving — is the domain where AI has made the most impressive progress.

Cognitive architecture

Kahneman (2011) distinguishes two modes of human cognition:

• System 1: fast, automatic, intuitive
  Efficient for routine decisions, but prone to bias and heuristic errors
• System 2: slow, deliberate, effortful
  Accurate for complex analysis, but resource-intensive and easily fatigued

Both modes have blind spots. AI augmentation can (under the right conditions) compensate for System 1 biases without overwhelming System 2 capacity.

This is motivation for AI augmentation from the human side. System 1 is why humans make predictable errors in forecasting, risk assessment, and pattern recognition at scale. AI can provide a kind of disciplined “second opinion.” However — and this is the critical caveat that Block E will develop — AI systems can introduce their own systematic biases, and explanations of AI reasoning can trigger new System 1 shortcuts (confirmation bias). Mention this as a preview so students keep it in mind.

Collective intelligence

Collective intelligence refers to “[…] groups of individuals acting collectively in ways that seem intelligent.” Malone (2015, p. 3)

The concept implies that under certain conditions, a (large) group of homogeneous individuals can outperform any single individual or even a single expert (Leimeister, 2010).

Today, research increasingly focuses on hybrid collective intelligence: connecting heterogeneous agents (e.g., humans and machines) so that they combine complementary intelligence and act more intelligently together (Malone, 2015).

The shift from homogeneous to heterogeneous collective intelligence is the conceptual pivot of the whole unit. Classical crowd wisdom (Galton’s ox-weight estimation, prediction markets) relies on averaging out human errors. Hybrid collective intelligence is more ambitious: it combines fundamentally different types of reasoning — statistical pattern recognition at scale on the machine side, contextual judgment and creativity on the human side. Emphasise that this combination is only superior under certain conditions, which is exactly what the research in Session 2 will specify.

Artificial intelligence

The term artificial intelligence describes systems that perform “[…] activities that we associate with human thinking, activities such as decision-making, problem solving, learning […]” Bellman (1978, p. 3)

AI can be defined as “[…] the art of creating machines that perform functions that require intelligence when performed by people […]” Kurzweil et al. (1990, p. 117)

The basic idea: systems that can analyse their environment, adapt to new circumstances, and act in ways that advance specified goals — without explicit programming for every situation.

Note the historical framing: both definitions are essentially functional — AI is whatever machines do that we previously considered to require human intelligence. This is a moving target (the “AI effect”: as soon as AI can do X, X is no longer considered to require intelligence). What is stable is the underlying aspiration: systems that can reason and act under uncertainty. Today’s large language models and agentic systems represent the latest frontier of that aspiration.

Complementary strengths

Figure 1: Complementary strengths of humans and machines (Dellermann et al., 2019, p. 640)

Walk through the figure. Humans bring: contextual understanding, creativity, ethical judgment, social/emotional intelligence, robustness to novelty. Machines bring: processing large volumes of structured data at high speed, consistency, pattern recognition at scale, perfect recall of stored information, and freedom from certain emotional biases. Neither side is uniformly superior. The value of hybrid systems lies precisely in combining both sets of strengths in tasks where they are complementary.

Bridge: Hemmer et al. (2025) (EJIS 2025) formalises this: two sources of complementarity drive performance advantages in hybrid systems. First, information asymmetry — humans and AI have access to different types of information. Second, capability asymmetry — humans and AI are differentially capable of certain tasks. Both must exist for genuine complementarity. Mention this for now; we will return to design implications in Session 2.

Discussion

What can AI do — and what can’t it?

• Where does AI surpass humans?
  
• Where do humans surpass AI?
  
• Where should they work together?

Think individually for two minutes, then discuss with a partner.

06:00

AI strengths: pattern recognition in large datasets, consistency, speed, 24/7 availability, recall.

Human strengths: moral judgment, handling true novelty, social interaction, accountability, creative leaps.

“Together” examples: medical diagnosis (AI on imaging, human on patient context), legal research (AI on case law, human on strategy), financial risk (AI on signals, human on judgment calls).

Agent architectures

Rational agents

Figure 2: Rational agents interact with environments through sensors and actuators

A rational agent is anything that (1) perceives its environment through sensors, (2) decides on actions by mapping percept sequences to outputs, and (3) acts through actuators. “Rational” means the agent acts to maximise its expected performance measure, given its built-in knowledge, its percept history, and its available actions. Key point: rationality is not the same as perfection — it requires no omniscience, only the best possible action given available information. This will matter when we discuss agentic AI acting under uncertainty.

Performance measure

If we use, to achieve our purposes, a mechanical agency with those operations we cannot interfere once we have started it […] we had better be quite sure that the purpose built into the machine is the purpose which we really desire. Wiener (1960, p. 1358)

Formulating a performance measure correctly is difficult — and a reason to be careful.

Wiener’s warning from 1960 is remarkably prescient for the age of agentic AI. The problem of misaligned goals — where an agent optimises the stated objective but not the intended one — is one of the central challenges in AI safety. Classic example: an agent instructed to “maximise paperclip production” converts all matter to paperclips. Less extreme but practically relevant: a recommendation algorithm optimised for engagement time maximises outrage content. The performance measure is specified by humans and is therefore only as good as human foresight. Bridge to governance in Block F.

Rationality vs. perfection

Rationality is not the same as perfection.

• Rationality maximizes expected performance.
• Perfection maximizes actual performance.
• Perfection requires omniscience.
• Rational choice depends only on the percept sequence to date.

Performance standards

To understand the engineering limits of AI, we distinguish between three standards:

Metric	Definition	Info Requirement	Feasibility
Rationality	Maximizing expected performance	Percept sequence + prior knowledge	High: The engineering standard
Omniscience	Knowing the actual outcome of actions	Complete future and present data	Impossible: Requires a “crystal ball”
Perfection	Maximizing actual performance	Requires Omniscience	Impossible in unpredictable worlds

Overcoming ignorance

To bridge the gap between initial ignorance and rational behavior, agents must utilize information gathering and learning.

Since agents lack omniscience, they must be designed to:

• Information gathering: take actions specifically to modify future percepts (e.g., looking both ways before crossing a street).
• Learning: modify their internal agent function based on experience to improve performance over time.

As the environment is usually not completely known a priori and not completely predictable, these are vital parts of rationality (Russel & Norvig, 2022, p. 59).

Example

The vacuum cleaner needs to explore an initially unknown environment (exploration) to maximize its expected performance. A vacuum cleaner that learns to predict where and when additional dirt will appear will do better than one that does not.

Simple reflex agents

Figure 3: A simple reflex agent

Simple reflex agents select actions based solely on the current percept — no memory, no history. They work only when the correct decision can be made from the present state alone, i.e., the environment is fully observable. Classic example: a thermostat responds only to the current temperature. Weakness: completely blind to temporal patterns. Business relevance: many legacy rule-based systems (e.g., simple fraud rules: if transaction > €5,000 in foreign country, flag) are simple reflex agents. They work reasonably in stable, predictable environments but fail in dynamic ones.

Model-based reflex agents

Figure 4: A model-based reflex agent

Model-based agents maintain an internal model of the world — they track how it changes independently of the agent (transition model) and how sensor readings relate to world states (sensor model). This allows handling of partially observable environments. Example: a self-driving car maintains a model of where other vehicles are, even when temporarily out of sensor range. Business relevance: a supply chain monitoring system that tracks inventory levels and supplier reliability over time. Key limitation: they still act on fixed condition–action rules — they do not have goals beyond “if condition, then action.”

Goal-based agents

Figure 5: A model-based, goal-based agent

Goal-based agents shift from “if-then rules” to “what state do I want to reach?” They reason about the future: “If I take this action, will I get closer to my goal?” This requires search and planning algorithms. The performance measure is now a binary goal state rather than a utility function. Key advantage: much more flexible — you change behaviour by changing the goal, not by reprogramming rules. Business relevance: a route-planning system that finds the fastest path to a delivery destination. Key limitation: cannot handle conflicting goals or partial goal achievement — only goal reached / not reached.

Utility-based agents

Figure 6: A model-based, utility-based agent

Utility-based agents replace binary goal success with a utility function — a real-valued measure of how “good” each state is. This allows: (1) handling conflicting goals (the utility function specifies the trade-off), (2) handling uncertainty (expected utility = probability-weighted average over outcomes). Real-world example: route planning that balances speed, cost, and scenic preference simultaneously. Business relevance: a trading algorithm that balances expected return, risk, and transaction costs. Key challenge: utility functions are hard to specify — Wiener’s warning again.

Learning agents

Figure 7: A learning agent

Learning agents can modify their own behaviour based on experience — they are not limited to their initial programming. Four components: (1) Learning element: improves performance through data, interactions, and feedback; (2) Performance element: the “acting” part, executes tasks based on current knowledge; (3) Critic: evaluates performance relative to a standard; (4) Problem generator: identifies opportunities for learning, including exploratory actions that may be suboptimal short-term. Key advance: greater autonomy. The agent can handle situations its designers did not anticipate. Bridge: this leads directly to agentic AI.

Evolution of agents

The evolution of AI agents

 

 

 

The trajectory is clear: each generation increases autonomy, reduces the need for human specification, and expands the range of situations the agent can handle.

• Simple reflex → fixed rules. Model-based → world tracking.
• Goal-based → flexible objective.
• Utility-based → handling trade-offs.
• Learning → self-improvement.

Agentic AI (next slide): extended autonomy over long-horizon, multi-step tasks with minimal supervision.

This evolution is why agentic AI is qualitatively different from earlier AI systems — not just faster or more accurate, but differently structured in its relationship to human oversight.

Agentic AI

Definition

Agentic AI is an emerging paradigm in AI that refers to autonomous systems designed to pursue complex goals with minimal human intervention. Acharya et al. (2025, p. 18912)

Core characteristics

• Autonomy & goal complexity: handles multiple complex goals simultaneously; operates independently over extended periods
• Adaptability: functions in dynamic and unpredictable environments; makes decisions with incomplete information
• Independent decision-making: learns from experience; reconceptualizes approaches based on new information

Stress the qualitative differences from classical AI: (1) multi-step, long-horizon tasks — not just responding to a single prompt but managing a workflow over time; (2) operating with minimal human intervention — the system takes initiative, not just responds; (3) real-world action — agentic systems can execute code, browse the web, send emails, call APIs. This changes the governance question fundamentally: we are no longer talking about a system that produces outputs humans act on; we are talking about a system that itself acts in the world.

Berente et al. (2021) (MIS Quarterly 2021) frame this in terms of three dimensions: autonomy (acting with less human guidance), learning (improving through experience), and inscrutability (opacity in reasoning). These three dimensions jointly create new management challenges that did not exist with traditional AI — they cannot be addressed with the same governance approaches used for rule-based systems.

Agentic AI vs. traditional AI

Comparison of traditional AI and agentic AI based on Acharya et al. (2025)

Feature	Traditional AI	Agentic AI
Primary purpose	Task-specific automation	Goal-oriented autonomy
Human intervention	High (predefined parameters)	Low (autonomous adaptability)
Adaptability	Limited	High
Environment interaction	Static or limited context	Dynamic and context-aware
Learning type	Primarily supervised	Reinforcement and self-supervised
Decision-making	Data-driven, static rules	Autonomous, contextual reasoning

Briefly walk through the table. The key column is “Human intervention”: moving from high (human must specify every parameter and exception) to low (the system navigates novel situations on its own). This is what creates business value — tasks that required constant human attention can now be delegated — but also what creates governance risk — the system may pursue its goal in ways that are technically correct but organisationally undesirable. Both sides of this trade-off are central to the rest of the course unit.

Workflow patterns in agentic systems

Anthrophic (2024) discusses five key patterns for designing agentic AI workflows:

1. Prompt chaining: output of one step becomes input to the next; creates complex multi-step reasoning flows
2. Routing: directs tasks to specialised components based on type; improves efficiency through targeted processing
3. Parallelisation: processes independent subtasks simultaneously; increases throughput
4. Orchestrator-workers: central orchestrator delegates to specialised worker agents; manages coordination and integration
5. Evaluator-optimizer: separate components generate, evaluate, and refine; enables iterative quality improvement

These patterns are important for students designing systems in their projects. The four most practically relevant: prompt chaining is the simplest — chain LLM calls like a workflow. Orchestrator-workers is the most powerful but hardest to govern: a central agent autonomously decomposes a problem and delegates to specialists. Evaluator-optimizer is most similar to how humans work: generate a draft, evaluate it, revise. Emphasise that the pattern choice affects both performance and how much human oversight is feasible — orchestrator-worker systems are harder to supervise than sequential prompt chains.

Workflow mapping exercise

Identify a process relevant to your project challenge.

Choose one process step from your project and work through these questions in pairs:

1. Which workflow pattern (chaining, routing, parallelisation, orchestrator-workers, evaluator-optimizer) could structure this process?
2. What would humans still need to do?
3. Where is the risk of things going wrong?

10:00

Walk around and prompt groups who are stuck. Expected difficulty: identifying what “humans still need to do” — this is often underspecified in initial designs. Encourage students to think about: where is contextual judgment irreplaceable? Where is accountability required? Where is the information structured enough for automation? The third question (risk) is a preview of Block F. Harvest 2–3 examples briefly. Connect: the pattern you choose determines both the value and the governance requirements of your solution.

Hybrid Intelligence

Concept

The idea is to combine the complementary capabilities of humans and computers to augment each other. Dellermann et al. (2019)

This is the pivot from “what is agentic AI?” to “how do we harness it to create value?”

The concept of hybrid intelligence frames AI not as a replacement for humans, but as a partner that is enhanced by human input and enhances human capability in return.

This framing shapes the entire design space — it leads to different questions (how do we best combine?) rather than purely optimisation questions (how do we automate most efficiently?).

There is also a technology-centric counter-perspective that we should acknowledge:

Peeters et al. (2021) note that some researchers argue human intelligence is biologically limited and that sufficiently advanced AI will eventually outperform humans across all domains. This view leads to a different design logic — but it remains contested and, for now, practically premature.

Definition

Hybrid intelligence is defined as the ability to achieve complex goals by combining human and artificial intelligence, thereby reaching superior results to those each of them could have accomplished separately, and continuously improving by learning from each other. Dellermann et al. (2019, p. 640)

Main characteristics:

• Collectively: tasks are performed jointly; activities are conditionally dependent
• Superior results: neither AI nor humans could have achieved the outcome without the other
• Continuous learning: all components of the socio-technical system learn from each other through experience

The third characteristic (continuous learning) needs to be emphasized as it distinguishes hybrid intelligence from simple human-tool interaction.

A hammer does not learn from how you use it. A hybrid intelligence system does: the AI component improves through human feedback; the human component develops new mental models through AI exposure.

This mutual adaptation is what makes hybrid systems dynamically valuable — they get better over time, not just in initial deployment.

Distribution of roles

Figure 8: Distribution of roles in hybrid intelligence (Dellermann et al., 2019, p. 640)

The visualisation shows a spectrum: fully human on one end, fully automated on the other, with hybrid systems in between. The point is not that “50-50 is best” but that the optimal split depends on the task, the context, and the capabilities of both human and AI components. Tasks where one side is clearly dominant should be allocated accordingly; tasks where both sides bring irreplaceable strengths warrant genuine hybrid designs. In practice, the split often shifts over time as AI capabilities develop and as humans adapt their working practices.

The Automation–augmentation paradox

Raisch & Krakowski (2021) argue that automation and augmentation are not opposing strategies — they are interdependent:

• Overemphasising automation (machines replacing humans) creates reinforcing cycles that erode human capability, ultimately making humans less able to provide value when it matters most
• Overemphasising augmentation (humans plus machines) can under-exploit AI capabilities and leave significant efficiency potential unrealised

Effective AI deployment requires holding both logics simultaneously, managing their tensions across time and space

The question is not “automate or augment?”
— but “when, where, and how to combine both?”

Raisch & Krakowski (2021) use paradox theory from organisation science to make this argument rigorous.

Their key insight: automation and augmentation create circular causality. Automation that erodes human skills eventually undermines the oversight capacity that makes automation safe. Augmentation that preserves all human roles prevents the scale and consistency advantages of automation from materialising. The productive resolution is not a compromise but a dialectical management of the tension: design systems that automate where appropriate while deliberately maintaining and developing human capabilities where they remain essential. This has direct implications for students’ project designs — they should ask: what human capabilities must not atrophy?

Managing AI

Berente et al. (2021) identify three interdependent dimensions that define the management challenge of AI systems:

• Autonomy: AI acts with progressively less human guidance; requires careful scoping of delegated decision authority
• Learning: AI behaviour changes over time through experience; creates challenges for quality control and accountability
• Inscrutability: AI reasoning is opaque; limits the ability to audit, explain, and correct decisions

These three dimensions interact: higher autonomy + higher inscrutability creates accountability gaps. Learning + higher inscrutability can produce invisible drift in system behaviour.

The interaction effects are the key insight here. Each dimension alone is manageable: autonomy can be scoped, learning can be monitored, inscrutability can be addressed through XAI. But all three together — which characterises agentic AI — create novel governance challenges. The “invisible drift” point is particularly important: a learning agentic system deployed today may behave differently six months from now in ways that are not immediately visible, and the opacity means the drift may not be detected until a significant error occurs. This is a direct preview of the governance section in Block F.

From tools to teammates

Seeber et al. (2020) highlight a fundamental shift in how AI systems are positioned in organisations:

Traditional AI	AI as Teammates
Role: Tool to be used	Role: Active collaboration partner
Interaction: Responds to commands	Interaction: Engages proactively
Function: Task automation	Function: Complex problem-solving
Agency: Limited / directed	Agency: Autonomous with initiative
Integration: Technical system integration	Integration: Social & team integration

This shift is not just conceptual — it changes what design means. Designing AI as a tool is primarily a software engineering problem. Designing AI as a teammate is a sociotechnical design problem that involves role clarity, trust calibration, communication protocols, and institutional structures. Seeber et al. (2020) argue this shift requires new frameworks — the old categories of “user” and “system” no longer apply when AI systems take initiative, participate in defining problems, and generate solutions unprompted. Ask students: “In your project, are you designing a tool or a teammate? And does the design reflect that choice?”

Critical design areas

Seeber et al. (2020) identify three interconnected design areas for AI teammates:

1. Machine artifact design: the AI system itself: appearance, capabilities, interaction modalities
2. Collaboration design: how humans and AI work together: team composition, task allocation, workflows, communication protocols
3. Institution design: the broader context: responsibility frameworks, liability, training requirements, governance structures

These areas are interdependent: decisions in one area constrain and shape the others. Effective design requires a holistic rather than purely technical approach.

People focus only on machine artifact design (the technology) and underestimate collaboration design (how work is organised) and institution design (who is responsible for what).

• A well-designed AI system deployed in a poorly designed workflow creates chaos.
• A well-designed workflow with no clear accountability framework creates risk.

All three areas must be addressed.

Implications for hybrid intelligence

According to Peeters et al. (2021):

• Intelligence should be studied at the group level of humans and AI-machines working together — not at the level of individual components
• Increasing system intelligence means increasing the quality of interaction between components — not merely improving individual components
• Both human and artificial intelligence are shallow when considered in isolation
• No AI is an island — value emerges from the system, not the artefact

This is the main theoretical takeaway for Session 1. “No AI is an island” is a memorable encapsulation of the whole unit’s argument. It counters the tendency to evaluate AI systems by their isolated capabilities (benchmark performance, parameter count, etc.) and reframes evaluation in terms of systemic performance: what does the combined human-AI system achieve in context? This has direct implications for how students should evaluate their project solutions — not “how capable is the AI?” but “how well does the combined system perform the task?”

Bridge to Session 2: We’ve established the conceptual framework. In Session 2, we ask: under what specific conditions does hybrid intelligence actually create business value, and what must be in place (design, governance, responsibility structures) for it to do so sustainably?

Value creation with Agentic AI

Revisiting the value chain

The IT value creation process (Soh & Markus, 1995):
IT investments only translate into performance if three linked processes work:

1. IT conversion: IT expenditure lead to IT assets (requires appropriate conversion)
2. IT use: IT assets create IT impacts (usage is the critical missing link)
3. Competitive process: IT impacts foster organisational performance (depends on context and competitors)

When AI agents close the mising link (i.e., IT usage) — what changes?

The key question: the model identifies IT use as the missing link — but it conceptualises use as human behaviour (adoption, effective use, workarounds).

When agentic AI replaces or augments the human as the primary “user,” this changes the value creation logic fundamentally.

The missing link is no longer (only) human adoption — it is system design quality, governance, and the quality of human-AI interaction.

Shifting the “missing link”

The shift from human use to agent action changes where value is created and where it can break down:

Characteristic	Traditional IT	Agentic AI
Missing link	Human adoption & use	Agent design & governance
Risk	Non-adoption, misuse, workarounds	Misaligned objectives, invisible errors, drift
Remedy	Training, UX design, change mgmt	Careful design, monitoring, oversight structures
Value driver	Effective human behaviour	System-level performance & accountability

With traditional IT, the biggest risk is that people don’t use the system, or use it incorrectly.

With agentic AI, the risk profile shifts: people may over-delegate to a system that is subtly misaligned, or fail to detect gradual behavioural drift.

The “invisible error” problem is particularly dangerous: agentic systems that perform well on average but fail systematically in edge cases can cause large-scale harm before the failure is detected. Connect to IS business value taxonomy: internal/external × tangible/intangible — agentic AI can create value in all four quadrants, but also creates risk in all four.

AI-augmented decisions

Herath et al. (2024) derive seven evidence-based design principles from action design research across three business decision contexts:

1. Transparent uncertainty communication: AI should signal its confidence, not just its recommendation
2. Explainable reasoning paths: users need to understand why, not just what
3. Scoped autonomy: AI should act autonomously only within well-defined task boundaries
4. Human override capability: human judgment must remain exercisable at every stage
5. Feedback integration: systems should learn from human corrections in near-real time
6. Accountability anchoring: every AI decision output must be linked to a responsible human
7. Context-sensitive presentation: recommendations should be tailored to the decision context, not generic

Herath et al. (2024) worked with actual companies in customer segmentation, retention, and portfolio design contexts — this is applied research, not theoretical. The most important principle for students to internalise is accountability anchoring: in a world where AI makes recommendations and humans act on them, there is a structural tendency for accountability to “fall through the gap” — the AI is not accountable, and humans claim they were only following the AI. Designing systems with explicit accountability anchoring — specific humans are responsible for verifying and owning AI outputs — prevents this. This connects directly to Block F.

Valueable collaboration

Fügener et al. (2022) conducted experiments on human-AI prediction tasks and found:

Human-AI teams achieve superior performance only when AI delegates to humans — not vice versa.

Human metaknowledge, i.e., the ability to assess your own reliability in a specific context (“knowing what you know”), seems to be the critical variable:

• AI can assess its own certainty well and delegates effectively (even to low-performing humans) because it knows what it knows and what it doesn’t
• Humans, by contrast, lack metaknowledge: they cannot accurately judge their own reliability, leading to poor delegation decisions despite genuine willingness to collaborate
• This metaknowledge deficit is unconscious and cannot be explained by algorithm aversion — subjects tried to follow delegation strategies diligently and appreciated the AI support

This is a counterintuitive and important result.

The popular narrative is: let humans decide when to delegate to AI, and they will learn to use it well.

Fügener et al. (2022) show that this fails — not because humans distrust AI, but because they cannot accurately assess their own capabilities. Subjects wanted to collaborate and tried to delegate rationally, but they were systematically wrong about which cases they could handle themselves.

The AI, by contrast, delegated effectively because it could assess its own certainty. It knew which cases were difficult for it and handed those to humans — and this worked even when the humans were low performers.

The asymmetry is key: AI has good self-knowledge, humans do not. This means effective collaboration requires designing interfaces that support human self-assessment — helping users judge their own reliability — rather than assuming humans will naturally calibrate their reliance on AI.

The delegation paradox

Traditional AI design assumes “top-down” delegation: humans decide when to hand tasks to AI. However, empirical evidence suggests this is often ineffective (Fügener et al., 2022).

Why human delegation fails

• Humans cannot accurately assess their own reliability as they are systematically wrong about which cases they can handle, leading to poor delegation decisions.
• This failure is not caused by distrust of AI (i.e., AI aversion). Subjects tried to follow delegation strategies diligently and appreciated AI support, but their lack of self-knowledge undermined collaboration.

Why AI delegation works

• AI can assess its own certainty and effectively hand off difficult cases to humans. This improved performance even when the humans were low performers.
• Interfaces should support human self-assessment rather than relying on humans to calibrate their own reliance on AI naturally.

Context-dependence of value

Revilla et al. (2023) conducted a field experiment in retail demand forecasting. Their results reveal the conditionality of hybrid intelligence value:

Effectiveness of human-AI collaboration in retail demand forecasting (Revilla et al., 2023)

Context	Superior Strategy	Explanation
Short horizon, high uncertainty	Automation (AI only)	AI extracts signal from noise better; humans “tinker at the edges” and add bias
Long horizon, low uncertainty	Augmentation (human + AI)	AIML model is well-grounded; humans add contextual knowledge the algorithm misses
Short horizon, low uncertainty	Adjustable automation	Some contextual knowledge helps, but short-horizon noise limits full augmentation benefit
Long horizon, high uncertainty	Mixed	Long horizons favor human input, high uncertainty favors AI — effects partially offset

There is no universal best practice. Task context determines optimal collaboration strategy.

This is the “it depends” result with specificity. Short horizon + high uncertainty: no time for human deliberation, and patterns are too volatile for human pattern-matching — AI wins. Long horizon + low uncertainty: human judgment about structural factors, business context, and anomalies adds genuine value. The practical implication: organisations should analyse their task context before choosing an automation vs. augmentation strategy, rather than applying a blanket policy. Connect to Raisch & Krakowski: the paradox is not just philosophical — it has concrete, measurable implications for which strategy to choose in specific operational contexts.

Exercise

Where does agentic AI create value in your project?

Map your project’s core AI component to the IS business value matrix (Schryen 2013):

• Internal tangible: Cost reductions, productivity gains, efficiency improvements
• Internal intangible: Better decisions, improved capabilities, organisational learning
• External tangible: Revenue growth, market share, customer retention
• External intangible: Brand trust, customer satisfaction, reputation

Where is the primary value? Where are the risks? What conditions must hold?

10:00

• Which process, by how much, under what conditions?
• What if the AI makes systematic errors? Who bears the cost? Where would errors be most visible / most damaging?
• What must be true about data quality, user behaviour, and governance for the value to materialise?

Explainability & trust

Effective hybrid intelligence

Peeters et al. (2021) identify four properties that human-AI systems must exhibit for effective collaboration:

• Observability: an actor should make its status, knowledge of the team, task, and environment visible to collaborators
• Predictability: an actor should behave consistently so others can anticipate its actions when planning their own
• Explainability: agents should be capable of explaining their behaviour to collaborators
• Directability: collaborators should be able to re-direct each other’s behaviour when necessary

These properties enable calibrated trust (i.e., humans trusting AI appropriately): neither too much nor too little.

Calibrated trust is the target state. Automation bias (over-trust) leads to humans accepting AI errors uncritically. Algorithm aversion (under-trust) leads to humans ignoring useful AI recommendations. Both destroy value. The four properties are essentially the design conditions for calibrated trust: if AI is observable, predictable, explainable, and directable, humans can rationally calibrate how much to rely on it in specific contexts. Note that these are also the properties that make governance feasible — you cannot govern what you cannot observe, predict, or redirect.

The XAI dilemma

Bauer et al. (2023) show that AI systems providing explanations (XAI) alongside predictions may:

• Draw users’ attention excessively to explanations that confirm prior beliefs (confirmation bias) rather than the prediction itself
• Diminish employees’ decision-making performance for the task at hand
• Lead individuals to carry over biased explanatory patterns to other domains
• Decrease individual-level noise (consistency increases) but increase systematic error
• Foster differences across subgroups with heterogeneous prior beliefs

Transparency ≠ better decisions.
How XAI is designed determines whether it helps or hurts.

This is a critical and counter-intuitive finding. The policy and regulatory direction (EU AI Act, financial services regulations) is toward mandatory XAI for high-stakes AI decisions. Bauer et al. (2023) show this can backfire: explanations trigger System 1 processing (fast, associative, confirmation-seeking) rather than System 2 deliberation, making decisions less accurate on average while appearing more legitimate. Key design implication: XAI should be designed to support deliberative reasoning (System 2), not just to provide transparency for its own sake. This means: targeted explanations for high-stakes exceptions rather than universal explanations for every decision; structured interfaces that require explicit reasoning before accepting AI recommendations; and careful testing of whether explanations actually improve decisions or just increase confidence.

Types of AI explanations

Different explanation types serve different cognitive needs (Miller, 2019; Wang et al., 2019):

1. How explanations: describe the AI’s process:
   “I used features X, Y, Z to reach this conclusion”
2. Why explanations: justify the AI’s reasoning:
   “This is the dominant factor because …”
3. What-if explanations: counterfactual analysis:
   “If feature X changed, the outcome would be …”
4. Confidence indicators: uncertainty communication:
   “I am 73% confident in this recommendation”

The most effective explanation type depends on user expertise, time pressure, decision stakes, and potential for bias activation.

Novice users often benefit most from why explanations (they help build mental models); experts often prefer what-if (counterfactuals support their existing reasoning patterns); time-pressured users need concise confidence indicators. The Bauer et al. (2023) caution applies especially to why explanations in contexts where users have strong prior beliefs — this is where confirmation bias is most likely to be activated. The safest approach is to test explanation formats with representative users rather than choosing based on theoretical appeal alone.

Designing for complementarity

Hemmer et al. (2025) identify the organisational factors that enable effective human-AI complementarity:

• Digital infrastructure: quality and accessibility of data and AI tools
• Governance mechanisms: clear rules for how AI outputs are used and overridden
• Change management: deliberate support for users adapting to hybrid workflows
• Trust calibration: training and feedback mechanisms that help users calibrate AI reliance

Optimal task allocation:

• AI automates easy tasks
• Augmentation on tasks of similar difficulty
• Humans handle difficult tasks alone.

The task allocation principle from Hemmer et al. (2025) is intuitive but practically underutilised. Organisations often default to either “automate everything AI can do” or “AI supports humans on every task.” The evidence suggests a more nuanced strategy: AI should fully automate tasks where it has a clear capability advantage; augment humans on tasks where both can contribute; and step back on tasks that are genuinely above current AI capability or where human judgment is irreplaceable. The four organisational enablers (infrastructure, governance, change management, trust calibration) are all necessary — the absence of any one can prevent the others from working.

AI affordances in teams

According to Dennis et al. (2023), AI agents provide three fundamental affordances to human teams:

• Communication support: coordination and reminders, review and feedback, delegation capabilities
• Information processing support: data cataloguing, search and retrieval, information analysis, content organisation
• Process structuring: planning and scheduling, task breakdown, delivery tracking, quality assurance

These affordances enable AI to contribute to team processes in ways that complement human team members and, thus, enable superior collective outcomes.

Dennis et al. (2023)‘s affordance framework is useful for students designing their project solutions: it shifts the question from “what can AI do?” to “what does the team need?” — and then asks which AI capabilities address those needs. Process structuring support is often underexplored: many students design AI for information processing (which is well understood) but overlook the potential of AI to improve how work is organised and tracked within the team or organisation. Connect this to students’ own team work: could an AI affordance reduce coordination friction in their own projects?

Governance & responsible AI

The stakes

As agentic AI systems act autonomously, safety and accountability are critical — not optional (Shavit et al., 2023).

Three compounding factors raise the stakes:

• Autonomy: systems act without direct human instruction; errors compound before detection
• Scale: agentic systems can act on thousands of cases before a human review cycle completes
• Opacity: inscrutability makes post-hoc attribution of errors difficult

The governance question is not “how do we prevent AI from making mistakes?” — but “how do we detect, correct, and account for mistakes when they inevitably occur?”

The framing shift in the last point is important: from prevention (unrealistic; all complex systems produce errors) to detection, correction, and accountability (realistic; what good governance can actually achieve). This is the “resilient AI governance” perspective. It accepts that agentic AI will sometimes act in ways that are wrong or harmful, and designs governance structures that limit the impact of those errors and ensure they are addressed. Connect to Wiener’s performance measure warning from Session 1: the difficulty of specifying what we actually want (performance measure formulation) means errors are nearly inevitable; governance must create the structures to catch and correct them.

Practices for safe operation

Shavit et al. (2023) proposes a series of practices for the responsible deployment of agentic AI:

• Suitability assessment: evaluate whether the agent is appropriate for the specific task and context
• Scope limitation: restrict agent action to well-defined domains; require approval for consequential actions
• Default behaviour establishment: define explicit defaults for ambiguous situations
• Traceability: ensure all agent actions can be logged and attributed
• Automated monitoring: implement real-time anomaly detection
• Attributability: every action must be linkable to an accountable actor (human or system)
• Interruptibility: the agent must be stoppable; human control must be maintainable at all times

Interruptibility is non-negotiable for any agentic system deployed in a real organisational context. An agentic system that cannot be stopped is not safe to deploy, regardless of its performance.

Attributability — the question “who is responsible?” must have a clear answer for every AI action. Without this, organisations face legal and reputational exposure that can far outweigh the value created by the system.

The principal-agent view

Jarrahi & Ritala (2025) apply principal-agent theory to reframe AI agents as delegated actors rather than autonomous systems:

• Principals (organisations, humans) delegate tasks to agents (AI systems) in exchange for performance
• The core problem: information asymmetry — agents have knowledge principals lack; interests may diverge

Three design principles follow:

• Guided autonomy: AI acts within principal-defined constraints, not freely
• Individualisation: AI behaviour adapts to the specific context and stakeholder
• Adaptability: AI can revise its approach as contexts change, within defined limits

This framing keeps accountability firmly with the principal — AI is an agent, not an autonomous actor with its own standing.

Principal-agent theory is well-established in economics and organisation science (e.g., executive compensation, outsourcing contracts). Jarrahi & Ritala (2025)’s contribution is to show it applies directly to AI: the agency problem (the agent acts in its own interest, or in ways the principal didn’t specify) is not uniquely a human problem — it is a structural feature of delegating complex tasks to any actor whose reasoning is not fully visible to the principal. The implication: organisations should treat agentic AI as they would treat a new employee in a high-trust role — establish clear scope, monitor performance against explicit criteria, build in escalation mechanisms, and maintain the right to override. This is emphatically not about distrust; it is about well-designed accountability structures.

Responsible AI governance

Papagiannidis et al. (2025) identify a systematic gap between AI principles and AI governance:

• AI principles: high-level commitments: ethics, transparency, fairness, accountability, privacy
• Governance mechanisms: operational structures: oversight processes, accountability roles, audit procedures, escalation paths

Their framework spans four phases:

• Design phase: embed governance requirements into system architecture from the start
• Execution phase: operational oversight during deployment; exception handling protocols
• Monitoring phase: continuous tracking of system behaviour, performance drift, and error patterns
• Evaluation phase: periodic review of whether the system is meeting its intended purpose

The principles-governance gap is well-documented and practically significant. Many organisations (and many AI providers) have published ethics principles — responsible AI, fair AI, trustworthy AI — but have not operationalised them into governance mechanisms that actually change how AI systems are built and operated. Papagiannidis et al. (2025) provide a framework for closing this gap. The four-phase structure maps onto the system development lifecycle: you cannot retrofit governance into production systems as an afterthought (evaluation-only governance); it must be embedded from the design phase. Connect to students’ projects: governance design is not a separate activity to be added after the prototype — it should be part of the design from day one.

Ethical dimensions

Agentic AI raises ethical questions that governance frameworks must address:

• Bias and fairness: AI trained on historical data can perpetuate and amplify existing inequalities; emergent effects at scale can be unforeseen (Peeters et al., 2021)
• Responsibility attribution: as AI acts more autonomously, the question of “who is responsible?” becomes harder — and more important
• Human agency: systems designed to reduce human effort may inadvertently reduce human capacity and meaningful judgment
• Regulatory context: EU AI Act: risk-based classification; high-risk AI systems require conformity assessment, human oversight, and transparency

The bias point connects to empirical evidence Peeters et al. (2021) cite: hiring software, credit scoring, and recidivism prediction systems have all produced documented discriminatory outcomes — not because they were designed to discriminate, but because they were trained on historically biased data and deployed in contexts that amplified rather than corrected those biases. The governance challenge is that these “emergent effects at the systemic level” are often invisible until they affect many people. The EU AI Act’s risk classification (prohibited, high-risk, limited-risk, minimal-risk) is the current EU regulatory response. For students’ projects: if their solution involves decisions affecting individuals (employment, credit, healthcare, etc.), it likely falls in the high-risk category and would require formal conformity assessment under the Act.

Exercise

What governance does your project solution need?

1. Suitability: Is agentic AI appropriate for this task? What are the risks of errors?
2. Scope: What decisions should the agent never make autonomously?
3. Accountability: Who is responsible for the agent’s outputs?
4. Monitoring: How will you detect errors or unwanted behaviour?
5. Override: Under what circumstances must a human be able to intervene?

10:00

Synthesis

An integrated model

Agentic AI creates value not through autonomy alone — but through thoughtful design of human-AI interaction and clear governance.

The key connections:

• Intelligence is complementary — neither humans nor AI alone are sufficient for complex, high-stakes tasks
• Agentic AI shifts the missing link from human adoption to system design and governance
• Hybrid intelligence is the productive frame — value emerges from the system, not the artefact
• Complementarity requires design across three levels: artifact, collaboration, and institution
• Governance is an enabling condition — without it, autonomy creates risk rather than value

Transfer to your projects

Three questions your project design must answer:

1. Complementarity: What do humans contribute that your AI system cannot? How does the design ensure this contribution is made?
2. Value: What IS business value does your solution create, and under what conditions does it actually materialise?
3. Governance: Who is accountable for AI actions in your solution? How are errors detected, corrected, and attributed?

A solution that cannot answer all three questions is not ready for deployment, regardless of its technical performance.

Looking ahead

Multi-agent systems, foundation models as agents, and AI-to-AI coordination represent the next wave — raising the same questions of complementarity, value, and governance at a larger scale.

The concepts we have discussed today (hybrid intelligence, managed autonomy, responsible governance) are not specific to current AI technology. They are enduring frameworks for navigating the evolving boundary between human and machine capability.

Q&A

Literature

Acharya, D. B., Kuppan, K., & Divya, B. (2025). Agentic AI: Autonomous intelligence for complex goals–a comprehensive survey. IEEE Access.

Anthrophic. (2024). Building effective agents. Anthropic Research Team; https://www.anthropic.com/engineering/building-effective-agents.

Bauer, K., Zahn, M. von, & Hinz, O. (2023). Expl (AI) ned: The impact of explainable artificial intelligence on users’ information processing. Information Systems Research.

Bellman, R. (1978). An introduction to artificial intelligence: Can computers think? Thomson Course Technology.

Berente, N., Gu, B., Recker, J., & Santhanam, R. (2021). Managing artificial intelligence. MIS Quarterly, 45(3), 1433–1450. https://doi.org/10.25300/MISQ/2021/16274

Dellermann, D., Ebel, P., Söllner, M., & Leimeister, J. M. (2019). Hybrid intelligence. Business & Information Systems Engineering, 61, 637–643.

Dennis, A. R., Lakhiwal, A., & Sachdeva, A. (2023). AI agents as team members: Effects on satisfaction, conflict, trustworthiness, and willingness to work with. Journal of Management Information Systems, 40(2), 307–337.

Fügener, A., Grahl, J., Gupta, A., & Ketter, W. (2022). Cognitive challenges in Human–Artificial Intelligence collaboration: Investigating the path toward productive delegation. Information Systems Research, 33(2), 678–696. https://doi.org/10.1287/isre.2021.1079

Gottfredson, L. S. (1997). Mainstream science on intelligence: An editorial with 52 signatories, history, and bibliography. In Intelligence (1; Vol. 24, pp. 13–23). JAI.

Hemmer, P., Schemmer, M., Kühl, N., Vössing, M., & Satzger, G. (2025). Complementarity in Human–AI collaboration: Concept, sources, and evidence. European Journal of Information Systems, 34(6), 979–1002. https://doi.org/10.1080/0960085X.2025.2475962

Herath, S., Shrestha, Y. R., & Krogh, G. von. (2024). Design principles for artificial intelligence-augmented decision making: An action design research study. European Journal of Information Systems, 34(2), 207–229. https://doi.org/10.1080/0960085X.2024.2330402

Jarrahi, M. H., & Ritala, P. (2025). Rethinking AI agents: A principal–agent perspective. California Management Review. https://doi.org/10.1177/00081256251320040

Kahneman, D. (2011). Thinking, fast and slow. macmillan.

Kurzweil, R., Richter, R., Kurzweil, R., & Schneider, M. L. (1990). The age of intelligent machines (Vol. 580). MIT press Cambridge.

Leimeister, J. M. (2010). Collective intelligence. Business & Information Systems Engineering, 2, 245–248.

Malone, T. (2015). Handbook of collective intelligence; bernstein, MS, ed. The MIT Press: Cambridge/London, UK.

Miller, T. (2019). Explanation in artificial intelligence: Insights from the social sciences. Artificial Intelligence, 267, 1–38.

Papagiannidis, E., Mikalef, P., & Conboy, K. (2025). Responsible artificial intelligence governance: A review and research framework. The Journal of Strategic Information Systems, 34(2), 101885. https://doi.org/10.1016/j.jsis.2024.101885

Peeters, M. M., Diggelen, J. van, Van Den Bosch, K., Bronkhorst, A., Neerincx, M. A., Schraagen, J. M., & Raaijmakers, S. (2021). Hybrid collective intelligence in a human–AI society. AI & Society, 36, 217–238.

Raisch, S., & Krakowski, S. (2021). Artificial intelligence and management: The automation–augmentation paradox. Academy of Management Review, 46(1), 192–210. https://doi.org/10.5465/amr.2018.0072

Revilla, E., Saenz, M. J., Seifert, M., & Ma, Y. (2023). Human–Artificial Intelligence collaboration in prediction: A field experiment in the retail industry. Journal of Management Information Systems, 40(4), 1071–1098. https://doi.org/10.1080/07421222.2023.2267317

Russel, S., & Norvig, P. (2022). Artificial intelligence: A modern approach. Pearson Education.

Seeber, I., Bittner, E., Briggs, R. O., De Vreede, T., De Vreede, G.-J., Elkins, A., Maier, R., Merz, A. B., Oeste-Reiß, S., Randrup, N., et al. (2020). Machines as teammates: A research agenda on AI in team collaboration. Information & Management, 57(2), 103174.

Shavit, Y., Agarwal, S., Brundage, M., Adler, S., O’Keefe, C., Campbell, R., Lee, T., Mishkin, P., Eloundou, T., Hickey, A., et al. (2023). Practices for governing agentic AI systems. Research Paper, OpenAI.

Soh, C., & Markus, M. L. (1995). How IT creates business value: A process theory synthesis. ICIS 1995 Proceedings, 4.

Sternberg, R. J. et al. (1985). Beyond IQ: A triarchic theory of human intelligence. CUP Archive.

Wang, D., Yang, Q., Abdul, A., & Lim, B. Y. (2019). Designing theory-driven user-centric explainable AI. Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, 1–15.

Wiener, N. (1960). Some moral and technical consequences of automation. Science, 131(3410), 1355–1358.