ST 2018: AI for the Transfer of Control from Autonomous Systems to Humans

Autonomous systems, like self-driving cars and collaborative robots, must occasionally ask people around them for help in anomalous situations. They form a new generation of interaction platforms that provide a comprehensive multimodal presentation of the current situation in real-time, so that a smooth Transfer of Control (ToC) to human agents is guaranteed. Several scientific questions are associated with this ToC, including what should cause a ToC, when and how to notify a user, and how to manage many of these situations. In this seminar, we will investigate several methods of Artificial Intelligence (AI) that may be applied to these challenges.

This seminar covers theoretical and practical aspects of the ToC challenges. Attendees will investigate a topic or particular scenario based on scientific publications, as well as experiment with real data or implement a small demonstration.

Registration is now closed. We will provide the final list of topics, presentation dates, references, and supervisors very soon.


  • Good English skills (Literature will be English)
  • Basic programming skills (the practical part will involve some programming)
  • For the Machine Learning topics, basic ML knowledge is recommended

Grading Criteria:

  • Presentation of a topic based on a scientific paper
  • Active participation in the discussion of presented topics and moderation of one session
  • Realization of a practical assignment, e.g. implementation or model creation (in groups of 2-3 people)



1) Cause of Transfer

Determining cases when a transfer of control should occur and to whom control should be transferred on the example of autonomous robots. Given their sensors, abilities (actuators) and the environment (human positions and skills), a decision is made.

AI Methods: Robot Programming, Context Modeling
Practical Assignments*: Monitoring and Decision Module

2) Time of Transfer

When should an autonomous system initiate transfer of control to a human? On the example of autonomous vehicles, we investigate what is the best time to transfer control from a system to a human based on personal (e.g. reaction time) and situational (e.g. driving speed) factors.

AI Methods: Machine Learning, Deep Learning
Practical Assignments*: Prediction Model

3) Mode of Transfer

The transfer can be communicated using a number of channels and interaction options. We investigate different multimodal concepts, cognitive aspects, and multi-level dialogue.

AI Methods: Multimodal Interaction Design, Dialogue Development
Practical Assignments*: Dialogue System for ToC

4) Management of Transfer

In an environment where multiple human agents can receive control from multiple robots on a regular basis, they need to be able to have an overview of the autonomous agents and their situation. On the example of a retail bot, we create a dashboard that gives human agents an overview, alerts, and a way to return control to the robot.

AI Methods: Situation Summary, Plan Generation, Return of Control
Practical Assignments*: Development of a Management Dashboard

* The current assignment topics are subject to change / extension.



In case you don’t have access to your paper, please contact your supervisor

Date Name Paper Moderator Supervisor
 07.05.18  Osama Haroo  A Review of Eye Gaze in Virtual Agents, Social Robotics and HCI Baris Cakar  Florian Daiber
 07.05.18  Tri Huynh  “Take over!” How long does it take to get the driver back into the loop? Hassan Kanso  Florian Daiber
 14.05.18  Baris Sönmez   Supporting Trust in Autonomous Driving Filip Josheski  Florian Daiber
 14.05.18  Hamza Anwar  Minimum Time to Situation Awareness in Scenarios Involving Transfer of Control from Automated Driving Suite Mohammed Adnan Sirus  Rafael Math
 28.05.18  Payman Goodarzi   Emergency, automation off: Unstructured transition timing for distracted drivers of automated vehicles Ruben Garcia Ucharima  Rafael Math
 28.05.18  Baris Cakar   Takeover Time in Highly Automated Vehicles: Noncritical Transitions to and From Manual Control Atika Akmal  Guillermo Reyes
 04.06.18  Hassan Kanso   Incremental learning algorithms and applications Johannes Schulz  Guillermo Reyes
 04.06.18  Filip Josheski   Human-level control through deep reinforcement learning Amr Gomaa  Winfried Schuffert
 11.06.18  Mohammed Adnan Sirus   Virtual to real reinforcement learning for autonomous driving Shiya Wang  Winfried Schuffert
11.06.18  Ruben Garcia Ucharima   Guiding attention in controlled real-world environments Maha Siddiqui  Michael Feld
 Atika Akmal   Natural language generation as incremental planning under uncertainty Adina Pohle  Magdalena Kaiser
 Johannes Schulz   Semantically conditioned lstm-based natural language generation for spoken dialogue systems Melvin Chelli  Magdalena Kaiser
 Amr Gomaa   A new model for generating multimodal referring expressions Kaleem Ullah  Magdalena Kaiser
 Shiya Wang   Adaptive probabilistic fission for multimodal systems Ibrahim Atwi  Yannick Körber
 Maha Siddiqui   Context-based generation of multimodal feedbacks for natural interaction in smart environments Khaleel Asyraaf Mat Sanusi  Yannick Körber
 Adina Pohle   An assistive robot to support dressing strategies for planning and error handling Osama Haroo  Tim Schwartz
 Melvin Chelli   Multimodal execution monitoring for anomaly detection during robot manipulation Tri Huynh  Tim Schwartz
 Kaleem Ullah  Enhancing Fault Tolerance of Autonomous Mobile Robots Baris Sönmez  Tim Schwartz
 Ibrahim Atwi   Improved human–robot team performance through cross-training Hamza Anwar  Tim Schwartz
 Khaleel Asyraaf Mat Sanusi   Effects of anticipatory action on human-robot teamwork efficiency, fluency, and perception of team Payman Goodarzi  Tim Schwartz


Practical Assignments