The Responsibility Quantification Model of Human Interaction with Automation

Nir Douer, Joachim Meyer*

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

Abstract

Intelligent systems and advanced automation are involved in information collection and evaluation, decision-making, and the implementation of chosen actions. In such systems, human responsibility becomes equivocal. Understanding human causal responsibility is particularly important when systems can harm people, as with autonomous vehicles or, most notably, with autonomous weapon systems (AWSs). Using information theory, we developed a responsibility quantification (ResQu) model of human causal responsibility in intelligent systems and demonstrated its applications on decisions regarding AWS. The analysis reveals that comparative human responsibility for outcomes is often low, even when major functions are allocated to the human. Thus, broadly stated policies of keeping humans in the loop and having meaningful human control are misleading and cannot truly direct decisions on how to involve humans in advanced automation. The current model assumes stationarity, full knowledge regarding the characteristic of the human and automation, and ignores temporal aspects. It is an initial step toward the development of a comprehensive responsibility model that will make it possible to quantify human causal responsibility. The model can serve as an additional tool in the analysis of system design alternatives and policy decisions regarding human causal responsibility, providing a novel, quantitative perspective on these matters. Note to Practitioners - We developed a theoretical model and a quantitative measure for computing the comparative human causal responsibility in the interaction with intelligent systems and advanced automation. Our responsibility measure can be applied by practitioners (system designers, regulators, and so on) to estimate user responsibility in specific system configurations. This can serve as an additional tool in the comparison between alternative system designs or deployment policies, by relating different automation design options to their predicted effect on the users' responsibility. To apply the model (which is based on entropy and mutual information) to real-world systems, one must deduce the underlying distributions, either from known system properties or from empirical observations, taken over time. The initial version of the model we present here assumes that the combined human-automation system is stationary and ergodic. Real-world systems may not be stationary and ergodic or cannot be observed sufficiently to allow accurate estimates of the required input of multivariate probabilities, in which case the computed responsibility values should be treated with caution. Nevertheless, the construction of a ResQu information flow model, combined with sensitivity analyses of how changes in the input probabilities and assumptions affect the responsibility measure, will often reveal important qualitative properties and supply valuable insights regarding the general level of meaningful human involvement and comparative responsibility in a system.

Original languageEnglish
Article number8978739
Pages (from-to)1044-1060
Number of pages17
JournalIEEE Transactions on Automation Science and Engineering
Volume17
Issue number2
DOIs
StatePublished - Apr 2020

Keywords

  • Analytical models
  • artificial intelligence
  • autonomous systems
  • decision making
  • human-computer interaction (HCI)
  • information theory
  • intelligent systems
  • responsibility

Fingerprint

Dive into the research topics of 'The Responsibility Quantification Model of Human Interaction with Automation'. Together they form a unique fingerprint.

Cite this