Robot & AI Governance

Setting a research agenda to mitigate overtrust in automation

Aroyo, A. M., de Bruyne, J., Dheu, O., Fosch-Villaronga, E., Gudkov, A., Hoch, H., Jones, S., Lutz, C., Sætra, H., Solberg, M., & Tamò-Larrieux, A. (2021) Overtrusting Robots: Setting a Research Agenda to Mitigate Overtrust in Automation. Paladyn Journal of Behavioral Robotics 12(1), 1-14.

There is increasing attention given to the concept of trustworthiness for artificial intelligence and robotics. However, trust is highly context-dependent, varies among cultures, and requires reflection on others’ trustworthiness, appraising whether there is enough evidence to conclude that these agents deserve to be trusted. Moreover, little research exists on what happens when too much trust is placed in robots and autonomous systems. Conceptual clarity and a shared framework for approaching overtrust are missing. In this contribution, we offer an overview of pressing topics in the context of overtrust and robots and autonomous systems. Our review mobilizes insights solicited from in-depth conversations from a multidisciplinary workshop on the subject of trust in human–robot interaction (HRI), held at a leading robotics conference in 2020. A broad range of participants brought in their expertise, allowing the formulation of a forward-looking research agenda on overtrust and automation biases in robotics and autonomous systems. Key points include the need for multidisciplinary understandings that are situated in an eco-system perspective, the consideration of adjacent concepts such as deception and anthropomorphization, a connection to ongoing legal discussions through the topic of liability, and a socially embedded understanding of overtrust in education and literacy matters. The article integrates diverse literature and provides a ground for common understanding for overtrust in the context of HRI.

Contribution to the revision of the Product Safety Directive: Serving the Consumer Safety Network

During Jan-Dec 2020 Eduard Fosch-Villaronga served as an expert at the Consumer Safety Network, advising the European Commission in the revision of the General Product Safety Directive in the Sub-Group on Artificial Intelligence (AI), connected products and other new challenges in product safety. See the opinion here.

Concepts, definitions, and considerations for healthcare robot governance

Fosch-Villaronga, E. & Drukarch, H. (2021). On Healthcare Robots. Concepts, Definitions, and considerations for healthcare robot governance. ArXiv pre-print, 1-87, https://arxiv.org/abs/2106.03468.

Although healthcare is a remarkably sensitive domain of application, and systems that exert direct control over the world can cause harm in a way that humans cannot necessarily correct or oversee, it is still unclear whether and how healthcare robots are currently regulated or should be regulated. Existing regulations are primarily unprepared to provide guidance for such a rapidly evolving field and accommodate devices that rely on machine learning and AI. Moreover, the field of healthcare robotics is very rich and extensive, but it is still very much scattered and unclear in terms of definitions, medical and technical classifications, product characteristics, purpose, and intended use. As a result, these devices often navigate between the medical device regulation or other non-medical norms, such as the ISO personal care standard. Before regulating the field of healthcare robots, it is therefore essential to map the major state-of-the-art developments in healthcare robotics, their capabilities and applications, and the challenges we face as a result of their integration within the healthcare environment. This contribution fills in this gap and lack of clarity currently experienced within healthcare robotics and its governance by providing a structured overview of and further elaboration on the main categories now established, their intended purpose, use, and main characteristics. We explicitly focus on surgical, assistive, and service robots to rightfully match the definition of healthcare as the organized provision of medical care to individuals, including efforts to maintain, treat, or restore physical, mental, or emotional well-being. We complement these findings with policy recommendations to help policymakers unravel an optimal regulatory framing for healthcare robot technologies.

On Healthcare Robots_ Concepts, definitions, and considerations for healthcare robot governance (1).pdf

Fosch-Villaronga, E. and Heldeweg, M. A., (2018) "Regulation, I Presume?" Said the Robot. Towards an Iterative Regulatory Process for Robot Governance. Computer Law and Security Review, 34(6), 1258-1277

This article envisions an iterative regulatory process for robot governance. In the article, we argue that what lacks in robot governance is actually a backstep mechanism that can coordinate and align robot and regulatory developers. In order to solve that problem, we present a theoretical model that represents a step forward in the coordination and alignment of robot and regulatory development. Our work builds on previous literature, and explores modes of alignment and iteration towards greater closeness in the nexus between research and development (R&D) and regulatory appraisal and channeling of robotics’ development. To illustrate practical challenges and solutions, we explore different examples of (related) types of communication processes between robot developers and regulatory bodies. These examples help illuminate the lack of formalization of the policymaking process, and the loss of time and resources that the waste of knowledge generated for future robot governance instruments implies. We argue that initiatives that fail to formalize the communication process between different actors and that propose the mere creation of coordinating agencies risk being seriously ineffective. We propose an iterative regulatory process for robot governance, which combines the use of an ex ante robot impact assessment for legal/ethical appraisal, and evaluation settings as data generators, and an ex post legislative evaluation instrument that eases the revision, modification and update of the normative instrument. In all, the model breathes the concept of creating dynamic evidence-based policies that can serve as temporary benchmark for future and/or new uses or robot developments. Our contribution seeks to provide a thoughtful proposal that avoids the current mismatch between existing governmental approaches and what is needed for effective ethical/legal oversight, in the hope that this will inform the policy debate and set the scene for further research.

Fosch-Villaronga, E., & Heldeweg, M.A. (2020) “Meet me halfway,” said the robot to the regulation. Linking ex-ante technology impact assessments to legislative ex-post evaluations via shared data repositories for robot governance. In: Pons-Rovira, J.L. (2020) Inclusive Robotics for a Better Society. INBOTS 2018. Biosystems & Biorobotics, vol. 25., Springer, Cham., 113-119.

Current legislation may apply to new developments. Even so, these developments may raise new challenges that call into question the applicability of this legislation. This paper explains what happens at this moment for new robot technologies. We argue that there is no formal communication process between robot developers and regulators from which policies could learn. To bridge this gap, we propose a model that links technology impact assessments to legislative ex-post evaluations via shared data repositories.

Fosch-Villaronga, E. (2015) Creation of a Care Robot Impact Assessment. WASET, International Science Journal of Social, Behavioral, Educational, Economic and Management Engineering, 9(6), 1817 – 1821.

Fosch-Villaronga, E. (2019). Robot Impact Assessment. In: Fosch-Villaronga, E. (2019). Healthcare, Robots, & the Law. Regulatin Automation in Personal Care, 39-56.

A roboticist building a robot that interacts with humans may be clueless about what regulations s/he needs to follow, whether the robot behavior needs to be regulated by design or after, or whether s/he is in charge of it. The methodology Robot Impact Assessment (ROBIA) helps robot engineers in the process to identify the legal aspects associated with their technology. A ROBIA involves setting the context, describing the robot, identifying the relevant framework, identify the risks and mitigate them. Following ROBIA, robot engineers may identify relevant safeguards to make robots safe to the whole extent of the meaning of the word.

Robot Impact Assessment_EFoschVillaronga.pdf

Fosch-Villaronga, E. and Golia, A. Jr. (2019) Robots, Standards and the Law. Rivalries between private standards and public policymaking for robot governance. Computer Law & Security Review, 35(2), 129-144

This article explains the complex intertwinement between public and private regulators in the case of robot technology. Public policymaking ensures broad multi-stakeholder protected scope, but its abstractness often fails in intelligibility and applicability. Private standards, on the contrary, are more concrete and applicable, but most of the times they are voluntary and reflect industry interests. The ‘better regulation’ approach of the EU may increase the use of evidence to inform policy and lawmaking, and the involvement of different stakeholders. Current hard-lawmaking instruments do not appear to take advantage of the knowledge produced by standard-based regulations, virtually wasting their potential benefits. This fact affects the legal certainty with regards to a fast-paced changing environment like robotics. In this paper, we investigate the challenges of overlapping public/private regulatory initiatives that govern robot technologies in general, and in the concrete of healthcare robot technologies. We wonder until what extent robotics should be governed only by standards. We also reflect on how public policymaking could increase their technical understanding of robot technology to devise an applicable and comprehensive framework for this technology. In this respect, we propose different ways to integrate the technical know-how into policymaking (e.g., collecting the data/knowledge generated from the impact assessments in shared data repositories, and using it for evidence-based policies) and to strengthen the legitimacy of standards.

Fosch-Villaronga, E. and Golia, A. Jr. (2019) The Intricate Relationships between Private Standards and Public Policymaking in the Case of Personal Care Robots. Who Cares More? In: Barattini, P., Vicentini, F., Virk, G. S., & Haidegger, T. (Eds.). (2019). Human-Robot Interaction: Safety, Standardization, and Benchmarking. CRC Press Taylor & Francis Group

One of the consequences of the fast development of technology is that public policymakers struggle to develop policies that adequately frame the technology impacts, challenges and opportunities in time. This favours private actors, who develop their own standards, decentralizing the power to regulate. Industrial standards have been developed to govern industrial robot technology and one type of service robots – personal care robots. This chapter explains the complex intertwinement between public and private regulators in the case of robot technology.

We argue that while safety requirements have been set for these types of robots, legal principles and values deeply embedded in the social environment where these are implemented – privacy, dignity, data protection, cognitive safety, autonomy or ethics – have been disregarded, leaving the users’ (fundamental) rights ignored. Moreover, there are currently no safety requirements for other types of robots, including surgical, therapeutic, rehabilitation, educational and sexual; specific requirements for different types of users are missing; and a code of practice for robot developers, even if being discussed, is currently unavailable.

Public policymaking can ensure a broader multi-stakeholder protected scope, but its abstractness often fails to intelligibility and applicability. On the other side, whereas private standards may be much more concrete, most of the time, they are made by voluntary work with no juridical guarantees, normally reflecting industry interests. To give comprehensive protection to robot users without losing technical concreteness, we advocate for a better intertwinement between private standard-setting and public lawmaking approaches.

This chapter explains the distinction between two different modes of regulation, i.e. standard-setting and lawmaking, highlighting their respective features and their reciprocal interrelationships. One of the consequences of the inability to keep up with technology is that industry, and more generally, private actors, usually take the lead and develop their own standards. The problem with such testing regimes is that they are conceived under the idea that new technology may need new testing zones, obviating the fact that machine learning capabilities and more “real” robots may challenge that. Lawmaking does not necessarily relate to “the achievement of the optimum degree of order in a given context,” but rather to the achievement of consensus and agreement among the relevant members of the community on the substantive content of the rules themselves. Setting a clear distinction between soft law and hard law, between standard-setting and lawmaking. This is further shown by their intertwinement and reciprocal cross-references, justified by factual and technical needs.






A self-guiding guide to conduct research with embodiment technologies responsibly

Aymerich-Franch & Fosch-Villaronga, E. (2020). A self-guiding tool to conduct research with embodiment technologies responsibly. Frontiers in Robotics and AI, Perspective, 7, 22

The extension of the sense of self to the avatar during experiences of avatar embodiment requires thorough ethical and legal consideration, especially in light of potential scenarios involving physical or psychological harm caused to, or by, embodied avatars. We provide researchers and developers working in the field of virtual and robot embodiment technologies with a self-guidance tool based on the principles of Responsible Research and Innovation (RRI). This tool will help them engage in ethical and responsible research and innovation in the area of embodiment technologies in a way that guarantees all the rights of the embodied users and their interactors, including safety, privacy, autonomy, and dignity.






Aymerich-Franch, L. and Fosch-Villaronga, E. (2019) What we learned from mediated embodiment experiments and why it should matter to policymakers. Presence: Teleoperators and Virtual Environments, MIT Press Journals, 27:1

When people embody a virtual or a robotic avatar, their sense of self extends to the body of that avatar. We argue that, as a consequence, if the avatar gets harmed, the person embodied in that avatar suffers the harm in the first person. Potential scenarios involving physical or psychological harm caused to avatars gives rise to legal, moral, and policy implications that need to be considered by policymakers. We maintain that the prevailing distinction in law between “property” and “person” categories compromises the legal protection of the embodied users. We advocate for the inclusion of robotic and virtual avatars in a double category, property–person, as the property and the person mingle in one: the avatar. This hybrid category is critical to protecting users of mediated embodiment experiences both from potential physical or psychological harm and property damage.

Martinetti, A., Chemweno, P., Nizamis, K., & Fosch-Villaronga, E. (2021) Redefining safety in light of human-robot interaction: a critical review of current standards and regulations. Frontiers in Chemical Engineering.

Policymakers need to consider the impacts that robots and artificial intelligence (AI) technologies have on humans beyond physical safety. Traditionally, the definition of safety has been interpreted to exclusively apply to risks that have a physical impact on persons' safety, such as, among others, mechanical or chemical risks. However, the current understanding is that the integration of AI in cyber-physical systems such as robots, thus increasing interconnectivity with several devices and cloud services, and influencing the growing human-robot interaction challenges how safety is currently conceptualised rather narrowly. Thus, to address safety comprehensively, AI demands a broader understanding of safety, extending beyond physical interaction, but covering aspects such as cybersecurity, and mental health. Moreover, the expanding use of machine learning techniques will more frequently demand evolving safety mechanisms to safeguard the substantial modifications taking place over time as robots embed more AI features. In this sense, our contribution brings forward the different dimensions of the concept of safety, including interaction (physical and social), psychosocial, cybersecurity, temporal, and societal. These dimensions aim to help policy and standard makers redefine the concept of safety in light of robots and AI's increasing capabilities, including human-robot interactions, cybersecurity, and machine learning.

Fosch-Villaronga, E. and Millard, C. (2019) Cloud Robotics Law and Regulation. Challenges in the Governance of Complex and Dynamic Cyber-Physical Ecosystems. Robotics and Autonomous Systems 119, 77-91

This paper assesses some of the key legal and regulatory questions arising from the integration of physical robotic systems with cloud-based services, also called “cloud robotics.” The literature on legal and ethical issues in robotics has a strong focus on the robot itself, but largely ignores the background information processing. Conversely, the literature on cloud computing rarely addresses human–machine interactions, which raise distinctive ethical and legal concerns. In this paper, we investigate, from a European legal and regulatory perspective, the growing interdependence and interactions of tangible and virtual elements in cloud robotics environments. We highlight specific problems and challenges in regulating such complex and dynamic ecosystems and explore potential solutions. To illustrate practical challenges, we consider several examples of cloud robotics ecosystems involving multiple parties, various physical devices, and various cloud services. These examples illuminate the complexity of interactions between relevant parties. By identifying pressing legal and regulatory issues in relation to cloud robotics, we hope to inform the policy debate and set the scene for further research.

Sætra, H. S. & Fosch-Villaronga, E. (2021) Research in AI has Implications for Society: How do we Respond? The Balance of Power between Science, Ethics, and Politics. Morals & Machines, 1(1), 62-75, https://doi.org/10.5771/2747-5182-2021-1-62.


Artificial intelligence (AI) offers previously unimaginable possibilities, solving problems faster and more creatively than before, representing and inviting hope and change, but also fear and resistance. Unfortunately, while the pace of technology development and application dramatically accelerates, the understanding of its implications does not follow suit. Moreover, while mechanisms to anticipate, control, and steer AI development to prevent adverse consequences seem necessary, the current power dynamics on which society should frame such development is causing much confusion. In this article, we ask whether AI advances should be restricted, modified, or adjusted based on their potential legal, ethical, societal consequences. We examine four possible arguments in favor of subjecting scientific activity to stricter ethical and political control and critically analyze them in light of the perspective that science, ethics, and politics should strive for a division of labor and balance of power rather than a conflation. We argue that the domains of science, ethics, and politics should not conflate if we are to retain the ability to adequately assess the adequate course of action in light of AI‘s implications. We do so because such conflation could lead to uncertain and questionable outcomes, such as politicized science or ethics washing, ethics constrained by corporate or scientific interests, insufficient regulation, and political activity due to a misplaced belief in industry self-regulation. As such, we argue that the different functions of science, ethics, and politics must be respected to ensure AI development serves the interests of society.