A RULE-BASED SHIELD: ACCUMULATING SAFETY RULES FROM CATASTROPHIC ACTION EFFECTS

Shahaf S. Shperberg, Bo Liu, Alessandro Allievi, Peter Stone

Research output: Contribution to journalConference articlepeer-review

1 Scopus citations

Abstract

Deploying autonomous agents in the real-world can lead to risks both to the agents and to the humans with whom they interact. As a result, it is essential for agents to try to achieve their objectives while acting as safely as possible. Thus, learning agents ought to learn not only about the effectiveness of actions, but also about their safety. While action effectiveness is task-dependent, information regarding the safety of actions can be preserved even if the task and/or the objective of the agent changes. The focus of this work is to leverage information from unsafe situations that the agent has experienced in order to obtain safety rules that identify which action from which state can lead to unsafe outcomes. These rules can be used for shielding the agent from repeating the same mistakes, as well as other mistakes that lead to the same catastrophic outcomes. In essence, before each action is selected for execution by the policy, actions which violate one of the safety rules from the current state are masked away and will not be selected. The cumulative set of safety rules can be used even when the agent faces multiple tasks, and can also be shared between different agents, so that mistakes that were made by one agent are not repeated by any of the agents that share the same rule-based shield. The process of learning a rule-based shield online is studied on a multi-task autonomous driving problem. Finally, the application of a rule-based shield to the Proximal Policy Optimization (PPO) algorithm is empirically evaluated and compared with the original PPO, with variants of PPO which use other online-learned shields, and with other baselines from the safe reinforcement learning literature. The results show that safety rules can significantly reduce the number of unsafe outcomes that agents experience, while even improving the cumulative rewards obtained by the agents.

Original languageEnglish
Pages (from-to)231-242
Number of pages12
JournalProceedings of Machine Learning Research
Volume199
StatePublished - 1 Jan 2022
Externally publishedYes
Event1st Conference on Lifelong Learning Agents, CoLLA 2022 - Montreal, Canada
Duration: 22 Aug 202224 Aug 2022

ASJC Scopus subject areas

  • Artificial Intelligence
  • Software
  • Control and Systems Engineering
  • Statistics and Probability

Fingerprint

Dive into the research topics of 'A RULE-BASED SHIELD: ACCUMULATING SAFETY RULES FROM CATASTROPHIC ACTION EFFECTS'. Together they form a unique fingerprint.

Cite this