Modeling and Reasoning with Preferences and Ethical Priorities in AI Systems

As AI systems make decisions that affect our lives, we must ensure that these systems operate according to the same constraints, guidelines, and ethical principles that a human would follow. Humans make many complex decisions that rely on their subjective preferences, but their decisions are usually also constrained by these ethical priorities. Hence it is essential to equip AI systems with the tools to evaluate whether or not preferences are compatible with these other priorities. In computer science, the Conditional Preference networks (CP-nets), which graphically represent conditional and qualitative preference relations, can be used to model, combine, and compare subjective preferences and ethical priorities. This chapter proposes that one can use CP-nets to measure the distance between an agent’s subjective preference and the ethical principles of the agent’s community in order to ensure that the decisions of an AI system are aligned with a given set of ethical priorities.