By Upol Ehsan, Ph.D. Student, School of Interactive Computing, Georgia Tech
(contributing writer; Joshua Preston, GVU Center)
If the power of AI is to be democratized, it needs to be accessible to anyone regardless of their technical abilities. As AI pervades all aspects of our lives, there is a distinct need for human-centered AI design that makes black-boxed AI systems explainable to everyday users.
Georgia Institute of Technology researchers, in collaboration with Cornell and University of Kentucky, have developed an artificially intelligent agent that can automatically generate natural language rationales to convey the motivations behind its actions. It’s a first step towards the aspirational goal of human-centered “Explainable AI” agents. The natural language mode of verbalization makes the human-AI communication accessible to most non-AI expert end-users. The technique allows human-understandable communication of the AI agent’s behavior, which can not only improve confidence in the system but also allow understanding of failure and unexpected behavior.
The agent also offers human-like rationales to make it more relatable or inspire confidence in those who might work regularly with AI machines or interact with them in social situations.
There is almost nothing artificial about artificial intelligence. It is designed by humans for humans, from training to testing to usage. Therefore, it’s essential that AI systems are human-understandable. Sadly, as AI-powered systems get more complex, their decision-making tends to be more “black-boxed” to the end-user, which inhibits trust. However, with this project, we have taken a formative step towards understanding the role of language-based explanations and how humans perceive them.
Humans use language to convey complex thoughts even when the internal neural firings aren’t consciously accessible. As the philosopher Jerry Fodor put it, the translation from “language of thought” to “language of communication” is an amazing feat that makes human-human communication and collaboration possible. So why not use this human-centered inspiration to design an AI agent that can think out loud even if the underlying system has a black-boxed architecture? This type of contextual design is ever so important in a world where humans will need to interact with AI. This project is more about understanding human perceptions and preferences of these AI systems than it is about building new technologies. At the heart of explainability is sensemaking. We are trying to understand that human factor.
Our research team enlisted participants to watch an AI agent play the classic arcade game Frogger and then asked the spectators to rate and rank three on-screen explanations that justified the agent’s actions.
The spectators judged the explanations based on:
- Confidence –the explanation makes the person confident in the agent’s ability to do its task.
- Human-likeness – the explanation looks like it was made by a human.
- Adequate justification – the explanation adequately justifies the action taken.
- Understandability –the explanation helps the person understand the AI’s behavior.
Choosing the game Frogger as the environment offered the researchers the chance to train an AI in a “sequential environment,” which is a significant research challenge because past decisions of the agent influence future ones. Explaining the chain of reasoning to experts is difficult and even more so when communicating with non-experts.
The human spectators understood the goal of Frogger in getting the frog safely home without being hit by moving vehicles or drowned in the river. The game mechanics of moving up, down, left or right, allowed the participants to clearly see what the AI was doing and evaluate if the explanations on the screen justified the move clearly.
Of the three anonymized explanations for each move – a human-generated response (the gold standard), the AI-agent’s response, and a randomly generated response (the bottom baseline) – the participants thought that the AI generated response handsomely beat the randomly generated one and came close to the gold standard. What this shows is that there is useful learning going on in the system that beats the random baseline; more notably, the explanations are plausible enough that it came close to the gold standard.
AI-generated explanations that had a positive influence on participant scores were those that showed recognition of environmental conditions and adaptability as well as those that communicated awareness of upcoming dangers and planned for them. Redundant information that just stated the obvious had a negative impact.
Knowing that the AI-generated explanations were perceived favorably by humans, the researchers then conducted a separate study to understand the preferences between two types of AI-generated responses: “focused” or concise rationales vs. “complete” or holistic rationales.
Only AI-generated answers were used in this study, and after participants watched each move in Frogger, they were asked which style of reasoning they would prefer if an AI were to make a mistake or behave unexpectedly.
By a 3-to-1 margin, participants favored explanations that were classified in the “complete” or holistic category. Responses showed that people appreciated the AI thinking about future steps rather than just what was in the moment, which might make them more prone to making another mistake. People also wanted to know more so that they might directly help the AI fix the errant behavior.
“The situated understanding of the perceptions and preferences by people in working with AI machines give us a powerful set of actionable insights that can help us design better human-centered, rationale-generating, autonomous agents,” said Mark Riedl, professor of Interactive Computing and lead faculty member on the project.
This project provided a foundational understanding of AI agents that can mimic thinking out loud. Possible future directions include understanding what happens when humans can contest an AI-generated explanation. Researchers will also look at how agents might respond in different scenarios, such as during an emergency response or when aiding teachers in the classroom.
The research is part of the proceedings of the Association for Computing Machinery’s Intelligent User Interfaces 2019 Conference that took place in mid-March. The paper is titled “Automated Rationale Generation: A Technique for Explainable AI and its Effects on Human Perceptions.”
Click here for a video of the talk at IUI.
mlatgt.blog 