|
Date: March 2, 2012 Speaker: Dr. Neil Yorke-Smith Abstract: We present experiences from evaluating the learning performance of a user-adaptive personal assistant agent. This work was part of the CALO project, which led to spin-out Siri acquired by Apple in 2010. We discuss the challenge of designing adequate evaluation and the tension of collecting adequate data without a fully functional, deployed system. Reflections on negative and positive experiences point to the challenges of evaluating user-adaptive agent systems. Lessons learned concern early consideration of evaluation and deployment, characteristics of AI technology and domains that make controlled evaluations appropriate or not, holistic experimental design, implications of "in the wild" evaluation, and the effect of AI-enabled functionality and its impact upon existing tools and work practices. Bio: Neil Yorke-Smith is an Assistant Professor of Business Information and Decision Systems at the Suliman S. Olayan School of Business, American University of Beirut, and a Research Scientist at SRI International, USA. His research focuses on technologies that assist human decision making, with interests including constraint-based reasoning, intelligent agents, preference modelling, intelligent user interfaces, machine learning and data mining, and their real-life applications. Publications and further information are available at: http://www.aub.edu.lb/~nysmith
|