A workshop affiliated with the 3rd ACM/IEEE International Conference on Human-Robot Interaction (HRI 2008)

March 12th, 2008, 13:30 - 17:30, Felix Meritis, Amsterdam

Overview
The evaluation of interactions between robots and humans, different types of interactions, and individual robot and human behaviors require adequate metrics and guidelines. These metrics should take into account a variety of factors, ranging from objective performance to social interaction. These metrics can code behaviors, ways of interaction, social and psychological aspects, and technical characteristics or objective measures (i.e. success rates, interaction time, error rates, etc.). There are metrics that can be acquired using objective measuring tools; others depend on the personal interpretation by the staff conducting and analysing experiments. When human beings are present, metrics for social human-robot interaction are of utmost interest in order to achieve robotic systems that can be intuitively handled by people without causing frustration and despair.

The following key questions are to be addressed by the intended workshop: Which guidelines should be followed for careful experimentation in HRI? Are there objective metrics applicable to HRI? Are there social metrics applicable to HRI? What is the relevance of subjective criteria for evaluation? Can subjectively categorized criteria be used to form objective metrics for social HRI? How can benchmarks (standardized tasks) be used to evaluate human-robot interactions?

The goal of the workshop is to propose guidelines for the analysis of human-robot experiments and forward a handbook of metrics that would be acceptable to the HRI community and allow researchers both to evaluate their own work and to better assess the progress of others. The format will combine information about different metrics and evaluation methods given in submitted and invited talks, panels, and moderated group discussions.

Invited talk
Julie A. Adams, Vanderbilt University, Statistical Validity Pitfalls, PDF, 604 KB

Conference Proceedings
Catherina R. Burghart & Aaron Steinfeld (Eds.), Proc. of Metrics for Human-Robot Interaction, a Workshop at ACM/IEEE HRI 2008,  Amsterdam, the Netherlands, 12 March 2008, Technical Report 471 University of Hertfordshire, Hatfield, UK. PDF, 8.3 MB

Proceedings Contents
Introduction

• Human-Robot Interaction Metrics and Future Directions
  C. R. Burghart & A. Steinfeld

Teams and Frameworks

• Steps to Creating Metrics for Human-like Movements and Communication Skills (of Robots)
  H. Holzapfel, R. Mikut, C. R. Burghart, & R. Häußling
• Identifying Generalizable Metric Classes to Evaluate Human-Robot Teams
  P. Pina, M. L. Cummings, J. W. Crandall, & M. Della Penna
• Toward Developing HRI Metrics For Teams: Pilot Testing In the Field
  J. Burke, K. S. Pratt, R. Murphy, M. Lineberry, M. Taing, & B. Day
  See also: USE-IT General Robot Usability Questionnaire
• Framing and Evaluating Human-Robot Interactions
  C. W. Nielsen, D. J. Bruemmer, D. A. Few, & D. I. Gertman

Social and Physical Interaction

• Measuring the Anthropomorphism, Animacy, Likeability, Perceived Intelligence, and Perceived Safety of Robots
  C. Bartneck, D. Kulic, & E. Croft
• Social Resonance: a Theoretical Framework and Benchmarks to Evaluate the Social Competence of Humanoid Robots
  T. Chaminade
• The Utility of Gaze in Spoken Human-Robot Interaction
  M. Staudte & M. Crocker
• A Visual Method for Robot Proxemics Measurements
  T. van Oosterhout & A. Visser

Documentation
The proceeding were made available as a printed report and are available at the link above. The aim of the workshop is to come up with a set of guidelines for experimental evaluation and a handbook of different metrics. It is intended to publish the results as well as selected papers in a special issue of an international journal. Input from this workshop will also be used in the construction of the www.hri-metrics.org portal.

Program committee
The co-chairs were supported by a distinguished list of organizers, all of whom have had significant experience both in organization and in relevant research. The list includes:

• Brian Scassellati, Yale University
• Alan Schultz, Naval Research Laboratory
• Chad (Odest) Jenkins, Brown University
• Gal Kaminka, Bar Ilan University
• Ralf Mikut, KIT

Organizers

-   -