<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"> <head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> <meta name="Generator" content="Microsoft Word 15 (filtered medium)"> <style></style> </head> <body lang="EN-GB" link="blue" vlink="purple"> <div class="WordSection1"> Call for Papers - Workshop on Methodology and/of Evaluation of IVAs “Replicating the replication crisis” <o:p></o:p> Merijn Bruijnes, University of Twente, the Netherlands, ✉️ <a href="mailto:m.bruijnes@utwente.nl">m.bruijnes@utwente.nl</a> <o:p></o:p> Ulysses Bernardet, Aston University, United Kingdom, <a href="mailto:u.bernardet@aston.ac.uk">u.bernardet@aston.ac.uk</a> <o:p></o:p> Willem-Paul Brinkman, Delft University, the Netherlands, <a href="mailto:W.P.Brinkman@tudelft.nl">w.p.brinkman@tudelft.nl</a> <o:p></o:p> Deborah Richards, Macquarie University, Australia, <a href="mailto:deborah.richards@mq.edu.au">deborah.richards@mq.edu.au</a><o:p></o:p> <o:p> </o:p> Website: <a href="https://iva2018methodologyworkshop.wordpress.com/">https://iva2018methodologyworkshop.wordpress.com/</a><o:p></o:p> <o:p> </o:p> The aim of the workshop is to critically but constructively discuss the empirical methods that are used in Human Computer Interaction, specifically in the area of Intelligent Virtual Agents. The social and life sciences are in a crisis of methodology as the results of many scientific studies are difficult or impossible to replicate in subsequent investigation (e.g. Pashler & Wagenmakers, 2012). The Open Science Collaboration (2015) observed, for example, that the effect size of replications was about half of the reported original effect size and that where 97% of the original studies had significant result, only 39% of the replication studies had significant results. In fact it has been suggested that more than 50% of psychological research results are likely false (i.e. theories hold no or very low verisimilitude) (Ioannidis, 2005). Many of the methods employed by HCI researchers come from the fields that are currently in a replication crisis. Hence, do our studies have similar issues? <o:p></o:p> <o:p> </o:p> Long before the replication crisis hit psychology, Meehl (1990) suggested ten obfuscating factors that make that research on psychological theories are often uninterpretable. Viewing these factors gives us an idea of the scope of the problems that our research methodology might face:<o:p></o:p> <ol style="margin-top:0cm" start="1" type="1"> <li class="MsoNormal" style="color:black;mso-list:l0 level1 lfo1;vertical-align:baseline"> Loose derivation chain: Very few derivation chains running from the theoretical premises to the predicted observational relation are deductively tight;<o:p></o:p></li><li class="MsoNormal" style="color:black;mso-list:l0 level1 lfo1;vertical-align:baseline"> Problematic auxiliary theories: each auxiliary theory is itself nearly as problematic as the main theory we are testing;<o:p></o:p></li><li class="MsoNormal" style="color:black;mso-list:l0 level1 lfo1;vertical-align:baseline"> Problematic ceteris paribus clause;<o:p></o:p></li><li class="MsoNormal" style="color:black;mso-list:l0 level1 lfo1;vertical-align:baseline"> Experimenter error;<o:p></o:p></li><li class="MsoNormal" style="color:black;mso-list:l0 level1 lfo1;vertical-align:baseline"> Inadequate statistical power;<o:p></o:p></li><li class="MsoNormal" style="color:black;mso-list:l0 level1 lfo1;vertical-align:baseline"> Crud factor: everything correlates with everything;<o:p></o:p></li><li class="MsoNormal" style="color:black;mso-list:l0 level1 lfo1;vertical-align:baseline"> Pilot studies: A true pilot study is a main study in the small. But these are often not published which can lead to line of research being dropped;<o:p></o:p></li><li class="MsoNormal" style="color:black;mso-list:l0 level1 lfo1;vertical-align:baseline"> Selective bias in submitting report;<o:p></o:p></li><li class="MsoNormal" style="color:black;mso-list:l0 level1 lfo1;vertical-align:baseline"> Selective editorial bias;<o:p></o:p></li><li class="MsoNormal" style="color:black;mso-list:l0 level1 lfo1;vertical-align:baseline"> Detached validation claim for psychometric instruments: claiming a measure is 'valid' without further consideration;<o:p></o:p></li></ol> <o:p> </o:p> A variety of ideas to improve research practices have been proposed and it is likely these ideas can be beneficial to the methods used in the field of HCI. Some actionable points leading to open and reproducible science are pre-registration of experiments, replication of findings, collaboration and education of researchers. The replication crisis needs our attention and as we reflect on our methods it makes sense to discuss in general our scientific methods and practises.<o:p></o:p> <o:p> </o:p> In the past years there has been limited activity on this topic at IVA conferences. A workshop aimed at improving the quality of IVA research and methods should be welcomed by all IVA researchers. During the workshop we will discuss the methodological challenges identified in other fields and how they relate to the methods we use in our field. Additionally, we will discuss the proposed remedies and whether these are applicable for the research we conduct. We will discuss whether questions such as those posed above are relevant and, if so, how to go about answering them. This workshop is intended as a starting point and it will be the first of a series of workshops (at IVA and other conferences in the field) on this topic.<o:p></o:p> <o:p> </o:p> The goal is to embrace a positive, proactive approach that is sustainable and will lead to better science (no naming and shaming). The idea is to foster discussion and one way to achieve this is by having provocative statements to respond to. <o:p></o:p> <o:p> </o:p> We invite participants to submit thought provoking statements about the methodology in HCI and/or respond to statements that we propose. Participants can posit their statements and/or discussions in an extended abstract (max 3 pages, excluding references). <o:p></o:p> <o:p> </o:p> Papers can be submitted via e-mail to: <a href="mailto:m.bruijnes@utwente.nl">m.bruijnes@utwente.nl</a> <o:p></o:p> <o:p> </o:p> Important dates:<o:p></o:p> <ul style="margin-top:0cm" type="disc"> <li class="MsoNormal" style="color:black;mso-list:l2 level1 lfo2;vertical-align:baseline"> 1 October – Submission deadline<o:p></o:p></li><li class="MsoNormal" style="color:black;mso-list:l2 level1 lfo2;vertical-align:baseline"> 15 October – Acceptance notification<o:p></o:p></li><li class="MsoNormal" style="color:black;mso-list:l2 level1 lfo2;vertical-align:baseline"> 5 November – Workshop<o:p></o:p></li></ul> <o:p> </o:p> Ten provocative statements to start the discussion:<o:p></o:p> <ol style="margin-top:0cm" start="1" type="1"> <li class="MsoNormal" style="color:black;mso-list:l3 level1 lfo3;vertical-align:baseline"> HCI research is too much novelty focussed.<o:p></o:p></li><li class="MsoNormal" style="color:black;mso-list:l3 level1 lfo3;vertical-align:baseline"> Sample size estimation is impossible to do when evaluating new technology.<o:p></o:p></li><li class="MsoNormal" style="color:black;mso-list:l3 level1 lfo3;vertical-align:baseline"> Experimental design/methodology are seen as necessary evil & boring by HCI researchers.<o:p></o:p></li><li class="MsoNormal" style="color:black;mso-list:l3 level1 lfo3;vertical-align:baseline"> Theory building is difficult because of technical implementation of auxiliary hypotheses.<o:p></o:p></li><li class="MsoNormal" style="color:black;mso-list:l3 level1 lfo3;vertical-align:baseline"> Knowledge of theories and concepts is insufficient in HCI (e.g. Basic emotions) and failures do not lead re-evaluation of assumptions.<o:p></o:p></li><li class="MsoNormal" style="color:black;mso-list:l3 level1 lfo3;vertical-align:baseline"> HCI relies on small corpora and ground truth does not exist.<o:p></o:p></li><li class="MsoNormal" style="color:black;mso-list:l3 level1 lfo3;vertical-align:baseline"> Technology focus creates legacy problems.<o:p></o:p></li><li class="MsoNormal" style="color:black;mso-list:l3 level1 lfo3;vertical-align:baseline"> Custom/proprietary technology prevents accurate replication.<o:p></o:p></li><li class="MsoNormal" style="color:black;mso-list:l3 level1 lfo3;vertical-align:baseline"> Open science is prevented by novelty focus, technical one-off solutions, and conference schedule.<o:p></o:p></li><li class="MsoNormal" style="color:black;mso-list:l3 level1 lfo3;vertical-align:baseline"> Valorisation & entrepreneurialism, for which HCI is a key field, are at odds with proper conduct of science.<o:p></o:p></li></ol> <o:p> </o:p> Participation is encouraged for all who are interested in good science. Contributions are welcome discussing methodology in HCI and/or related to the following topics:<o:p></o:p> <ul style="margin-top:0cm" type="disc"> <li class="MsoNormal" style="color:black;mso-list:l1 level1 lfo4;vertical-align:baseline"> Replicability of studies;<o:p></o:p></li></ul> <ul style="margin-top:0cm" type="disc"> <ul style="margin-top:0cm" type="circle"> <li class="MsoNormal" style="color:black;mso-list:l1 level2 lfo5;vertical-align:baseline"> Methodological pitfalls specific to HCI;<o:p></o:p></li><li class="MsoNormal" style="color:black;mso-list:l1 level2 lfo5;vertical-align:baseline"> Tools and procedure that can improve the replicability;<o:p></o:p></li></ul> <li class="MsoNormal" style="color:black;mso-list:l1 level1 lfo5;vertical-align:baseline"> Validity of HCI research;<o:p></o:p> <ul style="margin-top:0cm" type="circle"> <li class="MsoNormal" style="color:black;mso-list:l1 level2 lfo5;vertical-align:baseline"> What are we investigating (are the definitions clear)?<o:p></o:p></li><li class="MsoNormal" style="color:black;mso-list:l1 level2 lfo5;vertical-align:baseline"> Do we agree on definitions and what we are investigating?<o:p></o:p></li><li class="MsoNormal" style="color:black;mso-list:l1 level2 lfo5;vertical-align:baseline"> Are we asking the right questions?<o:p></o:p></li><li class="MsoNormal" style="color:black;mso-list:l1 level2 lfo5;vertical-align:baseline"> What are the answers worth?<o:p></o:p></li></ul> </li><li class="MsoNormal" style="color:black;mso-list:l1 level1 lfo5;vertical-align:baseline"> Generalisability of results;<o:p></o:p> <ul style="margin-top:0cm" type="circle"> <li class="MsoNormal" style="color:black;mso-list:l1 level2 lfo5;vertical-align:baseline"> From theoretical background to concrete predictions;<o:p></o:p></li><li class="MsoNormal" style="color:black;mso-list:l1 level2 lfo5;vertical-align:baseline"> Relating data from HCI experiments back to theory (e.g. Gratch, 2017);<o:p></o:p></li></ul> </li></ul> <o:p> </o:p> Tentative Schedule<o:p></o:p> 09.00 – 09.30 Conference registration and welcome coffee<o:p></o:p> 09.30 – 09.45 Introduction by Organizers<o:p></o:p> 09.45 – 10.45 Keynote: Alex Holcombe (University of Sydney) Methodological issues and solutions<o:p></o:p> 10:45 – 11:00 Coffee break<o:p></o:p> 11:00 – 12:00 Keynote: TBA<o:p></o:p> Tentative: Measurement instrument creation<o:p></o:p> 12:10 – 13:30 Lunch break<o:p></o:p> 13.30 – 15.00 Presentations of accepted abstracts (15 x 5 min)<o:p></o:p> 15.00 – 16.00 Panel discussion on the directions to take towards better methodology <o:p></o:p> 16.00 – 16.15 Coffee break<o:p></o:p> 16.15 – 17.00 Breakout Session on ‘measurement instrument creation’<o:p></o:p> 17.00 – 17.30 presentations of breakout sessions & finishing notes by organizers<o:p></o:p> 17.30 – 20.30 Social<o:p></o:p> <o:p> </o:p> Organizing Committee<o:p></o:p> Merijn Bruijnes (University of Twente)<o:p></o:p> Merijn Bruijnes is a Post-Doc researcher at the Human Media Interaction group at the University of Twente. He holds a masters in cognitive psychology and ergonomics and a doctorate in human media interaction. His research interests include, but are not limited to, artificial social agents, dialogue systems, and the effect technology has on humans.<o:p></o:p> <o:p> </o:p> Ulysses Bernardet (Aston University)<o:p></o:p> Ulysses Bernardet is a Lecturer in Computer Science at Aston University, Birmingham, UK. He holds a doctorate in psychology from the University of Zurich and has a background in psychology, computer science and neurobiology. In this research, Ulysses develops biologically grounded computational models of cognition and emotion that are used to control the behaviour of virtual and physical agent.<o:p></o:p> <o:p> </o:p> Willem-Paul Brinkman (Delft University)<o:p></o:p> In 2003 Willem-Paul Brinkman (1970) received his PhD degree in the area of human computer interaction from Eindhoven University of Technology, The Netherlands. His primary research interests are human-computer interaction, behavior change support systems, specifically eHealth systems including virtual reality therapy systems, and virtual health agents. He is fascinated by eHealth systems that include conversational agents that offer psychological support. <o:p></o:p> <o:p> </o:p> Deborah Richards (Macquarie University)<o:p></o:p> Deborah Richards is a Professor in the Department of Computing at Macquarie University. She joined academia in 1999 after nearly 20 years in the ICT industry during which she completed a BBus, MAppSc and a PhD in knowledge acquisition and reuse at UNSW. As an applied researcher and human-centred practitioner, her work is multidisciplinary and draws on theories from the cognitive science, biology, sociology, learning sciences, medicine, psychology and computer science to build and evaluate intelligent virtual agents to improve human learning and well-being.<o:p></o:p> <o:p> </o:p> Program committee<o:p></o:p> Douglas W. Cunningham (Cottbus) <o:p></o:p> <o:p> </o:p> Bibliography<o:p></o:p> Gratch, J. (2017). Understanding the mind by simulating the body: virtual humans as a tool for cognitive science research. (S. E. F. Chipman, Ed.)The Oxford Handbook of Cognitive Science (Vol. 1). Oxford University Press.<o:p></o:p> <o:p> </o:p> Ioannidis, J. P. (2005). Why most published research findings are false. PLoS medicine, 2(8), e124.<o:p></o:p> <o:p> </o:p> Meehl, P. E. (1990). Why summaries of research on psychological theories are often uninterpretable. Psychological reports, 66(1), 195-244.<o:p></o:p> <o:p> </o:p> Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716.<o:p></o:p> <o:p> </o:p> Pashler, H., & Wagenmakers, E. J. (2012). Editors’ introduction to the special section on replicability in psychological science: A crisis of confidence?. Perspectives on Psychological Science, 7(6), 528-530.<o:p></o:p> <o:p> </o:p> <div> <div> <o:p> </o:p> </div> </div> </div> </body> </html>