<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"> <head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> <meta name="Generator" content="Microsoft Word 15 (filtered medium)"> <style></style> </head> <body lang="EN-GB" link="#0563C1" vlink="#954F72"> <div class="WordSection1"> <o:p> </o:p> <o:p> </o:p> 2nd Call for Papers<o:p></o:p> <h2 style="mso-margin-top-alt:18.0pt;margin-right:0cm;margin-bottom:6.0pt;margin-left:0cm"> Workshop on Methodology and/of Evaluation of IVAs “Replicating the replication crisis”<o:p></o:p></h2> Merijn Bruijnes, University of Twente, the Netherlands, ✉️ <a href="mailto:m.bruijnes@utwente.nl">m.bruijnes@utwente.nl</a> <o:p></o:p> Annika Silvervarg, Linköping University, Sweden, <a href="mailto:annika.silvervarg@liu.se">annika.silvervarg@liu.se</a> <o:p></o:p> Willem-Paul Brinkman, Delft University, the Netherlands, <a href="mailto:W.P.Brinkman@tudelft.nl">w.p.brinkman@tudelft.nl</a> <o:p></o:p> Jelte van Waterschoot, University of Twente, the Netherlands, <a href="mailto:j.b.vanwaterschoot@utwente.nl">j.b.vanwaterschoot@utwente.nl</a> <o:p></o:p> Ulysses Bernardet, Aston University, United Kingdom, <a href="mailto:u.bernardet@aston.ac.uk">u.bernardet@aston.ac.uk</a> <o:p></o:p> <o:p> </o:p> Website: <a href="https://iva2019methodologyworkshop.wordpress.com/">https://iva2019methodologyworkshop.wordpress.com/</a> <o:p></o:p> <o:p> </o:p> DEADLINE EXTENDED 14 May 2019!<o:p></o:p> <o:p> </o:p> The aim of the 2nd workshop on methodology and evaluation is to critically but constructively discuss the empirical evaluation methods that are used in Human Computer Interaction, specifically in the area of Intelligent Virtual Agents. The social and life sciences are in a crisis of methodology as the results of many scientific studies are difficult or impossible to replicate in subsequent investigation (e.g. Pashler & Wagenmakers, 2012). The Open Science Collaboration (2015) observed, for example, that the effect size of replications was about half of the reported original effect size and that where 97% of the original studies had significant result, only 39% of the replication studies had significant results. In fact it has been suggested that more than 50% of psychological research results might be false (i.e. theories hold no or very low verisimilitude) (Ioannidis, 2005). Many of the methods employed by HCI researchers come from the fields that are currently in a replication crisis. Hence, do our studies have similar issues? <o:p></o:p> <o:p> </o:p> Long before the replication crisis hit psychology, Meehl (1990) suggested ten obfuscating factors that make that research on psychological theories are often uninterpretable. Viewing these factors gives us an idea of the scope of the problems that our research methodology might face:<o:p></o:p> <![if !supportLists]>1. <![endif]>Loose derivation chain: Very few derivation chains running from the theoretical premises to the predicted observational relation are deductively tight;<o:p></o:p> <![if !supportLists]>2. <![endif]>Problematic auxiliary theories: each auxiliary theory is itself nearly as problematic as the main theory we are testing;<o:p></o:p> <![if !supportLists]>3. <![endif]>Problematic ceteris paribus clause;<o:p></o:p> <![if !supportLists]>4. <![endif]>Experimenter error;<o:p></o:p> <![if !supportLists]>5. <![endif]>Inadequate statistical power;<o:p></o:p> <![if !supportLists]>6. <![endif]>Crud factor: everything correlates with everything;<o:p></o:p> <![if !supportLists]>7. <![endif]>Pilot studies: A true pilot study is a main study in the small. But these are often not published which can lead to line of research being dropped;<o:p></o:p> <![if !supportLists]>8. <![endif]>Selective bias in submitting report;<o:p></o:p> <![if !supportLists]>9. <![endif]>Selective editorial bias;<o:p></o:p> <![if !supportLists]>10. <![endif]>Detached validation claim for psychometric instruments: claiming a measure is 'valid' without further consideration;<o:p></o:p> <o:p> </o:p> A variety of ideas to improve research practices have been proposed and it is likely these ideas can be beneficial to the methods used in the field of HCI. Some actionable points leading to open and reproducible science are pre-registration of experiments, replication of findings, collaboration and education of researchers. For our field this could mean replicating our stimulus (such as an intelligent virtual agent) and the effect it has on users. The replication crisis needs our attention and as we reflect on our methods it makes sense to discuss in general our scientific methods and practises.<o:p></o:p> <o:p> </o:p> A workshop aimed at improving the quality of IVA research and methods should be welcomed by all IVA researchers. During the workshop we will discuss the methodological challenges identified in other fields and how they relate to the methods we use in our field. Additionally, we will discuss the proposed remedies and whether these are applicable for the research we conduct. We will discuss whether questions such as those posed above are relevant and, if so, how to go about answering them. This workshop is intended as a starting point and it will be the first of a series of workshops (at IVA and other conferences in the field) on this topic.<o:p></o:p> <o:p> </o:p> The goal is to embrace a positive, proactive approach that is sustainable and will lead to better science (no naming and shaming). The idea is to foster discussion and one way to achieve this is by having provocative statements to respond to. We invite participants to submit thought provoking statements about the methodology in HCI and/or respond to statements that we propose. Additionally, we invite (junior & senior) researchers to submit research ideas. Together with the participants and panel, we will offer practical support to improve the quality of their empirical work. Participants can posit their statements and/or discussions in an extended abstract (max 3 pages, excluding references). <o:p></o:p> <o:p> </o:p> Papers can be submitted via e-mail to: <a href="mailto:m.bruijnes@utwente.nl">m.bruijnes@utwente.nl</a> <o:p></o:p> <o:p> </o:p> Website of the first edition: <a href="https://iva2018methodologyworkshop.wordpress.com/">https://iva2018methodologyworkshop.wordpress.com/</a><o:p></o:p> <o:p> </o:p> Important dates:<o:p></o:p> <![if !supportLists]><s>· </s><![endif]><s>1 May – Submission deadline<o:p></o:p></s> <![if !supportLists]>· <![endif]>14 May – Submission deadline<o:p></o:p> <![if !supportLists]>· <![endif]>1 June – Acceptance notification<o:p></o:p> <![if !supportLists]>· <![endif]>1 July – Workshop<o:p></o:p> <o:p> </o:p> Ten provocative statements to start the discussion:<o:p></o:p> <![if !supportLists]>1. <![endif]>HCI research is too much novelty focussed.<o:p></o:p> <![if !supportLists]>2. <![endif]>Sample size estimation is impossible to do when evaluating new technology.<o:p></o:p> <![if !supportLists]>3. <![endif]>Experimental design/methodology are seen as necessary evil & boring by HCI researchers.<o:p></o:p> <![if !supportLists]>4. <![endif]>Theory building is difficult because of technical implementation of auxiliary hypotheses.<o:p></o:p> <![if !supportLists]>5. <![endif]>Knowledge of theories and concepts is insufficient in HCI (e.g. Basic emotions) and failures do not lead re-evaluation of assumptions.<o:p></o:p> <![if !supportLists]>6. <![endif]>HCI relies on small corpora and ground truth does not exist.<o:p></o:p> <![if !supportLists]>7. <![endif]>Technology focus creates legacy problems.<o:p></o:p> <![if !supportLists]>8. <![endif]>Custom/proprietary technology prevents accurate replication.<o:p></o:p> <![if !supportLists]>9. <![endif]>Open science is prevented by novelty focus, technical one-off solutions, and conference schedule.<o:p></o:p> <![if !supportLists]>10. <![endif]>Valorisation & entrepreneurialism, for which HCI is a key field, are at odds with proper conduct of science.<o:p></o:p> <o:p> </o:p> Participation is encouraged for all who are interested in good science. Contributions are welcome discussing methodology in HCI and/or related to the following topics:<o:p></o:p> <![if !supportLists]>· <![endif]>Replicability of studies;<o:p></o:p> <![if !supportLists]>· <![endif]>Methodological pitfalls specific to HCI;<o:p></o:p> <![if !supportLists]>· <![endif]>Tools and procedure that can improve the replicability;<o:p></o:p> <![if !supportLists]>· <![endif]>Validity of HCI research;<o:p></o:p> <![if !supportLists]>· <![endif]>What are we investigating (are the definitions clear)?<o:p></o:p> <![if !supportLists]>· <![endif]>Do we agree on definitions and what we are investigating?<o:p></o:p> <![if !supportLists]>· <![endif]>Are we asking the right questions?<o:p></o:p> <![if !supportLists]>· <![endif]>What are the answers worth?<o:p></o:p> <![if !supportLists]>· <![endif]>Generalisability of results;<o:p></o:p> <![if !supportLists]>· <![endif]>From theoretical background to concrete predictions;<o:p></o:p> <![if !supportLists]>· <![endif]>Relating data from HCI experiments back to theory (e.g. Gratch, 2017);<o:p></o:p> <o:p> </o:p> Tentative Schedule<o:p></o:p> 09.00 – 09.30 Conference registration and welcome coffee<o:p></o:p> 09.30 – 09.45 Introduction by Organizers, including a recap of last year’s workshop<o:p></o:p> 09.45 – 10.45 Keynote: TBA RepliCHI - Revisiting HCI findings<o:p></o:p> 10:45 – 11:00 Coffee break<o:p></o:p> 11:00 – 12:00 Keynote: TBA<o:p></o:p> Tentative: Measurement instrument creation<o:p></o:p> 12:10 – 13:30 Lunch break<o:p></o:p> 13.30 – 15.00 Presentations of accepted abstracts (6 x 15 min)<o:p></o:p> 15.00 – 15.30 Panel discussion on the directions to take towards better methodology <o:p></o:p> 15.30 – 15.45 Coffee break<o:p></o:p> 15.45 – 17.00 Breakout Session on ‘measurement instrument creation’<o:p></o:p> 17.00 – 17.30 presentations of breakout sessions & finishing notes by organizers<o:p></o:p> 17.30 – 21.00 Social dinner<o:p></o:p> <o:p> </o:p> Organizing Committee<o:p></o:p> Merijn Bruijnes (University of Twente)<o:p></o:p> Merijn Bruijnes is a Post-Doc researcher at the Human Media Interaction group at the University of Twente. He holds a masters in cognitive psychology and ergonomics and a doctorate in human media interaction. His research interests include, but are not limited to, artificial social agents, dialogue systems, and the effect technology has on humans.<o:p></o:p> <o:p> </o:p> Ulysses Bernardet (Aston University)<o:p></o:p> Ulysses Bernardet is a Lecturer in Computer Science at Aston University, Birmingham, UK. He holds a doctorate in psychology from the University of Zurich and has a background in psychology, computer science and neurobiology. In this research, Ulysses develops biologically grounded computational models of cognition and emotion that are used to control the behaviour of virtual and physical agent.<o:p></o:p> <o:p> </o:p> Willem-Paul Brinkman (Delft University)<o:p></o:p> In 2003 Willem-Paul Brinkman (1970) received his PhD degree in the area of human computer interaction from Eindhoven University of Technology, The Netherlands. His primary research interests are human-computer interaction, behavior change support systems, specifically eHealth systems including virtual reality therapy systems, and virtual health agents. He is fascinated by eHealth systems that include conversational agents that offer psychological support. <o:p></o:p> <o:p> </o:p> Deborah Richards (Macquarie University)<o:p></o:p> Deborah Richards is a Professor in the Department of Computing at Macquarie University. She joined academia in 1999 after nearly 20 years in the ICT industry during which she completed a BBus, MAppSc and a PhD in knowledge acquisition and reuse at UNSW. As an applied researcher and human-centred practitioner, her work is multidisciplinary and draws on theories from the cognitive science, biology, sociology, learning sciences, medicine, psychology and computer science to build and evaluate intelligent virtual agents to improve human learning and well-being.<o:p></o:p> <o:p> </o:p> Annika Silvervarg (Linköping University)<o:p></o:p> Annika Silvervarg is a Senior Lecturer in Cognitive Science at the Dept Computer and Information Science at Linköping University, Sweden. Her research interests include, conversational pedagogical agents, chatbots and dialogue systems, how humans perceive and interact with autonomous intelligent systems like virtual agents and social robots, and how to conduct empirical investigations of the users experiences and outcomes of interacting with such technologies.<o:p></o:p> Jelte van Waterschoot (University of Twente)<o:p></o:p> Jelte van Waterschoot is a PhD candidate at the Human Media Interaction Group at the University of Twente. He holds a masters in artificial intelligence, with a focus on language and linguistics. During his PhD he focuses on dialogue management for agents, specifically how we can create more topical and coherent conversations with the use of natural and non-verbal language processing. <o:p></o:p> <o:p> </o:p> Program committee<o:p></o:p> Douglas W. Cunningham (Cottbus) <o:p></o:p> More to be announced…<o:p></o:p> <o:p> </o:p> Bibliography<o:p></o:p> Gratch, J. (2017). Understanding the mind by simulating the body: virtual humans as a tool for cognitive science research. (S. E. F. Chipman, Ed.)The Oxford Handbook of Cognitive Science (Vol. 1). Oxford University Press.<o:p></o:p> <o:p> </o:p> Ioannidis, J. P. (2005). Why most published research findings are false. PLoS medicine, 2(8), e124.<o:p></o:p> <o:p> </o:p> Meehl, P. E. (1990). Why summaries of research on psychological theories are often uninterpretable. Psychological reports, 66(1), 195-244.<o:p></o:p> <o:p> </o:p> Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716.<o:p></o:p> <o:p> </o:p> Pashler, H., & Wagenmakers, E. J. (2012). Editors’ introduction to the special section on replicability in psychological science: A crisis of confidence?. Perspectives on Psychological Science, 7(6), 528-530.<o:p></o:p> <o:p> </o:p> </div> </body> </html>