Research and Development

Dialogue Grounded in Simulation


Situational Grounding within Multimodal Simulations by James Pustejovsky and Nikhil Krishnaswamy, Creating Common Ground through Multimodal Simulations by James Pustejovsky, Nikhil Krishnaswamy, Bruce Draper, Pradyumna Narayana and Rahul Bangar, Visual Dialog by Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, José M. F. Moura, Devi Parikh and Dhruv Batra, Visual Dialog: Towards Communicative Visual Agents by Satwik Kottur, End-to-end Optimization of Goal-driven and Visually Grounded Dialogue Systems by Florian Strub, Harm de Vries, Jeremie Mary, Bilal Piot, Aaron Courville and Olivier Pietquin and Neural Approaches to Conversational AI by Jianfeng Gao, Michel Galley and Lihong Li.

Computer Simulation

Learning and Instruction with Computer Simulations by Ton de Jong, Learning and Instruction with Computer Simulations: Learning Processes Involved by Ton de Jong and Melanie Njoo, Characteristics of Simulations for Instructional Settings by Wouter R. van Joolingen and Ton de Jong, Scientific Discovery Learning with Computer Simulations of Conceptual Domains by Ton de Jong and Wouter R. van Joolingen, The Learning Effects of Computer Simulations in Science Education by Nico Rutten, Wouter R. van Joolingen and Jan T. van der Veen, Computer Simulations to Support Science Instruction and Learning: A Critical Review of the Literature by Lara K. Smetana and Randy L. Bell, Interactive Model Building Environments by Anders Bouwer, Vânia B. Machado and Bert Bredeweg, Towards Interactive Tools for Constructing Articulate Simulations by Vânia B. Machado and Bert Bredeweg and Automatically Generating Tutoring Questions for Qualitative Simulations by Floor Goddijn, Anders Bouwer and Bert Bredeweg.

Artificial Mental Simulation

Handbook of Imagination and Mental Simulation by Keith D. Markman, William M. P. Klein and Julie A. Suhr, Semantic Memory as the Root of Imagination by Anna Abraham and Andreja Bubic, Extending Cognitive Architectures with Mental Imagery by Scott D. Lathrop and John E. Laird, Exploring the Functional Advantages of Spatial and Visual Cognition from an Architectural Perspective by Scott D. Lathrop, Samuel B. Wintermute and John E. Laird, Imagery in Cognitive Architecture: Representation and Control at Multiple Levels of Abstraction by Samuel B. Wintermute, Abstraction, Imagery, and Control in Cognitive Architecture by Samuel B. Wintermute, A Cognitive Architecture that Combines Internal Simulation with a Global Workspace by Murray Shanahan, From Episodic Memory to Narrative in a Cognitive Architecture by Tory Anderson, Towards Narrative-based Knowledge Representation in Cognitive Systems by Nicolas Szilas, An Architecture of Narrative Memory by Carlos León, Toward a Situation Model in a Cognitive Architecture by Stuart M. Rodgers, Christopher W. Myers, Jerry Ball and Mary D. Freiman, Dimensions of Situation Model Construction in Narrative Comprehension by Rolf A. Zwaan, Joseph P. Magliano and Arthur C. Graesser, Situation Models: An Avenue to the Modeling of Mental Models by Mica R. Endsley, Mental Models in Narrative Comprehension by Gordon H. Bower and Daniel G. Morrow and Narrative Representation and Comprehension by Arthur C. Graesser, Jonathan M. Golding and Debra L. Long.


Rhetorical Status of the Descriptive by Philippe Hamon and Patricia Baudoin, A Survey on Automatic Image Caption Generation by Shuang Bai and Shan An, A Comprehensive Survey of Deep Learning for Image Captioning by M. D. Zakir Hossain, Ferdous Sohel, Mohd F. Shiratuddin and Hamid Laga, Video Description: A Survey of Methods, Datasets, and Evaluation Metrics by Nayyer Aafaq, Ajmal Mian, Wei Liu, Syed Z. Gilani and Mubarak Shah, Translating Video Content to Natural Language Descriptions by Marcus Rohrbach, Wei Qiu, Ivan Titov, Stefan Thater, Manfred Pinkal and Bernt Schiele, Multimodal Video Description by Vasili Ramanishka, Abir Das, Dong H. Park, Subhashini Venugopalan, Lisa A. Hendricks, Marcus Rohrbach and Kate Saenko, Video Description Generation using Audio and Visual Cues by Qin Jin and Junwei Liang and Attention-based Multimodal Fusion for Video Description by Chiori Hori, Takaaki Hori, Teng-Yok Lee, Ziming Zhang, Bret Harsham, John R. Hershey, Tim K. Marks and Kazuhiko Sumi.


The Rhetoric of Explanation by Robert J. Connors, Explanation and Dialogue by Nigel Gilbert, Scientific Explanations: Characterizing and Evaluating the Effects of Teachers' Instructional Practices on Student Learning by Katherine L. McNeill and Joseph Krajcik, The Cognitive Basis of Model-based Reasoning in Science by Nancy J. Nersessian, How are Models and Explanations Related? by Yasha Rohwer and Collin Rice, Explanations with Models in Science Education by John K. Gilbert, Carolyn J. Boulter and Margaret Rutherford, Models for Understanding by Richard E. Mayer, Models and Explanation by Alisa Bokulich, How Scientific Models Can Explain by Alisa Bokulich, Models and Representation by Richard I. G. Hughes, How Models are Used to Represent Reality by Ronald N. Giere, Simulation and Similarity: Using Models to Understand the World by Michael Weisberg, The Role of Models and Analogies in Science Education: Implications from Research by Richard K. Coll, Bev France and Ian Taylor, Facilitating Conceptual Change using Analogies and Explanatory Models by David E. Brown, Teaching and Learning with Analogies by Allan G. Harrison and David F. Treagust, Goal‐based Explanation Evaluation by David B. Leake, Evaluating Explanations: A Content Theory by David B. Leake, Evaluation of Explanatory Hypotheses by Ashwin Ram and David B. Leake, Strategy Selection in Question Answering by Lynne M. Reder, An Analysis of Explanation Structures for Answering Complex Questions by Sarah Bourse and Patrick Saint-Dizier, Generating Explanations in Context by Giuseppe Carenini and Johanna D. Moore, Generating Explanations in Context: The System Perspective by Vibhu O. Mittal and Cécile L. Paris, The Interpretation of Questions in Dialogue by Alex Lascarides and Nicholas Asher, Participating in Explanatory Dialogues: Interpreting and Responding to Questions in Context by Johanna D. Moore and Joanna D. Moore, Explaining Simulated Phenomena: A Defense of the Epistemic Power of Computer Simulations by Juan M. Durán, Explaining with Simulations: Why Visual Representations Matter by Julie Jebeile, Model Histories: Narrative Explanation in Generative Simulation Modelling by James D. A. Millington, David O'Sullivan and George L. W. Perry and A Theoretical Framework for Narrative Explanation in Science by Stephen P. Norris, Sandra M. Guilbert, Martha L. Smith, Shahram Hakimelahi and Linda M. Phillips.


Verbalization of 3D Scenes Based on Natural Language Generation Techniques by Vassilios Golfinopoulos, Dimitrios Makris, Georgios Bardis, Georgios Miaoulis and Dimitri Plemenos, Translating Video Content to Natural Language Descriptions by Marcus Rohrbach, Wei Qiu, Ivan Titov, Stefan Thater, Manfred Pinkal and Bernt Schiele, Towards Coherent Natural Language Description of Video Streams by Muhammad U. G. Khan, Lei Zhang and Yoshihiko Gotoh, Coherent Multi-sentence Video Description with Variable Level of Detail by Anna Rohrbach, Marcus Rohrbach, Wei Qiu, Annemarie Friedrich, Manfred Pinkal and Bernt Schiele, A Framework for Summarizing Game Experiences as Narratives by Yun-Gyung Cheong and R. Michael Young, Automatized Summarization of Multiplayer Games by Peter Mindek, Ivan Viola, Eduard Gröller and Stefan Bruckner, Action Summary for Computer Games: Extracting Action for Spectator Modes and Summaries by Nick Halper and Maic Masuch, Automated Creation of Movie Summaries in Interactive Virtual Environments by Doron Friedman, Ariel Shamir, Yishai Feldman and Tsvi Dagan, Summarizing Narratives by Wendy G. Lehnert, John B. Black and Brian J. Reiser and Coherent Narrative Summarization with a Cognitive Model by Renxian Zhang, Wenjie Li, Naishi Liu and Dehong Gao.

Question Answering

VQA: Visual Question Answering by Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick and Devi Parikh, TGIF-QA: Toward Spatio-temporal Reasoning in Visual Question Answering by Yunseok Jang, Yale Song, Youngjae Yu, Youngjin Kim and Gunhee Kim, TVQA: Localized, Compositional Video Question Answering by Jie Lei, Licheng Yu, Mohit Bansal and Tamara L. Berg, MarioQA: Answering Questions by Watching Gameplay Videos by Jonghwan Mun, Paul Hongsuck Seo, Ilchae Jung and Bohyung Han, MovieQA: Understanding Stories in Movies through Question-answering by Makarand Tapaswi, Yukun Zhu, Rainer Stiefelhagen, Antonio Torralba, Raquel Urtasun and Sanja Fidler, DramaQA: Character-centered Video Story Understanding with Hierarchical QA by Seongho Choi, Kyoung-Woon On, Yu-Jung Heo, Ahjeong Seo, Youwon Jang, Seungchan Lee, Minsu Lee and Byoung-Tak Zhang, The NarrativeQA Reading Comprehension Challenge by Tomáš Kočiský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis and Edward Grefenstette and IQA: Visual Question Answering in Interactive Environments by Daniel Gordon, Aniruddha Kembhavi, Mohammad Rastegari, Joseph Redmon, Dieter Fox and Ali Farhadi.