Research and Development

Advancing Design and Engineering with Adaptive Instructional Systems

Adaptive Instructional Systems

Adaptive instructional systems, e.g., intelligent tutoring systems, can select and sequence design and engineering exercises for students. While so doing, these AI systems could measure and model students with respect to their proficiencies and adaptive expertise.

To achieve these objectives, adaptive instructional systems could receive and process real-time screencasts, streams of video and data generated as students complete design and engineering exercises. This real-time multimodal communication would enable adaptive instructional systems to better provide students with instruction, assessment, and contextual assistance.

This bulk educational data could be retained by adaptive instructional systems (1) to provide portions of recordings to other students, (2) to continuously retrain and improve, (3) to train other AI systems, (4) to provide resources for the advancement of education and science.

Screencast Streams and Recordings

State-of-the-art design and engineering software include features for recording and live-streaming screencasts. The formats of these streams and recordings can exceed those for simple video as screencast streams and recordings can include user-input events, e.g., keyboard and mouse events, and data representing users' commands and actions. Similarly, modern video communication technologies support screen-capturing and there are also remote desktop software to consider.

In (Grossman, Matejka, & Fitzmaurice, 2010), the authors state that "storing a document's workflow history, and providing tools for its visualization and exploration, could make any document a powerful learning tool."

In (Bao, Li, Xing, Wang, & Zhou, 2015), the authors present a computer-vision-based video-scraping technique to "automatically reverse-engineer time-series interaction data from screen-captured videos."

In (Frisson, Malacria, Bailly, & Dutoit, 2016), the authors describe a general-purpose tool for observing application usage and analyzing users' behaviors, combining computer-vision-based analyses of video-recordings with the collection of low-level interactions.

In (Sadeghi, Dargon, Rivest, & Pernot, 2016), the authors present a framework for fully capturing processes of computer-aided design and engineering.

Process Mining and Discovery

Process mining and discovery are relevant to the processing of screencast streams and recordings. Process mining is a family of techniques supporting the analysis of operational processes based on event logs. Process discovery aims to robustly obtain process models which describe event logs. Process models act as representations of processes (e.g., Petri nets, BPMN, activity diagrams, state diagrams).

In (Chang, Lafreniere, Kim, Fitzmaurice, & Grossman, 2020), the authors introduce workflow graphs which encode "how the approaches taken by multiple users performing a fixed 3D design task converge and diverge from one another."

Modeling Learners

Modeling learners is how adaptive instructional systems can best select and sequence design and engineering exercises for individualized learning.

Functions of adaptive instructional systems, e.g., intelligent tutoring systems, for design and engineering will include asessing students' (1) specific design and engineering proficiencies, (2) specific proficiencies pertaining to software use, and (3) adaptive expertise, as exhibited in screencast recordings of completions of exercises.

Adaptive expertise, a concept introduced by Hatano and Inagaki in 1986, has long been a useful construct though a challenging one to measure, requiring sophisticated analyses including based on self-reporting instruments such as Fisher and Peterson's Adaptive Expertise Beliefs survey.

In (Pierrakos, Anderson, & Welch, 2016), the authors state that "given the importance of adaptive expertise to the emerging engineer, it is imperative that a measure of adaptive expertise be able to provide meaningful information regarding how students perform on the various dimensions of the construct." The authors argue for "the development of a more sensitive, more direct measure of adaptive expertise that can be used across different design challenges."

Modeling Learning

Adaptive instructional systems for design and engineering will also need to model students' progressions, improvements, and learning.

In (Ozturk, Yalvac, Peng, Valverde, McGary, & Johnson, 2013), the authors indicate that, "as expected, when the students were more experienced with the modeling practice, their metacognitive adaptive expertise characteristics were enhanced."

Modeling Exercises

Modeling design and engineering exercises will be useful for adaptive instructional systems and for other consumers of resultant educational data.

Exercises include natural-language problem descriptions. These problem descriptions are a type of design specification or requirements.

Exercises can be presented to students in pedagogical contexts. Specific techniques may be desired to be shown by students, e.g., certain proficiencies in the use of CAD and CAE software.

Exercises can have metadata including those useful for categorizing screencast recordings of students' design and engineering workflows for subsequent processing. Metadata for exercises might include those relating exercises to types of courses or proficiencies.

The modeling of exercises encompasses measuring their efficacy or utility as they are presented to populations of students.

Computer-aided and Automatic Item Generation

Software tools can be envisioned with which to more rapidly develop design and engineering exercises for students.

A longer-term goal is that of automatic item generation, the automatic generation of efficacious design and engineering exercises such that retained screencast recordings of students' design and engineering workflows add value to educational and scientific data resources.

Educational Data

Vast amounts of data will be required for training next-generation AI systems (1) to tutor students in design and engineering, (2) to assist designers and engineers in their tasks, and (3) to automatically perform some design and engineering tasks. As envisioned, these requisite training data will include screencast recordings of design and engineering workflows and resultant 3D models.

Possibilities for obtaining these requisite training data include (1) creating resources for users to share their open-source screencast recordings and designs with one another, e.g., resembling YouTube or GitHub, and (2) retaining those screencast recordings and designs produced by students.

Adaptive instructional systems could record and subsequently retain these data for reasons including (1) to provide portions of recordings to other students, (2) to continuously retrain and improve, (3) to train other AI systems, (4) to provide resources for the advancement of education and science.

Adaptive instructional systems, e.g., intelligent tutoring systems, can be developed which optimize for multiple objectives while selecting and sequencing design and engineering exercises. These systems could achieve pedagogical goals for students while simultaneously bootstrapping and managing large-scale, open-source educational and scientific data resources. That is, all things being equal, adaptive instructional systems can intersperse design and engineering exercises to students such that resultant data, e.g., the various design processes utilized by students when completing the exercises, would be of use when collected at scale.


Intelligent Computer Systems in Engineering Design: Principles and Applications by Staffan Sunnersjö, Intelligent Computer-aided Design Systems: Past 20 Years and Future 20 Years by Tetsuo Tomiyama, The Ethical Use of Data in Education: Promoting Responsible Policies and Practices edited by Ellen B. Mandinach and Edith S. Gummer, Chronicle: Capture, Exploration, and Playback of Document Workflow Histories by Tovi Grossman, Justin Matejka and George Fitzmaurice, Reverse Engineering Time-series Interaction Data from Screen-captured Videos by Lingfeng Bao, Jing Li, Zhenchang Xing, Xinyu Wang and Bo Zhou, InspectorWidget: A System to Analyze Users Behaviors in their Applications by Christian Frisson, Sylvain Malacria, Gilles Bailly and Thierry Dutoit, Capturing and Analysing How Designers Use CAD Software by Samira Sadeghi, Thomas Dargon, Louis Rivest and Jean-Philippe Pernot, Workflow Graphs: A Computational Model of Collective Task Strategies for 3D Design Software by Minsuk Chang, Ben Lafreniere, Juho Kim, George Fitzmaurice and Tovi Grossman, Analysis of Contextual Computer-aided Design (CAD) Exercises by Elif Ozturk, Bugrahan Yalvac, Xiaobo Peng, Lauralee M. Valverde, Prentiss D. McGary and Michael Johnson, Examining the Role of Contextual Exercises and Adaptive Expertise on CAD Model Creation Procedures by Michael D. Johnson, Elif Ozturk, Lauralee M. Valverde, Bugrahan Yalvac and Xiaobo Peng, An Examination of the Effects of Contextual Computer-aided Design Exercises on Student Modeling Performance by Michael D. Johnson, Xiaobo Peng, Bugrahan Yalvac, Elif Ozturk and Ke Liu and Measuring Adaptive Expertise in Engineering Education by Olga Pierrakos, Robin D. Anderson and Cheryl A. Welch.