We can envision the utilization of psychometric and educational exercises and activities in the training of artificial intelligence systems. Suitable exercises and activities include: IQ test items, exercises and activities from textbooks, items from quizzes and exams, and items produced via automatic item generation.

Artificial intelligence techniques have long been applied to games and game playing.

Computer simulations, including simulations of realistic environments, can be of use for training artificial intelligence systems such as software agents, robots and autonomous vehicles.

In addition to their utility with respect to simulating environments, computer simulations can be of use to artificial intelligence systems for planning courses of action and imagining the results or consequences of actions.

We can envision the utilization of social simulation in the training of artificial intelligence systems. Such social simulations might resemble play or improvisational theatre, designed to develop the social cognition, theory of mind and mentalization capabilities of artificial intelligence systems.

Social simulations can also be of use to artificial intelligence systems for planning courses of action and imagining the results or consequences of actions.

Teacher-student architectures include two varieties of interacting artificial intelligence systems: teachers and students.

Typically, the students are machine learning systems and the teachers are responsible for tasks including selecting the appropriate data, loss functions and hypothesis spaces to utilize. Teacher-student systems can exhibit co-evolution, students and teachers improving together over the course of iterative scenarios.

Interactive narrative can build upon simulation and social simulation, utilizing digital directors, drama managers and/or experience managers to enhance the pedagogical value of scenarios with respect to the training of artificial intelligence systems.

Interactive narrative is well suited for teaching machine ethics to artificial intelligence systems.

Simulated learners can approximate the learning processes of humans to such a degree that instructional design and pedagogy can be advanced through their use.

Some technical topics are broached toward the crowdsourcing of dialogue system content and behavior.

The content and behavior of a dialogue system can be represented in a number of ways.

Firstly, the content and behavior of a dialogue system can be represented in programming language source code files. Collaborative authoring, in this case, is a matter of integrated development environments and source code repositories, version control systems.

Secondly, the content and behavior of a dialogue system can be separated from source code files as data stored in some data format or in a database. Collaborative authoring, in this case, could require custom software tools.

Thirdly, a number of services, cognitive services, can encapsulate the content and behavior of a dialogue system. Collaborative authoring, in this case, could require utilization of such services or related user interfaces.

Fourthly, the content and behavior of a dialogue system can be represented as a set of interrelated, URL-addressable, editable pages. Servers can provide content for a number of different content types, for example hypertext for content authors and other formats for dialogue system user agents. Server-side scripting can be utilized to generate pages and generated pages can contain client-side scripting.

Fifthly, the content and behavior of a dialogue system can be represented as a set of interrelated, URL-addressable, editable diagrams.

Sixthly, the content and behavior of a dialogue system can be represented in transcript form. Transcript-based user interfaces may resemble instant messaging applications, scrollable sequences of speech bubbles, with speech bubbles coming from the left and right sides, such that users can edit the content in dialogue systems’ speech bubbles. Users could opt to view more than plain text in speech bubbles. There could also be vertical, colored bands in one or both margins, visually indicating discourse behaviors, moves, objectives or plans which span one or multiple utterances.

Debugging dialogue systems is an important topic. Debugging scenarios include switching from interactions with dialogue systems to authoring processes such that dialogue context data is preserved.

Natural language generation can produce editable structured documents from the data stored in databases and knowledgebases. Generated content can contain, beyond natural language, data and program logic to facilitate the processing of constrained or unconstrained edits. Edits to generated content can result in changes to stored data.

Computer-aided writing can convenience content authors and assure quality. Software can, generally speaking, provide users with information, warnings and errors with regard to tentative edits. Software can support users including with regard to their spelling, grammar, word selection, readability, text coherence and cohesion. Software can measure the neutral point of view of natural language. Software can also process tentative edits with regard to their logical consistency with respect to data stored in databases and knowledgebases.

Exploration into the collaborative authoring and debugging of dialogue systems could result in new wiki technologies. Wiki dialogue systems could resemble spoken language dialogue systems with transcript-based user interfaces, users able to easily switch between dialogue-based interactions and the editing of dialogue system content and behavior.

Instructional design is the theory and practice of designing instructional experiences and encompasses the design of educational resources such as hypertext, images, animations, charts, graphs, infographics, audio, video, 3D graphics, computer simulations, games, learning objects, courses, textbooks and the contents of intelligent tutoring systems.

Diagrams are symbolic representations of information according to visualization techniques. Diagrams, conveying information visually, are a powerful and expressive medium. Diagram authoring software applications allow individuals without computer programming expertise to participate in the design of resources, the entry of data and the development of software.

Educational resources such as learning objects, courses, textbooks and the contents of intelligent tutoring systems can be represented as diagrams.

Crowdsourcing leverages the intelligence and wisdom of crowds toward the collaborative design of resources and the solving of problems.

The crowdsourcing of instructional design is facilitated by software applications for the collaborative authoring of extensible diagrams.

Relevant software systems include wiki software, collaborative document authoring software, integrated development environments, extensible workflow and diagram authoring software and version control systems.

Quality control is essential to crowdsourced endeavors. Learning management systems may interoperate with crowdsourced diagrams. Intelligent tutoring systems may draw dialogue, behaviors and other contents from crowdsourced diagrams.

With a model of quality – a definition of quality – quality control can be described as matters of quality assessment and quality assurance.

The quality of crowdsourced instructional design relates to the quality of other crowdsourced endeavors, such as encyclopedias, insofar as the veracity and factual accuracy of the resultant information is paramount. Also important to the quality of instructional design is its pedagogical efficacy. As education science is considered to be the study of improving the teaching process, quality with respect to crowdsourced instructional design can be phrased as a moving target.

The assessment of quality can utilize educational data mining, learning analytics, learner feedback and other techniques for obtaining educational metrics.

The assurance of quality is approached as a matter of computer-aided design. Software can apprise users of pertinent design constraints, educational standards and recommendations. Software can provide users with insights into what is working and why with respect to design elements, structures and patterns in the contexts of learning objectives, plans and strategies. Software can provide users with information, warnings and errors with respect to courses being designed and with respect to existing courses. Software bots can process collections of educational resources, for example performing verification and validation services.

We need to standardize models for the diagrams of educational resources such as learning objects, courses, textbooks and tutorial contents. Diagrams of such educational resources should interoperate with both learning management systems and intelligent tutoring systems.

We can envision educators and instructional designers utilizing diagram authoring software. Both real-time collaboration and larger-scale crowdsourcing scenarios should be considered with respect to such software.

One promising area of research and development is computer-aided crowdsourced instructional design which leverages the intelligence and wisdom of crowds toward the design of diagrams of educational resources while the recommendations of software tools and the insights of education science are brought to bear for quality assurance.

Another promising area of research and development is computer-automated instructional design which encompasses the synthesis, selection and sequencing of multimedia educational resources per learning objectives, plans and strategies. Approaches may utilize machine learning and discovery upon collections of educational resources, such algorithms informed by educational metrics.

Improving the modeling of exercises and activities is an important task, one which advances the theory and practice of intelligent tutoring systems. The variety of modeling discussed, herein, is domain-independent, spanning multiple courses and the development of cognition. The proper, rigorous modeling of exercises and activities is non-trivial, an inherently complex task; doing so involves the modeling of cognition, modeling cognitive processes for numerous, relevant paths of progression through the exercises or activities.

With properly modeled exercises and activities, possibly a standardized such modeling, intelligent tutoring systems could be described as querying repositories of modeled exercises and activities, modifying existing exercises and activities or generating new exercises and activities, as needed, and loading and interactively presenting sequences of such exercises and activities to students. Essentially, case-based reasoning.

Repositories of exercises and activities require a querying language, possibly a standardized such language. Such querying languages depend strongly upon the modeling of exercises and activities.

Repositories of exercises and activities are more than case-bases, in a number of ways, and require an update language with expressiveness for educational data mining, for providing educational-scientific measurements pertaining to the utility and quality of exercises and activities.

Repositories of exercises and activities are envisioned as interoperable with numerous intelligent tutoring systems and future versions thereof. The modeling of exercises and activities, then, must encompass the specific features and approaches of each contemporary intelligent tutoring system.

A number of approaches to the automatic generation of exercises and activities can be interoperable with repository architectures. While repositories may interface as collections of existing exercises and activities, some may, in response to some queries, generate new exercises and activities on the fly.

Learning objects and digital textbooks also contain exercises and activities. Software components can encapsulate learning objects and digital textbooks, or collections thereof, implementing the interfaces of repositories. That is, intelligent tutoring systems can utilize specific exercises and activities, e.g. from syllabi or course materials, and intelligent tutoring systems can query collections of learning objects and digital textbooks for exercises and activities.

In the upcoming years, we can envision artificial intelligence systems utilizing educational exercises and activities as training data.

Developmental cognitive neuroscience, educational neuroscience and cognitive modeling will continue to advance and curricula should expand to include computer-mediated exercises and activities which are designed to activate, strengthen, coordinate and integrate specific regions and processes of the developing human brain.

Exercises and activities for cognitive development and enrichment can be generated by computer technology. Both human- and machine-generated exercises can be sequenced and scheduled by computer technology.

A focus on cognitive development results in a variety of exercises and activities. Variety also arises from mixing exercises and activities from multiple courses during coursework. Variety and the strategic sequencing and scheduling of exercises and activities promote student engagement.

Student affect should be monitored during the performance of exercises and activities. So doing facilitates the strategic sequencing and scheduling of exercises and activities. Sequencing and scheduling can be responsive to student affect; for example, scheduling categories of exercises known to be enjoyable to specific students to elevate mood.

Cognitive load refers to the total amount of effort being used in the working memory. Advancements to student modeling and cognitive modeling will facilitate better estimation of cognitive load during the performance of exercises and activities.

Discussion of the topics of variety in the sequencing and scheduling of exercises and activities entails a consideration of multitasking and task switching during problem solving. Modeling multitasking and task switching facilitates smoother experiences during and transitions between groups of exercises and activities.

Transfer of learning refers to how learning resulting from one category of exercise or activity effects performance in others. As developmental cognitive neuroscience, educational neuroscience and cognitive modeling continue to advance, so too will our knowledge of transfer of learning and the cognitive neuroscience thereof.

The objectives of longer-scale exercise and activity sequencing and scheduling include preparing students for coursework days, weeks, months or years in advance. Students can, for example, enjoy: exercises and activities with boxes for arithmetical values, preparing them for eventual curricular topics from algebra; exercises and activities involving 3D visuospatial reasoning, preparing them for eventual curricular topics from trigonometry and geometry; and various exercises and activities designed to introduce concepts from calculus.

Intelligent tutoring systems facilitate computer-aided coursework, computer-mediated exercises and activities, providing mixed-initiative tutorial dialogue, explanation, hints and encouragement.

Intelligent tutoring systems can be advanced in a number of ways.

Intelligent tutoring at scale involves the tutoring of large populations of students. Interesting areas of research and development include: learning from interactions with students to improve the quality of tutoring, educational experimentation, A/B testing and multivariate testing.

Intelligent tutoring systems can be more interoperable with learning objects, digital textbooks and courseware. Intelligent tutoring systems could, for example, utilize exercises and activities from learning objects, digital textbooks and courseware.

The integration of multiple domains facilitates the mixing and scheduling of exercises and activities from multiple courses.

Tutoring across schoolyears involves longer-term student modeling and longer-term planning and scheduling of exercises and activities.

Intelligent tutoring systems can generate and sequence exercises and activities for purposes of activating, strengthening, coordinating and integrating specific regions and processes of the developing human brain.

Intelligent tutoring systems can tutor groups of students simultaneously.