How Many Times Should A Pedagogical Agent Simulation Model Be Run?

When using simulation modeling to explore pedagogical phenomena, there are several issues a designer/practitioner should consider. One of the most important decisions has to do with determining how many runs of a simulation to perform in order to be confident in the results produced by the simulation [1]. With a deterministic model, a single simulation run is adequate. This issue becomes more challenging when part of the simulation model is based on stochastic elements. One of the solutions that has been used to address this challenge in other research communities is the use of Monte Carlo simulation [2]. Within the AIED research community, however, this question of how many times should a pedagogical simulation model be run to produce predictions in which the designer can have confidence has received surprisingly little attention. The aim of this paper is to explore this issue using a pedagogical simulation model, SimDoc, designed to explore longer term mentoring issues [3]. In particular, we demonstrate how to run this simulation model over many iterations until the accumulated results of the iteration runs reach a statistically stable level that matches real world performance but also has appropriate variability among the runs. We believe this approach generalizes beyond our simulation environment and could be applied to other pedagogical simulations and would be especially useful for medium and high fidelity simulations where each run may take a long time.