What it is
An academic project that treats tutoring as a sequential decision problem: over a 24-hour clock, with a learner who has proficiency, fatigue and engagement, what should a tutor do at each step to maximise long-run learning?
The problem
Fixed tutoring schedules ignore the learner. Teaching into fatigue wastes the session; resting an engaged learner wastes the opportunity. The decision is sequential and the reward is delayed, which is the shape reinforcement learning exists for.
Chapter 01
A tutor that never looks at the learner
Tutoring schedules are fixed because fixed is easy to run, not because it works. Teaching into fatigue burns a session; resting an engaged learner wastes one. The decision repeats every step and the payoff arrives much later, which is precisely the shape reinforcement learning exists for.
The project began as a way to ask that question properly rather than to build a robot: over a 24-hour clock, with a learner who has proficiency, fatigue and engagement, what should a tutor do next?
Chapter 02
The environment turned out to be the contribution
Most of the work went into RobotTutor-v2 – a 1,440-state MDP wrapped as a Gymnasium environment. Making it Stable-Baselines3 compatible meant four algorithms could be compared without writing four training loops, and it leaves behind something someone else can run.
Soft Actor-Critic produced the best policy, reaching a 100% expert-proficiency rate and clearly beating random and fixed-schedule baselines. That a continuous-control method won on a discretised problem is the result I would most want to interrogate next.
Key decisions
03Formalise the problem as a 1,440-state MDP
Twenty-four hours crossed with proficiency, fatigue and engagement gives a state space large enough to be interesting and small enough to enumerate and reason about.
Trade-off · A discretised learner model is a caricature of a real one, so the results transfer as direction rather than as magnitude.
Build RobotTutor-v2 as a Gymnasium environment
Making it Stable-Baselines3 compatible meant four algorithms could be compared without writing four training loops, and the environment is reusable by anyone else.
Trade-off · Conforming to the interface constrained how the reward and the episode boundaries could be expressed.
Benchmark DQN, PPO, TRPO and SAC across independent seeds
A single run of a single algorithm proves nothing in RL – variance across seeds is often larger than the gap between methods.
Trade-off · Considerably more compute, for a result that is a comparison rather than one trained model.
Results
- A 1,440-state MDP over a 24-hour clock, learner proficiency, fatigue and engagement.
- RobotTutor-v2 – a reusable Gymnasium environment, compatible with Stable-Baselines3.
- DQN, PPO, TRPO and SAC benchmarked across independent seeds.
- Soft Actor-Critic produced the best policy, reaching a 100% expert-proficiency rate and clearly beating random and fixed-schedule baselines.
Looking back
The environment turned out to be the contribution. Once the MDP was honest about fatigue and engagement the algorithm comparison almost ran itself – and the fact that a continuous-control method won on a discretised problem is the part I would want to interrogate next.