Research
ResearchOptimizing Robot Tutor Strategies
When should a robot tutor teach, rest, or push harder? Formalised as a 1,440-state Markov Decision Process and solved with deep reinforcement learning.
- Role
- Researcher — Universitas Indonesia
- Timeline
- 2026
- Status
- Soft Actor-Critic reached a 100% expert-proficiency rate, far ahead of random and fixed-schedule baselines.
- Core stack
- Python · Gymnasium · Stable-Baselines3
What it is
An academic project that treats tutoring as a sequential decision problem: over a 24-hour clock, with a learner who has proficiency, fatigue and engagement, what should a tutor do at each step to maximise long-run learning?
The problem
Fixed tutoring schedules ignore the learner. Teaching into fatigue wastes the session; resting an engaged learner wastes the opportunity. The decision is sequential and the reward is delayed, which is the shape reinforcement learning exists for.
How it went
01A tutor that never looks at the learner
Tutoring schedules are fixed because fixed is easy to run, not because it works. Teaching into fatigue burns a session; resting an engaged learner wastes one. The decision repeats every step and the payoff arrives much later, which is precisely the shape reinforcement learning exists for.
The project began as a way to ask that question properly rather than to build a robot: over a 24-hour clock, with a learner who has proficiency, fatigue and engagement, what should a tutor do next?
02The environment turned out to be the contribution
Most of the work went into RobotTutor-v2 — a 1,440-state MDP wrapped as a Gymnasium environment. Making it Stable-Baselines3 compatible meant four algorithms could be compared without writing four training loops, and it leaves behind something someone else can run.
Soft Actor-Critic produced the best policy, reaching a 100% expert-proficiency rate and clearly beating random and fixed-schedule baselines. That a continuous-control method won on a discretised problem is the result I would most want to interrogate next.
Architecture
The decisions
Formalise the problem as a 1,440-state MDP
Twenty-four hours crossed with proficiency, fatigue and engagement gives a state space large enough to be interesting and small enough to enumerate and reason about.
Trade-off A discretised learner model is a caricature of a real one, so the results transfer as direction rather than as magnitude.
Build RobotTutor-v2 as a Gymnasium environment
Making it Stable-Baselines3 compatible meant four algorithms could be compared without writing four training loops, and the environment is reusable by anyone else.
Trade-off Conforming to the interface constrained how the reward and the episode boundaries could be expressed.
Benchmark DQN, PPO, TRPO and SAC across independent seeds
A single run of a single algorithm proves nothing in RL — variance across seeds is often larger than the gap between methods.
Trade-off Considerably more compute, for a result that is a comparison rather than one trained model.
Where it landed
- A 1,440-state MDP over a 24-hour clock, learner proficiency, fatigue and engagement.
- RobotTutor-v2 — a reusable Gymnasium environment, compatible with Stable-Baselines3.
- DQN, PPO, TRPO and SAC benchmarked across independent seeds.
- Soft Actor-Critic produced the best policy, reaching a 100% expert-proficiency rate and clearly beating random and fixed-schedule baselines.
In hindsight
The environment turned out to be the contribution. Once the MDP was honest about fatigue and engagement the algorithm comparison almost ran itself — and the fact that a continuous-control method won on a discretised problem is the part I would want to interrogate next.
Built with
- Python
- Gymnasium
- Stable-Baselines3
- DQN · PPO · TRPO · SAC
Next case study
Avagenc
A multi-agent assistant that acts across your inbox, calendar, contacts, music and home devices from a single conversation.