NeurIPS 2026 Tutorial · Sydney
Adam White · Martha White · Andrew Patterson
Hands-on sessions led by Andrew Patterson and Esraa Elelimy · Colab notebooks created by Esraa Elelimy, Jiamin He and Samuel Neumann
It is no secret that reinforcement learning algorithms are hard to apply in the real world, but why? The answer is not obvious, so we start with a concrete example: on a realistic industrial simulator, multiple widely used algorithms perform worse than a simple baseline. Attendees then dig into a hands-on session to find out why. Next, we step back and discuss what makes the real world hard. These challenges rarely show up in standard benchmarks, and our current algorithms were never designed for them. The rest of the tutorial is about bringing those concerns into the lab. We cover empirical methodology, new benchmarks, and the many ways researchers accidentally cheat. We also take a hard look at common bad practice in RL: too few random seeds, untuned baselines, and overconfident claims built on incorrect confidence intervals. A second hands-on session shows how easily these mistakes can mislead us about what will work in deployment. We do not offer a single checklist for running RL experiments. Instead, attendees leave with three key messages and practical tools they can use in their own work.
| Time (AEDT) | Session | Presenter |
|---|---|---|
| 9:00 – 9:05 | Introduction and plan for the day. If you take three messages away today… | Adam |
| 9:05 – 9:20 | Demonstration. Popular algorithms perform worse than a simple baseline on a realistic industrial simulator. | Adam |
| 9:20 – 9:40 | Hands-on session 1: You get one deployment. Get to the bottom of the failure. Details ↓ | Andrew & Esraa |
| 9:40 – 10:05 | Challenges unique to the real world. What deployment demands that popular RL benchmarks do not, and why current algorithms are not designed for it. | Adam |
| 10:05 – 10:30 | Break | |
| 10:30 – 11:00 | Designing laboratory experiments aligned with real-world concerns. Empirical methodology, new benchmarks, common accidental cheating, and more. | Martha |
| 11:00 – 11:15 | Common poor empirical practices in RL. Too few seeds, untuned baselines, and confidently incorrect claims (confidence intervals, distributions) that make us think our algorithms are ready for the real world. | Andrew |
| 11:15 – 11:40 | Hands-on session 2: Same agents, different conclusions. How these errors mislead us about what works in deployment. Details ↓ | Andrew & Esraa |
| 11:40 – 11:50 | Wrap-up. If you take three messages away today… | Martha |
Both sessions run in Colab in your browser. Bring a laptop and a Google account; nothing needs to be installed.
Hands-on session 1 · 20 minutes · Andrew & Esraa
You lead the ML team at a wastewater treatment plant, and the board has approved exactly one RL deployment. Your agent controls aeration: air costs money, but too little air lets ammonia through, and ammonia over the limit costs you your permit. You pick one algorithm and one learning rate, commit, and watch your agent run for years of plant time against a controller that never moves. Then we look at what it would have cost to find a better configuration on a real plant, and what happens to the best agent when the plant changes.
Hands-on session 2 · 25 minutes · Andrew & Esraa
In the first session you ran the plant; now you write the paper. Nothing trains. You get the runs from the experiments in Empirical Design in Reinforcement Learning and make the analysis choices yourself: how many seeds to report, which interval to trust, how to pick the best configuration from a sweep, and where to cut off an episode. Each common shortcut turns the same agents into a better-looking result. All of it runs on Mountain Car and PuddleWorld, on purpose: deployment-aligned evaluation is a protocol you can apply in any environment, not a reason to abandon simple problems.