What the AI Is Not Allowed to Decide

Every company building with AI publishes a page about responsibility. Most of them describe intentions. Intentions are not worth much, because the question a school actually has is narrower and more useful: what stops it, in the code, when something goes wrong at two in the morning with nobody watching?

So this is a list of decisions a model never makes here, and the mechanism in each case.

1. Whether a student sees it

The mechanism: generated quizzes are written to the database with the status pending_review. Every query that lists quizzes for a student filters on approved. Generated lessons are written as drafts, and the lesson player serves published lessons.

The difference between this and a policy matters. A policy is a sentence someone has to remember. This is the shape of the data: an unreviewed quiz does not appear for a student because no query asks for it. A new feature that listed quizzes would have to go out of its way to include unreviewed ones.

2. What the quiz is about

The mechanism: the topic is chosen before generation runs, from the catalogue, by the teacher or by the monthly schedule, and passed in as a parameter. The model writes to the topic it is given.

A model allowed to pick its own topic writes about whatever the chapter emphasised most, which is often a worked example rather than the concept. The result is well-formed content filed under the wrong heading, and a student's weak-topic record quietly becomes fiction.

3. How hard it is

The mechanism: the level comes in as a parameter and the engine stores it as easy, medium or skilled. The model is told what to write to. It does not label its own output.

Self-assessed difficulty is unreliable in a specific direction: models write middling questions and call them hard. Since difficulty drives the timer, the level ladder and what a student is allowed to attempt next, a model grading its own work would corrupt all three.

4. What the output may look like

The mechanism: structured schemas, validated on arrival. A lesson scene must be one of eight kinds. A question must be one of the types the player can render. A picture token must name a picture kind that exists. Anything outside the shape is dropped rather than repaired.

Dropping rather than repairing is the important half. An attempt to fix malformed output is an attempt to guess what was meant, and a guess about a child's answer key is not a guess worth making.

5. Whether a flagged question is fine

The mechanism: a keyword filter runs over every question and option in a teacher-submitted quiz. Anything it flags goes to an administrator-only queue instead of into peer review.

It routes rather than blocks, which is the right behaviour. A Class 9 biology question can contain vocabulary that looks alarming out of context and is entirely appropriate in it. The filter does not decide. It decides who decides.

6. Whether its own confidence means anything

The mechanism: there is no confidence threshold anywhere in the publication path.

This is on the list because it is the most tempting shortcut in the whole field. If the model could reliably say which of its outputs were trustworthy, you could auto-publish the top band and review the rest, and halve the human workload. It cannot. Model confidence tracks fluency, and a confidently stated wrong answer key reads exactly like a confidently stated right one. So no number the model produces about itself is allowed to affect what happens to its output.

7. Whether it stays published

The mechanism: students rate how hard each piece of content was to follow, from 1 to 5. The average is ignored below ten votes, so a couple of harsh votes cannot pull anything down. At or below an average of 2.0, the content goes back for review.

This catches what review misses. A reviewer is an adult who already knows the subject, and the one thing an adult cannot reliably judge is whether a sentence is clear to a child who does not know the answer yet. Thirty children can.

What is left

Having listed what it cannot do: the model drafts, and that is real. It turns a blank page into twenty questions, a chapter into a sequenced lesson, a topic into a worksheet. On a Wednesday evening that is the difference between a teacher building practice material and a teacher going home.

The constraints are what make that acceptable rather than reckless. A fast first draft with a human gate is a good system. The same draft with no gate is a liability wearing a product's clothes. Who staffs the gate, and what they are paid for it, is the next article.