Why Several Agents Instead of One Big Prompt

The first version of our content pipeline was one long prompt. Read this chapter, work out what it covers, write twenty questions at the right level for this class, vary the question types, add pictures where they help, and check your answers. It worked, in the sense that output came back and much of it was usable.

It also failed in a way that took us a while to name. When a question was wrong, there was no way to tell why. Had it misread the chapter? Misjudged the class? Picked the wrong sub-topic and written a perfectly good question about the wrong thing? One prompt gives you one blob of output and no evidence about how it got there.

What splitting it up actually buys

Breaking the work into steps, each with its own input, its own output and its own validation, changes the economics of being wrong. Every step leaves an artefact behind. When something is wrong downstream, you read the artefacts backwards until you find the first one that is wrong, and that is your fault, located.

Here are the steps that matter, and what each one leaves behind.

Extract

Input: a PDF. Output: plain text, or an explicit failure. This step has no judgement in it at all, and that is a feature. It either produced readable text or it did not, and you can read the text yourself.

Locate

Input: the text, the class, the subject. Output: which topic from the catalogue this chapter corresponds to. Small output, enormous consequences. Getting this wrong means everything downstream is well-made content about the wrong thing, and it is visible in one line rather than discovered in twenty questions.

Draft

Input: text, topic, class, level, count, and a schema. Output: structured questions. The schema does the heavy lifting here. The response is parsed against it and anything that does not fit is dropped rather than coerced, so a malformed question never reaches a database field.

Shape

Input: drafted questions. Output: questions with types and pictures assigned. A Class 2 quiz that is twenty multiple-choice questions is a worse quiz than the same content with ordering, sorting and picture questions mixed in, and this step is where that mix is enforced. Picture tokens are validated here too: a token for a picture kind that does not exist is removed rather than shipped as literal text in a question.

Challenge

Input: the drafted quiz. Output: a list of objections. This step is given one job, which is to argue that the previous step got something wrong. Which answer keys look suspicious. Which questions have two defensible answers. Which wording a child of this age will misread. Which two questions are the same question twice.

This is the step that pays for all the others. A model asked to write good questions writes questions it believes are good. The same model asked what is wrong with these questions finds problems it would never have avoided creating in the first place. The objections do not auto-apply; they go to the person reviewing, as a list of things to look at first.

Park

Input: a finished draft. Output: a row in the database with the status pending_review. No step in this pipeline publishes anything.

The principle underneath

Each step takes structured input and produces structured output that is validated before it moves on. That sounds like ordinary engineering, and it is, which is the point. The interesting thing about building with language models is how much of the work is ordinary engineering around an unreliable component.

An unvalidated step quietly corrupts everything after it. A validated step either produces something well-formed or stops. Stopping is a good outcome. A pipeline that cannot stop is a pipeline that ships whatever it has.

What we would do differently

Three things, learned the slow way.

Make the locate step explicit and early. We originally let the drafting step infer the topic as it went, and misattributed content is almost impossible to spot later because every individual question looks fine.

Keep the challenge step separate from the draft step. Asking a model to write questions and then check them in the same breath produces a check that agrees with itself. Fresh context, single instruction, better results.

Give the reviewer the objections as well as the output. A reviewer handed twenty questions reads them evenly. A reviewer handed twenty questions and a note saying numbers four, eleven and seventeen have contested answer keys reads those three properly and the rest normally. Same half-hour, better use of it.