Orchestration
Orchestration is the set of decisions that connects several tasks across subagents. It determines who does each task, what information each subagent receives, and when the work can move to the next step.
Writing an economics proposal with three subagents
In this minimal example, you give three AI agents a question in labor economics and ask them to develop a short research proposal. Each acts as a subagent with its own job. The librarian searches for related papers, the writer turns an open question into a research design, and the referee independently reviews the proposal.
The starting question is “How does giving customer support workers an AI assistant affect their productivity?” Here, productivity means the number of customer issues a worker resolves per hour. Your goal is a proposal that asks an open question, explains how to measure the effect, and identifies a feasible way to collect the data.
- Researcher (You)Choose the topic and confirm that the library covers it.Topic + library
- Librarian (subagent)Retrieve local passages and look outside the collection for competing papers.Evidence with sources
- Writer (subagent)Draft the proposal. On a revision, address the objections in the latest review.Gap + short proposal
- Referee (subagent)Check novelty, identification and feasibility against the evidence.Review with reasons
- Revise
- Send the draft and objections back to the writer. The referee reviews the new draft.
- Pursue
- All three checks pass. Return the proposal to you for a decision.
- Stop
- A decisive objection ends the run. Return the review to you.
The return arrow is a while loop. While the decision is revise and another revision is available, the writer and referee take another turn. If the limit is reached, you receive the latest draft and its outstanding objections. The notebook allows one revision by default.
In the worked example, the first draft came back for revision. The referee wanted to know how the researchers would recruit new hires and how they would score a support ticket. The second draft supplied both, and the referee recommended a feasibility pilot.
1 Start in a library you know
You choose a collection that covers the question. For this example, the librarian starts with the AI and work library from the vector database lesson.
The closest passage comes from Generative AI at Work (cosine similarity 0.936). The second comes from AI and Jobs (0.880), which studies hiring. Reading even these two passages helps a researcher judge whether the library fits. A score alone cannot establish that it contains the closest prior work.
2 Search for a paper that could defeat the idea
The first study already asks how many issues support workers resolve per hour with AI. The librarian therefore looked for related experiments that ask what workers can do on their own after using AI. The search also covered studies in which people attempt a problem before seeing a suggested solution. A working paper and its journal version count as one study.
- Generative AI at Work
- The authors examine how support workers perform during unexpected AI outages. They already ask whether experience with the assistant helps workers resolve issues when it is unavailable.
- Do People Engage Cognitively with AI? Impact of AI Assistance on Incidental Learning
- Asking people to make a choice before revealing AI advice has already been tested in a food-choice task. The order of advice alone is an existing experimental idea.
- Answer First or Google First? Using the Internet in ways that Enhance, not Impair, One’s Subsequent Retention of Needed Information
- Programming students have already been tested on transfer after trying a problem before seeing help. A new experiment needs to say what it could teach us about paid worker training.
These papers helped the writer narrow the question. Suppose a firm has one paid hour to train each new support worker. If workers try to diagnose a ticket before seeing the AI’s solution, will they resolve more unfamiliar tickets on their own two weeks later?
3 Turn the gap into a test
Estimate how the order of identical AI guidance within a fixed hour of paid training changes novice support workers’ unaided resolutions on issue families held out for two weeks, with practiced-family performance as a benchmark.
Both groups of support workers would practice with the same tickets, guidance and paid time. One group would attempt each diagnosis before reading the solution. The other would read the solution first. Two weeks later, a grader who did not know the assignments would score tickets completed without AI. Each worker would face some types of issues they had never practiced, so the firm could measure how well they apply what they learned to unfamiliar cases.
The writer uses the economics writing skill to state one question, compare it with the closest studies, and explain how random assignment would identify the effect of changing the training sequence. Read the worked proposal (PDF).
4 Send the proposal for independent review
The referee receives the draft and its sources, searches for the closest competing papers, and judges whether the proposed design would answer a different economic question. The search for a direct counterexample adapts the novelty check in Zero Paper. The comparison with the closest papers and the conditions for proceeding come from the Research Idea Evaluation Pipeline.
First draft · Revise
The referee found two closer comparisons. Effects of identical example–problem and problem–example pairs on learning had already reversed the order of identical problems and examples. Problem-solving in virtual environment simulations prior to direct instruction for differential diagnosis in medical education: An experimental study had already tested transfer to new diagnostic cases the next day. The writer narrowed the claim and addressed both requests from the referee by identifying a recruitment route and supplying a ticket with a verifiable answer.
Revised draft · Pursue a feasibility pilot
- Novelty · pass
- The narrower question survives this search: does the order of identical guidance affect how well new support workers resolve unfamiliar issues two weeks later? Each piece has precedents; the contribution depends on what this particular comparison can tell a firm about training.
- Identification · pass
- Random assignment, equal practice time and grading without knowing who received which training make the groups comparable. If only 10 of 12 workers return in each group, the missing outcomes could still change the estimated effect by far more than five percentage points.
- Feasibility · pass
- Try recruiting 24 eligible workers in four weeks. At least 10 of 12 per group should return, two graders must agree on at least 90% of ticket scores, and the scoring file must hide group assignments. These are proposed checks; no workers have been recruited.
Next decision. Price recruitment and ask two support practitioners to validate the tickets against the manual. A larger trial depends on the pilot’s cost, retention and scoring precision.
Run the example
Open the orchestration notebook in Jupyter or VS Code and select Run All. The default run searches the teaching index and takes you through both recorded drafts. Set MAX_REVISIONS to 0 to see where the first draft stopped. For a new topic, choose a matching library, inspect its results and confirm which source notes you can share. Set MODE to live to run the same loop with fresh Crossref searches and separate Codex sessions.