Glass Box dev log

What broke, and how it was fixed

Taken from the full lessons-learned document written after submission. Failures are listed as plainly as the successes, each with its cause and its fix.

What worked

Fable first, then Opus as the escalation rung (R9, R28).

• Over the whole jam, Fable served 202 records with 170 completed (84 %). Opus (the claude-code runner) served 176 with 116 completed (66 %), but it took more of the verification and merge work and, from 23:10 on 14 Nov, the hard tasks Fable could no longer take.
• Haiku served 40 records with 15 completed (38 %) (runs.jsonl). Its share of the 15 Nov trouble is in section 5.

The 1st AD gate (R10).

• 157 reviews; 725 files rejected and 127 passed.
• It caught real defects before Cristian saw them: the D04 mirror, HUD plates over the cast, Jesse's mitten hands, the white Shipping render, the wrong door in a contact sheet, a stage direction printed as a spoken subtitle.
• On 15 Nov an independent review of the store images dropped two of the six screenshots: one duplicated the cover, and one showed a take with every score at zero.
• Its cost is in section 6: review and fix rounds used 37 % of all Claude Code context tokens.

Lanes and the side car.

• Lane B, a second editor and checkout, completed 73 of its 94 records.
• The three side slots completed 182 of 247 records with no editor.
• Map ownership per lane (R34, R39) removed the ownership deadlocks.

One asset per task (R32). After R32 a rejected image cost one small task, not a rerun of a group of 12 to 43 files.

Rulings carried in every brief. Agents applied R14, R30, R33 and R40 without being told twice. They saw a ruling only from the next run, because a run holds the plan it loaded.

Recovery and recorded state.

• 210 recovery entries; the runner landed, merged and deployed with no hand-run (#422, #428).
• Every act is in a log with the time and who did it. That is how this document could be written at all.

Measured gates on the last day. None of these passed on a status code; each read the artifact:

• The cook: the first rc1 failure named the 18 animations.
• The zip measurement: tools/m112_zip.py and the agent's own no-pdb zip. Measured, not projected.
• The loudness gate: it refused a file that was 0.35 dB too hot.
• The SuiPlay smoke: it read 16 to 20 FPS on the device, with frames looked at.
• The logged-out page read: it showed the old itch text still live.

The lossless fix before the lossy one. The size problem ended with Oodle Leviathan pak compression and deflate 9, with no visual change. The texture caps alone had reached 1,001.5 MB.

What did not work, with cause and fix

The 14 Nov draft records these in full, and they still stand:

• lane A as the single editor bottleneck;
• the 45-minute end-of-run idle;
• leftovers stalling lane B;
• landing conflicts on records;
• grouped generation;
• the multi-attempt Mesh to MetaHuman task;
• the Shipping white render;
• card noise;
• restarts because the runner does not reload code;
• the workstation crash;
• a false DONE, an invented human gate, an id collision, CI near its ceiling, dropped work and wasted generation.

Their final state:

• The white render: fixed at 00:31 on 15 Nov. The cause was the streamed booth layer's old Manual-exposure post-process volume, fixed with a guard in TakeDirectorPawn.cpp (57ff420).
• The restarts: five more after the draft (23:10 and 23:39 on 14 Nov; 01:31, 01:48 and 08:20 on 15 Nov). The gap is still open.

Below are the failures of the last day, each with its cause and the fix it calls for.

Ship requirements found at the release candidate

The 1 GB zip limit.

• What happened: the first rc1 package zipped to 1,430 MiB against the jam's 1 GB limit. Three more packages and about 75 minutes went to getting under it (09:18 to 10:21).
• The warning was there: the size note had been in packages.jsonl since 14 Nov ("the zip may pass 1 GB") and in submission-pack.md ("The rc1 package must measure the zip"). Nothing gated on it.
• Cause: a submission limit lived in a note, not in a check, and the size grew with every asset landing (951 MB staged on 13 Nov, 1,872 MB at rc1).
• Fix: a size budget measured on every package, from day one, with the real zip tool and the real limit (1 GB decimal, not 1 GiB). Lossless compression (Oodle Leviathan) goes in the project template, not in the final hour.

Editor-only references in cooked content.

• What happened: two separate cook failures came from the same cause. 61 face animations (18 in lane A, 43 in lane B) referenced /MetaHuman/IdentityTemplate/Face_Archetype_Skeleton, which does not cook. The MetaHuman Animator audio solve writes to that skeleton by default.
• Cause: no package had been cooked since the lip-sync landed, and the second set was found only after the first was fixed.
• Fix: a reference scan for editor-only paths on every landing that touches content, plus a cook on every landing that adds a new asset type. The scan that finally ran (08:29) found no other offenders in one pass. It should have run at 22:00 on 14 Nov.

The itch page text had gone stale and false.

• What happened: the live page said no AI voices, images or audio files were in the game, and that the only third-party asset was a font. Both stopped being true when the MetaHuman cast, the ElevenLabs voice bank and the generated props landed. It was found only when the page was read logged out at 11:30 on 15 Nov.
• Fix: the page text lives in the repo next to the build. A check fails when the build's asset manifest names a source (a tool, a licence) that the page's AI-use section does not.

The gameplay video's audio path.

• What happened: the video task stopped on its first attempt. It believed that recording game audio needed a change to Windows audio routing.
• The answer existed: the continuous capture was already recording WASAPI loopback.
• Then the loudness gate: the watched cut was refused at -1.15 dBTP against the -1.5 ceiling, and the audio had to be mastered and re-approved.
• Fix: the capture recipe (loopback, the loudness master) belongs in the project template, proven on day one with a 10-second test clip.

The runner kept working on frozen work

• What happened: after the video was cleared and uploaded, the 1st AD review of its stills posted fix rounds. The runner reposted a re-cut (M16.2.fix1) twice; it was stopped both times (about 11:22 and 11:27) and a guard loop retired the reposts.
• The same after gold: after R40 the round-2 reviews of rc1 posted four set and character fixes. One of them reused the size task's id, M15.2.fix3, and so blocked the submission pack by id.
• Cause: the runner has no idea of a freeze. A review of a frozen artifact still produces fix tasks, and recovery still reposts a stopped task.
• Fix: a freeze is state in the plan (gold sha, video sha). Reviews of a frozen artifact are recorded as post-release notes, never as tasks, and recovery never reposts a task whose artifact is frozen.

Wrong model on a release-critical task

• What happened, four times on the last day:
• at 06:49, Haiku's interrupted checkout deleted 484 files in lane A's checkout;
• at 07:29, Haiku tagged rc1 and stopped at a tool outside its allowlist;
• at 09:49 and 10:12, a Haiku size task (M17.3.tfix1) ran stale work and requested its own packages while the Opus task owned the size cut (its lossless change turned out to be the winning one);
• at 11:16, Haiku's submission pack used the unmastered video and called the mastered one unexplained.
• Cause: Haiku was the default first runner (13 Nov), and a pin to the escalation rung lasted only for the first attempt, so recoveries and reposts fell back to Haiku.
• Fix: route by task type, not by order. Release, package, git-surgery and submission tasks are pinned for every attempt. Two tasks may not own the same goal: a recovery of a superseded task is closed, not reposted.

Fragile task identity

• What happened:
• dependencies pointed at mesh ids that went stale when a task was reposted ("waiting on task dfc52c4d (cannot tell)"), so finished work stayed held;
• a task titled M15.2.size was invisible to the runner, whose id parser accepts only .fix<n>, .tfix<n> and similar forms;
• fix ids are numbered per parent, so the 1st AD's M15.2.fix3 collided with the size task's M15.2.fix3.
• Fix: dependencies name task ids, not mesh ids. One id grammar is shared by every tool that posts tasks, and a post that does not parse is refused at post time. Id numbers are allocated by the runner, never chosen by the poster.

Breaking changes made by hand under time pressure

• Renaming the escalation rung (23:10 on 14 Nov) broke six tasks, because three modules hard-code the name claude-fable. Lesson recorded at the time: never rename a runner, change its model.
• build_runner.py run --dry-run is not side-effect free. Run during a live run (08:20 on 15 Nov), it requeued a running task as an orphan. The same mistake happened on 14 Nov.
• Index locks after a kill. Killing a run left index.lock files in worktrees, and two side slots crashed on them (23:11 on 14 Nov). The restart scripts now clear locks under 30 min old.
• Fix: a dry run that cannot write; runner names as stable ids with the model as a property; a stop path that cleans its own locks.

Leftover processes and machine state

• What happened:
• a game instance from an earlier check sat at the menu, its parent gone, until Cristian saw it (10:53);
• a Windows Security credential dialog held the foreground during the rc1 checks;
• the desk avatar's capture track wrote 124 empty files.
• Fix: every launched process is owned by its task and killed on the task's exit. Capture runs on a dedicated display or virtual desktop.

Capacity limits

• Fable's usage limit stopped every Fable task at 23:06 on 14 Nov ("resets Nov 18"). Opus carried the rest.
• Playwright's MCP refuses file uploads over 50 MB to a browser it treats as remote. The 941 MB zip went up through the Chrome DevTools protocol's DOM.setFileInputFiles with a local path.
• butler was not installed.
• Fix: probe every quota and upload path in the first hour, not the last.

Smaller ones

• Clock drift in the agent's own log stamps. Up to 10 minutes on 15 Nov morning; the log carries two corrections. Stamps should come from the system clock, not be written by hand.
• A hand-completed reconcile task did not release the runner's reconcile.json (00:13). Reconcile tasks belong to the runner.

Tokens: what was recorded

What was not recorded. The build runner did not record tokens or cost. runs.jsonl has no usage field, and neither has any other runner ledger. That is a finding in itself: we cannot say what one task, one review or one fix round cost.

What was recoverable. Every Claude Code agent the runner started left a session transcript with per-message usage. postjam_tokens.py sums them, de-duplicated by message id: 490 runner sessions, plus the two orchestrator sessions (this one and [session]).

By kind of work, as a share of cache-read tokens:

What the numbers say:

• The base context is large. The median runner session started at 81,770 tokens of context before doing any work (p90: 90,128), and ran 21 turns (p90: 63). Every turn re-reads that context. As an estimate, 17,134 messages at about 80 k each is about 1.4 B cache-read tokens, about 40 % of the total, spent on the fixed brief, rules and tool schemas.
• Review and repair cost as much as making. The 1st AD loop, reviews plus fix rounds, read more context than all the feature work.
• The orchestrator was the single largest consumer, at 27 %. It is one long session that re-reads its own growing history every turn.
• Not covered:
• opencode and Codex runs (GLM-5.2 via Ollama Cloud; 82 records), which wrote no usage we can read;
• the 1st AD fallback reviewers on Ollama;
• the Hugging Face and Meshy calls, whose call counts are in section 2.
• No money figure. Prices are not applied, because the billing basis for these runs (plan or metered) is not in the record.

Token counts

Recovered from the agent session transcripts, not logged by the runner. They cover the Claude models only; the open GLM runs and the review fallbacks on open models are not counted.

Claude Code transcripts in 140 project dirs; sessions by kind {'unattributed': 7, 'feature / task': 208, 'fix round': 130, 'review (1st AD)': 151, 'orchestrator': 2}
fresh-in = input + cache creation (billed at or above the input rate); cache-read = cached context re-read
TOTAL msgs  17134  fresh-in    92.13M  cache-read   3540.69M  out   14.15M
by model
  claude-opus-5-5              msgs  10435  fresh-in    48.59M  cache-read   2160.10M  out    6.85M
  claude-fable-5-2             msgs   5449  fresh-in    35.84M  cache-read   1188.94M  out    6.40M
  claude-haiku-5-5             msgs   1242  fresh-in     7.70M  cache-read    191.65M  out    0.90M
  <synthetic>                  msgs      8  fresh-in     0.00M  cache-read      0.00M  out    0.00M
by kind of work
  feature / task               msgs   6601  fresh-in    36.28M  cache-read   1287.48M  out    5.69M
  orchestrator (this session)  msgs   3149  fresh-in     4.76M  cache-read    946.83M  out    1.71M
  fix round                    msgs   4191  fresh-in    24.85M  cache-read    802.48M  out    4.00M
  review (1st AD)              msgs   3171  fresh-in    25.77M  cache-read    502.40M  out    2.75M
  unattributed                 msgs     22  fresh-in     0.47M  cache-read      1.49M  out    0.01M
by day (PT)
  Tue 10 Nov                   msgs      1  fresh-in     0.05M  cache-read      0.02M  out    0.00M
  Wed 11 Nov                   msgs    133  fresh-in     0.97M  cache-read     14.62M  out    0.05M
  Thu 12 Nov                   msgs   2003  fresh-in     6.40M  cache-read    404.51M  out    1.18M
  Fri 13 Nov                   msgs   3322  fresh-in    16.77M  cache-read    844.40M  out    3.55M
  Sat 14 Nov                   msgs   7151  fresh-in    45.30M  cache-read   1442.82M  out    6.39M
  Sun 15 Nov                   msgs   4524  fresh-in    22.64M  cache-read    834.32M  out    2.98M

If we started tomorrow

The 14 Nov list still stands:

1. Lanes own maps from hour one.
2. Three side slots.
3. One asset per task.
4. Fable or Opus for making.
5. An owner-aware 1st AD cap.
6. Per-slot budgets and long runs.
7. Reload code at a save point.
8. A Shipping smoke on every landing.
9. A clean editor after every task.
10. Batched decisions.
11. Mesh to MetaHuman on day one.
12. CI before a dependency enters the plan.

The last day adds:

1. Ship gates from hour one, not at rc. Measure the zip on every package against 1 GB decimal; scan cooked content for editor-only references; diff the store page against the build's asset manifest; record a 10-second capture with game audio and master it.
2. A freeze is state. After gold, reviews become notes, recovery stops reposting, and the runner refuses new fix tasks against frozen artifacts.
3. Pin release work to one strong model for every attempt. No cheap first try on package, release, git surgery or submission work.
4. One owner per goal. A superseded task's recovery closes; it never runs in parallel with its replacement.
5. Ids are allocated, parsed and checked by one module. Dependencies use task ids.
6. Measure tokens per task in the runner, and give each task a budget. Shrink the base brief: 80 k tokens before the first action is the largest single cost we can see.
7. Probe quotas and upload paths on day one: model limits, butler, the browser upload size, the store's size limits.
8. Every launched process is owned by its task and dies with it.

Written by an agent (Claude Code, Opus 5.5). Not reviewed by Cristian.