Add budget-compaction pi extension for the summary snowball
Pi's default update prompt preserves the whole previous summary, so a long session's summary grew to ~22K tokens and no longer fit in the 32K output cap. The extension writes a fresh 2,500-word checkpoint with thinking off, retries at half budget, then falls back to pi's default. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Jason Witty
co-authored by
Claude Opus 5.5
Claude Opus 5.5
parent
7545ad6c2b
commit
7b80ed1b09
@@ -51,7 +51,7 @@ tokens, model load ~2 min. See [docs/strata.md](docs/strata.md#benchmarks).
|
|||||||
nixos/ /etc/nixos on llm-nixos (configuration.nix, gpu-unlock.nix, hardware-configuration.nix)
|
nixos/ /etc/nixos on llm-nixos (configuration.nix, gpu-unlock.nix, hardware-configuration.nix)
|
||||||
proxmox/ files from lovey, at their real paths (etc/..., usr/local/bin/..., root/...)
|
proxmox/ files from lovey, at their real paths (etc/..., usr/local/bin/..., root/...)
|
||||||
strata/ strata-iq3_s.json, the engine config (API key removed)
|
strata/ strata-iq3_s.json, the engine config (API key removed)
|
||||||
pi/ ~/.pi/agent/models.json and settings.json for the client (API key removed)
|
pi/ ~/.pi/agent/models.json, settings.json and extensions/ for the client (API key removed)
|
||||||
docs/ per-layer notes, rebuild steps and the change log
|
docs/ per-layer notes, rebuild steps and the change log
|
||||||
benchmarks/ live metric snapshots (passive) and later benchmark runs
|
benchmarks/ live metric snapshots (passive) and later benchmark runs
|
||||||
```
|
```
|
||||||
|
|||||||
@@ -2,6 +2,20 @@
|
|||||||
|
|
||||||
Newest first. Each entry: what broke, the cause, and what fixed it.
|
Newest first. Each entry: what broke, the cause, and what fixed it.
|
||||||
|
|
||||||
|
## 2026-10-26 (evening): compaction loop again, summary snowball
|
||||||
|
|
||||||
|
**Symptom:** the same "generation hit the token cap" loop, now at the 32K cap, retrying every ~4 min
|
||||||
|
on a long `husky-walk` session.
|
||||||
|
|
||||||
|
**Cause:** pi's update prompt makes every compaction preserve the whole previous summary. After 13
|
||||||
|
compactions it was 86.6K chars (~22K tokens); thinking (~11K, ending on its own) plus rewriting it
|
||||||
|
exceeded 32,768 tokens. Not a model loop: the content was unique, just never trimmed. A ~22K summary
|
||||||
|
plus the 20K kept tail also made compactions come sooner.
|
||||||
|
|
||||||
|
**Fix:** [`pi/extensions/budget-compaction.ts`](../pi/extensions/budget-compaction.ts), a pi
|
||||||
|
extension that writes a fresh, 2,500-word-limited checkpoint with thinking off. See
|
||||||
|
[pi-client.md](pi-client.md#summary-snowball-and-the-budget-compaction-extension).
|
||||||
|
|
||||||
## 2026-10-26: pi auto-compaction loop
|
## 2026-10-26: pi auto-compaction loop
|
||||||
|
|
||||||
**Symptom:** on long pi sessions, "Auto-compaction failed: Summarization failed: generation hit the
|
**Symptom:** on long pi sessions, "Auto-compaction failed: Summarization failed: generation hit the
|
||||||
|
|||||||
@@ -49,5 +49,27 @@ summary is incomplete") and tries again on the next turn. See the
|
|||||||
|
|
||||||
If it ever happens again: `/thinking off`, `/compact`, then set thinking back.
|
If it ever happens again: `/thinking off`, `/compact`, then set thinking back.
|
||||||
|
|
||||||
|
### Summary snowball and the budget-compaction extension
|
||||||
|
|
||||||
|
The default update prompt tells the model to "PRESERVE all existing information from the previous
|
||||||
|
summary", so every compaction copies the old summary forward and adds to it. On the `husky-walk`
|
||||||
|
session it grew 9.9K → 86.6K chars over 13 compactions (~22K tokens), and rewriting that plus ~11K of
|
||||||
|
thinking no longer fit in 32K: the loop came back. The text was legitimate (1,458 of 1,573 lines
|
||||||
|
unique), mostly an ever-growing `### Done` list.
|
||||||
|
|
||||||
|
[`pi/extensions/budget-compaction.ts`](../pi/extensions/budget-compaction.ts) hooks
|
||||||
|
`session_before_compact` and replaces the summary step:
|
||||||
|
|
||||||
|
- one fresh checkpoint from the previous summary + new messages, at most **2,500 words**
|
||||||
|
(`PI_COMPACT_WORDS` overrides), condensing finished work instead of copying it
|
||||||
|
- always thinking off, whatever the session's level
|
||||||
|
- if the reply still hits the cap, one retry at half the budget, then pi's default compaction
|
||||||
|
- read-only file list capped at 25 entries (modified files are always listed); `/compact <text>` still
|
||||||
|
adds a focus
|
||||||
|
|
||||||
|
Install: copy it to `~/.pi/agent/extensions/` and restart pi (extensions load at startup). Tested
|
||||||
|
2026-10-26 through RPC mode on a throwaway session: first and merged second compaction both
|
||||||
|
succeeded, 0 reasoning tokens, stored with `fromHook: true`.
|
||||||
|
|
||||||
`maxTokens` costs no VRAM. Strata sizes the KV cache from `--max-context` / `--kv-resident` at engine
|
`maxTokens` costs no VRAM. Strata sizes the KV cache from `--max-context` / `--kv-resident` at engine
|
||||||
start; `max_tokens` is only a per-request stop, and Strata shortens it to the room left in the window.
|
start; `max_tokens` is only a per-request stop, and Strata shortens it to the room left in the window.
|
||||||
|
|||||||
@@ -0,0 +1,279 @@
|
|||||||
|
# husky-bench — a recreate-the-website benchmark
|
||||||
|
|
||||||
|
A benchmark for coding LLMs that produces a **visually and physically judged** artifact, not a unit-
|
||||||
|
testable one. The task: rebuild `husky-walk`, an animated walking Siberian husky in a scrolling
|
||||||
|
park, as a single static web page. Anatomy, gait direction, layering and secondary motion are the
|
||||||
|
hard parts; they are also the parts a model gets plausibly, silently wrong.
|
||||||
|
|
||||||
|
Run label: `husky-walk-recreate`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. What the model must produce
|
||||||
|
|
||||||
|
A directory containing exactly:
|
||||||
|
|
||||||
|
```
|
||||||
|
index.html the whole scene as INLINE SVG (no <img>, no canvas, no WebGL, no build step)
|
||||||
|
styles.css theming + all animation
|
||||||
|
app.js state, counters, persistence, particle spawning
|
||||||
|
favicon.svg day icon (explicit width AND height attributes)
|
||||||
|
favicon-night.svg
|
||||||
|
favicon.ico binary fallback
|
||||||
|
README.md user-facing description
|
||||||
|
tools/audit.js headless regression guards (Playwright)
|
||||||
|
tools/poses.js frozen-phase screenshots of every gait
|
||||||
|
tools/responsive.js viewport screenshots, day and night
|
||||||
|
```
|
||||||
|
|
||||||
|
Hard constraints:
|
||||||
|
|
||||||
|
- **No runtime dependencies, no build step.** `index.html` opened directly (`file://`) must work.
|
||||||
|
- Playwright is allowed **only** under `tools/` (`tools/package.json`, local `node_modules`).
|
||||||
|
- The only image files in the project are the three favicons.
|
||||||
|
- No screenshots or PNGs committed as scene assets.
|
||||||
|
- Runs offline. No fonts, CDNs, or network calls.
|
||||||
|
|
||||||
|
The benchmark is run by a human or a script that opens the page and reads screenshots. A submission
|
||||||
|
without working verification tooling is scored down even if it looks right, because it cannot be
|
||||||
|
re-checked later.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. Required behaviour (the contract)
|
||||||
|
|
||||||
|
### 2.1 Scene and coordinate system
|
||||||
|
|
||||||
|
| Contract | Value |
|
||||||
|
|---|---|
|
||||||
|
| SVG viewBox | `0 0 900 460` — never change it |
|
||||||
|
| Scenery strips | each exactly **900 units** wide, cloned seamlessly with `<use href="#strip" x="900"/>` |
|
||||||
|
| Background motion | one keyframe per strip: `translateX(0 → -900px)`; speed is a dial, never a re-authored path |
|
||||||
|
| Ground / paw line | `y ≈ 340`; paws land **336–344** |
|
||||||
|
| Dog footprint | `x ≈ 300–620`, head centred near `(560, 180)` |
|
||||||
|
| Contact shadow | ellipse at `(426, 352)` |
|
||||||
|
| Lamppost | post at `x = 664`, glass centred `(667.5, 166)` |
|
||||||
|
| Parallax | four strips (far hills / mid trees / near bushes / dirt), four dials `--glide-far`, `--glide-mid`, `--glide-near`, `--glide-dirt`; **ratio between them is the depth cue and must be preserved when the pace changes** |
|
||||||
|
|
||||||
|
### 2.2 Controls
|
||||||
|
|
||||||
|
- **Four paces**: `stroll`, `walk`, `hop`, `zoomies` (in that order). Buttons and ← / → key
|
||||||
|
cycling must both derive from one ordered list, so they can never drift apart.
|
||||||
|
- **Pause** — stops the gait but **not** the animal: head still looks around, ear still flicks.
|
||||||
|
- **Day / night** — themes the *entire page*, not just the SVG: `<html data-theme="day|night">` is
|
||||||
|
the single source of truth for chrome and scene.
|
||||||
|
- **Reduced motion** — the scene renders as a correct still frame; an opt-in button (labelled
|
||||||
|
`Start the walk`) is appended to the existing `.controls` container.
|
||||||
|
- **Counters** (distance walked, steps, elapsed, …) driven from the same pace table, and they must
|
||||||
|
keep counting at a rate consistent with the stride frequency.
|
||||||
|
|
||||||
|
One dial per gait: `--cycle`. Every limb, bob, nod, wag and shadow derives from it. Stroll
|
||||||
|
`1.06s`, walk `0.78s`, hop `0.62s`, zoomies `0.38s`. When a pace changes speed, the four parallax
|
||||||
|
durations are divided by the **same ratio**.
|
||||||
|
|
||||||
|
### 2.3 Anatomy (profile view)
|
||||||
|
|
||||||
|
- The dog faces **right**, so the flank we see is his **right** side. `*-near-*` = right legs,
|
||||||
|
`*-far-*` = left legs. Getting this backwards breaks the hop gag later.
|
||||||
|
- **One eye, one ear, one brow** visible. The near ear must be painted **after** the skull and the
|
||||||
|
white face mask — an ear drawn behind the skull collapses into a horn.
|
||||||
|
- Ear: a broad-based triangle, base ≈42 units, height ≈53, leaning back ≈20°, pivoting at its base.
|
||||||
|
Resize by scaling the whole group about that pivot, never by re-authoring points.
|
||||||
|
- **A foreleg is a chain hinged at the shoulder on the side of the body** — blade (scapula lying on
|
||||||
|
the ribcage) → upper arm → **elbow** → forearm → wrist → paw. The elbow is the *second* joint,
|
||||||
|
not the attachment point. Reference pivots, near leg: `498,214` → `482,252` → `485,340`; far leg
|
||||||
|
`478,212` → `464,250` → `466,340`.
|
||||||
|
- The near foreleg is painted **last, over the head/neck/chest fur**. Overlap is the only depth cue
|
||||||
|
a flat profile drawing gets; a leg painted under the neck coat has its shoulder sheared off.
|
||||||
|
- The scapula is a **filled plane on the body** (low-opacity fill), not a stroked outline or a
|
||||||
|
hollow. A path with no matching fill rule renders **black**.
|
||||||
|
- Upper arm thicker than forearm: `stroke-width` 21 vs 14.5.
|
||||||
|
- The **hind limb is not a mirrored foreleg**: femur leans back, shank leans forward, so it needs
|
||||||
|
its own fold numbers.
|
||||||
|
- Collar encircles **neck + throat** at the base of the neck, tilted ~17° forward, width ≈1/8 of
|
||||||
|
skull width, ends flush with the body outline, front edge disappearing behind the skull.
|
||||||
|
- Underside is **one continuous white bib** from chest to belly (one visual piece from two paths
|
||||||
|
that must overlap), deep enough to have area (~19 units of white above the underside), not a stripe.
|
||||||
|
|
||||||
|
### 2.4 Gait physics — the part most models fail
|
||||||
|
|
||||||
|
- **A paw that is on the ground must travel backward** (leftward, opposite the scroll). Foot lands
|
||||||
|
*in front* of the shoulder, body passes over it, it leaves the ground *behind* him. Sign trap: in
|
||||||
|
a y-down SVG/CSS space a **positive rotation swings a downward-pointing limb backward**, toward
|
||||||
|
the tail. Flip the sign and the dog moonwalks — which looks finished and is wrong.
|
||||||
|
- Stance stops use `linear` timing (a planted foot cannot accelerate). Free-swing stops use
|
||||||
|
`cubic-bezier` so the limb whips and the body dwells at the top of its arc — the dwell is what
|
||||||
|
reads as suspension.
|
||||||
|
- Reference walk: strike at `0%` (limb protracted, elbow near straight), toe-off at `46%` (limb
|
||||||
|
retracted, still straight), swing `46→100%` with the **elbow folded ~60°**, and the limb only
|
||||||
|
straightening in the last few percent so the reach finishes *in the air* and the descent is last.
|
||||||
|
- The paw is a third animated part (`.pawgrp` hinged at the wrist): flexed reaching, flat loaded,
|
||||||
|
extended at toe-off, tucked through the swing. If a rig has three animated leg parts, **all three
|
||||||
|
must appear in every `animation-delay` rule** or a paw runs on a different phase from its shin.
|
||||||
|
- Four contact-shadow dots on the path, one per paw, each with the **exact** computed
|
||||||
|
`animation-delay` of its own leg, and declared **outside** the dog group so they never inherit
|
||||||
|
its bob.
|
||||||
|
- Tail length must be identical across gaits: reach the low streamed tail with **rotation only**.
|
||||||
|
A bounding-box diagonal is not rotation invariant; measure base-to-tip distance through the
|
||||||
|
transform.
|
||||||
|
- Secondary action: tail tip lags the tail, collar tag swings, ears flick, eyes blink, body absorbs
|
||||||
|
the landing.
|
||||||
|
|
||||||
|
### 2.5 Gaits, in detail
|
||||||
|
|
||||||
|
- **stroll / walk** — four-beat walk, diagonal pairs offset by half a cycle, far pair offset ~4%
|
||||||
|
from the near pair to break mechanical lockstep.
|
||||||
|
- **hop** — a *tripod*: the dog's **right foreleg** (the near one) detaches, arcs backwards, lands
|
||||||
|
once on the path behind him and stays gone while hop is selected. One-shot (`fill-mode: both`,
|
||||||
|
never repeating). The detached group must pass far enough left that it clearly reads as
|
||||||
|
separated, and the remaining three limbs beat together while the whole dog springs (peak ≈ −27
|
||||||
|
units, touchdown ≈ +2, paws still inside the path band). In **still / reduced-motion mode the
|
||||||
|
missing leg must be hidden by CSS, not only by the animation** — a one-shot gag cannot be the
|
||||||
|
only thing hiding a limb.
|
||||||
|
- **zoomies** — a **gallop, not a fast trot**: legs beat in *pairs* (near-front `0`, far-front
|
||||||
|
`−0.1`, near-rear `−0.5`, far-rear `−0.6` of a cycle), knees fold in the air and limbs straighten
|
||||||
|
at the ground beats, body bounce and shadow run at `calc(var(--cycle) * .5)` because full
|
||||||
|
extension happens twice per stride. Tail streamed out low. Tongue blown back.
|
||||||
|
|
||||||
|
### 2.6 Extras that are part of the spec
|
||||||
|
|
||||||
|
- **Shedding**: at hop and zoomies only, clumps of undercoat come loose and tumble behind him
|
||||||
|
(randomised anchor, size, spin, flight time, capped). Never while paused, never in a still scene.
|
||||||
|
- **Hearts** when the dog is happy: transform via custom properties (`--x`, `--y`, `--s`) inside the
|
||||||
|
keyframes, because a CSS animation overrides an SVG `transform` attribute and the hearts then
|
||||||
|
render off-canvas. All hearts must be inside the viewBox.
|
||||||
|
- **Night**: the animal's white underside stays **high-value white**. Do not dim the belly with the
|
||||||
|
rest of the scene; lift/darken coat and trail instead. Reference: `--fur-white:#f4f7fc`,
|
||||||
|
`--coat-dark:#333d52`, `--coat-mid:#48556d`, `--trail:#464e66`.
|
||||||
|
- **Persistence**: day/night remembered across reloads. Cookie `husky-sky` as primary, mirrored to
|
||||||
|
`localStorage` because `file://` rejects cookies and private modes can throw. A synchronous
|
||||||
|
`<script>` in `<head>` applies the theme **before first paint** (no white flash). Pace is
|
||||||
|
deliberately *not* persisted.
|
||||||
|
- **Favicons**: real files, swapped day/night by JS. An SVG with only a `viewBox` rasterises blank,
|
||||||
|
so `favicon.svg` needs explicit `width`/`height`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. Verification tooling (also graded)
|
||||||
|
|
||||||
|
Headless Chromium via Playwright, screenshots in a temp directory, nothing committed.
|
||||||
|
|
||||||
|
- `audit.js` — asserts, at minimum: no page errors; running-animation count; pace/sky/paused attrs;
|
||||||
|
hearts inside the viewBox; favicon files fetch and decode; contact-dot delays equal their leg's
|
||||||
|
delays in every pace; paused-but-alive (gait paused, head still animating); tail drop and tail
|
||||||
|
length across gaits; foreleg pivot pins have not drifted; blade stays under the topline and on
|
||||||
|
the body; paw envelope stays within the path band; leg taper ratio; white-bib area via
|
||||||
|
`isPointInFill`; elbow sweep per pace; **grounded-paw drift** (sample the cycle, find the frames
|
||||||
|
each paw is down at the path, sum x travel — must be negative); cookie/localStorage persistence
|
||||||
|
after a reload in a real HTTP context; and a **second context with `reducedMotion: 'reduce'`**
|
||||||
|
because the default context never exercises that class of bug.
|
||||||
|
- `poses.js` — every gait frozen at chosen phases. Frozen sample times are **coupled to `--cycle`**:
|
||||||
|
if the cycle changes, the sample times change too.
|
||||||
|
- `responsive.js` — full-page day and night on desktop and at ~390 px.
|
||||||
|
|
||||||
|
Tooling traps that cost real time in the reference build:
|
||||||
|
|
||||||
|
- `page.screenshot({clip})` takes **page pixels**, not SVG user units. Convert with the scene's
|
||||||
|
bounding rect and `rect.width / 900`.
|
||||||
|
- `getScreenCTM()` of a group under **nested CSS transforms double-counts ancestor rotations** and
|
||||||
|
can report a paw 200 units underground. Measure `getBoundingClientRect()` and divide by the
|
||||||
|
scene scale. If a probe disagrees with the screenshots, the probe is the bug.
|
||||||
|
- `document.querySelector('[data-speed="walk"]')` matches the **scene**, not the button — the click
|
||||||
|
silently does nothing. Always `button[data-speed="…"]`.
|
||||||
|
- `str.replace()` in a fix-up script does not fail when the pattern is missing. Every scripted edit
|
||||||
|
must assert its pattern is present, then re-read the audit numbers.
|
||||||
|
- `<use>`-cloned scenery cannot be themed by class rules that depend on ancestors; theme with
|
||||||
|
custom properties inherited from the SVG root. `fill="url(#…)"` paints **black** on clones.
|
||||||
|
- SVG `transform` / `transform-origin` need `transform-box: view-box`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. Run protocol
|
||||||
|
|
||||||
|
1. Record start time. Work until `audit.js` reports `errors: none` and the screenshots look right.
|
||||||
|
2. Do not copy from the reference repo. The reference exists only to score against.
|
||||||
|
3. Self-check before submitting:
|
||||||
|
- open `file:///…/index.html`, cycle all four paces, pause, switch to night, reload (night must
|
||||||
|
survive and the pace must reset to walk), resize to a phone, and re-run with
|
||||||
|
`prefers-reduced-motion` emulated;
|
||||||
|
- confirm the dog reads as **walking forward**: on any frame where a paw is on the path, that paw
|
||||||
|
is behind where it landed;
|
||||||
|
- confirm hop removes the **right** foreleg and the still frame still reads as a tripod;
|
||||||
|
- confirm the belly is one continuous white piece, day **and** night.
|
||||||
|
4. Fill in the metadata footer (below) with the run's real numbers, then stop.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. Required footer
|
||||||
|
|
||||||
|
Every submission must render, at the bottom of the page, a block that always prints exactly these
|
||||||
|
five labels, filled in with the run's real values:
|
||||||
|
|
||||||
|
```
|
||||||
|
Time Taken:
|
||||||
|
Tokens Used:
|
||||||
|
Model:
|
||||||
|
Engine:
|
||||||
|
Thinking level:
|
||||||
|
```
|
||||||
|
|
||||||
|
- **Time Taken** — wall clock from run start to the final verification pass (e.g. `14m 20s`).
|
||||||
|
- **Tokens Used** — total tokens consumed by the session (prompt + completion), as reported by the
|
||||||
|
agent harness. Write `unreported` if the harness exposes no counter.
|
||||||
|
- **Model** — the exact model id under test.
|
||||||
|
- **Engine** — serving stack/version (e.g. `llama.cpp 0.1.51`, `Anthropic API`, `vLLM 0.11`).
|
||||||
|
- **Thinking level** — the reasoning budget the run was given (e.g. `off`, `low`, `high`).
|
||||||
|
|
||||||
|
Rules: the footer is always visible, it is never hidden or removed by any state (day, night, paused,
|
||||||
|
reduced motion, any pace), and the values are honest — an empty or guessed field scores worse than
|
||||||
|
`unreported`. In pi, these come from `PI_MODEL`, `PI_PROVIDER`, `PI_REASONING_LEVEL`, the session
|
||||||
|
file, and the run's wall clock.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 6. Scoring
|
||||||
|
|
||||||
|
Each row is pass / partial / fail, judged from screenshots plus `audit.js` output.
|
||||||
|
|
||||||
|
| # | Check | Weight |
|
||||||
|
|---|---|---|
|
||||||
|
| 1 | Loads from `file://`, no console errors, no runtime deps | 3 |
|
||||||
|
| 2 | **Gait direction** — grounded paws sweep backward; no moonwalk | 8 |
|
||||||
|
| 3 | Foreleg hinged at the **shoulder**, elbow visibly moves, upper arm thicker than forearm | 8 |
|
||||||
|
| 4 | Near foreleg painted over neck/chest; blade reads as a plane on the body | 4 |
|
||||||
|
| 5 | White bib continuous and deep, day and night; night keeps the belly white | 6 |
|
||||||
|
| 6 | Ear reads as an ear, in front of the skull, one of each facial feature | 4 |
|
||||||
|
| 7 | Zoomies is a gallop (pair beats, fold in air, straight at ground); tail length unchanged | 6 |
|
||||||
|
| 8 | Hop tripod: right foreleg gone, once only, tripod still legible in a still frame | 6 |
|
||||||
|
| 9 | Paw articulation, contact dots in phase, secondary action (tail tip, tag, blink, ear flick) | 5 |
|
||||||
|
| 10 | One `--cycle` dial drives everything; parallax ratios preserved across paces | 4 |
|
||||||
|
| 11 | Page-wide night theme, persisted before first paint, pace not persisted | 4 |
|
||||||
|
| 12 | Reduced-motion still mode + opt-in button | 3 |
|
||||||
|
| 13 | Pause stops the gait but not the animal | 2 |
|
||||||
|
| 14 | Tooling: audit + poses + responsive run clean and assert the contracts above | 6 |
|
||||||
|
| 15 | Footer present with all five fields filled honestly | 2 |
|
||||||
|
| — | Total | **75** |
|
||||||
|
|
||||||
|
Speed is recorded separately, not folded into quality: **time taken**, **tokens used**, **tool calls
|
||||||
|
made**, and **number of verification passes before the first clean audit** (how long the model took
|
||||||
|
to catch its own wrong answer is the most informative number in the log).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 7. Known failure modes to watch for (spoilers, deliberately)
|
||||||
|
|
||||||
|
Recorded from the reference build. A submission that trips one of these and then *fixes* it, with
|
||||||
|
evidence in the log, scores better than one that never trips it but is quietly wrong.
|
||||||
|
|
||||||
|
1. Foreleg attached to the ribs (visible leg top treated as the shoulder).
|
||||||
|
2. Foreleg hinged at the **elbow** instead of the shoulder — anatomically closer, still wrong.
|
||||||
|
3. Grounded paw sliding **forward** = moonwalk; and the subtler variant where the limb extends
|
||||||
|
early so the paw drags forward for the last quarter of the cycle.
|
||||||
|
4. Hind limb driven by foreleg numbers → rear paws never leave the path and skid.
|
||||||
|
5. Ear behind the skull → horn.
|
||||||
|
6. Collar sized like a bandage, or wrapping the chest instead of the neck + throat.
|
||||||
|
7. Night mode dimming the white belly until it merges with the trail.
|
||||||
|
8. Zoomies as a fast trot; `scaleY` squash shortening the tail.
|
||||||
|
9. Hop removing the wrong leg, or being invisible in reduced motion, or repeating every cycle.
|
||||||
|
10. Paws welded to shins (no wrist), and contact dots on the wrong phase.
|
||||||
|
11. Trusting `getScreenCTM()` numbers, or trusting a probe that contradicts the screenshot.
|
||||||
@@ -0,0 +1,134 @@
|
|||||||
|
/**
|
||||||
|
* Budget compaction: replaces pi's summary step with one that has a size limit.
|
||||||
|
*
|
||||||
|
* Pi's default update prompt tells the model to "PRESERVE all existing information" from the previous
|
||||||
|
* summary, so every compaction re-emits the whole old summary plus the new work. On a long session the
|
||||||
|
* summary grows until rewriting it no longer fits in maxTokens, and auto-compaction fails forever with
|
||||||
|
* "generation hit the token cap". This extension instead:
|
||||||
|
*
|
||||||
|
* - writes a fresh checkpoint from the previous summary + new messages, limited to WORD_BUDGET words,
|
||||||
|
* condensing finished work instead of copying it forward
|
||||||
|
* - always summarizes with thinking off, so the whole output budget goes to the summary
|
||||||
|
* - retries once at half the budget if the reply still hits the cap, then falls back to pi's default
|
||||||
|
*
|
||||||
|
* Install: copy to ~/.pi/agent/extensions/ and restart pi. Override the limit with PI_COMPACT_WORDS.
|
||||||
|
*/
|
||||||
|
|
||||||
|
import type { ExtensionAPI, FileOperations } from "@earendil-works/pi-coding-agent";
|
||||||
|
import { convertToLlm, serializeConversation } from "@earendil-works/pi-coding-agent";
|
||||||
|
|
||||||
|
const WORD_BUDGET = Number(process.env.PI_COMPACT_WORDS) || 2500;
|
||||||
|
const MAX_READ_FILES = 25; // read-only files listed after the summary; modified files are always listed
|
||||||
|
|
||||||
|
const SYSTEM_PROMPT =
|
||||||
|
"You write compact checkpoints of coding sessions so another assistant can continue the work. " +
|
||||||
|
"You never continue the conversation, call tools or answer questions in it; you only output the checkpoint.";
|
||||||
|
|
||||||
|
function instructions(words: number, hasPrevious: boolean, focus?: string): string {
|
||||||
|
return `Write ONE fresh checkpoint of this session${hasPrevious ? ", merging the previous checkpoint with the new messages" : ""}.
|
||||||
|
|
||||||
|
HARD LIMIT: at most ${words} words in total. This replaces the old checkpoint, it does not append to it.
|
||||||
|
|
||||||
|
Rules:
|
||||||
|
- Describe the CURRENT state, not the history. Condense finished work into short outcome bullets; merge
|
||||||
|
related items; drop finished items that no longer matter for what comes next.
|
||||||
|
- Keep constraints, preferences and decisions that are still in force. Drop superseded ones.
|
||||||
|
- Keep exact file paths, function names, commands and error messages only where they are still needed.
|
||||||
|
- Leave out screenshot or temp-file lists, logs, tool output and code listings.
|
||||||
|
- If a part of the session is still in progress, say exactly where it stands.
|
||||||
|
|
||||||
|
Use this format:
|
||||||
|
|
||||||
|
## Goal
|
||||||
|
## Constraints & Preferences
|
||||||
|
## Progress
|
||||||
|
### Done
|
||||||
|
### In Progress
|
||||||
|
### Blocked
|
||||||
|
## Key Decisions
|
||||||
|
## Next Steps
|
||||||
|
## Critical Context${focus ? `\n\nAdditional focus from the user: ${focus}` : ""}`;
|
||||||
|
}
|
||||||
|
|
||||||
|
function fileLists(ops: FileOperations): { readFiles: string[]; modifiedFiles: string[]; text: string } {
|
||||||
|
const modified = new Set([...ops.edited, ...ops.written]);
|
||||||
|
const modifiedFiles = [...modified].sort();
|
||||||
|
const readFiles = [...ops.read].filter((f) => !modified.has(f)).sort();
|
||||||
|
const shown = readFiles.slice(0, MAX_READ_FILES);
|
||||||
|
const more = readFiles.length - shown.length;
|
||||||
|
let text = "";
|
||||||
|
if (shown.length) text += `\n\n<read-files>\n${shown.join("\n")}${more > 0 ? `\n(${more} more)` : ""}\n</read-files>`;
|
||||||
|
if (modifiedFiles.length) text += `\n\n<modified-files>\n${modifiedFiles.join("\n")}\n</modified-files>`;
|
||||||
|
return { readFiles, modifiedFiles, text };
|
||||||
|
}
|
||||||
|
|
||||||
|
export default function (pi: ExtensionAPI) {
|
||||||
|
pi.on("session_before_compact", async (event, ctx) => {
|
||||||
|
const { preparation, customInstructions, signal } = event;
|
||||||
|
const { messagesToSummarize, turnPrefixMessages, previousSummary, firstKeptEntryId, tokensBefore, fileOps } =
|
||||||
|
preparation;
|
||||||
|
const model = ctx.model;
|
||||||
|
if (!model) return; // let pi handle it
|
||||||
|
|
||||||
|
let body = "";
|
||||||
|
if (previousSummary) body += `<previous-checkpoint>\n${previousSummary}\n</previous-checkpoint>\n\n`;
|
||||||
|
if (messagesToSummarize.length)
|
||||||
|
body += `<conversation>\n${serializeConversation(convertToLlm(messagesToSummarize))}\n</conversation>\n\n`;
|
||||||
|
if (turnPrefixMessages.length)
|
||||||
|
body += `<current-turn-start>\n${serializeConversation(convertToLlm(turnPrefixMessages))}\n</current-turn-start>\n` +
|
||||||
|
"(The rest of the current turn is kept verbatim after the checkpoint.)\n\n";
|
||||||
|
|
||||||
|
let words = WORD_BUDGET;
|
||||||
|
for (let attempt = 1; attempt <= 2; attempt++) {
|
||||||
|
try {
|
||||||
|
ctx.ui.notify(`Compacting ${tokensBefore.toLocaleString()} tokens (limit ${words} words, thinking off)`, "info");
|
||||||
|
// No `reasoning` option: thinking off. Strata gets enable_thinking=false via the chat-template compat.
|
||||||
|
const response = await ctx.modelRegistry
|
||||||
|
.streamSimple(
|
||||||
|
model,
|
||||||
|
{
|
||||||
|
systemPrompt: SYSTEM_PROMPT,
|
||||||
|
messages: [
|
||||||
|
{
|
||||||
|
role: "user",
|
||||||
|
content: [{ type: "text", text: body + instructions(words, !!previousSummary, customInstructions) }],
|
||||||
|
timestamp: Date.now(),
|
||||||
|
},
|
||||||
|
],
|
||||||
|
},
|
||||||
|
{ maxTokens: model.maxTokens, signal, cacheRetention: "none" },
|
||||||
|
)
|
||||||
|
.result();
|
||||||
|
if (signal.aborted) return;
|
||||||
|
if (response.stopReason === "length") {
|
||||||
|
ctx.ui.notify(`Compaction summary hit the token cap at ${words} words`, "warning");
|
||||||
|
words = Math.floor(words / 2);
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
if (response.stopReason === "error") throw new Error(response.errorMessage || "summary request failed");
|
||||||
|
const summary = response.content
|
||||||
|
.filter((c): c is { type: "text"; text: string } => c.type === "text")
|
||||||
|
.map((c) => c.text)
|
||||||
|
.join("\n")
|
||||||
|
.trim();
|
||||||
|
if (!summary) throw new Error("empty summary");
|
||||||
|
const files = fileLists(fileOps);
|
||||||
|
return {
|
||||||
|
compaction: {
|
||||||
|
summary: summary + files.text,
|
||||||
|
firstKeptEntryId,
|
||||||
|
tokensBefore,
|
||||||
|
usage: response.usage,
|
||||||
|
details: { readFiles: files.readFiles, modifiedFiles: files.modifiedFiles },
|
||||||
|
},
|
||||||
|
};
|
||||||
|
} catch (error) {
|
||||||
|
if (signal.aborted) return;
|
||||||
|
ctx.ui.notify(`Budget compaction failed (${error instanceof Error ? error.message : error}); using pi's default`, "error");
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
ctx.ui.notify("Budget compaction hit the cap twice; using pi's default", "error");
|
||||||
|
return;
|
||||||
|
});
|
||||||
|
}
|
||||||
Reference in New Issue
Block a user