Jason WittyandClaude Opus 5.5
cff8cf640e
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
92 lines
5.0 KiB
Markdown
92 lines
5.0 KiB
Markdown
# Change log and fixes
|
||
|
||
Newest first. Each entry: what broke, the cause, and what fixed it.
|
||
|
||
## 2026-10-27: power cap 250 W vs 275 W
|
||
|
||
With the case closed, alternating runs (250/275/250/275, fresh restart each) gave +4.6% prompt speed at
|
||
16K at 275 W (2,892 → 3,023 tok/s) and +1.6% at 4K; decode was unchanged within noise. Peak temps went
|
||
from 51/59°C to 53/63°C. **Kept:** `nvidia-power-limit.service` in
|
||
[`nixos/configuration.nix`](../nixos/configuration.nix) now sets 275 W at boot (applied with
|
||
`nixos-rebuild switch`; Strata kept running). Results:
|
||
[benchmarks/standard](../benchmarks/standard/README.md#power-cap-250-w-vs-275-w).
|
||
|
||
## 2026-10-26 (night): first standard benchmark and tuning arms
|
||
|
||
Ran Strata's `standard-v1` suite on the idle server, then one arm per tuning flag with a fresh restart
|
||
each. Baseline: 2,881–2,938 prompt tok/s at 16K, decode ~110–131. `--adapt-async 1` neutral,
|
||
`--pipeline-windows 2` −10–12% decode, `--prefill auto:32768` −22% prompt at 16K. No config change kept;
|
||
the original `strata-iq3_s.json` was restored (byte-identical, backup at
|
||
`/var/lib/strata/strata-iq3_s.json.orig-2026-10-26`). Results:
|
||
[benchmarks/standard](../benchmarks/standard/README.md).
|
||
|
||
## 2026-10-26 (evening): compaction loop again, summary snowball
|
||
|
||
**Symptom:** the same "generation hit the token cap" loop, now at the 32K cap, retrying every ~4 min
|
||
on a long `husky-walk` session.
|
||
|
||
**Cause:** pi's update prompt makes every compaction preserve the whole previous summary. After 13
|
||
compactions it was 86.6K chars (~22K tokens); thinking (~11K, ending on its own) plus rewriting it
|
||
exceeded 32,768 tokens. Not a model loop: the content was unique, just never trimmed. A ~22K summary
|
||
plus the 20K kept tail also made compactions come sooner.
|
||
|
||
**Fix:** [`pi/extensions/budget-compaction.ts`](../pi/extensions/budget-compaction.ts), a pi
|
||
extension that writes a fresh checkpoint with thinking off and enforces a 2,500-word limit (condense
|
||
passes, since the model ignored the limit when only asked). On a copy of the session: 44.5 s, 1,183
|
||
words, 23K tokens of context afterwards. See
|
||
[pi-client.md](pi-client.md#summary-snowball-and-the-budget-compaction-extension).
|
||
|
||
## 2026-10-26: pi auto-compaction loop
|
||
|
||
**Symptom:** on long pi sessions, "Auto-compaction failed: Summarization failed: generation hit the
|
||
token cap and the summary is incomplete", retried every ~2 minutes, forever. `/thinking off` +
|
||
`/compact` also failed.
|
||
|
||
**Cause:**
|
||
1. Pi caps the summary request at the model's `maxTokens` (was 16384). Strata gave thinking 75% of
|
||
that (12288), leaving ~4K for the summary, and every attempt ran to 16384 tokens (`length`).
|
||
2. Pi's thinking level never reached Strata: the compat had `supportsReasoningEffort: false` and no
|
||
`thinkingFormat`, so pi sent no thinking control and Strata thought at full effort on every
|
||
request. That's why the thinking-off workaround did nothing.
|
||
|
||
**Fix** (in [`pi/`](../pi)):
|
||
- `maxTokens` 16384 → 32768
|
||
- `compaction: { reserveTokens: 32768, keepRecentTokens: 20000 }` (compact at ~98K instead of ~115K)
|
||
- compat `thinkingFormat: "chat-template"` + `chatTemplateKwargs` for `enable_thinking` and
|
||
`reasoning_effort`
|
||
|
||
Verified: `pi -p --thinking off` produced no thinking; `--thinking medium` got a 24,576-token
|
||
budget; the stuck session compacted with thinking off.
|
||
|
||
## 2026-10-26: Strata, firewall, fans, GPU tuning
|
||
|
||
- Strata installed and serving IQ3_S ([strata.md](strata.md)). NixOS firewall turned on (22, 8080).
|
||
- `lovey-fans` fan control on the host, `gpu-temp-push` in the guest.
|
||
- 250 W cap and gpu-unlock in the guest ([nixos-guest.md](nixos-guest.md#gpu-tuning)).
|
||
`configuration.nix.pre-unlock` on the guest is the config from before; generation 1 is plain.
|
||
- Guest switched to static IP, NetworkManager removed.
|
||
|
||
## 2026-10-25: board swap and 4-DIMM instability
|
||
|
||
Moved to ASUS Maximus Z790 Hero + i9-14900KS, 4× 32 GB, both 3090s.
|
||
|
||
| Memory speed | Result |
|
||
|---|---|
|
||
| 5400 MT/s | Bootloop: kernel oopses at `RIP 0x0` in `pve-startall`, `qm`/`pveversion` segfaults |
|
||
| 5200 MT/s | Random `pvesm status` segfaults, host dropped off the network within minutes |
|
||
| **4800 JEDEC, 1.1 V** | Stable: 15-min soak ([`root/stab.sh`](../proxmox/root/stab.sh)) with 0 failures |
|
||
|
||
- Every PVE/Perl file checksum-verified clean: the corruption was in RAM, not on disk. Cause is
|
||
4-DIMM speed, not CPU degradation.
|
||
- Cheap tell: "Initramfs unpacking failed: ZSTD-compressed data is corrupt" at boot means memory is
|
||
unstable on this box. It disappears at 4800.
|
||
- `vmbr0` moved to `eno2` (Proxmox pins NIC names to MACs).
|
||
- NixOS 26.05 installed in VM 200, which got both GPUs and 112 GB.
|
||
|
||
## 2026-10-12: Z690 memory channel A fault
|
||
|
||
On the original MSI Z690 Carbon, memory channel A was contact-marginal: under load the box hard-froze
|
||
(purple/static on the iGPU, no logs). Memtest froze with 4 DIMMs or one DIMM in A after ~4 min with
|
||
0 errors; channel B alone was stable. CPU and RAM were cleared on another board. Resolved by moving
|
||
to the Z790 Hero. Crash-capture sysctls from this debugging are still on the host.
|