fix(agent): GPU worker thread, async journalctl, correctness + cache fixes

Lightweight:
- GPU collection moves to a dedicated worker thread that owns the gfxinfo
  handle for the process lifetime. gfxinfo's active_gpu() runs a full NVML
  init/teardown (~20ms, blocking) and we were paying it on the async
  runtime for every collect — measured at ~80% of the agent's entire
  active CPU on a GPU machine. The handle holds Rc<Nvml> (not Send), so a
  thread + mpsc/oneshot channel pair confines it; a zero-total-VRAM reply
  is treated as a dead session (driver reload) and re-probed.
- journalctl now runs via tokio::process instead of blocking one of the
  two runtime workers for the duration of the subprocess.
- TtlCell (state.rs) replaces the four hand-rolled static TTL caches; a
  cached negative result now counts as fresh, so hosts with no matching
  temp sensor or GPU stop rescanning every request. Single lock+clone on
  the GPU cache hit path (was two).

Correctness:
- Process/child CPU times are now microseconds as documented; they were
  milliseconds, rendering 1000x too small next to (correct) thread times.
- Non-Linux per-process CPU%% clamps AFTER dividing by core count; a
  4-cores-busy process on an 8-core box reported 12.5% instead of 50%.
- Journal timestamps are real RFC 3339 UTC plus an additive timestamp_us
  field (sorting is now numeric); the old strings were Debug-formatted
  SystemTime mangled by string replace.
- Partition detection uses /sys/block on Linux: whole-disk filesystems on
  names like nvme0n1 or zram1 are no longer misclassified as partitions.
  One shared parent_disk_name() replaces two inline copies.
- New sampled_at_ms on the metrics payload (additive) records when the
  snapshot was actually collected, so clients can compute exact rates
  across the agent's TTL cache.

Security/robustness:
- key.pem is created 0600 (was umask default 0644, world-readable) and
  pre-1.51 keys are tightened on startup.
- Per-PID detail/journal caches now evict (60s max age, 64 entries max);
  they previously grew without bound under PID-walking clients.
- The two per-PID ws handlers collapse into one generic helper.
- /proc/<pid>/stat parsing unified in one comm-safe module.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
jasonwitty
2026-08-19 14:43:37 -07:00
co-authored by Claude Fable 5
parent fbc788c799
commit e4b0d9b9f6
13 changed files with 511 additions and 373 deletions
+4
View File
@@ -54,6 +54,10 @@ pub struct GpuInfo {
#[derive(Debug, Clone, Deserialize, Serialize)]
pub struct Metrics {
/// Epoch ms when the agent actually collected this snapshot (agents may
/// serve TTL-cached data). Absent on agents older than 1.51.
#[serde(default)]
pub sampled_at_ms: Option<u64>,
pub cpu_total: f32,
pub cpu_per_core: Vec<f32>,
pub mem_total: u64,