fix: full review pass (main process, voice pipeline, Hermes client, HUD) and e2e-validated local voice (v2.1.0.1)

Electron main
- utility-process channel: sherpa output copied into V8 buffers (Electron rejects external
  buffers), audio exchanged as base64; worker restart on timeout, identity-safe exit handling,
  deterministic native unload by restart
- webhook: loopback by default without secret, UTF-8-safe body assembly, clean restart, port validation
- files IPC: realpath-based root check, openPath allow-list (reveal-only outside document folders),
  Windows reserved names; store: debounced async atomic writes with backup of corrupt files;
  logger: streaming writes with rotation; log message size cap
- HTTP stream proxy: socket released on idle timeout / renderer destroyed, id validation
- model manager: inactivity timeout, retrying rm/rename (Windows locks), engine stopped before
  replacing a model; WAV decoder handles float/24-bit; IPC payload validation
- window: opaque rounded window on Windows, navigation lock-down, visibility events;
  Ctrl+Alt+Escape instead of the Task Manager shortcut; single-instance guard; EVEFLOW_USER_DATA

Voice pipeline
- abort semantics (SendHandle.aborted), abort before the stream opens, session id prefixes per transport
- hands-free re-arm after replies without speech, start/stop race, no chime on auto re-arm,
  no silence shipped to STT (400 ms pre-roll), no transcription of empty manual stops
- TTS: bounded prefetch, cancellable segments, non-interrupting notices, volume applied at play time
- SSE CRLF split, usage in chat completions, finish_reason length, phonetic regex hoisted

HUD
- core renderer: no canvas shadows, cached colours, reusable spectrum buffer, theme read on change,
  30 fps idle, stops when the window is hidden; ping flashes on send / tool / speech
- deltas coalesced per animation frame; stable auto-scroll; narrow selectors everywhere
- bundled fonts (offline), reduce-motion fix, error toasts, ops drawer below 1180 px,
  interim transcript and first-token latency in the core caption, Ctrl+K, dialog semantics,
  switch/aria roles, compact widget cleanup, Whisper small recommended for French
- docs/ROADMAP.md: audit results, e2e results and the plan towards a real JARVIS

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017Wn5VX9HNbJ7N54hR24u9Y
This commit is contained in:
Claude committed 2026-09-03 18:08:09 +00:00
1 parent b1b8d0d5d4
commit 5f0fa7a1e2
55 files changed
+1181 -433

No files matched your search

+12 -1
View File
@@ -88,6 +88,7 @@ export class MicCapture {
private active = false;
private options: CaptureOptions | null = null;
private sampleRate = 16_000;
private hadSpeech = false;
get isActive(): boolean {
return this.active;
@@ -98,6 +99,7 @@ export class MicCapture {
this.options = options;
this.chunks = [];
this.totalSamples = 0;
this.hadSpeech = false;
this.vad = options.mode === 'auto' ? new EnergyVad(options.vad ?? DEFAULT_VAD) : null;
if (!navigator.mediaDevices?.getUserMedia) {
@@ -169,6 +171,12 @@ export class MicCapture {
if (!this.active || !this.options) return;
this.chunks.push(samples);
this.totalSamples += samples.length;
// Before speech starts only a short pre-roll (~400 ms) is kept, so silence is never shipped to STT.
if (this.vad && !this.hadSpeech) {
while (this.chunks.length > 1 && this.totalSamples - this.chunks[0].length > this.sampleRate * 0.4) {
this.totalSamples -= this.chunks.shift()!.length;
}
}
const level = rms(samples);
this.options.callbacks.onLevel?.(Math.min(1, level * 6));
@@ -184,6 +192,7 @@ export class MicCapture {
private handleVad(signal: VadSignal): void {
switch (signal) {
case 'speech-start':
this.hadSpeech = true;
this.options?.callbacks.onSpeechStart?.();
break;
case 'speech-end':
@@ -222,10 +231,12 @@ export class MicCapture {
/** Stop capture. In manual mode this finalises the utterance; otherwise reports the reason. */
stop(reason: 'stopped' | 'no-speech' | 'too-short' = 'stopped'): void {
if (!this.active) return;
if (reason === 'stopped') {
// Manual stop: transcribe only if something was actually said (or no VAD is running).
if (reason === 'stopped' && (!this.vad || this.hadSpeech)) {
this.finishUtterance('stopped');
return;
}
if (reason === 'stopped') reason = 'no-speech';
const opts = this.options;
this.teardown();
opts?.callbacks.onEnd(reason);