Écoute permanente
- Silero VAD dans le worker sherpa-onnx (vad.start/audio/stop) ; le renderer
envoie des trames de 128 ms pendant la commande et reçoit le segment WAV
complet (speech-start / segment). Repli sur le VAD énergétique si le modèle
n'est pas installé ou si l'option est désactivée.
- Modèle « silero-vad » (0,6 Mo) dans le catalogue, section « Fin de phrase »
dans Modèles locaux, réglage dans Paramètres → Micro.
Vision d'écran
- IPC system:screen-capture (desktopCapturer, JPEG 1600 px) ; bouton dans la
barre de commande ; « regarde mon écran… » joint la capture à la question
envoyée à Hermes (transport chat completions pour les images).
Actions locales
- IPC system:action à liste blanche : verrouiller la session, touches média
(volume, mute, lecture, piste), ouvrir une application connue ou une URL
http(s), presse-papiers, recherche de fichiers dans les dossiers utilisateur.
- Routeur d'intentions FR/EN exécuté avant Hermes (« coupe le son », « ouvre
Spotify », « verrouille la session »…), résultat affiché et lu ; désactivable.
Correctif
- Chat completions : les premiers fragments SSE arrivés avant l'événement de
démarrage étaient perdus (premier mot manquant) ; ils sont maintenant rejoués.
Tests : 26 tests unitaires (intentions locales ajoutées), e2e Electron sous
Xvfb avec les vrais modèles (Silero : un segment par phrase, 6,7 s d'audio
traités en 190 ms ; capture 109 ko reçue par Hermes ; intention locale
traitée sans Hermes).
Claude-Session: https://claude.ai/code/session_017Wn5VX9HNbJ7N54hR24u9Y
Co-authored-by: Claude <noreply@anthropic.com>
Electron main
- utility-process channel: sherpa output copied into V8 buffers (Electron rejects external
buffers), audio exchanged as base64; worker restart on timeout, identity-safe exit handling,
deterministic native unload by restart
- webhook: loopback by default without secret, UTF-8-safe body assembly, clean restart, port validation
- files IPC: realpath-based root check, openPath allow-list (reveal-only outside document folders),
Windows reserved names; store: debounced async atomic writes with backup of corrupt files;
logger: streaming writes with rotation; log message size cap
- HTTP stream proxy: socket released on idle timeout / renderer destroyed, id validation
- model manager: inactivity timeout, retrying rm/rename (Windows locks), engine stopped before
replacing a model; WAV decoder handles float/24-bit; IPC payload validation
- window: opaque rounded window on Windows, navigation lock-down, visibility events;
Ctrl+Alt+Escape instead of the Task Manager shortcut; single-instance guard; EVEFLOW_USER_DATA
Voice pipeline
- abort semantics (SendHandle.aborted), abort before the stream opens, session id prefixes per transport
- hands-free re-arm after replies without speech, start/stop race, no chime on auto re-arm,
no silence shipped to STT (400 ms pre-roll), no transcription of empty manual stops
- TTS: bounded prefetch, cancellable segments, non-interrupting notices, volume applied at play time
- SSE CRLF split, usage in chat completions, finish_reason length, phonetic regex hoisted
HUD
- core renderer: no canvas shadows, cached colours, reusable spectrum buffer, theme read on change,
30 fps idle, stops when the window is hidden; ping flashes on send / tool / speech
- deltas coalesced per animation frame; stable auto-scroll; narrow selectors everywhere
- bundled fonts (offline), reduce-motion fix, error toasts, ops drawer below 1180 px,
interim transcript and first-token latency in the core caption, Ctrl+K, dialog semantics,
switch/aria roles, compact widget cleanup, Whisper small recommended for French
- docs/ROADMAP.md: audit results, e2e results and the plan towards a real JARVIS
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017Wn5VX9HNbJ7N54hR24u9Y
Complete rewrite of the application:
- Toolchain: Electron 44, Vite 8, React 19, TypeScript 5.9, Zustand, Vitest;
main process rewritten in TypeScript (electron/), typed IPC contract (shared/),
sandboxed renderer with webSecurity on and an HTTP/SSE proxy in the main process.
- Voice: AudioWorklet microphone capture at 16 kHz with adaptive energy VAD and
auto-stop, WAV encoder, OpenAI-compatible STT, TTS queue with sentence-level
streaming, prefetch and Web Audio playback feeding an analyser; hands-free mode,
global push-to-talk hotkey, system-voice fallback.
- Hermes: full API client (capabilities, health, models, skills, toolsets,
sessions, jobs) with automatic transport selection: runs API (SSE lifecycle,
approvals, steer, stop) > sessions stream > chat completions with local tools;
tolerant event normalisation; webhook receiver hardened (secret, size limit).
- UI: JARVIS arc-reactor core on Canvas 2D reacting to the real audio signal,
holographic HUD layout (transcript, core, Hermes ops panel with tools/crons/
skills/sessions/link tabs, telemetry), settings drawer, approval modals,
compact floating widget, four themes, tray icon and global shortcuts.
- Removed the 3D Eve robot, three.js and legacy assets; migrated 1.x settings.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017Wn5VX9HNbJ7N54hR24u9Y
LogiFlowandClaude