mirror of
https://github.com/R0m1k3/EveFlow.git
synced 2026-10-11 17:29:03 +02:00
Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
  LogiFlowandClaude
|
af8bdc5e9d | ||
|
LogiFlow
|
4f6074183f | ||
|
Claude
|
0d284fea9e |
No files matched your search
@@ -1,7 +1,7 @@
|
||||
# EveFlow 2 — Interface vocale JARVIS pour Hermes Agent
|
||||
|
||||
[](https://github.com/R0m1k3/EveFlow/actions)
|
||||
[](https://github.com/R0m1k3/EveFlow/releases)
|
||||
[](https://github.com/R0m1k3/EveFlow/releases)
|
||||
[](LICENSE)
|
||||
|
||||
**EveFlow** est un compagnon de bureau Windows qui transforme [Hermes Agent](https://hermes-agent.nousresearch.com/) en assistant vocal à la JARVIS : un noyau holographique réactif au son, une conversation en streaming, les outils, sous-agents, approbations, crons, skills et sessions d'Hermes pilotés depuis un seul HUD.
|
||||
@@ -23,7 +23,10 @@ La version 2 est une réécriture complète : plus de robot 3D, un pipeline voca
|
||||
* **Détection d'activité vocale** (seuil adaptatif, sensibilité et silence de fin réglables) : l'enregistrement s'arrête tout seul quand vous avez fini de parler.
|
||||
* **Mains libres** : le micro se réactive après chaque réponse.
|
||||
* **Modèles intégrés, hors ligne** (sherpa-onnx dans un processus séparé) : reconnaissance Whisper (base, small, large-v3 turbo) ou SenseVoice, synthèse Kokoro v1.0 (voix française Siwis et voix anglaises) ou Piper (Siwis, Tom, UPMC). Les modèles se téléchargent depuis **Paramètres → Modèles locaux** et tournent sur le processeur.
|
||||
* **Mot d'activation** en mains libres : seules les phrases commençant par « Jarvis » (configurable) partent vers Hermes, le reste est ignoré ; un « Jarvis » seul ouvre une fenêtre d'écoute.
|
||||
* **Fin de phrase neuronale** : en écoute permanente, Silero VAD (0,6 Mo, sherpa-onnx) décide du début et de la fin de la commande à la place du seuil d'énergie ; moins de faux départs sur le bruit, coupure plus nette. Repli automatique sur le VAD énergétique si le modèle n'est pas installé.
|
||||
* **Vision d'écran** : « Jarvis, regarde mon écran » (ou le bouton de la barre de commande) joint une capture de l'écran principal à la question envoyée à Hermes.
|
||||
* **Actions locales instantanées** : « verrouille la session », « monte le son », « coupe le son », « piste suivante », « ouvre Spotify », « ouvre github.com »… exécutées sur le PC sans passer par Hermes, résultat lu à voix haute. Liste blanche d'actions dans le processus principal, désactivable dans les paramètres.
|
||||
* **Écoute permanente** : un détecteur de mot-clé de 3 Mo (sherpa-onnx, keyword spotting) tourne en continu sur le micro, quasi gratuit en CPU. « Jarvis » (ou n'importe quel mot-clé) ouvre l'écoute, « Jarvis, allume… » envoie directement la commande, et le mot coupe la voix en cours. Alternative : filtre du mot après transcription en mains libres.
|
||||
* **STT externe** : n'importe quelle API `/v1/audio/transcriptions` compatible OpenAI (Qwen3-ASR, Whisper, Speaches, faster-whisper-server, LocalAI, OpenAI). Repli sur la reconnaissance Chromium.
|
||||
* **TTS externe** : API `/v1/audio/speech` compatible OpenAI, voix système Windows ou Google Translate. Lecture phrase par phrase pendant le streaming, préchargement du segment suivant, coupure instantanée.
|
||||
* Raccourcis globaux : `Ctrl+Shift+Espace` (micro), `Ctrl+Shift+J` (afficher/masquer), `Ctrl+Shift+Échap` (couper la voix).
|
||||
|
||||
+22
-14
@@ -9,8 +9,11 @@
|
||||
| HUD arc-reactor réactif au son | Fait | Canvas 2D optimisé (pas d'ombres, couleurs en cache, 30 fps en veille, arrêt fenêtre masquée) |
|
||||
| Reconnaissance vocale locale | Fait | Whisper base/small/turbo via sherpa-onnx dans un processus utilitaire |
|
||||
| Synthèse vocale locale | Fait | Kokoro v1.0 (voix française Siwis) et Piper fr |
|
||||
| Mot d'activation | Fait, mode « après transcription » | Filtre « Jarvis … » en mains libres, tolérant aux erreurs de transcription |
|
||||
| Détection de fin de phrase | Fait | VAD énergétique adaptatif, pré-roll 400 ms |
|
||||
| Mot d'activation permanent | Fait (2.2.0) | Keyword spotting sherpa-onnx en continu ; mot-clé libre encodé en BPE ; validé sur audio réel (détection, zéro faux positif sur le test anglais) |
|
||||
| Mot d'activation après transcription | Fait | Filtre « Jarvis … » en mains libres, tolérant aux erreurs de transcription |
|
||||
| Détection de fin de phrase | Fait (2.3.0) | Silero VAD neuronal dans le worker (segment renvoyé au renderer), repli sur le VAD énergétique si le modèle manque |
|
||||
| Vision d'écran | Fait (2.3.0) | Capture `desktopCapturer` jointe à la requête Hermes (bouton, ou « regarde mon écran ») |
|
||||
| Actions système locales | Fait (2.3.0) | Verrouillage, volume et touches média, ouvrir une application ou une URL, presse-papiers, recherche de fichiers ; intentions courtes exécutées sans passer par Hermes |
|
||||
| Hermes : runs, sessions, chat completions | Fait | Transport choisi selon `/v1/capabilities` |
|
||||
| Approbations, steer, stop | Fait | Modales, injection de consigne en cours de run |
|
||||
| Crons, skills, toolsets, sessions | Fait | Panneau Hermes Ops |
|
||||
@@ -21,25 +24,27 @@
|
||||
|
||||
Sources : [jarvis-desktop-ai](https://github.com/ccarloshenri/jarvis-desktop-ai), [JarvisAi](https://github.com/PanPenek/JarvisAi), [bertrandmbanwi/Jarvis](https://github.com/bertrandmbanwi/Jarvis), [InterGenJLU/jarvis](https://github.com/InterGenJLU/jarvis), [livekit-wakeword](https://livekit.com/blog/livekit-wakeword), [sherpa-onnx keyword spotting](https://k2-fsa.github.io/sherpa/onnx/kws/index.html), [Hermes Agent features](https://hermes-agent.nousresearch.com/docs/user-guide/features/overview).
|
||||
|
||||
1. **Mot d'activation permanent, quasi gratuit en CPU.** Les projets de référence utilisent openWakeWord (« hey jarvis ») ou un modèle de keyword spotting qui écoute en continu, au lieu de transcrire chaque phrase. sherpa-onnx fournit un modèle KWS anglais de 3,3 Mo qui accepte n'importe quel mot-clé sans réentraînement ; « JARVIS » s'encode `▁JA R VI S` avec son modèle BPE (vérifié). C'est le prochain chantier prioritaire : streaming du micro vers le worker, détection en continu, puis capture de la commande.
|
||||
2. **VAD neuronal (Silero) au lieu du seuil d'énergie.** Fin de phrase plus nette (environ 500 ms gagnés) et beaucoup moins de faux départs sur le bruit ambiant. Silero est déjà livré dans sherpa-onnx (`silero_vad.onnx`, 0,6 Mo).
|
||||
1. **Mot d'activation permanent, quasi gratuit en CPU.** Les projets de référence utilisent openWakeWord (« hey jarvis ») ou un modèle de keyword spotting qui écoute en continu, au lieu de transcrire chaque phrase. sherpa-onnx fournit un modèle KWS anglais de 3,3 Mo qui accepte n'importe quel mot-clé sans réentraînement ; « JARVIS » s'encode `▁JA R VI S` avec son modèle BPE (vérifié). Livré en 2.2.0 (voir l'étape 1 ci-dessous).
|
||||
2. **VAD neuronal (Silero) au lieu du seuil d'énergie.** Fin de phrase plus nette (environ 500 ms gagnés) et beaucoup moins de faux départs sur le bruit ambiant. Silero est déjà livré dans sherpa-onnx (`silero_vad.onnx`, 0,6 Mo). Livré en 2.3.0.
|
||||
3. **Latence perçue sous la seconde.** Les références visent 1 s entre la fin de parole et le premier mot prononcé : STT rapide, premier token en streaming, TTS phrase par phrase (déjà en place), et un modèle Hermes rapide pour la conversation courante.
|
||||
4. **Vision d'écran.** Capture d'écran à la demande (« Jarvis, qu'est-ce que je regarde ? ») envoyée à Hermes comme image, ou lecture d'une fenêtre. Hermes accepte déjà les images inline.
|
||||
5. **Actions système locales.** Ouvrir une application, régler le volume, verrouiller la session, chercher un fichier ; ce sont des outils EveFlow côté client à exposer à Hermes (mode chat completions) ou un petit serveur MCP local que Hermes appelle.
|
||||
4. **Vision d'écran.** Capture d'écran à la demande (« Jarvis, qu'est-ce que je regarde ? ») envoyée à Hermes comme image, ou lecture d'une fenêtre. Hermes accepte déjà les images inline. Livré en 2.3.0 (transport chat completions pour les images).
|
||||
5. **Actions système locales.** Ouvrir une application, régler le volume, verrouiller la session, chercher un fichier. Livré en 2.3.0 sous forme d'intentions courtes exécutées localement avant Hermes ; reste à exposer les mêmes actions à Hermes via un serveur MCP local.
|
||||
6. **Proactivité.** Notifications parlées à l'arrivée d'un cron, rappel, événement webhook, avec un résumé plutôt que la lecture intégrale ; c'est en partie fait via le webhook, à enrichir avec des règles (heures calmes, priorité).
|
||||
7. **Mémoire et personnalisation.** Hermes gère la mémoire longue durée (`X-Hermes-Session-Key`) ; côté EveFlow, un profil (nom, préférences de voix, style de réponse) déjà transmis dans les instructions.
|
||||
|
||||
## Plan proposé
|
||||
|
||||
### Étape 1 (courte) : écoute permanente
|
||||
- Streamer l'audio du micro (16 kHz, blocs de 128 ms) du renderer vers le worker via IPC.
|
||||
- Dans le worker : `KeywordSpotter` sherpa-onnx (modèle gigaspeech 3,3 Mo, mot-clé configurable encodé automatiquement avec `bpe.model` via un petit encodeur BPE côté Node ou un dictionnaire pré-encodé pour « jarvis », « eve », « hey jarvis », « ok jarvis »).
|
||||
- À la détection : chime, capture de la commande avec Silero VAD, transcription, envoi.
|
||||
- Consommation attendue : quelques pourcents d'un cœur, pas de transcription en continu.
|
||||
### Étape 1 : écoute permanente — livrée en 2.2.0
|
||||
- Le renderer garde un seul flux micro (AudioWorklet 16 kHz) et envoie des blocs de 256 ms au processus principal, qui alimente le `KeywordSpotter` sherpa-onnx dans le worker.
|
||||
- Mots-clés encodés en BPE (table SentencePiece pour les mots courants, repli glouton sur le vocabulaire du modèle), sensibilité réglable (seuil 0,45 → 0,12).
|
||||
- À la détection : chime, capture de la commande sur le même flux (VAD énergétique, pré-roll 400 ms), transcription locale ou API, envoi à Hermes ; retour automatique à l'écoute.
|
||||
- Fin de phrase Silero livrée en 2.3.0 : le renderer envoie des trames de 128 ms au worker pendant la commande, le worker renvoie le segment WAV complet ; VAD énergétique en repli.
|
||||
|
||||
### Étape 2 : vision et actions locales
|
||||
- Outil `capture_screen` (Electron `desktopCapturer`) qui joint une capture à la requête Hermes.
|
||||
- Outils système : `open_app`, `set_volume`, `lock_session`, `find_file`, `clipboard` ; exposés en chat completions et via un serveur MCP local pour les transports runs/sessions.
|
||||
### Étape 2 : vision et actions locales — livrée en 2.3.0
|
||||
- Capture d'écran (`desktopCapturer`, JPEG 1600 px) jointe à la requête Hermes : bouton dans la barre de commande, ou phrase « regarde mon écran… » à l'oral comme à l'écrit.
|
||||
- Actions locales (processus principal, liste blanche) : verrouiller, volume/mute/lecture/piste, ouvrir une application connue ou une URL http(s), presse-papiers, recherche de fichiers dans Documents/Bureau/Téléchargements/Images.
|
||||
- Routeur d'intentions FR/EN (`src/services/localCommands.ts`) exécuté avant l'envoi à Hermes ; désactivable dans Paramètres → Micro. Les phrases composées (« ouvre X puis… ») partent à Hermes.
|
||||
- Reste à faire : exposer ces actions à Hermes lui-même (serveur MCP local) pour qu'un run puisse les enchaîner.
|
||||
|
||||
### Étape 3 : conversation plus naturelle
|
||||
- Barge-in réel : couper la voix dès que l'utilisateur parle (déjà préparé, à valider avec l'annulation d'écho Windows).
|
||||
@@ -60,5 +65,8 @@ Sources : [jarvis-desktop-ai](https://github.com/ccarloshenri/jarvis-desktop-ai)
|
||||
| Kokoro (fr) → Whisper base, phrase courte de 2 s | synthèse 2,9 s, transcription 2,5 s, texte approximatif |
|
||||
| Kokoro (fr) → Whisper base, phrase de 7 s | transcription correcte à un mot près |
|
||||
| Piper (fr) → Whisper base | transcription exacte |
|
||||
| Silero VAD sur phrase Kokoro de 2,3 s (2.3.0) | un seul segment, début et fin détectés, 6,7 s d'audio traités en 190 ms |
|
||||
| Capture d'écran → Hermes (2.3.0) | JPEG de 107 ko reçu côté Hermes (mock chat completions) |
|
||||
| Intention locale « coupe le son » (2.3.0) | traitée sans Hermes, résultat affiché dans le fil |
|
||||
|
||||
Sur un PC à 28 cœurs les temps sont nettement plus courts. Whisper small est maintenant recommandé pour le français.
|
||||
@@ -0,0 +1,184 @@
|
||||
/**
|
||||
* Local "JARVIS" actions: screenshot for vision, and an allow-list of system actions
|
||||
* (lock, open an application or URL, media keys, clipboard, file search).
|
||||
*/
|
||||
import { app, clipboard, desktopCapturer, ipcMain, screen, shell } from 'electron';
|
||||
import { execFile } from 'node:child_process';
|
||||
import fs from 'node:fs';
|
||||
import os from 'node:os';
|
||||
import path from 'node:path';
|
||||
import { IPC } from '../../shared/ipc';
|
||||
import type { SystemAction, SystemActionResult } from '../../shared/bridge';
|
||||
import { log } from '../logger';
|
||||
|
||||
const MEDIA_KEYS: Record<string, number> = {
|
||||
'volume-up': 175,
|
||||
'volume-down': 174,
|
||||
mute: 173,
|
||||
'play-pause': 179,
|
||||
next: 176,
|
||||
previous: 177
|
||||
};
|
||||
|
||||
/** Applications the assistant may launch by name (Windows aliases + common Linux/macOS names). */
|
||||
const APP_ALIASES: Record<string, string[]> = {
|
||||
'bloc-notes': ['notepad.exe', 'gedit', 'TextEdit'],
|
||||
notepad: ['notepad.exe', 'gedit', 'TextEdit'],
|
||||
calculatrice: ['calc.exe', 'gnome-calculator', 'Calculator'],
|
||||
calc: ['calc.exe', 'gnome-calculator', 'Calculator'],
|
||||
explorateur: ['explorer.exe', 'nautilus', 'Finder'],
|
||||
explorer: ['explorer.exe', 'nautilus', 'Finder'],
|
||||
terminal: ['wt.exe', 'cmd.exe', 'gnome-terminal', 'Terminal'],
|
||||
cmd: ['cmd.exe'],
|
||||
powershell: ['powershell.exe'],
|
||||
paint: ['mspaint.exe'],
|
||||
chrome: ['chrome', 'google-chrome', 'Google Chrome'],
|
||||
edge: ['msedge', 'microsoft-edge', 'Microsoft Edge'],
|
||||
firefox: ['firefox', 'Firefox'],
|
||||
vscode: ['code', 'Visual Studio Code'],
|
||||
code: ['code', 'Visual Studio Code'],
|
||||
spotify: ['spotify', 'Spotify'],
|
||||
discord: ['discord', 'Discord'],
|
||||
steam: ['steam', 'Steam'],
|
||||
word: ['winword', 'Microsoft Word'],
|
||||
excel: ['excel', 'Microsoft Excel'],
|
||||
outlook: ['outlook', 'Microsoft Outlook'],
|
||||
teams: ['ms-teams', 'Microsoft Teams'],
|
||||
'task manager': ['taskmgr.exe'],
|
||||
'gestionnaire des tâches': ['taskmgr.exe'],
|
||||
paramètres: ['ms-settings:'],
|
||||
settings: ['ms-settings:']
|
||||
};
|
||||
|
||||
function run(cmd: string, args: string[], timeoutMs = 8000): Promise<{ code: number; out: string }> {
|
||||
return new Promise((resolve) => {
|
||||
execFile(cmd, args, { timeout: timeoutMs, windowsHide: true }, (err, stdout, stderr) => {
|
||||
resolve({ code: err ? 1 : 0, out: `${stdout}${stderr}`.trim() });
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
async function openApp(name: string): Promise<SystemActionResult> {
|
||||
const key = name.trim().toLowerCase();
|
||||
if (!key || key.length > 40) return { ok: false, message: 'Nom d’application invalide' };
|
||||
const candidates = APP_ALIASES[key] ?? [key.replace(/[^a-z0-9 ._-]/gi, '')];
|
||||
if (process.platform === 'win32') {
|
||||
for (const candidate of candidates) {
|
||||
if (candidate.endsWith(':')) {
|
||||
await shell.openExternal(candidate);
|
||||
return { ok: true, message: `${name} ouvert` };
|
||||
}
|
||||
// `start` resolves App Paths, PATH and Start Menu names.
|
||||
const result = await run('cmd.exe', ['/c', 'start', '', candidate]);
|
||||
if (result.code === 0) return { ok: true, message: `${name} lancé` };
|
||||
}
|
||||
return { ok: false, message: `Impossible de lancer ${name}` };
|
||||
}
|
||||
for (const candidate of candidates) {
|
||||
const result = process.platform === 'darwin' ? await run('open', ['-a', candidate]) : await run('sh', ['-c', `command -v ${JSON.stringify(candidate)} >/dev/null && (nohup ${JSON.stringify(candidate)} >/dev/null 2>&1 &)`]);
|
||||
if (result.code === 0) return { ok: true, message: `${name} lancé` };
|
||||
}
|
||||
return { ok: false, message: `Application introuvable : ${name}` };
|
||||
}
|
||||
|
||||
async function lockSession(): Promise<SystemActionResult> {
|
||||
if (process.platform === 'win32') {
|
||||
const r = await run('rundll32.exe', ['user32.dll,LockWorkStation']);
|
||||
return { ok: r.code === 0, message: r.code === 0 ? 'Session verrouillée' : r.out };
|
||||
}
|
||||
if (process.platform === 'darwin') {
|
||||
const r = await run('osascript', ['-e', 'tell application "System Events" to keystroke "q" using {command down, control down}']);
|
||||
return { ok: r.code === 0, message: r.out };
|
||||
}
|
||||
const r = await run('sh', ['-c', 'loginctl lock-session || xdg-screensaver lock || gnome-screensaver-command -l']);
|
||||
return { ok: r.code === 0, message: r.code === 0 ? 'Session verrouillée' : r.out };
|
||||
}
|
||||
|
||||
async function mediaKey(key: string): Promise<SystemActionResult> {
|
||||
const code = MEDIA_KEYS[key];
|
||||
if (!code) return { ok: false, message: 'Touche inconnue' };
|
||||
if (process.platform === 'win32') {
|
||||
const script = `$s=Add-Type -MemberDefinition '[DllImport("user32.dll")] public static extern void keybd_event(byte b,byte s,uint f,UIntPtr e);' -Name K -Namespace W -PassThru; $s::keybd_event(${code},0,0,[UIntPtr]::Zero); $s::keybd_event(${code},0,2,[UIntPtr]::Zero)`;
|
||||
const r = await run('powershell.exe', ['-NoProfile', '-NonInteractive', '-Command', script]);
|
||||
return { ok: r.code === 0, message: r.code === 0 ? key : r.out };
|
||||
}
|
||||
const xdo: Record<string, string> = { 'volume-up': 'XF86AudioRaiseVolume', 'volume-down': 'XF86AudioLowerVolume', mute: 'XF86AudioMute', 'play-pause': 'XF86AudioPlay', next: 'XF86AudioNext', previous: 'XF86AudioPrev' };
|
||||
const r = await run('sh', ['-c', `xdotool key ${xdo[key]}`]);
|
||||
return { ok: r.code === 0, message: r.code === 0 ? key : 'xdotool indisponible' };
|
||||
}
|
||||
|
||||
async function findFiles(query: string): Promise<SystemActionResult> {
|
||||
const needle = query.trim().toLowerCase();
|
||||
if (needle.length < 2) return { ok: false, message: 'Recherche trop courte' };
|
||||
const roots = ['Documents', 'Desktop', 'Downloads', 'Pictures'].map((d) => path.join(os.homedir(), d));
|
||||
const hits: string[] = [];
|
||||
const deadline = Date.now() + 4000;
|
||||
const walk = async (dir: string, depth: number) => {
|
||||
if (depth > 4 || hits.length >= 25 || Date.now() > deadline) return;
|
||||
let entries: fs.Dirent[] = [];
|
||||
try {
|
||||
entries = await fs.promises.readdir(dir, { withFileTypes: true });
|
||||
} catch {
|
||||
return;
|
||||
}
|
||||
for (const entry of entries) {
|
||||
if (entry.name.startsWith('.') || entry.name === 'node_modules') continue;
|
||||
const full = path.join(dir, entry.name);
|
||||
if (entry.name.toLowerCase().includes(needle)) hits.push(full);
|
||||
if (entry.isDirectory()) await walk(full, depth + 1);
|
||||
if (hits.length >= 25) return;
|
||||
}
|
||||
};
|
||||
for (const root of roots) await walk(root, 0);
|
||||
return { ok: true, message: `${hits.length} résultat(s)`, data: hits };
|
||||
}
|
||||
|
||||
export async function captureScreen(maxWidth = 1600): Promise<string> {
|
||||
const display = screen.getPrimaryDisplay();
|
||||
const scale = Math.min(1, maxWidth / display.size.width);
|
||||
const sources = await desktopCapturer.getSources({
|
||||
types: ['screen'],
|
||||
thumbnailSize: { width: Math.round(display.size.width * scale), height: Math.round(display.size.height * scale) }
|
||||
});
|
||||
const primary = sources.find((s) => s.display_id === String(display.id)) ?? sources[0];
|
||||
if (!primary) throw new Error('Aucun écran capturable');
|
||||
return primary.thumbnail.toJPEG(82).toString('base64').replace(/^/, 'data:image/jpeg;base64,');
|
||||
}
|
||||
|
||||
export async function runSystemAction(action: SystemAction): Promise<SystemActionResult> {
|
||||
switch (action.type) {
|
||||
case 'lock':
|
||||
return lockSession();
|
||||
case 'open-app':
|
||||
return openApp(String(action.name ?? ''));
|
||||
case 'open-url': {
|
||||
const url = String(action.url ?? '');
|
||||
if (!/^https?:\/\//i.test(url) || url.length > 2048) return { ok: false, message: 'URL refusée' };
|
||||
await shell.openExternal(url);
|
||||
return { ok: true, message: 'Ouvert dans le navigateur' };
|
||||
}
|
||||
case 'media':
|
||||
return mediaKey(String(action.key));
|
||||
case 'clipboard-read':
|
||||
return { ok: true, data: (await clipboard.readText()).slice(0, 20_000) };
|
||||
case 'clipboard-write':
|
||||
await clipboard.writeText(String(action.text ?? '').slice(0, 100_000));
|
||||
return { ok: true, message: 'Copié dans le presse-papiers' };
|
||||
case 'find-files':
|
||||
return findFiles(String(action.query ?? ''));
|
||||
default:
|
||||
return { ok: false, message: 'Action inconnue' };
|
||||
}
|
||||
}
|
||||
|
||||
export function registerSystemIpc(): void {
|
||||
ipcMain.handle(IPC.screenCapture, (_e, maxWidth?: number) => captureScreen(Number(maxWidth) || 1600));
|
||||
ipcMain.handle(IPC.systemAction, async (_e, action: SystemAction) => {
|
||||
if (!action || typeof action.type !== 'string') throw new Error('Action invalide');
|
||||
log('INFO', 'system', `action ${action.type}`);
|
||||
const result = await runSystemAction(action);
|
||||
if (!result.ok) log('WARN', 'system', `action ${action.type} failed: ${result.message}`);
|
||||
return result;
|
||||
});
|
||||
void app;
|
||||
}
|
||||
@@ -9,6 +9,7 @@ import { getSharedDirectory, registerFilesIpc } from './ipc/files';
|
||||
import { createMainWindow, getMainWindow, getWindowMode, setWindowMode, toggleWindowVisibility, windowEvents } from './window';
|
||||
import { getWebhookStatus, startWebhookServer, stopWebhookServer } from './webhook';
|
||||
import { registerVoiceIpc } from './voice/ipc';
|
||||
import { registerSystemIpc } from './ipc/system';
|
||||
import { stopEngine } from './voice/engine';
|
||||
|
||||
// Audio playback must never be blocked behind a user gesture (TTS starts on incoming events).
|
||||
@@ -159,6 +160,7 @@ async function boot(): Promise<void> {
|
||||
registerFilesIpc();
|
||||
registerCoreIpc();
|
||||
registerVoiceIpc();
|
||||
registerSystemIpc();
|
||||
createMainWindow();
|
||||
createTray();
|
||||
registerShortcuts();
|
||||
|
||||
+14
-4
@@ -13,8 +13,8 @@ import {
|
||||
type WebhookStatus,
|
||||
type WindowMode
|
||||
} from '../shared/ipc';
|
||||
import type { EveFlowBridge, Unsubscribe } from '../shared/bridge';
|
||||
import { VOICE_IPC, type SynthesizeRequest, type SynthesizeResult, type TranscribeRequest, type TranscribeResult, type VoiceDownloadProgress, type VoiceEngineStatus, type VoiceModelStatus } from '../shared/voice';
|
||||
import type { EveFlowBridge, SystemAction, SystemActionResult, Unsubscribe } from '../shared/bridge';
|
||||
import { VOICE_IPC, type KwsDetection, type KwsStartRequest, type VadEvent, type VadStartRequest, type SynthesizeRequest, type SynthesizeResult, type TranscribeRequest, type TranscribeResult, type VoiceDownloadProgress, type VoiceEngineStatus, type VoiceModelStatus } from '../shared/voice';
|
||||
|
||||
function subscribe<T>(channel: string, callback: (payload: T) => void): Unsubscribe {
|
||||
const listener = (_event: Electron.IpcRendererEvent, payload: T) => callback(payload);
|
||||
@@ -43,7 +43,9 @@ const api: EveFlowBridge = {
|
||||
},
|
||||
system: {
|
||||
metrics: () => ipcRenderer.invoke(IPC.metrics) as Promise<SystemMetrics>,
|
||||
appInfo: () => ipcRenderer.invoke(IPC.appInfo) as Promise<AppInfo>
|
||||
appInfo: () => ipcRenderer.invoke(IPC.appInfo) as Promise<AppInfo>,
|
||||
captureScreen: (maxWidth?: number) => ipcRenderer.invoke(IPC.screenCapture, maxWidth) as Promise<string>,
|
||||
action: (action: SystemAction) => ipcRenderer.invoke(IPC.systemAction, action) as Promise<SystemActionResult>
|
||||
},
|
||||
files: {
|
||||
readLocal: (filePath: string) => ipcRenderer.invoke(IPC.readLocalFile, filePath) as Promise<string>,
|
||||
@@ -69,7 +71,15 @@ const api: EveFlowBridge = {
|
||||
onProgress: (cb: (progress: VoiceDownloadProgress) => void) => subscribe<VoiceDownloadProgress>(VOICE_IPC.modelsProgress, cb),
|
||||
transcribe: (req: TranscribeRequest) => ipcRenderer.invoke(VOICE_IPC.transcribe, req) as Promise<TranscribeResult>,
|
||||
synthesize: (req: SynthesizeRequest) => ipcRenderer.invoke(VOICE_IPC.synthesize, req) as Promise<SynthesizeResult>,
|
||||
unload: (id?: string) => ipcRenderer.invoke(VOICE_IPC.unload, id) as Promise<unknown>
|
||||
unload: (id?: string) => ipcRenderer.invoke(VOICE_IPC.unload, id) as Promise<unknown>,
|
||||
kwsStart: (req: KwsStartRequest) => ipcRenderer.invoke(VOICE_IPC.kwsStart, req) as Promise<{ accepted: string[]; rejected: string[] }>,
|
||||
kwsStop: () => ipcRenderer.invoke(VOICE_IPC.kwsStop) as Promise<void>,
|
||||
kwsAudio: (pcm: Uint8Array, sampleRate: number) => ipcRenderer.send(VOICE_IPC.kwsAudio, pcm, sampleRate),
|
||||
onKwsDetected: (cb: (detection: KwsDetection) => void) => subscribe<KwsDetection>(VOICE_IPC.kwsDetected, cb),
|
||||
vadStart: (req: VadStartRequest) => ipcRenderer.invoke(VOICE_IPC.vadStart, req) as Promise<void>,
|
||||
vadStop: () => ipcRenderer.invoke(VOICE_IPC.vadStop) as Promise<void>,
|
||||
vadAudio: (pcm: Uint8Array, sampleRate: number) => ipcRenderer.send(VOICE_IPC.vadAudio, pcm, sampleRate),
|
||||
onVadEvent: (cb: (event: VadEvent) => void) => subscribe<VadEvent>(VOICE_IPC.vadEvent, cb)
|
||||
}
|
||||
};
|
||||
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
import type { VoiceModelSpec, VoiceSpeaker } from '../../shared/voice';
|
||||
|
||||
const ASR = 'https://github.com/k2-fsa/sherpa-onnx/releases/download/asr-models';
|
||||
const KWS = 'https://github.com/k2-fsa/sherpa-onnx/releases/download/kws-models';
|
||||
const TTS = 'https://github.com/k2-fsa/sherpa-onnx/releases/download/tts-models';
|
||||
|
||||
const KOKORO_SPEAKERS: VoiceSpeaker[] = [
|
||||
@@ -67,6 +68,37 @@ export const VOICE_CATALOG: VoiceModelSpec[] = [
|
||||
dir: 'sherpa-onnx-sense-voice-zh-en-ja-ko-yue-int8-2024-07-17',
|
||||
files: ['model.int8.onnx', 'tokens.txt']
|
||||
},
|
||||
{
|
||||
id: 'kws-en',
|
||||
kind: 'kws',
|
||||
engine: 'kws-transducer',
|
||||
name: 'Détecteur de mot-clé (zipformer, 3 Mo)',
|
||||
description: 'Écoute permanente du mot d’activation, quasi gratuite en CPU. Mot-clé libre (« jarvis », « hey jarvis »…).',
|
||||
languages: ['en', 'fr'],
|
||||
sizeMb: 4,
|
||||
url: `${KWS}/sherpa-onnx-kws-zipformer-gigaspeech-3.3M-2024-01-01.tar.bz2`,
|
||||
dir: 'sherpa-onnx-kws-zipformer-gigaspeech-3.3M-2024-01-01',
|
||||
files: [
|
||||
'encoder-epoch-12-avg-2-chunk-16-left-64.int8.onnx',
|
||||
'decoder-epoch-12-avg-2-chunk-16-left-64.int8.onnx',
|
||||
'joiner-epoch-12-avg-2-chunk-16-left-64.int8.onnx',
|
||||
'tokens.txt'
|
||||
],
|
||||
recommended: true
|
||||
},
|
||||
{
|
||||
id: 'silero-vad',
|
||||
kind: 'vad',
|
||||
engine: 'silero',
|
||||
name: 'Silero VAD (fin de phrase neuronale, 0,6 Mo)',
|
||||
description: 'Détecte précisément le début et la fin de la parole pendant l’écoute permanente ; moins de faux départs sur le bruit.',
|
||||
languages: ['multi'],
|
||||
sizeMb: 1,
|
||||
url: `${ASR}/silero_vad.onnx`,
|
||||
dir: 'silero-vad',
|
||||
files: ['silero_vad.onnx'],
|
||||
recommended: true
|
||||
},
|
||||
{
|
||||
id: 'kokoro-v1',
|
||||
kind: 'tts',
|
||||
|
||||
+119
-3
@@ -2,13 +2,23 @@
|
||||
* Host side of the voice worker: spawns the utility process on demand, correlates
|
||||
* requests and responses, restarts the worker if it crashes.
|
||||
*/
|
||||
import { utilityProcess, type UtilityProcess } from 'electron';
|
||||
import { utilityProcess, type UtilityProcess, type WebContents } from 'electron';
|
||||
import { createHash } from 'node:crypto';
|
||||
import fs from 'node:fs';
|
||||
import path from 'node:path';
|
||||
import type { SynthesizeRequest, SynthesizeResult, TranscribeRequest, TranscribeResult, VoiceEngineStatus } from '../../shared/voice';
|
||||
import { VOICE_IPC, type KwsDetection, type KwsStartRequest, type SynthesizeRequest, type SynthesizeResult, type TranscribeRequest, type TranscribeResult, type VadEvent, type VadStartRequest, type VoiceEngineStatus } from '../../shared/voice';
|
||||
import { buildKeywordsFile, parseTokens } from '../../shared/keywords';
|
||||
import { findModel } from './catalog';
|
||||
import { isInstalled, modelDir, modelsDir } from './models';
|
||||
import { log } from '../logger';
|
||||
|
||||
/** Renderer that receives keyword detections while spotting is active. */
|
||||
let kwsSubscriber: WebContents | null = null;
|
||||
let kwsActive = false;
|
||||
let kwsRequest: KwsStartRequest | null = null;
|
||||
let vadSubscriber: WebContents | null = null;
|
||||
let vadActive = false;
|
||||
|
||||
interface Pending {
|
||||
resolve: (value: unknown) => void;
|
||||
reject: (err: Error) => void;
|
||||
@@ -34,7 +44,28 @@ function spawn(): UtilityProcess {
|
||||
child = proc;
|
||||
proc.stdout?.on('data', (d: Buffer) => log('DEBUG', 'voice-worker', d.toString().trim()));
|
||||
proc.stderr?.on('data', (d: Buffer) => log('WARN', 'voice-worker', d.toString().trim()));
|
||||
proc.on('message', (msg: { id: number; ok: boolean; result?: unknown; error?: string }) => {
|
||||
proc.on('message', (msg: { id?: number; type?: string; ok?: boolean; result?: unknown; error?: string; keyword?: string; at?: number; event?: { type: string; wav?: string; durationSec?: number } }) => {
|
||||
if (msg.type === 'vad.event' && msg.event) {
|
||||
if (vadSubscriber && !vadSubscriber.isDestroyed()) {
|
||||
const ev = msg.event;
|
||||
const payload: VadEvent =
|
||||
ev.type === 'segment' && ev.wav
|
||||
? { type: 'segment', wav: new Uint8Array(Buffer.from(ev.wav, 'base64')), durationSec: ev.durationSec ?? 0 }
|
||||
: ev.type === 'speech-start'
|
||||
? { type: 'speech-start' }
|
||||
: { type: 'error', message: 'événement VAD inconnu' };
|
||||
vadSubscriber.send(VOICE_IPC.vadEvent, payload);
|
||||
}
|
||||
return;
|
||||
}
|
||||
if (msg.type === 'kws.detected') {
|
||||
if (kwsSubscriber && !kwsSubscriber.isDestroyed()) {
|
||||
kwsSubscriber.send(VOICE_IPC.kwsDetected, { keyword: msg.keyword ?? '', at: msg.at ?? Date.now() } satisfies KwsDetection);
|
||||
}
|
||||
log('INFO', 'voice', `wake word detected: ${msg.keyword}`);
|
||||
return;
|
||||
}
|
||||
if (typeof msg.id !== 'number') return;
|
||||
const p = pending.get(msg.id);
|
||||
if (!p) return;
|
||||
pending.delete(msg.id);
|
||||
@@ -48,6 +79,8 @@ function spawn(): UtilityProcess {
|
||||
if (child !== proc) return;
|
||||
child = null;
|
||||
rejectAll('Le moteur vocal local s’est arrêté de façon inattendue.');
|
||||
// Keyword spotting survives a worker restart: re-arm on the next audio frame.
|
||||
if (kwsActive) kwsArmed = false;
|
||||
});
|
||||
log('INFO', 'voice', 'voice worker started');
|
||||
return proc;
|
||||
@@ -108,6 +141,89 @@ export async function synthesize(req: SynthesizeRequest): Promise<SynthesizeResu
|
||||
return { ...result, wav: new Uint8Array(buffer.buffer, buffer.byteOffset, buffer.byteLength) };
|
||||
}
|
||||
|
||||
let kwsArmed = false;
|
||||
const SENSITIVITY_THRESHOLD: Record<number, number> = { 1: 0.45, 2: 0.35, 3: 0.25, 4: 0.18, 5: 0.12 };
|
||||
const SENSITIVITY_SCORE: Record<number, number> = { 1: 1.0, 2: 1.0, 3: 1.2, 4: 1.5, 5: 2.0 };
|
||||
|
||||
/** Start keyword spotting; the keywords file is derived from the phrases and the model vocabulary. */
|
||||
export async function kwsStart(req: KwsStartRequest, sender: WebContents): Promise<{ accepted: string[]; rejected: string[] }> {
|
||||
const spec = findModel(req.modelId);
|
||||
if (!spec || spec.kind !== 'kws') throw new Error('Modèle de détection introuvable');
|
||||
if (!isInstalled(spec)) throw new Error('Détecteur de mot-clé non installé (Paramètres → Modèles locaux).');
|
||||
const dir = modelDir(spec);
|
||||
const vocab = parseTokens(fs.readFileSync(path.join(dir, 'tokens.txt'), 'utf8'));
|
||||
const phrases = req.keywords.map((k) => String(k).slice(0, 40)).filter(Boolean).slice(0, 8);
|
||||
const file = buildKeywordsFile(phrases, vocab);
|
||||
if (file.accepted.length === 0) throw new Error(`Aucun mot d’activation encodable : ${file.rejected.join(', ')}`);
|
||||
const hash = createHash('sha1').update(file.content).digest('hex').slice(0, 10);
|
||||
const keywordsFile = path.join(modelsDir(), `keywords-${hash}.txt`);
|
||||
fs.writeFileSync(keywordsFile, file.content, 'utf8');
|
||||
const sensitivity = Math.min(5, Math.max(1, Math.round(req.sensitivity))) as 1 | 2 | 3 | 4 | 5;
|
||||
kwsSubscriber = sender;
|
||||
kwsRequest = req;
|
||||
await request({ type: 'kws.start', model: { id: spec.id, engine: spec.engine, dir, files: spec.files }, keywordsFile, threshold: SENSITIVITY_THRESHOLD[sensitivity], score: SENSITIVITY_SCORE[sensitivity] }, 60_000);
|
||||
kwsActive = true;
|
||||
kwsArmed = true;
|
||||
log('INFO', 'voice', `keyword spotting on: ${file.accepted.join(', ')} (threshold ${SENSITIVITY_THRESHOLD[sensitivity]})`);
|
||||
return { accepted: file.accepted, rejected: file.rejected };
|
||||
}
|
||||
|
||||
export async function kwsStop(): Promise<void> {
|
||||
kwsActive = false;
|
||||
kwsArmed = false;
|
||||
kwsSubscriber = null;
|
||||
if (child) await request({ type: 'kws.stop' }, 10_000).catch(() => undefined);
|
||||
}
|
||||
|
||||
/** Feed 16-bit PCM from the renderer (fire-and-forget). */
|
||||
export function kwsFeed(pcm: Uint8Array, sampleRate: number): void {
|
||||
if (!kwsActive) return;
|
||||
const proc = spawn();
|
||||
if (!kwsArmed) {
|
||||
// Worker restarted: re-create the spotter before feeding audio.
|
||||
if (kwsRequest && kwsSubscriber) {
|
||||
kwsArmed = true;
|
||||
kwsStart(kwsRequest, kwsSubscriber).catch((err) => log('WARN', 'voice', `kws re-arm failed: ${(err as Error).message}`));
|
||||
}
|
||||
return;
|
||||
}
|
||||
const b64 = Buffer.from(pcm.buffer, pcm.byteOffset, pcm.byteLength).toString('base64');
|
||||
try {
|
||||
proc.postMessage({ id: -1, type: 'kws.audio', pcm: b64, sampleRate });
|
||||
} catch (err) {
|
||||
log('WARN', 'voice', `kws feed failed: ${(err as Error).message}`);
|
||||
}
|
||||
}
|
||||
|
||||
/** Neural end-of-speech detection (Silero) on frames streamed by the renderer. */
|
||||
export async function vadStart(req: VadStartRequest, sender: WebContents): Promise<void> {
|
||||
const spec = findModel(req.modelId);
|
||||
if (!spec || spec.kind !== 'vad') throw new Error('Modèle VAD introuvable');
|
||||
if (!isInstalled(spec)) throw new Error('Silero VAD non installé (Paramètres → Modèles locaux).');
|
||||
vadSubscriber = sender;
|
||||
await request(
|
||||
{ type: 'vad.start', model: { id: spec.id, engine: spec.engine, dir: modelDir(spec), files: spec.files }, silenceMs: req.silenceMs, threshold: req.threshold, maxUtteranceSec: req.maxUtteranceSec },
|
||||
30_000
|
||||
);
|
||||
vadActive = true;
|
||||
}
|
||||
|
||||
export async function vadStop(): Promise<void> {
|
||||
vadActive = false;
|
||||
vadSubscriber = null;
|
||||
if (child) await request({ type: 'vad.stop' }, 10_000).catch(() => undefined);
|
||||
}
|
||||
|
||||
export function vadFeed(pcm: Uint8Array, sampleRate: number): void {
|
||||
if (!vadActive || !child) return;
|
||||
const b64 = Buffer.from(pcm.buffer, pcm.byteOffset, pcm.byteLength).toString('base64');
|
||||
try {
|
||||
child.postMessage({ id: -1, type: 'vad.audio', pcm: b64, sampleRate });
|
||||
} catch (err) {
|
||||
log('WARN', 'voice', `vad feed failed: ${(err as Error).message}`);
|
||||
}
|
||||
}
|
||||
|
||||
/** Native model memory is only released deterministically by restarting the worker. */
|
||||
export function unload(_modelId?: string): Promise<unknown> {
|
||||
stopEngine();
|
||||
|
||||
+20
-2
@@ -1,6 +1,6 @@
|
||||
import { ipcMain } from 'electron';
|
||||
import { VOICE_IPC, type SynthesizeRequest, type TranscribeRequest } from '../../shared/voice';
|
||||
import { engineStatus, synthesize, transcribe, unload } from './engine';
|
||||
import { VOICE_IPC, type KwsStartRequest, type SynthesizeRequest, type TranscribeRequest, type VadStartRequest } from '../../shared/voice';
|
||||
import { engineStatus, kwsFeed, kwsStart, kwsStop, synthesize, transcribe, unload, vadFeed, vadStart, vadStop } from './engine';
|
||||
import { cancelDownload, downloadModel, listModels, removeModel } from './models';
|
||||
|
||||
export function registerVoiceIpc(): void {
|
||||
@@ -27,4 +27,22 @@ export function registerVoiceIpc(): void {
|
||||
return synthesize({ ...req, speaker: Number.isFinite(req.speaker) ? req.speaker : 0, speed: Number.isFinite(req.speed) ? req.speed : 1 });
|
||||
});
|
||||
ipcMain.handle(VOICE_IPC.unload, (_e, id?: string) => unload(id));
|
||||
ipcMain.handle(VOICE_IPC.kwsStart, (event, req: KwsStartRequest) => {
|
||||
if (!req || !Array.isArray(req.keywords) || typeof req.modelId !== 'string') throw new Error('Requête invalide');
|
||||
return kwsStart({ ...req, keywords: req.keywords.filter((k) => typeof k === 'string'), sensitivity: Number(req.sensitivity) || 3 }, event.sender);
|
||||
});
|
||||
ipcMain.handle(VOICE_IPC.kwsStop, () => kwsStop());
|
||||
ipcMain.handle(VOICE_IPC.vadStart, (event, req: VadStartRequest) => {
|
||||
if (!req || typeof req.modelId !== 'string') throw new Error('Requête invalide');
|
||||
return vadStart({ modelId: req.modelId, silenceMs: Number(req.silenceMs) || 700, threshold: Number(req.threshold) || 0.5, maxUtteranceSec: Number(req.maxUtteranceSec) || 30 }, event.sender);
|
||||
});
|
||||
ipcMain.handle(VOICE_IPC.vadStop, () => vadStop());
|
||||
ipcMain.on(VOICE_IPC.vadAudio, (_e, pcm: unknown, sampleRate: unknown) => {
|
||||
if (!(pcm instanceof Uint8Array) || pcm.byteLength === 0 || pcm.byteLength > 1024 * 1024) return;
|
||||
vadFeed(pcm, typeof sampleRate === 'number' && sampleRate > 0 ? sampleRate : 16000);
|
||||
});
|
||||
ipcMain.on(VOICE_IPC.kwsAudio, (_e, pcm: unknown, sampleRate: unknown) => {
|
||||
if (!(pcm instanceof Uint8Array) || pcm.byteLength === 0 || pcm.byteLength > 1024 * 1024) return;
|
||||
kwsFeed(pcm, typeof sampleRate === 'number' && sampleRate > 0 ? sampleRate : 16000);
|
||||
});
|
||||
}
|
||||
+144
-2
@@ -18,7 +18,13 @@ type Request =
|
||||
| { id: number; type: 'status' }
|
||||
| { id: number; type: 'transcribe'; model: ModelRef; wav: Uint8Array | string; language: string }
|
||||
| { id: number; type: 'synthesize'; model: ModelRef; text: string; speaker: number; speed: number }
|
||||
| { id: number; type: 'unload'; modelId?: string };
|
||||
| { id: number; type: 'unload'; modelId?: string }
|
||||
| { id: number; type: 'kws.start'; model: ModelRef; keywordsFile: string; threshold: number; score: number }
|
||||
| { id: number; type: 'kws.audio'; pcm: string; sampleRate: number }
|
||||
| { id: number; type: 'kws.stop' }
|
||||
| { id: number; type: 'vad.start'; model: ModelRef; silenceMs: number; threshold: number; maxUtteranceSec: number }
|
||||
| { id: number; type: 'vad.audio'; pcm: string; sampleRate: number }
|
||||
| { id: number; type: 'vad.stop' };
|
||||
|
||||
type Response = { id: number; ok: true; result: unknown } | { id: number; ok: false; error: string };
|
||||
|
||||
@@ -29,6 +35,23 @@ type Sherpa = {
|
||||
decode: (s: unknown) => void;
|
||||
getResult: (s: unknown) => { text: string; lang?: string };
|
||||
};
|
||||
Vad: new (config: unknown, bufferSizeInSeconds: number) => {
|
||||
acceptWaveform: (samples: Float32Array) => void;
|
||||
isEmpty: () => boolean;
|
||||
isDetected: () => boolean;
|
||||
pop: () => void;
|
||||
clear: () => void;
|
||||
front: (enableExternalBuffer?: boolean) => { start: number; samples: Float32Array };
|
||||
reset: () => void;
|
||||
flush: () => void;
|
||||
};
|
||||
KeywordSpotter: new (config: unknown) => {
|
||||
createStream: () => KwsStream;
|
||||
isReady: (s: KwsStream) => boolean;
|
||||
decode: (s: KwsStream) => void;
|
||||
reset: (s: KwsStream) => void;
|
||||
getResult: (s: KwsStream) => { keyword?: string };
|
||||
};
|
||||
OfflineTts: new (config: unknown) => {
|
||||
numSpeakers: number;
|
||||
sampleRate: number;
|
||||
@@ -37,7 +60,11 @@ type Sherpa = {
|
||||
version: string;
|
||||
};
|
||||
|
||||
type KwsStream = { acceptWaveform: (w: { sampleRate: number; samples: Float32Array }) => void };
|
||||
let sherpa: Sherpa | null = null;
|
||||
let kws: { spotter: InstanceType<Sherpa['KeywordSpotter']>; stream: KwsStream } | null = null;
|
||||
let vad: { detector: InstanceType<Sherpa['Vad']>; speaking: boolean; windowSize: number; carry: Float32Array } | null = null;
|
||||
let notify: ((message: unknown) => void) | null = null;
|
||||
let loadError: string | null = null;
|
||||
|
||||
function loadSherpa(): Sherpa {
|
||||
@@ -225,13 +252,124 @@ function encodeWav(samples: Float32Array, sampleRate: number): Uint8Array {
|
||||
return new Uint8Array(buffer);
|
||||
}
|
||||
|
||||
function startKws(model: ModelRef, keywordsFile: string, threshold: number, score: number): void {
|
||||
const s = loadSherpa();
|
||||
kws = null;
|
||||
const p = (f: string) => path.join(model.dir, f);
|
||||
const enc = model.files.find((f) => f.startsWith('encoder')) ?? 'encoder.int8.onnx';
|
||||
const dec = model.files.find((f) => f.startsWith('decoder')) ?? 'decoder.int8.onnx';
|
||||
const join = model.files.find((f) => f.startsWith('joiner')) ?? 'joiner.int8.onnx';
|
||||
const spotter = new s.KeywordSpotter({
|
||||
featConfig: { sampleRate: 16000, featureDim: 80 },
|
||||
modelConfig: { transducer: { encoder: p(enc), decoder: p(dec), joiner: p(join) }, tokens: p('tokens.txt'), numThreads: 1, provider: 'cpu', debug: 0 },
|
||||
maxActivePaths: 4,
|
||||
numTrailingBlanks: 1,
|
||||
keywordsScore: score,
|
||||
keywordsThreshold: threshold,
|
||||
keywordsFile
|
||||
});
|
||||
kws = { spotter, stream: spotter.createStream() };
|
||||
}
|
||||
|
||||
function feedKws(pcmBase64: string, sampleRate: number): void {
|
||||
if (!kws) return;
|
||||
const bytes = Buffer.from(pcmBase64, 'base64');
|
||||
const int16 = new Int16Array(bytes.buffer, bytes.byteOffset, Math.floor(bytes.byteLength / 2));
|
||||
let samples: Float32Array = new Float32Array(int16.length);
|
||||
for (let i = 0; i < int16.length; i++) samples[i] = int16[i] / 32768;
|
||||
if (sampleRate !== 16000) samples = resampleTo16k(samples, sampleRate);
|
||||
kws.stream.acceptWaveform({ sampleRate: 16000, samples });
|
||||
while (kws.spotter.isReady(kws.stream)) {
|
||||
kws.spotter.decode(kws.stream);
|
||||
const result = kws.spotter.getResult(kws.stream);
|
||||
if (result.keyword) {
|
||||
kws.spotter.reset(kws.stream);
|
||||
notify?.({ type: 'kws.detected', keyword: result.keyword, at: Date.now() });
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function startVad(model: ModelRef, silenceMs: number, threshold: number, maxUtteranceSec: number): void {
|
||||
const s = loadSherpa();
|
||||
const windowSize = 512;
|
||||
const detector = new s.Vad(
|
||||
{
|
||||
sileroVad: {
|
||||
model: path.join(model.dir, 'silero_vad.onnx'),
|
||||
threshold: Math.max(0.1, Math.min(0.95, threshold)),
|
||||
minSilenceDuration: Math.max(0.15, silenceMs / 1000),
|
||||
minSpeechDuration: 0.2,
|
||||
windowSize,
|
||||
maxSpeechDuration: Math.max(3, maxUtteranceSec)
|
||||
},
|
||||
sampleRate: 16000,
|
||||
numThreads: 1,
|
||||
provider: 'cpu',
|
||||
debug: 0
|
||||
},
|
||||
Math.max(10, maxUtteranceSec + 5)
|
||||
);
|
||||
vad = { detector, speaking: false, windowSize, carry: new Float32Array(0) };
|
||||
}
|
||||
|
||||
function feedVad(pcmBase64: string, sampleRate: number): void {
|
||||
if (!vad) return;
|
||||
const bytes = Buffer.from(pcmBase64, 'base64');
|
||||
const int16 = new Int16Array(bytes.buffer, bytes.byteOffset, Math.floor(bytes.byteLength / 2));
|
||||
let samples: Float32Array = new Float32Array(int16.length);
|
||||
for (let i = 0; i < int16.length; i++) samples[i] = int16[i] / 32768;
|
||||
if (sampleRate !== 16000) samples = resampleTo16k(samples, sampleRate);
|
||||
// Silero expects fixed windows: keep the remainder for the next frame.
|
||||
const merged = new Float32Array(vad.carry.length + samples.length);
|
||||
merged.set(vad.carry, 0);
|
||||
merged.set(samples, vad.carry.length);
|
||||
const usable = merged.length - (merged.length % vad.windowSize);
|
||||
for (let i = 0; i < usable; i += vad.windowSize) {
|
||||
vad.detector.acceptWaveform(merged.subarray(i, i + vad.windowSize));
|
||||
const detected = vad.detector.isDetected();
|
||||
if (detected && !vad.speaking) {
|
||||
vad.speaking = true;
|
||||
notify?.({ type: 'vad.event', event: { type: 'speech-start' } });
|
||||
}
|
||||
while (!vad.detector.isEmpty()) {
|
||||
const segment = vad.detector.front(false);
|
||||
vad.detector.pop();
|
||||
vad.speaking = false;
|
||||
const wav = encodeWav(segment.samples, 16000);
|
||||
notify?.({
|
||||
type: 'vad.event',
|
||||
event: { type: 'segment', wav: Buffer.from(wav.buffer, wav.byteOffset, wav.byteLength).toString('base64'), durationSec: segment.samples.length / 16000 }
|
||||
});
|
||||
}
|
||||
}
|
||||
vad.carry = merged.slice(usable);
|
||||
}
|
||||
|
||||
// ── request handling ───────────────────────────────────────────────────────
|
||||
function handle(req: Request): unknown {
|
||||
switch (req.type) {
|
||||
case 'vad.start':
|
||||
startVad(req.model, req.silenceMs, req.threshold, req.maxUtteranceSec);
|
||||
return { ok: true };
|
||||
case 'vad.audio':
|
||||
feedVad(req.pcm, req.sampleRate);
|
||||
return { ok: true };
|
||||
case 'vad.stop':
|
||||
vad = null;
|
||||
return { ok: true };
|
||||
case 'kws.start':
|
||||
startKws(req.model, req.keywordsFile, req.threshold, req.score);
|
||||
return { ok: true };
|
||||
case 'kws.audio':
|
||||
feedKws(req.pcm, req.sampleRate);
|
||||
return { ok: true };
|
||||
case 'kws.stop':
|
||||
kws = null;
|
||||
return { ok: true };
|
||||
case 'status': {
|
||||
try {
|
||||
const s = loadSherpa();
|
||||
return { available: true, version: s.version, loaded: [...recognizers.keys(), ...synthesizers.keys()] };
|
||||
return { available: true, version: s.version, loaded: [...recognizers.keys(), ...synthesizers.keys(), ...(kws ? ['kws'] : [])] };
|
||||
} catch (err) {
|
||||
return { available: false, error: (err as Error).message, loaded: [] };
|
||||
}
|
||||
@@ -271,6 +409,8 @@ function handle(req: Request): unknown {
|
||||
} else {
|
||||
recognizers.clear();
|
||||
synthesizers.clear();
|
||||
kws = null;
|
||||
vad = null;
|
||||
}
|
||||
return { ok: true };
|
||||
}
|
||||
@@ -290,7 +430,9 @@ function respond(req: Request): Response {
|
||||
// Electron utility process transport, with a child_process fallback for tests.
|
||||
const parentPort = (process as unknown as { parentPort?: { on: (ev: 'message', cb: (e: { data: Request }) => void) => void; postMessage: (m: unknown) => void } }).parentPort;
|
||||
if (parentPort) {
|
||||
notify = (m) => parentPort.postMessage(m);
|
||||
parentPort.on('message', (event) => parentPort.postMessage(respond(event.data)));
|
||||
} else if (process.send) {
|
||||
notify = (m) => process.send!(m);
|
||||
process.on('message', (msg: Request) => process.send!(respond(msg)));
|
||||
}
|
||||
+2
-2
@@ -1,7 +1,7 @@
|
||||
{
|
||||
"name": "eveflow",
|
||||
"version": "2.1.0",
|
||||
"releaseVersion": "2.1.0.1",
|
||||
"version": "2.3.0",
|
||||
"releaseVersion": "2.3.0",
|
||||
"description": "JARVIS-style desktop HUD for Hermes Agent: voice, streaming runs, scheduled jobs, skills and telemetry",
|
||||
"main": "dist-electron/main.js",
|
||||
"private": true,
|
||||
|
||||
+36
-4
@@ -12,6 +12,10 @@ import type {
|
||||
WindowMode
|
||||
} from './ipc';
|
||||
import type {
|
||||
KwsDetection,
|
||||
KwsStartRequest,
|
||||
VadEvent,
|
||||
VadStartRequest,
|
||||
SynthesizeRequest,
|
||||
SynthesizeResult,
|
||||
TranscribeRequest,
|
||||
@@ -44,10 +48,6 @@ export interface EveFlowBridge {
|
||||
streamAbort: (id: string) => void;
|
||||
onStreamEvent: (cb: (event: HttpStreamEvent) => void) => Unsubscribe;
|
||||
};
|
||||
system: {
|
||||
metrics: () => Promise<SystemMetrics>;
|
||||
appInfo: () => Promise<AppInfo>;
|
||||
};
|
||||
files: {
|
||||
readLocal: (filePath: string) => Promise<string>;
|
||||
writeShared: (filename: string, content: string, isBase64?: boolean) => Promise<{ path: string; url: string }>;
|
||||
@@ -72,5 +72,37 @@ export interface EveFlowBridge {
|
||||
transcribe: (req: TranscribeRequest) => Promise<TranscribeResult>;
|
||||
synthesize: (req: SynthesizeRequest) => Promise<SynthesizeResult>;
|
||||
unload: (id?: string) => Promise<unknown>;
|
||||
kwsStart: (req: KwsStartRequest) => Promise<{ accepted: string[]; rejected: string[] }>;
|
||||
kwsStop: () => Promise<void>;
|
||||
/** Fire-and-forget 16-bit PCM frames for the keyword spotter. */
|
||||
kwsAudio: (pcm: Uint8Array, sampleRate: number) => void;
|
||||
onKwsDetected: (cb: (detection: KwsDetection) => void) => Unsubscribe;
|
||||
vadStart: (req: VadStartRequest) => Promise<void>;
|
||||
vadStop: () => Promise<void>;
|
||||
vadAudio: (pcm: Uint8Array, sampleRate: number) => void;
|
||||
onVadEvent: (cb: (event: VadEvent) => void) => Unsubscribe;
|
||||
};
|
||||
system: {
|
||||
metrics: () => Promise<SystemMetrics>;
|
||||
appInfo: () => Promise<AppInfo>;
|
||||
/** Screenshot of the primary display as a JPEG data URL. */
|
||||
captureScreen: (maxWidth?: number) => Promise<string>;
|
||||
/** Allow-listed local actions (lock session, open app/url, media keys, clipboard). */
|
||||
action: (action: SystemAction) => Promise<SystemActionResult>;
|
||||
};
|
||||
}
|
||||
|
||||
export type SystemAction =
|
||||
| { type: 'lock' }
|
||||
| { type: 'open-app'; name: string }
|
||||
| { type: 'open-url'; url: string }
|
||||
| { type: 'media'; key: 'volume-up' | 'volume-down' | 'mute' | 'play-pause' | 'next' | 'previous' }
|
||||
| { type: 'clipboard-read' }
|
||||
| { type: 'clipboard-write'; text: string }
|
||||
| { type: 'find-files'; query: string };
|
||||
|
||||
export interface SystemActionResult {
|
||||
ok: boolean;
|
||||
message?: string;
|
||||
data?: unknown;
|
||||
}
|
||||
@@ -111,6 +111,8 @@ export const IPC = {
|
||||
httpStreamEvent: 'http:stream:event',
|
||||
metrics: 'system:metrics',
|
||||
appInfo: 'app:info',
|
||||
screenCapture: 'system:screen-capture',
|
||||
systemAction: 'system:action',
|
||||
readLocalFile: 'files:read-local',
|
||||
writeSharedFile: 'files:write-shared',
|
||||
openPath: 'files:open-path',
|
||||
|
||||
@@ -0,0 +1,108 @@
|
||||
/**
|
||||
* Encodes wake phrases into the BPE token sequences expected by the sherpa-onnx keyword
|
||||
* spotter (gigaspeech BPE-500 model). Common phrases use sequences produced by the real
|
||||
* SentencePiece model; anything else falls back to a greedy longest-match over the vocabulary,
|
||||
* which is a close approximation for short words.
|
||||
*/
|
||||
|
||||
const KNOWN: Record<string, string> = {
|
||||
'JARVIS': '▁JA R VI S',
|
||||
'HEY JARVIS': '▁HE Y ▁JA R VI S',
|
||||
'OK JARVIS': '▁O K ▁JA R VI S',
|
||||
'EVE': '▁E VE',
|
||||
'HEY EVE': '▁HE Y ▁E VE',
|
||||
'COMPUTER': '▁COMP U TER',
|
||||
'HEY COMPUTER': '▁HE Y ▁COMP U TER',
|
||||
'FRIDAY': '▁F RI DAY',
|
||||
'HEY FRIDAY': '▁HE Y ▁F RI DAY',
|
||||
'ALFRED': '▁A L F RE D',
|
||||
'HERMES': '▁HER ME S',
|
||||
'HEY HERMES': '▁HE Y ▁HER ME S',
|
||||
'ASSISTANT': '▁AS S IST ANT',
|
||||
'OK GOOGLE': '▁O K ▁GO O G LE',
|
||||
'ALEXA': '▁A LE X A',
|
||||
'SIRI': '▁S I RI',
|
||||
'HEY SIRI': '▁HE Y ▁S I RI',
|
||||
'NOVA': '▁NO V A',
|
||||
'ATLAS': '▁AT LA S',
|
||||
'HAL': '▁HA L'
|
||||
};
|
||||
|
||||
export function normalizeKeyword(phrase: string): string {
|
||||
return phrase
|
||||
.normalize('NFD')
|
||||
.replace(/[̀-ͯ]/g, '')
|
||||
.toUpperCase()
|
||||
.replace(/[^A-Z ]+/g, ' ')
|
||||
.replace(/\s+/g, ' ')
|
||||
.trim();
|
||||
}
|
||||
|
||||
/** Parse a sherpa tokens.txt ("piece id" per line) into the set of pieces. */
|
||||
export function parseTokens(tokensFile: string): Set<string> {
|
||||
const pieces = new Set<string>();
|
||||
for (const line of tokensFile.split(/\r?\n/)) {
|
||||
const piece = line.trim().split(/\s+/)[0];
|
||||
if (piece && !piece.startsWith('<')) pieces.add(piece);
|
||||
}
|
||||
return pieces;
|
||||
}
|
||||
|
||||
function greedyWord(word: string, vocab: Set<string>): string[] | null {
|
||||
const out: string[] = [];
|
||||
let i = 0;
|
||||
let first = true;
|
||||
while (i < word.length) {
|
||||
let matched = '';
|
||||
for (let len = word.length - i; len >= 1; len--) {
|
||||
const candidate = (first ? '▁' : '') + word.slice(i, i + len);
|
||||
if (vocab.has(candidate)) {
|
||||
matched = candidate;
|
||||
break;
|
||||
}
|
||||
}
|
||||
if (!matched && first) {
|
||||
// No word-initial piece: use a bare "▁" if available, then continue without the prefix.
|
||||
if (vocab.has('▁')) out.push('▁');
|
||||
first = false;
|
||||
continue;
|
||||
}
|
||||
if (!matched) return null;
|
||||
out.push(matched);
|
||||
i += matched.length - (first ? 1 : 0);
|
||||
first = false;
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
/** Encode a phrase into space-separated BPE pieces, or null when it cannot be represented. */
|
||||
export function encodeKeyword(phrase: string, vocab: Set<string>): string | null {
|
||||
const normalized = normalizeKeyword(phrase);
|
||||
if (!normalized) return null;
|
||||
if (KNOWN[normalized]) return KNOWN[normalized];
|
||||
const pieces: string[] = [];
|
||||
for (const word of normalized.split(' ')) {
|
||||
const encoded = greedyWord(word, vocab);
|
||||
if (!encoded) return null;
|
||||
pieces.push(...encoded);
|
||||
}
|
||||
return pieces.join(' ');
|
||||
}
|
||||
|
||||
/** Build the keywords file content: one line per phrase, with the readable label. */
|
||||
export function buildKeywordsFile(phrases: string[], vocab: Set<string>): { content: string; accepted: string[]; rejected: string[] } {
|
||||
const lines: string[] = [];
|
||||
const accepted: string[] = [];
|
||||
const rejected: string[] = [];
|
||||
for (const phrase of phrases) {
|
||||
const encoded = encodeKeyword(phrase, vocab);
|
||||
const label = normalizeKeyword(phrase).toLowerCase().replace(/ /g, '_');
|
||||
if (!encoded || !label) {
|
||||
rejected.push(phrase);
|
||||
continue;
|
||||
}
|
||||
lines.push(`${encoded} @${label}`);
|
||||
accepted.push(label);
|
||||
}
|
||||
return { content: lines.join('\n') + '\n', accepted, rejected };
|
||||
}
|
||||
+38
-3
@@ -1,7 +1,7 @@
|
||||
/** Local voice engine contract (sherpa-onnx in a utility process). Shared by main and renderer. */
|
||||
|
||||
export type VoiceModelKind = 'stt' | 'tts';
|
||||
export type VoiceEngineKind = 'whisper' | 'sense-voice' | 'nemo-transducer' | 'kokoro' | 'piper';
|
||||
export type VoiceModelKind = 'stt' | 'tts' | 'kws' | 'vad';
|
||||
export type VoiceEngineKind = 'whisper' | 'sense-voice' | 'nemo-transducer' | 'kokoro' | 'piper' | 'kws-transducer' | 'silero';
|
||||
|
||||
export interface VoiceSpeaker {
|
||||
id: number;
|
||||
@@ -69,6 +69,33 @@ export interface SynthesizeResult {
|
||||
audioSec: number;
|
||||
}
|
||||
|
||||
export interface KwsStartRequest {
|
||||
modelId: string;
|
||||
/** Wake phrases in plain text (e.g. "jarvis", "hey jarvis"). */
|
||||
keywords: string[];
|
||||
/** 1 (strict) .. 5 (eager) */
|
||||
sensitivity: number;
|
||||
}
|
||||
|
||||
export interface KwsDetection {
|
||||
keyword: string;
|
||||
at: number;
|
||||
}
|
||||
|
||||
export interface VadStartRequest {
|
||||
modelId: string;
|
||||
/** Silence that ends an utterance, in ms. */
|
||||
silenceMs: number;
|
||||
/** Detection threshold 0..1 (0.5 default). */
|
||||
threshold: number;
|
||||
maxUtteranceSec: number;
|
||||
}
|
||||
|
||||
export type VadEvent =
|
||||
| { type: 'speech-start' }
|
||||
| { type: 'segment'; wav: Uint8Array; durationSec: number }
|
||||
| { type: 'error'; message: string };
|
||||
|
||||
export interface VoiceEngineStatus {
|
||||
available: boolean;
|
||||
error?: string;
|
||||
@@ -86,5 +113,13 @@ export const VOICE_IPC = {
|
||||
modelsProgress: 'voice:models:progress',
|
||||
transcribe: 'voice:transcribe',
|
||||
synthesize: 'voice:synthesize',
|
||||
unload: 'voice:unload'
|
||||
unload: 'voice:unload',
|
||||
kwsStart: 'voice:kws:start',
|
||||
kwsStop: 'voice:kws:stop',
|
||||
kwsAudio: 'voice:kws:audio',
|
||||
kwsDetected: 'voice:kws:detected',
|
||||
vadStart: 'voice:vad:start',
|
||||
vadStop: 'voice:vad:stop',
|
||||
vadAudio: 'voice:vad:audio',
|
||||
vadEvent: 'voice:vad:event'
|
||||
} as const;
|
||||
+12
@@ -44,6 +44,18 @@ function useBoot(): boolean {
|
||||
if (api) {
|
||||
disposers.push(useVoiceModels.getState().subscribe());
|
||||
void useVoiceModels.getState().refresh();
|
||||
if (settings.voice.wakeMode === 'kws') void voiceController.startWakeMode();
|
||||
let lastWake = JSON.stringify([settings.voice.wakeMode, settings.voice.wakeWord, settings.voice.kwsSensitivity, settings.voice.micDeviceId]);
|
||||
disposers.push(
|
||||
useSettings.subscribe((s) => {
|
||||
const v = s.settings.voice;
|
||||
const key = JSON.stringify([v.wakeMode, v.wakeWord, v.kwsSensitivity, v.micDeviceId]);
|
||||
if (key === lastWake) return;
|
||||
lastWake = key;
|
||||
if (v.wakeMode === 'kws') void voiceController.restartWakeMode();
|
||||
else void voiceController.stopWakeMode();
|
||||
})
|
||||
);
|
||||
disposers.push(api.hermes.onPush(handlePush));
|
||||
disposers.push(
|
||||
api.hotkeys.on((event) => {
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
import { useEffect, useRef, useState } from 'react';
|
||||
import { Mic, Paperclip, Send, Square, X, Radio, Loader2, Volume2, VolumeX, Navigation } from 'lucide-react';
|
||||
import { sendMessage, steer, stopGeneration } from '../../services/conversation';
|
||||
import { Mic, Paperclip, Send, Square, X, Radio, Loader2, Volume2, VolumeX, Navigation, ScanEye } from 'lucide-react';
|
||||
import { sendMessage, steer, stopGeneration , captureScreen } from '../../services/conversation';
|
||||
import { speech } from '../../services/voice/speech';
|
||||
import { voiceController } from '../../services/voice/voiceController';
|
||||
import { useChat } from '../../state/chat';
|
||||
@@ -167,9 +167,20 @@ export function CommandBar({ compact }: Props) {
|
||||
spellCheck={false}
|
||||
/>
|
||||
{!compact && (
|
||||
<button className="icon-btn" title="Joindre une image" onClick={() => fileRef.current?.click()} style={{ height: 46, width: 40 }}>
|
||||
<Paperclip size={18} />
|
||||
</button>
|
||||
<>
|
||||
<button
|
||||
className="icon-btn"
|
||||
title="Joindre une capture de l’écran (Hermes la voit)"
|
||||
aria-label="Capturer l’écran"
|
||||
onClick={() => void captureScreen().then((shot) => { if (shot) setImages((prev) => [...prev, shot].slice(-4)); })}
|
||||
style={{ height: 46, width: 40 }}
|
||||
>
|
||||
<ScanEye size={18} />
|
||||
</button>
|
||||
<button className="icon-btn" title="Joindre une image" onClick={() => fileRef.current?.click()} style={{ height: 46, width: 40 }}>
|
||||
<Paperclip size={18} />
|
||||
</button>
|
||||
</>
|
||||
)}
|
||||
<input ref={fileRef} type="file" accept="image/*" multiple hidden onChange={(e) => void onFiles(e.target.files)} />
|
||||
{isSending && !steerMode ? (
|
||||
|
||||
@@ -35,6 +35,8 @@ export function CoreStage({ compact }: Props) {
|
||||
const transcript = useVoice((s) => s.lastTranscript);
|
||||
const interim = useVoice((s) => s.interim);
|
||||
const voiceError = useVoice((s) => s.error);
|
||||
const wake = useVoice((s) => s.wake);
|
||||
const wakeKeywords = useVoice((s) => s.wakeKeywords);
|
||||
const assistantName = useSettings((s) => s.settings.assistantName);
|
||||
const reduceMotion = useSettings((s) => s.settings.ui.reduceMotion);
|
||||
const transport = useHermes((s) => s.transport);
|
||||
@@ -57,6 +59,7 @@ export function CoreStage({ compact }: Props) {
|
||||
else if (hud === 'error') sub = previewText(error ?? voiceError ?? 'erreur', 90);
|
||||
else if (hud === 'speaking') sub = 'synthèse vocale';
|
||||
else if (transcript) sub = `« ${previewText(transcript, 80)} »`;
|
||||
else if (wake === 'spotting') sub = `dites « ${wakeKeywords[0] ?? 'jarvis'} »${link === 'online' ? ` · ${transport}` : link === 'offline' ? ' · Hermes hors ligne' : ''}`;
|
||||
else sub = link === 'online' ? `liaison Hermes · ${transport}` : link === 'offline' ? 'Hermes hors ligne' : '';
|
||||
|
||||
return (
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
import { useEffect } from 'react';
|
||||
import { Download, Trash2, X, CheckCircle2, Cpu, Mic, Volume2, AlertTriangle, RefreshCw } from 'lucide-react';
|
||||
import { Download, Trash2, X, CheckCircle2, Cpu, Mic, Volume2, AlertTriangle, RefreshCw, Ear, Activity } from 'lucide-react';
|
||||
import type { VoiceModelStatus } from '../../../shared/voice';
|
||||
import { bridge } from '../../lib/bridge';
|
||||
import { useShallow } from 'zustand/react/shallow';
|
||||
@@ -19,20 +19,25 @@ function ModelRow({ model }: { model: VoiceModelStatus }) {
|
||||
const isActive = model.kind === 'stt' ? activeStt === model.id : activeTts === model.id;
|
||||
const busy = !!progress && (progress.phase === 'download' || progress.phase === 'extract');
|
||||
|
||||
const wakeMode = useSettings((s) => s.settings.voice.wakeMode);
|
||||
const activate = () => {
|
||||
if (model.kind === 'stt') update({ voice: { localModel: model.id, provider: 'local' } });
|
||||
else if (model.kind === 'kws') update({ voice: { wakeMode: 'kws' } });
|
||||
else if (model.kind === 'vad') update({ voice: { neuralVad: true } });
|
||||
else update({ speech: { localModel: model.id, provider: 'local', localSpeaker: model.speakers?.[0]?.id ?? 0 } });
|
||||
};
|
||||
const neuralVad = useSettings((s) => s.settings.voice.neuralVad);
|
||||
const activeNow = model.kind === 'kws' ? wakeMode === 'kws' : model.kind === 'vad' ? neuralVad : isActive;
|
||||
|
||||
return (
|
||||
<div className="row-item" style={isActive && model.installed ? { borderColor: 'var(--accent)' } : undefined}>
|
||||
<div className="row-item" style={activeNow && model.installed ? { borderColor: 'var(--accent)' } : undefined}>
|
||||
<div className="main">
|
||||
<div className="row wrap" style={{ gap: 6 }}>
|
||||
<span className="name">{model.name}</span>
|
||||
{model.recommended && <span className="status-pill ok">recommandé</span>}
|
||||
{model.installed && <span className="status-pill online">installé · {formatMb(model.installedBytes)}</span>}
|
||||
{!model.installed && !busy && <span className="status-pill">{model.sizeMb} Mo</span>}
|
||||
{isActive && model.installed && <span className="status-pill active">actif</span>}
|
||||
{activeNow && model.installed && <span className="status-pill active">actif</span>}
|
||||
</div>
|
||||
<span className="desc">{model.description}</span>
|
||||
<span className="meta">langues : {model.languages.join(', ')}{model.speakers ? ` · ${model.speakers.length} voix` : ''}</span>
|
||||
@@ -49,7 +54,7 @@ function ModelRow({ model }: { model: VoiceModelStatus }) {
|
||||
<button className="icon-btn danger" title="Annuler" onClick={() => void cancel(model.id)}><X size={13} /></button>
|
||||
) : model.installed ? (
|
||||
<>
|
||||
{!isActive && <button className="btn small" onClick={activate}>Utiliser</button>}
|
||||
{!activeNow && <button className="btn small" onClick={activate}>Utiliser</button>}
|
||||
<button className="icon-btn danger" title="Supprimer du disque" onClick={() => { if (confirm(`Supprimer « ${model.name} » ?`)) void remove(model.id); }}><Trash2 size={13} /></button>
|
||||
</>
|
||||
) : (
|
||||
@@ -72,6 +77,8 @@ export function ModelsSection() {
|
||||
|
||||
const stt = models.filter((m) => m.kind === 'stt');
|
||||
const tts = models.filter((m) => m.kind === 'tts');
|
||||
const kws = models.filter((m) => m.kind === 'kws');
|
||||
const vad = models.filter((m) => m.kind === 'vad');
|
||||
return (
|
||||
<>
|
||||
<div className="card">
|
||||
@@ -91,6 +98,15 @@ export function ModelsSection() {
|
||||
<button className="btn small ghost" onClick={() => { void refresh(); void checkEngine(); }}><RefreshCw size={13} /> Actualiser</button>
|
||||
</div>
|
||||
</div>
|
||||
<div className="card">
|
||||
<div className="section-title"><Ear size={12} /> Mot d’activation</div>
|
||||
<div className="list">{kws.map((m) => <ModelRow key={m.id} model={m} />)}</div>
|
||||
</div>
|
||||
<div className="card">
|
||||
<div className="section-title"><Activity size={12} /> Fin de phrase</div>
|
||||
<div className="list">{vad.map((m) => <ModelRow key={m.id} model={m} />)}</div>
|
||||
<span className="hint" style={{ display: 'block', marginTop: 6 }}>Utilisé automatiquement en écoute permanente dès qu’il est installé ; sinon le VAD énergétique prend le relais.</span>
|
||||
</div>
|
||||
<div className="card">
|
||||
<div className="section-title"><Mic size={12} /> Reconnaissance vocale</div>
|
||||
<div className="list">{stt.map((m) => <ModelRow key={m.id} model={m} />)}</div>
|
||||
|
||||
@@ -33,6 +33,33 @@ function Toggle({ on, onChange, label, hint }: { on: boolean; onChange: (v: bool
|
||||
|
||||
type TestState = { status: 'idle' | 'running' | 'ok' | 'fail'; message: string };
|
||||
|
||||
function WakeStatus() {
|
||||
const wake = useVoice((s) => s.wake);
|
||||
const keywords = useVoice((s) => s.wakeKeywords);
|
||||
const error = useVoice((s) => s.error);
|
||||
const models = useVoiceModels((s) => s.models);
|
||||
const download = useVoiceModels((s) => s.download);
|
||||
const progress = useVoiceModels((s) => s.progress['kws-en']);
|
||||
const installed = models.find((m) => m.id === 'kws-en')?.installed;
|
||||
if (installed === false) {
|
||||
return (
|
||||
<div className="test-result fail" style={{ marginTop: 8 }}>
|
||||
<XCircle size={13} />
|
||||
<span>Détecteur non installé.</span>
|
||||
<button className="btn small primary" style={{ marginLeft: 'auto' }} disabled={!!progress} onClick={() => void download('kws-en')}>
|
||||
{progress ? `${progress.percent}%` : 'Télécharger (4 Mo)'}
|
||||
</button>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
return (
|
||||
<div className={`test-result ${wake === 'spotting' ? 'ok' : wake === 'error' ? 'fail' : ''}`} style={{ marginTop: 8 }}>
|
||||
{wake === 'spotting' ? <CheckCircle2 size={13} /> : wake === 'error' ? <XCircle size={13} /> : <Loader2 size={13} className="spin" />}
|
||||
<span>{wake === 'spotting' ? `à l’écoute de « ${keywords.join(' », « ')} »` : wake === 'error' ? error ?? 'erreur' : 'démarrage…'}</span>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
function TestResult({ t }: { t: TestState }) {
|
||||
if (t.status === 'idle') return null;
|
||||
return (
|
||||
@@ -278,13 +305,43 @@ export function SettingsDrawer({ onClose }: Props) {
|
||||
</div>
|
||||
<Toggle on={settings.voice.handsFree} onChange={(v) => { update({ voice: { handsFree: v } }); useVoice.getState().setHandsFree(v); }} label="Mains libres au démarrage" hint="Le micro se réactive automatiquement après chaque réponse." />
|
||||
<Toggle on={settings.voice.wakeChime} onChange={(v) => update({ voice: { wakeChime: v } })} label="Signal sonore d’écoute" />
|
||||
<Toggle on={settings.voice.wakeWordEnabled} onChange={(v) => update({ voice: { wakeWordEnabled: v } })} label="Mot d’activation en mains libres" hint="Seules les phrases commençant par ce mot sont envoyées à Hermes ; le reste est ignoré. Recommandé avec la reconnaissance locale." />
|
||||
{settings.voice.wakeWordEnabled && (
|
||||
<div className="field" style={{ marginTop: 8 }}>
|
||||
<label>Mot d’activation</label>
|
||||
<input className="input" value={settings.voice.wakeWord} placeholder="jarvis" onChange={(e) => update({ voice: { wakeWord: e.target.value.toLowerCase() } })} />
|
||||
<div className="field" style={{ marginTop: 10 }}>
|
||||
<label>Mot d’activation</label>
|
||||
<select className="select" value={settings.voice.wakeMode} onChange={(e) => update({ voice: { wakeMode: e.target.value as typeof settings.voice.wakeMode, wakeWordEnabled: e.target.value === 'transcript' } })}>
|
||||
<option value="off">Désactivé (bouton, raccourci ou mains libres)</option>
|
||||
<option value="kws">Écoute permanente : le micro reste ouvert et réagit au mot-clé (recommandé)</option>
|
||||
<option value="transcript">Filtre après transcription (mains libres) : les phrases sans le mot sont ignorées</option>
|
||||
</select>
|
||||
<span className="hint">
|
||||
{settings.voice.wakeMode === 'kws'
|
||||
? 'Détection locale par un modèle de 3 Mo, quasi gratuite en CPU. Dire le mot seul ouvre l’écoute ; dire le mot puis la commande envoie directement. Le mot coupe aussi la voix en cours.'
|
||||
: settings.voice.wakeMode === 'transcript'
|
||||
? 'Chaque phrase est transcrite puis filtrée : plus coûteux, à réserver à la reconnaissance locale.'
|
||||
: 'Le micro s’active avec le bouton, Ctrl+Shift+Espace ou la boucle mains libres.'}
|
||||
</span>
|
||||
</div>
|
||||
{settings.voice.wakeMode !== 'off' && (
|
||||
<div className="grid-2">
|
||||
<div className="field">
|
||||
<label>Mot-clé</label>
|
||||
<input className="input" value={settings.voice.wakeWord} placeholder="jarvis" onChange={(e) => update({ voice: { wakeWord: e.target.value.toLowerCase().slice(0, 40) } })} />
|
||||
<span className="hint">Prononciation anglaise conseillée (« jarvis », « hey jarvis », « computer », « friday »…).</span>
|
||||
</div>
|
||||
{settings.voice.wakeMode === 'kws' && (
|
||||
<div className="field">
|
||||
<label>Sensibilité du mot-clé : {settings.voice.kwsSensitivity}/5</label>
|
||||
<input className="range" type="range" min={1} max={5} step={1} value={settings.voice.kwsSensitivity} onChange={(e) => update({ voice: { kwsSensitivity: Number(e.target.value) } })} />
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
)}
|
||||
{settings.voice.wakeMode === 'kws' && (
|
||||
<>
|
||||
<WakeStatus />
|
||||
<Toggle on={settings.voice.neuralVad} onChange={(v) => update({ voice: { neuralVad: v } })} label="Fin de phrase neuronale (Silero)" hint="Coupe l’écoute au bon moment, même avec du bruit de fond. Nécessite le modèle Silero VAD (0,6 Mo) dans Modèles locaux." />
|
||||
</>
|
||||
)}
|
||||
<Toggle on={settings.voice.localCommands} onChange={(v) => update({ voice: { localCommands: v } })} label="Commandes locales instantanées" hint="« Verrouille la session », « monte le son », « ouvre Spotify », « regarde mon écran »… exécutées sur ce PC sans passer par Hermes." />
|
||||
<div className="row" style={{ marginTop: 10 }}>
|
||||
<button className="btn small" onClick={() => void testStt()} disabled={settings.voice.provider === 'browser'}><Mic size={13} /> Tester la reconnaissance</button>
|
||||
</div>
|
||||
|
||||
@@ -12,6 +12,8 @@ import { useSettings } from '../state/settings';
|
||||
import { executeLocalTool, LOCAL_TOOL_DEFINITIONS } from './hermes/localTools';
|
||||
import type { HermesStreamEvent, SendHandle } from './hermes/types';
|
||||
import { speech } from './voice/speech';
|
||||
import { bridge } from '../lib/bridge';
|
||||
import { parseLocalIntent, runLocalIntent } from './localCommands';
|
||||
|
||||
let active: SendHandle | null = null;
|
||||
let activeMessageId: string | null = null;
|
||||
@@ -165,9 +167,49 @@ function addPending(request: Omit<PendingRequest, 'id' | 'createdAt'>): void {
|
||||
speech.say(request.kind === 'approval' ? 'Autorisation requise.' : request.kind === 'clarify' ? request.description : 'Saisie requise.', { interrupt: false });
|
||||
}
|
||||
|
||||
/** Screenshot of the primary display as a data URL (Electron only). */
|
||||
export async function captureScreen(): Promise<string | null> {
|
||||
const api = bridge();
|
||||
if (!api) return null;
|
||||
try {
|
||||
return await api.system.captureScreen(1600);
|
||||
} catch (err) {
|
||||
useChat.getState().setError(`Capture d’écran : ${(err as Error).message}`);
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
/** Short system intents handled on the machine without a Hermes round trip. Returns true when consumed. */
|
||||
async function tryLocalIntent(text: string, source: string): Promise<{ handled: boolean; images?: string[]; text?: string }> {
|
||||
const settings = useSettings.getState().settings;
|
||||
if (!settings.voice.localCommands || !bridge()) return { handled: false };
|
||||
const intent = parseLocalIntent(text);
|
||||
if (!intent) return { handled: false };
|
||||
if (intent.kind === 'screenshot') {
|
||||
const shot = await captureScreen();
|
||||
if (!shot) return { handled: false };
|
||||
return { handled: false, images: [shot], text: intent.question || text };
|
||||
}
|
||||
const chat = useChat.getState();
|
||||
chat.addMessage({ role: 'user', content: text, source, status: 'done' });
|
||||
const result = await runLocalIntent(intent);
|
||||
chat.addMessage({ role: 'assistant', content: result.message, source: 'local', status: result.ok ? 'done' : 'error' });
|
||||
chat.setHud(result.ok ? 'idle' : 'error');
|
||||
if (settings.speech.autoSpeak) speech.say(result.message, { interrupt: true });
|
||||
return { handled: true };
|
||||
}
|
||||
|
||||
export async function sendMessage(text: string, images: string[] = [], source = 'eveflow'): Promise<void> {
|
||||
const trimmed = text.trim();
|
||||
let trimmed = text.trim();
|
||||
if ((!trimmed && images.length === 0) || active) return;
|
||||
if (trimmed && images.length === 0) {
|
||||
const local = await tryLocalIntent(trimmed, source);
|
||||
if (local.handled) return;
|
||||
if (local.images) {
|
||||
images = local.images;
|
||||
trimmed = (local.text ?? trimmed).trim();
|
||||
}
|
||||
}
|
||||
const chat = useChat.getState();
|
||||
const hermes = useHermes.getState();
|
||||
const settings = useSettings.getState().settings;
|
||||
|
||||
@@ -467,6 +467,8 @@ export class HermesClient {
|
||||
throw new HttpError(handle.start.status, detail);
|
||||
}
|
||||
ready = true;
|
||||
// Chunks that raced ahead of the start event were buffered as potential error bodies: replay them.
|
||||
for (const chunk of errorChunks.splice(0)) parser.feed(chunk);
|
||||
const rotated = handle.start.headers['x-hermes-session-id'];
|
||||
if (rotated && rotated !== options.sessionId) onEvent({ kind: 'session', sessionId: rotated });
|
||||
|
||||
|
||||
@@ -0,0 +1,75 @@
|
||||
/**
|
||||
* Instant local commands: short French/English intents handled on the machine without a
|
||||
* round trip to Hermes (lock, open app/url, volume, media, screenshot to Hermes).
|
||||
* Anything unmatched goes to Hermes as usual.
|
||||
*/
|
||||
import type { SystemAction } from '../../shared/bridge';
|
||||
import { bridge } from '../lib/bridge';
|
||||
import { Log } from '../lib/log';
|
||||
|
||||
export interface LocalIntent {
|
||||
kind: 'action' | 'screenshot';
|
||||
action?: SystemAction;
|
||||
/** Spoken confirmation. */
|
||||
reply: string;
|
||||
/** For screenshot: the question to send to Hermes with the image. */
|
||||
question?: string;
|
||||
}
|
||||
|
||||
const norm = (t: string) =>
|
||||
t
|
||||
.normalize('NFD')
|
||||
.replace(/[̀-ͯ]/g, '')
|
||||
.toLowerCase()
|
||||
.replace(/[’']/g, ' ')
|
||||
.replace(/[^a-z0-9 :/._-]+/g, ' ')
|
||||
.replace(/\s+/g, ' ')
|
||||
.trim();
|
||||
|
||||
const LOCK = /^(verrouille|verrouiller|bloque|lock)( (la |ma )?(session|le pc|l ordinateur|the (pc|computer|screen)))?$/;
|
||||
const VOL_UP = /^(monte|augmente|hausse|plus fort|volume plus|turn up|raise)( (le|the)? ?(son|volume))?( de \d+)?$/;
|
||||
const VOL_DOWN = /^(baisse|diminue|moins fort|volume moins|turn down|lower)( (le|the)? ?(son|volume))?( de \d+)?$/;
|
||||
const MUTE = /^(coupe|couper|mute|silence|desactive)( (le|the)? ?(son|volume|audio))?$/;
|
||||
const PLAY = /^(pause|play|lecture|reprends|reprendre|mets en pause|met en pause|stop la musique|arrete la musique)$/;
|
||||
const NEXT = /^(suivant|suivante|piste suivante|musique suivante|next|skip)$/;
|
||||
const PREV = /^(precedent|precedente|piste precedente|previous)$/;
|
||||
const OPEN_APP = /^(ouvre|ouvrir|lance|lancer|demarre|open|launch|start) ((l application|l appli|le logiciel|le programme|the app|le|la|les|l|un|une|the|moi) )*(.+)$/;
|
||||
const OPEN_URL = /^(ouvre|ouvrir|va sur|open|go to) (https?:\/\/\S+|(www\.)?[a-z0-9-]+\.[a-z]{2,}(\/\S*)?)$/;
|
||||
const SCREEN = /(regarde|regardes|analyse|decris|decrit|lis|explique|qu est ce qu il y a sur|que vois tu sur|what is on|look at|read) (mon |l |the )?(ecran|screen)|capture (d )?ecran|screenshot/;
|
||||
|
||||
export function parseLocalIntent(text: string): LocalIntent | null {
|
||||
const t = norm(text).replace(/^(jarvis|hey jarvis|ok jarvis)[ ,]*/, '');
|
||||
if (!t) return null;
|
||||
if (LOCK.test(t)) return { kind: 'action', action: { type: 'lock' }, reply: 'Session verrouillée.' };
|
||||
if (MUTE.test(t)) return { kind: 'action', action: { type: 'media', key: 'mute' }, reply: 'Son coupé.' };
|
||||
if (VOL_UP.test(t)) return { kind: 'action', action: { type: 'media', key: 'volume-up' }, reply: 'Volume augmenté.' };
|
||||
if (VOL_DOWN.test(t)) return { kind: 'action', action: { type: 'media', key: 'volume-down' }, reply: 'Volume baissé.' };
|
||||
if (PLAY.test(t)) return { kind: 'action', action: { type: 'media', key: 'play-pause' }, reply: 'Lecture.' };
|
||||
if (NEXT.test(t)) return { kind: 'action', action: { type: 'media', key: 'next' }, reply: 'Piste suivante.' };
|
||||
if (PREV.test(t)) return { kind: 'action', action: { type: 'media', key: 'previous' }, reply: 'Piste précédente.' };
|
||||
if (SCREEN.test(t)) return { kind: 'screenshot', reply: 'Je regarde votre écran.', question: text.trim() };
|
||||
const url = OPEN_URL.exec(t);
|
||||
if (url) {
|
||||
const target = url[2].startsWith('http') ? url[2] : `https://${url[2]}`;
|
||||
return { kind: 'action', action: { type: 'open-url', url: target }, reply: `J’ouvre ${url[2]}.` };
|
||||
}
|
||||
const app = OPEN_APP.exec(t);
|
||||
if (app) {
|
||||
const name = app[app.length - 1].trim();
|
||||
if (name.length <= 40 && !/\s(et|puis|and)\s/.test(name)) return { kind: 'action', action: { type: 'open-app', name }, reply: `J’ouvre ${name}.` };
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
/** Execute an action intent through the bridge. Returns the spoken outcome. */
|
||||
export async function runLocalIntent(intent: LocalIntent): Promise<{ ok: boolean; message: string }> {
|
||||
const api = bridge();
|
||||
if (!api || !intent.action) return { ok: false, message: 'Actions locales indisponibles hors Electron.' };
|
||||
try {
|
||||
const result = await api.system.action(intent.action);
|
||||
Log.info('local', `${intent.action.type}: ${result.ok ? 'ok' : result.message}`);
|
||||
return { ok: result.ok, message: result.ok ? intent.reply : result.message || 'Échec de la commande locale.' };
|
||||
} catch (err) {
|
||||
return { ok: false, message: (err as Error).message };
|
||||
}
|
||||
}
|
||||
@@ -10,7 +10,9 @@ import { audioBus } from './audioBus';
|
||||
import { listMicrophones, MicCapture } from './capture';
|
||||
import { speech } from './speech';
|
||||
import { BrowserRecognizer, transcribeWav } from './stt';
|
||||
import { WakeListener } from './wakeListener';
|
||||
import type { WavResult } from './wav';
|
||||
import { bridge } from '../../lib/bridge';
|
||||
|
||||
const SENSITIVITY_RATIO: Record<number, number> = { 1: 4.5, 2: 3.4, 3: 2.6, 4: 2.0, 5: 1.6 };
|
||||
const SENSITIVITY_MIN_RMS: Record<number, number> = { 1: 0.03, 2: 0.02, 3: 0.012, 4: 0.008, 5: 0.005 };
|
||||
@@ -46,6 +48,8 @@ export function matchWakeWord(text: string, wakeWord: string): { matched: boolea
|
||||
|
||||
class VoiceController {
|
||||
private capture = new MicCapture();
|
||||
private wake = new WakeListener();
|
||||
private unsubscribeKws: (() => void) | null = null;
|
||||
/** After a bare wake word, the next utterance is accepted without the wake word. */
|
||||
private attentionUntil = 0;
|
||||
private browser = new BrowserRecognizer();
|
||||
@@ -74,17 +78,122 @@ class VoiceController {
|
||||
dispose(): void {
|
||||
this.unsubscribeVoice?.();
|
||||
this.stop();
|
||||
void this.stopWakeMode();
|
||||
}
|
||||
|
||||
get isListening(): boolean {
|
||||
return useVoice.getState().phase !== 'off';
|
||||
}
|
||||
|
||||
get wakeActive(): boolean {
|
||||
return this.wake.isActive;
|
||||
}
|
||||
|
||||
toggle(): void {
|
||||
if (this.wake.isActive) {
|
||||
// Always-on mode: the button starts/cancels a command capture on the shared microphone.
|
||||
if (this.wake.currentPhase === 'command' || this.wake.currentPhase === 'speech') this.wake.cancelCommand();
|
||||
else this.onWakeDetected('manuel');
|
||||
return;
|
||||
}
|
||||
if (this.isListening) this.stop();
|
||||
else void this.start();
|
||||
}
|
||||
|
||||
/** Always-on keyword spotting: the microphone stays open and the main process spots the wake word. */
|
||||
async startWakeMode(): Promise<void> {
|
||||
const api = bridge();
|
||||
const voice = useVoice.getState();
|
||||
if (!api || this.wake.isActive) return;
|
||||
const settings = useSettings.getState().settings.voice;
|
||||
voice.setWake('starting');
|
||||
try {
|
||||
const phrases = [settings.wakeWord || 'jarvis', `hey ${settings.wakeWord || 'jarvis'}`];
|
||||
const result = await api.voice.kwsStart({ modelId: 'kws-en', keywords: phrases, sensitivity: settings.kwsSensitivity });
|
||||
this.unsubscribeKws?.();
|
||||
this.unsubscribeKws = api.voice.onKwsDetected((d) => this.onWakeDetected(d.keyword));
|
||||
const sensitivity = Math.min(5, Math.max(1, Math.round(settings.sensitivity))) as 1 | 2 | 3 | 4 | 5;
|
||||
let neuralVad = false;
|
||||
if (settings.neuralVad) {
|
||||
try {
|
||||
const installed = (await api.voice.listModels()).some((m) => m.id === 'silero-vad' && m.installed);
|
||||
if (installed) {
|
||||
await api.voice.vadStart({ modelId: 'silero-vad', silenceMs: Math.max(300, Math.min(1500, settings.silenceMs - 200)), threshold: 0.5, maxUtteranceSec: 25 });
|
||||
neuralVad = true;
|
||||
}
|
||||
} catch (err) {
|
||||
Log.warn('voice', `silero unavailable, energy VAD fallback: ${(err as Error).message}`);
|
||||
}
|
||||
}
|
||||
await this.wake.start({
|
||||
deviceId: settings.micDeviceId || undefined,
|
||||
neuralVad,
|
||||
vad: { silenceMs: settings.silenceMs, speechRatio: SENSITIVITY_RATIO[sensitivity], minRms: SENSITIVITY_MIN_RMS[sensitivity] },
|
||||
callbacks: {
|
||||
onPhase: (phase) => {
|
||||
const v = useVoice.getState();
|
||||
if (phase === 'spotting') {
|
||||
v.setPhase('off');
|
||||
const chat = useChat.getState();
|
||||
if (chat.hud === 'listening') chat.setHud(chat.isSending ? 'thinking' : 'idle');
|
||||
} else if (phase === 'command') {
|
||||
v.setPhase('listening');
|
||||
useChat.getState().setHud('listening');
|
||||
} else if (phase === 'speech') {
|
||||
v.setPhase('speech');
|
||||
useChat.getState().ping();
|
||||
}
|
||||
},
|
||||
onLevel: (level) => {
|
||||
const q = Math.round(level * 20) / 20;
|
||||
if (q !== useVoice.getState().inputLevel) useVoice.getState().setInputLevel(q);
|
||||
},
|
||||
onUtterance: (wav) => void this.transcribe(wav, true),
|
||||
onNoSpeech: () => useVoice.getState().setInputLevel(0),
|
||||
onError: (message) => useVoice.getState().setError(message)
|
||||
}
|
||||
});
|
||||
voice.setWake('spotting', result.accepted.map((k) => k.replace(/_/g, ' ')));
|
||||
voice.setNeuralVad(neuralVad);
|
||||
Log.info('voice', `wake mode on: ${result.accepted.join(', ')}${result.rejected.length ? ` (rejetés : ${result.rejected.join(', ')})` : ''}${neuralVad ? ' · Silero' : ''}`);
|
||||
} catch (err) {
|
||||
const message = (err as Error).message;
|
||||
voice.setWake('error');
|
||||
voice.setError(message);
|
||||
useChat.getState().setError(`Mot d’activation : ${message}`);
|
||||
Log.error('voice', `wake mode failed: ${message}`);
|
||||
await api.voice.kwsStop().catch(() => undefined);
|
||||
await api.voice.vadStop().catch(() => undefined);
|
||||
}
|
||||
}
|
||||
|
||||
async stopWakeMode(): Promise<void> {
|
||||
this.unsubscribeKws?.();
|
||||
this.unsubscribeKws = null;
|
||||
this.wake.stop();
|
||||
useVoice.getState().setWake('off');
|
||||
useVoice.getState().setNeuralVad(false);
|
||||
const api = bridge();
|
||||
await api?.voice.kwsStop().catch(() => undefined);
|
||||
await api?.voice.vadStop().catch(() => undefined);
|
||||
}
|
||||
|
||||
/** Restart spotting with the current settings (wake word, sensitivity, microphone). */
|
||||
async restartWakeMode(): Promise<void> {
|
||||
await this.stopWakeMode();
|
||||
if (useSettings.getState().settings.voice.wakeMode === 'kws') await this.startWakeMode();
|
||||
}
|
||||
|
||||
private onWakeDetected(keyword: string): void {
|
||||
if (!this.wake.isActive || this.wake.currentPhase !== 'spotting') return;
|
||||
if (useChat.getState().isSending) return;
|
||||
Log.info('voice', `wake: ${keyword}`);
|
||||
speech.stop();
|
||||
useChat.getState().ping();
|
||||
this.playChime(true);
|
||||
this.wake.beginCommand();
|
||||
}
|
||||
|
||||
setHandsFree(on: boolean): void {
|
||||
useVoice.getState().setHandsFree(on);
|
||||
useSettings.getState().update({ voice: { handsFree: on } });
|
||||
@@ -113,6 +222,11 @@ class VoiceController {
|
||||
async start(auto = false): Promise<void> {
|
||||
const voice = useVoice.getState();
|
||||
if (voice.phase !== 'off') return;
|
||||
if (this.wake.isActive) {
|
||||
// The always-on listener owns the microphone: open a command capture on it instead.
|
||||
if (!auto) this.onWakeDetected('manuel');
|
||||
return;
|
||||
}
|
||||
const settings = useSettings.getState().settings.voice;
|
||||
const seq = ++this.startSeq;
|
||||
voice.setError(null);
|
||||
@@ -218,7 +332,7 @@ class VoiceController {
|
||||
}
|
||||
}
|
||||
|
||||
private async transcribe(wav: WavResult): Promise<void> {
|
||||
private async transcribe(wav: WavResult, fromWake = false): Promise<void> {
|
||||
const voice = useVoice.getState();
|
||||
voice.setPhase('transcribing');
|
||||
useChat.getState().setHud('thinking');
|
||||
@@ -229,7 +343,8 @@ class VoiceController {
|
||||
voice.setTranscript(text);
|
||||
voice.setPhase('off');
|
||||
Log.info('voice', `transcript (${wav.durationSec.toFixed(1)}s): ${text}`);
|
||||
if (voice.handsFree && settings.wakeWordEnabled && Date.now() > this.attentionUntil) {
|
||||
const filterByTranscript = settings.wakeMode === 'transcript' || (settings.wakeMode === 'off' && settings.wakeWordEnabled);
|
||||
if (!fromWake && voice.handsFree && filterByTranscript && Date.now() > this.attentionUntil) {
|
||||
const { matched, rest } = matchWakeWord(text, settings.wakeWord || 'jarvis');
|
||||
if (!matched) {
|
||||
Log.debug('voice', 'utterance ignored: no wake word');
|
||||
|
||||
@@ -0,0 +1,273 @@
|
||||
/**
|
||||
* Always-on microphone for the keyword spotter. Streams 16 kHz PCM to the main process
|
||||
* (which feeds the sherpa-onnx spotter); once the wake word is detected it captures the
|
||||
* following utterance with the energy VAD and hands back a WAV, then resumes spotting.
|
||||
* One microphone stream, no re-opening between phrases.
|
||||
*/
|
||||
import { bridge } from '../../lib/bridge';
|
||||
import { Log } from '../../lib/log';
|
||||
import { audioBus } from './audioBus';
|
||||
import { DEFAULT_VAD, EnergyVad, type VadOptions } from './vad';
|
||||
import { buildWav16k, rms, type WavResult } from './wav';
|
||||
import type { VadEvent } from '../../../shared/voice';
|
||||
|
||||
const WORKLET_SOURCE = `
|
||||
class EveFlowWakeProcessor extends AudioWorkletProcessor {
|
||||
constructor() { super(); this.buffer = new Float32Array(2048); this.offset = 0; }
|
||||
process(inputs) {
|
||||
const channel = inputs[0] && inputs[0][0];
|
||||
if (!channel) return true;
|
||||
let i = 0;
|
||||
while (i < channel.length) {
|
||||
const n = Math.min(channel.length - i, this.buffer.length - this.offset);
|
||||
this.buffer.set(channel.subarray(i, i + n), this.offset);
|
||||
this.offset += n; i += n;
|
||||
if (this.offset === this.buffer.length) {
|
||||
this.port.postMessage(this.buffer, [this.buffer.buffer]);
|
||||
this.buffer = new Float32Array(2048); this.offset = 0;
|
||||
}
|
||||
}
|
||||
return true;
|
||||
}
|
||||
}
|
||||
registerProcessor('eveflow-wake', EveFlowWakeProcessor);
|
||||
`;
|
||||
|
||||
export type WakePhase = 'off' | 'spotting' | 'command' | 'speech';
|
||||
|
||||
export interface WakeCallbacks {
|
||||
onPhase: (phase: WakePhase) => void;
|
||||
onLevel?: (level: number) => void;
|
||||
onUtterance: (wav: WavResult) => void;
|
||||
onNoSpeech: () => void;
|
||||
onError: (message: string) => void;
|
||||
}
|
||||
|
||||
let workletUrl: string | null = null;
|
||||
|
||||
export class WakeListener {
|
||||
private ctx: AudioContext | null = null;
|
||||
private stream: MediaStream | null = null;
|
||||
private node: AudioWorkletNode | null = null;
|
||||
private phase: WakePhase = 'off';
|
||||
private vad: EnergyVad | null = null;
|
||||
private chunks: Float32Array[] = [];
|
||||
private totalSamples = 0;
|
||||
private sampleRate = 16_000;
|
||||
private pending: Float32Array[] = [];
|
||||
private pendingSamples = 0;
|
||||
private callbacks: WakeCallbacks | null = null;
|
||||
private vadOptions: Partial<VadOptions> = {};
|
||||
private noSpeechTimer: ReturnType<typeof setTimeout> | null = null;
|
||||
/** Neural end-of-speech (Silero in the main process) instead of the energy VAD. */
|
||||
private neural = false;
|
||||
private unsubscribeVad: (() => void) | null = null;
|
||||
private vadPending: Float32Array[] = [];
|
||||
private vadPendingSamples = 0;
|
||||
|
||||
get isActive(): boolean {
|
||||
return this.phase !== 'off';
|
||||
}
|
||||
|
||||
get currentPhase(): WakePhase {
|
||||
return this.phase;
|
||||
}
|
||||
|
||||
async start(options: { deviceId?: string; vad?: Partial<VadOptions>; neuralVad?: boolean; callbacks: WakeCallbacks }): Promise<void> {
|
||||
if (this.phase !== 'off') return;
|
||||
this.callbacks = options.callbacks;
|
||||
this.vadOptions = options.vad ?? {};
|
||||
this.neural = !!options.neuralVad;
|
||||
if (this.neural) {
|
||||
this.unsubscribeVad?.();
|
||||
this.unsubscribeVad = bridge()?.voice.onVadEvent((event) => this.onVadEvent(event)) ?? null;
|
||||
}
|
||||
this.stream = await navigator.mediaDevices.getUserMedia({
|
||||
audio: {
|
||||
deviceId: options.deviceId ? { exact: options.deviceId } : undefined,
|
||||
echoCancellation: true,
|
||||
noiseSuppression: true,
|
||||
autoGainControl: true,
|
||||
channelCount: 1
|
||||
}
|
||||
});
|
||||
try {
|
||||
this.ctx = new AudioContext({ sampleRate: 16_000, latencyHint: 'playback' });
|
||||
} catch {
|
||||
this.ctx = new AudioContext({ latencyHint: 'playback' });
|
||||
}
|
||||
if (this.ctx.state === 'suspended') await this.ctx.resume().catch(() => undefined);
|
||||
this.sampleRate = this.ctx.sampleRate;
|
||||
const source = this.ctx.createMediaStreamSource(this.stream);
|
||||
const analyser = this.ctx.createAnalyser();
|
||||
analyser.fftSize = 512;
|
||||
source.connect(analyser);
|
||||
audioBus.setInputAnalyser(analyser);
|
||||
if (!workletUrl) workletUrl = URL.createObjectURL(new Blob([WORKLET_SOURCE], { type: 'application/javascript' }));
|
||||
await this.ctx.audioWorklet.addModule(workletUrl);
|
||||
this.node = new AudioWorkletNode(this.ctx, 'eveflow-wake', { numberOfInputs: 1, numberOfOutputs: 0, channelCount: 1 });
|
||||
this.node.port.onmessage = (event: MessageEvent<Float32Array>) => this.onSamples(event.data);
|
||||
source.connect(this.node);
|
||||
this.setPhase('spotting');
|
||||
Log.info('wake', `listener started (${this.sampleRate} Hz, fin de phrase ${this.neural ? 'Silero' : 'énergie'})`);
|
||||
}
|
||||
|
||||
private onVadEvent(event: VadEvent): void {
|
||||
if (!this.neural || (this.phase !== 'command' && this.phase !== 'speech')) return;
|
||||
if (event.type === 'speech-start') {
|
||||
if (this.phase === 'command') {
|
||||
this.setPhase('speech');
|
||||
if (this.noSpeechTimer) clearTimeout(this.noSpeechTimer);
|
||||
this.noSpeechTimer = null;
|
||||
}
|
||||
} else if (event.type === 'segment') {
|
||||
const wav: WavResult = { bytes: event.wav, sampleRate: 16_000, durationSec: event.durationSec };
|
||||
this.resumeSpotting();
|
||||
if (wav.durationSec >= 0.25) this.callbacks?.onUtterance(wav);
|
||||
else this.callbacks?.onNoSpeech();
|
||||
} else if (event.type === 'error') {
|
||||
this.callbacks?.onError(event.message);
|
||||
this.resumeSpotting();
|
||||
}
|
||||
}
|
||||
|
||||
/** Called by the controller when the main process reports the wake word (or on manual trigger). */
|
||||
beginCommand(): void {
|
||||
if (this.phase === 'off' || this.phase === 'command' || this.phase === 'speech') return;
|
||||
this.vad = this.neural ? null : new EnergyVad({ ...DEFAULT_VAD, ...this.vadOptions, noSpeechTimeoutMs: Number.POSITIVE_INFINITY });
|
||||
this.vadPending = [];
|
||||
this.vadPendingSamples = 0;
|
||||
this.chunks = [];
|
||||
this.totalSamples = 0;
|
||||
this.setPhase('command');
|
||||
if (this.noSpeechTimer) clearTimeout(this.noSpeechTimer);
|
||||
this.noSpeechTimer = setTimeout(() => {
|
||||
if (this.phase === 'command') {
|
||||
this.resumeSpotting();
|
||||
this.callbacks?.onNoSpeech();
|
||||
}
|
||||
}, 8000);
|
||||
}
|
||||
|
||||
/** Abort a command capture and go back to spotting. */
|
||||
cancelCommand(): void {
|
||||
if (this.phase === 'command' || this.phase === 'speech') this.resumeSpotting();
|
||||
}
|
||||
|
||||
stop(): void {
|
||||
if (this.phase === 'off') return;
|
||||
if (this.noSpeechTimer) clearTimeout(this.noSpeechTimer);
|
||||
this.noSpeechTimer = null;
|
||||
this.unsubscribeVad?.();
|
||||
this.unsubscribeVad = null;
|
||||
audioBus.setInputAnalyser(null);
|
||||
if (this.node) {
|
||||
this.node.port.onmessage = null;
|
||||
this.node.disconnect();
|
||||
this.node = null;
|
||||
}
|
||||
this.stream?.getTracks().forEach((t) => t.stop());
|
||||
this.stream = null;
|
||||
void this.ctx?.close().catch(() => undefined);
|
||||
this.ctx = null;
|
||||
this.chunks = [];
|
||||
this.pending = [];
|
||||
this.pendingSamples = 0;
|
||||
this.vadPending = [];
|
||||
this.vadPendingSamples = 0;
|
||||
this.setPhase('off');
|
||||
Log.info('wake', 'listener stopped');
|
||||
}
|
||||
|
||||
private setPhase(phase: WakePhase): void {
|
||||
this.phase = phase;
|
||||
this.callbacks?.onPhase(phase);
|
||||
}
|
||||
|
||||
private resumeSpotting(): void {
|
||||
if (this.noSpeechTimer) clearTimeout(this.noSpeechTimer);
|
||||
this.noSpeechTimer = null;
|
||||
this.vad = null;
|
||||
this.chunks = [];
|
||||
this.totalSamples = 0;
|
||||
this.setPhase('spotting');
|
||||
}
|
||||
|
||||
private onSamples(samples: Float32Array): void {
|
||||
if (this.phase === 'off') return;
|
||||
const level = rms(samples);
|
||||
this.callbacks?.onLevel?.(Math.min(1, level * 6));
|
||||
|
||||
if (this.phase === 'spotting') {
|
||||
// Batch ~256 ms of audio per IPC message for the keyword spotter.
|
||||
this.pending.push(samples);
|
||||
this.pendingSamples += samples.length;
|
||||
if (this.pendingSamples >= this.sampleRate * 0.25) this.flushToSpotter();
|
||||
return;
|
||||
}
|
||||
|
||||
if (this.neural) {
|
||||
// command / speech with Silero: stream ~128 ms frames to the main process, which returns the segment.
|
||||
this.vadPending.push(samples);
|
||||
this.vadPendingSamples += samples.length;
|
||||
if (this.vadPendingSamples >= this.sampleRate * 0.128) this.flushToVad();
|
||||
return;
|
||||
}
|
||||
|
||||
// command / speech: collect the utterance
|
||||
this.chunks.push(samples);
|
||||
this.totalSamples += samples.length;
|
||||
if (this.vad && this.phase === 'command') {
|
||||
// keep only a short pre-roll before speech starts
|
||||
while (this.chunks.length > 1 && this.totalSamples - this.chunks[0].length > this.sampleRate * 0.4) {
|
||||
this.totalSamples -= this.chunks.shift()!.length;
|
||||
}
|
||||
}
|
||||
if (!this.vad) return;
|
||||
const signal = this.vad.feed(level, (samples.length / this.sampleRate) * 1000);
|
||||
if (signal === 'speech-start') {
|
||||
this.setPhase('speech');
|
||||
if (this.noSpeechTimer) clearTimeout(this.noSpeechTimer);
|
||||
this.noSpeechTimer = null;
|
||||
} else if (signal === 'speech-end' || signal === 'max-length') {
|
||||
const wav = buildWav16k(this.chunks, this.sampleRate);
|
||||
this.resumeSpotting();
|
||||
if (wav.durationSec >= 0.25) this.callbacks?.onUtterance(wav);
|
||||
else this.callbacks?.onNoSpeech();
|
||||
} else if (signal === 'too-short') {
|
||||
this.resumeSpotting();
|
||||
this.callbacks?.onNoSpeech();
|
||||
}
|
||||
}
|
||||
|
||||
private static toInt16(chunks: Float32Array[], total: number): Uint8Array {
|
||||
const merged = new Int16Array(total);
|
||||
let offset = 0;
|
||||
for (const chunk of chunks) {
|
||||
for (let i = 0; i < chunk.length; i++) {
|
||||
const s = Math.max(-1, Math.min(1, chunk[i]));
|
||||
merged[offset + i] = s < 0 ? s * 0x8000 : s * 0x7fff;
|
||||
}
|
||||
offset += chunk.length;
|
||||
}
|
||||
return new Uint8Array(merged.buffer);
|
||||
}
|
||||
|
||||
private flushToSpotter(): void {
|
||||
const api = bridge();
|
||||
if (!api) return;
|
||||
const bytes = WakeListener.toInt16(this.pending, this.pendingSamples);
|
||||
this.pending = [];
|
||||
this.pendingSamples = 0;
|
||||
api.voice.kwsAudio(bytes, this.sampleRate);
|
||||
}
|
||||
|
||||
private flushToVad(): void {
|
||||
const api = bridge();
|
||||
if (!api) return;
|
||||
const bytes = WakeListener.toInt16(this.vadPending, this.vadPendingSamples);
|
||||
this.vadPending = [];
|
||||
this.vadPendingSamples = 0;
|
||||
api.voice.vadAudio(bytes, this.sampleRate);
|
||||
}
|
||||
}
|
||||
@@ -19,6 +19,13 @@ export interface VoiceSettings extends SttConfig {
|
||||
/** Hands-free: only react to utterances starting with this word (local STT recommended). */
|
||||
wakeWordEnabled: boolean;
|
||||
wakeWord: string;
|
||||
/** off = push-to-talk / hands-free; transcript = filter after transcription; kws = always-on keyword spotting. */
|
||||
wakeMode: 'off' | 'transcript' | 'kws';
|
||||
kwsSensitivity: number; // 1..5
|
||||
/** Use Silero VAD for end-of-speech in always-on mode when the model is installed. */
|
||||
neuralVad: boolean;
|
||||
/** Execute short system intents locally (lock, volume, open app…) instead of asking Hermes. */
|
||||
localCommands: boolean;
|
||||
}
|
||||
|
||||
export interface SpeechSettings extends TtsConfig {
|
||||
@@ -83,6 +90,10 @@ export const DEFAULT_SETTINGS: Settings = {
|
||||
wakeChime: true,
|
||||
wakeWordEnabled: false,
|
||||
wakeWord: 'jarvis',
|
||||
wakeMode: 'off',
|
||||
kwsSensitivity: 3,
|
||||
neuralVad: true,
|
||||
localCommands: true,
|
||||
localModel: 'whisper-base'
|
||||
},
|
||||
speech: {
|
||||
@@ -181,6 +192,7 @@ export const useSettings = create<SettingsStore>((set, get) => ({
|
||||
}
|
||||
const merged = merge(DEFAULT_SETTINGS, saved);
|
||||
if (!merged.hermesSessionId) merged.hermesSessionId = uid('eveflow');
|
||||
if (saved && saved.voice && saved.voice.wakeMode === undefined && saved.voice.wakeWordEnabled) merged.voice.wakeMode = 'transcript';
|
||||
set({ settings: merged, loaded: true });
|
||||
persistSet(STORAGE_KEY, merged);
|
||||
},
|
||||
|
||||
@@ -2,6 +2,7 @@ import { create } from 'zustand';
|
||||
import type { TtsState } from '../services/voice/tts';
|
||||
|
||||
export type ListenPhase = 'off' | 'arming' | 'listening' | 'speech' | 'transcribing';
|
||||
export type WakeState = 'off' | 'starting' | 'spotting' | 'error';
|
||||
|
||||
interface VoiceStore {
|
||||
phase: ListenPhase;
|
||||
@@ -12,6 +13,11 @@ interface VoiceStore {
|
||||
interim: string;
|
||||
error: string | null;
|
||||
micDevices: Array<{ deviceId: string; label: string }>;
|
||||
wake: WakeState;
|
||||
wakeKeywords: string[];
|
||||
neuralVad: boolean;
|
||||
setNeuralVad: (on: boolean) => void;
|
||||
setWake: (state: WakeState, keywords?: string[]) => void;
|
||||
setPhase: (phase: ListenPhase) => void;
|
||||
setInputLevel: (level: number) => void;
|
||||
setTts: (state: TtsState) => void;
|
||||
@@ -31,6 +37,11 @@ export const useVoice = create<VoiceStore>((set) => ({
|
||||
interim: '',
|
||||
error: null,
|
||||
micDevices: [],
|
||||
wake: 'off',
|
||||
wakeKeywords: [],
|
||||
neuralVad: false,
|
||||
setNeuralVad: (neuralVad) => set({ neuralVad }),
|
||||
setWake: (wake, wakeKeywords) => set(wakeKeywords ? { wake, wakeKeywords } : { wake }),
|
||||
setPhase: (phase) => set({ phase }),
|
||||
setInputLevel: (inputLevel) => set({ inputLevel }),
|
||||
setTts: (tts) => set({ tts }),
|
||||
|
||||
@@ -0,0 +1,25 @@
|
||||
import { describe, expect, it } from 'vitest';
|
||||
import { buildKeywordsFile, encodeKeyword, normalizeKeyword, parseTokens } from '../shared/keywords';
|
||||
|
||||
const vocab = parseTokens(['<blk> 0', '▁ 3', '▁JA 4', 'R 5', 'VI 6', 'S 7', '▁HE 8', 'Y 9', '▁NO 10', 'V 11', 'A 12', '▁MA 13', 'X 14', '▁T 15', 'ON 16', 'Y 17'].join('\n'));
|
||||
|
||||
describe('keywords', () => {
|
||||
it('normalises accents, case and punctuation', () => {
|
||||
expect(normalizeKeyword(' Hé, Jarvis ! ')).toBe('HE JARVIS');
|
||||
});
|
||||
it('uses the known SentencePiece encodings', () => {
|
||||
expect(encodeKeyword('jarvis', vocab)).toBe('▁JA R VI S');
|
||||
expect(encodeKeyword('Hey Jarvis', vocab)).toBe('▁HE Y ▁JA R VI S');
|
||||
});
|
||||
it('falls back to greedy longest match', () => {
|
||||
expect(encodeKeyword('max', vocab)).toBe('▁MA X');
|
||||
expect(encodeKeyword('tony', vocab)).toBe('▁T ON Y');
|
||||
expect(encodeKeyword('zzz', vocab)).toBeNull();
|
||||
});
|
||||
it('builds a keywords file with labels', () => {
|
||||
const file = buildKeywordsFile(['jarvis', 'hey jarvis', 'zzz'], vocab);
|
||||
expect(file.content).toBe('▁JA R VI S @jarvis\n▁HE Y ▁JA R VI S @hey_jarvis\n');
|
||||
expect(file.accepted).toEqual(['jarvis', 'hey_jarvis']);
|
||||
expect(file.rejected).toEqual(['zzz']);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,24 @@
|
||||
import { describe, expect, it } from 'vitest';
|
||||
import { parseLocalIntent } from '../src/services/localCommands';
|
||||
|
||||
describe('parseLocalIntent', () => {
|
||||
it('recognises system actions in French and English', () => {
|
||||
expect(parseLocalIntent('Jarvis, verrouille la session')?.action).toEqual({ type: 'lock' });
|
||||
expect(parseLocalIntent('monte le son')?.action).toEqual({ type: 'media', key: 'volume-up' });
|
||||
expect(parseLocalIntent('Baisse le volume')?.action).toEqual({ type: 'media', key: 'volume-down' });
|
||||
expect(parseLocalIntent('coupe le son')?.action).toEqual({ type: 'media', key: 'mute' });
|
||||
expect(parseLocalIntent('piste suivante')?.action).toEqual({ type: 'media', key: 'next' });
|
||||
expect(parseLocalIntent('ouvre le bloc-notes')?.action).toEqual({ type: 'open-app', name: 'bloc-notes' });
|
||||
expect(parseLocalIntent('open spotify')?.action).toEqual({ type: 'open-app', name: 'spotify' });
|
||||
expect(parseLocalIntent('ouvre github.com')?.action).toEqual({ type: 'open-url', url: 'https://github.com' });
|
||||
});
|
||||
it('detects screen questions', () => {
|
||||
expect(parseLocalIntent('Jarvis, regarde mon écran et dis-moi ce que tu vois')?.kind).toBe('screenshot');
|
||||
expect(parseLocalIntent('fais une capture d’écran')?.kind).toBe('screenshot');
|
||||
});
|
||||
it('leaves everything else to Hermes', () => {
|
||||
expect(parseLocalIntent('Quelle est la météo à Paris demain ?')).toBeNull();
|
||||
expect(parseLocalIntent('ouvre le fichier puis envoie-le à Marc')).toBeNull();
|
||||
expect(parseLocalIntent('')).toBeNull();
|
||||
});
|
||||
});
|
||||
Reference in new issue
Block a user