mirror of
https://github.com/R0m1k3/Loki.git
synced 2026-10-16 19:59:56 +02:00
Lot 27 : banc d'essai tenu par la façade (LOKI_BENCH=rust)
Avec LOKI_BENCH=rust (expérimental), la façade sert /api/bench, /api/bench/status, /api/bench/cancel et /api/bench/last, et déroule le protocole de mesure contre le moteur local (rust/loki-rs/src/bench) : mode rapide (préchauffage, ligne courte) et mode complet (prefill à froid à profondeur réelle, trois tours qui reprennent le cache). Corpus, requêtes, calculs et messages sont ceux du Go ; seule une mesure complète est enregistrée (last_bench, bench_presets, empreinte de configuration du Go). Le verrou de génération reste celui de la conversation du Go : la façade le prend par le bail du moteur, avec le motif « bench ». Le Go note alors benching et donne aux autres le refus du benchmark ; changer de discussion ne rompt pas le bail ; benchRunning() le voit (clients /v1, préchauffage, optimiseur). Un stop du chat, un Reset ou un bail perdu annulent le bench. Le Go prête aussi le travail annexe (effacement du slot à la fin, /internal/v1/engine/sidejob) et la télémétrie (/internal/v1/bench/perf). Correction des tests du Go : trois tests du préchauffage échouaient dix minutes par jour, avant minuit (prewarmNearMidnight lisait l'heure réelle). L'heure du garde-fou devient remplaçable (prewarmNow), fixée à midi dans les tests. Parité : parity_bench_test.go tire du Go un journal rejoué par tests/bench.rs contre le même faux moteur ; chaque requête reçue par le moteur est comparée (longs textes réduits à leur taille et à leur empreinte). Il couvre les refus, les deux modes, les échecs de chaque phase, le raisonnement refusé puis rejoué, la profondeur partielle et l'annulation. Les tests du Go sont portés (bench/tests.rs). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016tcGfJWefW5kUGoC2wL8cv
This commit is contained in:
27 files changed
+7580
-12
No files matched your search
@@ -241,6 +241,7 @@ redémarrages (volume `/data`). Variables d'environnement du conteneur :
|
||||
| `LOKI_BACKENDS` | `rust` : les backends llama.cpp — statut, vérification git, binaires précompilés, bascule fast/opt, backends custom, GPU vus par un moteur — et le job des moteurs sont tenus par la façade Rust ; les jobs du Go (compilation, moteur de l'image, Strata) lui rapportent leurs étapes (expérimental, migration en cours) | Go |
|
||||
| `LOKI_STRATA` | `rust` (avec `LOKI_BACKENDS=rust`) : l'écran de Strata — machine, ce qui y tient, recommandation, modèles installés —, ses réglages et son installation sont tenus par la façade Rust ; le lancement reste au Go (expérimental, migration en cours) | Go |
|
||||
| `LOKI_TASKS` | `rust` : les tâches planifiées — liste, enregistrement, bascule, interrupteur maître, lancement, arrêt —, le planificateur et les tâches « script seul » sont tenus par la façade Rust ; une tâche IA reste exécutée par le Go, à la demande de la façade (expérimental, migration en cours) | Go |
|
||||
| `LOKI_BENCH` | `rust` : le banc d'essai — lancement, progression, annulation, dernière mesure, mode rapide et mode complet en profondeur — est tenu par la façade Rust ; le verrou de génération reste celui du Go, pris par le bail du moteur (expérimental, migration en cours) | Go |
|
||||
| `LOKI_ENGINE_IMAGE` | `rust` (avec `LOKI_BACKENDS=rust`) : le panneau « moteur de l'image » — version qui tourne, dernier build publié, version recommandée, mise à jour depuis le registre officiel avec essai à blanc, bascules, retour à la version précédente, vérification du rendu du gabarit — est tenu par la façade Rust (expérimental, migration en cours) | Go |
|
||||
| `LOKI_PROJECTS` | `rust` : le hub des projets (liste et comptes, création, renommage, description, suppression, bascule) est tenu par la façade Rust ; les discussions restent à qui tient la conversation (expérimental, migration en cours) | Go |
|
||||
| `LOKI_SYSTEM` | `rust` : les interrupteurs système — accès internet et Crawl4AI, pilotage du navigateur, clé de complétion, écoute réseau du moteur — sont tenus par la façade Rust ; le navigateur piloté reste au Go (expérimental, migration en cours) | Go |
|
||||
|
||||
@@ -0,0 +1,125 @@
|
||||
# Lot 27 — Banc d'essai tenu par la façade
|
||||
|
||||
## Contexte
|
||||
|
||||
Le banc d'essai mesure le moteur local tel que le chat s'en sert, sur
|
||||
`/v1/chat/completions`, avec le gabarit, le raisonnement et
|
||||
l'échantillonnage du preset (`llm_bench.go`, `llm_bench_job.go`). Il a deux
|
||||
modes :
|
||||
|
||||
- **rapide** : un préchauffage jeté, puis la ligne courte (~2000 jetons de
|
||||
prompt, ~300 générés) ;
|
||||
- **complet** : la ligne courte, puis une discussion à profondeur réelle —
|
||||
un prefill à froid de D jetons, et trois tours qui y ajoutent chacun ~2048
|
||||
jetons neufs. La reprise du cache est lue dans la réponse, jamais
|
||||
supposée.
|
||||
|
||||
Il tourne en tâche de fond derrière quatre routes :
|
||||
|
||||
- `POST /api/bench` lance (202 et un id) ;
|
||||
- `GET /api/bench/status` dit la phase, puis le résultat ou l'erreur ;
|
||||
- `POST /api/bench/cancel` annule ;
|
||||
- `GET /api/bench/last` rend la dernière mesure enregistrée.
|
||||
|
||||
Pendant la mesure, le bench tient le verrou de génération de la
|
||||
conversation. Un message, une tâche planifiée ou un client `/v1` reçoivent
|
||||
le refus du bench. Seule une mesure complète est enregistrée : la
|
||||
dernière, et celle du preset actif.
|
||||
|
||||
## Le lot
|
||||
|
||||
Avec `LOKI_BENCH=rust` (expérimental), la façade (`rust/loki-rs/src/bench`)
|
||||
sert les quatre routes et déroule le protocole contre le moteur :
|
||||
|
||||
- `bench/mod.rs` : le protocole, porté du Go à l'octet près. On y trouve :
|
||||
- les mêmes corpus : la prose du Go et l'interface web embarquée, de la
|
||||
même source ;
|
||||
- les mêmes requêtes ;
|
||||
- le même calcul de profondeur, de reprise du cache et de débit ;
|
||||
- les mêmes messages d'erreur ;
|
||||
- `bench/job.rs` : le job, les routes et l'enregistrement (`last_bench`,
|
||||
`bench_presets`, avec l'empreinte de configuration du Go).
|
||||
|
||||
Le verrou de génération reste celui de la conversation du Go. La façade le
|
||||
prend par le **bail du moteur**, avec le motif `bench` :
|
||||
|
||||
- le Go note alors `benching` et donne aux autres le refus du benchmark ;
|
||||
- changer de discussion ne rompt pas le bail ;
|
||||
- `benchRunning()` le voit aussi : les clients `/v1`, le préchauffage et
|
||||
l'optimiseur attendent ;
|
||||
- un stop du chat côté Go (signalé au renouvellement), un Reset ou un bail
|
||||
perdu annulent le bench.
|
||||
|
||||
Le Go prête aussi à la façade :
|
||||
|
||||
- **le travail annexe** (`/internal/v1/engine/sidejob`) : à la fin, le slot
|
||||
du bench est effacé si c'est sans risque, comme pour un bench du Go ;
|
||||
- **la télémétrie** (`/internal/v1/bench/perf`) : la trace du bench dans
|
||||
`perf_log`.
|
||||
|
||||
Le port, la clé de l'API locale et le second slot (`SIDE_SLOT`) viennent de
|
||||
`/internal/v1/oai/route`. Les niveaux de raisonnement appris sont ceux de la
|
||||
conversation de la façade si elle la tient, sinon ceux du bench.
|
||||
|
||||
La ligne de commande (`loki bench`) reste au Go.
|
||||
|
||||
## Correction des tests du Go
|
||||
|
||||
Trois tests du préchauffage échouaient dix minutes par jour, avant minuit :
|
||||
le préchauffage se tait alors (`prewarmNearMidnight`), et ils lisaient
|
||||
l'heure réelle. L'heure de ce garde-fou devient remplaçable (`prewarmNow`),
|
||||
et les tests la fixent à midi.
|
||||
|
||||
## Parité
|
||||
|
||||
`internal/loki/parity_bench_test.go` (Linux) tire du Go un journal rejoué
|
||||
par `tests/bench.rs` contre le même faux moteur (`fakeBenchEngine`, porté).
|
||||
Chaque requête reçue par le moteur est comparée ; les longs textes (corpus)
|
||||
sont réduits à leur taille et à leur empreinte. Le journal couvre :
|
||||
|
||||
- **les routes** : statut et dernière mesure sans bench, GET refusés,
|
||||
annulation sans bench ;
|
||||
- **les refus au lancement** : preset externe, optimisation en cours, modèle
|
||||
pas prêt, génération en cours, second lancement pendant un bench ;
|
||||
- **les échecs dans le job** : moteur occupé par un autre (`/slots`), preset
|
||||
Strata ;
|
||||
- **les mesures** :
|
||||
- mode rapide, puis mode complet (profondeur, trois tours, reprise du
|
||||
cache, brouillon) ;
|
||||
- contexte trop petit (profondeur sautée), contexte inconnu du moteur
|
||||
(celui du preset) ;
|
||||
- paramètres bornés, petits paramètres, corps illisible ;
|
||||
- **les échecs d'une mesure** :
|
||||
- préchauffage, ligne courte, prefill à froid ou tour en profondeur en
|
||||
erreur ;
|
||||
- réponse sans timings ;
|
||||
- **le raisonnement** : niveau refusé par le gabarit puis rejoué, puis le
|
||||
niveau appris envoyé d'emblée ;
|
||||
- **budget et annulation** : budget d'une requête dépassé (profondeur
|
||||
partielle, non enregistrée), annulation pendant la profondeur ;
|
||||
- **après chaque étape** : la dernière mesure et les mesures par preset.
|
||||
|
||||
Autres tests :
|
||||
|
||||
- ceux du Go (`llm_bench_test.go`), portés (`bench/tests.rs`) : profondeur,
|
||||
types de cache, requête, morceaux du corpus, reprise du cache, relectures
|
||||
du disque, empreinte de l'enregistrement ;
|
||||
- la réponse du moteur (timings, cache, brouillon, nombres illisibles),
|
||||
`Duration.String` du Go, les flottants écrits à la Go ;
|
||||
- la façade : le bail « bench », le travail annexe (slot servi ou non), la
|
||||
trace, la clé de l'API locale, le second slot, le stop du chat et le bail
|
||||
perdu, les routes sans la façade ;
|
||||
- le Go (`TestFacadeBench`) :
|
||||
- les quatre routes partent à la façade, et elles seules ;
|
||||
- le bail « bench » : refus, bascule de discussion, libération,
|
||||
expiration ;
|
||||
- le travail annexe et la trace.
|
||||
|
||||
## Écarts assumés
|
||||
|
||||
- Une erreur réseau vers le moteur garde le texte de `reqwest` après le
|
||||
préfixe du Go (`Post "url": …`).
|
||||
- Un niveau de raisonnement appris par le bench de la façade n'est pas
|
||||
transmis au chat du Go, quand le Go tient la conversation (et
|
||||
inversement) : la première requête paie le refus, puis la traduction est
|
||||
retenue.
|
||||
@@ -245,6 +245,8 @@ func prewarmNearMidnight(now time.Time) bool {
|
||||
|
||||
// Points d'accroche remplaçables par les tests.
|
||||
var (
|
||||
// prewarmNow : l'heure du garde-fou de minuit.
|
||||
prewarmNow = time.Now
|
||||
// prewarmAsync : false = le préchauffage tourne dans l'appelant (tests).
|
||||
prewarmAsync = true
|
||||
// prewarmSlots : nombre de slots du moteur local (/props total_slots).
|
||||
@@ -375,7 +377,7 @@ func (c *Conversation) prewarm(why string) {
|
||||
return
|
||||
}
|
||||
}
|
||||
if sideJobs.Load() > 0 || benchRunning() || tuneGuard() != nil || prewarmNearMidnight(time.Now()) {
|
||||
if sideJobs.Load() > 0 || benchRunning() || tuneGuard() != nil || prewarmNearMidnight(prewarmNow()) {
|
||||
return
|
||||
}
|
||||
if !healthCheck() {
|
||||
|
||||
@@ -79,6 +79,11 @@ func (m *moteurPrewarm) completions() []string {
|
||||
// prewarmSetup : base neuve, discussion active, faux moteur à un slot,
|
||||
// préchauffage synchrone. keys : réglages posés.
|
||||
func prewarmSetup(t *testing.T, keys map[string]string) *moteurPrewarm {
|
||||
// Le préchauffage se tait dans les dix minutes avant minuit : midi ici,
|
||||
// pour que les tests ne dépendent pas de l'heure à laquelle ils tournent.
|
||||
oldNow := prewarmNow
|
||||
prewarmNow = func() time.Time { return time.Date(2026, 1, 1, 12, 0, 0, 0, time.Local) }
|
||||
t.Cleanup(func() { prewarmNow = oldNow })
|
||||
t.Helper()
|
||||
withWorkspace(t)
|
||||
activeProjMu.Lock()
|
||||
|
||||
@@ -0,0 +1,95 @@
|
||||
package loki
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
"os"
|
||||
"strings"
|
||||
"sync"
|
||||
)
|
||||
|
||||
// facade_bench.go — le banc d'essai tenu par la façade Rust (LOKI_BENCH=rust,
|
||||
// lot 27 de la migration ; expérimental, le Go reste le défaut) : les routes
|
||||
// /api/bench*, le job et le protocole de mesure (rust/loki-rs/src/bench). Le
|
||||
// verrou de génération reste celui de la conversation du Go : la façade le
|
||||
// prend par le bail du moteur, avec le motif « bench » (benching, et le refus
|
||||
// errBenchBusy pour les autres). Le Go lui prête aussi l'isolation des travaux
|
||||
// annexes (effacement du slot à la fin) et la télémétrie (perf_log.go).
|
||||
|
||||
const facadeBenchEnv = "LOKI_BENCH"
|
||||
|
||||
// facadeBench : la façade tient-elle le banc d'essai ?
|
||||
func facadeBench() bool {
|
||||
return strings.EqualFold(strings.TrimSpace(os.Getenv(facadeBenchEnv)), "rust") &&
|
||||
facadeAddr() != "" && internalToken() != ""
|
||||
}
|
||||
|
||||
// facadeBenchRoutes : les routes que la façade sert alors.
|
||||
var facadeBenchRoutes = map[string]bool{
|
||||
"/api/bench": true,
|
||||
"/api/bench/status": true,
|
||||
"/api/bench/cancel": true,
|
||||
"/api/bench/last": true,
|
||||
}
|
||||
|
||||
// facadeSideJobs : les travaux annexes ouverts par la façade (id → fin).
|
||||
var facadeSideJobs = struct {
|
||||
mu sync.Mutex
|
||||
seq int
|
||||
m map[int]func()
|
||||
}{m: map[int]func(){}}
|
||||
|
||||
// handleInternalSideJob : POST {} ouvre un travail annexe (engineSideJob) et
|
||||
// rend son id ; POST {"end": id, "served": bool} le ferme — served : le
|
||||
// moteur a servi au moins une requête du travail (engineServed), le slot
|
||||
// porte donc son état et peut être effacé.
|
||||
func handleInternalSideJob(w http.ResponseWriter, r *http.Request) {
|
||||
var in struct {
|
||||
End int `json:"end"`
|
||||
Served bool `json:"served"`
|
||||
}
|
||||
if !readLeaseBody(w, r, &in) {
|
||||
return
|
||||
}
|
||||
if in.End == 0 {
|
||||
done := engineSideJob()
|
||||
facadeSideJobs.mu.Lock()
|
||||
facadeSideJobs.seq++
|
||||
id := facadeSideJobs.seq
|
||||
facadeSideJobs.m[id] = done
|
||||
facadeSideJobs.mu.Unlock()
|
||||
sendJSON(w, http.StatusOK, map[string]any{"id": id})
|
||||
return
|
||||
}
|
||||
facadeSideJobs.mu.Lock()
|
||||
done := facadeSideJobs.m[in.End]
|
||||
delete(facadeSideJobs.m, in.End)
|
||||
facadeSideJobs.mu.Unlock()
|
||||
if done == nil {
|
||||
sendJSON(w, http.StatusNotFound, map[string]any{"error": "travail annexe inconnu"})
|
||||
return
|
||||
}
|
||||
if in.Served {
|
||||
engineServed()
|
||||
}
|
||||
done()
|
||||
w.WriteHeader(http.StatusNoContent)
|
||||
}
|
||||
|
||||
// handleInternalBenchPerf : la trace d'un bench de la façade dans la
|
||||
// télémétrie (le bench a pris le slot : la perte de cache du tour suivant lui
|
||||
// revient).
|
||||
func handleInternalBenchPerf(w http.ResponseWriter, r *http.Request) {
|
||||
var in struct {
|
||||
PromptN int `json:"prompt_n"`
|
||||
PromptMs float64 `json:"prompt_ms"`
|
||||
PromptPerSecond float64 `json:"prompt_per_second"`
|
||||
PredictedN int `json:"predicted_n"`
|
||||
PredictedPerSec float64 `json:"predicted_per_second"`
|
||||
}
|
||||
if !readLeaseBody(w, r, &in) {
|
||||
return
|
||||
}
|
||||
perfRecord(perfRec{Kind: perfBench, Complete: true, Total: in.PromptN, New: in.PromptN,
|
||||
PPms: in.PromptMs, PPtps: in.PromptPerSecond, Gen: in.PredictedN, TGtps: in.PredictedPerSec}, nil)
|
||||
w.WriteHeader(http.StatusNoContent)
|
||||
}
|
||||
@@ -0,0 +1,167 @@
|
||||
package loki
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Le banc d'essai part à la façade avec LOKI_BENCH=rust (lot 27) ; le reste,
|
||||
// non. La façade tient le verrou de génération par le bail du moteur, motif
|
||||
// « bench », et le Go lui prête le travail annexe et la télémétrie.
|
||||
func TestFacadeBench(t *testing.T) {
|
||||
testHome(t)
|
||||
var mu sync.Mutex
|
||||
var calls []string
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if r.Header.Get(internalTokenHeader) != jetonTest {
|
||||
w.WriteHeader(403)
|
||||
return
|
||||
}
|
||||
mu.Lock()
|
||||
calls = append(calls, r.Method+" "+r.URL.Path)
|
||||
mu.Unlock()
|
||||
io.WriteString(w, `{"ok":true,"de":"facade"}`)
|
||||
}))
|
||||
defer srv.Close()
|
||||
t.Setenv(internalTokenEnv, jetonTest)
|
||||
t.Setenv(facadeAddrEnv, strings.TrimPrefix(srv.URL, "http://"))
|
||||
local := func(w http.ResponseWriter, r *http.Request) { io.WriteString(w, "go") }
|
||||
|
||||
t.Setenv(facadeBenchEnv, "")
|
||||
if facadeBench() {
|
||||
t.Fatal("tenu sans LOKI_BENCH")
|
||||
}
|
||||
for p := range facadeBenchRoutes {
|
||||
w := httptest.NewRecorder()
|
||||
withFacadeChat(p, local)(w, httptest.NewRequest("POST", p, strings.NewReader("{}")))
|
||||
if w.Body.String() != "go" {
|
||||
t.Fatalf("sans LOKI_BENCH, %s : %q", p, w.Body.String())
|
||||
}
|
||||
}
|
||||
t.Setenv(facadeBenchEnv, " RUST")
|
||||
if len(facadeBenchRoutes) != 4 {
|
||||
t.Fatalf("routes : %v", facadeBenchRoutes)
|
||||
}
|
||||
for p := range facadeBenchRoutes {
|
||||
w := httptest.NewRecorder()
|
||||
withFacadeChat(p, local)(w, httptest.NewRequest("POST", p, strings.NewReader("{}")))
|
||||
if !strings.Contains(w.Body.String(), "facade") {
|
||||
t.Fatalf("%s non relayé : %q", p, w.Body.String())
|
||||
}
|
||||
}
|
||||
mu.Lock()
|
||||
n := len(calls)
|
||||
mu.Unlock()
|
||||
if n != len(facadeBenchRoutes) {
|
||||
t.Fatalf("relais : %v", calls)
|
||||
}
|
||||
w := httptest.NewRecorder()
|
||||
withFacadeChat("/api/tune", local)(w, httptest.NewRequest("POST", "/api/tune", nil))
|
||||
if w.Body.String() != "go" {
|
||||
t.Fatalf("l'optimiseur reste au Go : %q", w.Body.String())
|
||||
}
|
||||
|
||||
// Le bail « bench » : le verrou, le refus du bench pour les autres.
|
||||
post := func(h http.HandlerFunc, path, body string) (int, map[string]any) {
|
||||
rec := httptest.NewRecorder()
|
||||
h(rec, httptest.NewRequest("POST", path, strings.NewReader(body)))
|
||||
var out map[string]any
|
||||
_ = json.Unmarshal(rec.Body.Bytes(), &out)
|
||||
return rec.Code, out
|
||||
}
|
||||
code, out := post(handleInternalLease, "/internal/v1/engine/lease", `{"ttl_ms":60000,"why":"bench"}`)
|
||||
if code != 200 {
|
||||
t.Fatalf("bail : %d %v", code, out)
|
||||
}
|
||||
id, _ := out["lease"].(string)
|
||||
if !benchRunning() || conv.busyReason() != errBenchBusy {
|
||||
t.Fatalf("bail du bench : running=%v refus=%v", benchRunning(), conv.busyReason())
|
||||
}
|
||||
if code, out := post(handleInternalLease, "/internal/v1/engine/lease", `{}`); code != 409 || out["error"] != errBenchBusy.Error() {
|
||||
t.Fatalf("second bail : %d %v", code, out)
|
||||
}
|
||||
// Changer de discussion ne rompt pas le bail d'un bench.
|
||||
conv.loadFrom("autre", nil)
|
||||
if !benchRunning() || !conv.isGenerating() {
|
||||
t.Fatal("le bail du bench a sauté à la bascule de discussion")
|
||||
}
|
||||
if code, _ := post(handleInternalLeaseRelease, "/internal/v1/engine/lease/release", `{"lease":"`+id+`"}`); code != 204 {
|
||||
t.Fatalf("libération : %d", code)
|
||||
}
|
||||
if benchRunning() || conv.isGenerating() || conv.busyReason() == errBenchBusy {
|
||||
t.Fatal("bail rendu : le verrou devait être libre")
|
||||
}
|
||||
// Un bail ordinaire n'est pas un bench ; un bail de bench expiré rend
|
||||
// le verrou et le refus.
|
||||
code, out = post(handleInternalLease, "/internal/v1/engine/lease", `{}`)
|
||||
if code != 200 || benchRunning() {
|
||||
t.Fatalf("bail ordinaire : %d %v bench=%v", code, out, benchRunning())
|
||||
}
|
||||
post(handleInternalLeaseRelease, "/internal/v1/engine/lease/release", `{"lease":"`+out["lease"].(string)+`"}`)
|
||||
conv.mu.Lock()
|
||||
conv.extLeaseClearLocked()
|
||||
conv.Generating = false
|
||||
conv.mu.Unlock()
|
||||
if _, err := conv.extLeaseAcquire(5*time.Second, true); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
conv.mu.Lock()
|
||||
st := conv.extLease
|
||||
conv.mu.Unlock()
|
||||
conv.extLeaseExpire(st)
|
||||
if benchRunning() || conv.busyReason() == errBenchBusy {
|
||||
t.Fatal("bail de bench expiré : refus du bench encore là")
|
||||
}
|
||||
|
||||
// Le travail annexe ouvert puis fermé par la façade.
|
||||
code, out = post(handleInternalSideJob, "/internal/v1/engine/sidejob", `{}`)
|
||||
job, _ := out["id"].(float64)
|
||||
if code != 200 || job <= 0 || sideJobs.Load() != 1 {
|
||||
t.Fatalf("ouverture : %d %v (en cours : %d)", code, out, sideJobs.Load())
|
||||
}
|
||||
served := engineServedCount()
|
||||
if code, _ := post(handleInternalSideJob, "/internal/v1/engine/sidejob", `{"end":`+strconv.Itoa(int(job))+`,"served":true}`); code != 204 {
|
||||
t.Fatalf("fermeture : %d", code)
|
||||
}
|
||||
if sideJobs.Load() != 0 || engineServedCount() != served+1 {
|
||||
t.Fatalf("après la fermeture : en cours %d, servies %d", sideJobs.Load(), engineServedCount()-served)
|
||||
}
|
||||
if code, _ := post(handleInternalSideJob, "/internal/v1/engine/sidejob", `{"end":999}`); code != 404 {
|
||||
t.Fatalf("travail inconnu : %d", code)
|
||||
}
|
||||
rec := httptest.NewRecorder()
|
||||
handleInternalSideJob(rec, httptest.NewRequest("GET", "/internal/v1/engine/sidejob", nil))
|
||||
if rec.Code != 405 {
|
||||
t.Fatalf("GET : %d", rec.Code)
|
||||
}
|
||||
|
||||
// La trace du bench dans la télémétrie.
|
||||
var before uint64
|
||||
if recs := perfLog.snapshot(); len(recs) > 0 {
|
||||
before = recs[len(recs)-1].Seq
|
||||
}
|
||||
if code, _ := post(handleInternalBenchPerf, "/internal/v1/bench/perf",
|
||||
`{"prompt_n":2036,"prompt_ms":1018,"prompt_per_second":2000,"predicted_n":300,"predicted_per_second":50}`); code != 204 {
|
||||
t.Fatalf("trace : %d", code)
|
||||
}
|
||||
recs := perfLog.snapshot()
|
||||
if len(recs) == 0 || recs[len(recs)-1].Seq != before+1 {
|
||||
t.Fatalf("trace absente (dernière avant : %d)", before)
|
||||
}
|
||||
r := recs[len(recs)-1]
|
||||
if r.Kind != perfBench || !r.Complete || r.Total != 2036 || r.New != 2036 || r.Gen != 300 || r.TGtps != 50 || r.PPtps != 2000 || r.PPms != 1018 {
|
||||
t.Fatalf("trace : %+v", r)
|
||||
}
|
||||
|
||||
t.Setenv(internalTokenEnv, "")
|
||||
if facadeBench() {
|
||||
t.Fatal("tenu sans jeton interne")
|
||||
}
|
||||
}
|
||||
@@ -181,6 +181,8 @@ func withFacadeChat(path string, h http.HandlerFunc) http.HandlerFunc {
|
||||
held, unavailable = facadeStrata, "Strata indisponible : "
|
||||
case facadeTasksRoutes[path]:
|
||||
held, unavailable = facadeTasks, "tâches indisponibles : "
|
||||
case facadeBenchRoutes[path]:
|
||||
held, unavailable = facadeBench, "banc d'essai indisponible : "
|
||||
default:
|
||||
return h
|
||||
}
|
||||
|
||||
@@ -81,11 +81,13 @@ var benchJob struct {
|
||||
canceled bool
|
||||
}
|
||||
|
||||
// benchRunning : un benchmark de ce processus est-il en cours ?
|
||||
// benchRunning : un benchmark est-il en cours, de ce processus ou de la
|
||||
// façade (LOKI_BENCH=rust, qui tient alors le bail du moteur) ?
|
||||
func benchRunning() bool {
|
||||
benchJob.mu.Lock()
|
||||
defer benchJob.mu.Unlock()
|
||||
return benchJob.running
|
||||
running := benchJob.running
|
||||
benchJob.mu.Unlock()
|
||||
return running || benchLeased()
|
||||
}
|
||||
|
||||
// benchRunner est runBench, remplaçable dans les tests.
|
||||
|
||||
@@ -0,0 +1,375 @@
|
||||
//go:build linux
|
||||
|
||||
package loki
|
||||
|
||||
import (
|
||||
"crypto/sha256"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"net"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"regexp"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// parity_bench_test.go — journal de référence du banc d'essai (lot 27 de la
|
||||
// migration Rust) : les quatre routes, chaque refus au lancement, le mode
|
||||
// rapide et le mode complet (profondeur, tours, reprise du cache), les
|
||||
// échecs d'une mesure, le niveau de raisonnement refusé puis rejoué, la
|
||||
// profondeur partielle et l'annulation, contre le faux moteur des tests du
|
||||
// bench (llm_bench_test.go). Chaque requête reçue par le moteur est
|
||||
// enregistrée — les longs textes (corpus) réduits à leur taille et à leur
|
||||
// empreinte. Rangé dans rust/loki-rs/tests/parity_bench/bench.json ;
|
||||
// tests/bench.rs le rejoue par la façade.
|
||||
//
|
||||
// LOKI_PARITY_UPDATE=1 go test ./internal/loki -run TestPariteBench
|
||||
|
||||
const parityBenchFile = "../../rust/loki-rs/tests/parity_bench/bench.json"
|
||||
|
||||
// parityBenchEngine : le réglage du faux moteur pour une étape (remis à zéro
|
||||
// à chaque étape qui en donne un).
|
||||
type parityBenchEngine struct {
|
||||
NCtx int `json:"n_ctx"`
|
||||
Busy bool `json:"busy,omitempty"`
|
||||
FailAt int `json:"fail_at,omitempty"`
|
||||
NoTiming bool `json:"no_timing,omitempty"`
|
||||
RejectEffort string `json:"reject_effort,omitempty"`
|
||||
SlowHotMs int `json:"slow_hot_ms,omitempty"`
|
||||
TimeoutMs int `json:"timeout_ms,omitempty"` // benchReqTimeout
|
||||
}
|
||||
|
||||
type parityBenchStep struct {
|
||||
Name string `json:"name"`
|
||||
Config map[string]string `json:"config,omitempty"`
|
||||
Files map[string]string `json:"files,omitempty"`
|
||||
Engine *parityBenchEngine `json:"engine,omitempty"`
|
||||
Tune bool `json:"tune,omitempty"`
|
||||
Generating bool `json:"generating,omitempty"`
|
||||
Method string `json:"method"`
|
||||
Path string `json:"path"`
|
||||
Body string `json:"body,omitempty"`
|
||||
Status int `json:"status"`
|
||||
Out any `json:"out"`
|
||||
// Again : un second lancement pendant le bench ; CancelAt : la phase
|
||||
// (préfixe) à laquelle l'annuler.
|
||||
Again bool `json:"again,omitempty"`
|
||||
AgainOut any `json:"again_out,omitempty"`
|
||||
CancelAt string `json:"cancel_at,omitempty"`
|
||||
Wait bool `json:"wait,omitempty"`
|
||||
Job any `json:"job,omitempty"` // état final du job
|
||||
Calls []any `json:"calls"` // requêtes reçues par le moteur
|
||||
Last any `json:"last"` // state/last_bench
|
||||
Presets any `json:"presets"` // state/bench_presets
|
||||
}
|
||||
|
||||
var (
|
||||
parityBenchIDRe = regexp.MustCompile(`"b[0-9]+-[0-9]+"`)
|
||||
parityBenchTRe = regexp.MustCompile(`"(elapsed_sec|at)":[0-9.e+-]+`)
|
||||
parityBenchFPRe = regexp.MustCompile(`"fingerprint":"[0-9a-f]+"`)
|
||||
)
|
||||
|
||||
// parityBenchDigest : une valeur JSON dont les longs textes sont réduits à
|
||||
// leur taille et à leur empreinte.
|
||||
func parityBenchDigest(v any) any {
|
||||
switch x := v.(type) {
|
||||
case map[string]any:
|
||||
out := map[string]any{}
|
||||
for k, e := range x {
|
||||
out[k] = parityBenchDigest(e)
|
||||
}
|
||||
return out
|
||||
case []any:
|
||||
out := make([]any, len(x))
|
||||
for i, e := range x {
|
||||
out[i] = parityBenchDigest(e)
|
||||
}
|
||||
return out
|
||||
case string:
|
||||
if len(x) > 64 {
|
||||
sum := sha256.Sum256([]byte(x))
|
||||
return fmt.Sprintf("<%d octets %x>", len(x), sum[:6])
|
||||
}
|
||||
return x
|
||||
}
|
||||
return v
|
||||
}
|
||||
|
||||
func TestPariteBench(t *testing.T) {
|
||||
home := testHome(t)
|
||||
oldTimeout := benchReqTimeout
|
||||
// Un état laissé par un autre test du paquet : ni niveau appris, ni bench
|
||||
// précédent.
|
||||
resetBench := func() {
|
||||
effortFallbacks.Range(func(k, _ any) bool {
|
||||
effortFallbacks.Delete(k)
|
||||
return true
|
||||
})
|
||||
benchJob.mu.Lock()
|
||||
benchJob.running, benchJob.id, benchJob.mode, benchJob.phase = false, "", "", ""
|
||||
benchJob.step, benchJob.steps, benchJob.cancel, benchJob.result = 0, 0, nil, nil
|
||||
benchJob.err, benchJob.canceled = "", false
|
||||
benchJob.mu.Unlock()
|
||||
}
|
||||
resetBench()
|
||||
t.Cleanup(func() {
|
||||
benchReqTimeout = oldTimeout
|
||||
// Les niveaux de raisonnement appris ici ne valent que pour ce test.
|
||||
effortFallbacks.Range(func(k, _ any) bool {
|
||||
effortFallbacks.Delete(k)
|
||||
return true
|
||||
})
|
||||
})
|
||||
|
||||
// Le faux moteur : celui des tests du bench, derrière un journal de
|
||||
// toutes les requêtes reçues.
|
||||
var mu sync.Mutex
|
||||
var calls []any
|
||||
fake := &fakeBenchEngine{}
|
||||
inner := http.Handler(nil)
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
b, _ := io.ReadAll(r.Body)
|
||||
entry := map[string]any{"method": r.Method, "path": r.URL.Path}
|
||||
if len(b) > 0 {
|
||||
var body any
|
||||
if json.Unmarshal(b, &body) == nil {
|
||||
entry["body"] = parityBenchDigest(body)
|
||||
}
|
||||
}
|
||||
mu.Lock()
|
||||
calls = append(calls, entry)
|
||||
h := inner
|
||||
mu.Unlock()
|
||||
r.Body = io.NopCloser(strings.NewReader(string(b)))
|
||||
h.ServeHTTP(w, r)
|
||||
}))
|
||||
defer srv.Close()
|
||||
port := strings.TrimPrefix(srv.URL, "http://127.0.0.1:")
|
||||
l, err := net.Listen("tcp", "127.0.0.1:0")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
closed := strconv.Itoa(l.Addr().(*net.TCPAddr).Port)
|
||||
l.Close()
|
||||
|
||||
mask := func(s string) string {
|
||||
s = strings.ReplaceAll(s, home, "{{HOME}}")
|
||||
s = parityBenchIDRe.ReplaceAllString(s, `"{{ID}}"`)
|
||||
s = parityBenchTRe.ReplaceAllString(s, `"$1":"{{T}}"`)
|
||||
return parityBenchFPRe.ReplaceAllString(s, `"fingerprint":"{{FP}}"`)
|
||||
}
|
||||
unmask := func(s string) string {
|
||||
s = strings.ReplaceAll(s, "{{PORT}}", port)
|
||||
s = strings.ReplaceAll(s, "{{CLOSED}}", closed)
|
||||
return strings.ReplaceAll(s, "{{HOME}}", home)
|
||||
}
|
||||
decode := func(raw string) any {
|
||||
var v any
|
||||
if raw == "" {
|
||||
return nil
|
||||
}
|
||||
if err := json.Unmarshal([]byte(mask(raw)), &v); err != nil {
|
||||
t.Fatalf("illisible : %q", raw)
|
||||
}
|
||||
return v
|
||||
}
|
||||
handlers := map[string]http.HandlerFunc{
|
||||
"/api/bench": handleBench,
|
||||
"/api/bench/status": handleBenchStatus,
|
||||
"/api/bench/cancel": handleBenchCancel,
|
||||
"/api/bench/last": handleBenchLast,
|
||||
}
|
||||
call := func(method, path, body string) (int, any) {
|
||||
rec := httptest.NewRecorder()
|
||||
handlers[path](rec, httptest.NewRequest(method, path, strings.NewReader(body)))
|
||||
return rec.Code, decode(rec.Body.String())
|
||||
}
|
||||
status := func() map[string]any {
|
||||
_, v := call("GET", "/api/bench/status", "")
|
||||
return v.(map[string]any)
|
||||
}
|
||||
wait := func(name string) {
|
||||
deadline := time.Now().Add(60 * time.Second)
|
||||
for status()["running"] == true {
|
||||
if time.Now().After(deadline) {
|
||||
t.Fatalf("%s : le bench ne finit pas", name)
|
||||
}
|
||||
time.Sleep(5 * time.Millisecond)
|
||||
}
|
||||
}
|
||||
|
||||
var steps []parityBenchStep
|
||||
run := func(st parityBenchStep) {
|
||||
for k, v := range st.Config {
|
||||
if err := SetConfigKey(k, unmask(v)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
files := map[string]string{}
|
||||
for k, v := range st.Files {
|
||||
files[k] = unmask(v)
|
||||
}
|
||||
parityApplyFiles(t, home, files)
|
||||
if e := st.Engine; e != nil {
|
||||
f := &fakeBenchEngine{nCtx: e.NCtx, busy: e.Busy, failAt: e.FailAt, noTiming: e.NoTiming,
|
||||
rejectEffort: e.RejectEffort, slowHot: time.Duration(e.SlowHotMs) * time.Millisecond}
|
||||
mu.Lock()
|
||||
fake = f
|
||||
inner = f.handler(t)
|
||||
mu.Unlock()
|
||||
benchReqTimeout = oldTimeout
|
||||
if e.TimeoutMs > 0 {
|
||||
benchReqTimeout = time.Duration(e.TimeoutMs) * time.Millisecond
|
||||
}
|
||||
}
|
||||
mu.Lock()
|
||||
calls = nil
|
||||
mu.Unlock()
|
||||
var lock *tuneLock
|
||||
if st.Tune {
|
||||
l, err := tuneLockAcquire("web", false)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
lock = l
|
||||
}
|
||||
if st.Generating {
|
||||
conv.mu.Lock()
|
||||
conv.Generating = true
|
||||
conv.mu.Unlock()
|
||||
}
|
||||
st.Status, st.Out = call(st.Method, st.Path, unmask(st.Body))
|
||||
if st.Generating {
|
||||
conv.mu.Lock()
|
||||
conv.Generating = false
|
||||
conv.mu.Unlock()
|
||||
}
|
||||
if lock != nil {
|
||||
lock.release()
|
||||
}
|
||||
if st.Again {
|
||||
_, st.AgainOut = call("POST", "/api/bench", "{}")
|
||||
}
|
||||
if st.CancelAt != "" {
|
||||
deadline := time.Now().Add(30 * time.Second)
|
||||
for {
|
||||
p, _ := status()["phase"].(string)
|
||||
if strings.HasPrefix(p, st.CancelAt) {
|
||||
break
|
||||
}
|
||||
if time.Now().After(deadline) {
|
||||
t.Fatalf("%s : phase %q jamais atteinte", st.Name, st.CancelAt)
|
||||
}
|
||||
time.Sleep(2 * time.Millisecond)
|
||||
}
|
||||
// Le moteur a reçu la requête de cette phase avant l'annulation.
|
||||
for {
|
||||
mu.Lock()
|
||||
n := len(fake.calls)
|
||||
mu.Unlock()
|
||||
if n >= 4 {
|
||||
break
|
||||
}
|
||||
time.Sleep(2 * time.Millisecond)
|
||||
}
|
||||
call("POST", "/api/bench/cancel", "")
|
||||
}
|
||||
if st.Wait || st.CancelAt != "" {
|
||||
wait(st.Name)
|
||||
st.Job = status()
|
||||
}
|
||||
mu.Lock()
|
||||
st.Calls = append([]any{}, calls...)
|
||||
mu.Unlock()
|
||||
st.Last = decode(string(getBytes(bkState, "last_bench")))
|
||||
st.Presets = decode(string(getBytes(bkState, "bench_presets")))
|
||||
steps = append(steps, st)
|
||||
}
|
||||
|
||||
preset := "MODEL=/models/qwen.gguf\nPORT={{PORT}}\nCTX=16384\nTOP_K=40\nMIN_P=0.05\nREASONING=off\nCACHE_ISOLATE=off\n"
|
||||
base := map[string]string{"MODEL": "/models/qwen.gguf", "PORT": "{{PORT}}", "CTX": "16384",
|
||||
"TOP_K": "40", "MIN_P": "0.05", "REASONING": "off", "CACHE_ISOLATE": "off"}
|
||||
for _, st := range []parityBenchStep{
|
||||
{Name: "statut sans bench", Config: base, Files: map[string]string{"presets/qwen.env": preset},
|
||||
Engine: &parityBenchEngine{NCtx: 16384}, Method: "GET", Path: "/api/bench/status"},
|
||||
{Name: "dernier bench : aucun", Method: "GET", Path: "/api/bench/last"},
|
||||
{Name: "lancement en GET", Method: "GET", Path: "/api/bench"},
|
||||
{Name: "annulation en GET", Method: "GET", Path: "/api/bench/cancel"},
|
||||
{Name: "annulation sans bench", Method: "POST", Path: "/api/bench/cancel"},
|
||||
{Name: "preset externe", Config: map[string]string{"EXTERNAL": "1"}, Method: "POST", Path: "/api/bench", Body: `{}`},
|
||||
{Name: "optimisation en cours", Config: map[string]string{"EXTERNAL": ""}, Tune: true, Method: "POST", Path: "/api/bench", Body: `{}`},
|
||||
{Name: "modèle pas prêt", Config: map[string]string{"PORT": "{{CLOSED}}"}, Method: "POST", Path: "/api/bench", Body: `{}`},
|
||||
{Name: "génération en cours", Config: map[string]string{"PORT": "{{PORT}}"}, Generating: true, Method: "POST", Path: "/api/bench", Body: `{}`},
|
||||
{Name: "moteur occupé par un autre", Engine: &parityBenchEngine{NCtx: 16384, Busy: true},
|
||||
Method: "POST", Path: "/api/bench", Body: `{}`, Wait: true},
|
||||
{Name: "mode rapide", Engine: &parityBenchEngine{NCtx: 16384}, Method: "POST", Path: "/api/bench", Body: `{}`, Wait: true},
|
||||
{Name: "dernier bench", Method: "GET", Path: "/api/bench/last"},
|
||||
{Name: "mode complet", Engine: &parityBenchEngine{NCtx: 16384}, Method: "POST", Path: "/api/bench", Body: `{"mode":" FULL "}`, Wait: true},
|
||||
{Name: "mode complet, contexte trop petit", Engine: &parityBenchEngine{NCtx: 4096},
|
||||
Method: "POST", Path: "/api/bench", Body: `{"mode":"full"}`, Wait: true},
|
||||
{Name: "contexte inconnu du moteur : celui du preset", Engine: &parityBenchEngine{NCtx: 0},
|
||||
Method: "POST", Path: "/api/bench", Body: `{"mode":"full"}`, Wait: true},
|
||||
{Name: "paramètres bornés", Engine: &parityBenchEngine{NCtx: 65536},
|
||||
Method: "POST", Path: "/api/bench", Body: `{"prompt":99999,"n":9999,"mode":"quick"}`, Wait: true},
|
||||
{Name: "petits paramètres", Engine: &parityBenchEngine{NCtx: 16384},
|
||||
Method: "POST", Path: "/api/bench", Body: `{"prompt":100,"n":8}`, Wait: true},
|
||||
{Name: "corps illisible : défauts", Engine: &parityBenchEngine{NCtx: 16384},
|
||||
Method: "POST", Path: "/api/bench", Body: `{"prompt":"beaucoup"}`, Wait: true},
|
||||
{Name: "le préchauffage échoue", Engine: &parityBenchEngine{NCtx: 16384, FailAt: 1},
|
||||
Method: "POST", Path: "/api/bench", Body: `{}`, Wait: true},
|
||||
{Name: "la ligne courte échoue", Engine: &parityBenchEngine{NCtx: 16384, FailAt: 2},
|
||||
Method: "POST", Path: "/api/bench", Body: `{}`, Wait: true},
|
||||
{Name: "le prefill à froid échoue", Engine: &parityBenchEngine{NCtx: 16384, FailAt: 3},
|
||||
Method: "POST", Path: "/api/bench", Body: `{"mode":"full"}`, Wait: true},
|
||||
{Name: "un tour en profondeur échoue", Engine: &parityBenchEngine{NCtx: 16384, FailAt: 5},
|
||||
Method: "POST", Path: "/api/bench", Body: `{"mode":"full"}`, Wait: true},
|
||||
{Name: "sans timings", Engine: &parityBenchEngine{NCtx: 16384, NoTiming: true},
|
||||
Method: "POST", Path: "/api/bench", Body: `{}`, Wait: true},
|
||||
{Name: "niveau de raisonnement refusé puis rejoué",
|
||||
Config: map[string]string{"REASONING": "", "REASONING_EFFORT": "high"},
|
||||
Engine: &parityBenchEngine{NCtx: 16384, RejectEffort: "high"},
|
||||
Method: "POST", Path: "/api/bench", Body: `{}`, Wait: true},
|
||||
{Name: "niveau appris : envoyé d'emblée", Engine: &parityBenchEngine{NCtx: 16384, RejectEffort: "high"},
|
||||
Method: "POST", Path: "/api/bench", Body: `{}`, Wait: true},
|
||||
{Name: "budget d'une requête dépassé : profondeur partielle",
|
||||
Config: map[string]string{"REASONING_EFFORT": "", "REASONING": "off"},
|
||||
Engine: &parityBenchEngine{NCtx: 16384, SlowHotMs: 400, TimeoutMs: 150},
|
||||
Method: "POST", Path: "/api/bench", Body: `{"mode":"full"}`, Wait: true},
|
||||
{Name: "annulation pendant la profondeur", Engine: &parityBenchEngine{NCtx: 16384, SlowHotMs: 20000},
|
||||
Method: "POST", Path: "/api/bench", Body: `{"mode":"full"}`, Again: true, CancelAt: "tour 1/"},
|
||||
{Name: "preset Strata", Config: map[string]string{"ENGINE": "strata"}, Engine: &parityBenchEngine{NCtx: 16384},
|
||||
Method: "POST", Path: "/api/bench", Body: `{}`, Wait: true},
|
||||
{Name: "dernier bench, inchangé", Config: map[string]string{"ENGINE": ""}, Method: "GET", Path: "/api/bench/last"},
|
||||
} {
|
||||
run(st)
|
||||
}
|
||||
|
||||
got, err := parityJSON(map[string]any{"steps": steps})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if os.Getenv("LOKI_PARITY_UPDATE") == "1" {
|
||||
if err := os.MkdirAll("../../rust/loki-rs/tests/parity_bench", 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(parityBenchFile, got, 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return
|
||||
}
|
||||
want, err := os.ReadFile(parityBenchFile)
|
||||
if err != nil {
|
||||
t.Fatalf("journal de référence absent (%v) : LOKI_PARITY_UPDATE=1 pour le créer", err)
|
||||
}
|
||||
if string(got) != string(want) {
|
||||
t.Errorf("le banc d'essai ne produit plus le journal rangé dans %s.\n"+
|
||||
"Si c'est voulu : LOKI_PARITY_UPDATE=1 go test ./internal/loki -run TestPariteBench,\n"+
|
||||
"puis cargo test --test bench dit si le port Rust suit.\n--- obtenu ---\n%s", parityBenchFile, got)
|
||||
}
|
||||
}
|
||||
@@ -129,6 +129,10 @@ func withInternalAPI(next http.Handler) http.Handler {
|
||||
handleInternalProjectSwitch(w, r)
|
||||
case "/internal/v1/project/drop-convs":
|
||||
handleInternalProjectDropConvs(w, r)
|
||||
case "/internal/v1/engine/sidejob":
|
||||
handleInternalSideJob(w, r)
|
||||
case "/internal/v1/bench/perf":
|
||||
handleInternalBenchPerf(w, r)
|
||||
case "/internal/v1/tasks/run":
|
||||
handleInternalTaskRun(w, r)
|
||||
case "/internal/v1/push/send":
|
||||
|
||||
@@ -6,6 +6,7 @@ import (
|
||||
"io"
|
||||
"net/http"
|
||||
"os"
|
||||
"sync/atomic"
|
||||
"time"
|
||||
)
|
||||
|
||||
@@ -43,18 +44,29 @@ type extLeaseState struct {
|
||||
stop bool
|
||||
ttl time.Duration
|
||||
timer *time.Timer
|
||||
// bench : le bail d'un benchmark de la façade (LOKI_BENCH=rust) — les
|
||||
// autres reçoivent errBenchBusy, et changer de discussion ne le rompt pas.
|
||||
bench bool
|
||||
}
|
||||
|
||||
// extLeaseClearLocked oublie le bail de la façade. c.mu détenu.
|
||||
// benchLeaseHeld : un bail « bench » de la façade est tenu. Lu sans le verrou
|
||||
// de la conversation (benchRunning est appelé par des chemins qui le tiennent
|
||||
// déjà, le préchauffage par exemple).
|
||||
var benchLeaseHeld atomic.Bool
|
||||
|
||||
func (c *Conversation) extLeaseClearLocked() {
|
||||
if c.extLease != nil {
|
||||
if c.extLease.bench {
|
||||
benchLeaseHeld.Store(false)
|
||||
}
|
||||
c.extLease.timer.Stop()
|
||||
c.extLease = nil
|
||||
}
|
||||
}
|
||||
|
||||
// extLeaseAcquire prend le verrou de génération pour la façade.
|
||||
func (c *Conversation) extLeaseAcquire(ttl time.Duration) (string, error) {
|
||||
func (c *Conversation) extLeaseAcquire(ttl time.Duration, bench bool) (string, error) {
|
||||
if err := tuneGuard(); err != nil {
|
||||
return "", err
|
||||
}
|
||||
@@ -67,9 +79,13 @@ func (c *Conversation) extLeaseAcquire(ttl time.Duration) (string, error) {
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
st := &extLeaseState{id: fmt.Sprintf("%x", b), ttl: ttl}
|
||||
st := &extLeaseState{id: fmt.Sprintf("%x", b), ttl: ttl, bench: bench}
|
||||
st.timer = time.AfterFunc(ttl, func() { c.extLeaseExpire(st) })
|
||||
c.Generating = true
|
||||
if bench {
|
||||
c.benching, c.leaseWhy = true, errBenchBusy
|
||||
benchLeaseHeld.Store(true)
|
||||
}
|
||||
c.cancel = func() {
|
||||
c.mu.Lock()
|
||||
st.stop = true
|
||||
@@ -88,6 +104,10 @@ func (c *Conversation) extLeaseExpire(st *extLeaseState) {
|
||||
}
|
||||
c.extLease = nil
|
||||
c.Generating, c.cancel = false, nil
|
||||
if st.bench {
|
||||
c.benching, c.leaseWhy = false, nil
|
||||
benchLeaseHeld.Store(false)
|
||||
}
|
||||
fmt.Fprintf(os.Stderr, "[lease] bail de la façade expiré sans renouvellement (%s) : verrou du moteur rendu\n", st.ttl)
|
||||
}
|
||||
|
||||
@@ -110,10 +130,16 @@ func (c *Conversation) extLeaseRelease(id string) {
|
||||
if c.extLease == nil || c.extLease.id != id {
|
||||
return
|
||||
}
|
||||
if c.extLease.bench {
|
||||
c.benching, c.leaseWhy = false, nil
|
||||
}
|
||||
c.extLeaseClearLocked()
|
||||
c.Generating, c.cancel = false, nil
|
||||
}
|
||||
|
||||
// benchLeased : un benchmark de la façade tient-il le verrou ?
|
||||
func benchLeased() bool { return benchLeaseHeld.Load() }
|
||||
|
||||
func readLeaseBody(w http.ResponseWriter, r *http.Request, v any) bool {
|
||||
if r.Method != http.MethodPost {
|
||||
w.Header().Set("Allow", http.MethodPost)
|
||||
@@ -134,7 +160,8 @@ func readLeaseBody(w http.ResponseWriter, r *http.Request, v any) bool {
|
||||
|
||||
func handleInternalLease(w http.ResponseWriter, r *http.Request) {
|
||||
var in struct {
|
||||
TTLms int64 `json:"ttl_ms"`
|
||||
TTLms int64 `json:"ttl_ms"`
|
||||
Why string `json:"why"` // "bench" : le bail d'un benchmark
|
||||
}
|
||||
if !readLeaseBody(w, r, &in) {
|
||||
return
|
||||
@@ -143,7 +170,7 @@ func handleInternalLease(w http.ResponseWriter, r *http.Request) {
|
||||
if in.TTLms > 0 {
|
||||
ttl = min(max(time.Duration(in.TTLms)*time.Millisecond, extLeaseMinTTL), extLeaseMaxTTL)
|
||||
}
|
||||
id, err := conv.extLeaseAcquire(ttl)
|
||||
id, err := conv.extLeaseAcquire(ttl, in.Why == "bench")
|
||||
if err != nil {
|
||||
if err == ErrBusy {
|
||||
err = conv.busyReason() // dit QUI occupe le moteur
|
||||
|
||||
@@ -86,7 +86,7 @@ func TestBailInterneVerrouPartage(t *testing.T) {
|
||||
func TestBailInterneNeLibereQueLeSien(t *testing.T) {
|
||||
testHome(t)
|
||||
libérerVerrou(t)
|
||||
id, err := conv.extLeaseAcquire(time.Minute)
|
||||
id, err := conv.extLeaseAcquire(time.Minute, false)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
@@ -110,7 +110,7 @@ func TestBailInterneNeLibereQueLeSien(t *testing.T) {
|
||||
func TestBailInterneExpire(t *testing.T) {
|
||||
testHome(t)
|
||||
libérerVerrou(t)
|
||||
id, err := conv.extLeaseAcquire(50 * time.Millisecond)
|
||||
id, err := conv.extLeaseAcquire(50*time.Millisecond, false)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
@@ -65,6 +65,8 @@ pub struct Ctx {
|
||||
/// Tâches planifiées (routes, planificateur, tâches script) tenues par la
|
||||
/// façade (`LOKI_TASKS=rust`).
|
||||
pub tasks: Option<std::sync::Arc<crate::tasks::Tasks>>,
|
||||
/// Banc d'essai tenu par la façade (`LOKI_BENCH=rust`).
|
||||
pub bench: Option<std::sync::Arc<crate::bench::job::Bench>>,
|
||||
/// Jeton de l'API interne : une requête qui le porte vient du Go.
|
||||
pub internal_token: String,
|
||||
}
|
||||
@@ -90,6 +92,7 @@ impl Ctx {
|
||||
|| self.engineimg.is_some()
|
||||
|| self.strata.is_some()
|
||||
|| self.tasks.is_some()
|
||||
|| self.bench.is_some()
|
||||
}
|
||||
|
||||
/// La requête vient du Go (jeton interne valide) : déjà authentifiée par
|
||||
|
||||
@@ -0,0 +1,10 @@
|
||||
In the early hours of an October morning, Camille walked along the canal, watching the cargo barges slip past the iron bridge that spanned the water. She thought about the meeting she had skipped, the unanswered messages on her phone, the way the city always seemed to forget her name after summer ended. Three streets away, a pâtisserie opened its shutters and the smell of warm butter mixed with diesel exhaust from the waiting bus.
|
||||
Pendant ce temps, à Marseille, un chercheur en biologie marine prépare son matériel pour une plongée. Il étudie les herbiers de posidonie, ces prairies sous-marines vieilles de plusieurs milliers d'années qui stockent autant de carbone qu'une forêt amazonienne. Le bateau quitte le port à six heures vingt-trois.
|
||||
Quantum computers, properly engineered, can solve certain classes of problems exponentially faster than classical machines. The catch is that decoherence ruins everything. Engineers use dilution refrigerators to drop superconducting qubits to fifteen millikelvin, colder than deep space. The wires connecting the chip to room-temperature electronics must dissipate almost no heat, or the qubit state collapses before any useful computation finishes.
|
||||
Le boulanger lève la pâte à quatre heures. Il regarde la balance numérique en plissant les yeux : six cent vingt-trois grammes, presque le compte. Son chien dort sur le tapis de farine près du four. Dehors, deux chats se disputent un poisson abandonné par le pêcheur de nuit.
|
||||
Consider a recursive descent parser written in Go. The lexer emits tokens; the parser consumes them and produces an abstract syntax tree. Error recovery is hard: after a syntax error, the parser must resynchronize at a known boundary—a semicolon, a closing brace—without losing track of subsequent diagnostics. Tree-sitter solves this with incremental parsing and a glr-like algorithm.
|
||||
Le philosophe stoïcien disait : "Ce qui nous trouble, ce n'est pas ce qui nous arrive, mais l'opinion que nous nous en faisons." Vingt siècles plus tard, la phrase apparaît dans un livre de poche au rayon développement personnel d'une librairie d'aéroport, à côté d'un roman policier suédois.
|
||||
Mitochondria descended from ancient bacteria engulfed by archaeal cells roughly two billion years ago. They still keep their own ring of DNA, separate from the nuclear genome. Mutations in mitochondrial DNA accumulate with age and have been implicated in everything from Parkinson's disease to ordinary muscle fatigue. Yet they remain stubbornly difficult to repair therapeutically because each cell contains hundreds.
|
||||
La marée descend lentement, exposant des rochers couverts d'huîtres et d'algues vertes. Un héron immobile surveille les flaques laissées par l'eau. Plus loin, deux enfants courent avec un cerf-volant rouge qui refuse de monter à cause de l'humidité dans la voile.
|
||||
Compilers translate high-level languages into machine code through several intermediate representations. LLVM IR sits in the middle: typed, mostly static-single-assignment, suitable for both aggressive optimization and direct lowering to x86 or ARM. The optimizer runs dozens of passes—dead code elimination, loop-invariant code motion, induction variable simplification—each touching the IR in carefully ordered ways.
|
||||
Le cuisinier ferme les yeux pour goûter la sauce. Trop salée. Il ajoute une pomme de terre crue coupée en quartiers, sachant qu'elle absorbera l'excès en mijotant vingt minutes. Sa grand-mère lui a appris ce geste un dimanche de novembre il y a très longtemps.
|
||||
@@ -0,0 +1,684 @@
|
||||
//! Le banc d'essai en tâche de fond (`llm_bench_job.go`), ses quatre routes
|
||||
//! et l'enregistrement des mesures (`llm_bench.go`, « enregistrement ») :
|
||||
//!
|
||||
//! - `POST /api/bench` lance (202 + id) ; refus clairs avant de lancer ;
|
||||
//! - `GET /api/bench/status` : la phase en cours, puis le résultat ou l'erreur ;
|
||||
//! - `POST /api/bench/cancel` annule ;
|
||||
//! - `GET /api/bench/last` : la dernière mesure enregistrée.
|
||||
//!
|
||||
//! Pendant la mesure, le bench tient le verrou de génération du Go par le
|
||||
//! bail du moteur, motif « bench » : un message, une tâche ou un client /v1
|
||||
//! reçoivent le refus du bench. Un stop du chat (ou un Reset) côté Go
|
||||
//! interrompt le bail : le bench est annulé.
|
||||
|
||||
use std::collections::{BTreeMap, HashMap};
|
||||
use std::path::PathBuf;
|
||||
use std::sync::atomic::{AtomicU64, Ordering};
|
||||
use std::sync::{Arc, Mutex};
|
||||
use std::time::{Duration, Instant};
|
||||
|
||||
use http_body_util::{BodyExt, Limited};
|
||||
use hyper::body::Incoming;
|
||||
use hyper::{Method, Request, Response, StatusCode};
|
||||
use serde::{Deserialize, Serialize};
|
||||
use serde_json::{Value, json};
|
||||
use tokio::sync::watch;
|
||||
|
||||
use super::{BenchResult, Corpora, Engine, Opts, Setup};
|
||||
use crate::api::Ctx;
|
||||
use crate::auth::json as send_json;
|
||||
use crate::chat::lease::EngineLease;
|
||||
use crate::chat::wire::{EffortFallbacks, log_effort_fallback};
|
||||
use crate::gojson::{self, Kind};
|
||||
use crate::proxy::Body;
|
||||
use crate::store::BK_STATE;
|
||||
|
||||
type Cfg = HashMap<String, String>;
|
||||
|
||||
const LAST_KEY: &str = "last_bench";
|
||||
const PRESETS_KEY: &str = "bench_presets";
|
||||
const BENCH_REQ: &str =
|
||||
r#"struct { Mode string "json:\"mode\""; Prompt int "json:\"prompt\""; N int "json:\"n\"" }"#;
|
||||
|
||||
/// Le job : le bench en cours ou le dernier terminé (un seul à la fois).
|
||||
#[derive(Default)]
|
||||
struct JobState {
|
||||
seq: i64,
|
||||
running: bool,
|
||||
/// Lancement en cours (le bail se prend hors du verrou).
|
||||
starting: bool,
|
||||
id: String,
|
||||
mode: String,
|
||||
phase: String,
|
||||
step: i64,
|
||||
steps: i64,
|
||||
started: Option<Instant>,
|
||||
finished: Option<Instant>,
|
||||
cancel: Option<watch::Sender<bool>>,
|
||||
result: Option<BenchResult>,
|
||||
err: String,
|
||||
canceled: bool,
|
||||
}
|
||||
|
||||
/// Le banc d'essai tenu par la façade.
|
||||
pub struct Bench {
|
||||
pub home: PathBuf,
|
||||
state: Mutex<JobState>,
|
||||
/// Budget d'une requête de chat (`benchReqTimeout`).
|
||||
pub req_timeout: Mutex<Duration>,
|
||||
/// Durée du bail du moteur (renouvelé au tiers).
|
||||
pub lease_ttl: Duration,
|
||||
/// Niveaux de raisonnement appris quand la façade ne tient pas la
|
||||
/// conversation (sinon, ceux de sa conversation).
|
||||
effort: EffortFallbacks,
|
||||
client: reqwest::Client,
|
||||
/// Le service du moteur (PID, pour l'indication « possible thrash »).
|
||||
engine: crate::engine::Paths,
|
||||
}
|
||||
|
||||
impl Bench {
|
||||
pub fn new(engine: crate::engine::Paths) -> Self {
|
||||
Bench {
|
||||
home: engine.home.clone(),
|
||||
state: Mutex::new(JobState::default()),
|
||||
req_timeout: Mutex::new(super::REQ_TIMEOUT),
|
||||
lease_ttl: Duration::from_secs(30),
|
||||
effort: EffortFallbacks::new(),
|
||||
client: {
|
||||
crate::chat::engine::install_crypto();
|
||||
reqwest::Client::builder().build().unwrap_or_default()
|
||||
},
|
||||
engine,
|
||||
}
|
||||
}
|
||||
|
||||
fn lock(&self) -> std::sync::MutexGuard<'_, JobState> {
|
||||
self.state.lock().unwrap_or_else(|e| e.into_inner())
|
||||
}
|
||||
|
||||
/// `benchRunning`.
|
||||
pub fn running(&self) -> bool {
|
||||
self.lock().running
|
||||
}
|
||||
|
||||
/// `benchCancel` : false si aucun ne tournait.
|
||||
pub fn cancel(&self) -> bool {
|
||||
let st = self.lock();
|
||||
match (&st.cancel, st.running) {
|
||||
(Some(c), true) => {
|
||||
let _ = c.send(true);
|
||||
true
|
||||
}
|
||||
_ => false,
|
||||
}
|
||||
}
|
||||
|
||||
/// `benchStatus`.
|
||||
pub fn status(&self) -> Value {
|
||||
let st = self.lock();
|
||||
let mut out = json!({ "ok": true, "running": st.running, "id": st.id, "mode": st.mode });
|
||||
if st.id.is_empty() {
|
||||
return out;
|
||||
}
|
||||
let o = out.as_object_mut().expect("objet");
|
||||
o.insert("phase".into(), json!(st.phase));
|
||||
o.insert("step".into(), json!(st.step));
|
||||
o.insert("steps".into(), json!(st.steps));
|
||||
let end = if st.running {
|
||||
Some(Instant::now())
|
||||
} else {
|
||||
st.finished
|
||||
};
|
||||
let elapsed = match (st.started, end) {
|
||||
(Some(s), Some(e)) => e.saturating_duration_since(s).as_secs_f64(),
|
||||
_ => 0.0,
|
||||
};
|
||||
o.insert("elapsed_sec".into(), json!(elapsed));
|
||||
if !st.running {
|
||||
if let Some(r) = &st.result {
|
||||
o.insert(
|
||||
"result".into(),
|
||||
serde_json::to_value(r).unwrap_or(Value::Null),
|
||||
);
|
||||
o.insert("saved".into(), json!(savable(r)));
|
||||
}
|
||||
if !st.err.is_empty() {
|
||||
o.insert("error".into(), json!(st.err));
|
||||
o.insert("canceled".into(), json!(st.canceled));
|
||||
}
|
||||
}
|
||||
out
|
||||
}
|
||||
|
||||
/// `benchStart` : le statut HTTP et la réponse.
|
||||
pub async fn start(self: &Arc<Self>, ctx: &Ctx, opts: Opts) -> (StatusCode, Value) {
|
||||
let store = &ctx.store;
|
||||
let fail = |code: StatusCode, e: &str| (code, json!({ "ok": false, "error": e }));
|
||||
let cfg = store.config().await;
|
||||
if crate::engstatus::is_external(&cfg) {
|
||||
return fail(
|
||||
StatusCode::BAD_REQUEST,
|
||||
"benchmark indisponible : le preset actif est une API externe",
|
||||
);
|
||||
}
|
||||
if let Some(e) = crate::backends::tune_busy(store).await {
|
||||
return fail(StatusCode::CONFLICT, &e);
|
||||
}
|
||||
if !crate::engstatus::health_check(&cfg).await {
|
||||
return fail(
|
||||
StatusCode::SERVICE_UNAVAILABLE,
|
||||
crate::tasks::ERR_MODEL_LOADING,
|
||||
);
|
||||
}
|
||||
{
|
||||
let mut st = self.lock();
|
||||
if st.running || st.starting {
|
||||
return fail(StatusCode::CONFLICT, "un benchmark tourne déjà");
|
||||
}
|
||||
st.starting = true;
|
||||
}
|
||||
let lease = match EngineLease::acquire_for(store.clone(), self.lease_ttl, "bench").await {
|
||||
Ok(l) => l,
|
||||
Err(e) => {
|
||||
self.lock().starting = false;
|
||||
return fail(StatusCode::CONFLICT, &e.to_string());
|
||||
}
|
||||
};
|
||||
let (tx, rx) = watch::channel(false);
|
||||
let id = {
|
||||
let mut st = self.lock();
|
||||
st.seq += 1;
|
||||
let id = format!("b{}-{}", unix_secs(), st.seq);
|
||||
*st = JobState {
|
||||
seq: st.seq,
|
||||
running: true,
|
||||
starting: false,
|
||||
id: id.clone(),
|
||||
mode: opts.mode.clone(),
|
||||
phase: "démarrage".into(),
|
||||
step: 0,
|
||||
steps: 0,
|
||||
started: Some(Instant::now()),
|
||||
finished: None,
|
||||
cancel: Some(tx),
|
||||
result: None,
|
||||
err: String::new(),
|
||||
canceled: false,
|
||||
};
|
||||
id
|
||||
};
|
||||
let (me, ctx2, id2) = (self.clone(), ctx.clone(), id.clone());
|
||||
tokio::spawn(async move { me.job(ctx2, id2, opts, rx, lease).await });
|
||||
(StatusCode::ACCEPTED, json!({ "ok": true, "id": id }))
|
||||
}
|
||||
|
||||
/// Le bench en tâche de fond : le verrou est rendu AVANT d'annoncer la
|
||||
/// fin — l'interface qui lit « terminé » et envoie aussitôt un message ne
|
||||
/// doit pas se voir refuser « benchmark en cours ».
|
||||
async fn job(
|
||||
self: Arc<Self>,
|
||||
ctx: Ctx,
|
||||
id: String,
|
||||
opts: Opts,
|
||||
rx: watch::Receiver<bool>,
|
||||
mut lease: EngineLease,
|
||||
) {
|
||||
let me = self.clone();
|
||||
let id2 = id.clone();
|
||||
let progress = move |phase: &str, step: i64, steps: i64| {
|
||||
let mut st = me.lock();
|
||||
if st.id == id2 {
|
||||
st.phase = phase.to_string();
|
||||
st.step = step;
|
||||
st.steps = steps;
|
||||
}
|
||||
};
|
||||
let run = self.run_bench(&ctx, opts, rx.clone(), &progress);
|
||||
tokio::pin!(run);
|
||||
let mut interrupted = false;
|
||||
let res = loop {
|
||||
tokio::select! {
|
||||
r = &mut run => break r,
|
||||
_ = lease.interrupted(), if !interrupted => {
|
||||
// Stop du chat, Reset ou bail perdu côté Go : le bench
|
||||
// s'arrête (le verrou n'est plus, ou ne sera plus, à lui).
|
||||
interrupted = true;
|
||||
if let Some(c) = &self.lock().cancel {
|
||||
let _ = c.send(true);
|
||||
}
|
||||
}
|
||||
}
|
||||
};
|
||||
let canceled = *rx.borrow();
|
||||
lease.release().await;
|
||||
let mut st = self.lock();
|
||||
if st.id != id {
|
||||
return;
|
||||
}
|
||||
st.running = false;
|
||||
st.finished = Some(Instant::now());
|
||||
st.cancel = None;
|
||||
match res {
|
||||
Ok(r) => st.result = Some(r),
|
||||
Err(e) => {
|
||||
st.result = None;
|
||||
st.canceled = canceled;
|
||||
st.err = if canceled {
|
||||
"benchmark annulé".into()
|
||||
} else {
|
||||
e
|
||||
};
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// `runBench` : mesure le moteur local et enregistre le résultat s'il
|
||||
/// est complet.
|
||||
async fn run_bench(
|
||||
&self,
|
||||
ctx: &Ctx,
|
||||
opts: Opts,
|
||||
cancel: watch::Receiver<bool>,
|
||||
progress: &(dyn Fn(&str, i64, i64) + Send + Sync),
|
||||
) -> Result<BenchResult, String> {
|
||||
let store = &ctx.store;
|
||||
let cfg = store.config().await;
|
||||
if crate::engstatus::is_external(&cfg) {
|
||||
return Err("benchmark indisponible : le preset actif est une API externe".into());
|
||||
}
|
||||
if crate::strata::is_strata(&cfg) {
|
||||
return Err("benchmark indisponible : le preset actif tourne sur Strata (le débit s'affiche sous chaque réponse)".into());
|
||||
}
|
||||
if let Some(e) = crate::backends::tune_busy(store).await {
|
||||
return Err(e);
|
||||
}
|
||||
let port = crate::engstatus::llm_port_raw(&cfg);
|
||||
if !crate::engstatus::health_check(&cfg).await {
|
||||
return Err(format!("serveur injoignable sur :{port}"));
|
||||
}
|
||||
let arg_env: HashMap<String, String> = super::ARG_ENV
|
||||
.iter()
|
||||
.filter_map(|k| {
|
||||
std::env::var(k)
|
||||
.ok()
|
||||
.filter(|v| !v.is_empty())
|
||||
.map(|v| (k.to_string(), v))
|
||||
})
|
||||
.collect();
|
||||
let effort: &EffortFallbacks = match &ctx.chat {
|
||||
Some(c) => &c.shared.effort,
|
||||
None => &self.effort,
|
||||
};
|
||||
let mut setup: Setup = super::setup_from(&cfg, &arg_env, effort);
|
||||
setup.hybrid = model_hybrid(ctx, &cfg).await;
|
||||
let route = store
|
||||
.internal_json(
|
||||
Method::GET,
|
||||
"/internal/v1/oai/route?method=POST&path=%2Fv1%2Fchat%2Fcompletions",
|
||||
None,
|
||||
)
|
||||
.await
|
||||
.unwrap_or(Value::Null);
|
||||
setup.side_slot = route.get("side_live").and_then(Value::as_bool) == Some(true);
|
||||
let served = Arc::new(AtomicU64::new(0));
|
||||
let engine = Engine {
|
||||
base: format!("http://localhost:{port}"),
|
||||
key: route
|
||||
.get("key")
|
||||
.and_then(Value::as_str)
|
||||
.unwrap_or("")
|
||||
.to_string(),
|
||||
client: self.client.clone(),
|
||||
cancel,
|
||||
req_timeout: *self.req_timeout.lock().unwrap_or_else(|e| e.into_inner()),
|
||||
served: served.clone(),
|
||||
};
|
||||
let cfg2 = cfg.clone();
|
||||
let learn = move |want: &str, got: &str| {
|
||||
effort.remember(&cfg2, want, got);
|
||||
log_effort_fallback(want, got);
|
||||
};
|
||||
let paths = self.engine.clone();
|
||||
let io = move || engine_io_sample(&paths);
|
||||
// Le bench prend le slot de la conversation : une fois fini, le Go
|
||||
// l'efface si c'est sans risque (engineSideJob).
|
||||
let side = store
|
||||
.internal_post("/internal/v1/engine/sidejob", &json!({}))
|
||||
.await
|
||||
.ok()
|
||||
.and_then(|(_, v)| v.get("id").and_then(Value::as_i64));
|
||||
let res = super::run(
|
||||
&engine,
|
||||
opts,
|
||||
&mut setup,
|
||||
&Corpora::default_set(),
|
||||
progress,
|
||||
&learn,
|
||||
&io,
|
||||
)
|
||||
.await;
|
||||
if let Some(id) = side {
|
||||
let _ = store
|
||||
.internal_post(
|
||||
"/internal/v1/engine/sidejob",
|
||||
&json!({ "end": id, "served": served.load(Ordering::SeqCst) > 0 }),
|
||||
)
|
||||
.await;
|
||||
}
|
||||
let res = res?;
|
||||
let _ = store
|
||||
.internal_post(
|
||||
"/internal/v1/bench/perf",
|
||||
&json!({
|
||||
"prompt_n": res.prompt_n,
|
||||
"prompt_ms": res.prompt_ms,
|
||||
"prompt_per_second": res.prompt_per_second,
|
||||
"predicted_n": res.predicted_n,
|
||||
"predicted_per_second": res.predicted_per_second,
|
||||
}),
|
||||
)
|
||||
.await;
|
||||
if savable(&res) {
|
||||
save_last(ctx, &res, &cfg).await;
|
||||
self.save_for_active_preset(ctx, &res, &cfg).await;
|
||||
}
|
||||
Ok(res)
|
||||
}
|
||||
|
||||
/// `saveBenchForActivePreset` : sous l'id du preset actif ; sans effet si
|
||||
/// aucun ne correspond.
|
||||
async fn save_for_active_preset(&self, ctx: &Ctx, res: &BenchResult, cfg: &Cfg) {
|
||||
let store = &ctx.store;
|
||||
let order = crate::presets::parse_preset_order(
|
||||
store
|
||||
.get(BK_STATE, crate::presets::PRESET_ORDER_KEY)
|
||||
.await
|
||||
.ok()
|
||||
.flatten()
|
||||
.as_deref(),
|
||||
);
|
||||
let Ok(list) = crate::presets::list_presets(&self.home, cfg, &order) else {
|
||||
return;
|
||||
};
|
||||
let Some(id) = list.into_iter().find(|p| p.active).map(|p| p.id) else {
|
||||
return;
|
||||
};
|
||||
let mut m = load_bench_store(ctx).await;
|
||||
m.insert(id, saved_bench(res, cfg));
|
||||
let _ = store
|
||||
.put(BK_STATE, PRESETS_KEY, gojson::marshal(&m).into_bytes())
|
||||
.await;
|
||||
}
|
||||
}
|
||||
|
||||
fn unix_secs() -> i64 {
|
||||
std::time::SystemTime::now()
|
||||
.duration_since(std::time::UNIX_EPOCH)
|
||||
.map_or(0, |d| d.as_secs() as i64)
|
||||
}
|
||||
|
||||
/// `benchSavable` : seul un résultat COMPLET est enregistré.
|
||||
pub fn savable(res: &BenchResult) -> bool {
|
||||
res.depth.as_ref().is_none_or(|d| d.partial.is_empty())
|
||||
}
|
||||
|
||||
/// `savedBench`.
|
||||
#[derive(Clone, Debug, Default, PartialEq, Serialize, Deserialize)]
|
||||
pub struct Saved {
|
||||
#[serde(default)]
|
||||
pub result: BenchResult,
|
||||
#[serde(default)]
|
||||
pub model: String,
|
||||
#[serde(default)]
|
||||
pub at: i64,
|
||||
#[serde(default, skip_serializing_if = "String::is_empty")]
|
||||
pub fingerprint: String,
|
||||
}
|
||||
|
||||
/// `newSavedBench` : l'indication du moment (thrash) n'est pas enregistrée.
|
||||
pub fn saved_bench(res: &BenchResult, cfg: &Cfg) -> Saved {
|
||||
let mut r = res.clone();
|
||||
if let Some(d) = &mut r.depth {
|
||||
d.hint.clear();
|
||||
}
|
||||
Saved {
|
||||
result: r,
|
||||
model: crate::paths::go_base(cfg.get("MODEL").map(String::as_str).unwrap_or("")),
|
||||
at: unix_secs(),
|
||||
fingerprint: crate::presets_api::bench_fingerprint(cfg, cfg),
|
||||
}
|
||||
}
|
||||
|
||||
/// `saveLastBench`.
|
||||
async fn save_last(ctx: &Ctx, res: &BenchResult, cfg: &Cfg) {
|
||||
let _ = ctx
|
||||
.store
|
||||
.put(
|
||||
BK_STATE,
|
||||
LAST_KEY,
|
||||
gojson::marshal(&saved_bench(res, cfg)).into_bytes(),
|
||||
)
|
||||
.await;
|
||||
}
|
||||
|
||||
/// `loadBenchStore` : les mesures par preset (vide s'il n'y en a pas).
|
||||
async fn load_bench_store(ctx: &Ctx) -> BTreeMap<String, Saved> {
|
||||
ctx.store
|
||||
.get(BK_STATE, PRESETS_KEY)
|
||||
.await
|
||||
.ok()
|
||||
.flatten()
|
||||
.and_then(|b| serde_json::from_slice::<Option<BTreeMap<String, Saved>>>(&b).ok())
|
||||
.flatten()
|
||||
.unwrap_or_default()
|
||||
}
|
||||
|
||||
/// `loadLastBench`.
|
||||
async fn load_last(ctx: &Ctx) -> Option<Saved> {
|
||||
let b = ctx.store.get(BK_STATE, LAST_KEY).await.ok().flatten()?;
|
||||
serde_json::from_slice(&b).ok()
|
||||
}
|
||||
|
||||
/// Le modèle du preset est-il hybride (reprise du cache aux points de
|
||||
/// contrôle) ?
|
||||
async fn model_hybrid(ctx: &Ctx, cfg: &Cfg) -> bool {
|
||||
let m = cfg.get("MODEL").map(|s| s.trim()).unwrap_or("");
|
||||
if m.is_empty() {
|
||||
return false;
|
||||
}
|
||||
let stored = crate::gguf::parse_stored_model_dirs(
|
||||
ctx.store
|
||||
.get(BK_STATE, crate::gguf::STATE_MODEL_DIRS_KEY)
|
||||
.await
|
||||
.ok()
|
||||
.flatten()
|
||||
.as_deref(),
|
||||
);
|
||||
let env = std::env::var(crate::gguf::MODEL_DIRS_ENV).ok();
|
||||
let Ok(p) = crate::gguf::resolve_serve_model_path(m, &ctx.engine.home, &stored, env.as_deref())
|
||||
else {
|
||||
return false;
|
||||
};
|
||||
match crate::gguf::gguf_meta(&p) {
|
||||
Ok(g) => crate::serve::ckpt::gguf_hybrid(Some(&g)),
|
||||
Err(_) => false,
|
||||
}
|
||||
}
|
||||
|
||||
/// `engineIOSample` : octets lus du disque et RSS du moteur ; `None` si le
|
||||
/// PID ou /proc est illisible.
|
||||
fn engine_io_sample(paths: &crate::engine::Paths) -> Option<(i64, i64)> {
|
||||
let pid = crate::engine::read_pid(&paths.pid());
|
||||
if pid <= 0 {
|
||||
return None;
|
||||
}
|
||||
let io = std::fs::read_to_string(format!("/proc/{pid}/io")).ok()?;
|
||||
for line in io.split('\n') {
|
||||
if let Some(v) = line.strip_prefix("read_bytes:") {
|
||||
let n = v.trim().parse::<i64>().ok()?;
|
||||
let rss = std::fs::read_to_string(format!("/proc/{pid}/status"))
|
||||
.map(|s| crate::engstatus::parse_vm_rss(&s))
|
||||
.unwrap_or(0);
|
||||
return Some((n, rss));
|
||||
}
|
||||
}
|
||||
None
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Routes
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/// Les routes servies par ce module.
|
||||
pub fn handles(path: &str) -> bool {
|
||||
matches!(
|
||||
path,
|
||||
"/api/bench" | "/api/bench/status" | "/api/bench/cancel" | "/api/bench/last"
|
||||
)
|
||||
}
|
||||
|
||||
/// `postOnly`.
|
||||
fn post_only(method: &Method) -> Option<Response<Body>> {
|
||||
if method == Method::POST {
|
||||
return None;
|
||||
}
|
||||
let mut r = send_json(
|
||||
StatusCode::METHOD_NOT_ALLOWED,
|
||||
&json!({ "ok": false, "error": format!("méthode {method} refusée : POST attendu") }),
|
||||
);
|
||||
r.headers_mut().insert(
|
||||
hyper::header::ALLOW,
|
||||
hyper::header::HeaderValue::from_static("POST"),
|
||||
);
|
||||
Some(r)
|
||||
}
|
||||
|
||||
/// `from_go` : remise par le Go (tunnel), déjà authentifiée par lui.
|
||||
pub async fn serve(
|
||||
ctx: &Ctx,
|
||||
bench: &Arc<Bench>,
|
||||
req: Request<Incoming>,
|
||||
from_go: bool,
|
||||
) -> Response<Body> {
|
||||
let (parts, body) = req.into_parts();
|
||||
if !from_go && let Err(refus) = ctx.guard(&parts.headers).await {
|
||||
return *refus;
|
||||
}
|
||||
match parts.uri.path() {
|
||||
"/api/bench" => {
|
||||
if let Some(r) = post_only(&parts.method) {
|
||||
return r;
|
||||
}
|
||||
let raw = Limited::new(body, 1 << 20)
|
||||
.collect()
|
||||
.await
|
||||
.map(|b| b.to_bytes().to_vec())
|
||||
.unwrap_or_default();
|
||||
// Les erreurs de décodage sont ignorées (champs facultatifs).
|
||||
let d = gojson::decode_struct(
|
||||
&raw,
|
||||
BENCH_REQ,
|
||||
&[("mode", Kind::Str), ("prompt", Kind::Int), ("n", Kind::Int)],
|
||||
);
|
||||
let mut opts = Opts {
|
||||
mode: super::MODE_QUICK.into(),
|
||||
prompt: 2000,
|
||||
predict: 300,
|
||||
depth: 0,
|
||||
};
|
||||
if d.str("mode").trim().eq_ignore_ascii_case(super::MODE_FULL) {
|
||||
opts.mode = super::MODE_FULL.into();
|
||||
}
|
||||
if d.int("prompt") > 0 {
|
||||
opts.prompt = d.int("prompt").min(32768);
|
||||
}
|
||||
if d.int("n") > 0 {
|
||||
opts.predict = d.int("n").min(4096);
|
||||
}
|
||||
let (code, out) = bench.start(ctx, opts).await;
|
||||
send_json(code, &out)
|
||||
}
|
||||
"/api/bench/status" => send_json(StatusCode::OK, &bench.status()),
|
||||
"/api/bench/cancel" => {
|
||||
if let Some(r) = post_only(&parts.method) {
|
||||
return r;
|
||||
}
|
||||
send_json(StatusCode::OK, &json!({ "ok": bench.cancel() }))
|
||||
}
|
||||
"/api/bench/last" => match load_last(ctx).await {
|
||||
None => send_json(StatusCode::OK, &json!({ "ok": false })),
|
||||
Some(sb) => send_json(
|
||||
StatusCode::OK,
|
||||
&json!({ "ok": true, "result": sb.result, "model": sb.model, "at": sb.at }),
|
||||
),
|
||||
},
|
||||
_ => send_json(
|
||||
StatusCode::NOT_FOUND,
|
||||
&json!({ "ok": false, "error": "route inconnue" }),
|
||||
),
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[cfg(target_os = "linux")]
|
||||
#[test]
|
||||
fn lecture_du_moteur() {
|
||||
let tmp = tempfile::tempdir().unwrap();
|
||||
let paths = crate::engine::Paths {
|
||||
home: tmp.path().to_path_buf(),
|
||||
service: "loki-engine".into(),
|
||||
};
|
||||
assert_eq!(engine_io_sample(&paths), None, "sans PID");
|
||||
std::fs::create_dir_all(paths.pid().parent().unwrap()).unwrap();
|
||||
std::fs::write(paths.pid(), "0\n").unwrap();
|
||||
assert_eq!(engine_io_sample(&paths), None, "PID nul");
|
||||
std::fs::write(paths.pid(), format!("{}\n", std::process::id())).unwrap();
|
||||
let (read, rss) = engine_io_sample(&paths).expect("ce processus");
|
||||
assert!(read >= 0 && rss > 0, "{read} {rss}");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn mesure_enregistree() {
|
||||
let cfg: Cfg = [("MODEL", "/models/Q.gguf"), ("CTX", "32768")]
|
||||
.iter()
|
||||
.map(|(k, v)| (k.to_string(), v.to_string()))
|
||||
.collect();
|
||||
let res = BenchResult {
|
||||
prompt_per_second: 1.0,
|
||||
depth: Some(super::super::Depth {
|
||||
hint: "possible thrash".into(),
|
||||
..Default::default()
|
||||
}),
|
||||
..Default::default()
|
||||
};
|
||||
let sb = saved_bench(&res, &cfg);
|
||||
assert_eq!(sb.model, "Q.gguf");
|
||||
assert!(!sb.fingerprint.is_empty());
|
||||
assert_eq!(
|
||||
sb.fingerprint,
|
||||
crate::presets_api::bench_fingerprint(&cfg, &cfg)
|
||||
);
|
||||
assert_eq!(
|
||||
sb.result.depth.as_ref().unwrap().hint,
|
||||
"",
|
||||
"l'indication ne s'enregistre pas"
|
||||
);
|
||||
assert_eq!(res.depth.as_ref().unwrap().hint, "possible thrash");
|
||||
assert!(sb.at > 0);
|
||||
assert_eq!(saved_bench(&res, &Cfg::new()).model, ".");
|
||||
// Seul un résultat complet s'enregistre.
|
||||
assert!(savable(&BenchResult::default()));
|
||||
assert!(savable(&res));
|
||||
let partial = BenchResult {
|
||||
depth: Some(super::super::Depth {
|
||||
partial: "tour 1 interrompu".into(),
|
||||
..Default::default()
|
||||
}),
|
||||
..Default::default()
|
||||
};
|
||||
assert!(!savable(&partial));
|
||||
}
|
||||
}
|
||||
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,344 @@
|
||||
//! Tests du bench : ceux du Go (`llm_bench_test.go`), portés, et les bornes.
|
||||
|
||||
use super::*;
|
||||
|
||||
fn cfg(pairs: &[(&str, &str)]) -> Cfg {
|
||||
pairs
|
||||
.iter()
|
||||
.map(|(k, v)| (k.to_string(), v.to_string()))
|
||||
.collect()
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn profondeur_choisie() {
|
||||
let warm = 2 * 512 + WARM_GEN;
|
||||
let turns = TURNS * (APPEND_TOKENS + GEN_TOKENS);
|
||||
for (ctx, cpu, want, skipped, what) in [
|
||||
(131072, false, DEPTH_MAX, false, "grand contexte : plafond"),
|
||||
(
|
||||
131072,
|
||||
true,
|
||||
DEPTH_CPU,
|
||||
false,
|
||||
"poids sur CPU : plafond réduit",
|
||||
),
|
||||
(65536, false, 32768, false, "moitié du contexte"),
|
||||
(
|
||||
16384,
|
||||
false,
|
||||
16384 - turns - GEN_TOKENS - warm - MARGIN,
|
||||
false,
|
||||
"les tours doivent tenir",
|
||||
),
|
||||
(8192, false, 0, true, "trop petit"),
|
||||
(0, false, 0, true, "inconnu"),
|
||||
] {
|
||||
let (got, why) = depth_for(ctx, cpu, warm);
|
||||
assert_eq!((got, !why.is_empty()), (want, skipped), "{what}");
|
||||
if got > 0 {
|
||||
assert!(got + turns + GEN_TOKENS <= ctx, "{what} : déborde");
|
||||
}
|
||||
}
|
||||
assert_eq!(depth_for(0, false, 0).1, "taille de contexte inconnue");
|
||||
assert_eq!(
|
||||
depth_for(8192, false, warm).1,
|
||||
"contexte de 8192 jetons trop petit pour la mesure en profondeur"
|
||||
);
|
||||
// Juste au seuil : 4096 tient, 4095 non.
|
||||
let at = |d: i64| d + turns + GEN_TOKENS + warm + MARGIN;
|
||||
assert_eq!(depth_for(at(4096), false, warm).0, 4096);
|
||||
assert_eq!(depth_for(at(4095), false, warm).0, 0);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn types_de_cache() {
|
||||
for (k, v, label, quant) in [
|
||||
("", "", "f16", false),
|
||||
("f16", "", "f16", false),
|
||||
("bf16", "bf16", "bf16", false),
|
||||
("q8_0", "q8_0", "q8_0", true),
|
||||
("q8_0", "", "q8_0/f16", true),
|
||||
("", "q4_0", "f16/q4_0", true),
|
||||
(" Q8_0 ", "q8_0", "q8_0", true),
|
||||
] {
|
||||
assert_eq!(kv_label(k, v), (label.to_string(), quant), "{k:?} {v:?}");
|
||||
}
|
||||
// Le -ctk d'EXTRA_ARGS l'emporte, comme au lancement.
|
||||
let s = setup_from(
|
||||
&cfg(&[
|
||||
("KV_TYPE", "f16"),
|
||||
("EXTRA_ARGS", "-ctk q8_0 -ctv q8_0 --n-cpu-moe 20"),
|
||||
]),
|
||||
&HashMap::new(),
|
||||
&EffortFallbacks::new(),
|
||||
);
|
||||
assert_eq!(
|
||||
(s.kv.as_str(), s.kv_quant, s.cpu_placed),
|
||||
("q8_0", true, true)
|
||||
);
|
||||
let s = setup_from(&cfg(&[]), &HashMap::new(), &EffortFallbacks::new());
|
||||
assert!(!s.cpu_placed);
|
||||
// Placement par l'environnement de llama-server.
|
||||
let env = HashMap::from([("LLAMA_ARG_CPU_MOE".to_string(), "1".to_string())]);
|
||||
let s = setup_from(&cfg(&[]), &env, &EffortFallbacks::new());
|
||||
assert!(s.cpu_placed);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn micro_lot_du_preset() {
|
||||
let ub = |pairs: &[(&str, &str)]| {
|
||||
setup_from(&cfg(pairs), &HashMap::new(), &EffortFallbacks::new()).ub
|
||||
};
|
||||
assert_eq!(ub(&[]), 512);
|
||||
assert_eq!(ub(&[("UBATCH", "1024")]), 1024);
|
||||
assert_eq!(ub(&[("UBATCH", " 2048 ")]), 2048);
|
||||
assert_eq!(ub(&[("UBATCH", "0")]), 512);
|
||||
assert_eq!(ub(&[("UBATCH", "x")]), 512);
|
||||
assert_eq!(ub(&[("EXTRA_ARGS", "-ub 256"), ("UBATCH", "1024")]), 256);
|
||||
assert_eq!(ub(&[("EXTRA_ARGS", "--ubatch-size 128")]), 128);
|
||||
assert_eq!(ub(&[("EXTRA_ARGS", "-ub -4"), ("UBATCH", "64")]), 64);
|
||||
let gpus = setup_from(
|
||||
&cfg(&[("CUDA_VISIBLE_DEVICES", " 0,1 ")]),
|
||||
&HashMap::new(),
|
||||
&EffortFallbacks::new(),
|
||||
)
|
||||
.gpus;
|
||||
assert_eq!(gpus, "0,1");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn requete_du_bench() {
|
||||
let msgs = vec![("user".to_string(), "x".to_string())];
|
||||
let fb = EffortFallbacks::new();
|
||||
let p = setup_from(
|
||||
&cfg(&[("TEMP", "0.6"), ("TOP_P", "0.95"), ("TOP_K", "20")]),
|
||||
&HashMap::new(),
|
||||
&fb,
|
||||
)
|
||||
.payload(&msgs, 10, true);
|
||||
assert_eq!(p["temperature"], json!(0.6));
|
||||
assert_eq!(p["top_p"], json!(0.95));
|
||||
assert_eq!(p["top_k"], json!(20));
|
||||
assert_eq!(p["seed"], json!(SEED));
|
||||
assert_eq!(p["max_tokens"], json!(10));
|
||||
assert_eq!(p["model"], json!("loki"));
|
||||
assert_eq!(p["messages"], json!([{ "role": "user", "content": "x" }]));
|
||||
assert_eq!(
|
||||
(p["cache_prompt"].clone(), p["stream"].clone()),
|
||||
(json!(true), json!(false))
|
||||
);
|
||||
for k in [
|
||||
"chat_template_kwargs",
|
||||
"reasoning_effort",
|
||||
"ignore_eos",
|
||||
"id_slot",
|
||||
] {
|
||||
assert!(p.get(k).is_none(), "{k}");
|
||||
}
|
||||
let s = setup_from(&cfg(&[]), &HashMap::new(), &fb);
|
||||
let p = s.payload(&msgs, 10, false);
|
||||
assert_eq!(p["temperature"], json!(0.7));
|
||||
assert_eq!(p["cache_prompt"], json!(false));
|
||||
assert!(p.get("top_k").is_none());
|
||||
let mut s = setup_from(&cfg(&[("REASONING_EFFORT", "high")]), &HashMap::new(), &fb);
|
||||
assert_eq!(s.payload(&msgs, 1, true)["reasoning_effort"], json!("high"));
|
||||
s.side_slot = true;
|
||||
assert_eq!(s.payload(&msgs, 1, true)["id_slot"], json!(1));
|
||||
// Raisonnement coupé : le gabarit le dit.
|
||||
let s = setup_from(&cfg(&[("REASONING", "off")]), &HashMap::new(), &fb);
|
||||
assert_eq!(
|
||||
s.payload(&msgs, 1, true)["chat_template_kwargs"],
|
||||
json!({ "enable_thinking": false })
|
||||
);
|
||||
let s = setup_from(&cfg(&[("REASONING_EFFORT", "none")]), &HashMap::new(), &fb);
|
||||
assert!(s.off);
|
||||
// Une traduction apprise est envoyée d'emblée.
|
||||
let c = cfg(&[("REASONING_EFFORT", "high"), ("MODEL", "/m/Q.gguf")]);
|
||||
fb.remember(&c, "high", "xhigh");
|
||||
let s = setup_from(&c, &HashMap::new(), &fb);
|
||||
assert_eq!((s.want.as_str(), s.effort.as_str()), ("high", "xhigh"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn morceaux_du_corpus() {
|
||||
let text = "ééé ligne\n".repeat(100);
|
||||
let mut off = 0;
|
||||
while off < text.len() {
|
||||
let (chunk, next) = slice(&text, off, 37);
|
||||
assert!(!chunk.is_empty(), "à {off}");
|
||||
if next < text.len() {
|
||||
assert!(chunk.ends_with('\n'), "coupé hors fin de ligne : {chunk:?}");
|
||||
}
|
||||
off = next;
|
||||
}
|
||||
assert_eq!(slice(&text, text.len(), 10).0, "");
|
||||
assert_eq!(slice(&text, 0, 0).0, "");
|
||||
assert_eq!(slice("abc", 0, 99), ("abc", 3));
|
||||
// Sans fin de ligne assez loin : coupé à la taille, jamais dans un
|
||||
// caractère.
|
||||
assert_eq!(slice("éééééé", 0, 3), ("é", 2));
|
||||
assert_eq!(slice("ab\ncdefgh", 0, 8), ("ab\ncdefg", 8));
|
||||
assert_eq!(slice("abcde\nfgh", 0, 8), ("abcde\n", 6));
|
||||
assert_eq!(slice("abcd\nfgh", 0, 8), ("abcd\nfgh", 8));
|
||||
assert_eq!(line_start("ab\ncd", 1), 3);
|
||||
assert_eq!(line_start("ab\ncd", 3), 3);
|
||||
assert_eq!(line_start("ab\ncd", 0), 0);
|
||||
assert_eq!(line_start("ab\ncd", -5), 0);
|
||||
assert_eq!(line_start("ab\ncd", 4), 5);
|
||||
assert_eq!(line_start("ab\ncd", 9), 5);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn reprise_du_cache() {
|
||||
let t = |cached, expected| Turn {
|
||||
cached,
|
||||
expected,
|
||||
..Default::default()
|
||||
};
|
||||
let (r, note) = reuse(&[t(1000, 1004), t(3000, 3000)], false);
|
||||
assert!(r > 0.99 && note.is_empty(), "{r} {note}");
|
||||
assert_eq!(reuse(&[t(1000, 1016)], false).1, "");
|
||||
assert!(!reuse(&[t(1000, 1017)], false).1.is_empty());
|
||||
let part = [t(512, 3000)];
|
||||
assert!(reuse(&part, true).1.contains("hybride"));
|
||||
assert!(reuse(&part, false).1.contains("gabarit"));
|
||||
assert_eq!(reuse(&part, false).0, 512.0 / 3000.0);
|
||||
let (r, note) = reuse(&[t(-1, 3000)], false);
|
||||
assert_eq!(r, -1.0);
|
||||
assert!(!note.is_empty());
|
||||
assert_eq!(reuse(&[t(10, 0)], false).0, -1.0);
|
||||
// Un cache plus grand que l'attendu compte pour l'attendu.
|
||||
assert_eq!(reuse(&[t(5000, 3000)], false).0, 1.0);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn relectures_du_disque() {
|
||||
assert_eq!(thrash_hint(100 << 20, 256, 0), "");
|
||||
assert_eq!(thrash_hint(0, 256, 0), "");
|
||||
assert_eq!(thrash_hint(1 << 40, 0, 0), "");
|
||||
assert_eq!(thrash_hint(512 << 20, 256, 0), "");
|
||||
let h = thrash_hint(3 << 30, 768, 60 << 30);
|
||||
assert_eq!(
|
||||
h,
|
||||
"possible thrash : 1.0 Go relus du disque par 256 jetons générés (RSS 60.0 Go) — le modèle ne tient sans doute pas en RAM"
|
||||
);
|
||||
assert!(!thrash_hint((512 << 20) + 1, 256, 0).is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn brouillon() {
|
||||
assert_eq!(draft_text(None), "");
|
||||
assert_eq!(draft_text(Some(&Draft { n: 0, accepted: 0 })), "");
|
||||
assert_eq!(
|
||||
draft_text(Some(&Draft {
|
||||
n: 30,
|
||||
accepted: 21
|
||||
})),
|
||||
" brouillon accepté 70 % (21/30)"
|
||||
);
|
||||
let d = draft_add(None, Some(Draft { n: 10, accepted: 7 }));
|
||||
let d = draft_add(d, None);
|
||||
let d = draft_add(d, Some(Draft { n: 5, accepted: 1 }));
|
||||
assert_eq!(d, Some(Draft { n: 15, accepted: 8 }));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn durees_du_go() {
|
||||
let ms = Duration::from_millis;
|
||||
assert_eq!(go_dur(Duration::ZERO), "0s");
|
||||
assert_eq!(go_dur(Duration::from_nanos(7)), "7ns");
|
||||
assert_eq!(go_dur(Duration::from_nanos(1500)), "1.5µs");
|
||||
assert_eq!(go_dur(ms(150)), "150ms");
|
||||
assert_eq!(go_dur(Duration::from_micros(1500)), "1.5ms");
|
||||
assert_eq!(go_dur(ms(1000)), "1s");
|
||||
assert_eq!(go_dur(ms(1500)), "1.5s");
|
||||
assert_eq!(go_dur(Duration::from_secs(6 * 60)), "6m0s");
|
||||
assert_eq!(go_dur(Duration::from_secs(3600)), "1h0m0s");
|
||||
assert_eq!(go_dur(Duration::from_secs(3661)), "1h1m1s");
|
||||
assert_eq!(
|
||||
go_dur(Duration::from_nanos(61_000_000_001)),
|
||||
"1m1.000000001s"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn reponse_du_moteur() {
|
||||
let full = br#"{"choices":[{"message":{"content":"ok"}}],"usage":{"prompt_tokens":120,"prompt_tokens_details":{"cached_tokens":30}},"timings":{"prompt_n":100,"prompt_ms":50,"predicted_n":10,"predicted_ms":200,"draft_n":4,"draft_n_accepted":3}}"#;
|
||||
let r = parse_reply(full).unwrap();
|
||||
assert_eq!(r.turn.new, 100);
|
||||
assert_eq!(r.turn.prompt_per_second, 2000.0);
|
||||
assert_eq!(r.turn.predicted_per_second, 50.0);
|
||||
assert_eq!(r.turn.cached, 30);
|
||||
assert_eq!(r.turn.draft, Some(Draft { n: 4, accepted: 3 }));
|
||||
assert_eq!((r.total, r.content.as_str()), (120, "ok"));
|
||||
assert_eq!(r.prompt_total(), 120);
|
||||
// cache_n l'emporte ; débits donnés par le moteur gardés.
|
||||
let r = parse_reply(br#"{"timings":{"prompt_n":"8","prompt_ms":4,"predicted_n":2,"predicted_ms":0,"cache_n":5,"prompt_per_second":7,"draft_n":0,"draft_n_accepted":0}}"#).unwrap();
|
||||
assert_eq!((r.turn.cached, r.turn.prompt_per_second), (5, 7.0));
|
||||
assert_eq!(r.turn.predicted_per_second, 0.0);
|
||||
assert_eq!(r.turn.draft, None);
|
||||
assert_eq!(r.prompt_total(), 13);
|
||||
// Cache inconnu : -1, et le total sans lui.
|
||||
let r = parse_reply(
|
||||
br#"{"timings":{"prompt_n":8,"prompt_ms":4,"predicted_n":2,"predicted_ms":1}} trailing"#,
|
||||
)
|
||||
.unwrap();
|
||||
assert_eq!(
|
||||
(r.turn.cached, r.prompt_total(), r.content.as_str()),
|
||||
(-1, 8, "")
|
||||
);
|
||||
for bad in [
|
||||
&br#"{"timings":{"prompt_n":8,"prompt_ms":4,"predicted_n":2}}"#[..],
|
||||
br#"{"timings":{"prompt_n":-1,"prompt_ms":4,"predicted_n":2,"predicted_ms":1}}"#,
|
||||
br#"{"timings":"x"}"#,
|
||||
b"pas du json",
|
||||
br#"{"timings":{"prompt_n":1e13,"prompt_ms":4,"predicted_n":2,"predicted_ms":1}}"#,
|
||||
] {
|
||||
assert_eq!(
|
||||
parse_reply(bad),
|
||||
Err(ReqError::Other(ERR_NO_TIMINGS.into())),
|
||||
"{}",
|
||||
String::from_utf8_lossy(bad)
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn erreur_http() {
|
||||
let e = ReqError::Http {
|
||||
method: "POST".into(),
|
||||
path: "/v1/chat/completions".into(),
|
||||
status: "500 Internal Server Error".into(),
|
||||
body: format!(" {}é{} ", "x".repeat(299), "y".repeat(10)),
|
||||
};
|
||||
// 300 octets au plus, jamais au milieu d'un caractère.
|
||||
assert_eq!(
|
||||
e.to_string(),
|
||||
format!(
|
||||
"POST /v1/chat/completions : 500 Internal Server Error {}",
|
||||
"x".repeat(299)
|
||||
)
|
||||
);
|
||||
assert_eq!(go_op("POST"), "Post");
|
||||
assert_eq!(go_op("get"), "Get");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn flottants_du_go() {
|
||||
let t = Turn {
|
||||
prompt_ms: 3836.0,
|
||||
prompt_per_second: 1037.5,
|
||||
..Default::default()
|
||||
};
|
||||
let v = serde_json::to_string(&t).unwrap();
|
||||
assert!(v.contains(r#""prompt_ms":3836,"#), "{v}");
|
||||
assert!(v.contains(r#""prompt_per_second":1037.5"#), "{v}");
|
||||
let d = Depth {
|
||||
reuse: -1.0,
|
||||
..Default::default()
|
||||
};
|
||||
assert_eq!(
|
||||
serde_json::to_value(&d).unwrap(),
|
||||
json!({ "ctx": 0, "target": 0, "reuse": -1 })
|
||||
);
|
||||
}
|
||||
@@ -76,7 +76,17 @@ pub struct EngineLease {
|
||||
impl EngineLease {
|
||||
/// Prend le verrou de génération pour `ttl` (borné par le Go à 5 s – 5 min).
|
||||
pub async fn acquire(store: Store, ttl: Duration) -> Result<Self, LeaseError> {
|
||||
let body = json!({ "ttl_ms": ttl.as_millis() as u64 });
|
||||
Self::acquire_for(store, ttl, "").await
|
||||
}
|
||||
|
||||
/// Comme [`Self::acquire`], avec un motif : « bench » — le Go refuse alors
|
||||
/// les autres avec le message du benchmark, et changer de discussion ne
|
||||
/// rompt pas le bail.
|
||||
pub async fn acquire_for(store: Store, ttl: Duration, why: &str) -> Result<Self, LeaseError> {
|
||||
let mut body = json!({ "ttl_ms": ttl.as_millis() as u64 });
|
||||
if !why.is_empty() {
|
||||
body["why"] = json!(why);
|
||||
}
|
||||
let (status, v) = store
|
||||
.internal_post(PATH_LEASE, &body)
|
||||
.await
|
||||
|
||||
@@ -184,7 +184,7 @@ pub fn service_log_tail(paths: &Paths, n: usize) -> String {
|
||||
|
||||
/// Valeur brute de PORT comme `LLMPort` du Go (`strconv.Atoi`, > 0), sans
|
||||
/// borne haute : un port > 65535 doit faire échouer la sonde, pas viser 8080.
|
||||
fn llm_port_raw(cfg: &HashMap<String, String>) -> i64 {
|
||||
pub fn llm_port_raw(cfg: &HashMap<String, String>) -> i64 {
|
||||
if let Some(p) = cfg.get("PORT")
|
||||
&& let Ok(n) = p.parse::<i64>()
|
||||
&& n > 0
|
||||
|
||||
@@ -13,6 +13,7 @@
|
||||
pub mod api;
|
||||
pub mod auth;
|
||||
pub mod backends;
|
||||
pub mod bench;
|
||||
pub mod chat;
|
||||
pub mod chatapi;
|
||||
pub mod chatx;
|
||||
|
||||
@@ -183,6 +183,9 @@ fn serve() -> ExitCode {
|
||||
// LOKI_TASKS=rust : les tâches planifiées (routes, planificateur,
|
||||
// tâches script) ; une tâche IA reste exécutée par le Go.
|
||||
let tasks_rust = held_by_facade("LOKI_TASKS");
|
||||
// LOKI_BENCH=rust : le banc d'essai (routes, job, protocole) ; le
|
||||
// verrou de génération reste celui du Go, pris par le bail du moteur.
|
||||
let bench_rust = held_by_facade("LOKI_BENCH");
|
||||
let engine_rust = held_by_facade("LOKI_ENGINE")
|
||||
&& match loki_rs::enginectl::usable(
|
||||
std::env::var("LOKI_CONTAINER").ok().as_deref(),
|
||||
@@ -218,6 +221,7 @@ fn serve() -> ExitCode {
|
||||
("LOKI_ENGINE_IMAGE", engineimg_rust),
|
||||
("LOKI_STRATA", strata_rust),
|
||||
("LOKI_TASKS", tasks_rust),
|
||||
("LOKI_BENCH", bench_rust),
|
||||
];
|
||||
for (var, on) in held {
|
||||
if on {
|
||||
@@ -351,6 +355,8 @@ fn serve() -> ExitCode {
|
||||
});
|
||||
let tasks = tasks_rust
|
||||
.then(|| std::sync::Arc::new(loki_rs::tasks::Tasks::new(engine_paths.home.clone())));
|
||||
let bench = bench_rust
|
||||
.then(|| std::sync::Arc::new(loki_rs::bench::job::Bench::new(engine_paths.clone())));
|
||||
let cfg = proxy::Config {
|
||||
backend,
|
||||
internal_token: token,
|
||||
@@ -373,6 +379,7 @@ fn serve() -> ExitCode {
|
||||
engineimg,
|
||||
strata,
|
||||
tasks,
|
||||
bench,
|
||||
mcp: mcp_rust.then(|| std::sync::Arc::new(loki_rs::mcp::Mcp::new(mem_home))),
|
||||
};
|
||||
let res = proxy::serve(listener, cfg, shutdown_signal()).await;
|
||||
|
||||
@@ -106,6 +106,9 @@ pub struct Config {
|
||||
/// Tâches planifiées (routes, planificateur, tâches script) tenues par la
|
||||
/// façade (`LOKI_TASKS=rust`) ; `None` : au Go.
|
||||
pub tasks: Option<Arc<crate::tasks::Tasks>>,
|
||||
/// Banc d'essai (routes, job, protocole de mesure) tenu par la façade
|
||||
/// (`LOKI_BENCH=rust`) ; `None` : au Go.
|
||||
pub bench: Option<Arc<crate::bench::job::Bench>>,
|
||||
}
|
||||
|
||||
/// Arrête la tâche de fond tenue à sa sortie de portée.
|
||||
@@ -160,6 +163,7 @@ pub async fn serve(
|
||||
engineimg: cfg.engineimg,
|
||||
strata: cfg.strata,
|
||||
tasks: cfg.tasks,
|
||||
bench: cfg.bench,
|
||||
internal_token: cfg.internal_token.clone(),
|
||||
},
|
||||
});
|
||||
@@ -287,6 +291,11 @@ impl Proxy {
|
||||
{
|
||||
return crate::tasks::serve(&self.ctx, t, req, from_go).await;
|
||||
}
|
||||
if let Some(b) = &self.ctx.bench
|
||||
&& crate::bench::job::handles(path)
|
||||
{
|
||||
return crate::bench::job::serve(&self.ctx, b, req, from_go).await;
|
||||
}
|
||||
if let Some(b) = &self.ctx.backends
|
||||
&& crate::backends::handles(path)
|
||||
{
|
||||
|
||||
@@ -0,0 +1,547 @@
|
||||
//! Lot 27 : banc d'essai tenu par la façade (`LOKI_BENCH=rust`).
|
||||
//!
|
||||
//! Le journal tiré du Go (`internal/loki/parity_bench_test.go` →
|
||||
//! `tests/parity_bench/bench.json`) est rejoué par une façade contre le même
|
||||
//! faux moteur (port de `fakeBenchEngine`) : chaque requête qu'il reçoit est
|
||||
//! comparée, longs textes réduits à leur taille et à leur empreinte.
|
||||
|
||||
mod common;
|
||||
|
||||
use std::convert::Infallible;
|
||||
use std::net::SocketAddr;
|
||||
use std::path::PathBuf;
|
||||
use std::sync::{Arc, Mutex};
|
||||
use std::time::Duration;
|
||||
|
||||
use bytes::Bytes;
|
||||
use common::fakego;
|
||||
use common::parity::apply_files;
|
||||
use http_body_util::{BodyExt, Full};
|
||||
use hyper::service::service_fn;
|
||||
use hyper::{Request, Response};
|
||||
use hyper_util::rt::TokioIo;
|
||||
use loki_rs::bench::job::Bench;
|
||||
use regex::Regex;
|
||||
use serde_json::{Map, Value, json};
|
||||
use sha2::Digest;
|
||||
use tokio::net::TcpListener;
|
||||
|
||||
fn fixture() -> Value {
|
||||
serde_json::from_str(include_str!("parity_bench/bench.json")).expect("journal")
|
||||
}
|
||||
|
||||
fn s(v: &Value) -> &str {
|
||||
v.as_str().unwrap_or("")
|
||||
}
|
||||
|
||||
const TUNE_BUSY: &str = "optimisation en cours (loki tune) : le moteur est réservé aux essais — attends la fin ou annule-la";
|
||||
|
||||
/// `parityBenchDigest`.
|
||||
fn digest(v: &Value) -> Value {
|
||||
match v {
|
||||
Value::Object(o) => Value::Object(o.iter().map(|(k, e)| (k.clone(), digest(e))).collect()),
|
||||
Value::Array(a) => Value::Array(a.iter().map(digest).collect()),
|
||||
Value::String(x) if x.len() > 64 => {
|
||||
let sum = sha2::Sha256::digest(x.as_bytes());
|
||||
Value::String(format!("<{} octets {}>", x.len(), hex::encode(&sum[..6])))
|
||||
}
|
||||
_ => v.clone(),
|
||||
}
|
||||
}
|
||||
|
||||
/// Le faux moteur (`fakeBenchEngine`) et le journal des requêtes reçues.
|
||||
#[derive(Default)]
|
||||
struct Fake {
|
||||
n_ctx: i64,
|
||||
busy: bool,
|
||||
fail_at: usize,
|
||||
no_timing: bool,
|
||||
reject_effort: String,
|
||||
slow_hot: Duration,
|
||||
prev: i64,
|
||||
chats: usize,
|
||||
log: Vec<Value>,
|
||||
/// En-têtes Authorization reçus.
|
||||
auth: Vec<String>,
|
||||
}
|
||||
|
||||
fn reply(code: u16, body: String) -> Response<Full<Bytes>> {
|
||||
let mut r = Response::new(Full::new(Bytes::from(body)));
|
||||
*r.status_mut() = hyper::StatusCode::from_u16(code).unwrap();
|
||||
r
|
||||
}
|
||||
|
||||
async fn fake_engine(f: Arc<Mutex<Fake>>) -> u16 {
|
||||
let l = TcpListener::bind("127.0.0.1:0").await.unwrap();
|
||||
let port = l.local_addr().unwrap().port();
|
||||
tokio::spawn(async move {
|
||||
while let Ok((st, _)) = l.accept().await {
|
||||
let f = f.clone();
|
||||
tokio::spawn(async move {
|
||||
let svc = service_fn(move |req: Request<hyper::body::Incoming>| {
|
||||
let f = f.clone();
|
||||
async move { Ok::<_, Infallible>(handle(f, req).await) }
|
||||
});
|
||||
let _ = hyper::server::conn::http1::Builder::new()
|
||||
.serve_connection(TokioIo::new(st), svc)
|
||||
.await;
|
||||
});
|
||||
}
|
||||
});
|
||||
port
|
||||
}
|
||||
|
||||
async fn handle(f: Arc<Mutex<Fake>>, req: Request<hyper::body::Incoming>) -> Response<Full<Bytes>> {
|
||||
let method = req.method().to_string();
|
||||
let path = req.uri().path().to_string();
|
||||
if let Some(a) = req.headers().get("authorization") {
|
||||
f.lock()
|
||||
.unwrap()
|
||||
.auth
|
||||
.push(String::from_utf8_lossy(a.as_bytes()).into_owned());
|
||||
}
|
||||
let raw = req.into_body().collect().await.unwrap().to_bytes();
|
||||
let body: Option<Value> = serde_json::from_slice(&raw).ok();
|
||||
{
|
||||
let mut entry = Map::new();
|
||||
entry.insert("method".into(), json!(method));
|
||||
entry.insert("path".into(), json!(path));
|
||||
if let Some(b) = &body {
|
||||
entry.insert("body".into(), digest(b));
|
||||
}
|
||||
f.lock().unwrap().log.push(Value::Object(entry));
|
||||
}
|
||||
match path.as_str() {
|
||||
"/health" => reply(200, r#"{"status":"ok"}"#.into()),
|
||||
"/slots" => {
|
||||
let busy = f.lock().unwrap().busy;
|
||||
reply(200, format!(r#"[{{"id":0,"is_processing":{busy}}}]"#))
|
||||
}
|
||||
"/props" => {
|
||||
let n = f.lock().unwrap().n_ctx;
|
||||
reply(
|
||||
200,
|
||||
format!(
|
||||
r#"{{"build_info":"b9999-test","total_slots":1,"default_generation_settings":{{"n_ctx":{n}}}}}"#
|
||||
),
|
||||
)
|
||||
}
|
||||
"/tokenize" => {
|
||||
let n = body
|
||||
.as_ref()
|
||||
.and_then(|b| b["content"].as_str())
|
||||
.map_or(0, |c| c.len() / 4);
|
||||
reply(200, json!({ "tokens": vec![0; n] }).to_string() + "\n")
|
||||
}
|
||||
"/v1/chat/completions" => {
|
||||
let p = body.unwrap_or(Value::Null);
|
||||
let (n, fail_at, reject, slow) = {
|
||||
let mut g = f.lock().unwrap();
|
||||
g.chats += 1;
|
||||
(g.chats, g.fail_at, g.reject_effort.clone(), g.slow_hot)
|
||||
};
|
||||
if fail_at == n {
|
||||
return reply(500, "{\"error\":\"boom\"}\n".into());
|
||||
}
|
||||
let effort = p["reasoning_effort"].as_str().unwrap_or("");
|
||||
if !reject.is_empty() && effort == reject {
|
||||
return reply(
|
||||
500,
|
||||
format!(
|
||||
"{{\"error\":{{\"code\":500,\"message\":\"{}Unexpected reasoning effort {effort}. Supported types are xhigh (default), medium, and low.\"}}}}\n",
|
||||
"jinja trace ".repeat(40)
|
||||
),
|
||||
);
|
||||
}
|
||||
let cache = p["cache_prompt"].as_bool() == Some(true);
|
||||
if cache && !slow.is_zero() {
|
||||
tokio::time::sleep(slow).await;
|
||||
}
|
||||
let total: i64 = p["messages"]
|
||||
.as_array()
|
||||
.into_iter()
|
||||
.flatten()
|
||||
.map(|m| m["content"].as_str().unwrap_or("").len() as i64 / 4 + 4)
|
||||
.sum();
|
||||
let (cached, no_timing) = {
|
||||
let mut g = f.lock().unwrap();
|
||||
let cached = if cache { g.prev.min(total) } else { 0 };
|
||||
g.prev = total;
|
||||
(cached, g.no_timing)
|
||||
};
|
||||
let gen_n = p["max_tokens"].as_f64().unwrap_or(0.0) as i64;
|
||||
let mut resp = json!({
|
||||
"choices": [{ "message": { "role": "assistant", "content": "réponse" } }],
|
||||
"usage": { "prompt_tokens": total, "completion_tokens": gen_n },
|
||||
});
|
||||
if !no_timing {
|
||||
resp["timings"] = json!({
|
||||
"prompt_n": total - cached,
|
||||
"prompt_ms": (total - cached) as f64 / 2.0,
|
||||
"prompt_per_second": 2000.0,
|
||||
"predicted_n": gen_n,
|
||||
"predicted_ms": gen_n as f64 * 20.0,
|
||||
"predicted_per_second": 50.0,
|
||||
"cache_n": cached,
|
||||
"draft_n": 10,
|
||||
"draft_n_accepted": 7,
|
||||
});
|
||||
}
|
||||
reply(200, resp.to_string() + "\n")
|
||||
}
|
||||
_ => reply(404, "404 page not found\n".into()),
|
||||
}
|
||||
}
|
||||
|
||||
struct Bn {
|
||||
facade: SocketAddr,
|
||||
go: Arc<fakego::FakeGo>,
|
||||
_stop: tokio::sync::oneshot::Sender<()>,
|
||||
home: PathBuf,
|
||||
bench: Arc<Bench>,
|
||||
fake: Arc<Mutex<Fake>>,
|
||||
port: u16,
|
||||
closed: u16,
|
||||
_tmp: tempfile::TempDir,
|
||||
}
|
||||
|
||||
async fn start() -> Bn {
|
||||
start_with(|_| {}).await
|
||||
}
|
||||
|
||||
async fn start_with(tweak: impl FnOnce(&mut Bench)) -> Bn {
|
||||
let tmp = tempfile::tempdir().unwrap();
|
||||
let home = tmp.path().to_path_buf();
|
||||
let fake = Arc::new(Mutex::new(Fake::default()));
|
||||
let port = fake_engine(fake.clone()).await;
|
||||
let closed = std::net::TcpListener::bind("127.0.0.1:0")
|
||||
.unwrap()
|
||||
.local_addr()
|
||||
.unwrap()
|
||||
.port();
|
||||
let paths = loki_rs::engine::Paths {
|
||||
home: home.clone(),
|
||||
service: "loki-engine".into(),
|
||||
};
|
||||
let mut bench = Bench::new(paths);
|
||||
tweak(&mut bench);
|
||||
let bench = Arc::new(bench);
|
||||
let (facade, go, stop, _) = fakego::start_with(
|
||||
home.clone(),
|
||||
fakego::Opts {
|
||||
bench: Some(bench.clone()),
|
||||
..Default::default()
|
||||
},
|
||||
)
|
||||
.await;
|
||||
Bn {
|
||||
facade,
|
||||
go,
|
||||
_stop: stop,
|
||||
home,
|
||||
bench,
|
||||
fake,
|
||||
port,
|
||||
closed,
|
||||
_tmp: tmp,
|
||||
}
|
||||
}
|
||||
|
||||
impl Bn {
|
||||
fn mask(&self, v: &str) -> String {
|
||||
let id = Regex::new(r#""b[0-9]+-[0-9]+""#).unwrap();
|
||||
let t = Regex::new(r#""(elapsed_sec|at)":[0-9.e+-]+"#).unwrap();
|
||||
let fp = Regex::new(r#""fingerprint":"[0-9a-f]+""#).unwrap();
|
||||
let v = v.replace(&self.home.to_string_lossy().into_owned(), "{{HOME}}");
|
||||
let v = id.replace_all(&v, r#""{{ID}}""#);
|
||||
let v = t.replace_all(&v, r#""$1":"{{T}}""#);
|
||||
fp.replace_all(&v, r#""fingerprint":"{{FP}}""#).into_owned()
|
||||
}
|
||||
|
||||
fn unmask(&self, v: &str) -> String {
|
||||
v.replace("{{PORT}}", &self.port.to_string())
|
||||
.replace("{{CLOSED}}", &self.closed.to_string())
|
||||
.replace("{{HOME}}", &self.home.to_string_lossy())
|
||||
}
|
||||
|
||||
fn decode(&self, raw: &str) -> Value {
|
||||
if raw.is_empty() {
|
||||
return Value::Null;
|
||||
}
|
||||
serde_json::from_str(&self.mask(raw)).unwrap_or_else(|_| panic!("illisible : {raw:?}"))
|
||||
}
|
||||
|
||||
async fn call(&self, method: &str, path: &str, body: &str) -> (u16, Value) {
|
||||
let (code, b) = fakego::send(self.facade, method, path, &[], body).await;
|
||||
(code.as_u16(), self.decode(&String::from_utf8_lossy(&b)))
|
||||
}
|
||||
|
||||
async fn status(&self) -> Value {
|
||||
self.call("GET", "/api/bench/status", "").await.1
|
||||
}
|
||||
|
||||
async fn wait(&self, name: &str) {
|
||||
for _ in 0..12000 {
|
||||
if self.status().await["running"] != json!(true) {
|
||||
return;
|
||||
}
|
||||
tokio::time::sleep(Duration::from_millis(5)).await;
|
||||
}
|
||||
panic!("{name} : le bench ne finit pas");
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
|
||||
async fn parite_du_banc_d_essai() {
|
||||
let fx = fixture();
|
||||
let bn = start().await;
|
||||
for st in fx["steps"].as_array().unwrap() {
|
||||
let name = s(&st["name"]);
|
||||
for (k, v) in st["config"].as_object().into_iter().flatten() {
|
||||
// putStr du Go : une valeur vide retire la clé.
|
||||
match bn.unmask(s(v)) {
|
||||
v if v.is_empty() => bn.go.del("config", k),
|
||||
v => bn.go.set("config", k, &v),
|
||||
}
|
||||
}
|
||||
let files: Map<String, Value> = st["files"]
|
||||
.as_object()
|
||||
.into_iter()
|
||||
.flatten()
|
||||
.map(|(k, v)| (k.clone(), Value::String(bn.unmask(s(v)))))
|
||||
.collect();
|
||||
apply_files(&bn.home, &Value::Object(files));
|
||||
if let Some(e) = st["engine"].as_object() {
|
||||
let n = |k: &str| e.get(k).and_then(Value::as_i64).unwrap_or(0);
|
||||
*bn.fake.lock().unwrap() = Fake {
|
||||
n_ctx: n("n_ctx"),
|
||||
busy: e.get("busy").and_then(Value::as_bool) == Some(true),
|
||||
fail_at: n("fail_at") as usize,
|
||||
no_timing: e.get("no_timing").and_then(Value::as_bool) == Some(true),
|
||||
reject_effort: e
|
||||
.get("reject_effort")
|
||||
.and_then(Value::as_str)
|
||||
.unwrap_or("")
|
||||
.to_string(),
|
||||
slow_hot: Duration::from_millis(n("slow_hot_ms") as u64),
|
||||
..Default::default()
|
||||
};
|
||||
*bn.bench.req_timeout.lock().unwrap() = match n("timeout_ms") {
|
||||
0 => loki_rs::bench::REQ_TIMEOUT,
|
||||
ms => Duration::from_millis(ms as u64),
|
||||
};
|
||||
}
|
||||
bn.fake.lock().unwrap().log.clear();
|
||||
*bn.go.tune_busy.lock().unwrap() =
|
||||
(st["tune"].as_bool() == Some(true)).then(|| TUNE_BUSY.to_string());
|
||||
*bn.go.go_busy.lock().unwrap() =
|
||||
(st["generating"].as_bool() == Some(true)).then(|| "génération en cours".to_string());
|
||||
let (code, out) = bn
|
||||
.call(s(&st["method"]), s(&st["path"]), &bn.unmask(s(&st["body"])))
|
||||
.await;
|
||||
*bn.go.tune_busy.lock().unwrap() = None;
|
||||
*bn.go.go_busy.lock().unwrap() = None;
|
||||
assert_eq!(code as i64, st["status"].as_i64().unwrap(), "{name}");
|
||||
assert_eq!(out, st["out"], "{name}");
|
||||
if st["again"].as_bool() == Some(true) {
|
||||
let (_, again) = bn.call("POST", "/api/bench", "{}").await;
|
||||
assert_eq!(again, st["again_out"], "{name} : second lancement");
|
||||
}
|
||||
if let Some(phase) = st["cancel_at"].as_str() {
|
||||
let mut ok = false;
|
||||
for _ in 0..6000 {
|
||||
let p = bn.status().await["phase"]
|
||||
.as_str()
|
||||
.unwrap_or("")
|
||||
.to_string();
|
||||
if p.starts_with(phase) {
|
||||
ok = true;
|
||||
break;
|
||||
}
|
||||
tokio::time::sleep(Duration::from_millis(2)).await;
|
||||
}
|
||||
assert!(ok, "{name} : phase {phase} jamais atteinte");
|
||||
while bn.fake.lock().unwrap().chats < 4 {
|
||||
tokio::time::sleep(Duration::from_millis(2)).await;
|
||||
}
|
||||
bn.call("POST", "/api/bench/cancel", "").await;
|
||||
}
|
||||
if st["wait"].as_bool() == Some(true) || st["cancel_at"].is_string() {
|
||||
bn.wait(name).await;
|
||||
assert_eq!(bn.status().await, st["job"], "{name} : job");
|
||||
}
|
||||
// Les requêtes reçues : dans l'ordre, sauf les sondes /health d'un
|
||||
// second lancement, concurrentes de celles du bench.
|
||||
let mut got = bn.fake.lock().unwrap().log.clone();
|
||||
let mut want = st["calls"].as_array().cloned().unwrap_or_default();
|
||||
let key = |v: &Value| v.to_string();
|
||||
if st["again"].as_bool() == Some(true) {
|
||||
got.sort_by_key(key);
|
||||
want.sort_by_key(key);
|
||||
}
|
||||
assert_eq!(
|
||||
Value::Array(got),
|
||||
Value::Array(want),
|
||||
"{name} : requêtes au moteur"
|
||||
);
|
||||
let stored = |k: &str| {
|
||||
bn.go
|
||||
.get("state", k)
|
||||
.map_or(Value::Null, |raw| bn.decode(&raw))
|
||||
};
|
||||
assert_eq!(stored("last_bench"), st["last"], "{name} : dernier bench");
|
||||
assert_eq!(
|
||||
stored("bench_presets"),
|
||||
st["presets"],
|
||||
"{name} : par preset"
|
||||
);
|
||||
if name == "mode rapide" {
|
||||
// L'empreinte masquée du journal est bien celle de la
|
||||
// configuration, et le Go a ouvert puis fermé le travail annexe.
|
||||
let sb: Value =
|
||||
serde_json::from_str(&bn.go.get("state", "last_bench").unwrap()).unwrap();
|
||||
let cfg = bn.go.config_map();
|
||||
assert_eq!(
|
||||
sb["fingerprint"],
|
||||
json!(loki_rs::presets_api::bench_fingerprint(&cfg, &cfg))
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl Bn {
|
||||
/// La configuration de base du journal, le faux moteur réglé.
|
||||
fn base(&self, n_ctx: i64, slow_hot: Duration) {
|
||||
for (k, v) in [
|
||||
("MODEL", "/models/qwen.gguf"),
|
||||
("CTX", "16384"),
|
||||
("REASONING", "off"),
|
||||
("CACHE_ISOLATE", "off"),
|
||||
] {
|
||||
self.go.set("config", k, v);
|
||||
}
|
||||
self.go.set("config", "PORT", &self.port.to_string());
|
||||
*self.fake.lock().unwrap() = Fake {
|
||||
n_ctx,
|
||||
slow_hot,
|
||||
..Default::default()
|
||||
};
|
||||
}
|
||||
|
||||
async fn until_turn(&self) {
|
||||
for _ in 0..6000 {
|
||||
let p = self.status().await["phase"]
|
||||
.as_str()
|
||||
.unwrap_or("")
|
||||
.to_string();
|
||||
if p.starts_with("tour 1/") && self.fake.lock().unwrap().chats >= 4 {
|
||||
return;
|
||||
}
|
||||
tokio::time::sleep(Duration::from_millis(2)).await;
|
||||
}
|
||||
panic!("le tour 1 n'arrive jamais");
|
||||
}
|
||||
}
|
||||
|
||||
/// Le bench prend le bail du moteur au motif « bench », ouvre puis ferme un
|
||||
/// travail annexe (slot servi) et laisse sa trace dans la télémétrie.
|
||||
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
|
||||
async fn bail_travail_annexe_et_trace() {
|
||||
let bn = start().await;
|
||||
bn.base(16384, Duration::ZERO);
|
||||
bn.go.set("state", "api_key", "sk-moteur");
|
||||
let (code, _) = bn.call("POST", "/api/bench", "{}").await;
|
||||
assert_eq!(code, 202);
|
||||
bn.wait("bench").await;
|
||||
assert_eq!(bn.status().await["saved"], json!(true));
|
||||
assert_eq!(*bn.go.lease_whys.lock().unwrap(), vec!["bench".to_string()]);
|
||||
let log = bn.go.lease_log.lock().unwrap().clone();
|
||||
assert!(
|
||||
log.first().is_some_and(|l| l.starts_with("acquire 30000")),
|
||||
"{log:?}"
|
||||
);
|
||||
assert!(
|
||||
log.last().is_some_and(|l| l.starts_with("release ")),
|
||||
"{log:?}"
|
||||
);
|
||||
assert!(bn.go.lease.lock().unwrap().is_none(), "bail rendu");
|
||||
assert_eq!(
|
||||
*bn.go.side_jobs.lock().unwrap(),
|
||||
vec!["begin 1".to_string(), "end 1 true".to_string()]
|
||||
);
|
||||
assert_eq!(
|
||||
*bn.go.bench_perfs.lock().unwrap(),
|
||||
vec![json!({
|
||||
"prompt_n": 2036, "prompt_ms": 1018.0, "prompt_per_second": 2000.0,
|
||||
"predicted_n": 300, "predicted_per_second": 50.0,
|
||||
})]
|
||||
);
|
||||
// La clé de l'API locale accompagne les requêtes du bench.
|
||||
let auth = bn.fake.lock().unwrap().auth.clone();
|
||||
assert!(!auth.is_empty());
|
||||
assert!(auth.iter().all(|a| a == "Bearer sk-moteur"), "{auth:?}");
|
||||
// Le second slot en service : le bench y passe.
|
||||
*bn.go.side_live.lock().unwrap() = true;
|
||||
bn.fake.lock().unwrap().log.clear();
|
||||
bn.call("POST", "/api/bench", "{}").await;
|
||||
bn.wait("bench").await;
|
||||
let chats: Vec<Value> = bn
|
||||
.fake
|
||||
.lock()
|
||||
.unwrap()
|
||||
.log
|
||||
.iter()
|
||||
.filter(|c| c["path"] == json!("/v1/chat/completions"))
|
||||
.map(|c| c["body"]["id_slot"].clone())
|
||||
.collect();
|
||||
assert_eq!(chats, vec![json!(1), json!(1)]);
|
||||
// Échec avant tout envoi : le travail annexe est fermé sans slot servi.
|
||||
bn.fake.lock().unwrap().busy = true;
|
||||
bn.go.side_jobs.lock().unwrap().clear();
|
||||
bn.call("POST", "/api/bench", "{}").await;
|
||||
bn.wait("bench").await;
|
||||
assert_eq!(
|
||||
*bn.go.side_jobs.lock().unwrap(),
|
||||
vec!["begin 1".to_string(), "end 1 false".to_string()]
|
||||
);
|
||||
}
|
||||
|
||||
/// Un stop du chat côté Go (signalé au renouvellement du bail) annule le
|
||||
/// bench ; un bail perdu aussi.
|
||||
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
|
||||
async fn stop_du_chat_et_bail_perdu() {
|
||||
let bn = start_with(|b| b.lease_ttl = Duration::from_millis(300)).await;
|
||||
bn.base(16384, Duration::from_secs(20));
|
||||
bn.call("POST", "/api/bench", r#"{"mode":"full"}"#).await;
|
||||
bn.until_turn().await;
|
||||
*bn.go.lease_stop.lock().unwrap() = true;
|
||||
bn.wait("stop").await;
|
||||
let st = bn.status().await;
|
||||
assert_eq!(
|
||||
(st["error"].clone(), st["canceled"].clone()),
|
||||
(json!("benchmark annulé"), json!(true))
|
||||
);
|
||||
assert!(bn.go.lease.lock().unwrap().is_none(), "bail rendu");
|
||||
*bn.go.lease_stop.lock().unwrap() = false;
|
||||
|
||||
bn.base(16384, Duration::from_secs(20));
|
||||
bn.call("POST", "/api/bench", r#"{"mode":"full"}"#).await;
|
||||
bn.until_turn().await;
|
||||
*bn.go.lease.lock().unwrap() = None;
|
||||
bn.wait("perdu").await;
|
||||
let st = bn.status().await;
|
||||
assert_eq!(
|
||||
(st["error"].clone(), st["canceled"].clone()),
|
||||
(json!("benchmark annulé"), json!(true))
|
||||
);
|
||||
}
|
||||
|
||||
/// Sans la façade pour le bench, ses routes partent au Go.
|
||||
#[tokio::test]
|
||||
async fn sans_la_facade() {
|
||||
let tmp = tempfile::tempdir().unwrap();
|
||||
let (facade, _go, _stop) = fakego::start(tmp.path().to_path_buf()).await;
|
||||
let (c, b) = fakego::send(facade, "GET", "/api/bench/status", &[], "").await;
|
||||
assert_eq!(c.as_u16(), 200);
|
||||
assert_eq!(String::from_utf8_lossy(&b), "go:/api/bench/status");
|
||||
}
|
||||
@@ -109,6 +109,12 @@ pub struct FakeGo {
|
||||
pub task_run_delay: Mutex<Option<std::time::Duration>>,
|
||||
/// /internal/v1/push/send reçus : (titre, corps).
|
||||
pub push_sends: Mutex<Vec<(String, String)>>,
|
||||
/// /internal/v1/engine/lease reçus avec un motif (« bench »).
|
||||
pub lease_whys: Mutex<Vec<String>>,
|
||||
/// /internal/v1/engine/sidejob : « begin n » ou « end n served ».
|
||||
pub side_jobs: Mutex<Vec<String>>,
|
||||
/// /internal/v1/bench/perf reçus.
|
||||
pub bench_perfs: Mutex<Vec<serde_json::Value>>,
|
||||
}
|
||||
|
||||
impl FakeGo {
|
||||
@@ -120,6 +126,19 @@ impl FakeGo {
|
||||
.or_default()
|
||||
.insert(key.into(), val.as_bytes().to_vec());
|
||||
}
|
||||
/// La configuration rangée.
|
||||
pub fn config_map(&self) -> HashMap<String, String> {
|
||||
self.store
|
||||
.lock()
|
||||
.unwrap()
|
||||
.get("config")
|
||||
.map(|m| {
|
||||
m.iter()
|
||||
.map(|(k, v)| (k.clone(), String::from_utf8_lossy(v).into_owned()))
|
||||
.collect()
|
||||
})
|
||||
.unwrap_or_default()
|
||||
}
|
||||
pub fn del(&self, bucket: &str, key: &str) {
|
||||
if let Some(b) = self.store.lock().unwrap().get_mut(bucket) {
|
||||
b.remove(key);
|
||||
@@ -227,6 +246,9 @@ async fn handle(st: Arc<FakeGo>, req: Request<Incoming>) -> Response<Full<Bytes>
|
||||
"/internal/v1/engine/lease" => {
|
||||
let body = req.into_body().collect().await.unwrap().to_bytes();
|
||||
let v: serde_json::Value = serde_json::from_slice(&body).unwrap_or_default();
|
||||
if let Some(why) = v["why"].as_str() {
|
||||
st.lease_whys.lock().unwrap().push(why.to_string());
|
||||
}
|
||||
if let Some(why) = st.go_busy.lock().unwrap().clone() {
|
||||
let out = serde_json::json!({"error": why, "task_id": "t1", "task_name": "veille"});
|
||||
return resp(409, out.to_string());
|
||||
@@ -508,6 +530,28 @@ async fn handle(st: Arc<FakeGo>, req: Request<Incoming>) -> Response<Full<Bytes>
|
||||
}
|
||||
resp(200, r#"{"busy":true}"#)
|
||||
}
|
||||
"/internal/v1/engine/sidejob" => {
|
||||
let body = req.into_body().collect().await.unwrap().to_bytes();
|
||||
let v: serde_json::Value = serde_json::from_slice(&body).unwrap_or_default();
|
||||
let mut log = st.side_jobs.lock().unwrap();
|
||||
match v["end"].as_i64() {
|
||||
Some(id) if id != 0 => {
|
||||
log.push(format!("end {id} {}", v["served"]));
|
||||
resp(204, "")
|
||||
}
|
||||
_ => {
|
||||
let id = log.len() + 1;
|
||||
log.push(format!("begin {id}"));
|
||||
resp(200, serde_json::json!({ "id": id }).to_string())
|
||||
}
|
||||
}
|
||||
}
|
||||
"/internal/v1/bench/perf" => {
|
||||
let body = req.into_body().collect().await.unwrap().to_bytes();
|
||||
let v: serde_json::Value = serde_json::from_slice(&body).unwrap_or_default();
|
||||
st.bench_perfs.lock().unwrap().push(v);
|
||||
resp(204, "")
|
||||
}
|
||||
"/internal/v1/push/send" => {
|
||||
let body = req.into_body().collect().await.unwrap().to_bytes();
|
||||
let v: serde_json::Value = serde_json::from_slice(&body).unwrap_or_default();
|
||||
@@ -705,6 +749,7 @@ pub struct Opts {
|
||||
pub engineimg: Option<Arc<loki_rs::engineimg::EngineImg>>,
|
||||
pub strata: Option<Arc<loki_rs::strata::api::Strata>>,
|
||||
pub tasks: Option<Arc<loki_rs::tasks::Tasks>>,
|
||||
pub bench: Option<Arc<loki_rs::bench::job::Bench>>,
|
||||
/// Base de départ, posée avant que la façade démarre : (bucket, clé,
|
||||
/// valeur).
|
||||
pub seed: Vec<(String, String, String)>,
|
||||
@@ -816,6 +861,7 @@ pub async fn start_with(
|
||||
engineimg: opts.engineimg,
|
||||
strata: opts.strata,
|
||||
tasks: opts.tasks,
|
||||
bench: opts.bench,
|
||||
};
|
||||
tokio::spawn(loki_rs::proxy::serve(fl, cfg, async move {
|
||||
let _ = rx.await;
|
||||
|
||||
@@ -110,6 +110,7 @@ async fn banc(cle: Option<&str>) -> Banc {
|
||||
engineimg: None,
|
||||
strata: None,
|
||||
tasks: None,
|
||||
bench: None,
|
||||
};
|
||||
tokio::spawn(loki_rs::proxy::serve(l, cfg, async move {
|
||||
let _ = rx.await;
|
||||
|
||||
File diff suppressed because it is too large.
Load diff
@@ -116,6 +116,7 @@ async fn facade(backend: SocketAddr) -> (SocketAddr, oneshot::Sender<()>) {
|
||||
engineimg: None,
|
||||
strata: None,
|
||||
tasks: None,
|
||||
bench: None,
|
||||
};
|
||||
tokio::spawn(loki_rs::proxy::serve(l, cfg, async move {
|
||||
let _ = rx.await;
|
||||
|
||||
Reference in new issue
Block a user