diff --git a/README.md b/README.md index 7dd8e11..0fddd7e 100644 --- a/README.md +++ b/README.md @@ -241,6 +241,7 @@ redémarrages (volume `/data`). Variables d'environnement du conteneur : | `LOKI_BACKENDS` | `rust` : les backends llama.cpp — statut, vérification git, binaires précompilés, bascule fast/opt, backends custom, GPU vus par un moteur — et le job des moteurs sont tenus par la façade Rust ; les jobs du Go (compilation, moteur de l'image, Strata) lui rapportent leurs étapes (expérimental, migration en cours) | Go | | `LOKI_STRATA` | `rust` (avec `LOKI_BACKENDS=rust`) : l'écran de Strata — machine, ce qui y tient, recommandation, modèles installés —, ses réglages et son installation sont tenus par la façade Rust ; le lancement reste au Go (expérimental, migration en cours) | Go | | `LOKI_TASKS` | `rust` : les tâches planifiées — liste, enregistrement, bascule, interrupteur maître, lancement, arrêt —, le planificateur et les tâches « script seul » sont tenus par la façade Rust ; une tâche IA reste exécutée par le Go, à la demande de la façade (expérimental, migration en cours) | Go | +| `LOKI_BENCH` | `rust` : le banc d'essai — lancement, progression, annulation, dernière mesure, mode rapide et mode complet en profondeur — est tenu par la façade Rust ; le verrou de génération reste celui du Go, pris par le bail du moteur (expérimental, migration en cours) | Go | | `LOKI_ENGINE_IMAGE` | `rust` (avec `LOKI_BACKENDS=rust`) : le panneau « moteur de l'image » — version qui tourne, dernier build publié, version recommandée, mise à jour depuis le registre officiel avec essai à blanc, bascules, retour à la version précédente, vérification du rendu du gabarit — est tenu par la façade Rust (expérimental, migration en cours) | Go | | `LOKI_PROJECTS` | `rust` : le hub des projets (liste et comptes, création, renommage, description, suppression, bascule) est tenu par la façade Rust ; les discussions restent à qui tient la conversation (expérimental, migration en cours) | Go | | `LOKI_SYSTEM` | `rust` : les interrupteurs système — accès internet et Crawl4AI, pilotage du navigateur, clé de complétion, écoute réseau du moteur — sont tenus par la façade Rust ; le navigateur piloté reste au Go (expérimental, migration en cours) | Go | diff --git a/docs/superpowers/specs/2026-10-15-rust-migration-lot27-banc-d-essai-design.md b/docs/superpowers/specs/2026-10-15-rust-migration-lot27-banc-d-essai-design.md new file mode 100644 index 0000000..756c841 --- /dev/null +++ b/docs/superpowers/specs/2026-10-15-rust-migration-lot27-banc-d-essai-design.md @@ -0,0 +1,125 @@ +# Lot 27 — Banc d'essai tenu par la façade + +## Contexte + +Le banc d'essai mesure le moteur local tel que le chat s'en sert, sur +`/v1/chat/completions`, avec le gabarit, le raisonnement et +l'échantillonnage du preset (`llm_bench.go`, `llm_bench_job.go`). Il a deux +modes : + +- **rapide** : un préchauffage jeté, puis la ligne courte (~2000 jetons de + prompt, ~300 générés) ; +- **complet** : la ligne courte, puis une discussion à profondeur réelle — + un prefill à froid de D jetons, et trois tours qui y ajoutent chacun ~2048 + jetons neufs. La reprise du cache est lue dans la réponse, jamais + supposée. + +Il tourne en tâche de fond derrière quatre routes : + +- `POST /api/bench` lance (202 et un id) ; +- `GET /api/bench/status` dit la phase, puis le résultat ou l'erreur ; +- `POST /api/bench/cancel` annule ; +- `GET /api/bench/last` rend la dernière mesure enregistrée. + +Pendant la mesure, le bench tient le verrou de génération de la +conversation. Un message, une tâche planifiée ou un client `/v1` reçoivent +le refus du bench. Seule une mesure complète est enregistrée : la +dernière, et celle du preset actif. + +## Le lot + +Avec `LOKI_BENCH=rust` (expérimental), la façade (`rust/loki-rs/src/bench`) +sert les quatre routes et déroule le protocole contre le moteur : + +- `bench/mod.rs` : le protocole, porté du Go à l'octet près. On y trouve : + - les mêmes corpus : la prose du Go et l'interface web embarquée, de la + même source ; + - les mêmes requêtes ; + - le même calcul de profondeur, de reprise du cache et de débit ; + - les mêmes messages d'erreur ; +- `bench/job.rs` : le job, les routes et l'enregistrement (`last_bench`, + `bench_presets`, avec l'empreinte de configuration du Go). + +Le verrou de génération reste celui de la conversation du Go. La façade le +prend par le **bail du moteur**, avec le motif `bench` : + +- le Go note alors `benching` et donne aux autres le refus du benchmark ; +- changer de discussion ne rompt pas le bail ; +- `benchRunning()` le voit aussi : les clients `/v1`, le préchauffage et + l'optimiseur attendent ; +- un stop du chat côté Go (signalé au renouvellement), un Reset ou un bail + perdu annulent le bench. + +Le Go prête aussi à la façade : + +- **le travail annexe** (`/internal/v1/engine/sidejob`) : à la fin, le slot + du bench est effacé si c'est sans risque, comme pour un bench du Go ; +- **la télémétrie** (`/internal/v1/bench/perf`) : la trace du bench dans + `perf_log`. + +Le port, la clé de l'API locale et le second slot (`SIDE_SLOT`) viennent de +`/internal/v1/oai/route`. Les niveaux de raisonnement appris sont ceux de la +conversation de la façade si elle la tient, sinon ceux du bench. + +La ligne de commande (`loki bench`) reste au Go. + +## Correction des tests du Go + +Trois tests du préchauffage échouaient dix minutes par jour, avant minuit : +le préchauffage se tait alors (`prewarmNearMidnight`), et ils lisaient +l'heure réelle. L'heure de ce garde-fou devient remplaçable (`prewarmNow`), +et les tests la fixent à midi. + +## Parité + +`internal/loki/parity_bench_test.go` (Linux) tire du Go un journal rejoué +par `tests/bench.rs` contre le même faux moteur (`fakeBenchEngine`, porté). +Chaque requête reçue par le moteur est comparée ; les longs textes (corpus) +sont réduits à leur taille et à leur empreinte. Le journal couvre : + +- **les routes** : statut et dernière mesure sans bench, GET refusés, + annulation sans bench ; +- **les refus au lancement** : preset externe, optimisation en cours, modèle + pas prêt, génération en cours, second lancement pendant un bench ; +- **les échecs dans le job** : moteur occupé par un autre (`/slots`), preset + Strata ; +- **les mesures** : + - mode rapide, puis mode complet (profondeur, trois tours, reprise du + cache, brouillon) ; + - contexte trop petit (profondeur sautée), contexte inconnu du moteur + (celui du preset) ; + - paramètres bornés, petits paramètres, corps illisible ; +- **les échecs d'une mesure** : + - préchauffage, ligne courte, prefill à froid ou tour en profondeur en + erreur ; + - réponse sans timings ; +- **le raisonnement** : niveau refusé par le gabarit puis rejoué, puis le + niveau appris envoyé d'emblée ; +- **budget et annulation** : budget d'une requête dépassé (profondeur + partielle, non enregistrée), annulation pendant la profondeur ; +- **après chaque étape** : la dernière mesure et les mesures par preset. + +Autres tests : + +- ceux du Go (`llm_bench_test.go`), portés (`bench/tests.rs`) : profondeur, + types de cache, requête, morceaux du corpus, reprise du cache, relectures + du disque, empreinte de l'enregistrement ; +- la réponse du moteur (timings, cache, brouillon, nombres illisibles), + `Duration.String` du Go, les flottants écrits à la Go ; +- la façade : le bail « bench », le travail annexe (slot servi ou non), la + trace, la clé de l'API locale, le second slot, le stop du chat et le bail + perdu, les routes sans la façade ; +- le Go (`TestFacadeBench`) : + - les quatre routes partent à la façade, et elles seules ; + - le bail « bench » : refus, bascule de discussion, libération, + expiration ; + - le travail annexe et la trace. + +## Écarts assumés + +- Une erreur réseau vers le moteur garde le texte de `reqwest` après le + préfixe du Go (`Post "url": …`). +- Un niveau de raisonnement appris par le bench de la façade n'est pas + transmis au chat du Go, quand le Go tient la conversation (et + inversement) : la première requête paie le refus, puis la traduction est + retenue. diff --git a/internal/loki/chat_prewarm.go b/internal/loki/chat_prewarm.go index 15ab32d..0f96093 100644 --- a/internal/loki/chat_prewarm.go +++ b/internal/loki/chat_prewarm.go @@ -245,6 +245,8 @@ func prewarmNearMidnight(now time.Time) bool { // Points d'accroche remplaçables par les tests. var ( + // prewarmNow : l'heure du garde-fou de minuit. + prewarmNow = time.Now // prewarmAsync : false = le préchauffage tourne dans l'appelant (tests). prewarmAsync = true // prewarmSlots : nombre de slots du moteur local (/props total_slots). @@ -375,7 +377,7 @@ func (c *Conversation) prewarm(why string) { return } } - if sideJobs.Load() > 0 || benchRunning() || tuneGuard() != nil || prewarmNearMidnight(time.Now()) { + if sideJobs.Load() > 0 || benchRunning() || tuneGuard() != nil || prewarmNearMidnight(prewarmNow()) { return } if !healthCheck() { diff --git a/internal/loki/chat_prewarm_test.go b/internal/loki/chat_prewarm_test.go index 9af4ae0..65cab72 100644 --- a/internal/loki/chat_prewarm_test.go +++ b/internal/loki/chat_prewarm_test.go @@ -79,6 +79,11 @@ func (m *moteurPrewarm) completions() []string { // prewarmSetup : base neuve, discussion active, faux moteur à un slot, // préchauffage synchrone. keys : réglages posés. func prewarmSetup(t *testing.T, keys map[string]string) *moteurPrewarm { + // Le préchauffage se tait dans les dix minutes avant minuit : midi ici, + // pour que les tests ne dépendent pas de l'heure à laquelle ils tournent. + oldNow := prewarmNow + prewarmNow = func() time.Time { return time.Date(2026, 1, 1, 12, 0, 0, 0, time.Local) } + t.Cleanup(func() { prewarmNow = oldNow }) t.Helper() withWorkspace(t) activeProjMu.Lock() diff --git a/internal/loki/facade_bench.go b/internal/loki/facade_bench.go new file mode 100644 index 0000000..318f770 --- /dev/null +++ b/internal/loki/facade_bench.go @@ -0,0 +1,95 @@ +package loki + +import ( + "net/http" + "os" + "strings" + "sync" +) + +// facade_bench.go — le banc d'essai tenu par la façade Rust (LOKI_BENCH=rust, +// lot 27 de la migration ; expérimental, le Go reste le défaut) : les routes +// /api/bench*, le job et le protocole de mesure (rust/loki-rs/src/bench). Le +// verrou de génération reste celui de la conversation du Go : la façade le +// prend par le bail du moteur, avec le motif « bench » (benching, et le refus +// errBenchBusy pour les autres). Le Go lui prête aussi l'isolation des travaux +// annexes (effacement du slot à la fin) et la télémétrie (perf_log.go). + +const facadeBenchEnv = "LOKI_BENCH" + +// facadeBench : la façade tient-elle le banc d'essai ? +func facadeBench() bool { + return strings.EqualFold(strings.TrimSpace(os.Getenv(facadeBenchEnv)), "rust") && + facadeAddr() != "" && internalToken() != "" +} + +// facadeBenchRoutes : les routes que la façade sert alors. +var facadeBenchRoutes = map[string]bool{ + "/api/bench": true, + "/api/bench/status": true, + "/api/bench/cancel": true, + "/api/bench/last": true, +} + +// facadeSideJobs : les travaux annexes ouverts par la façade (id → fin). +var facadeSideJobs = struct { + mu sync.Mutex + seq int + m map[int]func() +}{m: map[int]func(){}} + +// handleInternalSideJob : POST {} ouvre un travail annexe (engineSideJob) et +// rend son id ; POST {"end": id, "served": bool} le ferme — served : le +// moteur a servi au moins une requête du travail (engineServed), le slot +// porte donc son état et peut être effacé. +func handleInternalSideJob(w http.ResponseWriter, r *http.Request) { + var in struct { + End int `json:"end"` + Served bool `json:"served"` + } + if !readLeaseBody(w, r, &in) { + return + } + if in.End == 0 { + done := engineSideJob() + facadeSideJobs.mu.Lock() + facadeSideJobs.seq++ + id := facadeSideJobs.seq + facadeSideJobs.m[id] = done + facadeSideJobs.mu.Unlock() + sendJSON(w, http.StatusOK, map[string]any{"id": id}) + return + } + facadeSideJobs.mu.Lock() + done := facadeSideJobs.m[in.End] + delete(facadeSideJobs.m, in.End) + facadeSideJobs.mu.Unlock() + if done == nil { + sendJSON(w, http.StatusNotFound, map[string]any{"error": "travail annexe inconnu"}) + return + } + if in.Served { + engineServed() + } + done() + w.WriteHeader(http.StatusNoContent) +} + +// handleInternalBenchPerf : la trace d'un bench de la façade dans la +// télémétrie (le bench a pris le slot : la perte de cache du tour suivant lui +// revient). +func handleInternalBenchPerf(w http.ResponseWriter, r *http.Request) { + var in struct { + PromptN int `json:"prompt_n"` + PromptMs float64 `json:"prompt_ms"` + PromptPerSecond float64 `json:"prompt_per_second"` + PredictedN int `json:"predicted_n"` + PredictedPerSec float64 `json:"predicted_per_second"` + } + if !readLeaseBody(w, r, &in) { + return + } + perfRecord(perfRec{Kind: perfBench, Complete: true, Total: in.PromptN, New: in.PromptN, + PPms: in.PromptMs, PPtps: in.PromptPerSecond, Gen: in.PredictedN, TGtps: in.PredictedPerSec}, nil) + w.WriteHeader(http.StatusNoContent) +} diff --git a/internal/loki/facade_bench_test.go b/internal/loki/facade_bench_test.go new file mode 100644 index 0000000..e4b085c --- /dev/null +++ b/internal/loki/facade_bench_test.go @@ -0,0 +1,167 @@ +package loki + +import ( + "encoding/json" + "io" + "net/http" + "net/http/httptest" + "strconv" + "strings" + "sync" + "testing" + "time" +) + +// Le banc d'essai part à la façade avec LOKI_BENCH=rust (lot 27) ; le reste, +// non. La façade tient le verrou de génération par le bail du moteur, motif +// « bench », et le Go lui prête le travail annexe et la télémétrie. +func TestFacadeBench(t *testing.T) { + testHome(t) + var mu sync.Mutex + var calls []string + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.Header.Get(internalTokenHeader) != jetonTest { + w.WriteHeader(403) + return + } + mu.Lock() + calls = append(calls, r.Method+" "+r.URL.Path) + mu.Unlock() + io.WriteString(w, `{"ok":true,"de":"facade"}`) + })) + defer srv.Close() + t.Setenv(internalTokenEnv, jetonTest) + t.Setenv(facadeAddrEnv, strings.TrimPrefix(srv.URL, "http://")) + local := func(w http.ResponseWriter, r *http.Request) { io.WriteString(w, "go") } + + t.Setenv(facadeBenchEnv, "") + if facadeBench() { + t.Fatal("tenu sans LOKI_BENCH") + } + for p := range facadeBenchRoutes { + w := httptest.NewRecorder() + withFacadeChat(p, local)(w, httptest.NewRequest("POST", p, strings.NewReader("{}"))) + if w.Body.String() != "go" { + t.Fatalf("sans LOKI_BENCH, %s : %q", p, w.Body.String()) + } + } + t.Setenv(facadeBenchEnv, " RUST") + if len(facadeBenchRoutes) != 4 { + t.Fatalf("routes : %v", facadeBenchRoutes) + } + for p := range facadeBenchRoutes { + w := httptest.NewRecorder() + withFacadeChat(p, local)(w, httptest.NewRequest("POST", p, strings.NewReader("{}"))) + if !strings.Contains(w.Body.String(), "facade") { + t.Fatalf("%s non relayé : %q", p, w.Body.String()) + } + } + mu.Lock() + n := len(calls) + mu.Unlock() + if n != len(facadeBenchRoutes) { + t.Fatalf("relais : %v", calls) + } + w := httptest.NewRecorder() + withFacadeChat("/api/tune", local)(w, httptest.NewRequest("POST", "/api/tune", nil)) + if w.Body.String() != "go" { + t.Fatalf("l'optimiseur reste au Go : %q", w.Body.String()) + } + + // Le bail « bench » : le verrou, le refus du bench pour les autres. + post := func(h http.HandlerFunc, path, body string) (int, map[string]any) { + rec := httptest.NewRecorder() + h(rec, httptest.NewRequest("POST", path, strings.NewReader(body))) + var out map[string]any + _ = json.Unmarshal(rec.Body.Bytes(), &out) + return rec.Code, out + } + code, out := post(handleInternalLease, "/internal/v1/engine/lease", `{"ttl_ms":60000,"why":"bench"}`) + if code != 200 { + t.Fatalf("bail : %d %v", code, out) + } + id, _ := out["lease"].(string) + if !benchRunning() || conv.busyReason() != errBenchBusy { + t.Fatalf("bail du bench : running=%v refus=%v", benchRunning(), conv.busyReason()) + } + if code, out := post(handleInternalLease, "/internal/v1/engine/lease", `{}`); code != 409 || out["error"] != errBenchBusy.Error() { + t.Fatalf("second bail : %d %v", code, out) + } + // Changer de discussion ne rompt pas le bail d'un bench. + conv.loadFrom("autre", nil) + if !benchRunning() || !conv.isGenerating() { + t.Fatal("le bail du bench a sauté à la bascule de discussion") + } + if code, _ := post(handleInternalLeaseRelease, "/internal/v1/engine/lease/release", `{"lease":"`+id+`"}`); code != 204 { + t.Fatalf("libération : %d", code) + } + if benchRunning() || conv.isGenerating() || conv.busyReason() == errBenchBusy { + t.Fatal("bail rendu : le verrou devait être libre") + } + // Un bail ordinaire n'est pas un bench ; un bail de bench expiré rend + // le verrou et le refus. + code, out = post(handleInternalLease, "/internal/v1/engine/lease", `{}`) + if code != 200 || benchRunning() { + t.Fatalf("bail ordinaire : %d %v bench=%v", code, out, benchRunning()) + } + post(handleInternalLeaseRelease, "/internal/v1/engine/lease/release", `{"lease":"`+out["lease"].(string)+`"}`) + conv.mu.Lock() + conv.extLeaseClearLocked() + conv.Generating = false + conv.mu.Unlock() + if _, err := conv.extLeaseAcquire(5*time.Second, true); err != nil { + t.Fatal(err) + } + conv.mu.Lock() + st := conv.extLease + conv.mu.Unlock() + conv.extLeaseExpire(st) + if benchRunning() || conv.busyReason() == errBenchBusy { + t.Fatal("bail de bench expiré : refus du bench encore là") + } + + // Le travail annexe ouvert puis fermé par la façade. + code, out = post(handleInternalSideJob, "/internal/v1/engine/sidejob", `{}`) + job, _ := out["id"].(float64) + if code != 200 || job <= 0 || sideJobs.Load() != 1 { + t.Fatalf("ouverture : %d %v (en cours : %d)", code, out, sideJobs.Load()) + } + served := engineServedCount() + if code, _ := post(handleInternalSideJob, "/internal/v1/engine/sidejob", `{"end":`+strconv.Itoa(int(job))+`,"served":true}`); code != 204 { + t.Fatalf("fermeture : %d", code) + } + if sideJobs.Load() != 0 || engineServedCount() != served+1 { + t.Fatalf("après la fermeture : en cours %d, servies %d", sideJobs.Load(), engineServedCount()-served) + } + if code, _ := post(handleInternalSideJob, "/internal/v1/engine/sidejob", `{"end":999}`); code != 404 { + t.Fatalf("travail inconnu : %d", code) + } + rec := httptest.NewRecorder() + handleInternalSideJob(rec, httptest.NewRequest("GET", "/internal/v1/engine/sidejob", nil)) + if rec.Code != 405 { + t.Fatalf("GET : %d", rec.Code) + } + + // La trace du bench dans la télémétrie. + var before uint64 + if recs := perfLog.snapshot(); len(recs) > 0 { + before = recs[len(recs)-1].Seq + } + if code, _ := post(handleInternalBenchPerf, "/internal/v1/bench/perf", + `{"prompt_n":2036,"prompt_ms":1018,"prompt_per_second":2000,"predicted_n":300,"predicted_per_second":50}`); code != 204 { + t.Fatalf("trace : %d", code) + } + recs := perfLog.snapshot() + if len(recs) == 0 || recs[len(recs)-1].Seq != before+1 { + t.Fatalf("trace absente (dernière avant : %d)", before) + } + r := recs[len(recs)-1] + if r.Kind != perfBench || !r.Complete || r.Total != 2036 || r.New != 2036 || r.Gen != 300 || r.TGtps != 50 || r.PPtps != 2000 || r.PPms != 1018 { + t.Fatalf("trace : %+v", r) + } + + t.Setenv(internalTokenEnv, "") + if facadeBench() { + t.Fatal("tenu sans jeton interne") + } +} diff --git a/internal/loki/facade_chat.go b/internal/loki/facade_chat.go index a897095..f2795c2 100644 --- a/internal/loki/facade_chat.go +++ b/internal/loki/facade_chat.go @@ -181,6 +181,8 @@ func withFacadeChat(path string, h http.HandlerFunc) http.HandlerFunc { held, unavailable = facadeStrata, "Strata indisponible : " case facadeTasksRoutes[path]: held, unavailable = facadeTasks, "tâches indisponibles : " + case facadeBenchRoutes[path]: + held, unavailable = facadeBench, "banc d'essai indisponible : " default: return h } diff --git a/internal/loki/llm_bench_job.go b/internal/loki/llm_bench_job.go index 09666ae..44b528f 100644 --- a/internal/loki/llm_bench_job.go +++ b/internal/loki/llm_bench_job.go @@ -81,11 +81,13 @@ var benchJob struct { canceled bool } -// benchRunning : un benchmark de ce processus est-il en cours ? +// benchRunning : un benchmark est-il en cours, de ce processus ou de la +// façade (LOKI_BENCH=rust, qui tient alors le bail du moteur) ? func benchRunning() bool { benchJob.mu.Lock() - defer benchJob.mu.Unlock() - return benchJob.running + running := benchJob.running + benchJob.mu.Unlock() + return running || benchLeased() } // benchRunner est runBench, remplaçable dans les tests. diff --git a/internal/loki/parity_bench_test.go b/internal/loki/parity_bench_test.go new file mode 100644 index 0000000..bfa2af1 --- /dev/null +++ b/internal/loki/parity_bench_test.go @@ -0,0 +1,375 @@ +//go:build linux + +package loki + +import ( + "crypto/sha256" + "encoding/json" + "fmt" + "io" + "net" + "net/http" + "net/http/httptest" + "os" + "regexp" + "strconv" + "strings" + "sync" + "testing" + "time" +) + +// parity_bench_test.go — journal de référence du banc d'essai (lot 27 de la +// migration Rust) : les quatre routes, chaque refus au lancement, le mode +// rapide et le mode complet (profondeur, tours, reprise du cache), les +// échecs d'une mesure, le niveau de raisonnement refusé puis rejoué, la +// profondeur partielle et l'annulation, contre le faux moteur des tests du +// bench (llm_bench_test.go). Chaque requête reçue par le moteur est +// enregistrée — les longs textes (corpus) réduits à leur taille et à leur +// empreinte. Rangé dans rust/loki-rs/tests/parity_bench/bench.json ; +// tests/bench.rs le rejoue par la façade. +// +// LOKI_PARITY_UPDATE=1 go test ./internal/loki -run TestPariteBench + +const parityBenchFile = "../../rust/loki-rs/tests/parity_bench/bench.json" + +// parityBenchEngine : le réglage du faux moteur pour une étape (remis à zéro +// à chaque étape qui en donne un). +type parityBenchEngine struct { + NCtx int `json:"n_ctx"` + Busy bool `json:"busy,omitempty"` + FailAt int `json:"fail_at,omitempty"` + NoTiming bool `json:"no_timing,omitempty"` + RejectEffort string `json:"reject_effort,omitempty"` + SlowHotMs int `json:"slow_hot_ms,omitempty"` + TimeoutMs int `json:"timeout_ms,omitempty"` // benchReqTimeout +} + +type parityBenchStep struct { + Name string `json:"name"` + Config map[string]string `json:"config,omitempty"` + Files map[string]string `json:"files,omitempty"` + Engine *parityBenchEngine `json:"engine,omitempty"` + Tune bool `json:"tune,omitempty"` + Generating bool `json:"generating,omitempty"` + Method string `json:"method"` + Path string `json:"path"` + Body string `json:"body,omitempty"` + Status int `json:"status"` + Out any `json:"out"` + // Again : un second lancement pendant le bench ; CancelAt : la phase + // (préfixe) à laquelle l'annuler. + Again bool `json:"again,omitempty"` + AgainOut any `json:"again_out,omitempty"` + CancelAt string `json:"cancel_at,omitempty"` + Wait bool `json:"wait,omitempty"` + Job any `json:"job,omitempty"` // état final du job + Calls []any `json:"calls"` // requêtes reçues par le moteur + Last any `json:"last"` // state/last_bench + Presets any `json:"presets"` // state/bench_presets +} + +var ( + parityBenchIDRe = regexp.MustCompile(`"b[0-9]+-[0-9]+"`) + parityBenchTRe = regexp.MustCompile(`"(elapsed_sec|at)":[0-9.e+-]+`) + parityBenchFPRe = regexp.MustCompile(`"fingerprint":"[0-9a-f]+"`) +) + +// parityBenchDigest : une valeur JSON dont les longs textes sont réduits à +// leur taille et à leur empreinte. +func parityBenchDigest(v any) any { + switch x := v.(type) { + case map[string]any: + out := map[string]any{} + for k, e := range x { + out[k] = parityBenchDigest(e) + } + return out + case []any: + out := make([]any, len(x)) + for i, e := range x { + out[i] = parityBenchDigest(e) + } + return out + case string: + if len(x) > 64 { + sum := sha256.Sum256([]byte(x)) + return fmt.Sprintf("<%d octets %x>", len(x), sum[:6]) + } + return x + } + return v +} + +func TestPariteBench(t *testing.T) { + home := testHome(t) + oldTimeout := benchReqTimeout + // Un état laissé par un autre test du paquet : ni niveau appris, ni bench + // précédent. + resetBench := func() { + effortFallbacks.Range(func(k, _ any) bool { + effortFallbacks.Delete(k) + return true + }) + benchJob.mu.Lock() + benchJob.running, benchJob.id, benchJob.mode, benchJob.phase = false, "", "", "" + benchJob.step, benchJob.steps, benchJob.cancel, benchJob.result = 0, 0, nil, nil + benchJob.err, benchJob.canceled = "", false + benchJob.mu.Unlock() + } + resetBench() + t.Cleanup(func() { + benchReqTimeout = oldTimeout + // Les niveaux de raisonnement appris ici ne valent que pour ce test. + effortFallbacks.Range(func(k, _ any) bool { + effortFallbacks.Delete(k) + return true + }) + }) + + // Le faux moteur : celui des tests du bench, derrière un journal de + // toutes les requêtes reçues. + var mu sync.Mutex + var calls []any + fake := &fakeBenchEngine{} + inner := http.Handler(nil) + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + b, _ := io.ReadAll(r.Body) + entry := map[string]any{"method": r.Method, "path": r.URL.Path} + if len(b) > 0 { + var body any + if json.Unmarshal(b, &body) == nil { + entry["body"] = parityBenchDigest(body) + } + } + mu.Lock() + calls = append(calls, entry) + h := inner + mu.Unlock() + r.Body = io.NopCloser(strings.NewReader(string(b))) + h.ServeHTTP(w, r) + })) + defer srv.Close() + port := strings.TrimPrefix(srv.URL, "http://127.0.0.1:") + l, err := net.Listen("tcp", "127.0.0.1:0") + if err != nil { + t.Fatal(err) + } + closed := strconv.Itoa(l.Addr().(*net.TCPAddr).Port) + l.Close() + + mask := func(s string) string { + s = strings.ReplaceAll(s, home, "{{HOME}}") + s = parityBenchIDRe.ReplaceAllString(s, `"{{ID}}"`) + s = parityBenchTRe.ReplaceAllString(s, `"$1":"{{T}}"`) + return parityBenchFPRe.ReplaceAllString(s, `"fingerprint":"{{FP}}"`) + } + unmask := func(s string) string { + s = strings.ReplaceAll(s, "{{PORT}}", port) + s = strings.ReplaceAll(s, "{{CLOSED}}", closed) + return strings.ReplaceAll(s, "{{HOME}}", home) + } + decode := func(raw string) any { + var v any + if raw == "" { + return nil + } + if err := json.Unmarshal([]byte(mask(raw)), &v); err != nil { + t.Fatalf("illisible : %q", raw) + } + return v + } + handlers := map[string]http.HandlerFunc{ + "/api/bench": handleBench, + "/api/bench/status": handleBenchStatus, + "/api/bench/cancel": handleBenchCancel, + "/api/bench/last": handleBenchLast, + } + call := func(method, path, body string) (int, any) { + rec := httptest.NewRecorder() + handlers[path](rec, httptest.NewRequest(method, path, strings.NewReader(body))) + return rec.Code, decode(rec.Body.String()) + } + status := func() map[string]any { + _, v := call("GET", "/api/bench/status", "") + return v.(map[string]any) + } + wait := func(name string) { + deadline := time.Now().Add(60 * time.Second) + for status()["running"] == true { + if time.Now().After(deadline) { + t.Fatalf("%s : le bench ne finit pas", name) + } + time.Sleep(5 * time.Millisecond) + } + } + + var steps []parityBenchStep + run := func(st parityBenchStep) { + for k, v := range st.Config { + if err := SetConfigKey(k, unmask(v)); err != nil { + t.Fatal(err) + } + } + files := map[string]string{} + for k, v := range st.Files { + files[k] = unmask(v) + } + parityApplyFiles(t, home, files) + if e := st.Engine; e != nil { + f := &fakeBenchEngine{nCtx: e.NCtx, busy: e.Busy, failAt: e.FailAt, noTiming: e.NoTiming, + rejectEffort: e.RejectEffort, slowHot: time.Duration(e.SlowHotMs) * time.Millisecond} + mu.Lock() + fake = f + inner = f.handler(t) + mu.Unlock() + benchReqTimeout = oldTimeout + if e.TimeoutMs > 0 { + benchReqTimeout = time.Duration(e.TimeoutMs) * time.Millisecond + } + } + mu.Lock() + calls = nil + mu.Unlock() + var lock *tuneLock + if st.Tune { + l, err := tuneLockAcquire("web", false) + if err != nil { + t.Fatal(err) + } + lock = l + } + if st.Generating { + conv.mu.Lock() + conv.Generating = true + conv.mu.Unlock() + } + st.Status, st.Out = call(st.Method, st.Path, unmask(st.Body)) + if st.Generating { + conv.mu.Lock() + conv.Generating = false + conv.mu.Unlock() + } + if lock != nil { + lock.release() + } + if st.Again { + _, st.AgainOut = call("POST", "/api/bench", "{}") + } + if st.CancelAt != "" { + deadline := time.Now().Add(30 * time.Second) + for { + p, _ := status()["phase"].(string) + if strings.HasPrefix(p, st.CancelAt) { + break + } + if time.Now().After(deadline) { + t.Fatalf("%s : phase %q jamais atteinte", st.Name, st.CancelAt) + } + time.Sleep(2 * time.Millisecond) + } + // Le moteur a reçu la requête de cette phase avant l'annulation. + for { + mu.Lock() + n := len(fake.calls) + mu.Unlock() + if n >= 4 { + break + } + time.Sleep(2 * time.Millisecond) + } + call("POST", "/api/bench/cancel", "") + } + if st.Wait || st.CancelAt != "" { + wait(st.Name) + st.Job = status() + } + mu.Lock() + st.Calls = append([]any{}, calls...) + mu.Unlock() + st.Last = decode(string(getBytes(bkState, "last_bench"))) + st.Presets = decode(string(getBytes(bkState, "bench_presets"))) + steps = append(steps, st) + } + + preset := "MODEL=/models/qwen.gguf\nPORT={{PORT}}\nCTX=16384\nTOP_K=40\nMIN_P=0.05\nREASONING=off\nCACHE_ISOLATE=off\n" + base := map[string]string{"MODEL": "/models/qwen.gguf", "PORT": "{{PORT}}", "CTX": "16384", + "TOP_K": "40", "MIN_P": "0.05", "REASONING": "off", "CACHE_ISOLATE": "off"} + for _, st := range []parityBenchStep{ + {Name: "statut sans bench", Config: base, Files: map[string]string{"presets/qwen.env": preset}, + Engine: &parityBenchEngine{NCtx: 16384}, Method: "GET", Path: "/api/bench/status"}, + {Name: "dernier bench : aucun", Method: "GET", Path: "/api/bench/last"}, + {Name: "lancement en GET", Method: "GET", Path: "/api/bench"}, + {Name: "annulation en GET", Method: "GET", Path: "/api/bench/cancel"}, + {Name: "annulation sans bench", Method: "POST", Path: "/api/bench/cancel"}, + {Name: "preset externe", Config: map[string]string{"EXTERNAL": "1"}, Method: "POST", Path: "/api/bench", Body: `{}`}, + {Name: "optimisation en cours", Config: map[string]string{"EXTERNAL": ""}, Tune: true, Method: "POST", Path: "/api/bench", Body: `{}`}, + {Name: "modèle pas prêt", Config: map[string]string{"PORT": "{{CLOSED}}"}, Method: "POST", Path: "/api/bench", Body: `{}`}, + {Name: "génération en cours", Config: map[string]string{"PORT": "{{PORT}}"}, Generating: true, Method: "POST", Path: "/api/bench", Body: `{}`}, + {Name: "moteur occupé par un autre", Engine: &parityBenchEngine{NCtx: 16384, Busy: true}, + Method: "POST", Path: "/api/bench", Body: `{}`, Wait: true}, + {Name: "mode rapide", Engine: &parityBenchEngine{NCtx: 16384}, Method: "POST", Path: "/api/bench", Body: `{}`, Wait: true}, + {Name: "dernier bench", Method: "GET", Path: "/api/bench/last"}, + {Name: "mode complet", Engine: &parityBenchEngine{NCtx: 16384}, Method: "POST", Path: "/api/bench", Body: `{"mode":" FULL "}`, Wait: true}, + {Name: "mode complet, contexte trop petit", Engine: &parityBenchEngine{NCtx: 4096}, + Method: "POST", Path: "/api/bench", Body: `{"mode":"full"}`, Wait: true}, + {Name: "contexte inconnu du moteur : celui du preset", Engine: &parityBenchEngine{NCtx: 0}, + Method: "POST", Path: "/api/bench", Body: `{"mode":"full"}`, Wait: true}, + {Name: "paramètres bornés", Engine: &parityBenchEngine{NCtx: 65536}, + Method: "POST", Path: "/api/bench", Body: `{"prompt":99999,"n":9999,"mode":"quick"}`, Wait: true}, + {Name: "petits paramètres", Engine: &parityBenchEngine{NCtx: 16384}, + Method: "POST", Path: "/api/bench", Body: `{"prompt":100,"n":8}`, Wait: true}, + {Name: "corps illisible : défauts", Engine: &parityBenchEngine{NCtx: 16384}, + Method: "POST", Path: "/api/bench", Body: `{"prompt":"beaucoup"}`, Wait: true}, + {Name: "le préchauffage échoue", Engine: &parityBenchEngine{NCtx: 16384, FailAt: 1}, + Method: "POST", Path: "/api/bench", Body: `{}`, Wait: true}, + {Name: "la ligne courte échoue", Engine: &parityBenchEngine{NCtx: 16384, FailAt: 2}, + Method: "POST", Path: "/api/bench", Body: `{}`, Wait: true}, + {Name: "le prefill à froid échoue", Engine: &parityBenchEngine{NCtx: 16384, FailAt: 3}, + Method: "POST", Path: "/api/bench", Body: `{"mode":"full"}`, Wait: true}, + {Name: "un tour en profondeur échoue", Engine: &parityBenchEngine{NCtx: 16384, FailAt: 5}, + Method: "POST", Path: "/api/bench", Body: `{"mode":"full"}`, Wait: true}, + {Name: "sans timings", Engine: &parityBenchEngine{NCtx: 16384, NoTiming: true}, + Method: "POST", Path: "/api/bench", Body: `{}`, Wait: true}, + {Name: "niveau de raisonnement refusé puis rejoué", + Config: map[string]string{"REASONING": "", "REASONING_EFFORT": "high"}, + Engine: &parityBenchEngine{NCtx: 16384, RejectEffort: "high"}, + Method: "POST", Path: "/api/bench", Body: `{}`, Wait: true}, + {Name: "niveau appris : envoyé d'emblée", Engine: &parityBenchEngine{NCtx: 16384, RejectEffort: "high"}, + Method: "POST", Path: "/api/bench", Body: `{}`, Wait: true}, + {Name: "budget d'une requête dépassé : profondeur partielle", + Config: map[string]string{"REASONING_EFFORT": "", "REASONING": "off"}, + Engine: &parityBenchEngine{NCtx: 16384, SlowHotMs: 400, TimeoutMs: 150}, + Method: "POST", Path: "/api/bench", Body: `{"mode":"full"}`, Wait: true}, + {Name: "annulation pendant la profondeur", Engine: &parityBenchEngine{NCtx: 16384, SlowHotMs: 20000}, + Method: "POST", Path: "/api/bench", Body: `{"mode":"full"}`, Again: true, CancelAt: "tour 1/"}, + {Name: "preset Strata", Config: map[string]string{"ENGINE": "strata"}, Engine: &parityBenchEngine{NCtx: 16384}, + Method: "POST", Path: "/api/bench", Body: `{}`, Wait: true}, + {Name: "dernier bench, inchangé", Config: map[string]string{"ENGINE": ""}, Method: "GET", Path: "/api/bench/last"}, + } { + run(st) + } + + got, err := parityJSON(map[string]any{"steps": steps}) + if err != nil { + t.Fatal(err) + } + if os.Getenv("LOKI_PARITY_UPDATE") == "1" { + if err := os.MkdirAll("../../rust/loki-rs/tests/parity_bench", 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(parityBenchFile, got, 0o644); err != nil { + t.Fatal(err) + } + return + } + want, err := os.ReadFile(parityBenchFile) + if err != nil { + t.Fatalf("journal de référence absent (%v) : LOKI_PARITY_UPDATE=1 pour le créer", err) + } + if string(got) != string(want) { + t.Errorf("le banc d'essai ne produit plus le journal rangé dans %s.\n"+ + "Si c'est voulu : LOKI_PARITY_UPDATE=1 go test ./internal/loki -run TestPariteBench,\n"+ + "puis cargo test --test bench dit si le port Rust suit.\n--- obtenu ---\n%s", parityBenchFile, got) + } +} diff --git a/internal/loki/web_internal.go b/internal/loki/web_internal.go index 6d715de..1764e1a 100644 --- a/internal/loki/web_internal.go +++ b/internal/loki/web_internal.go @@ -129,6 +129,10 @@ func withInternalAPI(next http.Handler) http.Handler { handleInternalProjectSwitch(w, r) case "/internal/v1/project/drop-convs": handleInternalProjectDropConvs(w, r) + case "/internal/v1/engine/sidejob": + handleInternalSideJob(w, r) + case "/internal/v1/bench/perf": + handleInternalBenchPerf(w, r) case "/internal/v1/tasks/run": handleInternalTaskRun(w, r) case "/internal/v1/push/send": diff --git a/internal/loki/web_internal_lease.go b/internal/loki/web_internal_lease.go index ee1eb1c..0e8979e 100644 --- a/internal/loki/web_internal_lease.go +++ b/internal/loki/web_internal_lease.go @@ -6,6 +6,7 @@ import ( "io" "net/http" "os" + "sync/atomic" "time" ) @@ -43,18 +44,29 @@ type extLeaseState struct { stop bool ttl time.Duration timer *time.Timer + // bench : le bail d'un benchmark de la façade (LOKI_BENCH=rust) — les + // autres reçoivent errBenchBusy, et changer de discussion ne le rompt pas. + bench bool } // extLeaseClearLocked oublie le bail de la façade. c.mu détenu. +// benchLeaseHeld : un bail « bench » de la façade est tenu. Lu sans le verrou +// de la conversation (benchRunning est appelé par des chemins qui le tiennent +// déjà, le préchauffage par exemple). +var benchLeaseHeld atomic.Bool + func (c *Conversation) extLeaseClearLocked() { if c.extLease != nil { + if c.extLease.bench { + benchLeaseHeld.Store(false) + } c.extLease.timer.Stop() c.extLease = nil } } // extLeaseAcquire prend le verrou de génération pour la façade. -func (c *Conversation) extLeaseAcquire(ttl time.Duration) (string, error) { +func (c *Conversation) extLeaseAcquire(ttl time.Duration, bench bool) (string, error) { if err := tuneGuard(); err != nil { return "", err } @@ -67,9 +79,13 @@ func (c *Conversation) extLeaseAcquire(ttl time.Duration) (string, error) { if err != nil { return "", err } - st := &extLeaseState{id: fmt.Sprintf("%x", b), ttl: ttl} + st := &extLeaseState{id: fmt.Sprintf("%x", b), ttl: ttl, bench: bench} st.timer = time.AfterFunc(ttl, func() { c.extLeaseExpire(st) }) c.Generating = true + if bench { + c.benching, c.leaseWhy = true, errBenchBusy + benchLeaseHeld.Store(true) + } c.cancel = func() { c.mu.Lock() st.stop = true @@ -88,6 +104,10 @@ func (c *Conversation) extLeaseExpire(st *extLeaseState) { } c.extLease = nil c.Generating, c.cancel = false, nil + if st.bench { + c.benching, c.leaseWhy = false, nil + benchLeaseHeld.Store(false) + } fmt.Fprintf(os.Stderr, "[lease] bail de la façade expiré sans renouvellement (%s) : verrou du moteur rendu\n", st.ttl) } @@ -110,10 +130,16 @@ func (c *Conversation) extLeaseRelease(id string) { if c.extLease == nil || c.extLease.id != id { return } + if c.extLease.bench { + c.benching, c.leaseWhy = false, nil + } c.extLeaseClearLocked() c.Generating, c.cancel = false, nil } +// benchLeased : un benchmark de la façade tient-il le verrou ? +func benchLeased() bool { return benchLeaseHeld.Load() } + func readLeaseBody(w http.ResponseWriter, r *http.Request, v any) bool { if r.Method != http.MethodPost { w.Header().Set("Allow", http.MethodPost) @@ -134,7 +160,8 @@ func readLeaseBody(w http.ResponseWriter, r *http.Request, v any) bool { func handleInternalLease(w http.ResponseWriter, r *http.Request) { var in struct { - TTLms int64 `json:"ttl_ms"` + TTLms int64 `json:"ttl_ms"` + Why string `json:"why"` // "bench" : le bail d'un benchmark } if !readLeaseBody(w, r, &in) { return @@ -143,7 +170,7 @@ func handleInternalLease(w http.ResponseWriter, r *http.Request) { if in.TTLms > 0 { ttl = min(max(time.Duration(in.TTLms)*time.Millisecond, extLeaseMinTTL), extLeaseMaxTTL) } - id, err := conv.extLeaseAcquire(ttl) + id, err := conv.extLeaseAcquire(ttl, in.Why == "bench") if err != nil { if err == ErrBusy { err = conv.busyReason() // dit QUI occupe le moteur diff --git a/internal/loki/web_internal_lease_test.go b/internal/loki/web_internal_lease_test.go index 1d0c442..d8ae60b 100644 --- a/internal/loki/web_internal_lease_test.go +++ b/internal/loki/web_internal_lease_test.go @@ -86,7 +86,7 @@ func TestBailInterneVerrouPartage(t *testing.T) { func TestBailInterneNeLibereQueLeSien(t *testing.T) { testHome(t) libérerVerrou(t) - id, err := conv.extLeaseAcquire(time.Minute) + id, err := conv.extLeaseAcquire(time.Minute, false) if err != nil { t.Fatal(err) } @@ -110,7 +110,7 @@ func TestBailInterneNeLibereQueLeSien(t *testing.T) { func TestBailInterneExpire(t *testing.T) { testHome(t) libérerVerrou(t) - id, err := conv.extLeaseAcquire(50 * time.Millisecond) + id, err := conv.extLeaseAcquire(50*time.Millisecond, false) if err != nil { t.Fatal(err) } diff --git a/rust/loki-rs/src/api.rs b/rust/loki-rs/src/api.rs index f212fdf..ba14a1d 100644 --- a/rust/loki-rs/src/api.rs +++ b/rust/loki-rs/src/api.rs @@ -65,6 +65,8 @@ pub struct Ctx { /// Tâches planifiées (routes, planificateur, tâches script) tenues par la /// façade (`LOKI_TASKS=rust`). pub tasks: Option>, + /// Banc d'essai tenu par la façade (`LOKI_BENCH=rust`). + pub bench: Option>, /// Jeton de l'API interne : une requête qui le porte vient du Go. pub internal_token: String, } @@ -90,6 +92,7 @@ impl Ctx { || self.engineimg.is_some() || self.strata.is_some() || self.tasks.is_some() + || self.bench.is_some() } /// La requête vient du Go (jeton interne valide) : déjà authentifiée par diff --git a/rust/loki-rs/src/bench/corpus.txt b/rust/loki-rs/src/bench/corpus.txt new file mode 100644 index 0000000..8a61c6f --- /dev/null +++ b/rust/loki-rs/src/bench/corpus.txt @@ -0,0 +1,10 @@ +In the early hours of an October morning, Camille walked along the canal, watching the cargo barges slip past the iron bridge that spanned the water. She thought about the meeting she had skipped, the unanswered messages on her phone, the way the city always seemed to forget her name after summer ended. Three streets away, a pâtisserie opened its shutters and the smell of warm butter mixed with diesel exhaust from the waiting bus. +Pendant ce temps, à Marseille, un chercheur en biologie marine prépare son matériel pour une plongée. Il étudie les herbiers de posidonie, ces prairies sous-marines vieilles de plusieurs milliers d'années qui stockent autant de carbone qu'une forêt amazonienne. Le bateau quitte le port à six heures vingt-trois. +Quantum computers, properly engineered, can solve certain classes of problems exponentially faster than classical machines. The catch is that decoherence ruins everything. Engineers use dilution refrigerators to drop superconducting qubits to fifteen millikelvin, colder than deep space. The wires connecting the chip to room-temperature electronics must dissipate almost no heat, or the qubit state collapses before any useful computation finishes. +Le boulanger lève la pâte à quatre heures. Il regarde la balance numérique en plissant les yeux : six cent vingt-trois grammes, presque le compte. Son chien dort sur le tapis de farine près du four. Dehors, deux chats se disputent un poisson abandonné par le pêcheur de nuit. +Consider a recursive descent parser written in Go. The lexer emits tokens; the parser consumes them and produces an abstract syntax tree. Error recovery is hard: after a syntax error, the parser must resynchronize at a known boundary—a semicolon, a closing brace—without losing track of subsequent diagnostics. Tree-sitter solves this with incremental parsing and a glr-like algorithm. +Le philosophe stoïcien disait : "Ce qui nous trouble, ce n'est pas ce qui nous arrive, mais l'opinion que nous nous en faisons." Vingt siècles plus tard, la phrase apparaît dans un livre de poche au rayon développement personnel d'une librairie d'aéroport, à côté d'un roman policier suédois. +Mitochondria descended from ancient bacteria engulfed by archaeal cells roughly two billion years ago. They still keep their own ring of DNA, separate from the nuclear genome. Mutations in mitochondrial DNA accumulate with age and have been implicated in everything from Parkinson's disease to ordinary muscle fatigue. Yet they remain stubbornly difficult to repair therapeutically because each cell contains hundreds. +La marée descend lentement, exposant des rochers couverts d'huîtres et d'algues vertes. Un héron immobile surveille les flaques laissées par l'eau. Plus loin, deux enfants courent avec un cerf-volant rouge qui refuse de monter à cause de l'humidité dans la voile. +Compilers translate high-level languages into machine code through several intermediate representations. LLVM IR sits in the middle: typed, mostly static-single-assignment, suitable for both aggressive optimization and direct lowering to x86 or ARM. The optimizer runs dozens of passes—dead code elimination, loop-invariant code motion, induction variable simplification—each touching the IR in carefully ordered ways. +Le cuisinier ferme les yeux pour goûter la sauce. Trop salée. Il ajoute une pomme de terre crue coupée en quartiers, sachant qu'elle absorbera l'excès en mijotant vingt minutes. Sa grand-mère lui a appris ce geste un dimanche de novembre il y a très longtemps. diff --git a/rust/loki-rs/src/bench/job.rs b/rust/loki-rs/src/bench/job.rs new file mode 100644 index 0000000..64603d8 --- /dev/null +++ b/rust/loki-rs/src/bench/job.rs @@ -0,0 +1,684 @@ +//! Le banc d'essai en tâche de fond (`llm_bench_job.go`), ses quatre routes +//! et l'enregistrement des mesures (`llm_bench.go`, « enregistrement ») : +//! +//! - `POST /api/bench` lance (202 + id) ; refus clairs avant de lancer ; +//! - `GET /api/bench/status` : la phase en cours, puis le résultat ou l'erreur ; +//! - `POST /api/bench/cancel` annule ; +//! - `GET /api/bench/last` : la dernière mesure enregistrée. +//! +//! Pendant la mesure, le bench tient le verrou de génération du Go par le +//! bail du moteur, motif « bench » : un message, une tâche ou un client /v1 +//! reçoivent le refus du bench. Un stop du chat (ou un Reset) côté Go +//! interrompt le bail : le bench est annulé. + +use std::collections::{BTreeMap, HashMap}; +use std::path::PathBuf; +use std::sync::atomic::{AtomicU64, Ordering}; +use std::sync::{Arc, Mutex}; +use std::time::{Duration, Instant}; + +use http_body_util::{BodyExt, Limited}; +use hyper::body::Incoming; +use hyper::{Method, Request, Response, StatusCode}; +use serde::{Deserialize, Serialize}; +use serde_json::{Value, json}; +use tokio::sync::watch; + +use super::{BenchResult, Corpora, Engine, Opts, Setup}; +use crate::api::Ctx; +use crate::auth::json as send_json; +use crate::chat::lease::EngineLease; +use crate::chat::wire::{EffortFallbacks, log_effort_fallback}; +use crate::gojson::{self, Kind}; +use crate::proxy::Body; +use crate::store::BK_STATE; + +type Cfg = HashMap; + +const LAST_KEY: &str = "last_bench"; +const PRESETS_KEY: &str = "bench_presets"; +const BENCH_REQ: &str = + r#"struct { Mode string "json:\"mode\""; Prompt int "json:\"prompt\""; N int "json:\"n\"" }"#; + +/// Le job : le bench en cours ou le dernier terminé (un seul à la fois). +#[derive(Default)] +struct JobState { + seq: i64, + running: bool, + /// Lancement en cours (le bail se prend hors du verrou). + starting: bool, + id: String, + mode: String, + phase: String, + step: i64, + steps: i64, + started: Option, + finished: Option, + cancel: Option>, + result: Option, + err: String, + canceled: bool, +} + +/// Le banc d'essai tenu par la façade. +pub struct Bench { + pub home: PathBuf, + state: Mutex, + /// Budget d'une requête de chat (`benchReqTimeout`). + pub req_timeout: Mutex, + /// Durée du bail du moteur (renouvelé au tiers). + pub lease_ttl: Duration, + /// Niveaux de raisonnement appris quand la façade ne tient pas la + /// conversation (sinon, ceux de sa conversation). + effort: EffortFallbacks, + client: reqwest::Client, + /// Le service du moteur (PID, pour l'indication « possible thrash »). + engine: crate::engine::Paths, +} + +impl Bench { + pub fn new(engine: crate::engine::Paths) -> Self { + Bench { + home: engine.home.clone(), + state: Mutex::new(JobState::default()), + req_timeout: Mutex::new(super::REQ_TIMEOUT), + lease_ttl: Duration::from_secs(30), + effort: EffortFallbacks::new(), + client: { + crate::chat::engine::install_crypto(); + reqwest::Client::builder().build().unwrap_or_default() + }, + engine, + } + } + + fn lock(&self) -> std::sync::MutexGuard<'_, JobState> { + self.state.lock().unwrap_or_else(|e| e.into_inner()) + } + + /// `benchRunning`. + pub fn running(&self) -> bool { + self.lock().running + } + + /// `benchCancel` : false si aucun ne tournait. + pub fn cancel(&self) -> bool { + let st = self.lock(); + match (&st.cancel, st.running) { + (Some(c), true) => { + let _ = c.send(true); + true + } + _ => false, + } + } + + /// `benchStatus`. + pub fn status(&self) -> Value { + let st = self.lock(); + let mut out = json!({ "ok": true, "running": st.running, "id": st.id, "mode": st.mode }); + if st.id.is_empty() { + return out; + } + let o = out.as_object_mut().expect("objet"); + o.insert("phase".into(), json!(st.phase)); + o.insert("step".into(), json!(st.step)); + o.insert("steps".into(), json!(st.steps)); + let end = if st.running { + Some(Instant::now()) + } else { + st.finished + }; + let elapsed = match (st.started, end) { + (Some(s), Some(e)) => e.saturating_duration_since(s).as_secs_f64(), + _ => 0.0, + }; + o.insert("elapsed_sec".into(), json!(elapsed)); + if !st.running { + if let Some(r) = &st.result { + o.insert( + "result".into(), + serde_json::to_value(r).unwrap_or(Value::Null), + ); + o.insert("saved".into(), json!(savable(r))); + } + if !st.err.is_empty() { + o.insert("error".into(), json!(st.err)); + o.insert("canceled".into(), json!(st.canceled)); + } + } + out + } + + /// `benchStart` : le statut HTTP et la réponse. + pub async fn start(self: &Arc, ctx: &Ctx, opts: Opts) -> (StatusCode, Value) { + let store = &ctx.store; + let fail = |code: StatusCode, e: &str| (code, json!({ "ok": false, "error": e })); + let cfg = store.config().await; + if crate::engstatus::is_external(&cfg) { + return fail( + StatusCode::BAD_REQUEST, + "benchmark indisponible : le preset actif est une API externe", + ); + } + if let Some(e) = crate::backends::tune_busy(store).await { + return fail(StatusCode::CONFLICT, &e); + } + if !crate::engstatus::health_check(&cfg).await { + return fail( + StatusCode::SERVICE_UNAVAILABLE, + crate::tasks::ERR_MODEL_LOADING, + ); + } + { + let mut st = self.lock(); + if st.running || st.starting { + return fail(StatusCode::CONFLICT, "un benchmark tourne déjà"); + } + st.starting = true; + } + let lease = match EngineLease::acquire_for(store.clone(), self.lease_ttl, "bench").await { + Ok(l) => l, + Err(e) => { + self.lock().starting = false; + return fail(StatusCode::CONFLICT, &e.to_string()); + } + }; + let (tx, rx) = watch::channel(false); + let id = { + let mut st = self.lock(); + st.seq += 1; + let id = format!("b{}-{}", unix_secs(), st.seq); + *st = JobState { + seq: st.seq, + running: true, + starting: false, + id: id.clone(), + mode: opts.mode.clone(), + phase: "démarrage".into(), + step: 0, + steps: 0, + started: Some(Instant::now()), + finished: None, + cancel: Some(tx), + result: None, + err: String::new(), + canceled: false, + }; + id + }; + let (me, ctx2, id2) = (self.clone(), ctx.clone(), id.clone()); + tokio::spawn(async move { me.job(ctx2, id2, opts, rx, lease).await }); + (StatusCode::ACCEPTED, json!({ "ok": true, "id": id })) + } + + /// Le bench en tâche de fond : le verrou est rendu AVANT d'annoncer la + /// fin — l'interface qui lit « terminé » et envoie aussitôt un message ne + /// doit pas se voir refuser « benchmark en cours ». + async fn job( + self: Arc, + ctx: Ctx, + id: String, + opts: Opts, + rx: watch::Receiver, + mut lease: EngineLease, + ) { + let me = self.clone(); + let id2 = id.clone(); + let progress = move |phase: &str, step: i64, steps: i64| { + let mut st = me.lock(); + if st.id == id2 { + st.phase = phase.to_string(); + st.step = step; + st.steps = steps; + } + }; + let run = self.run_bench(&ctx, opts, rx.clone(), &progress); + tokio::pin!(run); + let mut interrupted = false; + let res = loop { + tokio::select! { + r = &mut run => break r, + _ = lease.interrupted(), if !interrupted => { + // Stop du chat, Reset ou bail perdu côté Go : le bench + // s'arrête (le verrou n'est plus, ou ne sera plus, à lui). + interrupted = true; + if let Some(c) = &self.lock().cancel { + let _ = c.send(true); + } + } + } + }; + let canceled = *rx.borrow(); + lease.release().await; + let mut st = self.lock(); + if st.id != id { + return; + } + st.running = false; + st.finished = Some(Instant::now()); + st.cancel = None; + match res { + Ok(r) => st.result = Some(r), + Err(e) => { + st.result = None; + st.canceled = canceled; + st.err = if canceled { + "benchmark annulé".into() + } else { + e + }; + } + } + } + + /// `runBench` : mesure le moteur local et enregistre le résultat s'il + /// est complet. + async fn run_bench( + &self, + ctx: &Ctx, + opts: Opts, + cancel: watch::Receiver, + progress: &(dyn Fn(&str, i64, i64) + Send + Sync), + ) -> Result { + let store = &ctx.store; + let cfg = store.config().await; + if crate::engstatus::is_external(&cfg) { + return Err("benchmark indisponible : le preset actif est une API externe".into()); + } + if crate::strata::is_strata(&cfg) { + return Err("benchmark indisponible : le preset actif tourne sur Strata (le débit s'affiche sous chaque réponse)".into()); + } + if let Some(e) = crate::backends::tune_busy(store).await { + return Err(e); + } + let port = crate::engstatus::llm_port_raw(&cfg); + if !crate::engstatus::health_check(&cfg).await { + return Err(format!("serveur injoignable sur :{port}")); + } + let arg_env: HashMap = super::ARG_ENV + .iter() + .filter_map(|k| { + std::env::var(k) + .ok() + .filter(|v| !v.is_empty()) + .map(|v| (k.to_string(), v)) + }) + .collect(); + let effort: &EffortFallbacks = match &ctx.chat { + Some(c) => &c.shared.effort, + None => &self.effort, + }; + let mut setup: Setup = super::setup_from(&cfg, &arg_env, effort); + setup.hybrid = model_hybrid(ctx, &cfg).await; + let route = store + .internal_json( + Method::GET, + "/internal/v1/oai/route?method=POST&path=%2Fv1%2Fchat%2Fcompletions", + None, + ) + .await + .unwrap_or(Value::Null); + setup.side_slot = route.get("side_live").and_then(Value::as_bool) == Some(true); + let served = Arc::new(AtomicU64::new(0)); + let engine = Engine { + base: format!("http://localhost:{port}"), + key: route + .get("key") + .and_then(Value::as_str) + .unwrap_or("") + .to_string(), + client: self.client.clone(), + cancel, + req_timeout: *self.req_timeout.lock().unwrap_or_else(|e| e.into_inner()), + served: served.clone(), + }; + let cfg2 = cfg.clone(); + let learn = move |want: &str, got: &str| { + effort.remember(&cfg2, want, got); + log_effort_fallback(want, got); + }; + let paths = self.engine.clone(); + let io = move || engine_io_sample(&paths); + // Le bench prend le slot de la conversation : une fois fini, le Go + // l'efface si c'est sans risque (engineSideJob). + let side = store + .internal_post("/internal/v1/engine/sidejob", &json!({})) + .await + .ok() + .and_then(|(_, v)| v.get("id").and_then(Value::as_i64)); + let res = super::run( + &engine, + opts, + &mut setup, + &Corpora::default_set(), + progress, + &learn, + &io, + ) + .await; + if let Some(id) = side { + let _ = store + .internal_post( + "/internal/v1/engine/sidejob", + &json!({ "end": id, "served": served.load(Ordering::SeqCst) > 0 }), + ) + .await; + } + let res = res?; + let _ = store + .internal_post( + "/internal/v1/bench/perf", + &json!({ + "prompt_n": res.prompt_n, + "prompt_ms": res.prompt_ms, + "prompt_per_second": res.prompt_per_second, + "predicted_n": res.predicted_n, + "predicted_per_second": res.predicted_per_second, + }), + ) + .await; + if savable(&res) { + save_last(ctx, &res, &cfg).await; + self.save_for_active_preset(ctx, &res, &cfg).await; + } + Ok(res) + } + + /// `saveBenchForActivePreset` : sous l'id du preset actif ; sans effet si + /// aucun ne correspond. + async fn save_for_active_preset(&self, ctx: &Ctx, res: &BenchResult, cfg: &Cfg) { + let store = &ctx.store; + let order = crate::presets::parse_preset_order( + store + .get(BK_STATE, crate::presets::PRESET_ORDER_KEY) + .await + .ok() + .flatten() + .as_deref(), + ); + let Ok(list) = crate::presets::list_presets(&self.home, cfg, &order) else { + return; + }; + let Some(id) = list.into_iter().find(|p| p.active).map(|p| p.id) else { + return; + }; + let mut m = load_bench_store(ctx).await; + m.insert(id, saved_bench(res, cfg)); + let _ = store + .put(BK_STATE, PRESETS_KEY, gojson::marshal(&m).into_bytes()) + .await; + } +} + +fn unix_secs() -> i64 { + std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .map_or(0, |d| d.as_secs() as i64) +} + +/// `benchSavable` : seul un résultat COMPLET est enregistré. +pub fn savable(res: &BenchResult) -> bool { + res.depth.as_ref().is_none_or(|d| d.partial.is_empty()) +} + +/// `savedBench`. +#[derive(Clone, Debug, Default, PartialEq, Serialize, Deserialize)] +pub struct Saved { + #[serde(default)] + pub result: BenchResult, + #[serde(default)] + pub model: String, + #[serde(default)] + pub at: i64, + #[serde(default, skip_serializing_if = "String::is_empty")] + pub fingerprint: String, +} + +/// `newSavedBench` : l'indication du moment (thrash) n'est pas enregistrée. +pub fn saved_bench(res: &BenchResult, cfg: &Cfg) -> Saved { + let mut r = res.clone(); + if let Some(d) = &mut r.depth { + d.hint.clear(); + } + Saved { + result: r, + model: crate::paths::go_base(cfg.get("MODEL").map(String::as_str).unwrap_or("")), + at: unix_secs(), + fingerprint: crate::presets_api::bench_fingerprint(cfg, cfg), + } +} + +/// `saveLastBench`. +async fn save_last(ctx: &Ctx, res: &BenchResult, cfg: &Cfg) { + let _ = ctx + .store + .put( + BK_STATE, + LAST_KEY, + gojson::marshal(&saved_bench(res, cfg)).into_bytes(), + ) + .await; +} + +/// `loadBenchStore` : les mesures par preset (vide s'il n'y en a pas). +async fn load_bench_store(ctx: &Ctx) -> BTreeMap { + ctx.store + .get(BK_STATE, PRESETS_KEY) + .await + .ok() + .flatten() + .and_then(|b| serde_json::from_slice::>>(&b).ok()) + .flatten() + .unwrap_or_default() +} + +/// `loadLastBench`. +async fn load_last(ctx: &Ctx) -> Option { + let b = ctx.store.get(BK_STATE, LAST_KEY).await.ok().flatten()?; + serde_json::from_slice(&b).ok() +} + +/// Le modèle du preset est-il hybride (reprise du cache aux points de +/// contrôle) ? +async fn model_hybrid(ctx: &Ctx, cfg: &Cfg) -> bool { + let m = cfg.get("MODEL").map(|s| s.trim()).unwrap_or(""); + if m.is_empty() { + return false; + } + let stored = crate::gguf::parse_stored_model_dirs( + ctx.store + .get(BK_STATE, crate::gguf::STATE_MODEL_DIRS_KEY) + .await + .ok() + .flatten() + .as_deref(), + ); + let env = std::env::var(crate::gguf::MODEL_DIRS_ENV).ok(); + let Ok(p) = crate::gguf::resolve_serve_model_path(m, &ctx.engine.home, &stored, env.as_deref()) + else { + return false; + }; + match crate::gguf::gguf_meta(&p) { + Ok(g) => crate::serve::ckpt::gguf_hybrid(Some(&g)), + Err(_) => false, + } +} + +/// `engineIOSample` : octets lus du disque et RSS du moteur ; `None` si le +/// PID ou /proc est illisible. +fn engine_io_sample(paths: &crate::engine::Paths) -> Option<(i64, i64)> { + let pid = crate::engine::read_pid(&paths.pid()); + if pid <= 0 { + return None; + } + let io = std::fs::read_to_string(format!("/proc/{pid}/io")).ok()?; + for line in io.split('\n') { + if let Some(v) = line.strip_prefix("read_bytes:") { + let n = v.trim().parse::().ok()?; + let rss = std::fs::read_to_string(format!("/proc/{pid}/status")) + .map(|s| crate::engstatus::parse_vm_rss(&s)) + .unwrap_or(0); + return Some((n, rss)); + } + } + None +} + +// --------------------------------------------------------------------------- +// Routes +// --------------------------------------------------------------------------- + +/// Les routes servies par ce module. +pub fn handles(path: &str) -> bool { + matches!( + path, + "/api/bench" | "/api/bench/status" | "/api/bench/cancel" | "/api/bench/last" + ) +} + +/// `postOnly`. +fn post_only(method: &Method) -> Option> { + if method == Method::POST { + return None; + } + let mut r = send_json( + StatusCode::METHOD_NOT_ALLOWED, + &json!({ "ok": false, "error": format!("méthode {method} refusée : POST attendu") }), + ); + r.headers_mut().insert( + hyper::header::ALLOW, + hyper::header::HeaderValue::from_static("POST"), + ); + Some(r) +} + +/// `from_go` : remise par le Go (tunnel), déjà authentifiée par lui. +pub async fn serve( + ctx: &Ctx, + bench: &Arc, + req: Request, + from_go: bool, +) -> Response { + let (parts, body) = req.into_parts(); + if !from_go && let Err(refus) = ctx.guard(&parts.headers).await { + return *refus; + } + match parts.uri.path() { + "/api/bench" => { + if let Some(r) = post_only(&parts.method) { + return r; + } + let raw = Limited::new(body, 1 << 20) + .collect() + .await + .map(|b| b.to_bytes().to_vec()) + .unwrap_or_default(); + // Les erreurs de décodage sont ignorées (champs facultatifs). + let d = gojson::decode_struct( + &raw, + BENCH_REQ, + &[("mode", Kind::Str), ("prompt", Kind::Int), ("n", Kind::Int)], + ); + let mut opts = Opts { + mode: super::MODE_QUICK.into(), + prompt: 2000, + predict: 300, + depth: 0, + }; + if d.str("mode").trim().eq_ignore_ascii_case(super::MODE_FULL) { + opts.mode = super::MODE_FULL.into(); + } + if d.int("prompt") > 0 { + opts.prompt = d.int("prompt").min(32768); + } + if d.int("n") > 0 { + opts.predict = d.int("n").min(4096); + } + let (code, out) = bench.start(ctx, opts).await; + send_json(code, &out) + } + "/api/bench/status" => send_json(StatusCode::OK, &bench.status()), + "/api/bench/cancel" => { + if let Some(r) = post_only(&parts.method) { + return r; + } + send_json(StatusCode::OK, &json!({ "ok": bench.cancel() })) + } + "/api/bench/last" => match load_last(ctx).await { + None => send_json(StatusCode::OK, &json!({ "ok": false })), + Some(sb) => send_json( + StatusCode::OK, + &json!({ "ok": true, "result": sb.result, "model": sb.model, "at": sb.at }), + ), + }, + _ => send_json( + StatusCode::NOT_FOUND, + &json!({ "ok": false, "error": "route inconnue" }), + ), + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[cfg(target_os = "linux")] + #[test] + fn lecture_du_moteur() { + let tmp = tempfile::tempdir().unwrap(); + let paths = crate::engine::Paths { + home: tmp.path().to_path_buf(), + service: "loki-engine".into(), + }; + assert_eq!(engine_io_sample(&paths), None, "sans PID"); + std::fs::create_dir_all(paths.pid().parent().unwrap()).unwrap(); + std::fs::write(paths.pid(), "0\n").unwrap(); + assert_eq!(engine_io_sample(&paths), None, "PID nul"); + std::fs::write(paths.pid(), format!("{}\n", std::process::id())).unwrap(); + let (read, rss) = engine_io_sample(&paths).expect("ce processus"); + assert!(read >= 0 && rss > 0, "{read} {rss}"); + } + + #[test] + fn mesure_enregistree() { + let cfg: Cfg = [("MODEL", "/models/Q.gguf"), ("CTX", "32768")] + .iter() + .map(|(k, v)| (k.to_string(), v.to_string())) + .collect(); + let res = BenchResult { + prompt_per_second: 1.0, + depth: Some(super::super::Depth { + hint: "possible thrash".into(), + ..Default::default() + }), + ..Default::default() + }; + let sb = saved_bench(&res, &cfg); + assert_eq!(sb.model, "Q.gguf"); + assert!(!sb.fingerprint.is_empty()); + assert_eq!( + sb.fingerprint, + crate::presets_api::bench_fingerprint(&cfg, &cfg) + ); + assert_eq!( + sb.result.depth.as_ref().unwrap().hint, + "", + "l'indication ne s'enregistre pas" + ); + assert_eq!(res.depth.as_ref().unwrap().hint, "possible thrash"); + assert!(sb.at > 0); + assert_eq!(saved_bench(&res, &Cfg::new()).model, "."); + // Seul un résultat complet s'enregistre. + assert!(savable(&BenchResult::default())); + assert!(savable(&res)); + let partial = BenchResult { + depth: Some(super::super::Depth { + partial: "tour 1 interrompu".into(), + ..Default::default() + }), + ..Default::default() + }; + assert!(!savable(&partial)); + } +} diff --git a/rust/loki-rs/src/bench/mod.rs b/rust/loki-rs/src/bench/mod.rs new file mode 100644 index 0000000..774f6dc --- /dev/null +++ b/rust/loki-rs/src/bench/mod.rs @@ -0,0 +1,1120 @@ +//! Banc d'essai tenu par la façade (lot 27 de la migration, `LOKI_BENCH=rust` +//! ; expérimental, le Go reste le défaut) — port de `llm_bench.go` et +//! `llm_bench_job.go`. +//! +//! Le bench mesure le moteur LOCAL tel que le chat s'en sert, sur +//! `/v1/chat/completions`, avec le gabarit, le raisonnement et +//! l'échantillonnage du preset : +//! +//! - « rapide » : un préchauffage jeté, puis la ligne courte (~2000 jetons de +//! prompt, ~300 générés) ; +//! - « complet » : la ligne courte, puis une discussion à profondeur réelle — +//! un prefill à froid de D jetons et trois tours qui y ajoutent chacun +//! ~2048 jetons neufs. La reprise du cache est LUE dans la réponse. +//! +//! Une réponse non-200 ou sans timings fait échouer la phase : rien n'est +//! inventé, rien n'est enregistré. Le job, les routes et l'enregistrement +//! sont dans [`job`]. + +pub mod job; + +use std::collections::HashMap; +use std::time::Duration; + +use serde::{Deserialize, Serialize}; +use serde_json::{Map, Value, json}; +use tokio::sync::watch; + +use crate::chat::wire::{ + EffortFallbacks, apply_sampling_from, effort_from_rejection, reasoning_effort_value, + reasoning_explicitly_off, reasoning_template_kwargs, +}; + +type Cfg = HashMap; + +/// `benchProtocol`. +pub const PROTOCOL: i64 = 2; +pub const MODE_QUICK: &str = "quick"; +pub const MODE_FULL: &str = "full"; + +const TURNS: i64 = 3; +const APPEND_TOKENS: i64 = 2048; +const GEN_TOKENS: i64 = 256; +const DEPTH_MAX: i64 = 32768; +const DEPTH_CPU: i64 = 16384; +const DEPTH_MIN: i64 = 4096; +const MARGIN: i64 = 512; +const WARM_GEN: i64 = 32; +const SEED: i64 = 1234; +const REUSE_SLACK: i64 = 16; +const SAMPLE_BYTES: usize = 160 << 10; +const FALLBACK_RATIO: f64 = 2.5; +const THRASH_BYTES: i64 = 512 << 20; + +/// `benchReqTimeout` : budget d'une requête. +pub const REQ_TIMEOUT: Duration = Duration::from_secs(6 * 60); + +/// `errBenchNoTimings`. +pub const ERR_NO_TIMINGS: &str = + "le moteur n'a pas renvoyé de mesures (timings) : rien à enregistrer"; + +/// `benchCorpus` : la prose de la ligne courte (le texte même du Go). +pub const CORPUS: &str = include_str!("corpus.txt"); + +const ASK_PROSE: &str = "\n\nContinue this passage with another 1000+ words of original varied prose, mixing French and English narrative paragraphs on different topics."; +const ASK_FIRST: &str = "Résume en quelques phrases le rôle de ce fichier."; +const ASK_NEXT: &str = + "Écris en JavaScript une fonction courte inspirée de cette partie, sans explication."; + +/// `benchArgEnv` : les LLAMA_ARG_* qui décident des types de cache et du +/// placement des poids. +pub const ARG_ENV: [&str; 6] = [ + "LLAMA_ARG_CACHE_TYPE_K", + "LLAMA_ARG_CACHE_TYPE_V", + "LLAMA_ARG_OVERRIDE_TENSOR", + "LLAMA_ARG_CPU_MOE", + "LLAMA_ARG_N_CPU_MOE", + "LLAMA_ARG_N_CPU_FFN", +]; + +/// Un flottant écrit comme le `json.Marshal` du Go : entier sans « .0 ». +pub fn ser_go_f64(v: &f64, s: S) -> Result { + if v.is_finite() && v.fract() == 0.0 && v.abs() < 1e15 { + s.serialize_i64(*v as i64) + } else { + s.serialize_f64(*v) + } +} + +fn is_zero_f(v: &f64) -> bool { + *v == 0.0 +} + +fn is_zero_i(v: &i64) -> bool { + *v == 0 +} + +fn is_false(v: &bool) -> bool { + !*v +} + +/// `benchDraft` : acceptation du brouillon. +#[derive(Clone, Copy, Debug, Default, PartialEq, Serialize, Deserialize)] +pub struct Draft { + #[serde(default)] + pub n: i64, + #[serde(default)] + pub accepted: i64, +} + +/// `(*benchDraft).add`. +pub fn draft_add(d: Option, o: Option) -> Option { + let Some(o) = o else { return d }; + let mut d = d.unwrap_or_default(); + d.n += o.n; + d.accepted += o.accepted; + Some(d) +} + +/// `benchTurn`. +#[derive(Clone, Debug, Default, PartialEq, Serialize, Deserialize)] +pub struct Turn { + #[serde(default)] + pub new: i64, + #[serde(default)] + pub cached: i64, + #[serde(default)] + pub expected: i64, + #[serde(default, serialize_with = "ser_go_f64")] + pub prompt_ms: f64, + #[serde(default, serialize_with = "ser_go_f64")] + pub prompt_per_second: f64, + #[serde(default)] + pub predicted_n: i64, + #[serde(default, serialize_with = "ser_go_f64")] + pub predicted_ms: f64, + #[serde(default, serialize_with = "ser_go_f64")] + pub predicted_per_second: f64, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub draft: Option, +} + +/// `benchDepth`. +#[derive(Clone, Debug, Default, PartialEq, Serialize, Deserialize)] +pub struct Depth { + #[serde(default)] + pub ctx: i64, + #[serde(default)] + pub target: i64, + #[serde(default, skip_serializing_if = "String::is_empty")] + pub skipped: String, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub cold: Option, + #[serde(default, skip_serializing_if = "Vec::is_empty")] + pub turns: Vec, + #[serde( + default, + skip_serializing_if = "is_zero_f", + serialize_with = "ser_go_f64" + )] + pub cached_per_second: f64, + #[serde( + default, + skip_serializing_if = "is_zero_f", + serialize_with = "ser_go_f64" + )] + pub decode_per_second: f64, + #[serde(default, serialize_with = "ser_go_f64")] + pub reuse: f64, + #[serde(default, skip_serializing_if = "String::is_empty")] + pub reuse_note: String, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub draft: Option, + #[serde(default, skip_serializing_if = "String::is_empty")] + pub partial: String, + #[serde(default, skip_serializing_if = "String::is_empty")] + pub hint: String, +} + +/// `benchResult`. +#[derive(Clone, Debug, Default, PartialEq, Serialize, Deserialize)] +pub struct BenchResult { + #[serde(default)] + pub prompt_n: i64, + #[serde(default, serialize_with = "ser_go_f64")] + pub prompt_ms: f64, + #[serde(default, serialize_with = "ser_go_f64")] + pub prompt_per_second: f64, + #[serde(default)] + pub predicted_n: i64, + #[serde(default, serialize_with = "ser_go_f64")] + pub predicted_ms: f64, + #[serde(default, serialize_with = "ser_go_f64")] + pub predicted_per_second: f64, + #[serde(default, rename = "elapsed_sec", serialize_with = "ser_go_f64")] + pub elapsed: f64, + #[serde(default, skip_serializing_if = "String::is_empty")] + pub mode: String, + #[serde(default, skip_serializing_if = "is_zero_i")] + pub protocol: i64, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub draft: Option, + #[serde(default, skip_serializing_if = "String::is_empty")] + pub kv: String, + #[serde(default, skip_serializing_if = "is_false")] + pub kv_quantized: bool, + #[serde(default, skip_serializing_if = "String::is_empty")] + pub engine: String, + #[serde(default, skip_serializing_if = "String::is_empty")] + pub gpus: String, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub depth: Option, +} + +/// `benchOpts`. +#[derive(Clone, Debug, Default, PartialEq)] +pub struct Opts { + pub mode: String, + pub prompt: i64, + pub predict: i64, + /// Profondeur imposée (0 : choisie selon le contexte). + pub depth: i64, +} + +impl Opts { + pub fn full(&self) -> bool { + self.mode == MODE_FULL + } +} + +/// `benchCorpora` : prose pour la ligne courte, code pour la profondeur. +pub struct Corpora { + pub prose: String, + pub code: String, +} + +impl Corpora { + /// `defaultBenchCorpora` : le code est l'interface web embarquée. + pub fn default_set() -> Self { + Corpora { + prose: CORPUS.to_string(), + code: String::from_utf8_lossy(crate::ui::index_bytes()).into_owned(), + } + } +} + +// --------------------------------------------------------------------------- +// Le moteur, vu du bench +// --------------------------------------------------------------------------- + +/// Erreur d'une requête au moteur. +#[derive(Clone, Debug, PartialEq)] +pub enum ReqError { + /// Réponse non-200 (`benchHTTPError`), corps gardé en entier. + Http { + method: String, + path: String, + status: String, + body: String, + }, + /// Le budget de la requête est dépassé (`context.DeadlineExceeded`). + Timeout(String), + /// Le bench est annulé. + Canceled(String), + /// Autre (réseau, réponse sans timings…). + Other(String), +} + +impl std::fmt::Display for ReqError { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + match self { + ReqError::Http { + method, + path, + status, + body, + } => { + let snip = body.trim(); + let snip = if snip.len() > 300 { + let mut cut = 300; + while cut > 0 && !snip.is_char_boundary(cut) { + cut -= 1; + } + &snip[..cut] + } else { + snip + }; + write!(f, "{method} {path} : {status} {snip}") + } + ReqError::Timeout(e) | ReqError::Canceled(e) | ReqError::Other(e) => f.write_str(e), + } + } +} + +/// `benchEngine` : le moteur local. +#[derive(Clone)] +pub struct Engine { + pub base: String, + pub key: String, + pub client: reqwest::Client, + /// Annulation du bench. + pub cancel: watch::Receiver, + /// Budget d'une requête de chat. + pub req_timeout: Duration, + /// Requêtes de chat servies (le slot porte alors l'état du bench). + pub served: std::sync::Arc, +} + +/// Le nom de méthode d'une `url.Error` du Go (« Post », « Get »). +fn go_op(method: &str) -> String { + let m = method.to_ascii_lowercase(); + let mut c = m.chars(); + match c.next() { + Some(f) => f.to_uppercase().collect::() + c.as_str(), + None => String::new(), + } +} + +impl Engine { + /// `do` : le corps d'une réponse 200 (lu jusqu'à 8 Mo) ; toute autre + /// réponse est une erreur. + async fn send( + &self, + method: &str, + path: &str, + payload: Option<&Value>, + limit: Duration, + ) -> Result, ReqError> { + let url = format!("{}{path}", self.base); + let m = reqwest::Method::from_bytes(method.as_bytes()).unwrap_or(reqwest::Method::GET); + let mut rq = self.client.request(m, &url); + if let Some(p) = payload { + rq = rq + .header("Content-Type", "application/json") + .body(serde_json::to_vec(p).unwrap_or_default()); + } + if !self.key.is_empty() { + rq = rq.header("Authorization", format!("Bearer {}", self.key)); + } + let op = go_op(method); + let fut = async { + let resp = rq + .send() + .await + .map_err(|e| ReqError::Other(format!("{op} \"{url}\": {e}")))?; + let status = resp.status(); + let mut data = Vec::new(); + let mut resp = resp; + while let Some(chunk) = resp + .chunk() + .await + .map_err(|e| ReqError::Other(format!("{op} \"{url}\": {e}")))? + { + let room = (8usize << 20).saturating_sub(data.len()); + data.extend_from_slice(&chunk[..chunk.len().min(room)]); + if room <= chunk.len() { + break; + } + } + if status != reqwest::StatusCode::OK { + return Err(ReqError::Http { + method: method.to_string(), + path: path.to_string(), + status: format!( + "{} {}", + status.as_u16(), + status.canonical_reason().unwrap_or("") + ), + body: String::from_utf8_lossy(&data).into_owned(), + }); + } + Ok(data) + }; + let mut cancel = self.cancel.clone(); + if *cancel.borrow() { + return Err(ReqError::Canceled(format!( + "{op} \"{url}\": context canceled" + ))); + } + tokio::select! { + r = tokio::time::timeout(limit, fut) => match r { + Ok(r) => r, + Err(_) => Err(ReqError::Timeout(format!( + "{op} \"{url}\": context deadline exceeded" + ))), + }, + _ = async { + // Émetteur disparu : plus personne pour annuler. + if cancel.wait_for(|c| *c).await.is_err() { + std::future::pending::<()>().await; + } + } => { + Err(ReqError::Canceled(format!("{op} \"{url}\": context canceled"))) + } + } + } + + /// `idle` : un slot qui travaille, c'est la requête d'un autre. `/slots` + /// coupé ou illisible : on ne sait pas, on n'empêche rien. + pub async fn idle(&self) -> Result<(), String> { + let Ok(data) = self + .send("GET", "/slots", None, Duration::from_secs(3)) + .await + else { + return Ok(()); + }; + let Ok(Value::Array(slots)) = serde_json::from_slice::(&data) else { + return Ok(()); + }; + let busy = slots + .iter() + .any(|s| s.get("is_processing").and_then(Value::as_bool) == Some(true)); + if busy { + return Err( + "le moteur traite déjà une requête : relance le bench une fois libre".into(), + ); + } + Ok(()) + } + + /// `props` : la taille de contexte et le build. Absents : 0 et "". + pub async fn props(&self) -> (i64, String) { + let Ok(data) = self + .send("GET", "/props", None, Duration::from_secs(5)) + .await + else { + return (0, String::new()); + }; + let Ok(Value::Object(p)) = serde_json::from_slice::(&data) else { + return (0, String::new()); + }; + let int = |v: Option<&Value>| v.and_then(Value::as_i64).unwrap_or(0); + let mut n = int(p + .get("default_generation_settings") + .and_then(|d| d.get("n_ctx"))); + if n <= 0 { + n = int(p.get("n_ctx")); + } + let build = p + .get("build_info") + .and_then(Value::as_str) + .unwrap_or("") + .to_string(); + (n, build) + } + + /// `bytesPerToken` : octets par jeton d'un échantillon (`/tokenize`). + pub async fn bytes_per_token(&self, sample: &str) -> f64 { + if sample.is_empty() { + return FALLBACK_RATIO; + } + let Ok(data) = self + .send( + "POST", + "/tokenize", + Some(&json!({ "content": sample })), + Duration::from_secs(30), + ) + .await + else { + return FALLBACK_RATIO; + }; + match serde_json::from_slice::(&data) { + Ok(v) => match v.get("tokens").and_then(Value::as_array) { + Some(t) if !t.is_empty() => sample.len() as f64 / t.len() as f64, + _ => FALLBACK_RATIO, + }, + Err(_) => FALLBACK_RATIO, + } + } + + /// `chat` : une complétion et ses timings ; sans timings, pas de mesure. + pub async fn chat(&self, payload: &Value) -> Result { + let data = self + .send( + "POST", + "/v1/chat/completions", + Some(payload), + self.req_timeout, + ) + .await?; + self.served + .fetch_add(1, std::sync::atomic::Ordering::SeqCst); + parse_reply(&data) + } +} + +/// `perfNum` : un nombre (ou une chaîne qui en contient un), fini, positif et +/// < 1e12 ; sinon inconnu. +fn perf_num(v: Option<&Value>) -> Option { + let f = match v? { + Value::Number(n) => n.as_f64()?, + Value::String(s) => s.parse::().ok()?, + _ => return None, + }; + (f.is_finite() && (0.0..1e12).contains(&f)).then_some(f) +} + +/// `benchReply` : une réponse du chat. `total` : usage.prompt_tokens (0 = +/// inconnu). +#[derive(Clone, Debug, Default, PartialEq)] +pub struct Reply { + pub turn: Turn, + pub total: i64, + pub content: String, +} + +impl Reply { + /// `promptTotal`. + pub fn prompt_total(&self) -> i64 { + if self.total > 0 { + return self.total; + } + self.turn.new + self.turn.cached.max(0) + } +} + +/// La réponse du moteur en mesure (`decodePerfWire` et le contenu). +pub fn parse_reply(data: &[u8]) -> Result { + let first = serde_json::Deserializer::from_slice(data) + .into_iter::() + .next(); + let o = match first { + Some(Ok(Value::Object(o))) => o, + _ => Map::new(), + }; + let t = o.get("timings").and_then(Value::as_object); + let num = |k: &str| t.and_then(|t| perf_num(t.get(k))); + let (Some(pn), Some(pms), Some(gn), Some(gms)) = ( + num("prompt_n"), + num("prompt_ms"), + num("predicted_n"), + num("predicted_ms"), + ) else { + return Err(ReqError::Other(ERR_NO_TIMINGS.into())); + }; + let rate = |n: f64, ms: f64, given: Option| match given { + Some(g) if g > 0.0 => g, + _ if ms > 0.0 => n / (ms / 1000.0), + _ => 0.0, + }; + let mut turn = Turn { + new: pn as i64, + cached: -1, + prompt_ms: pms, + prompt_per_second: rate(pn, pms, num("prompt_per_second")), + predicted_n: gn as i64, + predicted_ms: gms, + predicted_per_second: rate(gn, gms, num("predicted_per_second")), + ..Default::default() + }; + let usage = o.get("usage").and_then(Value::as_object); + let cached_tokens = usage + .and_then(|u| u.get("prompt_tokens_details")) + .and_then(Value::as_object) + .and_then(|d| perf_num(d.get("cached_tokens"))); + if let Some(c) = num("cache_n") { + turn.cached = c as i64; + } else if let Some(c) = cached_tokens { + turn.cached = c as i64; + } + if let (Some(n), Some(a)) = (num("draft_n"), num("draft_n_accepted")) + && n as i64 > 0 + { + turn.draft = Some(Draft { + n: n as i64, + accepted: a as i64, + }); + } + let total = usage + .and_then(|u| perf_num(u.get("prompt_tokens"))) + .map_or(0, |f| f as i64); + // json.Unmarshal du message entier : un octet de trop, pas de contenu. + let content = serde_json::from_slice::(data) + .ok() + .and_then(|v| { + v.get("choices")? + .get(0)? + .get("message")? + .get("content")? + .as_str() + .map(str::to_string) + }) + .unwrap_or_default(); + Ok(Reply { + turn, + total, + content, + }) +} + +// --------------------------------------------------------------------------- +// Réglages du preset +// --------------------------------------------------------------------------- + +/// `benchSetup` : ce que le bench tire du preset, une fois. +pub struct Setup { + pub cfg: Cfg, + pub want: String, + pub off: bool, + pub effort: String, + pub kwargs: Option>, + pub retried: bool, + pub ub: i64, + pub cpu_placed: bool, + pub hybrid: bool, + pub kv: String, + pub kv_quant: bool, + pub gpus: String, + pub side_slot: bool, +} + +/// `benchSetupFrom`. +pub fn setup_from( + cfg: &Cfg, + arg_env: &HashMap, + fallbacks: &EffortFallbacks, +) -> Setup { + let get = |k: &str| cfg.get(k).map(String::as_str).unwrap_or(""); + let extra = crate::models::split_args(get("EXTRA_ARGS")); + let want = reasoning_effort_value(get("REASONING_EFFORT")); + let effort = fallbacks.resolve(cfg, &want); + let off = reasoning_explicitly_off(get("REASONING")) || want == "none"; + let kwargs = reasoning_template_kwargs(off, &effort); + let mut ub = 512; + for v in [ + crate::serve::flag_value(&extra, &["-ub", "--ubatch-size"]), + get("UBATCH").to_string(), + ] { + if let Some(n) = go_atoi(v.trim()) + && n > 0 + { + ub = n; + break; + } + } + let cpu_placed = !crate::serve::queues::tensor_override(&extra, arg_env).is_empty(); + let (k, v, _) = crate::serve::fidelity::effective_kv_types(cfg, &extra, arg_env); + let (kv, kv_quant) = kv_label(&k, &v); + Setup { + cfg: cfg.clone(), + want, + off, + effort, + kwargs, + retried: false, + ub, + cpu_placed, + // Le modèle hybride se lit dans le GGUF : posé par l'appelant, qui sait + // où sont les modèles. + hybrid: false, + kv, + kv_quant, + gpus: get("CUDA_VISIBLE_DEVICES").trim().to_string(), + side_slot: false, + } +} + +/// `strconv.Atoi`. +fn go_atoi(s: &str) -> Option { + let digits = s.strip_prefix(['+', '-']).unwrap_or(s); + if digits.is_empty() || !digits.bytes().all(|c| c.is_ascii_digit()) { + return None; + } + s.parse().ok() +} + +/// `benchKVLabel` : les types de cache (« f16 », « q8_0/f16 »…), quantifiés +/// ou non (rang 2 et plus de kvFidelity). +pub fn kv_label(k: &str, v: &str) -> (String, bool) { + let norm = |s: &str| match s.trim().to_lowercase() { + s if s.is_empty() => "f16".to_string(), + s => s, + }; + let (k, v) = (norm(k), norm(v)); + let (rk, _) = crate::serve::fidelity::kv_fidelity(&k); + let (rv, _) = crate::serve::fidelity::kv_fidelity(&v); + let label = if k == v { k } else { format!("{k}/{v}") }; + (label, rk.max(rv) >= 2) +} + +impl Setup { + /// `payload` : une requête de chat du bench. + pub fn payload(&self, msgs: &[(String, String)], max_tokens: i64, cache: bool) -> Value { + let messages: Vec = msgs + .iter() + .map(|(r, c)| json!({ "role": r, "content": c })) + .collect(); + let mut p = Map::new(); + p.insert("model".into(), json!("loki")); + p.insert("messages".into(), Value::Array(messages)); + p.insert("max_tokens".into(), json!(max_tokens)); + p.insert("stream".into(), json!(false)); + p.insert("temperature".into(), json!(0.7)); + p.insert("seed".into(), json!(SEED)); + p.insert("cache_prompt".into(), json!(cache)); + apply_sampling_from(&mut p, &self.cfg); + if !self.effort.is_empty() { + p.insert("reasoning_effort".into(), json!(self.effort)); + } + if let Some(k) = &self.kwargs { + p.insert("chat_template_kwargs".into(), Value::Object(k.clone())); + } + if self.side_slot { + p.insert("id_slot".into(), json!(1)); + } + Value::Object(p) + } + + /// `ask` : un gabarit qui refuse le niveau de raisonnement est rejoué une + /// fois avec celui qu'il accepte (ou sans le champ), traduction retenue. + pub async fn ask( + &mut self, + e: &Engine, + msgs: &[(String, String)], + max_tokens: i64, + cache: bool, + learn: &(dyn Fn(&str, &str) + Send + Sync), + ) -> Result { + let r = e.chat(&self.payload(msgs, max_tokens, cache)).await; + let body = match &r { + Err(ReqError::Http { body, .. }) if !self.retried && !self.want.is_empty() => { + body.clone() + } + _ => return r, + }; + let Some(fixed) = effort_from_rejection(&body, &self.want) else { + return r; + }; + self.retried = true; + learn(&self.want, &fixed); + self.kwargs = reasoning_template_kwargs(self.off, &fixed); + self.effort = fixed; + e.chat(&self.payload(msgs, max_tokens, cache)).await + } +} + +// --------------------------------------------------------------------------- +// Le protocole +// --------------------------------------------------------------------------- + +/// `benchDepthFor` : la moitié du contexte, plafonnée, et assez basse pour +/// que les tours y tiennent. Trop petit : la phase est sautée. +pub fn depth_for(n_ctx: i64, cpu_placed: bool, warm: i64) -> (i64, String) { + if n_ctx <= 0 { + return (0, "taille de contexte inconnue".into()); + } + let mut d = (n_ctx / 2).min(DEPTH_MAX); + if cpu_placed { + d = d.min(DEPTH_CPU); + } + d = d.min(n_ctx - TURNS * (APPEND_TOKENS + GEN_TOKENS) - GEN_TOKENS - warm - MARGIN); + if d < DEPTH_MIN { + return ( + 0, + format!("contexte de {n_ctx} jetons trop petit pour la mesure en profondeur"), + ); + } + (d, String::new()) +} + +/// `benchSlice` : à partir de `off`, environ `n` octets arrêtés à une fin de +/// ligne — jamais au milieu d'un caractère. "" : corpus épuisé. +pub fn slice(text: &str, off: usize, n: i64) -> (&str, usize) { + if off >= text.len() || n <= 0 { + return ("", off); + } + let n = n as usize; + let mut end = off.saturating_add(n); + if end >= text.len() { + return (&text[off..], text.len()); + } + if let Some(i) = text.as_bytes()[off..end].iter().rposition(|&b| b == b'\n') + && i > n / 2 + { + end = off + i + 1; + } + while end > off && !text.is_char_boundary(end) { + end -= 1; + } + (&text[off..end], end) +} + +/// `benchLineStart` : le début de la ligne qui suit `off`. +pub fn line_start(text: &str, off: i64) -> usize { + if off <= 0 { + return 0; + } + let off = off as usize; + if off >= text.len() { + return text.len(); + } + let b = text.as_bytes(); + if b[off - 1] == b'\n' { + return off; + } + match b[off..].iter().position(|&c| c == b'\n') { + Some(i) => off + i + 1, + None => text.len(), + } +} + +/// `benchReuse` : la part du prompt précédent reprise du cache, et ce +/// qu'elle veut dire. +pub fn reuse(turns: &[Turn], hybrid: bool) -> (f64, String) { + let (mut cached, mut expected) = (0i64, 0i64); + let mut full = true; + for t in turns { + if t.cached < 0 || t.expected <= 0 { + continue; + } + cached += t.cached.min(t.expected); + expected += t.expected; + if t.cached < t.expected - REUSE_SLACK { + full = false; + } + } + if expected == 0 { + return ( + -1.0, + "le moteur ne dit pas combien de jetons il reprend du cache".into(), + ); + } + let ratio = cached as f64 / expected as f64; + if full { + (ratio, String::new()) + } else if hybrid { + ( + ratio, + "reprise partielle : modèle hybride, le cache ne reprend qu'aux points de contrôle" + .into(), + ) + } else { + ( + ratio, + "reprise partielle : le gabarit réécrit sans doute la fin de l'historique (raisonnement retiré)" + .into(), + ) + } +} + +/// `benchThrashHint` : un moteur qui relit beaucoup le disque pendant les +/// tours n'a pas le modèle en RAM (indication seulement). +pub fn thrash_hint(read: i64, generated: i64, rss: i64) -> String { + if read <= 0 || generated <= 0 { + return String::new(); + } + let per = read * 256 / generated; + if per <= THRASH_BYTES { + return String::new(); + } + format!( + "possible thrash : {:.1} Go relus du disque par 256 jetons générés (RSS {:.1} Go) — le modèle ne tient sans doute pas en RAM", + per as f64 / (1u64 << 30) as f64, + rss as f64 / (1u64 << 30) as f64 + ) +} + +/// `benchDraftText`. +pub fn draft_text(d: Option<&Draft>) -> String { + match d { + Some(d) if d.n != 0 => format!( + " brouillon accepté {:.0} % ({}/{})", + d.accepted as f64 * 100.0 / d.n as f64, + d.accepted, + d.n + ), + _ => String::new(), + } +} + +/// `time.Duration.String` du Go, exact (« 150ms », « 1.5µs », « 6m0s »). +pub fn go_dur(d: Duration) -> String { + let u = d.as_nanos(); + if u == 0 { + return "0s".into(); + } + // La fraction de v / 10^prec, zéros de fin omis (sans point si nulle). + let frac = |v: u128, prec: u32| -> (u128, String) { + let p = 10u128.pow(prec); + let f = v % p; + if f == 0 { + return (v / p, String::new()); + } + let digits = format!("{f:0width$}", width = prec as usize); + (v / p, format!(".{}", digits.trim_end_matches('0'))) + }; + if u < 1_000_000_000 { + let (unit, prec) = match u { + u if u < 1_000 => ("ns", 0), + u if u < 1_000_000 => ("µs", 3), + _ => ("ms", 6), + }; + let (i, f) = frac(u, prec); + return format!("{i}{f}{unit}"); + } + let (secs, f) = frac(u, 9); + let (h, m, s) = (secs / 3600, secs / 60 % 60, secs % 60); + match (h, m) { + (0, 0) => format!("{s}{f}s"), + (0, m) => format!("{m}m{s}{f}s"), + (h, m) => format!("{h}h{m}m{s}{f}s"), + } +} + +/// La lecture disque et la RSS du moteur (`engineIOSample`) ; `None` hors +/// Linux ou sans moteur. +pub type IoSample<'a> = &'a (dyn Fn() -> Option<(i64, i64)> + Send + Sync); + +/// `benchRun` : le protocole contre un moteur. `Err` = une phase de mesure a +/// échoué, rien à enregistrer ; une phase en profondeur interrompue par son +/// budget ou par la taille du contexte rend un résultat PARTIEL. +pub async fn run( + e: &Engine, + mut opts: Opts, + s: &mut Setup, + corp: &Corpora, + progress: &(dyn Fn(&str, i64, i64) + Send + Sync), + learn: &(dyn Fn(&str, &str) + Send + Sync), + io: IoSample<'_>, +) -> Result { + if opts.prompt <= 0 { + opts.prompt = 2000; + } + if opts.predict <= 0 { + opts.predict = 300; + } + e.idle().await?; + let steps = if opts.full() { 2 + 1 + TURNS } else { 2 }; + let step = std::sync::atomic::AtomicI64::new(0); + let next = |phase: &str| { + let n = step.fetch_add(1, std::sync::atomic::Ordering::SeqCst) + 1; + progress(phase, n, steps); + }; + let t0 = std::time::Instant::now(); + let (mut n_ctx, build) = e.props().await; + if n_ctx <= 0 { + n_ctx = go_atoi(s.cfg.get("CTX").map(String::as_str).unwrap_or("").trim()).unwrap_or(0); + } + let mode = if opts.full() { MODE_FULL } else { MODE_QUICK }; + let mut res = BenchResult { + mode: mode.into(), + protocol: PROTOCOL, + kv: s.kv.clone(), + kv_quantized: s.kv_quant, + engine: build, + gpus: s.gpus.clone(), + ..Default::default() + }; + let code = corp.code.as_str(); + let (sample, _) = slice(code, 0, SAMPLE_BYTES as i64); + let code_ratio = e.bytes_per_token(sample).await; + let prose_ratio = e.bytes_per_token(&corp.prose).await; + + // 1. Préchauffage, jeté. + next("préchauffage"); + let mut warm_tok = 2 * s.ub; + if n_ctx > 0 { + warm_tok = warm_tok.min(n_ctx / 4).max(64); + } + let warm = warm_tok + WARM_GEN; + let from = line_start( + code, + code.len() as i64 - (warm_tok as f64 * code_ratio) as i64, + ); + let (warm_text, _) = slice(code, from, code.len() as i64); + let msgs = vec![("user".to_string(), format!("{warm_text}\n\n{ASK_FIRST}"))]; + if let Err(err) = s.ask(e, &msgs, WARM_GEN, false, learn).await { + return Err(format!("préchauffage : {err}")); + } + + // 2. La ligne courte : de la prose, puis du code inédit. + next(&format!( + "ligne courte ({} + {} jetons)", + opts.prompt, opts.predict + )); + let mut prompt = corp.prose.clone(); + let want = (opts.prompt as f64 * prose_ratio) as i64; + if want < prompt.len() as i64 { + prompt = slice(&corp.prose, 0, want).0.to_string(); + } else { + let rest = opts.prompt - (prompt.len() as f64 / prose_ratio) as i64; + if rest > 0 { + let (chunk, _) = slice( + code, + line_start(code, (code.len() / 2) as i64), + (rest as f64 * code_ratio) as i64, + ); + prompt = format!("{prompt}\n\n{chunk}"); + } + } + let msgs = vec![("user".to_string(), format!("{}{ASK_PROSE}", prompt.trim()))]; + let quick = s + .ask(e, &msgs, opts.predict, false, learn) + .await + .map_err(|err| format!("ligne courte : {err}"))?; + let q = quick.turn; + res.prompt_n = q.new; + res.prompt_ms = q.prompt_ms; + res.prompt_per_second = q.prompt_per_second; + res.predicted_n = q.predicted_n; + res.predicted_ms = q.predicted_ms; + res.predicted_per_second = q.predicted_per_second; + res.draft = q.draft; + if !opts.full() { + res.elapsed = t0.elapsed().as_secs_f64(); + return Ok(res); + } + + // 3. La profondeur. + let (mut d, mut why) = depth_for(n_ctx, s.cpu_placed, warm); + if opts.depth > 0 { + d = opts.depth; + why = String::new(); + } + let mut depth = Depth { + ctx: n_ctx, + target: d, + reuse: -1.0, + ..Default::default() + }; + if !why.is_empty() { + depth.skipped = why; + res.depth = Some(depth); + res.elapsed = t0.elapsed().as_secs_f64(); + return Ok(res); + } + depth_run(e, s, code, code_ratio, &mut depth, &next, learn, io).await?; + res.depth = Some(depth); + res.elapsed = t0.elapsed().as_secs_f64(); + Ok(res) +} + +/// `benchDepthRun` : prefill à froid à D, puis trois tours qui prolongent la +/// même discussion avec du code jamais vu. +#[allow(clippy::too_many_arguments)] +async fn depth_run( + e: &Engine, + s: &mut Setup, + code: &str, + ratio: f64, + depth: &mut Depth, + next: &(dyn Fn(&str) + Send + Sync), + learn: &(dyn Fn(&str, &str) + Send + Sync), + io: IoSample<'_>, +) -> Result<(), String> { + next(&format!("prefill à froid ({} jetons)", depth.target)); + let (prefix, mut off) = slice(code, 0, (depth.target as f64 * ratio) as i64); + let mut msgs = vec![( + "user".to_string(), + format!( + "Voici un fichier source de l'interface web de Loki :\n\n```html\n{prefix}\n```\n\n{ASK_FIRST}" + ), + )]; + let cold = match s.ask(e, &msgs, GEN_TOKENS, false, learn).await { + Err(ReqError::Timeout(_)) => { + depth.partial = format!("prefill à froid interrompu après {}", go_dur(e.req_timeout)); + return Ok(()); + } + Err(err) => return Err(format!("prefill à froid : {err}")), + Ok(r) => r, + }; + depth.cold = Some(cold.turn.clone()); + let (mut prev, mut answer) = (cold.prompt_total(), cold.content.clone()); + // Relectures du disque comptées sur les TOURS seulement. + let before = io(); + let (mut new_n, mut generated) = (0i64, 0i64); + let (mut prompt_ms, mut gen_ms) = (0f64, 0f64); + for r in 1..=TURNS { + if prev + 2 * GEN_TOKENS + APPEND_TOKENS * 5 / 4 + MARGIN > depth.ctx { + depth.partial = format!("contexte plein après {} tour(s)", r - 1); + break; + } + let (chunk, next_off) = slice(code, off, (APPEND_TOKENS as f64 * ratio) as i64); + if chunk.is_empty() { + depth.partial = format!("corpus épuisé après {} tour(s)", r - 1); + break; + } + off = next_off; + next(&format!("tour {r}/{TURNS} en profondeur")); + msgs.push(("assistant".to_string(), answer.clone())); + msgs.push(( + "user".to_string(), + format!("Suite du fichier :\n\n```html\n{chunk}\n```\n\n{ASK_NEXT}"), + )); + let rep = match s.ask(e, &msgs, GEN_TOKENS, true, learn).await { + Err(ReqError::Timeout(_)) => { + depth.partial = format!("tour {r} interrompu après {}", go_dur(e.req_timeout)); + break; + } + Err(err) => return Err(format!("tour {r} en profondeur : {err}")), + Ok(rep) => rep, + }; + let mut t = rep.turn.clone(); + t.expected = prev; + depth.draft = draft_add(depth.draft, t.draft); + new_n += t.new; + prompt_ms += t.prompt_ms; + generated += t.predicted_n; + gen_ms += t.predicted_ms; + depth.turns.push(t); + prev = rep.prompt_total(); + answer = rep.content; + } + if prompt_ms > 0.0 { + depth.cached_per_second = new_n as f64 / (prompt_ms / 1000.0); + } + if gen_ms > 0.0 { + depth.decode_per_second = generated as f64 / (gen_ms / 1000.0); + } + (depth.reuse, depth.reuse_note) = reuse(&depth.turns, s.hybrid); + if let (Some((io0, rss0)), Some((io1, rss1))) = (before, io()) + && generated > 0 + { + depth.hint = thrash_hint(io1 - io0, generated, rss0.max(rss1)); + } + Ok(()) +} + +#[cfg(test)] +mod tests; diff --git a/rust/loki-rs/src/bench/tests.rs b/rust/loki-rs/src/bench/tests.rs new file mode 100644 index 0000000..219ec0b --- /dev/null +++ b/rust/loki-rs/src/bench/tests.rs @@ -0,0 +1,344 @@ +//! Tests du bench : ceux du Go (`llm_bench_test.go`), portés, et les bornes. + +use super::*; + +fn cfg(pairs: &[(&str, &str)]) -> Cfg { + pairs + .iter() + .map(|(k, v)| (k.to_string(), v.to_string())) + .collect() +} + +#[test] +fn profondeur_choisie() { + let warm = 2 * 512 + WARM_GEN; + let turns = TURNS * (APPEND_TOKENS + GEN_TOKENS); + for (ctx, cpu, want, skipped, what) in [ + (131072, false, DEPTH_MAX, false, "grand contexte : plafond"), + ( + 131072, + true, + DEPTH_CPU, + false, + "poids sur CPU : plafond réduit", + ), + (65536, false, 32768, false, "moitié du contexte"), + ( + 16384, + false, + 16384 - turns - GEN_TOKENS - warm - MARGIN, + false, + "les tours doivent tenir", + ), + (8192, false, 0, true, "trop petit"), + (0, false, 0, true, "inconnu"), + ] { + let (got, why) = depth_for(ctx, cpu, warm); + assert_eq!((got, !why.is_empty()), (want, skipped), "{what}"); + if got > 0 { + assert!(got + turns + GEN_TOKENS <= ctx, "{what} : déborde"); + } + } + assert_eq!(depth_for(0, false, 0).1, "taille de contexte inconnue"); + assert_eq!( + depth_for(8192, false, warm).1, + "contexte de 8192 jetons trop petit pour la mesure en profondeur" + ); + // Juste au seuil : 4096 tient, 4095 non. + let at = |d: i64| d + turns + GEN_TOKENS + warm + MARGIN; + assert_eq!(depth_for(at(4096), false, warm).0, 4096); + assert_eq!(depth_for(at(4095), false, warm).0, 0); +} + +#[test] +fn types_de_cache() { + for (k, v, label, quant) in [ + ("", "", "f16", false), + ("f16", "", "f16", false), + ("bf16", "bf16", "bf16", false), + ("q8_0", "q8_0", "q8_0", true), + ("q8_0", "", "q8_0/f16", true), + ("", "q4_0", "f16/q4_0", true), + (" Q8_0 ", "q8_0", "q8_0", true), + ] { + assert_eq!(kv_label(k, v), (label.to_string(), quant), "{k:?} {v:?}"); + } + // Le -ctk d'EXTRA_ARGS l'emporte, comme au lancement. + let s = setup_from( + &cfg(&[ + ("KV_TYPE", "f16"), + ("EXTRA_ARGS", "-ctk q8_0 -ctv q8_0 --n-cpu-moe 20"), + ]), + &HashMap::new(), + &EffortFallbacks::new(), + ); + assert_eq!( + (s.kv.as_str(), s.kv_quant, s.cpu_placed), + ("q8_0", true, true) + ); + let s = setup_from(&cfg(&[]), &HashMap::new(), &EffortFallbacks::new()); + assert!(!s.cpu_placed); + // Placement par l'environnement de llama-server. + let env = HashMap::from([("LLAMA_ARG_CPU_MOE".to_string(), "1".to_string())]); + let s = setup_from(&cfg(&[]), &env, &EffortFallbacks::new()); + assert!(s.cpu_placed); +} + +#[test] +fn micro_lot_du_preset() { + let ub = |pairs: &[(&str, &str)]| { + setup_from(&cfg(pairs), &HashMap::new(), &EffortFallbacks::new()).ub + }; + assert_eq!(ub(&[]), 512); + assert_eq!(ub(&[("UBATCH", "1024")]), 1024); + assert_eq!(ub(&[("UBATCH", " 2048 ")]), 2048); + assert_eq!(ub(&[("UBATCH", "0")]), 512); + assert_eq!(ub(&[("UBATCH", "x")]), 512); + assert_eq!(ub(&[("EXTRA_ARGS", "-ub 256"), ("UBATCH", "1024")]), 256); + assert_eq!(ub(&[("EXTRA_ARGS", "--ubatch-size 128")]), 128); + assert_eq!(ub(&[("EXTRA_ARGS", "-ub -4"), ("UBATCH", "64")]), 64); + let gpus = setup_from( + &cfg(&[("CUDA_VISIBLE_DEVICES", " 0,1 ")]), + &HashMap::new(), + &EffortFallbacks::new(), + ) + .gpus; + assert_eq!(gpus, "0,1"); +} + +#[test] +fn requete_du_bench() { + let msgs = vec![("user".to_string(), "x".to_string())]; + let fb = EffortFallbacks::new(); + let p = setup_from( + &cfg(&[("TEMP", "0.6"), ("TOP_P", "0.95"), ("TOP_K", "20")]), + &HashMap::new(), + &fb, + ) + .payload(&msgs, 10, true); + assert_eq!(p["temperature"], json!(0.6)); + assert_eq!(p["top_p"], json!(0.95)); + assert_eq!(p["top_k"], json!(20)); + assert_eq!(p["seed"], json!(SEED)); + assert_eq!(p["max_tokens"], json!(10)); + assert_eq!(p["model"], json!("loki")); + assert_eq!(p["messages"], json!([{ "role": "user", "content": "x" }])); + assert_eq!( + (p["cache_prompt"].clone(), p["stream"].clone()), + (json!(true), json!(false)) + ); + for k in [ + "chat_template_kwargs", + "reasoning_effort", + "ignore_eos", + "id_slot", + ] { + assert!(p.get(k).is_none(), "{k}"); + } + let s = setup_from(&cfg(&[]), &HashMap::new(), &fb); + let p = s.payload(&msgs, 10, false); + assert_eq!(p["temperature"], json!(0.7)); + assert_eq!(p["cache_prompt"], json!(false)); + assert!(p.get("top_k").is_none()); + let mut s = setup_from(&cfg(&[("REASONING_EFFORT", "high")]), &HashMap::new(), &fb); + assert_eq!(s.payload(&msgs, 1, true)["reasoning_effort"], json!("high")); + s.side_slot = true; + assert_eq!(s.payload(&msgs, 1, true)["id_slot"], json!(1)); + // Raisonnement coupé : le gabarit le dit. + let s = setup_from(&cfg(&[("REASONING", "off")]), &HashMap::new(), &fb); + assert_eq!( + s.payload(&msgs, 1, true)["chat_template_kwargs"], + json!({ "enable_thinking": false }) + ); + let s = setup_from(&cfg(&[("REASONING_EFFORT", "none")]), &HashMap::new(), &fb); + assert!(s.off); + // Une traduction apprise est envoyée d'emblée. + let c = cfg(&[("REASONING_EFFORT", "high"), ("MODEL", "/m/Q.gguf")]); + fb.remember(&c, "high", "xhigh"); + let s = setup_from(&c, &HashMap::new(), &fb); + assert_eq!((s.want.as_str(), s.effort.as_str()), ("high", "xhigh")); +} + +#[test] +fn morceaux_du_corpus() { + let text = "ééé ligne\n".repeat(100); + let mut off = 0; + while off < text.len() { + let (chunk, next) = slice(&text, off, 37); + assert!(!chunk.is_empty(), "à {off}"); + if next < text.len() { + assert!(chunk.ends_with('\n'), "coupé hors fin de ligne : {chunk:?}"); + } + off = next; + } + assert_eq!(slice(&text, text.len(), 10).0, ""); + assert_eq!(slice(&text, 0, 0).0, ""); + assert_eq!(slice("abc", 0, 99), ("abc", 3)); + // Sans fin de ligne assez loin : coupé à la taille, jamais dans un + // caractère. + assert_eq!(slice("éééééé", 0, 3), ("é", 2)); + assert_eq!(slice("ab\ncdefgh", 0, 8), ("ab\ncdefg", 8)); + assert_eq!(slice("abcde\nfgh", 0, 8), ("abcde\n", 6)); + assert_eq!(slice("abcd\nfgh", 0, 8), ("abcd\nfgh", 8)); + assert_eq!(line_start("ab\ncd", 1), 3); + assert_eq!(line_start("ab\ncd", 3), 3); + assert_eq!(line_start("ab\ncd", 0), 0); + assert_eq!(line_start("ab\ncd", -5), 0); + assert_eq!(line_start("ab\ncd", 4), 5); + assert_eq!(line_start("ab\ncd", 9), 5); +} + +#[test] +fn reprise_du_cache() { + let t = |cached, expected| Turn { + cached, + expected, + ..Default::default() + }; + let (r, note) = reuse(&[t(1000, 1004), t(3000, 3000)], false); + assert!(r > 0.99 && note.is_empty(), "{r} {note}"); + assert_eq!(reuse(&[t(1000, 1016)], false).1, ""); + assert!(!reuse(&[t(1000, 1017)], false).1.is_empty()); + let part = [t(512, 3000)]; + assert!(reuse(&part, true).1.contains("hybride")); + assert!(reuse(&part, false).1.contains("gabarit")); + assert_eq!(reuse(&part, false).0, 512.0 / 3000.0); + let (r, note) = reuse(&[t(-1, 3000)], false); + assert_eq!(r, -1.0); + assert!(!note.is_empty()); + assert_eq!(reuse(&[t(10, 0)], false).0, -1.0); + // Un cache plus grand que l'attendu compte pour l'attendu. + assert_eq!(reuse(&[t(5000, 3000)], false).0, 1.0); +} + +#[test] +fn relectures_du_disque() { + assert_eq!(thrash_hint(100 << 20, 256, 0), ""); + assert_eq!(thrash_hint(0, 256, 0), ""); + assert_eq!(thrash_hint(1 << 40, 0, 0), ""); + assert_eq!(thrash_hint(512 << 20, 256, 0), ""); + let h = thrash_hint(3 << 30, 768, 60 << 30); + assert_eq!( + h, + "possible thrash : 1.0 Go relus du disque par 256 jetons générés (RSS 60.0 Go) — le modèle ne tient sans doute pas en RAM" + ); + assert!(!thrash_hint((512 << 20) + 1, 256, 0).is_empty()); +} + +#[test] +fn brouillon() { + assert_eq!(draft_text(None), ""); + assert_eq!(draft_text(Some(&Draft { n: 0, accepted: 0 })), ""); + assert_eq!( + draft_text(Some(&Draft { + n: 30, + accepted: 21 + })), + " brouillon accepté 70 % (21/30)" + ); + let d = draft_add(None, Some(Draft { n: 10, accepted: 7 })); + let d = draft_add(d, None); + let d = draft_add(d, Some(Draft { n: 5, accepted: 1 })); + assert_eq!(d, Some(Draft { n: 15, accepted: 8 })); +} + +#[test] +fn durees_du_go() { + let ms = Duration::from_millis; + assert_eq!(go_dur(Duration::ZERO), "0s"); + assert_eq!(go_dur(Duration::from_nanos(7)), "7ns"); + assert_eq!(go_dur(Duration::from_nanos(1500)), "1.5µs"); + assert_eq!(go_dur(ms(150)), "150ms"); + assert_eq!(go_dur(Duration::from_micros(1500)), "1.5ms"); + assert_eq!(go_dur(ms(1000)), "1s"); + assert_eq!(go_dur(ms(1500)), "1.5s"); + assert_eq!(go_dur(Duration::from_secs(6 * 60)), "6m0s"); + assert_eq!(go_dur(Duration::from_secs(3600)), "1h0m0s"); + assert_eq!(go_dur(Duration::from_secs(3661)), "1h1m1s"); + assert_eq!( + go_dur(Duration::from_nanos(61_000_000_001)), + "1m1.000000001s" + ); +} + +#[test] +fn reponse_du_moteur() { + let full = br#"{"choices":[{"message":{"content":"ok"}}],"usage":{"prompt_tokens":120,"prompt_tokens_details":{"cached_tokens":30}},"timings":{"prompt_n":100,"prompt_ms":50,"predicted_n":10,"predicted_ms":200,"draft_n":4,"draft_n_accepted":3}}"#; + let r = parse_reply(full).unwrap(); + assert_eq!(r.turn.new, 100); + assert_eq!(r.turn.prompt_per_second, 2000.0); + assert_eq!(r.turn.predicted_per_second, 50.0); + assert_eq!(r.turn.cached, 30); + assert_eq!(r.turn.draft, Some(Draft { n: 4, accepted: 3 })); + assert_eq!((r.total, r.content.as_str()), (120, "ok")); + assert_eq!(r.prompt_total(), 120); + // cache_n l'emporte ; débits donnés par le moteur gardés. + let r = parse_reply(br#"{"timings":{"prompt_n":"8","prompt_ms":4,"predicted_n":2,"predicted_ms":0,"cache_n":5,"prompt_per_second":7,"draft_n":0,"draft_n_accepted":0}}"#).unwrap(); + assert_eq!((r.turn.cached, r.turn.prompt_per_second), (5, 7.0)); + assert_eq!(r.turn.predicted_per_second, 0.0); + assert_eq!(r.turn.draft, None); + assert_eq!(r.prompt_total(), 13); + // Cache inconnu : -1, et le total sans lui. + let r = parse_reply( + br#"{"timings":{"prompt_n":8,"prompt_ms":4,"predicted_n":2,"predicted_ms":1}} trailing"#, + ) + .unwrap(); + assert_eq!( + (r.turn.cached, r.prompt_total(), r.content.as_str()), + (-1, 8, "") + ); + for bad in [ + &br#"{"timings":{"prompt_n":8,"prompt_ms":4,"predicted_n":2}}"#[..], + br#"{"timings":{"prompt_n":-1,"prompt_ms":4,"predicted_n":2,"predicted_ms":1}}"#, + br#"{"timings":"x"}"#, + b"pas du json", + br#"{"timings":{"prompt_n":1e13,"prompt_ms":4,"predicted_n":2,"predicted_ms":1}}"#, + ] { + assert_eq!( + parse_reply(bad), + Err(ReqError::Other(ERR_NO_TIMINGS.into())), + "{}", + String::from_utf8_lossy(bad) + ); + } +} + +#[test] +fn erreur_http() { + let e = ReqError::Http { + method: "POST".into(), + path: "/v1/chat/completions".into(), + status: "500 Internal Server Error".into(), + body: format!(" {}é{} ", "x".repeat(299), "y".repeat(10)), + }; + // 300 octets au plus, jamais au milieu d'un caractère. + assert_eq!( + e.to_string(), + format!( + "POST /v1/chat/completions : 500 Internal Server Error {}", + "x".repeat(299) + ) + ); + assert_eq!(go_op("POST"), "Post"); + assert_eq!(go_op("get"), "Get"); +} + +#[test] +fn flottants_du_go() { + let t = Turn { + prompt_ms: 3836.0, + prompt_per_second: 1037.5, + ..Default::default() + }; + let v = serde_json::to_string(&t).unwrap(); + assert!(v.contains(r#""prompt_ms":3836,"#), "{v}"); + assert!(v.contains(r#""prompt_per_second":1037.5"#), "{v}"); + let d = Depth { + reuse: -1.0, + ..Default::default() + }; + assert_eq!( + serde_json::to_value(&d).unwrap(), + json!({ "ctx": 0, "target": 0, "reuse": -1 }) + ); +} diff --git a/rust/loki-rs/src/chat/lease.rs b/rust/loki-rs/src/chat/lease.rs index f17fe44..52b89a4 100644 --- a/rust/loki-rs/src/chat/lease.rs +++ b/rust/loki-rs/src/chat/lease.rs @@ -76,7 +76,17 @@ pub struct EngineLease { impl EngineLease { /// Prend le verrou de génération pour `ttl` (borné par le Go à 5 s – 5 min). pub async fn acquire(store: Store, ttl: Duration) -> Result { - let body = json!({ "ttl_ms": ttl.as_millis() as u64 }); + Self::acquire_for(store, ttl, "").await + } + + /// Comme [`Self::acquire`], avec un motif : « bench » — le Go refuse alors + /// les autres avec le message du benchmark, et changer de discussion ne + /// rompt pas le bail. + pub async fn acquire_for(store: Store, ttl: Duration, why: &str) -> Result { + let mut body = json!({ "ttl_ms": ttl.as_millis() as u64 }); + if !why.is_empty() { + body["why"] = json!(why); + } let (status, v) = store .internal_post(PATH_LEASE, &body) .await diff --git a/rust/loki-rs/src/engstatus.rs b/rust/loki-rs/src/engstatus.rs index 7ea8bdd..cd019c8 100644 --- a/rust/loki-rs/src/engstatus.rs +++ b/rust/loki-rs/src/engstatus.rs @@ -184,7 +184,7 @@ pub fn service_log_tail(paths: &Paths, n: usize) -> String { /// Valeur brute de PORT comme `LLMPort` du Go (`strconv.Atoi`, > 0), sans /// borne haute : un port > 65535 doit faire échouer la sonde, pas viser 8080. -fn llm_port_raw(cfg: &HashMap) -> i64 { +pub fn llm_port_raw(cfg: &HashMap) -> i64 { if let Some(p) = cfg.get("PORT") && let Ok(n) = p.parse::() && n > 0 diff --git a/rust/loki-rs/src/lib.rs b/rust/loki-rs/src/lib.rs index fa08264..3070278 100644 --- a/rust/loki-rs/src/lib.rs +++ b/rust/loki-rs/src/lib.rs @@ -13,6 +13,7 @@ pub mod api; pub mod auth; pub mod backends; +pub mod bench; pub mod chat; pub mod chatapi; pub mod chatx; diff --git a/rust/loki-rs/src/main.rs b/rust/loki-rs/src/main.rs index 102e89d..7b00547 100644 --- a/rust/loki-rs/src/main.rs +++ b/rust/loki-rs/src/main.rs @@ -183,6 +183,9 @@ fn serve() -> ExitCode { // LOKI_TASKS=rust : les tâches planifiées (routes, planificateur, // tâches script) ; une tâche IA reste exécutée par le Go. let tasks_rust = held_by_facade("LOKI_TASKS"); + // LOKI_BENCH=rust : le banc d'essai (routes, job, protocole) ; le + // verrou de génération reste celui du Go, pris par le bail du moteur. + let bench_rust = held_by_facade("LOKI_BENCH"); let engine_rust = held_by_facade("LOKI_ENGINE") && match loki_rs::enginectl::usable( std::env::var("LOKI_CONTAINER").ok().as_deref(), @@ -218,6 +221,7 @@ fn serve() -> ExitCode { ("LOKI_ENGINE_IMAGE", engineimg_rust), ("LOKI_STRATA", strata_rust), ("LOKI_TASKS", tasks_rust), + ("LOKI_BENCH", bench_rust), ]; for (var, on) in held { if on { @@ -351,6 +355,8 @@ fn serve() -> ExitCode { }); let tasks = tasks_rust .then(|| std::sync::Arc::new(loki_rs::tasks::Tasks::new(engine_paths.home.clone()))); + let bench = bench_rust + .then(|| std::sync::Arc::new(loki_rs::bench::job::Bench::new(engine_paths.clone()))); let cfg = proxy::Config { backend, internal_token: token, @@ -373,6 +379,7 @@ fn serve() -> ExitCode { engineimg, strata, tasks, + bench, mcp: mcp_rust.then(|| std::sync::Arc::new(loki_rs::mcp::Mcp::new(mem_home))), }; let res = proxy::serve(listener, cfg, shutdown_signal()).await; diff --git a/rust/loki-rs/src/proxy.rs b/rust/loki-rs/src/proxy.rs index 17ca653..5311db4 100644 --- a/rust/loki-rs/src/proxy.rs +++ b/rust/loki-rs/src/proxy.rs @@ -106,6 +106,9 @@ pub struct Config { /// Tâches planifiées (routes, planificateur, tâches script) tenues par la /// façade (`LOKI_TASKS=rust`) ; `None` : au Go. pub tasks: Option>, + /// Banc d'essai (routes, job, protocole de mesure) tenu par la façade + /// (`LOKI_BENCH=rust`) ; `None` : au Go. + pub bench: Option>, } /// Arrête la tâche de fond tenue à sa sortie de portée. @@ -160,6 +163,7 @@ pub async fn serve( engineimg: cfg.engineimg, strata: cfg.strata, tasks: cfg.tasks, + bench: cfg.bench, internal_token: cfg.internal_token.clone(), }, }); @@ -287,6 +291,11 @@ impl Proxy { { return crate::tasks::serve(&self.ctx, t, req, from_go).await; } + if let Some(b) = &self.ctx.bench + && crate::bench::job::handles(path) + { + return crate::bench::job::serve(&self.ctx, b, req, from_go).await; + } if let Some(b) = &self.ctx.backends && crate::backends::handles(path) { diff --git a/rust/loki-rs/tests/bench.rs b/rust/loki-rs/tests/bench.rs new file mode 100644 index 0000000..48dde66 --- /dev/null +++ b/rust/loki-rs/tests/bench.rs @@ -0,0 +1,547 @@ +//! Lot 27 : banc d'essai tenu par la façade (`LOKI_BENCH=rust`). +//! +//! Le journal tiré du Go (`internal/loki/parity_bench_test.go` → +//! `tests/parity_bench/bench.json`) est rejoué par une façade contre le même +//! faux moteur (port de `fakeBenchEngine`) : chaque requête qu'il reçoit est +//! comparée, longs textes réduits à leur taille et à leur empreinte. + +mod common; + +use std::convert::Infallible; +use std::net::SocketAddr; +use std::path::PathBuf; +use std::sync::{Arc, Mutex}; +use std::time::Duration; + +use bytes::Bytes; +use common::fakego; +use common::parity::apply_files; +use http_body_util::{BodyExt, Full}; +use hyper::service::service_fn; +use hyper::{Request, Response}; +use hyper_util::rt::TokioIo; +use loki_rs::bench::job::Bench; +use regex::Regex; +use serde_json::{Map, Value, json}; +use sha2::Digest; +use tokio::net::TcpListener; + +fn fixture() -> Value { + serde_json::from_str(include_str!("parity_bench/bench.json")).expect("journal") +} + +fn s(v: &Value) -> &str { + v.as_str().unwrap_or("") +} + +const TUNE_BUSY: &str = "optimisation en cours (loki tune) : le moteur est réservé aux essais — attends la fin ou annule-la"; + +/// `parityBenchDigest`. +fn digest(v: &Value) -> Value { + match v { + Value::Object(o) => Value::Object(o.iter().map(|(k, e)| (k.clone(), digest(e))).collect()), + Value::Array(a) => Value::Array(a.iter().map(digest).collect()), + Value::String(x) if x.len() > 64 => { + let sum = sha2::Sha256::digest(x.as_bytes()); + Value::String(format!("<{} octets {}>", x.len(), hex::encode(&sum[..6]))) + } + _ => v.clone(), + } +} + +/// Le faux moteur (`fakeBenchEngine`) et le journal des requêtes reçues. +#[derive(Default)] +struct Fake { + n_ctx: i64, + busy: bool, + fail_at: usize, + no_timing: bool, + reject_effort: String, + slow_hot: Duration, + prev: i64, + chats: usize, + log: Vec, + /// En-têtes Authorization reçus. + auth: Vec, +} + +fn reply(code: u16, body: String) -> Response> { + let mut r = Response::new(Full::new(Bytes::from(body))); + *r.status_mut() = hyper::StatusCode::from_u16(code).unwrap(); + r +} + +async fn fake_engine(f: Arc>) -> u16 { + let l = TcpListener::bind("127.0.0.1:0").await.unwrap(); + let port = l.local_addr().unwrap().port(); + tokio::spawn(async move { + while let Ok((st, _)) = l.accept().await { + let f = f.clone(); + tokio::spawn(async move { + let svc = service_fn(move |req: Request| { + let f = f.clone(); + async move { Ok::<_, Infallible>(handle(f, req).await) } + }); + let _ = hyper::server::conn::http1::Builder::new() + .serve_connection(TokioIo::new(st), svc) + .await; + }); + } + }); + port +} + +async fn handle(f: Arc>, req: Request) -> Response> { + let method = req.method().to_string(); + let path = req.uri().path().to_string(); + if let Some(a) = req.headers().get("authorization") { + f.lock() + .unwrap() + .auth + .push(String::from_utf8_lossy(a.as_bytes()).into_owned()); + } + let raw = req.into_body().collect().await.unwrap().to_bytes(); + let body: Option = serde_json::from_slice(&raw).ok(); + { + let mut entry = Map::new(); + entry.insert("method".into(), json!(method)); + entry.insert("path".into(), json!(path)); + if let Some(b) = &body { + entry.insert("body".into(), digest(b)); + } + f.lock().unwrap().log.push(Value::Object(entry)); + } + match path.as_str() { + "/health" => reply(200, r#"{"status":"ok"}"#.into()), + "/slots" => { + let busy = f.lock().unwrap().busy; + reply(200, format!(r#"[{{"id":0,"is_processing":{busy}}}]"#)) + } + "/props" => { + let n = f.lock().unwrap().n_ctx; + reply( + 200, + format!( + r#"{{"build_info":"b9999-test","total_slots":1,"default_generation_settings":{{"n_ctx":{n}}}}}"# + ), + ) + } + "/tokenize" => { + let n = body + .as_ref() + .and_then(|b| b["content"].as_str()) + .map_or(0, |c| c.len() / 4); + reply(200, json!({ "tokens": vec![0; n] }).to_string() + "\n") + } + "/v1/chat/completions" => { + let p = body.unwrap_or(Value::Null); + let (n, fail_at, reject, slow) = { + let mut g = f.lock().unwrap(); + g.chats += 1; + (g.chats, g.fail_at, g.reject_effort.clone(), g.slow_hot) + }; + if fail_at == n { + return reply(500, "{\"error\":\"boom\"}\n".into()); + } + let effort = p["reasoning_effort"].as_str().unwrap_or(""); + if !reject.is_empty() && effort == reject { + return reply( + 500, + format!( + "{{\"error\":{{\"code\":500,\"message\":\"{}Unexpected reasoning effort {effort}. Supported types are xhigh (default), medium, and low.\"}}}}\n", + "jinja trace ".repeat(40) + ), + ); + } + let cache = p["cache_prompt"].as_bool() == Some(true); + if cache && !slow.is_zero() { + tokio::time::sleep(slow).await; + } + let total: i64 = p["messages"] + .as_array() + .into_iter() + .flatten() + .map(|m| m["content"].as_str().unwrap_or("").len() as i64 / 4 + 4) + .sum(); + let (cached, no_timing) = { + let mut g = f.lock().unwrap(); + let cached = if cache { g.prev.min(total) } else { 0 }; + g.prev = total; + (cached, g.no_timing) + }; + let gen_n = p["max_tokens"].as_f64().unwrap_or(0.0) as i64; + let mut resp = json!({ + "choices": [{ "message": { "role": "assistant", "content": "réponse" } }], + "usage": { "prompt_tokens": total, "completion_tokens": gen_n }, + }); + if !no_timing { + resp["timings"] = json!({ + "prompt_n": total - cached, + "prompt_ms": (total - cached) as f64 / 2.0, + "prompt_per_second": 2000.0, + "predicted_n": gen_n, + "predicted_ms": gen_n as f64 * 20.0, + "predicted_per_second": 50.0, + "cache_n": cached, + "draft_n": 10, + "draft_n_accepted": 7, + }); + } + reply(200, resp.to_string() + "\n") + } + _ => reply(404, "404 page not found\n".into()), + } +} + +struct Bn { + facade: SocketAddr, + go: Arc, + _stop: tokio::sync::oneshot::Sender<()>, + home: PathBuf, + bench: Arc, + fake: Arc>, + port: u16, + closed: u16, + _tmp: tempfile::TempDir, +} + +async fn start() -> Bn { + start_with(|_| {}).await +} + +async fn start_with(tweak: impl FnOnce(&mut Bench)) -> Bn { + let tmp = tempfile::tempdir().unwrap(); + let home = tmp.path().to_path_buf(); + let fake = Arc::new(Mutex::new(Fake::default())); + let port = fake_engine(fake.clone()).await; + let closed = std::net::TcpListener::bind("127.0.0.1:0") + .unwrap() + .local_addr() + .unwrap() + .port(); + let paths = loki_rs::engine::Paths { + home: home.clone(), + service: "loki-engine".into(), + }; + let mut bench = Bench::new(paths); + tweak(&mut bench); + let bench = Arc::new(bench); + let (facade, go, stop, _) = fakego::start_with( + home.clone(), + fakego::Opts { + bench: Some(bench.clone()), + ..Default::default() + }, + ) + .await; + Bn { + facade, + go, + _stop: stop, + home, + bench, + fake, + port, + closed, + _tmp: tmp, + } +} + +impl Bn { + fn mask(&self, v: &str) -> String { + let id = Regex::new(r#""b[0-9]+-[0-9]+""#).unwrap(); + let t = Regex::new(r#""(elapsed_sec|at)":[0-9.e+-]+"#).unwrap(); + let fp = Regex::new(r#""fingerprint":"[0-9a-f]+""#).unwrap(); + let v = v.replace(&self.home.to_string_lossy().into_owned(), "{{HOME}}"); + let v = id.replace_all(&v, r#""{{ID}}""#); + let v = t.replace_all(&v, r#""$1":"{{T}}""#); + fp.replace_all(&v, r#""fingerprint":"{{FP}}""#).into_owned() + } + + fn unmask(&self, v: &str) -> String { + v.replace("{{PORT}}", &self.port.to_string()) + .replace("{{CLOSED}}", &self.closed.to_string()) + .replace("{{HOME}}", &self.home.to_string_lossy()) + } + + fn decode(&self, raw: &str) -> Value { + if raw.is_empty() { + return Value::Null; + } + serde_json::from_str(&self.mask(raw)).unwrap_or_else(|_| panic!("illisible : {raw:?}")) + } + + async fn call(&self, method: &str, path: &str, body: &str) -> (u16, Value) { + let (code, b) = fakego::send(self.facade, method, path, &[], body).await; + (code.as_u16(), self.decode(&String::from_utf8_lossy(&b))) + } + + async fn status(&self) -> Value { + self.call("GET", "/api/bench/status", "").await.1 + } + + async fn wait(&self, name: &str) { + for _ in 0..12000 { + if self.status().await["running"] != json!(true) { + return; + } + tokio::time::sleep(Duration::from_millis(5)).await; + } + panic!("{name} : le bench ne finit pas"); + } +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn parite_du_banc_d_essai() { + let fx = fixture(); + let bn = start().await; + for st in fx["steps"].as_array().unwrap() { + let name = s(&st["name"]); + for (k, v) in st["config"].as_object().into_iter().flatten() { + // putStr du Go : une valeur vide retire la clé. + match bn.unmask(s(v)) { + v if v.is_empty() => bn.go.del("config", k), + v => bn.go.set("config", k, &v), + } + } + let files: Map = st["files"] + .as_object() + .into_iter() + .flatten() + .map(|(k, v)| (k.clone(), Value::String(bn.unmask(s(v))))) + .collect(); + apply_files(&bn.home, &Value::Object(files)); + if let Some(e) = st["engine"].as_object() { + let n = |k: &str| e.get(k).and_then(Value::as_i64).unwrap_or(0); + *bn.fake.lock().unwrap() = Fake { + n_ctx: n("n_ctx"), + busy: e.get("busy").and_then(Value::as_bool) == Some(true), + fail_at: n("fail_at") as usize, + no_timing: e.get("no_timing").and_then(Value::as_bool) == Some(true), + reject_effort: e + .get("reject_effort") + .and_then(Value::as_str) + .unwrap_or("") + .to_string(), + slow_hot: Duration::from_millis(n("slow_hot_ms") as u64), + ..Default::default() + }; + *bn.bench.req_timeout.lock().unwrap() = match n("timeout_ms") { + 0 => loki_rs::bench::REQ_TIMEOUT, + ms => Duration::from_millis(ms as u64), + }; + } + bn.fake.lock().unwrap().log.clear(); + *bn.go.tune_busy.lock().unwrap() = + (st["tune"].as_bool() == Some(true)).then(|| TUNE_BUSY.to_string()); + *bn.go.go_busy.lock().unwrap() = + (st["generating"].as_bool() == Some(true)).then(|| "génération en cours".to_string()); + let (code, out) = bn + .call(s(&st["method"]), s(&st["path"]), &bn.unmask(s(&st["body"]))) + .await; + *bn.go.tune_busy.lock().unwrap() = None; + *bn.go.go_busy.lock().unwrap() = None; + assert_eq!(code as i64, st["status"].as_i64().unwrap(), "{name}"); + assert_eq!(out, st["out"], "{name}"); + if st["again"].as_bool() == Some(true) { + let (_, again) = bn.call("POST", "/api/bench", "{}").await; + assert_eq!(again, st["again_out"], "{name} : second lancement"); + } + if let Some(phase) = st["cancel_at"].as_str() { + let mut ok = false; + for _ in 0..6000 { + let p = bn.status().await["phase"] + .as_str() + .unwrap_or("") + .to_string(); + if p.starts_with(phase) { + ok = true; + break; + } + tokio::time::sleep(Duration::from_millis(2)).await; + } + assert!(ok, "{name} : phase {phase} jamais atteinte"); + while bn.fake.lock().unwrap().chats < 4 { + tokio::time::sleep(Duration::from_millis(2)).await; + } + bn.call("POST", "/api/bench/cancel", "").await; + } + if st["wait"].as_bool() == Some(true) || st["cancel_at"].is_string() { + bn.wait(name).await; + assert_eq!(bn.status().await, st["job"], "{name} : job"); + } + // Les requêtes reçues : dans l'ordre, sauf les sondes /health d'un + // second lancement, concurrentes de celles du bench. + let mut got = bn.fake.lock().unwrap().log.clone(); + let mut want = st["calls"].as_array().cloned().unwrap_or_default(); + let key = |v: &Value| v.to_string(); + if st["again"].as_bool() == Some(true) { + got.sort_by_key(key); + want.sort_by_key(key); + } + assert_eq!( + Value::Array(got), + Value::Array(want), + "{name} : requêtes au moteur" + ); + let stored = |k: &str| { + bn.go + .get("state", k) + .map_or(Value::Null, |raw| bn.decode(&raw)) + }; + assert_eq!(stored("last_bench"), st["last"], "{name} : dernier bench"); + assert_eq!( + stored("bench_presets"), + st["presets"], + "{name} : par preset" + ); + if name == "mode rapide" { + // L'empreinte masquée du journal est bien celle de la + // configuration, et le Go a ouvert puis fermé le travail annexe. + let sb: Value = + serde_json::from_str(&bn.go.get("state", "last_bench").unwrap()).unwrap(); + let cfg = bn.go.config_map(); + assert_eq!( + sb["fingerprint"], + json!(loki_rs::presets_api::bench_fingerprint(&cfg, &cfg)) + ); + } + } +} + +impl Bn { + /// La configuration de base du journal, le faux moteur réglé. + fn base(&self, n_ctx: i64, slow_hot: Duration) { + for (k, v) in [ + ("MODEL", "/models/qwen.gguf"), + ("CTX", "16384"), + ("REASONING", "off"), + ("CACHE_ISOLATE", "off"), + ] { + self.go.set("config", k, v); + } + self.go.set("config", "PORT", &self.port.to_string()); + *self.fake.lock().unwrap() = Fake { + n_ctx, + slow_hot, + ..Default::default() + }; + } + + async fn until_turn(&self) { + for _ in 0..6000 { + let p = self.status().await["phase"] + .as_str() + .unwrap_or("") + .to_string(); + if p.starts_with("tour 1/") && self.fake.lock().unwrap().chats >= 4 { + return; + } + tokio::time::sleep(Duration::from_millis(2)).await; + } + panic!("le tour 1 n'arrive jamais"); + } +} + +/// Le bench prend le bail du moteur au motif « bench », ouvre puis ferme un +/// travail annexe (slot servi) et laisse sa trace dans la télémétrie. +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn bail_travail_annexe_et_trace() { + let bn = start().await; + bn.base(16384, Duration::ZERO); + bn.go.set("state", "api_key", "sk-moteur"); + let (code, _) = bn.call("POST", "/api/bench", "{}").await; + assert_eq!(code, 202); + bn.wait("bench").await; + assert_eq!(bn.status().await["saved"], json!(true)); + assert_eq!(*bn.go.lease_whys.lock().unwrap(), vec!["bench".to_string()]); + let log = bn.go.lease_log.lock().unwrap().clone(); + assert!( + log.first().is_some_and(|l| l.starts_with("acquire 30000")), + "{log:?}" + ); + assert!( + log.last().is_some_and(|l| l.starts_with("release ")), + "{log:?}" + ); + assert!(bn.go.lease.lock().unwrap().is_none(), "bail rendu"); + assert_eq!( + *bn.go.side_jobs.lock().unwrap(), + vec!["begin 1".to_string(), "end 1 true".to_string()] + ); + assert_eq!( + *bn.go.bench_perfs.lock().unwrap(), + vec![json!({ + "prompt_n": 2036, "prompt_ms": 1018.0, "prompt_per_second": 2000.0, + "predicted_n": 300, "predicted_per_second": 50.0, + })] + ); + // La clé de l'API locale accompagne les requêtes du bench. + let auth = bn.fake.lock().unwrap().auth.clone(); + assert!(!auth.is_empty()); + assert!(auth.iter().all(|a| a == "Bearer sk-moteur"), "{auth:?}"); + // Le second slot en service : le bench y passe. + *bn.go.side_live.lock().unwrap() = true; + bn.fake.lock().unwrap().log.clear(); + bn.call("POST", "/api/bench", "{}").await; + bn.wait("bench").await; + let chats: Vec = bn + .fake + .lock() + .unwrap() + .log + .iter() + .filter(|c| c["path"] == json!("/v1/chat/completions")) + .map(|c| c["body"]["id_slot"].clone()) + .collect(); + assert_eq!(chats, vec![json!(1), json!(1)]); + // Échec avant tout envoi : le travail annexe est fermé sans slot servi. + bn.fake.lock().unwrap().busy = true; + bn.go.side_jobs.lock().unwrap().clear(); + bn.call("POST", "/api/bench", "{}").await; + bn.wait("bench").await; + assert_eq!( + *bn.go.side_jobs.lock().unwrap(), + vec!["begin 1".to_string(), "end 1 false".to_string()] + ); +} + +/// Un stop du chat côté Go (signalé au renouvellement du bail) annule le +/// bench ; un bail perdu aussi. +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn stop_du_chat_et_bail_perdu() { + let bn = start_with(|b| b.lease_ttl = Duration::from_millis(300)).await; + bn.base(16384, Duration::from_secs(20)); + bn.call("POST", "/api/bench", r#"{"mode":"full"}"#).await; + bn.until_turn().await; + *bn.go.lease_stop.lock().unwrap() = true; + bn.wait("stop").await; + let st = bn.status().await; + assert_eq!( + (st["error"].clone(), st["canceled"].clone()), + (json!("benchmark annulé"), json!(true)) + ); + assert!(bn.go.lease.lock().unwrap().is_none(), "bail rendu"); + *bn.go.lease_stop.lock().unwrap() = false; + + bn.base(16384, Duration::from_secs(20)); + bn.call("POST", "/api/bench", r#"{"mode":"full"}"#).await; + bn.until_turn().await; + *bn.go.lease.lock().unwrap() = None; + bn.wait("perdu").await; + let st = bn.status().await; + assert_eq!( + (st["error"].clone(), st["canceled"].clone()), + (json!("benchmark annulé"), json!(true)) + ); +} + +/// Sans la façade pour le bench, ses routes partent au Go. +#[tokio::test] +async fn sans_la_facade() { + let tmp = tempfile::tempdir().unwrap(); + let (facade, _go, _stop) = fakego::start(tmp.path().to_path_buf()).await; + let (c, b) = fakego::send(facade, "GET", "/api/bench/status", &[], "").await; + assert_eq!(c.as_u16(), 200); + assert_eq!(String::from_utf8_lossy(&b), "go:/api/bench/status"); +} diff --git a/rust/loki-rs/tests/common/fakego.rs b/rust/loki-rs/tests/common/fakego.rs index 39abcee..c633790 100644 --- a/rust/loki-rs/tests/common/fakego.rs +++ b/rust/loki-rs/tests/common/fakego.rs @@ -109,6 +109,12 @@ pub struct FakeGo { pub task_run_delay: Mutex>, /// /internal/v1/push/send reçus : (titre, corps). pub push_sends: Mutex>, + /// /internal/v1/engine/lease reçus avec un motif (« bench »). + pub lease_whys: Mutex>, + /// /internal/v1/engine/sidejob : « begin n » ou « end n served ». + pub side_jobs: Mutex>, + /// /internal/v1/bench/perf reçus. + pub bench_perfs: Mutex>, } impl FakeGo { @@ -120,6 +126,19 @@ impl FakeGo { .or_default() .insert(key.into(), val.as_bytes().to_vec()); } + /// La configuration rangée. + pub fn config_map(&self) -> HashMap { + self.store + .lock() + .unwrap() + .get("config") + .map(|m| { + m.iter() + .map(|(k, v)| (k.clone(), String::from_utf8_lossy(v).into_owned())) + .collect() + }) + .unwrap_or_default() + } pub fn del(&self, bucket: &str, key: &str) { if let Some(b) = self.store.lock().unwrap().get_mut(bucket) { b.remove(key); @@ -227,6 +246,9 @@ async fn handle(st: Arc, req: Request) -> Response "/internal/v1/engine/lease" => { let body = req.into_body().collect().await.unwrap().to_bytes(); let v: serde_json::Value = serde_json::from_slice(&body).unwrap_or_default(); + if let Some(why) = v["why"].as_str() { + st.lease_whys.lock().unwrap().push(why.to_string()); + } if let Some(why) = st.go_busy.lock().unwrap().clone() { let out = serde_json::json!({"error": why, "task_id": "t1", "task_name": "veille"}); return resp(409, out.to_string()); @@ -508,6 +530,28 @@ async fn handle(st: Arc, req: Request) -> Response } resp(200, r#"{"busy":true}"#) } + "/internal/v1/engine/sidejob" => { + let body = req.into_body().collect().await.unwrap().to_bytes(); + let v: serde_json::Value = serde_json::from_slice(&body).unwrap_or_default(); + let mut log = st.side_jobs.lock().unwrap(); + match v["end"].as_i64() { + Some(id) if id != 0 => { + log.push(format!("end {id} {}", v["served"])); + resp(204, "") + } + _ => { + let id = log.len() + 1; + log.push(format!("begin {id}")); + resp(200, serde_json::json!({ "id": id }).to_string()) + } + } + } + "/internal/v1/bench/perf" => { + let body = req.into_body().collect().await.unwrap().to_bytes(); + let v: serde_json::Value = serde_json::from_slice(&body).unwrap_or_default(); + st.bench_perfs.lock().unwrap().push(v); + resp(204, "") + } "/internal/v1/push/send" => { let body = req.into_body().collect().await.unwrap().to_bytes(); let v: serde_json::Value = serde_json::from_slice(&body).unwrap_or_default(); @@ -705,6 +749,7 @@ pub struct Opts { pub engineimg: Option>, pub strata: Option>, pub tasks: Option>, + pub bench: Option>, /// Base de départ, posée avant que la façade démarre : (bucket, clé, /// valeur). pub seed: Vec<(String, String, String)>, @@ -816,6 +861,7 @@ pub async fn start_with( engineimg: opts.engineimg, strata: opts.strata, tasks: opts.tasks, + bench: opts.bench, }; tokio::spawn(loki_rs::proxy::serve(fl, cfg, async move { let _ = rx.await; diff --git a/rust/loki-rs/tests/lot2.rs b/rust/loki-rs/tests/lot2.rs index 2022452..83863b2 100644 --- a/rust/loki-rs/tests/lot2.rs +++ b/rust/loki-rs/tests/lot2.rs @@ -110,6 +110,7 @@ async fn banc(cle: Option<&str>) -> Banc { engineimg: None, strata: None, tasks: None, + bench: None, }; tokio::spawn(loki_rs::proxy::serve(l, cfg, async move { let _ = rx.await; diff --git a/rust/loki-rs/tests/parity_bench/bench.json b/rust/loki-rs/tests/parity_bench/bench.json new file mode 100644 index 0000000..9f15507 --- /dev/null +++ b/rust/loki-rs/tests/parity_bench/bench.json @@ -0,0 +1,3980 @@ +{ + "steps": [ + { + "name": "statut sans bench", + "config": { + "CACHE_ISOLATE": "off", + "CTX": "16384", + "MIN_P": "0.05", + "MODEL": "/models/qwen.gguf", + "PORT": "{{PORT}}", + "REASONING": "off", + "TOP_K": "40" + }, + "files": { + "presets/qwen.env": "MODEL=/models/qwen.gguf\nPORT={{PORT}}\nCTX=16384\nTOP_K=40\nMIN_P=0.05\nREASONING=off\nCACHE_ISOLATE=off\n" + }, + "engine": { + "n_ctx": 16384 + }, + "method": "GET", + "path": "/api/bench/status", + "status": 200, + "out": { + "id": "", + "mode": "", + "ok": true, + "running": false + }, + "calls": [], + "last": null, + "presets": null + }, + { + "name": "dernier bench : aucun", + "method": "GET", + "path": "/api/bench/last", + "status": 200, + "out": { + "ok": false + }, + "calls": [], + "last": null, + "presets": null + }, + { + "name": "lancement en GET", + "method": "GET", + "path": "/api/bench", + "status": 405, + "out": { + "error": "méthode GET refusée : POST attendu", + "ok": false + }, + "calls": [], + "last": null, + "presets": null + }, + { + "name": "annulation en GET", + "method": "GET", + "path": "/api/bench/cancel", + "status": 405, + "out": { + "error": "méthode GET refusée : POST attendu", + "ok": false + }, + "calls": [], + "last": null, + "presets": null + }, + { + "name": "annulation sans bench", + "method": "POST", + "path": "/api/bench/cancel", + "status": 200, + "out": { + "ok": false + }, + "calls": [], + "last": null, + "presets": null + }, + { + "name": "preset externe", + "config": { + "EXTERNAL": "1" + }, + "method": "POST", + "path": "/api/bench", + "body": "{}", + "status": 400, + "out": { + "error": "benchmark indisponible : le preset actif est une API externe", + "ok": false + }, + "calls": [], + "last": null, + "presets": null + }, + { + "name": "optimisation en cours", + "config": { + "EXTERNAL": "" + }, + "tune": true, + "method": "POST", + "path": "/api/bench", + "body": "{}", + "status": 409, + "out": { + "error": "optimisation en cours (loki tune) : le moteur est réservé aux essais — attends la fin ou annule-la", + "ok": false + }, + "calls": [], + "last": null, + "presets": null + }, + { + "name": "modèle pas prêt", + "config": { + "PORT": "{{CLOSED}}" + }, + "method": "POST", + "path": "/api/bench", + "body": "{}", + "status": 503, + "out": { + "error": "⏳ Le modèle est encore en train de charger — réessaie dans quelques secondes.", + "ok": false + }, + "calls": [], + "last": null, + "presets": null + }, + { + "name": "génération en cours", + "config": { + "PORT": "{{PORT}}" + }, + "generating": true, + "method": "POST", + "path": "/api/bench", + "body": "{}", + "status": 409, + "out": { + "error": "génération en cours", + "ok": false + }, + "calls": [ + { + "method": "GET", + "path": "/health" + } + ], + "last": null, + "presets": null + }, + { + "name": "moteur occupé par un autre", + "engine": { + "n_ctx": 16384, + "busy": true + }, + "method": "POST", + "path": "/api/bench", + "body": "{}", + "status": 202, + "out": { + "id": "{{ID}}", + "ok": true + }, + "wait": true, + "job": { + "canceled": false, + "elapsed_sec": "{{T}}", + "error": "le moteur traite déjà une requête : relance le bench une fois libre", + "id": "{{ID}}", + "mode": "quick", + "ok": true, + "phase": "démarrage", + "running": false, + "step": 0, + "steps": 0 + }, + "calls": [ + { + "method": "GET", + "path": "/health" + }, + { + "method": "GET", + "path": "/health" + }, + { + "method": "GET", + "path": "/slots" + } + ], + "last": null, + "presets": null + }, + { + "name": "mode rapide", + "engine": { + "n_ctx": 16384 + }, + "method": "POST", + "path": "/api/bench", + "body": "{}", + "status": 202, + "out": { + "id": "{{ID}}", + "ok": true + }, + "wait": true, + "job": { + "elapsed_sec": "{{T}}", + "id": "{{ID}}", + "mode": "quick", + "ok": true, + "phase": "ligne courte (2000 + 300 jetons)", + "result": { + "draft": { + "accepted": 7, + "n": 10 + }, + "elapsed_sec": "{{T}}", + "engine": "b9999-test", + "kv": "f16", + "mode": "quick", + "predicted_ms": 6000, + "predicted_n": 300, + "predicted_per_second": 50, + "prompt_ms": 1018, + "prompt_n": 2036, + "prompt_per_second": 2000, + "protocol": 2 + }, + "running": false, + "saved": true, + "step": 2, + "steps": 2 + }, + "calls": [ + { + "method": "GET", + "path": "/health" + }, + { + "method": "GET", + "path": "/health" + }, + { + "method": "GET", + "path": "/slots" + }, + { + "method": "GET", + "path": "/props" + }, + { + "body": { + "content": "<163664 octets b7452851885a>" + }, + "method": "POST", + "path": "/tokenize" + }, + { + "body": { + "content": "<3561 octets f977dc7d4950>" + }, + "method": "POST", + "path": "/tokenize" + }, + { + "body": { + "cache_prompt": false, + "chat_template_kwargs": { + "enable_thinking": false + }, + "max_tokens": 32, + "messages": [ + { + "content": "<4110 octets c386aa380c09>", + "role": "user" + } + ], + "min_p": 0.05, + "model": "loki", + "seed": 1234, + "stream": false, + "temperature": 0.7, + "top_k": 40 + }, + "method": "POST", + "path": "/v1/chat/completions" + }, + { + "body": { + "cache_prompt": false, + "chat_template_kwargs": { + "enable_thinking": false + }, + "max_tokens": 300, + "messages": [ + { + "content": "<8129 octets ae9cc838495d>", + "role": "user" + } + ], + "min_p": 0.05, + "model": "loki", + "seed": 1234, + "stream": false, + "temperature": 0.7, + "top_k": 40 + }, + "method": "POST", + "path": "/v1/chat/completions" + } + ], + "last": { + "at": "{{T}}", + "fingerprint": "{{FP}}", + "model": "qwen.gguf", + "result": { + "draft": { + "accepted": 7, + "n": 10 + }, + "elapsed_sec": "{{T}}", + "engine": "b9999-test", + "kv": "f16", + "mode": "quick", + "predicted_ms": 6000, + "predicted_n": 300, + "predicted_per_second": 50, + "prompt_ms": 1018, + "prompt_n": 2036, + "prompt_per_second": 2000, + "protocol": 2 + } + }, + "presets": { + "qwen": { + "at": "{{T}}", + "fingerprint": "{{FP}}", + "model": "qwen.gguf", + "result": { + "draft": { + "accepted": 7, + "n": 10 + }, + "elapsed_sec": "{{T}}", + "engine": "b9999-test", + "kv": "f16", + "mode": "quick", + "predicted_ms": 6000, + "predicted_n": 300, + "predicted_per_second": 50, + "prompt_ms": 1018, + "prompt_n": 2036, + "prompt_per_second": 2000, + "protocol": 2 + } + } + } + }, + { + "name": "dernier bench", + "method": "GET", + "path": "/api/bench/last", + "status": 200, + "out": { + "at": "{{T}}", + "model": "qwen.gguf", + "ok": true, + "result": { + "draft": { + "accepted": 7, + "n": 10 + }, + "elapsed_sec": "{{T}}", + "engine": "b9999-test", + "kv": "f16", + "mode": "quick", + "predicted_ms": 6000, + "predicted_n": 300, + "predicted_per_second": 50, + "prompt_ms": 1018, + "prompt_n": 2036, + "prompt_per_second": 2000, + "protocol": 2 + } + }, + "calls": [], + "last": { + "at": "{{T}}", + "fingerprint": "{{FP}}", + "model": "qwen.gguf", + "result": { + "draft": { + "accepted": 7, + "n": 10 + }, + "elapsed_sec": "{{T}}", + "engine": "b9999-test", + "kv": "f16", + "mode": "quick", + "predicted_ms": 6000, + "predicted_n": 300, + "predicted_per_second": 50, + "prompt_ms": 1018, + "prompt_n": 2036, + "prompt_per_second": 2000, + "protocol": 2 + } + }, + "presets": { + "qwen": { + "at": "{{T}}", + "fingerprint": "{{FP}}", + "model": "qwen.gguf", + "result": { + "draft": { + "accepted": 7, + "n": 10 + }, + "elapsed_sec": "{{T}}", + "engine": "b9999-test", + "kv": "f16", + "mode": "quick", + "predicted_ms": 6000, + "predicted_n": 300, + "predicted_per_second": 50, + "prompt_ms": 1018, + "prompt_n": 2036, + "prompt_per_second": 2000, + "protocol": 2 + } + } + } + }, + { + "name": "mode complet", + "engine": { + "n_ctx": 16384 + }, + "method": "POST", + "path": "/api/bench", + "body": "{\"mode\":\" FULL \"}", + "status": 202, + "out": { + "id": "{{ID}}", + "ok": true + }, + "wait": true, + "job": { + "elapsed_sec": "{{T}}", + "id": "{{ID}}", + "mode": "full", + "ok": true, + "phase": "tour 3/3 en profondeur", + "result": { + "depth": { + "cached_per_second": 2000, + "cold": { + "cached": 0, + "draft": { + "accepted": 7, + "n": 10 + }, + "expected": 0, + "new": 7672, + "predicted_ms": 5120, + "predicted_n": 256, + "predicted_per_second": 50, + "prompt_ms": 3836, + "prompt_per_second": 2000 + }, + "ctx": 16384, + "decode_per_second": 50, + "draft": { + "accepted": 21, + "n": 30 + }, + "reuse": 1, + "target": 7648, + "turns": [ + { + "cached": 7672, + "draft": { + "accepted": 7, + "n": 10 + }, + "expected": 7672, + "new": 2076, + "predicted_ms": 5120, + "predicted_n": 256, + "predicted_per_second": 50, + "prompt_ms": 1038, + "prompt_per_second": 2000 + }, + { + "cached": 9748, + "draft": { + "accepted": 7, + "n": 10 + }, + "expected": 9748, + "new": 2075, + "predicted_ms": 5120, + "predicted_n": 256, + "predicted_per_second": 50, + "prompt_ms": 1037.5, + "prompt_per_second": 2000 + }, + { + "cached": 11823, + "draft": { + "accepted": 7, + "n": 10 + }, + "expected": 11823, + "new": 2078, + "predicted_ms": 5120, + "predicted_n": 256, + "predicted_per_second": 50, + "prompt_ms": 1039, + "prompt_per_second": 2000 + } + ] + }, + "draft": { + "accepted": 7, + "n": 10 + }, + "elapsed_sec": "{{T}}", + "engine": "b9999-test", + "kv": "f16", + "mode": "full", + "predicted_ms": 6000, + "predicted_n": 300, + "predicted_per_second": 50, + "prompt_ms": 1018, + "prompt_n": 2036, + "prompt_per_second": 2000, + "protocol": 2 + }, + "running": false, + "saved": true, + "step": 6, + "steps": 6 + }, + "calls": [ + { + "method": "GET", + "path": "/health" + }, + { + "method": "GET", + "path": "/health" + }, + { + "method": "GET", + "path": "/slots" + }, + { + "method": "GET", + "path": "/props" + }, + { + "body": { + "content": "<163664 octets b7452851885a>" + }, + "method": "POST", + "path": "/tokenize" + }, + { + "body": { + "content": "<3561 octets f977dc7d4950>" + }, + "method": "POST", + "path": "/tokenize" + }, + { + "body": { + "cache_prompt": false, + "chat_template_kwargs": { + "enable_thinking": false + }, + "max_tokens": 32, + "messages": [ + { + "content": "<4110 octets c386aa380c09>", + "role": "user" + } + ], + "min_p": 0.05, + "model": "loki", + "seed": 1234, + "stream": false, + "temperature": 0.7, + "top_k": 40 + }, + "method": "POST", + "path": "/v1/chat/completions" + }, + { + "body": { + "cache_prompt": false, + "chat_template_kwargs": { + "enable_thinking": false + }, + "max_tokens": 300, + "messages": [ + { + "content": "<8129 octets ae9cc838495d>", + "role": "user" + } + ], + "min_p": 0.05, + "model": "loki", + "seed": 1234, + "stream": false, + "temperature": 0.7, + "top_k": 40 + }, + "method": "POST", + "path": "/v1/chat/completions" + }, + { + "body": { + "cache_prompt": false, + "chat_template_kwargs": { + "enable_thinking": false + }, + "max_tokens": 256, + "messages": [ + { + "content": "<30673 octets 41db43c5b144>", + "role": "user" + } + ], + "min_p": 0.05, + "model": "loki", + "seed": 1234, + "stream": false, + "temperature": 0.7, + "top_k": 40 + }, + "method": "POST", + "path": "/v1/chat/completions" + }, + { + "body": { + "cache_prompt": true, + "chat_template_kwargs": { + "enable_thinking": false + }, + "max_tokens": 256, + "messages": [ + { + "content": "<30673 octets 41db43c5b144>", + "role": "user" + }, + { + "content": "réponse", + "role": "assistant" + }, + { + "content": "<8264 octets eb0a24749ba3>", + "role": "user" + } + ], + "min_p": 0.05, + "model": "loki", + "seed": 1234, + "stream": false, + "temperature": 0.7, + "top_k": 40 + }, + "method": "POST", + "path": "/v1/chat/completions" + }, + { + "body": { + "cache_prompt": true, + "chat_template_kwargs": { + "enable_thinking": false + }, + "max_tokens": 256, + "messages": [ + { + "content": "<30673 octets 41db43c5b144>", + "role": "user" + }, + { + "content": "réponse", + "role": "assistant" + }, + { + "content": "<8264 octets eb0a24749ba3>", + "role": "user" + }, + { + "content": "réponse", + "role": "assistant" + }, + { + "content": "<8263 octets 273e9def4ac3>", + "role": "user" + } + ], + "min_p": 0.05, + "model": "loki", + "seed": 1234, + "stream": false, + "temperature": 0.7, + "top_k": 40 + }, + "method": "POST", + "path": "/v1/chat/completions" + }, + { + "body": { + "cache_prompt": true, + "chat_template_kwargs": { + "enable_thinking": false + }, + "max_tokens": 256, + "messages": [ + { + "content": "<30673 octets 41db43c5b144>", + "role": "user" + }, + { + "content": "réponse", + "role": "assistant" + }, + { + "content": "<8264 octets eb0a24749ba3>", + "role": "user" + }, + { + "content": "réponse", + "role": "assistant" + }, + { + "content": "<8263 octets 273e9def4ac3>", + "role": "user" + }, + { + "content": "réponse", + "role": "assistant" + }, + { + "content": "<8273 octets 6c5196e1654d>", + "role": "user" + } + ], + "min_p": 0.05, + "model": "loki", + "seed": 1234, + "stream": false, + "temperature": 0.7, + "top_k": 40 + }, + "method": "POST", + "path": "/v1/chat/completions" + } + ], + "last": { + "at": "{{T}}", + "fingerprint": "{{FP}}", + "model": "qwen.gguf", + "result": { + "depth": { + "cached_per_second": 2000, + "cold": { + "cached": 0, + "draft": { + "accepted": 7, + "n": 10 + }, + "expected": 0, + "new": 7672, + "predicted_ms": 5120, + "predicted_n": 256, + "predicted_per_second": 50, + "prompt_ms": 3836, + "prompt_per_second": 2000 + }, + "ctx": 16384, + "decode_per_second": 50, + "draft": { + "accepted": 21, + "n": 30 + }, + "reuse": 1, + "target": 7648, + "turns": [ + { + "cached": 7672, + "draft": { + "accepted": 7, + "n": 10 + }, + "expected": 7672, + "new": 2076, + "predicted_ms": 5120, + "predicted_n": 256, + "predicted_per_second": 50, + "prompt_ms": 1038, + "prompt_per_second": 2000 + }, + { + "cached": 9748, + "draft": { + "accepted": 7, + "n": 10 + }, + "expected": 9748, + "new": 2075, + "predicted_ms": 5120, + "predicted_n": 256, + "predicted_per_second": 50, + "prompt_ms": 1037.5, + "prompt_per_second": 2000 + }, + { + "cached": 11823, + "draft": { + "accepted": 7, + "n": 10 + }, + "expected": 11823, + "new": 2078, + "predicted_ms": 5120, + "predicted_n": 256, + "predicted_per_second": 50, + "prompt_ms": 1039, + "prompt_per_second": 2000 + } + ] + }, + "draft": { + "accepted": 7, + "n": 10 + }, + "elapsed_sec": "{{T}}", + "engine": "b9999-test", + "kv": "f16", + "mode": "full", + "predicted_ms": 6000, + "predicted_n": 300, + "predicted_per_second": 50, + "prompt_ms": 1018, + "prompt_n": 2036, + "prompt_per_second": 2000, + "protocol": 2 + } + }, + "presets": { + "qwen": { + "at": "{{T}}", + "fingerprint": "{{FP}}", + "model": "qwen.gguf", + "result": { + "depth": { + "cached_per_second": 2000, + "cold": { + "cached": 0, + "draft": { + "accepted": 7, + "n": 10 + }, + "expected": 0, + "new": 7672, + "predicted_ms": 5120, + "predicted_n": 256, + "predicted_per_second": 50, + "prompt_ms": 3836, + "prompt_per_second": 2000 + }, + "ctx": 16384, + "decode_per_second": 50, + "draft": { + "accepted": 21, + "n": 30 + }, + "reuse": 1, + "target": 7648, + "turns": [ + { + "cached": 7672, + "draft": { + "accepted": 7, + "n": 10 + }, + "expected": 7672, + "new": 2076, + "predicted_ms": 5120, + "predicted_n": 256, + "predicted_per_second": 50, + "prompt_ms": 1038, + "prompt_per_second": 2000 + }, + { + "cached": 9748, + "draft": { + "accepted": 7, + "n": 10 + }, + "expected": 9748, + "new": 2075, + "predicted_ms": 5120, + "predicted_n": 256, + "predicted_per_second": 50, + "prompt_ms": 1037.5, + "prompt_per_second": 2000 + }, + { + "cached": 11823, + "draft": { + "accepted": 7, + "n": 10 + }, + "expected": 11823, + "new": 2078, + "predicted_ms": 5120, + "predicted_n": 256, + "predicted_per_second": 50, + "prompt_ms": 1039, + "prompt_per_second": 2000 + } + ] + }, + "draft": { + "accepted": 7, + "n": 10 + }, + "elapsed_sec": "{{T}}", + "engine": "b9999-test", + "kv": "f16", + "mode": "full", + "predicted_ms": 6000, + "predicted_n": 300, + "predicted_per_second": 50, + "prompt_ms": 1018, + "prompt_n": 2036, + "prompt_per_second": 2000, + "protocol": 2 + } + } + } + }, + { + "name": "mode complet, contexte trop petit", + "engine": { + "n_ctx": 4096 + }, + "method": "POST", + "path": "/api/bench", + "body": "{\"mode\":\"full\"}", + "status": 202, + "out": { + "id": "{{ID}}", + "ok": true + }, + "wait": true, + "job": { + "elapsed_sec": "{{T}}", + "id": "{{ID}}", + "mode": "full", + "ok": true, + "phase": "ligne courte (2000 + 300 jetons)", + "result": { + "depth": { + "ctx": 4096, + "reuse": -1, + "skipped": "contexte de 4096 jetons trop petit pour la mesure en profondeur", + "target": 0 + }, + "draft": { + "accepted": 7, + "n": 10 + }, + "elapsed_sec": "{{T}}", + "engine": "b9999-test", + "kv": "f16", + "mode": "full", + "predicted_ms": 6000, + "predicted_n": 300, + "predicted_per_second": 50, + "prompt_ms": 1018, + "prompt_n": 2036, + "prompt_per_second": 2000, + "protocol": 2 + }, + "running": false, + "saved": true, + "step": 2, + "steps": 6 + }, + "calls": [ + { + "method": "GET", + "path": "/health" + }, + { + "method": "GET", + "path": "/health" + }, + { + "method": "GET", + "path": "/slots" + }, + { + "method": "GET", + "path": "/props" + }, + { + "body": { + "content": "<163664 octets b7452851885a>" + }, + "method": "POST", + "path": "/tokenize" + }, + { + "body": { + "content": "<3561 octets f977dc7d4950>" + }, + "method": "POST", + "path": "/tokenize" + }, + { + "body": { + "cache_prompt": false, + "chat_template_kwargs": { + "enable_thinking": false + }, + "max_tokens": 32, + "messages": [ + { + "content": "<4110 octets c386aa380c09>", + "role": "user" + } + ], + "min_p": 0.05, + "model": "loki", + "seed": 1234, + "stream": false, + "temperature": 0.7, + "top_k": 40 + }, + "method": "POST", + "path": "/v1/chat/completions" + }, + { + "body": { + "cache_prompt": false, + "chat_template_kwargs": { + "enable_thinking": false + }, + "max_tokens": 300, + "messages": [ + { + "content": "<8129 octets ae9cc838495d>", + "role": "user" + } + ], + "min_p": 0.05, + "model": "loki", + "seed": 1234, + "stream": false, + "temperature": 0.7, + "top_k": 40 + }, + "method": "POST", + "path": "/v1/chat/completions" + } + ], + "last": { + "at": "{{T}}", + "fingerprint": "{{FP}}", + "model": "qwen.gguf", + "result": { + "depth": { + "ctx": 4096, + "reuse": -1, + "skipped": "contexte de 4096 jetons trop petit pour la mesure en profondeur", + "target": 0 + }, + "draft": { + "accepted": 7, + "n": 10 + }, + "elapsed_sec": "{{T}}", + "engine": "b9999-test", + "kv": "f16", + "mode": "full", + "predicted_ms": 6000, + "predicted_n": 300, + "predicted_per_second": 50, + "prompt_ms": 1018, + "prompt_n": 2036, + "prompt_per_second": 2000, + "protocol": 2 + } + }, + "presets": { + "qwen": { + "at": "{{T}}", + "fingerprint": "{{FP}}", + "model": "qwen.gguf", + "result": { + "depth": { + "ctx": 4096, + "reuse": -1, + "skipped": "contexte de 4096 jetons trop petit pour la mesure en profondeur", + "target": 0 + }, + "draft": { + "accepted": 7, + "n": 10 + }, + "elapsed_sec": "{{T}}", + "engine": "b9999-test", + "kv": "f16", + "mode": "full", + "predicted_ms": 6000, + "predicted_n": 300, + "predicted_per_second": 50, + "prompt_ms": 1018, + "prompt_n": 2036, + "prompt_per_second": 2000, + "protocol": 2 + } + } + } + }, + { + "name": "contexte inconnu du moteur : celui du preset", + "engine": { + "n_ctx": 0 + }, + "method": "POST", + "path": "/api/bench", + "body": "{\"mode\":\"full\"}", + "status": 202, + "out": { + "id": "{{ID}}", + "ok": true + }, + "wait": true, + "job": { + "elapsed_sec": "{{T}}", + "id": "{{ID}}", + "mode": "full", + "ok": true, + "phase": "tour 3/3 en profondeur", + "result": { + "depth": { + "cached_per_second": 2000, + "cold": { + "cached": 0, + "draft": { + "accepted": 7, + "n": 10 + }, + "expected": 0, + "new": 7672, + "predicted_ms": 5120, + "predicted_n": 256, + "predicted_per_second": 50, + "prompt_ms": 3836, + "prompt_per_second": 2000 + }, + "ctx": 16384, + "decode_per_second": 50, + "draft": { + "accepted": 21, + "n": 30 + }, + "reuse": 1, + "target": 7648, + "turns": [ + { + "cached": 7672, + "draft": { + "accepted": 7, + "n": 10 + }, + "expected": 7672, + "new": 2076, + "predicted_ms": 5120, + "predicted_n": 256, + "predicted_per_second": 50, + "prompt_ms": 1038, + "prompt_per_second": 2000 + }, + { + "cached": 9748, + "draft": { + "accepted": 7, + "n": 10 + }, + "expected": 9748, + "new": 2075, + "predicted_ms": 5120, + "predicted_n": 256, + "predicted_per_second": 50, + "prompt_ms": 1037.5, + "prompt_per_second": 2000 + }, + { + "cached": 11823, + "draft": { + "accepted": 7, + "n": 10 + }, + "expected": 11823, + "new": 2078, + "predicted_ms": 5120, + "predicted_n": 256, + "predicted_per_second": 50, + "prompt_ms": 1039, + "prompt_per_second": 2000 + } + ] + }, + "draft": { + "accepted": 7, + "n": 10 + }, + "elapsed_sec": "{{T}}", + "engine": "b9999-test", + "kv": "f16", + "mode": "full", + "predicted_ms": 6000, + "predicted_n": 300, + "predicted_per_second": 50, + "prompt_ms": 1018, + "prompt_n": 2036, + "prompt_per_second": 2000, + "protocol": 2 + }, + "running": false, + "saved": true, + "step": 6, + "steps": 6 + }, + "calls": [ + { + "method": "GET", + "path": "/health" + }, + { + "method": "GET", + "path": "/health" + }, + { + "method": "GET", + "path": "/slots" + }, + { + "method": "GET", + "path": "/props" + }, + { + "body": { + "content": "<163664 octets b7452851885a>" + }, + "method": "POST", + "path": "/tokenize" + }, + { + "body": { + "content": "<3561 octets f977dc7d4950>" + }, + "method": "POST", + "path": "/tokenize" + }, + { + "body": { + "cache_prompt": false, + "chat_template_kwargs": { + "enable_thinking": false + }, + "max_tokens": 32, + "messages": [ + { + "content": "<4110 octets c386aa380c09>", + "role": "user" + } + ], + "min_p": 0.05, + "model": "loki", + "seed": 1234, + "stream": false, + "temperature": 0.7, + "top_k": 40 + }, + "method": "POST", + "path": "/v1/chat/completions" + }, + { + "body": { + "cache_prompt": false, + "chat_template_kwargs": { + "enable_thinking": false + }, + "max_tokens": 300, + "messages": [ + { + "content": "<8129 octets ae9cc838495d>", + "role": "user" + } + ], + "min_p": 0.05, + "model": "loki", + "seed": 1234, + "stream": false, + "temperature": 0.7, + "top_k": 40 + }, + "method": "POST", + "path": "/v1/chat/completions" + }, + { + "body": { + "cache_prompt": false, + "chat_template_kwargs": { + "enable_thinking": false + }, + "max_tokens": 256, + "messages": [ + { + "content": "<30673 octets 41db43c5b144>", + "role": "user" + } + ], + "min_p": 0.05, + "model": "loki", + "seed": 1234, + "stream": false, + "temperature": 0.7, + "top_k": 40 + }, + "method": "POST", + "path": "/v1/chat/completions" + }, + { + "body": { + "cache_prompt": true, + "chat_template_kwargs": { + "enable_thinking": false + }, + "max_tokens": 256, + "messages": [ + { + "content": "<30673 octets 41db43c5b144>", + "role": "user" + }, + { + "content": "réponse", + "role": "assistant" + }, + { + "content": "<8264 octets eb0a24749ba3>", + "role": "user" + } + ], + "min_p": 0.05, + "model": "loki", + "seed": 1234, + "stream": false, + "temperature": 0.7, + "top_k": 40 + }, + "method": "POST", + "path": "/v1/chat/completions" + }, + { + "body": { + "cache_prompt": true, + "chat_template_kwargs": { + "enable_thinking": false + }, + "max_tokens": 256, + "messages": [ + { + "content": "<30673 octets 41db43c5b144>", + "role": "user" + }, + { + "content": "réponse", + "role": "assistant" + }, + { + "content": "<8264 octets eb0a24749ba3>", + "role": "user" + }, + { + "content": "réponse", + "role": "assistant" + }, + { + "content": "<8263 octets 273e9def4ac3>", + "role": "user" + } + ], + "min_p": 0.05, + "model": "loki", + "seed": 1234, + "stream": false, + "temperature": 0.7, + "top_k": 40 + }, + "method": "POST", + "path": "/v1/chat/completions" + }, + { + "body": { + "cache_prompt": true, + "chat_template_kwargs": { + "enable_thinking": false + }, + "max_tokens": 256, + "messages": [ + { + "content": "<30673 octets 41db43c5b144>", + "role": "user" + }, + { + "content": "réponse", + "role": "assistant" + }, + { + "content": "<8264 octets eb0a24749ba3>", + "role": "user" + }, + { + "content": "réponse", + "role": "assistant" + }, + { + "content": "<8263 octets 273e9def4ac3>", + "role": "user" + }, + { + "content": "réponse", + "role": "assistant" + }, + { + "content": "<8273 octets 6c5196e1654d>", + "role": "user" + } + ], + "min_p": 0.05, + "model": "loki", + "seed": 1234, + "stream": false, + "temperature": 0.7, + "top_k": 40 + }, + "method": "POST", + "path": "/v1/chat/completions" + } + ], + "last": { + "at": "{{T}}", + "fingerprint": "{{FP}}", + "model": "qwen.gguf", + "result": { + "depth": { + "cached_per_second": 2000, + "cold": { + "cached": 0, + "draft": { + "accepted": 7, + "n": 10 + }, + "expected": 0, + "new": 7672, + "predicted_ms": 5120, + "predicted_n": 256, + "predicted_per_second": 50, + "prompt_ms": 3836, + "prompt_per_second": 2000 + }, + "ctx": 16384, + "decode_per_second": 50, + "draft": { + "accepted": 21, + "n": 30 + }, + "reuse": 1, + "target": 7648, + "turns": [ + { + "cached": 7672, + "draft": { + "accepted": 7, + "n": 10 + }, + "expected": 7672, + "new": 2076, + "predicted_ms": 5120, + "predicted_n": 256, + "predicted_per_second": 50, + "prompt_ms": 1038, + "prompt_per_second": 2000 + }, + { + "cached": 9748, + "draft": { + "accepted": 7, + "n": 10 + }, + "expected": 9748, + "new": 2075, + "predicted_ms": 5120, + "predicted_n": 256, + "predicted_per_second": 50, + "prompt_ms": 1037.5, + "prompt_per_second": 2000 + }, + { + "cached": 11823, + "draft": { + "accepted": 7, + "n": 10 + }, + "expected": 11823, + "new": 2078, + "predicted_ms": 5120, + "predicted_n": 256, + "predicted_per_second": 50, + "prompt_ms": 1039, + "prompt_per_second": 2000 + } + ] + }, + "draft": { + "accepted": 7, + "n": 10 + }, + "elapsed_sec": "{{T}}", + "engine": "b9999-test", + "kv": "f16", + "mode": "full", + "predicted_ms": 6000, + "predicted_n": 300, + "predicted_per_second": 50, + "prompt_ms": 1018, + "prompt_n": 2036, + "prompt_per_second": 2000, + "protocol": 2 + } + }, + "presets": { + "qwen": { + "at": "{{T}}", + "fingerprint": "{{FP}}", + "model": "qwen.gguf", + "result": { + "depth": { + "cached_per_second": 2000, + "cold": { + "cached": 0, + "draft": { + "accepted": 7, + "n": 10 + }, + "expected": 0, + "new": 7672, + "predicted_ms": 5120, + "predicted_n": 256, + "predicted_per_second": 50, + "prompt_ms": 3836, + "prompt_per_second": 2000 + }, + "ctx": 16384, + "decode_per_second": 50, + "draft": { + "accepted": 21, + "n": 30 + }, + "reuse": 1, + "target": 7648, + "turns": [ + { + "cached": 7672, + "draft": { + "accepted": 7, + "n": 10 + }, + "expected": 7672, + "new": 2076, + "predicted_ms": 5120, + "predicted_n": 256, + "predicted_per_second": 50, + "prompt_ms": 1038, + "prompt_per_second": 2000 + }, + { + "cached": 9748, + "draft": { + "accepted": 7, + "n": 10 + }, + "expected": 9748, + "new": 2075, + "predicted_ms": 5120, + "predicted_n": 256, + "predicted_per_second": 50, + "prompt_ms": 1037.5, + "prompt_per_second": 2000 + }, + { + "cached": 11823, + "draft": { + "accepted": 7, + "n": 10 + }, + "expected": 11823, + "new": 2078, + "predicted_ms": 5120, + "predicted_n": 256, + "predicted_per_second": 50, + "prompt_ms": 1039, + "prompt_per_second": 2000 + } + ] + }, + "draft": { + "accepted": 7, + "n": 10 + }, + "elapsed_sec": "{{T}}", + "engine": "b9999-test", + "kv": "f16", + "mode": "full", + "predicted_ms": 6000, + "predicted_n": 300, + "predicted_per_second": 50, + "prompt_ms": 1018, + "prompt_n": 2036, + "prompt_per_second": 2000, + "protocol": 2 + } + } + } + }, + { + "name": "paramètres bornés", + "engine": { + "n_ctx": 65536 + }, + "method": "POST", + "path": "/api/bench", + "body": "{\"prompt\":99999,\"n\":9999,\"mode\":\"quick\"}", + "status": 202, + "out": { + "id": "{{ID}}", + "ok": true + }, + "wait": true, + "job": { + "elapsed_sec": "{{T}}", + "id": "{{ID}}", + "mode": "quick", + "ok": true, + "phase": "ligne courte (32768 + 4096 jetons)", + "result": { + "draft": { + "accepted": 7, + "n": 10 + }, + "elapsed_sec": "{{T}}", + "engine": "b9999-test", + "kv": "f16", + "mode": "quick", + "predicted_ms": 81920, + "predicted_n": 4096, + "predicted_per_second": 50, + "prompt_ms": 16403, + "prompt_n": 32806, + "prompt_per_second": 2000, + "protocol": 2 + }, + "running": false, + "saved": true, + "step": 2, + "steps": 2 + }, + "calls": [ + { + "method": "GET", + "path": "/health" + }, + { + "method": "GET", + "path": "/health" + }, + { + "method": "GET", + "path": "/slots" + }, + { + "method": "GET", + "path": "/props" + }, + { + "body": { + "content": "<163664 octets b7452851885a>" + }, + "method": "POST", + "path": "/tokenize" + }, + { + "body": { + "content": "<3561 octets f977dc7d4950>" + }, + "method": "POST", + "path": "/tokenize" + }, + { + "body": { + "cache_prompt": false, + "chat_template_kwargs": { + "enable_thinking": false + }, + "max_tokens": 32, + "messages": [ + { + "content": "<4110 octets c386aa380c09>", + "role": "user" + } + ], + "min_p": 0.05, + "model": "loki", + "seed": 1234, + "stream": false, + "temperature": 0.7, + "top_k": 40 + }, + "method": "POST", + "path": "/v1/chat/completions" + }, + { + "body": { + "cache_prompt": false, + "chat_template_kwargs": { + "enable_thinking": false + }, + "max_tokens": 4096, + "messages": [ + { + "content": "<131209 octets a079a2b6a1db>", + "role": "user" + } + ], + "min_p": 0.05, + "model": "loki", + "seed": 1234, + "stream": false, + "temperature": 0.7, + "top_k": 40 + }, + "method": "POST", + "path": "/v1/chat/completions" + } + ], + "last": { + "at": "{{T}}", + "fingerprint": "{{FP}}", + "model": "qwen.gguf", + "result": { + "draft": { + "accepted": 7, + "n": 10 + }, + "elapsed_sec": "{{T}}", + "engine": "b9999-test", + "kv": "f16", + "mode": "quick", + "predicted_ms": 81920, + "predicted_n": 4096, + "predicted_per_second": 50, + "prompt_ms": 16403, + "prompt_n": 32806, + "prompt_per_second": 2000, + "protocol": 2 + } + }, + "presets": { + "qwen": { + "at": "{{T}}", + "fingerprint": "{{FP}}", + "model": "qwen.gguf", + "result": { + "draft": { + "accepted": 7, + "n": 10 + }, + "elapsed_sec": "{{T}}", + "engine": "b9999-test", + "kv": "f16", + "mode": "quick", + "predicted_ms": 81920, + "predicted_n": 4096, + "predicted_per_second": 50, + "prompt_ms": 16403, + "prompt_n": 32806, + "prompt_per_second": 2000, + "protocol": 2 + } + } + } + }, + { + "name": "petits paramètres", + "engine": { + "n_ctx": 16384 + }, + "method": "POST", + "path": "/api/bench", + "body": "{\"prompt\":100,\"n\":8}", + "status": 202, + "out": { + "id": "{{ID}}", + "ok": true + }, + "wait": true, + "job": { + "elapsed_sec": "{{T}}", + "id": "{{ID}}", + "mode": "quick", + "ok": true, + "phase": "ligne courte (100 + 8 jetons)", + "result": { + "draft": { + "accepted": 7, + "n": 10 + }, + "elapsed_sec": "{{T}}", + "engine": "b9999-test", + "kv": "f16", + "mode": "quick", + "predicted_ms": 160, + "predicted_n": 8, + "predicted_per_second": 50, + "prompt_ms": 69.5, + "prompt_n": 139, + "prompt_per_second": 2000, + "protocol": 2 + }, + "running": false, + "saved": true, + "step": 2, + "steps": 2 + }, + "calls": [ + { + "method": "GET", + "path": "/health" + }, + { + "method": "GET", + "path": "/health" + }, + { + "method": "GET", + "path": "/slots" + }, + { + "method": "GET", + "path": "/props" + }, + { + "body": { + "content": "<163664 octets b7452851885a>" + }, + "method": "POST", + "path": "/tokenize" + }, + { + "body": { + "content": "<3561 octets f977dc7d4950>" + }, + "method": "POST", + "path": "/tokenize" + }, + { + "body": { + "cache_prompt": false, + "chat_template_kwargs": { + "enable_thinking": false + }, + "max_tokens": 32, + "messages": [ + { + "content": "<4110 octets c386aa380c09>", + "role": "user" + } + ], + "min_p": 0.05, + "model": "loki", + "seed": 1234, + "stream": false, + "temperature": 0.7, + "top_k": 40 + }, + "method": "POST", + "path": "/v1/chat/completions" + }, + { + "body": { + "cache_prompt": false, + "chat_template_kwargs": { + "enable_thinking": false + }, + "max_tokens": 8, + "messages": [ + { + "content": "<542 octets 4782ba6c4bb7>", + "role": "user" + } + ], + "min_p": 0.05, + "model": "loki", + "seed": 1234, + "stream": false, + "temperature": 0.7, + "top_k": 40 + }, + "method": "POST", + "path": "/v1/chat/completions" + } + ], + "last": { + "at": "{{T}}", + "fingerprint": "{{FP}}", + "model": "qwen.gguf", + "result": { + "draft": { + "accepted": 7, + "n": 10 + }, + "elapsed_sec": "{{T}}", + "engine": "b9999-test", + "kv": "f16", + "mode": "quick", + "predicted_ms": 160, + "predicted_n": 8, + "predicted_per_second": 50, + "prompt_ms": 69.5, + "prompt_n": 139, + "prompt_per_second": 2000, + "protocol": 2 + } + }, + "presets": { + "qwen": { + "at": "{{T}}", + "fingerprint": "{{FP}}", + "model": "qwen.gguf", + "result": { + "draft": { + "accepted": 7, + "n": 10 + }, + "elapsed_sec": "{{T}}", + "engine": "b9999-test", + "kv": "f16", + "mode": "quick", + "predicted_ms": 160, + "predicted_n": 8, + "predicted_per_second": 50, + "prompt_ms": 69.5, + "prompt_n": 139, + "prompt_per_second": 2000, + "protocol": 2 + } + } + } + }, + { + "name": "corps illisible : défauts", + "engine": { + "n_ctx": 16384 + }, + "method": "POST", + "path": "/api/bench", + "body": "{\"prompt\":\"beaucoup\"}", + "status": 202, + "out": { + "id": "{{ID}}", + "ok": true + }, + "wait": true, + "job": { + "elapsed_sec": "{{T}}", + "id": "{{ID}}", + "mode": "quick", + "ok": true, + "phase": "ligne courte (2000 + 300 jetons)", + "result": { + "draft": { + "accepted": 7, + "n": 10 + }, + "elapsed_sec": "{{T}}", + "engine": "b9999-test", + "kv": "f16", + "mode": "quick", + "predicted_ms": 6000, + "predicted_n": 300, + "predicted_per_second": 50, + "prompt_ms": 1018, + "prompt_n": 2036, + "prompt_per_second": 2000, + "protocol": 2 + }, + "running": false, + "saved": true, + "step": 2, + "steps": 2 + }, + "calls": [ + { + "method": "GET", + "path": "/health" + }, + { + "method": "GET", + "path": "/health" + }, + { + "method": "GET", + "path": "/slots" + }, + { + "method": "GET", + "path": "/props" + }, + { + "body": { + "content": "<163664 octets b7452851885a>" + }, + "method": "POST", + "path": "/tokenize" + }, + { + "body": { + "content": "<3561 octets f977dc7d4950>" + }, + "method": "POST", + "path": "/tokenize" + }, + { + "body": { + "cache_prompt": false, + "chat_template_kwargs": { + "enable_thinking": false + }, + "max_tokens": 32, + "messages": [ + { + "content": "<4110 octets c386aa380c09>", + "role": "user" + } + ], + "min_p": 0.05, + "model": "loki", + "seed": 1234, + "stream": false, + "temperature": 0.7, + "top_k": 40 + }, + "method": "POST", + "path": "/v1/chat/completions" + }, + { + "body": { + "cache_prompt": false, + "chat_template_kwargs": { + "enable_thinking": false + }, + "max_tokens": 300, + "messages": [ + { + "content": "<8129 octets ae9cc838495d>", + "role": "user" + } + ], + "min_p": 0.05, + "model": "loki", + "seed": 1234, + "stream": false, + "temperature": 0.7, + "top_k": 40 + }, + "method": "POST", + "path": "/v1/chat/completions" + } + ], + "last": { + "at": "{{T}}", + "fingerprint": "{{FP}}", + "model": "qwen.gguf", + "result": { + "draft": { + "accepted": 7, + "n": 10 + }, + "elapsed_sec": "{{T}}", + "engine": "b9999-test", + "kv": "f16", + "mode": "quick", + "predicted_ms": 6000, + "predicted_n": 300, + "predicted_per_second": 50, + "prompt_ms": 1018, + "prompt_n": 2036, + "prompt_per_second": 2000, + "protocol": 2 + } + }, + "presets": { + "qwen": { + "at": "{{T}}", + "fingerprint": "{{FP}}", + "model": "qwen.gguf", + "result": { + "draft": { + "accepted": 7, + "n": 10 + }, + "elapsed_sec": "{{T}}", + "engine": "b9999-test", + "kv": "f16", + "mode": "quick", + "predicted_ms": 6000, + "predicted_n": 300, + "predicted_per_second": 50, + "prompt_ms": 1018, + "prompt_n": 2036, + "prompt_per_second": 2000, + "protocol": 2 + } + } + } + }, + { + "name": "le préchauffage échoue", + "engine": { + "n_ctx": 16384, + "fail_at": 1 + }, + "method": "POST", + "path": "/api/bench", + "body": "{}", + "status": 202, + "out": { + "id": "{{ID}}", + "ok": true + }, + "wait": true, + "job": { + "canceled": false, + "elapsed_sec": "{{T}}", + "error": "préchauffage : POST /v1/chat/completions : 500 Internal Server Error {\"error\":\"boom\"}", + "id": "{{ID}}", + "mode": "quick", + "ok": true, + "phase": "préchauffage", + "running": false, + "step": 1, + "steps": 2 + }, + "calls": [ + { + "method": "GET", + "path": "/health" + }, + { + "method": "GET", + "path": "/health" + }, + { + "method": "GET", + "path": "/slots" + }, + { + "method": "GET", + "path": "/props" + }, + { + "body": { + "content": "<163664 octets b7452851885a>" + }, + "method": "POST", + "path": "/tokenize" + }, + { + "body": { + "content": "<3561 octets f977dc7d4950>" + }, + "method": "POST", + "path": "/tokenize" + }, + { + "body": { + "cache_prompt": false, + "chat_template_kwargs": { + "enable_thinking": false + }, + "max_tokens": 32, + "messages": [ + { + "content": "<4110 octets c386aa380c09>", + "role": "user" + } + ], + "min_p": 0.05, + "model": "loki", + "seed": 1234, + "stream": false, + "temperature": 0.7, + "top_k": 40 + }, + "method": "POST", + "path": "/v1/chat/completions" + } + ], + "last": { + "at": "{{T}}", + "fingerprint": "{{FP}}", + "model": "qwen.gguf", + "result": { + "draft": { + "accepted": 7, + "n": 10 + }, + "elapsed_sec": "{{T}}", + "engine": "b9999-test", + "kv": "f16", + "mode": "quick", + "predicted_ms": 6000, + "predicted_n": 300, + "predicted_per_second": 50, + "prompt_ms": 1018, + "prompt_n": 2036, + "prompt_per_second": 2000, + "protocol": 2 + } + }, + "presets": { + "qwen": { + "at": "{{T}}", + "fingerprint": "{{FP}}", + "model": "qwen.gguf", + "result": { + "draft": { + "accepted": 7, + "n": 10 + }, + "elapsed_sec": "{{T}}", + "engine": "b9999-test", + "kv": "f16", + "mode": "quick", + "predicted_ms": 6000, + "predicted_n": 300, + "predicted_per_second": 50, + "prompt_ms": 1018, + "prompt_n": 2036, + "prompt_per_second": 2000, + "protocol": 2 + } + } + } + }, + { + "name": "la ligne courte échoue", + "engine": { + "n_ctx": 16384, + "fail_at": 2 + }, + "method": "POST", + "path": "/api/bench", + "body": "{}", + "status": 202, + "out": { + "id": "{{ID}}", + "ok": true + }, + "wait": true, + "job": { + "canceled": false, + "elapsed_sec": "{{T}}", + "error": "ligne courte : POST /v1/chat/completions : 500 Internal Server Error {\"error\":\"boom\"}", + "id": "{{ID}}", + "mode": "quick", + "ok": true, + "phase": "ligne courte (2000 + 300 jetons)", + "running": false, + "step": 2, + "steps": 2 + }, + "calls": [ + { + "method": "GET", + "path": "/health" + }, + { + "method": "GET", + "path": "/health" + }, + { + "method": "GET", + "path": "/slots" + }, + { + "method": "GET", + "path": "/props" + }, + { + "body": { + "content": "<163664 octets b7452851885a>" + }, + "method": "POST", + "path": "/tokenize" + }, + { + "body": { + "content": "<3561 octets f977dc7d4950>" + }, + "method": "POST", + "path": "/tokenize" + }, + { + "body": { + "cache_prompt": false, + "chat_template_kwargs": { + "enable_thinking": false + }, + "max_tokens": 32, + "messages": [ + { + "content": "<4110 octets c386aa380c09>", + "role": "user" + } + ], + "min_p": 0.05, + "model": "loki", + "seed": 1234, + "stream": false, + "temperature": 0.7, + "top_k": 40 + }, + "method": "POST", + "path": "/v1/chat/completions" + }, + { + "body": { + "cache_prompt": false, + "chat_template_kwargs": { + "enable_thinking": false + }, + "max_tokens": 300, + "messages": [ + { + "content": "<8129 octets ae9cc838495d>", + "role": "user" + } + ], + "min_p": 0.05, + "model": "loki", + "seed": 1234, + "stream": false, + "temperature": 0.7, + "top_k": 40 + }, + "method": "POST", + "path": "/v1/chat/completions" + } + ], + "last": { + "at": "{{T}}", + "fingerprint": "{{FP}}", + "model": "qwen.gguf", + "result": { + "draft": { + "accepted": 7, + "n": 10 + }, + "elapsed_sec": "{{T}}", + "engine": "b9999-test", + "kv": "f16", + "mode": "quick", + "predicted_ms": 6000, + "predicted_n": 300, + "predicted_per_second": 50, + "prompt_ms": 1018, + "prompt_n": 2036, + "prompt_per_second": 2000, + "protocol": 2 + } + }, + "presets": { + "qwen": { + "at": "{{T}}", + "fingerprint": "{{FP}}", + "model": "qwen.gguf", + "result": { + "draft": { + "accepted": 7, + "n": 10 + }, + "elapsed_sec": "{{T}}", + "engine": "b9999-test", + "kv": "f16", + "mode": "quick", + "predicted_ms": 6000, + "predicted_n": 300, + "predicted_per_second": 50, + "prompt_ms": 1018, + "prompt_n": 2036, + "prompt_per_second": 2000, + "protocol": 2 + } + } + } + }, + { + "name": "le prefill à froid échoue", + "engine": { + "n_ctx": 16384, + "fail_at": 3 + }, + "method": "POST", + "path": "/api/bench", + "body": "{\"mode\":\"full\"}", + "status": 202, + "out": { + "id": "{{ID}}", + "ok": true + }, + "wait": true, + "job": { + "canceled": false, + "elapsed_sec": "{{T}}", + "error": "prefill à froid : POST /v1/chat/completions : 500 Internal Server Error {\"error\":\"boom\"}", + "id": "{{ID}}", + "mode": "full", + "ok": true, + "phase": "prefill à froid (7648 jetons)", + "running": false, + "step": 3, + "steps": 6 + }, + "calls": [ + { + "method": "GET", + "path": "/health" + }, + { + "method": "GET", + "path": "/health" + }, + { + "method": "GET", + "path": "/slots" + }, + { + "method": "GET", + "path": "/props" + }, + { + "body": { + "content": "<163664 octets b7452851885a>" + }, + "method": "POST", + "path": "/tokenize" + }, + { + "body": { + "content": "<3561 octets f977dc7d4950>" + }, + "method": "POST", + "path": "/tokenize" + }, + { + "body": { + "cache_prompt": false, + "chat_template_kwargs": { + "enable_thinking": false + }, + "max_tokens": 32, + "messages": [ + { + "content": "<4110 octets c386aa380c09>", + "role": "user" + } + ], + "min_p": 0.05, + "model": "loki", + "seed": 1234, + "stream": false, + "temperature": 0.7, + "top_k": 40 + }, + "method": "POST", + "path": "/v1/chat/completions" + }, + { + "body": { + "cache_prompt": false, + "chat_template_kwargs": { + "enable_thinking": false + }, + "max_tokens": 300, + "messages": [ + { + "content": "<8129 octets ae9cc838495d>", + "role": "user" + } + ], + "min_p": 0.05, + "model": "loki", + "seed": 1234, + "stream": false, + "temperature": 0.7, + "top_k": 40 + }, + "method": "POST", + "path": "/v1/chat/completions" + }, + { + "body": { + "cache_prompt": false, + "chat_template_kwargs": { + "enable_thinking": false + }, + "max_tokens": 256, + "messages": [ + { + "content": "<30673 octets 41db43c5b144>", + "role": "user" + } + ], + "min_p": 0.05, + "model": "loki", + "seed": 1234, + "stream": false, + "temperature": 0.7, + "top_k": 40 + }, + "method": "POST", + "path": "/v1/chat/completions" + } + ], + "last": { + "at": "{{T}}", + "fingerprint": "{{FP}}", + "model": "qwen.gguf", + "result": { + "draft": { + "accepted": 7, + "n": 10 + }, + "elapsed_sec": "{{T}}", + "engine": "b9999-test", + "kv": "f16", + "mode": "quick", + "predicted_ms": 6000, + "predicted_n": 300, + "predicted_per_second": 50, + "prompt_ms": 1018, + "prompt_n": 2036, + "prompt_per_second": 2000, + "protocol": 2 + } + }, + "presets": { + "qwen": { + "at": "{{T}}", + "fingerprint": "{{FP}}", + "model": "qwen.gguf", + "result": { + "draft": { + "accepted": 7, + "n": 10 + }, + "elapsed_sec": "{{T}}", + "engine": "b9999-test", + "kv": "f16", + "mode": "quick", + "predicted_ms": 6000, + "predicted_n": 300, + "predicted_per_second": 50, + "prompt_ms": 1018, + "prompt_n": 2036, + "prompt_per_second": 2000, + "protocol": 2 + } + } + } + }, + { + "name": "un tour en profondeur échoue", + "engine": { + "n_ctx": 16384, + "fail_at": 5 + }, + "method": "POST", + "path": "/api/bench", + "body": "{\"mode\":\"full\"}", + "status": 202, + "out": { + "id": "{{ID}}", + "ok": true + }, + "wait": true, + "job": { + "canceled": false, + "elapsed_sec": "{{T}}", + "error": "tour 2 en profondeur : POST /v1/chat/completions : 500 Internal Server Error {\"error\":\"boom\"}", + "id": "{{ID}}", + "mode": "full", + "ok": true, + "phase": "tour 2/3 en profondeur", + "running": false, + "step": 5, + "steps": 6 + }, + "calls": [ + { + "method": "GET", + "path": "/health" + }, + { + "method": "GET", + "path": "/health" + }, + { + "method": "GET", + "path": "/slots" + }, + { + "method": "GET", + "path": "/props" + }, + { + "body": { + "content": "<163664 octets b7452851885a>" + }, + "method": "POST", + "path": "/tokenize" + }, + { + "body": { + "content": "<3561 octets f977dc7d4950>" + }, + "method": "POST", + "path": "/tokenize" + }, + { + "body": { + "cache_prompt": false, + "chat_template_kwargs": { + "enable_thinking": false + }, + "max_tokens": 32, + "messages": [ + { + "content": "<4110 octets c386aa380c09>", + "role": "user" + } + ], + "min_p": 0.05, + "model": "loki", + "seed": 1234, + "stream": false, + "temperature": 0.7, + "top_k": 40 + }, + "method": "POST", + "path": "/v1/chat/completions" + }, + { + "body": { + "cache_prompt": false, + "chat_template_kwargs": { + "enable_thinking": false + }, + "max_tokens": 300, + "messages": [ + { + "content": "<8129 octets ae9cc838495d>", + "role": "user" + } + ], + "min_p": 0.05, + "model": "loki", + "seed": 1234, + "stream": false, + "temperature": 0.7, + "top_k": 40 + }, + "method": "POST", + "path": "/v1/chat/completions" + }, + { + "body": { + "cache_prompt": false, + "chat_template_kwargs": { + "enable_thinking": false + }, + "max_tokens": 256, + "messages": [ + { + "content": "<30673 octets 41db43c5b144>", + "role": "user" + } + ], + "min_p": 0.05, + "model": "loki", + "seed": 1234, + "stream": false, + "temperature": 0.7, + "top_k": 40 + }, + "method": "POST", + "path": "/v1/chat/completions" + }, + { + "body": { + "cache_prompt": true, + "chat_template_kwargs": { + "enable_thinking": false + }, + "max_tokens": 256, + "messages": [ + { + "content": "<30673 octets 41db43c5b144>", + "role": "user" + }, + { + "content": "réponse", + "role": "assistant" + }, + { + "content": "<8264 octets eb0a24749ba3>", + "role": "user" + } + ], + "min_p": 0.05, + "model": "loki", + "seed": 1234, + "stream": false, + "temperature": 0.7, + "top_k": 40 + }, + "method": "POST", + "path": "/v1/chat/completions" + }, + { + "body": { + "cache_prompt": true, + "chat_template_kwargs": { + "enable_thinking": false + }, + "max_tokens": 256, + "messages": [ + { + "content": "<30673 octets 41db43c5b144>", + "role": "user" + }, + { + "content": "réponse", + "role": "assistant" + }, + { + "content": "<8264 octets eb0a24749ba3>", + "role": "user" + }, + { + "content": "réponse", + "role": "assistant" + }, + { + "content": "<8263 octets 273e9def4ac3>", + "role": "user" + } + ], + "min_p": 0.05, + "model": "loki", + "seed": 1234, + "stream": false, + "temperature": 0.7, + "top_k": 40 + }, + "method": "POST", + "path": "/v1/chat/completions" + } + ], + "last": { + "at": "{{T}}", + "fingerprint": "{{FP}}", + "model": "qwen.gguf", + "result": { + "draft": { + "accepted": 7, + "n": 10 + }, + "elapsed_sec": "{{T}}", + "engine": "b9999-test", + "kv": "f16", + "mode": "quick", + "predicted_ms": 6000, + "predicted_n": 300, + "predicted_per_second": 50, + "prompt_ms": 1018, + "prompt_n": 2036, + "prompt_per_second": 2000, + "protocol": 2 + } + }, + "presets": { + "qwen": { + "at": "{{T}}", + "fingerprint": "{{FP}}", + "model": "qwen.gguf", + "result": { + "draft": { + "accepted": 7, + "n": 10 + }, + "elapsed_sec": "{{T}}", + "engine": "b9999-test", + "kv": "f16", + "mode": "quick", + "predicted_ms": 6000, + "predicted_n": 300, + "predicted_per_second": 50, + "prompt_ms": 1018, + "prompt_n": 2036, + "prompt_per_second": 2000, + "protocol": 2 + } + } + } + }, + { + "name": "sans timings", + "engine": { + "n_ctx": 16384, + "no_timing": true + }, + "method": "POST", + "path": "/api/bench", + "body": "{}", + "status": 202, + "out": { + "id": "{{ID}}", + "ok": true + }, + "wait": true, + "job": { + "canceled": false, + "elapsed_sec": "{{T}}", + "error": "préchauffage : le moteur n'a pas renvoyé de mesures (timings) : rien à enregistrer", + "id": "{{ID}}", + "mode": "quick", + "ok": true, + "phase": "préchauffage", + "running": false, + "step": 1, + "steps": 2 + }, + "calls": [ + { + "method": "GET", + "path": "/health" + }, + { + "method": "GET", + "path": "/health" + }, + { + "method": "GET", + "path": "/slots" + }, + { + "method": "GET", + "path": "/props" + }, + { + "body": { + "content": "<163664 octets b7452851885a>" + }, + "method": "POST", + "path": "/tokenize" + }, + { + "body": { + "content": "<3561 octets f977dc7d4950>" + }, + "method": "POST", + "path": "/tokenize" + }, + { + "body": { + "cache_prompt": false, + "chat_template_kwargs": { + "enable_thinking": false + }, + "max_tokens": 32, + "messages": [ + { + "content": "<4110 octets c386aa380c09>", + "role": "user" + } + ], + "min_p": 0.05, + "model": "loki", + "seed": 1234, + "stream": false, + "temperature": 0.7, + "top_k": 40 + }, + "method": "POST", + "path": "/v1/chat/completions" + } + ], + "last": { + "at": "{{T}}", + "fingerprint": "{{FP}}", + "model": "qwen.gguf", + "result": { + "draft": { + "accepted": 7, + "n": 10 + }, + "elapsed_sec": "{{T}}", + "engine": "b9999-test", + "kv": "f16", + "mode": "quick", + "predicted_ms": 6000, + "predicted_n": 300, + "predicted_per_second": 50, + "prompt_ms": 1018, + "prompt_n": 2036, + "prompt_per_second": 2000, + "protocol": 2 + } + }, + "presets": { + "qwen": { + "at": "{{T}}", + "fingerprint": "{{FP}}", + "model": "qwen.gguf", + "result": { + "draft": { + "accepted": 7, + "n": 10 + }, + "elapsed_sec": "{{T}}", + "engine": "b9999-test", + "kv": "f16", + "mode": "quick", + "predicted_ms": 6000, + "predicted_n": 300, + "predicted_per_second": 50, + "prompt_ms": 1018, + "prompt_n": 2036, + "prompt_per_second": 2000, + "protocol": 2 + } + } + } + }, + { + "name": "niveau de raisonnement refusé puis rejoué", + "config": { + "REASONING": "", + "REASONING_EFFORT": "high" + }, + "engine": { + "n_ctx": 16384, + "reject_effort": "high" + }, + "method": "POST", + "path": "/api/bench", + "body": "{}", + "status": 202, + "out": { + "id": "{{ID}}", + "ok": true + }, + "wait": true, + "job": { + "elapsed_sec": "{{T}}", + "id": "{{ID}}", + "mode": "quick", + "ok": true, + "phase": "ligne courte (2000 + 300 jetons)", + "result": { + "draft": { + "accepted": 7, + "n": 10 + }, + "elapsed_sec": "{{T}}", + "engine": "b9999-test", + "kv": "f16", + "mode": "quick", + "predicted_ms": 6000, + "predicted_n": 300, + "predicted_per_second": 50, + "prompt_ms": 1018, + "prompt_n": 2036, + "prompt_per_second": 2000, + "protocol": 2 + }, + "running": false, + "saved": true, + "step": 2, + "steps": 2 + }, + "calls": [ + { + "method": "GET", + "path": "/health" + }, + { + "method": "GET", + "path": "/health" + }, + { + "method": "GET", + "path": "/slots" + }, + { + "method": "GET", + "path": "/props" + }, + { + "body": { + "content": "<163664 octets b7452851885a>" + }, + "method": "POST", + "path": "/tokenize" + }, + { + "body": { + "content": "<3561 octets f977dc7d4950>" + }, + "method": "POST", + "path": "/tokenize" + }, + { + "body": { + "cache_prompt": false, + "chat_template_kwargs": { + "reasoning_effort": "high" + }, + "max_tokens": 32, + "messages": [ + { + "content": "<4110 octets c386aa380c09>", + "role": "user" + } + ], + "min_p": 0.05, + "model": "loki", + "reasoning_effort": "high", + "seed": 1234, + "stream": false, + "temperature": 0.7, + "top_k": 40 + }, + "method": "POST", + "path": "/v1/chat/completions" + }, + { + "body": { + "cache_prompt": false, + "chat_template_kwargs": { + "reasoning_effort": "xhigh" + }, + "max_tokens": 32, + "messages": [ + { + "content": "<4110 octets c386aa380c09>", + "role": "user" + } + ], + "min_p": 0.05, + "model": "loki", + "reasoning_effort": "xhigh", + "seed": 1234, + "stream": false, + "temperature": 0.7, + "top_k": 40 + }, + "method": "POST", + "path": "/v1/chat/completions" + }, + { + "body": { + "cache_prompt": false, + "chat_template_kwargs": { + "reasoning_effort": "xhigh" + }, + "max_tokens": 300, + "messages": [ + { + "content": "<8129 octets ae9cc838495d>", + "role": "user" + } + ], + "min_p": 0.05, + "model": "loki", + "reasoning_effort": "xhigh", + "seed": 1234, + "stream": false, + "temperature": 0.7, + "top_k": 40 + }, + "method": "POST", + "path": "/v1/chat/completions" + } + ], + "last": { + "at": "{{T}}", + "fingerprint": "{{FP}}", + "model": "qwen.gguf", + "result": { + "draft": { + "accepted": 7, + "n": 10 + }, + "elapsed_sec": "{{T}}", + "engine": "b9999-test", + "kv": "f16", + "mode": "quick", + "predicted_ms": 6000, + "predicted_n": 300, + "predicted_per_second": 50, + "prompt_ms": 1018, + "prompt_n": 2036, + "prompt_per_second": 2000, + "protocol": 2 + } + }, + "presets": { + "qwen": { + "at": "{{T}}", + "fingerprint": "{{FP}}", + "model": "qwen.gguf", + "result": { + "draft": { + "accepted": 7, + "n": 10 + }, + "elapsed_sec": "{{T}}", + "engine": "b9999-test", + "kv": "f16", + "mode": "quick", + "predicted_ms": 6000, + "predicted_n": 300, + "predicted_per_second": 50, + "prompt_ms": 1018, + "prompt_n": 2036, + "prompt_per_second": 2000, + "protocol": 2 + } + } + } + }, + { + "name": "niveau appris : envoyé d'emblée", + "engine": { + "n_ctx": 16384, + "reject_effort": "high" + }, + "method": "POST", + "path": "/api/bench", + "body": "{}", + "status": 202, + "out": { + "id": "{{ID}}", + "ok": true + }, + "wait": true, + "job": { + "elapsed_sec": "{{T}}", + "id": "{{ID}}", + "mode": "quick", + "ok": true, + "phase": "ligne courte (2000 + 300 jetons)", + "result": { + "draft": { + "accepted": 7, + "n": 10 + }, + "elapsed_sec": "{{T}}", + "engine": "b9999-test", + "kv": "f16", + "mode": "quick", + "predicted_ms": 6000, + "predicted_n": 300, + "predicted_per_second": 50, + "prompt_ms": 1018, + "prompt_n": 2036, + "prompt_per_second": 2000, + "protocol": 2 + }, + "running": false, + "saved": true, + "step": 2, + "steps": 2 + }, + "calls": [ + { + "method": "GET", + "path": "/health" + }, + { + "method": "GET", + "path": "/health" + }, + { + "method": "GET", + "path": "/slots" + }, + { + "method": "GET", + "path": "/props" + }, + { + "body": { + "content": "<163664 octets b7452851885a>" + }, + "method": "POST", + "path": "/tokenize" + }, + { + "body": { + "content": "<3561 octets f977dc7d4950>" + }, + "method": "POST", + "path": "/tokenize" + }, + { + "body": { + "cache_prompt": false, + "chat_template_kwargs": { + "reasoning_effort": "xhigh" + }, + "max_tokens": 32, + "messages": [ + { + "content": "<4110 octets c386aa380c09>", + "role": "user" + } + ], + "min_p": 0.05, + "model": "loki", + "reasoning_effort": "xhigh", + "seed": 1234, + "stream": false, + "temperature": 0.7, + "top_k": 40 + }, + "method": "POST", + "path": "/v1/chat/completions" + }, + { + "body": { + "cache_prompt": false, + "chat_template_kwargs": { + "reasoning_effort": "xhigh" + }, + "max_tokens": 300, + "messages": [ + { + "content": "<8129 octets ae9cc838495d>", + "role": "user" + } + ], + "min_p": 0.05, + "model": "loki", + "reasoning_effort": "xhigh", + "seed": 1234, + "stream": false, + "temperature": 0.7, + "top_k": 40 + }, + "method": "POST", + "path": "/v1/chat/completions" + } + ], + "last": { + "at": "{{T}}", + "fingerprint": "{{FP}}", + "model": "qwen.gguf", + "result": { + "draft": { + "accepted": 7, + "n": 10 + }, + "elapsed_sec": "{{T}}", + "engine": "b9999-test", + "kv": "f16", + "mode": "quick", + "predicted_ms": 6000, + "predicted_n": 300, + "predicted_per_second": 50, + "prompt_ms": 1018, + "prompt_n": 2036, + "prompt_per_second": 2000, + "protocol": 2 + } + }, + "presets": { + "qwen": { + "at": "{{T}}", + "fingerprint": "{{FP}}", + "model": "qwen.gguf", + "result": { + "draft": { + "accepted": 7, + "n": 10 + }, + "elapsed_sec": "{{T}}", + "engine": "b9999-test", + "kv": "f16", + "mode": "quick", + "predicted_ms": 6000, + "predicted_n": 300, + "predicted_per_second": 50, + "prompt_ms": 1018, + "prompt_n": 2036, + "prompt_per_second": 2000, + "protocol": 2 + } + } + } + }, + { + "name": "budget d'une requête dépassé : profondeur partielle", + "config": { + "REASONING": "off", + "REASONING_EFFORT": "" + }, + "engine": { + "n_ctx": 16384, + "slow_hot_ms": 400, + "timeout_ms": 150 + }, + "method": "POST", + "path": "/api/bench", + "body": "{\"mode\":\"full\"}", + "status": 202, + "out": { + "id": "{{ID}}", + "ok": true + }, + "wait": true, + "job": { + "elapsed_sec": "{{T}}", + "id": "{{ID}}", + "mode": "full", + "ok": true, + "phase": "tour 1/3 en profondeur", + "result": { + "depth": { + "cold": { + "cached": 0, + "draft": { + "accepted": 7, + "n": 10 + }, + "expected": 0, + "new": 7672, + "predicted_ms": 5120, + "predicted_n": 256, + "predicted_per_second": 50, + "prompt_ms": 3836, + "prompt_per_second": 2000 + }, + "ctx": 16384, + "partial": "tour 1 interrompu après 150ms", + "reuse": -1, + "reuse_note": "le moteur ne dit pas combien de jetons il reprend du cache", + "target": 7648 + }, + "draft": { + "accepted": 7, + "n": 10 + }, + "elapsed_sec": "{{T}}", + "engine": "b9999-test", + "kv": "f16", + "mode": "full", + "predicted_ms": 6000, + "predicted_n": 300, + "predicted_per_second": 50, + "prompt_ms": 1018, + "prompt_n": 2036, + "prompt_per_second": 2000, + "protocol": 2 + }, + "running": false, + "saved": false, + "step": 4, + "steps": 6 + }, + "calls": [ + { + "method": "GET", + "path": "/health" + }, + { + "method": "GET", + "path": "/health" + }, + { + "method": "GET", + "path": "/slots" + }, + { + "method": "GET", + "path": "/props" + }, + { + "body": { + "content": "<163664 octets b7452851885a>" + }, + "method": "POST", + "path": "/tokenize" + }, + { + "body": { + "content": "<3561 octets f977dc7d4950>" + }, + "method": "POST", + "path": "/tokenize" + }, + { + "body": { + "cache_prompt": false, + "chat_template_kwargs": { + "enable_thinking": false + }, + "max_tokens": 32, + "messages": [ + { + "content": "<4110 octets c386aa380c09>", + "role": "user" + } + ], + "min_p": 0.05, + "model": "loki", + "seed": 1234, + "stream": false, + "temperature": 0.7, + "top_k": 40 + }, + "method": "POST", + "path": "/v1/chat/completions" + }, + { + "body": { + "cache_prompt": false, + "chat_template_kwargs": { + "enable_thinking": false + }, + "max_tokens": 300, + "messages": [ + { + "content": "<8129 octets ae9cc838495d>", + "role": "user" + } + ], + "min_p": 0.05, + "model": "loki", + "seed": 1234, + "stream": false, + "temperature": 0.7, + "top_k": 40 + }, + "method": "POST", + "path": "/v1/chat/completions" + }, + { + "body": { + "cache_prompt": false, + "chat_template_kwargs": { + "enable_thinking": false + }, + "max_tokens": 256, + "messages": [ + { + "content": "<30673 octets 41db43c5b144>", + "role": "user" + } + ], + "min_p": 0.05, + "model": "loki", + "seed": 1234, + "stream": false, + "temperature": 0.7, + "top_k": 40 + }, + "method": "POST", + "path": "/v1/chat/completions" + }, + { + "body": { + "cache_prompt": true, + "chat_template_kwargs": { + "enable_thinking": false + }, + "max_tokens": 256, + "messages": [ + { + "content": "<30673 octets 41db43c5b144>", + "role": "user" + }, + { + "content": "réponse", + "role": "assistant" + }, + { + "content": "<8264 octets eb0a24749ba3>", + "role": "user" + } + ], + "min_p": 0.05, + "model": "loki", + "seed": 1234, + "stream": false, + "temperature": 0.7, + "top_k": 40 + }, + "method": "POST", + "path": "/v1/chat/completions" + } + ], + "last": { + "at": "{{T}}", + "fingerprint": "{{FP}}", + "model": "qwen.gguf", + "result": { + "draft": { + "accepted": 7, + "n": 10 + }, + "elapsed_sec": "{{T}}", + "engine": "b9999-test", + "kv": "f16", + "mode": "quick", + "predicted_ms": 6000, + "predicted_n": 300, + "predicted_per_second": 50, + "prompt_ms": 1018, + "prompt_n": 2036, + "prompt_per_second": 2000, + "protocol": 2 + } + }, + "presets": { + "qwen": { + "at": "{{T}}", + "fingerprint": "{{FP}}", + "model": "qwen.gguf", + "result": { + "draft": { + "accepted": 7, + "n": 10 + }, + "elapsed_sec": "{{T}}", + "engine": "b9999-test", + "kv": "f16", + "mode": "quick", + "predicted_ms": 6000, + "predicted_n": 300, + "predicted_per_second": 50, + "prompt_ms": 1018, + "prompt_n": 2036, + "prompt_per_second": 2000, + "protocol": 2 + } + } + } + }, + { + "name": "annulation pendant la profondeur", + "engine": { + "n_ctx": 16384, + "slow_hot_ms": 20000 + }, + "method": "POST", + "path": "/api/bench", + "body": "{\"mode\":\"full\"}", + "status": 202, + "out": { + "id": "{{ID}}", + "ok": true + }, + "again": true, + "again_out": { + "error": "un benchmark tourne déjà", + "ok": false + }, + "cancel_at": "tour 1/", + "job": { + "canceled": true, + "elapsed_sec": "{{T}}", + "error": "benchmark annulé", + "id": "{{ID}}", + "mode": "full", + "ok": true, + "phase": "tour 1/3 en profondeur", + "running": false, + "step": 4, + "steps": 6 + }, + "calls": [ + { + "method": "GET", + "path": "/health" + }, + { + "method": "GET", + "path": "/health" + }, + { + "method": "GET", + "path": "/health" + }, + { + "method": "GET", + "path": "/slots" + }, + { + "method": "GET", + "path": "/props" + }, + { + "body": { + "content": "<163664 octets b7452851885a>" + }, + "method": "POST", + "path": "/tokenize" + }, + { + "body": { + "content": "<3561 octets f977dc7d4950>" + }, + "method": "POST", + "path": "/tokenize" + }, + { + "body": { + "cache_prompt": false, + "chat_template_kwargs": { + "enable_thinking": false + }, + "max_tokens": 32, + "messages": [ + { + "content": "<4110 octets c386aa380c09>", + "role": "user" + } + ], + "min_p": 0.05, + "model": "loki", + "seed": 1234, + "stream": false, + "temperature": 0.7, + "top_k": 40 + }, + "method": "POST", + "path": "/v1/chat/completions" + }, + { + "body": { + "cache_prompt": false, + "chat_template_kwargs": { + "enable_thinking": false + }, + "max_tokens": 300, + "messages": [ + { + "content": "<8129 octets ae9cc838495d>", + "role": "user" + } + ], + "min_p": 0.05, + "model": "loki", + "seed": 1234, + "stream": false, + "temperature": 0.7, + "top_k": 40 + }, + "method": "POST", + "path": "/v1/chat/completions" + }, + { + "body": { + "cache_prompt": false, + "chat_template_kwargs": { + "enable_thinking": false + }, + "max_tokens": 256, + "messages": [ + { + "content": "<30673 octets 41db43c5b144>", + "role": "user" + } + ], + "min_p": 0.05, + "model": "loki", + "seed": 1234, + "stream": false, + "temperature": 0.7, + "top_k": 40 + }, + "method": "POST", + "path": "/v1/chat/completions" + }, + { + "body": { + "cache_prompt": true, + "chat_template_kwargs": { + "enable_thinking": false + }, + "max_tokens": 256, + "messages": [ + { + "content": "<30673 octets 41db43c5b144>", + "role": "user" + }, + { + "content": "réponse", + "role": "assistant" + }, + { + "content": "<8264 octets eb0a24749ba3>", + "role": "user" + } + ], + "min_p": 0.05, + "model": "loki", + "seed": 1234, + "stream": false, + "temperature": 0.7, + "top_k": 40 + }, + "method": "POST", + "path": "/v1/chat/completions" + } + ], + "last": { + "at": "{{T}}", + "fingerprint": "{{FP}}", + "model": "qwen.gguf", + "result": { + "draft": { + "accepted": 7, + "n": 10 + }, + "elapsed_sec": "{{T}}", + "engine": "b9999-test", + "kv": "f16", + "mode": "quick", + "predicted_ms": 6000, + "predicted_n": 300, + "predicted_per_second": 50, + "prompt_ms": 1018, + "prompt_n": 2036, + "prompt_per_second": 2000, + "protocol": 2 + } + }, + "presets": { + "qwen": { + "at": "{{T}}", + "fingerprint": "{{FP}}", + "model": "qwen.gguf", + "result": { + "draft": { + "accepted": 7, + "n": 10 + }, + "elapsed_sec": "{{T}}", + "engine": "b9999-test", + "kv": "f16", + "mode": "quick", + "predicted_ms": 6000, + "predicted_n": 300, + "predicted_per_second": 50, + "prompt_ms": 1018, + "prompt_n": 2036, + "prompt_per_second": 2000, + "protocol": 2 + } + } + } + }, + { + "name": "preset Strata", + "config": { + "ENGINE": "strata" + }, + "engine": { + "n_ctx": 16384 + }, + "method": "POST", + "path": "/api/bench", + "body": "{}", + "status": 202, + "out": { + "id": "{{ID}}", + "ok": true + }, + "wait": true, + "job": { + "canceled": false, + "elapsed_sec": "{{T}}", + "error": "benchmark indisponible : le preset actif tourne sur Strata (le débit s'affiche sous chaque réponse)", + "id": "{{ID}}", + "mode": "quick", + "ok": true, + "phase": "démarrage", + "running": false, + "step": 0, + "steps": 0 + }, + "calls": [ + { + "method": "GET", + "path": "/health" + } + ], + "last": { + "at": "{{T}}", + "fingerprint": "{{FP}}", + "model": "qwen.gguf", + "result": { + "draft": { + "accepted": 7, + "n": 10 + }, + "elapsed_sec": "{{T}}", + "engine": "b9999-test", + "kv": "f16", + "mode": "quick", + "predicted_ms": 6000, + "predicted_n": 300, + "predicted_per_second": 50, + "prompt_ms": 1018, + "prompt_n": 2036, + "prompt_per_second": 2000, + "protocol": 2 + } + }, + "presets": { + "qwen": { + "at": "{{T}}", + "fingerprint": "{{FP}}", + "model": "qwen.gguf", + "result": { + "draft": { + "accepted": 7, + "n": 10 + }, + "elapsed_sec": "{{T}}", + "engine": "b9999-test", + "kv": "f16", + "mode": "quick", + "predicted_ms": 6000, + "predicted_n": 300, + "predicted_per_second": 50, + "prompt_ms": 1018, + "prompt_n": 2036, + "prompt_per_second": 2000, + "protocol": 2 + } + } + } + }, + { + "name": "dernier bench, inchangé", + "config": { + "ENGINE": "" + }, + "method": "GET", + "path": "/api/bench/last", + "status": 200, + "out": { + "at": "{{T}}", + "model": "qwen.gguf", + "ok": true, + "result": { + "draft": { + "accepted": 7, + "n": 10 + }, + "elapsed_sec": "{{T}}", + "engine": "b9999-test", + "kv": "f16", + "mode": "quick", + "predicted_ms": 6000, + "predicted_n": 300, + "predicted_per_second": 50, + "prompt_ms": 1018, + "prompt_n": 2036, + "prompt_per_second": 2000, + "protocol": 2 + } + }, + "calls": [], + "last": { + "at": "{{T}}", + "fingerprint": "{{FP}}", + "model": "qwen.gguf", + "result": { + "draft": { + "accepted": 7, + "n": 10 + }, + "elapsed_sec": "{{T}}", + "engine": "b9999-test", + "kv": "f16", + "mode": "quick", + "predicted_ms": 6000, + "predicted_n": 300, + "predicted_per_second": 50, + "prompt_ms": 1018, + "prompt_n": 2036, + "prompt_per_second": 2000, + "protocol": 2 + } + }, + "presets": { + "qwen": { + "at": "{{T}}", + "fingerprint": "{{FP}}", + "model": "qwen.gguf", + "result": { + "draft": { + "accepted": 7, + "n": 10 + }, + "elapsed_sec": "{{T}}", + "engine": "b9999-test", + "kv": "f16", + "mode": "quick", + "predicted_ms": 6000, + "predicted_n": 300, + "predicted_per_second": 50, + "prompt_ms": 1018, + "prompt_n": 2036, + "prompt_per_second": 2000, + "protocol": 2 + } + } + } + } + ] +} diff --git a/rust/loki-rs/tests/proxy.rs b/rust/loki-rs/tests/proxy.rs index c9989ed..3a639ac 100644 --- a/rust/loki-rs/tests/proxy.rs +++ b/rust/loki-rs/tests/proxy.rs @@ -116,6 +116,7 @@ async fn facade(backend: SocketAddr) -> (SocketAddr, oneshot::Sender<()>) { engineimg: None, strata: None, tasks: None, + bench: None, }; tokio::spawn(loki_rs::proxy::serve(l, cfg, async move { let _ = rx.await;