mirror of
https://github.com/R0m1k3/Loki.git
synced 2026-10-16 19:59:56 +02:00
Lot 42 : outils web de l'agent exécutés par la façade
Les outils web des tours de la façade (web_search, web_images, web_open,
web_read, web_grep) ne passent plus par /tool/exec du Go : src/webtools.rs
porte chat_internet_tools.go, chat_internet.go et chat_web_images.go. Le
cache des pages (10 min, clé MD5 des options comme le Go) est dans la
façade ; client Crawl4AI au corps près, réponses lues comme json.Unmarshal ;
résultats DuckDuckGo tirés du markdown (uddg, entités, extraits) ;
recherche d'images avec le jeton vqd et les cookies gardés par hôte ;
web_grep sur les regex du Go, erreurs au texte de regexp.
Le moteur intégré lit le HTML avec go-readability et html-to-markdown, que
rien en Rust ne reproduit au caractère près : la conversion d'une page et la
lecture des résultats restent au Go, derrière /internal/v1/web/fetch et
/internal/v1/web/search (sans état). web_screenshot reste au Go.
Le traducteur go_regex refuse la répétition d'une répétition et les comptes
au-delà de 1000, et rend littéral un « { » qui n'ouvre pas de répétition,
comme le Go (motifs retry_patterns compris).
Parité : journal de 74 appels tiré du Go (parity_web_tools_test.go →
tests/parity_web/outils.json : résultats, requêtes reçues par les faux
Crawl4AI et DuckDuckGo, pages demandées au moteur intégré), rejoué par
tests/webtools.rs ; un tour de la façade qui ouvre une page (tests/lot6.rs) ;
TestInternalMoteurWebIntegre côté Go.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016tcGfJWefW5kUGoC2wL8cv
This commit is contained in:
19 files changed
+4561
-7
No files matched your search
@@ -0,0 +1,121 @@
|
||||
# Lot 42 — Outils web de l'agent dans la façade
|
||||
|
||||
## Contexte
|
||||
|
||||
La façade tient la conversation par défaut. Les outils web de ses tours
|
||||
partaient pourtant au Go (`/internal/v1/tool/exec`) :
|
||||
|
||||
- `web_search` ;
|
||||
- `web_images` ;
|
||||
- `web_open`, `web_read`, `web_grep`.
|
||||
|
||||
Le cache des pages (10 min) était donc au Go : `web_open` puis `web_read`
|
||||
faisaient chacun un aller-retour.
|
||||
|
||||
## Le lot
|
||||
|
||||
`src/webtools.rs` porte les outils. Sources Go :
|
||||
|
||||
- `chat_internet_tools.go` : les quatre outils de lecture et de recherche ;
|
||||
- `chat_internet.go` : client Crawl4AI, cache, lecture des résultats
|
||||
DuckDuckGo ;
|
||||
- `chat_web_images.go` : recherche d'images.
|
||||
|
||||
Ce qui est porté :
|
||||
|
||||
- **le cache** :
|
||||
- clé : l'URL, suivie de l'empreinte MD5 des options qui changent la page
|
||||
(actions, attente, bandeaux), à l'échappement HTML près du Go ;
|
||||
- `web_read` et `web_grep` lisent l'ouverture la plus récente d'une URL,
|
||||
toutes options confondues ;
|
||||
- 10 minutes ;
|
||||
- **Crawl4AI** :
|
||||
- corps envoyé au Go près (clés triées, échappement HTML) ;
|
||||
- clé d'accès en `Bearer` ;
|
||||
- réponse `{results:[…]}` ou résultat seul, lue comme `json.Unmarshal`
|
||||
(types faux, `null`, casse des clés) ;
|
||||
- markdown en chaîne ou `{raw_markdown, fit_markdown}` ;
|
||||
- erreurs : HTTP (fin du corps), échec du crawl, réponse vide, serveur
|
||||
injoignable (texte de l'erreur du client Go) ;
|
||||
- **les lignes** : `\r\n`, blancs de fin, vides et doublons consécutifs
|
||||
réduits ; plan des titres ;
|
||||
- **la recherche par Crawl4AI** : titres, liens `uddg` décodés deux fois
|
||||
comme le Go, entités HTML, extrait tiré des liens qui suivent, doublons et
|
||||
liens DuckDuckGo écartés, défi anti-bot ;
|
||||
- **les images** :
|
||||
- jeton `vqd` de la page, puis `i.js` ;
|
||||
- mêmes en-têtes ;
|
||||
- cookies gardés par hôte, comme le jar du client Go ;
|
||||
- réponse lue comme `json.Unmarshal` (erreurs de syntaxe et de type au
|
||||
texte du Go) ;
|
||||
- **`web_grep`** :
|
||||
- regex du Go (`(?i)`, classes ASCII) ;
|
||||
- erreurs au texte de `regexp` pour les fautes courantes ;
|
||||
- **le plafond** de 8000 caractères ;
|
||||
- **les arguments** : nombres tronqués comme `int(float64)` du Go.
|
||||
|
||||
Le moteur intégré lit le HTML avec go-readability, html-to-markdown et le
|
||||
parseur HTML5 de x/net : rien en Rust ne les reproduit au caractère près.
|
||||
Ces deux étapes restent au Go, derrière deux appels sans état :
|
||||
|
||||
- `POST /internal/v1/web/fetch` `{url}` → `{markdown}` ou `{error}`
|
||||
(`goFetchMarkdown`) ;
|
||||
- `POST /internal/v1/web/search` `{query, limit}` → `{results}` ou `{error}`
|
||||
(`goSearch`).
|
||||
|
||||
`ChatShared.web` porte les outils ; `GoTurnHost::exec_tool` les exécute
|
||||
avant tout le reste. `web_screenshot` reste au Go (navigateur piloté).
|
||||
|
||||
Le traducteur des regex du Go (`go_regex`) refuse désormais :
|
||||
|
||||
- la répétition d'une répétition (`a**`, `a*??`, `a{2}*`) ;
|
||||
- un compte au-delà de 1000 ou inversé.
|
||||
|
||||
Un `{` qui n'ouvre pas de répétition y devient littéral, comme au Go. Les
|
||||
motifs `retry_patterns` en profitent.
|
||||
|
||||
Côté Go, pour le journal :
|
||||
|
||||
- `webNow` : l'horloge du cache ;
|
||||
- `ddgImagesBase` et `ddgHTMLBase` : les adresses de DuckDuckGo.
|
||||
|
||||
## Écart connu
|
||||
|
||||
reqwest pose `Accept: */*` sur une requête qui n'en a pas ; le client Go
|
||||
n'en pose aucun. Seule la page de recherche d'images est concernée, sans
|
||||
effet sur DuckDuckGo.
|
||||
|
||||
## Parité
|
||||
|
||||
`internal/loki/parity_web_tools_test.go` tire du Go un journal de 74
|
||||
appels (`tests/parity_web/outils.json`). Chaque appel porte :
|
||||
|
||||
- le résultat ;
|
||||
- les requêtes reçues par le faux Crawl4AI (méthode, clé, type, corps) ;
|
||||
- les requêtes reçues par le faux DuckDuckGo (chemin, en-têtes, cookies) ;
|
||||
- les pages et recherches demandées au moteur intégré.
|
||||
|
||||
Il couvre :
|
||||
|
||||
- recherche : limites, anti-bot, aucun résultat, panne ;
|
||||
- lecture : plages, bornes, au-delà, page jamais ouverte ;
|
||||
- grep : contexte fusionné, plafond, sans résultat, classes ASCII, sept
|
||||
regex invalides ;
|
||||
- cache : réouverture, forcée, options, la plus récente, péremption ;
|
||||
- GitHub ;
|
||||
- les réponses de Crawl4AI ;
|
||||
- serveur injoignable, sans serveur, sans clé ;
|
||||
- moteur intégré : ouverture, texte brut, page absente, recherche ;
|
||||
- images : doublons, cookie gardé, jeton absent, pannes, réponses
|
||||
illisibles ou mal typées, `null`.
|
||||
|
||||
`tests/webtools.rs` le rejoue contre le port. Le faux Go sert ce que le
|
||||
moteur intégré a tiré des pages.
|
||||
|
||||
D'autres tests complètent le journal :
|
||||
|
||||
- `tests/lot6.rs` (`outil_web_par_la_facade`) : un tour de la façade ouvre
|
||||
une page sans passer par `/tool/exec` ;
|
||||
- `TestInternalMoteurWebIntegre` : les deux appels internes du Go ;
|
||||
- tests unitaires : MD5, clés du cache, URL, entités, nombres, erreurs de
|
||||
regex, réponses de Crawl4AI.
|
||||
@@ -347,6 +347,9 @@ var (
|
||||
|
||||
const pageCacheTTL = 10 * time.Minute
|
||||
|
||||
// webNow : l'horloge du cache des pages (fixée par le journal de parité).
|
||||
var webNow = time.Now
|
||||
|
||||
type fetchOptions struct {
|
||||
force bool
|
||||
actions []string
|
||||
@@ -371,7 +374,7 @@ func findCached(rawURL string) *cacheEntry {
|
||||
var best *cacheEntry
|
||||
for k, v := range pageCache {
|
||||
if k == u || strings.HasPrefix(k, u+"#") {
|
||||
if time.Since(v.fetchedAt) > pageCacheTTL {
|
||||
if webNow().Sub(v.fetchedAt) > pageCacheTTL {
|
||||
continue
|
||||
}
|
||||
if best == nil || v.fetchedAt.After(best.fetchedAt) {
|
||||
@@ -388,7 +391,7 @@ func getPage(rawURL string, opts fetchOptions) (*cacheEntry, error) {
|
||||
pageCacheMu.Lock()
|
||||
cached := pageCache[key]
|
||||
pageCacheMu.Unlock()
|
||||
if !opts.force && cached != nil && time.Since(cached.fetchedAt) < pageCacheTTL {
|
||||
if !opts.force && cached != nil && webNow().Sub(cached.fetchedAt) < pageCacheTTL {
|
||||
return cached, nil
|
||||
}
|
||||
|
||||
@@ -418,7 +421,7 @@ func getPage(rawURL string, opts fetchOptions) (*cacheEntry, error) {
|
||||
if strings.TrimSpace(md) == "" {
|
||||
return nil, fmt.Errorf("page vide")
|
||||
}
|
||||
entry := &cacheEntry{url: u, lines: normalizeLines(md), fetchedAt: time.Now()}
|
||||
entry := &cacheEntry{url: u, lines: normalizeLines(md), fetchedAt: webNow()}
|
||||
pageCacheMu.Lock()
|
||||
pageCache[key] = entry
|
||||
pageCacheMu.Unlock()
|
||||
@@ -493,6 +496,10 @@ func decodeHTMLEntities(s string) string {
|
||||
})
|
||||
}
|
||||
|
||||
// ddgHTMLBase : le point d'entrée HTML de DuckDuckGo lu par le moteur intégré
|
||||
// (goSearch ; un faux pour les tests).
|
||||
var ddgHTMLBase = "https://html.duckduckgo.com"
|
||||
|
||||
// decodeUddg : DDG enrobe les résultats en //duckduckgo.com/l/?uddg=URL_ENCODÉE
|
||||
func decodeUddg(raw string) string {
|
||||
s := raw
|
||||
|
||||
@@ -46,6 +46,9 @@ type imageResult struct {
|
||||
|
||||
var ddgVqdRe = regexp.MustCompile(`vqd=["']?([0-9-]+)`)
|
||||
|
||||
// ddgImagesBase : DuckDuckGo (un faux pour les tests).
|
||||
var ddgImagesBase = "https://duckduckgo.com"
|
||||
|
||||
func ddgGet(ctx context.Context, target, referer string) ([]byte, error) {
|
||||
req, err := http.NewRequestWithContext(ctx, http.MethodGet, target, nil)
|
||||
if err != nil {
|
||||
@@ -75,7 +78,7 @@ func ddgGet(ctx context.Context, target, referer string) ([]byte, error) {
|
||||
func duckduckgoImages(query string, limit int) ([]imageResult, error) {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), goFetchTimeout)
|
||||
defer cancel()
|
||||
page := "https://duckduckgo.com/?q=" + url.QueryEscape(query) + "&iax=images&ia=images"
|
||||
page := ddgImagesBase + "/?q=" + url.QueryEscape(query) + "&iax=images&ia=images"
|
||||
body, err := ddgGet(ctx, page, "")
|
||||
if err != nil {
|
||||
return nil, err
|
||||
@@ -84,7 +87,7 @@ func duckduckgoImages(query string, limit int) ([]imageResult, error) {
|
||||
if m == nil {
|
||||
return nil, fmt.Errorf("jeton DuckDuckGo introuvable (défi anti-bot ?)")
|
||||
}
|
||||
api := "https://duckduckgo.com/i.js?l=fr-fr&o=json&f=,,,,,&p=1&q=" + url.QueryEscape(query) + "&vqd=" + string(m[1])
|
||||
api := ddgImagesBase + "/i.js?l=fr-fr&o=json&f=,,,,,&p=1&q=" + url.QueryEscape(query) + "&vqd=" + string(m[1])
|
||||
body, err = ddgGet(ctx, api, "https://duckduckgo.com/")
|
||||
if err != nil {
|
||||
return nil, err
|
||||
|
||||
@@ -0,0 +1,542 @@
|
||||
package loki
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"net"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// parity_web_tools_test.go — journal de référence des outils web de l'agent
|
||||
// (lot 42 de la migration Rust) : web_search, web_images, web_open, web_read
|
||||
// et web_grep, sur les deux moteurs.
|
||||
//
|
||||
// Un faux Crawl4AI rend des réponses rangées dans le journal, URL par URL ;
|
||||
// un faux DuckDuckGo sert la recherche d'images. Le moteur intégré (Go) lit
|
||||
// de vraies pages HTML d'un faux site et de faux résultats DuckDuckGo : ce
|
||||
// qu'il en tire (goFetchMarkdown, goSearch) est rangé tel quel, la façade le
|
||||
// demandant au Go. Chaque appel porte son résultat et les requêtes reçues
|
||||
// par les faux services. Rangé dans rust/loki-rs/tests/parity_web/outils.json ;
|
||||
// tests/webtools.rs le rejoue contre le port.
|
||||
//
|
||||
// LOKI_PARITY_UPDATE=1 go test ./internal/loki -run TestPariteOutilsWeb
|
||||
|
||||
const parityWebFile = "../../rust/loki-rs/tests/parity_web/outils.json"
|
||||
|
||||
type parityCrawlResp struct {
|
||||
Status int `json:"status"`
|
||||
Body string `json:"body"`
|
||||
}
|
||||
|
||||
type parityImgResp struct {
|
||||
PageStatus int `json:"page_status"`
|
||||
Page string `json:"page"`
|
||||
Cookie string `json:"cookie,omitempty"`
|
||||
APIStatus int `json:"api_status"`
|
||||
API string `json:"api"`
|
||||
}
|
||||
|
||||
type parityWebReq struct {
|
||||
Path string `json:"path"`
|
||||
Auth string `json:"auth,omitempty"`
|
||||
CType string `json:"ctype,omitempty"`
|
||||
Body string `json:"body,omitempty"`
|
||||
// DuckDuckGo
|
||||
Referer string `json:"referer,omitempty"`
|
||||
Accept string `json:"accept,omitempty"`
|
||||
Lang string `json:"lang,omitempty"`
|
||||
UA string `json:"ua,omitempty"`
|
||||
Cookie string `json:"cookie,omitempty"`
|
||||
}
|
||||
|
||||
type parityWebCall struct {
|
||||
Name string `json:"name"`
|
||||
Setup map[string]string `json:"setup,omitempty"` // WEB_ENGINE, CRAWL4AI_URL, crawl_key ("-" efface)
|
||||
Advance int `json:"advance_s,omitempty"`
|
||||
Tool string `json:"tool"`
|
||||
Args map[string]any `json:"args"`
|
||||
Result string `json:"result"`
|
||||
Crawls []parityWebReq `json:"crawls"`
|
||||
Images []parityWebReq `json:"images"`
|
||||
GoFetch []string `json:"go_fetch"`
|
||||
GoSearch []string `json:"go_search"`
|
||||
}
|
||||
|
||||
type parityGoOut struct {
|
||||
Markdown string `json:"markdown,omitempty"`
|
||||
Results []map[string]string `json:"results,omitempty"`
|
||||
Error string `json:"error,omitempty"`
|
||||
}
|
||||
|
||||
type parityWebJournal struct {
|
||||
Crawl map[string][]parityCrawlResp `json:"crawl"`
|
||||
Images map[string]parityImgResp `json:"images"`
|
||||
GoFetch map[string]parityGoOut `json:"go_fetch"`
|
||||
GoSearch map[string]parityGoOut `json:"go_search"`
|
||||
Calls []parityWebCall `json:"calls"`
|
||||
}
|
||||
|
||||
// Les réponses du faux Crawl4AI, par URL demandée (consommées dans l'ordre,
|
||||
// la dernière reste).
|
||||
func parityCrawlFixtures() map[string][]parityCrawlResp {
|
||||
ddg := func(q string) string {
|
||||
return "https://html.duckduckgo.com/html/?q=" + q
|
||||
}
|
||||
obj := func(v any) string {
|
||||
b, _ := json.Marshal(v)
|
||||
return string(b)
|
||||
}
|
||||
long := strings.Repeat("une ligne assez longue pour remplir la fenêtre de lecture, encore et encore. ", 3)
|
||||
var longLines []string
|
||||
for i := 0; i < 60; i++ {
|
||||
longLines = append(longLines, fmt.Sprintf("%d %s", i, long))
|
||||
}
|
||||
searchMD := strings.Join([]string{
|
||||
"# DuckDuckGo",
|
||||
"",
|
||||
"## [Loki & **Rust**](//duckduckgo.com/l/?uddg=https%3A%2F%2Floki.example%2Frust%3Fa%3D1%26b%3D2&rut=abc)",
|
||||
"",
|
||||
"[loki.example/rust](//duckduckgo.com/l/?uddg=x) [](x) [Un port en **Rust** étape par étape](//duckduckgo.com/l/?uddg=y)",
|
||||
"## [Publicité](https://duckduckgo.com/y.js?ad=1)",
|
||||
"## [Deuxième "résultat"](https://deux.example/page)",
|
||||
"",
|
||||
"",
|
||||
"",
|
||||
"",
|
||||
"",
|
||||
"[trop loin pour le résumé](x)",
|
||||
"## [Loki encore](//duckduckgo.com/l/?uddg=https%3A%2F%2Floki.example%2Frust%3Fa%3D1%26b%3D2)",
|
||||
"## [Troisième](https://trois.example/) ",
|
||||
"[!bang ignoré](x) [vraie description](y)",
|
||||
"## [Quatrième](//duckduckgo.com/l/?uddg=%ZZ)",
|
||||
"## [](https://vide.example/)",
|
||||
}, "\n")
|
||||
docMD := strings.Join([]string{
|
||||
"",
|
||||
"# Documentation ",
|
||||
"",
|
||||
"Intro ligne 1",
|
||||
"Intro ligne 1",
|
||||
"",
|
||||
"",
|
||||
"",
|
||||
"## Installation",
|
||||
"Lancer `loki install`.\t",
|
||||
"Erreur : moteur absent",
|
||||
"### Détails",
|
||||
"ligne a",
|
||||
"ligne b",
|
||||
"Erreur : port pris",
|
||||
"ligne c",
|
||||
"ligne d",
|
||||
"ligne e",
|
||||
"ligne f",
|
||||
"Erreur : disque plein",
|
||||
"####### pas un titre",
|
||||
"## Fin\r",
|
||||
"",
|
||||
}, "\n")
|
||||
return map[string][]parityCrawlResp{
|
||||
ddg("loki+rust"): {{200, obj(map[string]any{"results": []any{map[string]any{"success": true,
|
||||
"markdown": map[string]any{"raw_markdown": searchMD, "fit_markdown": "ignoré"}}}})}},
|
||||
ddg("anti-bot"): {{200, obj(map[string]any{"results": []any{map[string]any{"success": true,
|
||||
"markdown": "<div class=\"anomaly-modal\">"}}})}},
|
||||
ddg("rien"): {{200, obj(map[string]any{"results": []any{map[string]any{"success": true, "markdown": "# Rien\n\npas de résultat"}}})}},
|
||||
ddg("panne"): {{500, "Internal Server Error: " + strings.Repeat("é", 320)}},
|
||||
"https://exemple.test/doc": {
|
||||
{200, obj(map[string]any{"results": []any{map[string]any{"success": true,
|
||||
"markdown": map[string]any{"raw_markdown": "brut", "fit_markdown": docMD}}}})},
|
||||
{200, obj(map[string]any{"results": []any{map[string]any{"success": true,
|
||||
"markdown": "# Version 2\n\nnouveau contenu"}}})},
|
||||
{200, obj(map[string]any{"results": []any{map[string]any{"success": true,
|
||||
"markdown": "# Avec actions\n\nbouton cliqué"}}})},
|
||||
},
|
||||
"https://raw.githubusercontent.com/r0m1k3/loki/HEAD/README.md": {{200, obj(map[string]any{"results": []any{map[string]any{"success": true, "markdown": "# Loki\n\nLe README."}}})}},
|
||||
"https://exemple.test/echec": {{200, obj(map[string]any{"results": []any{map[string]any{"success": false,
|
||||
"error_message": "net::ERR_NAME_NOT_RESOLVED " + strings.Repeat("x", 310)}}})}},
|
||||
"https://exemple.test/echec-muet": {{200, `{"results":[{"success":false}]}`}},
|
||||
"https://exemple.test/vide": {{200, `{"results":[]}`}},
|
||||
"https://exemple.test/tableau": {{200, `[]`}},
|
||||
"https://exemple.test/illisible": {{200, `pas du json`}},
|
||||
"https://exemple.test/seul": {{200, `{"success":true,"markdown":"# Seul\n\nun résultat seul"}`}},
|
||||
"https://exemple.test/mal-type": {{200, `{"results":[{"success":"oui","markdown":"x"}]}`}},
|
||||
"https://exemple.test/blanc": {{200, `{"results":[{"success":true,"markdown":" \n\t\n"}]}`}},
|
||||
"https://exemple.test/objet-faux": {{200, `{"results":[{"success":true,"markdown":{"raw_markdown":5}}]}`}},
|
||||
"https://exemple.test/longue": {{200, obj(map[string]any{"results": []any{map[string]any{"success": true, "markdown": strings.Join(longLines, "\n")}}})}},
|
||||
"https://exemple.test/brut-absent": {{200, `{"results":[{"success":true,"markdown":{"fit_markdown":"# Fit seul\n\nok"}}]}`}},
|
||||
}
|
||||
}
|
||||
|
||||
func parityImageFixtures() map[string]parityImgResp {
|
||||
page := func(vqd string) string {
|
||||
return `<html><script>var x = 1; vqd="` + vqd + `"; </script></html>`
|
||||
}
|
||||
res := func(items ...map[string]any) string {
|
||||
b, _ := json.Marshal(map[string]any{"results": items})
|
||||
return string(b)
|
||||
}
|
||||
it := func(img, title, u string, w, h int) map[string]any {
|
||||
return map[string]any{"image": img, "title": title, "url": u, "width": w, "height": h, "thumbnail": "t"}
|
||||
}
|
||||
return map[string]parityImgResp{
|
||||
"chats": {200, page("4-123-456"), "dcm=7; Path=/; HttpOnly", 200, res(
|
||||
it("https://img.example/1.jpg", "Chat roux", "https://page.example/1", 800, 600),
|
||||
it("https://img.example/1.jpg", "Doublon", "https://page.example/1b", 1, 1),
|
||||
it("", "Sans image", "https://page.example/2", 1, 1),
|
||||
it("data:image/png;base64,xx", "Pas http", "https://page.example/3", 1, 1),
|
||||
it("http://img.example/4.png", "Chat & chien <3", "https://page.example/4", 1024, 768),
|
||||
it("https://img.example/5.jpg", "Cinquième", "https://page.example/5", 10, 20),
|
||||
)},
|
||||
"sans jeton": {200, "<html>rien</html>", "", 200, "{}"},
|
||||
"api en panne": {200, page("1"), "", 500, "oops"},
|
||||
"page en panne": {403, "interdit", "", 200, "{}"},
|
||||
"illisible": {200, page("2"), "", 200, "pas du json"},
|
||||
"apres": {200, page("3"), "", 200, `{"results":[]} x`},
|
||||
"largeur texte": {200, page("4"), "", 200, `{"results":[{"image":"https://i/1","width":"800"}]}`},
|
||||
"vide": {200, page("5"), "", 200, `{"results":[]}`},
|
||||
"null": {200, page("6"), "", 200, `{"results":null,"next":"i.js?x"}`},
|
||||
"resultats faux": {200, page("7"), "", 200, `{"results":"non"}`},
|
||||
"apostrophe": {200, `vqd='77-88'`, "", 200, `{"results":[{"image":"https://i/a","title":"A","url":"https://p/a","width":1,"height":2}]}`},
|
||||
}
|
||||
}
|
||||
|
||||
// Le faux site du moteur intégré.
|
||||
var parityWebPages = map[string]string{
|
||||
"/article.html": `<!doctype html><html><head><title>Guide de Loki</title></head><body>
|
||||
<nav><a href="https://loki.example/">Accueil</a> · <a href="https://loki.example/blog">Blog</a></nav>
|
||||
<article><h1>Guide de Loki</h1>
|
||||
<p>Loki fait tourner un modèle de langage sur votre machine, sans rien envoyer ailleurs. Ce guide explique l'installation, la configuration et les premiers pas avec l'agent.</p>
|
||||
<h2>Installation</h2>
|
||||
<p>Téléchargez le binaire depuis la <a href="https://loki.example/releases">page des versions</a>, puis lancez-le. Le moteur llama.cpp est installé au premier démarrage, précompilé pour votre carte graphique.</p>
|
||||
<h2>Configuration</h2>
|
||||
<p>Les réglages se font dans l'interface : modèle, contexte, outils. Erreur fréquente : un port déjà pris par un autre service. Changez alors le port dans les réglages avancés.</p>
|
||||
<p>Erreur moins fréquente : un disque plein empêche le téléchargement du modèle.</p>
|
||||
</article><footer>© Loki</footer></body></html>`,
|
||||
"/texte.txt": "Ligne brute 1\nLigne brute 2\n",
|
||||
}
|
||||
|
||||
// Les faux résultats DuckDuckGo (page HTML) du moteur intégré.
|
||||
var parityWebDDGHTML = map[string]string{
|
||||
"loki": `<html><body><div class="results">
|
||||
<div class="result results_links"><h2 class="result__title"><a class="result__a" href="//duckduckgo.com/l/?uddg=https%3A%2F%2Floki.example%2F&rut=x">Loki <b>local</b></a></h2>
|
||||
<a class="result__snippet" href="x">Un assistant <b>local</b> et privé.</a></div>
|
||||
<div class="result"><a class="result__a" href="https://duckduckgo.com/ad">Pub</a></div>
|
||||
<div class="result"><a class="result__a" href="https://deux.example/">Deux</a></div>
|
||||
<div class="result"><a class="result__a" href="https://loki.example/">Doublon</a></div>
|
||||
</div></body></html>`,
|
||||
"bot": `<html><script src="/anomaly.js"></script></html>`,
|
||||
}
|
||||
|
||||
func TestPariteOutilsWeb(t *testing.T) {
|
||||
testHome(t)
|
||||
crawlFx := parityCrawlFixtures()
|
||||
imgFx := parityImageFixtures()
|
||||
|
||||
var mu sync.Mutex
|
||||
var crawls, images []parityWebReq
|
||||
served := map[string]int{}
|
||||
crawlSrv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
body, _ := io.ReadAll(r.Body)
|
||||
var in struct {
|
||||
URLs []string `json:"urls"`
|
||||
}
|
||||
_ = json.Unmarshal(body, &in)
|
||||
mu.Lock()
|
||||
crawls = append(crawls, parityWebReq{Path: r.Method + " " + r.URL.Path, Auth: r.Header.Get("Authorization"),
|
||||
CType: r.Header.Get("Content-Type"), Body: string(body)})
|
||||
target := ""
|
||||
if len(in.URLs) > 0 {
|
||||
target = in.URLs[0]
|
||||
}
|
||||
list := crawlFx[target]
|
||||
n := served[target]
|
||||
served[target]++
|
||||
mu.Unlock()
|
||||
if len(list) == 0 {
|
||||
w.WriteHeader(404)
|
||||
_, _ = io.WriteString(w, "inconnu : "+target)
|
||||
return
|
||||
}
|
||||
if n >= len(list) {
|
||||
n = len(list) - 1
|
||||
}
|
||||
w.WriteHeader(list[n].Status)
|
||||
_, _ = io.WriteString(w, list[n].Body)
|
||||
}))
|
||||
defer crawlSrv.Close()
|
||||
imgSrv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
mu.Lock()
|
||||
images = append(images, parityWebReq{Path: r.URL.RequestURI(), Referer: r.Header.Get("Referer"),
|
||||
Accept: r.Header.Get("Accept"), Lang: r.Header.Get("Accept-Language"),
|
||||
UA: r.Header.Get("User-Agent"), Cookie: r.Header.Get("Cookie")})
|
||||
mu.Unlock()
|
||||
fx := imgFx[r.URL.Query().Get("q")]
|
||||
if r.URL.Path == "/i.js" {
|
||||
w.WriteHeader(fx.APIStatus)
|
||||
_, _ = io.WriteString(w, fx.API)
|
||||
return
|
||||
}
|
||||
if fx.Cookie != "" {
|
||||
w.Header().Set("Set-Cookie", fx.Cookie)
|
||||
}
|
||||
w.WriteHeader(fx.PageStatus)
|
||||
_, _ = io.WriteString(w, fx.Page)
|
||||
}))
|
||||
defer imgSrv.Close()
|
||||
var pagesHit, searchHit []string
|
||||
pagesSrv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
mu.Lock()
|
||||
pagesHit = append(pagesHit, r.URL.Path)
|
||||
mu.Unlock()
|
||||
p, ok := parityWebPages[r.URL.Path]
|
||||
if !ok {
|
||||
http.NotFound(w, r)
|
||||
return
|
||||
}
|
||||
if strings.HasSuffix(r.URL.Path, ".txt") {
|
||||
w.Header().Set("Content-Type", "text/plain; charset=utf-8")
|
||||
} else {
|
||||
w.Header().Set("Content-Type", "text/html; charset=utf-8")
|
||||
}
|
||||
_, _ = io.WriteString(w, p)
|
||||
}))
|
||||
defer pagesSrv.Close()
|
||||
ddgHTMLSrv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
q := r.URL.Query().Get("q")
|
||||
mu.Lock()
|
||||
searchHit = append(searchHit, q)
|
||||
mu.Unlock()
|
||||
w.Header().Set("Content-Type", "text/html; charset=utf-8")
|
||||
_, _ = io.WriteString(w, parityWebDDGHTML[q])
|
||||
}))
|
||||
defer ddgHTMLSrv.Close()
|
||||
dead, err := net.Listen("tcp", "127.0.0.1:0")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
deadAddr := dead.Addr().String()
|
||||
dead.Close()
|
||||
|
||||
oldImg, oldHTML, oldNow, oldJar := ddgImagesBase, ddgHTMLBase, webNow, goHTTPClient.Jar
|
||||
ddgImagesBase, ddgHTMLBase = imgSrv.URL, ddgHTMLSrv.URL
|
||||
goHTTPClient.Jar = newCookieJar() // les cookies d'un passage précédent ne comptent pas
|
||||
base := time.Now()
|
||||
var skew time.Duration
|
||||
webNow = func() time.Time { return base.Add(skew) }
|
||||
pageCacheMu.Lock()
|
||||
pageCache = map[string]*cacheEntry{}
|
||||
pageCacheMu.Unlock()
|
||||
defer func() {
|
||||
ddgImagesBase, ddgHTMLBase, webNow, goHTTPClient.Jar = oldImg, oldHTML, oldNow, oldJar
|
||||
pageCacheMu.Lock()
|
||||
pageCache = map[string]*cacheEntry{}
|
||||
pageCacheMu.Unlock()
|
||||
}()
|
||||
|
||||
mask := func(s string) string {
|
||||
s = strings.ReplaceAll(s, crawlSrv.URL, "{{CRAWL}}")
|
||||
s = strings.ReplaceAll(s, imgSrv.URL, "{{DDG}}")
|
||||
s = strings.ReplaceAll(s, pagesSrv.URL, "{{PAGES}}")
|
||||
s = strings.ReplaceAll(s, "http://"+deadAddr, "{{DEAD}}")
|
||||
return strings.ReplaceAll(s, deadAddr, "{{DEADADDR}}")
|
||||
}
|
||||
unmask := func(s string) string {
|
||||
s = strings.ReplaceAll(s, "{{CRAWL}}", crawlSrv.URL)
|
||||
s = strings.ReplaceAll(s, "{{PAGES}}", pagesSrv.URL)
|
||||
return strings.ReplaceAll(s, "{{DEAD}}", "http://"+deadAddr)
|
||||
}
|
||||
maskReqs := func(rs []parityWebReq) []parityWebReq {
|
||||
out := []parityWebReq{}
|
||||
for _, r := range rs {
|
||||
r.Path, r.Body, r.Referer = mask(r.Path), mask(r.Body), mask(r.Referer)
|
||||
out = append(out, r)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// Ce que le moteur intégré tire des pages et des résultats, rangé pour la
|
||||
// façade (qui le demande au Go).
|
||||
goFetch := map[string]parityGoOut{}
|
||||
for _, p := range []string{"/article.html", "/texte.txt", "/absente"} {
|
||||
md, err := goFetchMarkdown(pagesSrv.URL + p)
|
||||
o := parityGoOut{Markdown: mask(md)}
|
||||
if err != nil {
|
||||
o = parityGoOut{Error: mask(err.Error())}
|
||||
}
|
||||
goFetch["{{PAGES}}"+p] = o
|
||||
}
|
||||
goSearchOut := map[string]parityGoOut{}
|
||||
for _, q := range []struct {
|
||||
q string
|
||||
n int
|
||||
}{{"loki", 8}, {"loki", 1}, {"bot", 8}} {
|
||||
res, err := goSearch(q.q, q.n)
|
||||
o := parityGoOut{Results: []map[string]string{}}
|
||||
for _, r := range res {
|
||||
o.Results = append(o.Results, map[string]string{"title": r.Title, "url": r.URL, "snippet": r.Snippet})
|
||||
}
|
||||
if err != nil {
|
||||
o = parityGoOut{Error: err.Error()}
|
||||
}
|
||||
goSearchOut[fmt.Sprintf("%s|%d", q.q, q.n)] = o
|
||||
}
|
||||
mu.Lock()
|
||||
pagesHit, searchHit = nil, nil
|
||||
mu.Unlock()
|
||||
|
||||
crawlSetup := map[string]string{"WEB_ENGINE": "crawl4ai", "CRAWL4AI_URL": "{{CRAWL}}", "crawl_key": "sesame"}
|
||||
var calls []parityWebCall
|
||||
run := func(c parityWebCall) {
|
||||
for k, v := range c.Setup {
|
||||
switch k {
|
||||
case "crawl_key":
|
||||
if v == "-" {
|
||||
v = ""
|
||||
}
|
||||
if err := putStr(bkState, "crawl_key", v); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
default:
|
||||
if v == "-" {
|
||||
v = ""
|
||||
}
|
||||
if err := SetConfigKey(k, unmask(v)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
}
|
||||
skew += time.Duration(c.Advance) * time.Second
|
||||
args := map[string]any{}
|
||||
raw, _ := json.Marshal(c.Args)
|
||||
_ = json.Unmarshal([]byte(unmask(string(raw))), &args)
|
||||
res := execTool(t.Context(), c.Tool, args, toolCallLabel(c.Tool, args), Caps{Agent: true})
|
||||
mu.Lock()
|
||||
c.Result = mask(res.Result)
|
||||
c.Crawls, c.Images = maskReqs(crawls), maskReqs(images)
|
||||
c.GoFetch, c.GoSearch = []string{}, []string{}
|
||||
for _, p := range pagesHit {
|
||||
c.GoFetch = append(c.GoFetch, "{{PAGES}}"+p)
|
||||
}
|
||||
c.GoSearch = append(c.GoSearch, searchHit...)
|
||||
crawls, images, pagesHit, searchHit = nil, nil, nil, nil
|
||||
mu.Unlock()
|
||||
if c.Args == nil {
|
||||
c.Args = map[string]any{}
|
||||
}
|
||||
calls = append(calls, c)
|
||||
}
|
||||
a := func(kv ...any) map[string]any {
|
||||
m := map[string]any{}
|
||||
for i := 0; i+1 < len(kv); i += 2 {
|
||||
m[kv[i].(string)] = kv[i+1]
|
||||
}
|
||||
return m
|
||||
}
|
||||
|
||||
// Crawl4AI.
|
||||
run(parityWebCall{Name: "recherche", Setup: crawlSetup, Tool: "web_search", Args: a("query", "loki rust")})
|
||||
run(parityWebCall{Name: "recherche limitée", Tool: "web_search", Args: a("query", "loki rust", "limit", 1.0)})
|
||||
run(parityWebCall{Name: "recherche, limite hors bornes", Tool: "web_search", Args: a("query", "loki rust", "limit", 99.0)})
|
||||
run(parityWebCall{Name: "recherche, limite nulle", Tool: "web_search", Args: a("query", "loki rust", "limit", -3.0)})
|
||||
run(parityWebCall{Name: "recherche, limite texte", Tool: "web_search", Args: a("query", "loki rust", "limit", "2")})
|
||||
run(parityWebCall{Name: "défi anti-bot", Tool: "web_search", Args: a("query", "anti-bot")})
|
||||
run(parityWebCall{Name: "aucun résultat", Tool: "web_search", Args: a("query", "rien")})
|
||||
run(parityWebCall{Name: "Crawl4AI en panne", Tool: "web_search", Args: a("query", "panne")})
|
||||
run(parityWebCall{Name: "ouverture", Tool: "web_open", Args: a("url", "https://exemple.test/doc")})
|
||||
run(parityWebCall{Name: "lecture", Tool: "web_read", Args: a("url", "https://exemple.test/doc")})
|
||||
run(parityWebCall{Name: "lecture d'une plage", Tool: "web_read", Args: a("url", "https://exemple.test/doc", "offset", 5.0, "limit", 3.0)})
|
||||
run(parityWebCall{Name: "lecture au-delà", Tool: "web_read", Args: a("url", "https://exemple.test/doc", "offset", 400.0)})
|
||||
run(parityWebCall{Name: "lecture, bornes", Tool: "web_read", Args: a("url", "https://exemple.test/doc", "offset", -2.0, "limit", 0.0)})
|
||||
run(parityWebCall{Name: "grep", Tool: "web_grep", Args: a("url", "https://exemple.test/doc", "pattern", "erreur")})
|
||||
run(parityWebCall{Name: "grep sans contexte", Tool: "web_grep", Args: a("url", "https://exemple.test/doc", "pattern", "^ligne [ae]$", "context", 0.0)})
|
||||
run(parityWebCall{Name: "grep plafonné", Tool: "web_grep", Args: a("url", "https://exemple.test/doc", "pattern", "ligne", "max_matches", 2.0, "context", 1.0)})
|
||||
run(parityWebCall{Name: "grep, contexte large", Tool: "web_grep", Args: a("url", "https://exemple.test/doc", "pattern", "Détails", "context", 50.0)})
|
||||
run(parityWebCall{Name: "grep sans résultat", Tool: "web_grep", Args: a("url", "https://exemple.test/doc", "pattern", "introuvable")})
|
||||
run(parityWebCall{Name: "grep, ascii", Tool: "web_grep", Args: a("url", "https://exemple.test/doc", "pattern", `\bD\w+ils`)})
|
||||
for i, p := range []string{"(", "a)", "[abc", "*x", "a**", `\q`, `a\`} {
|
||||
run(parityWebCall{Name: fmt.Sprintf("regex invalide %d", i+1), Tool: "web_grep", Args: a("url", "https://exemple.test/doc", "pattern", p)})
|
||||
}
|
||||
run(parityWebCall{Name: "page jamais ouverte", Tool: "web_read", Args: a("url", "https://exemple.test/autre")})
|
||||
run(parityWebCall{Name: "grep, page jamais ouverte", Tool: "web_grep", Args: a("url", "https://exemple.test/autre", "pattern", "x")})
|
||||
run(parityWebCall{Name: "réouverture : cache", Tool: "web_open", Args: a("url", "https://exemple.test/doc")})
|
||||
run(parityWebCall{Name: "réouverture forcée", Tool: "web_open", Args: a("url", "https://exemple.test/doc", "refresh", true)})
|
||||
run(parityWebCall{Name: "ouverture avec actions", Advance: 5, Tool: "web_open", Args: a("url", "https://exemple.test/doc",
|
||||
"actions", []any{"document.querySelector('#plus').click();", 3.0, "x && y < z"}, "wait_for", "css:#fin", "dismiss_popups", false)})
|
||||
run(parityWebCall{Name: "lecture : la plus récente", Tool: "web_read", Args: a("url", "https://exemple.test/doc")})
|
||||
run(parityWebCall{Name: "attente seule", Tool: "web_open", Args: a("url", "https://exemple.test/seul", "wait_for", "js:() => true")})
|
||||
run(parityWebCall{Name: "sans fermer les bandeaux", Tool: "web_open", Args: a("url", "https://exemple.test/brut-absent", "dismiss_popups", false)})
|
||||
run(parityWebCall{Name: "cache périmé", Advance: 601, Tool: "web_read", Args: a("url", "https://exemple.test/doc")})
|
||||
run(parityWebCall{Name: "ouverture après péremption", Tool: "web_open", Args: a("url", "https://exemple.test/doc")})
|
||||
run(parityWebCall{Name: "dépôt GitHub", Tool: "web_open", Args: a("url", "https://github.com/r0m1k3/loki/")})
|
||||
run(parityWebCall{Name: "lecture du dépôt", Tool: "web_read", Args: a("url", "https://github.com/r0m1k3/loki")})
|
||||
run(parityWebCall{Name: "GitHub, pas un dépôt", Tool: "web_open", Args: a("url", "https://github.com/r0m1k3")})
|
||||
run(parityWebCall{Name: "crawl en échec", Tool: "web_open", Args: a("url", "https://exemple.test/echec")})
|
||||
run(parityWebCall{Name: "crawl en échec, sans message", Tool: "web_open", Args: a("url", "https://exemple.test/echec-muet")})
|
||||
run(parityWebCall{Name: "réponse vide", Tool: "web_open", Args: a("url", "https://exemple.test/vide")})
|
||||
run(parityWebCall{Name: "réponse tableau", Tool: "web_open", Args: a("url", "https://exemple.test/tableau")})
|
||||
run(parityWebCall{Name: "réponse illisible", Tool: "web_open", Args: a("url", "https://exemple.test/illisible")})
|
||||
run(parityWebCall{Name: "résultat mal typé", Tool: "web_open", Args: a("url", "https://exemple.test/mal-type")})
|
||||
run(parityWebCall{Name: "page blanche", Tool: "web_open", Args: a("url", "https://exemple.test/blanc")})
|
||||
run(parityWebCall{Name: "markdown objet faux", Tool: "web_open", Args: a("url", "https://exemple.test/objet-faux")})
|
||||
run(parityWebCall{Name: "page longue", Tool: "web_open", Args: a("url", "https://exemple.test/longue")})
|
||||
run(parityWebCall{Name: "lecture tronquée", Tool: "web_read", Args: a("url", "https://exemple.test/longue", "limit", 500.0)})
|
||||
run(parityWebCall{Name: "URL inconnue du faux", Tool: "web_open", Args: a("url", "https://exemple.test/inconnue")})
|
||||
run(parityWebCall{Name: "sans clé", Setup: map[string]string{"crawl_key": "-"}, Tool: "web_open", Args: a("url", "https://exemple.test/seul", "refresh", true)})
|
||||
run(parityWebCall{Name: "Crawl4AI injoignable", Setup: map[string]string{"CRAWL4AI_URL": "{{DEAD}}/"}, Tool: "web_open", Args: a("url", "https://exemple.test/x")})
|
||||
run(parityWebCall{Name: "sans serveur Crawl4AI", Setup: map[string]string{"CRAWL4AI_URL": "-"}, Tool: "web_open", Args: a("url", "https://exemple.test/x")})
|
||||
run(parityWebCall{Name: "lecture d'une page vue par Crawl4AI", Tool: "web_read", Args: a("url", "https://exemple.test/seul", "limit", 1.0)})
|
||||
|
||||
// Moteur intégré.
|
||||
goSetup := map[string]string{"WEB_ENGINE": "go"}
|
||||
run(parityWebCall{Name: "intégré : ouverture", Setup: goSetup, Tool: "web_open", Args: a("url", "{{PAGES}}/article.html")})
|
||||
run(parityWebCall{Name: "intégré : grep", Tool: "web_grep", Args: a("url", "{{PAGES}}/article.html", "pattern", "erreur", "context", 0.0)})
|
||||
run(parityWebCall{Name: "intégré : texte brut", Tool: "web_open", Args: a("url", "{{PAGES}}/texte.txt", "actions", []any{"x()"})})
|
||||
run(parityWebCall{Name: "intégré : lecture", Tool: "web_read", Args: a("url", "{{PAGES}}/texte.txt")})
|
||||
run(parityWebCall{Name: "intégré : page absente", Tool: "web_open", Args: a("url", "{{PAGES}}/absente")})
|
||||
run(parityWebCall{Name: "intégré : recherche", Tool: "web_search", Args: a("query", "loki")})
|
||||
run(parityWebCall{Name: "intégré : recherche limitée", Tool: "web_search", Args: a("query", "loki", "limit", 1.0)})
|
||||
run(parityWebCall{Name: "intégré : anti-bot", Tool: "web_search", Args: a("query", "bot")})
|
||||
|
||||
// Images (même client quel que soit le moteur).
|
||||
run(parityWebCall{Name: "images", Tool: "web_images", Args: a("query", "chats")})
|
||||
run(parityWebCall{Name: "images, cookie gardé", Tool: "web_images", Args: a("query", "chats", "limit", 1.0)})
|
||||
for _, q := range []string{"sans jeton", "api en panne", "page en panne", "illisible", "apres", "largeur texte", "vide", "null", "resultats faux", "apostrophe"} {
|
||||
run(parityWebCall{Name: "images : " + q, Tool: "web_images", Args: a("query", q)})
|
||||
}
|
||||
|
||||
j := parityWebJournal{Crawl: crawlFx, Images: imgFx, GoFetch: goFetch, GoSearch: goSearchOut, Calls: calls}
|
||||
var buf bytes.Buffer
|
||||
enc := json.NewEncoder(&buf)
|
||||
enc.SetEscapeHTML(false)
|
||||
enc.SetIndent("", " ")
|
||||
if err := enc.Encode(j); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
got := buf.Bytes()
|
||||
if os.Getenv("LOKI_PARITY_UPDATE") == "1" {
|
||||
if err := os.MkdirAll(filepath.Dir(parityWebFile), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(parityWebFile, got, 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return
|
||||
}
|
||||
want, err := os.ReadFile(parityWebFile)
|
||||
if err != nil {
|
||||
t.Fatalf("journal de référence absent (%v) : LOKI_PARITY_UPDATE=1 pour le créer", err)
|
||||
}
|
||||
if !bytes.Equal(got, want) {
|
||||
t.Errorf("les outils web ne produisent plus le journal rangé dans %s.\n"+
|
||||
"LOKI_PARITY_UPDATE=1 pour le régénérer si le changement est voulu.", parityWebFile)
|
||||
}
|
||||
}
|
||||
@@ -309,7 +309,7 @@ func goSearch(query string, limit int) ([]searchResult, error) {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), goFetchTimeout)
|
||||
defer cancel()
|
||||
|
||||
target := "https://html.duckduckgo.com/html/?q=" + url.QueryEscape(query)
|
||||
target := ddgHTMLBase + "/html/?q=" + url.QueryEscape(query)
|
||||
body, contentType, _, err := goFetch(ctx, target)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
|
||||
@@ -137,6 +137,10 @@ func withInternalAPI(next http.Handler) http.Handler {
|
||||
handleInternalDictateStop(w, r)
|
||||
case "/internal/v1/web/reach/reset":
|
||||
handleInternalReachReset(w, r)
|
||||
case "/internal/v1/web/fetch":
|
||||
handleInternalWebFetch(w, r)
|
||||
case "/internal/v1/web/search":
|
||||
handleInternalWebSearch(w, r)
|
||||
case "/internal/v1/cdp/shutdown":
|
||||
handleInternalCDPShutdown(w, r)
|
||||
case "/internal/v1/service/action":
|
||||
|
||||
@@ -0,0 +1,52 @@
|
||||
package loki
|
||||
|
||||
import "net/http"
|
||||
|
||||
// API interne du moteur web intégré (lot 42 de la migration Rust).
|
||||
//
|
||||
// La façade exécute les outils web de ses tours (web_search, web_images,
|
||||
// web_open, web_read, web_grep) : cache des pages, client Crawl4AI, lecture
|
||||
// des résultats, mise en forme. Le moteur intégré lit le HTML avec
|
||||
// go-readability, html-to-markdown et le parseur HTML5 de x/net : rien en
|
||||
// Rust ne les reproduit au caractère près, ils restent donc ici, derrière
|
||||
// deux appels sans état.
|
||||
//
|
||||
// POST /internal/v1/web/fetch {"url":"…"}
|
||||
// → {"markdown":"…"} | {"error":"…"}
|
||||
// POST /internal/v1/web/search {"query":"…","limit":8}
|
||||
// → {"results":[{"title":"…","url":"…","snippet":"…"}]} | {"error":"…"}
|
||||
|
||||
func handleInternalWebFetch(w http.ResponseWriter, r *http.Request) {
|
||||
var in struct {
|
||||
URL string `json:"url"`
|
||||
}
|
||||
if !readInternalJSON(w, r, &in) {
|
||||
return
|
||||
}
|
||||
md, err := goFetchMarkdown(in.URL)
|
||||
if err != nil {
|
||||
sendJSON(w, http.StatusOK, map[string]any{"error": err.Error()})
|
||||
return
|
||||
}
|
||||
sendJSON(w, http.StatusOK, map[string]any{"markdown": md})
|
||||
}
|
||||
|
||||
func handleInternalWebSearch(w http.ResponseWriter, r *http.Request) {
|
||||
var in struct {
|
||||
Query string `json:"query"`
|
||||
Limit int `json:"limit"`
|
||||
}
|
||||
if !readInternalJSON(w, r, &in) {
|
||||
return
|
||||
}
|
||||
results, err := goSearch(in.Query, in.Limit)
|
||||
if err != nil {
|
||||
sendJSON(w, http.StatusOK, map[string]any{"error": err.Error()})
|
||||
return
|
||||
}
|
||||
out := make([]map[string]string, 0, len(results))
|
||||
for _, r := range results {
|
||||
out = append(out, map[string]string{"title": r.Title, "url": r.URL, "snippet": r.Snippet})
|
||||
}
|
||||
sendJSON(w, http.StatusOK, map[string]any{"results": out})
|
||||
}
|
||||
@@ -0,0 +1,58 @@
|
||||
package loki
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// L'API interne du moteur web intégré (lot 42) : la conversion d'une page et
|
||||
// la recherche DuckDuckGo, telles que la façade les demande.
|
||||
func TestInternalMoteurWebIntegre(t *testing.T) {
|
||||
site := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
switch r.URL.Path {
|
||||
case "/page.txt":
|
||||
w.Header().Set("Content-Type", "text/plain")
|
||||
_, _ = io.WriteString(w, "bonjour\n")
|
||||
case "/html/":
|
||||
w.Header().Set("Content-Type", "text/html")
|
||||
_, _ = io.WriteString(w, `<div class="result"><a class="result__a" href="https://r.example/">R</a>`+
|
||||
`<a class="result__snippet">résumé</a></div>`)
|
||||
default:
|
||||
http.NotFound(w, r)
|
||||
}
|
||||
}))
|
||||
defer site.Close()
|
||||
old := ddgHTMLBase
|
||||
ddgHTMLBase = site.URL
|
||||
defer func() { ddgHTMLBase = old }()
|
||||
|
||||
call := func(h http.HandlerFunc, method, body string) (int, map[string]any) {
|
||||
w := httptest.NewRecorder()
|
||||
h(w, httptest.NewRequest(method, "/x", strings.NewReader(body)))
|
||||
var out map[string]any
|
||||
_ = json.Unmarshal(w.Body.Bytes(), &out)
|
||||
return w.Code, out
|
||||
}
|
||||
if code, out := call(handleInternalWebFetch, "POST", `{"url":"`+site.URL+`/page.txt"}`); code != 200 || out["markdown"] != "bonjour\n" {
|
||||
t.Fatalf("fetch : %d %v", code, out)
|
||||
}
|
||||
if code, out := call(handleInternalWebFetch, "POST", `{"url":"`+site.URL+`/absente"}`); code != 200 ||
|
||||
out["error"] != "HTTP 404 sur "+site.URL+"/absente" {
|
||||
t.Fatalf("fetch absente : %d %v", code, out)
|
||||
}
|
||||
code, out := call(handleInternalWebSearch, "POST", `{"query":"q","limit":5}`)
|
||||
res, _ := json.Marshal(out["results"])
|
||||
if code != 200 || string(res) != `[{"snippet":"résumé","title":"R","url":"https://r.example/"}]` {
|
||||
t.Fatalf("search : %d %s", code, res)
|
||||
}
|
||||
if code, _ := call(handleInternalWebSearch, "GET", ""); code != http.StatusMethodNotAllowed {
|
||||
t.Fatalf("GET : %d", code)
|
||||
}
|
||||
if code, _ := call(handleInternalWebFetch, "POST", "pas du json"); code != http.StatusBadRequest {
|
||||
t.Fatalf("corps illisible : %d", code)
|
||||
}
|
||||
}
|
||||
@@ -1322,6 +1322,7 @@ impl ChatShared {
|
||||
mcp: None,
|
||||
tasks: None,
|
||||
memory: None,
|
||||
web: Default::default(),
|
||||
go_version: Default::default(),
|
||||
}
|
||||
}
|
||||
|
||||
@@ -347,6 +347,8 @@ pub struct ChatShared {
|
||||
/// La mémoire, quand la façade la tient (LOKI_MEM=rust) : les outils
|
||||
/// `mem_*` et `tracker` du tour s'exécutent ici (lot 40).
|
||||
pub memory: Option<Arc<crate::memory::Memory>>,
|
||||
/// Les outils web (lot 42) : cache des pages, Crawl4AI, DuckDuckGo.
|
||||
pub web: Arc<crate::webtools::Web>,
|
||||
/// La version du Go, annoncée aux serveurs MCP à une reconnexion ; lue
|
||||
/// une fois.
|
||||
pub go_version: tokio::sync::OnceCell<String>,
|
||||
@@ -698,6 +700,13 @@ impl TurnHost for GoTurnHost {
|
||||
};
|
||||
}
|
||||
}
|
||||
// Outils web (lot 42) : le cache des pages est ici.
|
||||
if let Some(result) = self.shared.web.exec(&self.shared.store, name, args).await {
|
||||
return ToolOutcome {
|
||||
result,
|
||||
..Default::default()
|
||||
};
|
||||
}
|
||||
// Outils task_*, quand la façade tient les tâches (lot 37).
|
||||
if crate::tasks::tools::is_tool(name)
|
||||
&& let Some(tasks) = &self.shared.tasks
|
||||
|
||||
@@ -968,8 +968,67 @@ pub fn go_regex(p: &str) -> Option<String> {
|
||||
let cs: Vec<char> = p.chars().collect();
|
||||
let mut out = String::with_capacity(p.len() + 16);
|
||||
let (mut i, mut class, mut class_start) = (0, false, false);
|
||||
// Répétition qui précède : 0 aucune, 1 une répétition (un « ? » la
|
||||
// rend paresseuse), 2 déjà paresseuse. Le Go refuse d'en répéter une.
|
||||
let mut rep = 0u8;
|
||||
while i < cs.len() {
|
||||
let c = cs[i];
|
||||
if !class {
|
||||
match c {
|
||||
'*' | '+' | '?' => {
|
||||
if rep == 1 && c == '?' {
|
||||
rep = 2;
|
||||
} else if rep != 0 {
|
||||
return None;
|
||||
} else {
|
||||
rep = 1;
|
||||
}
|
||||
out.push(c);
|
||||
i += 1;
|
||||
continue;
|
||||
}
|
||||
'{' => {
|
||||
// `{n}`, `{n,}`, `{n,m}` (1000 au plus) ; sinon un « { »
|
||||
// littéral pour le Go.
|
||||
let rest: String = cs[i + 1..].iter().collect();
|
||||
let body = rest.split_once('}').map(|(b, _)| b);
|
||||
let parsed = body.and_then(|b| {
|
||||
let (lo, hi) = match b.split_once(',') {
|
||||
Some((lo, hi)) => (lo, Some(hi)),
|
||||
None => (b, None),
|
||||
};
|
||||
let num = |t: &str| {
|
||||
(!t.is_empty() && t.bytes().all(|d| d.is_ascii_digit()))
|
||||
.then(|| t.parse::<u64>().unwrap_or(u64::MAX))
|
||||
};
|
||||
let lo = num(lo)?;
|
||||
let hi = match hi {
|
||||
None => Some(lo),
|
||||
Some("") => None,
|
||||
Some(h) => Some(num(h)?),
|
||||
};
|
||||
Some((b.chars().count(), lo, hi))
|
||||
});
|
||||
match parsed {
|
||||
Some((len, lo, hi)) => {
|
||||
if lo > 1000 || hi.is_some_and(|h| h > 1000 || h < lo) || rep != 0 {
|
||||
return None;
|
||||
}
|
||||
rep = 1;
|
||||
out.extend(&cs[i..i + len + 2]);
|
||||
i += len + 2;
|
||||
}
|
||||
None => {
|
||||
rep = 0;
|
||||
out.push_str(r"\{");
|
||||
i += 1;
|
||||
}
|
||||
}
|
||||
continue;
|
||||
}
|
||||
_ => rep = 0,
|
||||
}
|
||||
}
|
||||
if c == '\\' && i + 1 < cs.len() {
|
||||
let n = cs[i + 1];
|
||||
i += 2;
|
||||
@@ -990,6 +1049,18 @@ pub fn go_regex(p: &str) -> Option<String> {
|
||||
class_start = false;
|
||||
continue;
|
||||
}
|
||||
// `\p{…}`, `\P{…}`, `\x{…}` : les accolades sont à l'échappement.
|
||||
if matches!(n, 'p' | 'P' | 'x')
|
||||
&& cs.get(i) == Some(&'{')
|
||||
&& let Some(k) = cs[i..].iter().position(|&c| c == '}')
|
||||
{
|
||||
out.push('\\');
|
||||
out.push(n);
|
||||
out.extend(&cs[i..=i + k]);
|
||||
i += k + 1;
|
||||
class_start = false;
|
||||
continue;
|
||||
}
|
||||
// Un chiffre seul est une référence arrière pour le Go, qui la
|
||||
// refuse ; avec l'octal permis, `regex` l'accepterait.
|
||||
if ('1'..='7').contains(&n) && !cs.get(i).is_some_and(|d| ('0'..='7').contains(d)) {
|
||||
@@ -2872,6 +2943,19 @@ mod tests {
|
||||
assert_eq!(m(r"\Q(x", "a(x"), Some("(x".into()));
|
||||
// Le Go refuse \Q dans une classe.
|
||||
assert_eq!(go_regex(r"[\Q]-\E]+"), None);
|
||||
// Répétition d'une répétition : refusée par le Go ; « ? » paresseux
|
||||
// permis une fois ; « { » hors répétition littéral ; bornes ≤ 1000.
|
||||
for p in ["a**", "a+*", "a*??", "a{2}*", "a*{2}", "a{1001}", "a{3,2}"] {
|
||||
assert_eq!(go_regex(p), None, "{p}");
|
||||
}
|
||||
assert_eq!(go_regex("a*?b+?c??").as_deref(), Some("a*?b+?c??"));
|
||||
assert_eq!(go_regex("a{2,}{").as_deref(), Some(r"a{2,}\{"));
|
||||
assert_eq!(go_regex("x{a}").as_deref(), Some(r"x\{a}"));
|
||||
assert!(compile_go_regex("x{,3}").unwrap().is_match("x{,3}"));
|
||||
assert_eq!(
|
||||
go_regex(r"\p{Greek}+\x{41}{2}").as_deref(),
|
||||
Some(r"\p{Greek}+\x{41}{2}")
|
||||
);
|
||||
// Dans une classe, « [ », « && », « -- » et « ~~ » sont littéraux.
|
||||
assert_eq!(m(r"[a[]+", "x[a["), Some("[a[".into()));
|
||||
assert_eq!(m(r"[a&&b]+", "x&ab"), Some("&ab".into()));
|
||||
|
||||
@@ -304,7 +304,7 @@ fn kind_of(v: &Value) -> &'static str {
|
||||
}
|
||||
|
||||
/// `quoteChar` du scanner Go.
|
||||
fn quote_char(c: u8) -> String {
|
||||
pub(crate) fn quote_char(c: u8) -> String {
|
||||
match c {
|
||||
b'\'' => r"'\''".into(),
|
||||
b'"' => "'\"'".into(),
|
||||
|
||||
@@ -52,3 +52,4 @@ pub mod tune;
|
||||
pub mod tz;
|
||||
pub mod ui;
|
||||
pub mod update;
|
||||
pub mod webtools;
|
||||
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,192 @@
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn md5_rfc1321() {
|
||||
for (input, want) in [
|
||||
("", "d41d8cd98f00b204e9800998ecf8427e"),
|
||||
("a", "0cc175b9c0f1b6a831c399e269772661"),
|
||||
("abc", "900150983cd24fb0d6963f7d28e17f72"),
|
||||
("message digest", "f96b697d7cb7938d525a2f31aaf161d0"),
|
||||
(
|
||||
"12345678901234567890123456789012345678901234567890123456789012345678901234567890",
|
||||
"57edf4a22be3c955ac49da2e2107b67a",
|
||||
),
|
||||
] {
|
||||
assert_eq!(hex::encode(md5(input.as_bytes())), want, "{input}");
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn plafond_de_sortie() {
|
||||
let s = "é".repeat(MAX_OUTPUT);
|
||||
assert_eq!(cap_output(&s), s);
|
||||
let long = format!("{s}x");
|
||||
let out = cap_output(&long);
|
||||
assert!(out.starts_with(&s));
|
||||
assert!(out.ends_with("ou cible avec web_grep.]"));
|
||||
assert!(!out.contains('x'));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn cles_du_cache() {
|
||||
let base = FetchOpts {
|
||||
dismiss_popups: true,
|
||||
..Default::default()
|
||||
};
|
||||
assert_eq!(cache_key_for("u", &base), "u");
|
||||
let o = FetchOpts {
|
||||
dismiss_popups: false,
|
||||
..Default::default()
|
||||
};
|
||||
// Valeurs tirées de cacheKeyFor du Go.
|
||||
assert_eq!(cache_key_for("u", &o), "u#77fc370a");
|
||||
let o = FetchOpts {
|
||||
dismiss_popups: true,
|
||||
actions: vec!["a < b && c".into()],
|
||||
..Default::default()
|
||||
};
|
||||
assert_eq!(cache_key_for("u", &o), "u#d1533b44");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn urls() {
|
||||
for (raw, want) in [
|
||||
(
|
||||
"https://github.com/a/b",
|
||||
"https://raw.githubusercontent.com/a/b/HEAD/README.md",
|
||||
),
|
||||
(
|
||||
"//github.com/a/b/",
|
||||
"https://raw.githubusercontent.com/a/b/HEAD/README.md",
|
||||
),
|
||||
(
|
||||
"https://github.com/a%20x/b?tab=1#f",
|
||||
"https://raw.githubusercontent.com/a x/b/HEAD/README.md",
|
||||
),
|
||||
("https://GitHub.com/a/b", "https://GitHub.com/a/b"),
|
||||
("https://github.com:443/a/b", "https://github.com:443/a/b"),
|
||||
("https://github.com/a", "https://github.com/a"),
|
||||
("https://github.com/a/b/c", "https://github.com/a/b/c"),
|
||||
("https://github.com/%zz/b", "https://github.com/%zz/b"),
|
||||
("github.com/a/b", "github.com/a/b"),
|
||||
(":github.com/a/b", ":github.com/a/b"),
|
||||
] {
|
||||
assert_eq!(normalize_crawl_url(raw), want, "{raw}");
|
||||
}
|
||||
for (raw, want) in [
|
||||
(
|
||||
"//duckduckgo.com/l/?uddg=https%3A%2F%2Fa.example%2F%3Fx%3D1%26y%3D2&rut=z",
|
||||
"https://a.example/?x=1&y=2",
|
||||
),
|
||||
("https://b.example/", "https://b.example/"),
|
||||
(
|
||||
"//duckduckgo.com/l/?uddg=%ZZ",
|
||||
"https://duckduckgo.com/l/?uddg=%ZZ",
|
||||
),
|
||||
("//duckduckgo.com/l/?x;y=1&uddg=a%2Bb+c", "a b c"),
|
||||
("/l/?uddg=", "/l/?uddg="),
|
||||
] {
|
||||
assert_eq!(decode_uddg(raw), want, "{raw}");
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn entites() {
|
||||
assert_eq!(
|
||||
decode_html_entities(
|
||||
"a &lt; été � � � ""
|
||||
),
|
||||
"a < été � A � \""
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn nombres_du_go() {
|
||||
assert_eq!(go_int(3.9), 3);
|
||||
assert_eq!(go_int(-3.9), -3);
|
||||
if cfg!(target_arch = "x86_64") {
|
||||
assert_eq!(go_int(1e300), i64::MIN);
|
||||
assert_eq!(go_int(f64::NAN), i64::MIN);
|
||||
}
|
||||
assert_eq!(format_bytes(1023), "1023 B");
|
||||
assert_eq!(format_bytes(1536), "1.5 KB");
|
||||
assert_eq!(format_bytes(3 * 1024 * 1024 + 1), "3.00 MB");
|
||||
assert_eq!(tail_runes("abcdé", 2), "dé");
|
||||
assert_eq!(tail_runes("ab", 5), "ab");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn erreurs_de_regex() {
|
||||
for (expr, want) in [
|
||||
("(?i)(", "missing closing ): `(?i)(`"),
|
||||
("(?i)a)", "unexpected ): `(?i)a)`"),
|
||||
("(?i)[abc", "missing closing ]: `[abc`"),
|
||||
("(?i)[[:alpha:]", "missing closing ]: `[[:alpha:]`"),
|
||||
("(?i)*x", "missing argument to repetition operator: `*`"),
|
||||
("(?i)a|+", "missing argument to repetition operator: `+`"),
|
||||
("(?i)a**", "invalid nested repetition operator: `**`"),
|
||||
("(?i)a*??", "invalid nested repetition operator: `*??`"),
|
||||
("(?i)\\q", "invalid escape sequence: `\\q`"),
|
||||
("(?i)\\1", "invalid escape sequence: `\\1`"),
|
||||
("(?i)a\\", "trailing backslash at end of expression: ``"),
|
||||
] {
|
||||
assert_eq!(
|
||||
go_regexp_error(expr),
|
||||
format!("error parsing regexp: {want}"),
|
||||
"{expr}"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn reponses_de_crawl4ai() {
|
||||
let md = |raw: &str, pr: bool| crawl_markdown(raw.as_bytes(), pr);
|
||||
assert_eq!(
|
||||
md(
|
||||
r#"{"results":[{"success":true,"markdown":{"raw_markdown":"r","fit_markdown":"f"}}]}"#,
|
||||
true
|
||||
),
|
||||
Ok("r".into())
|
||||
);
|
||||
assert_eq!(
|
||||
md(
|
||||
r#"{"results":[{"success":true,"markdown":{"raw_markdown":"","fit_markdown":"f"}}]}"#,
|
||||
true
|
||||
),
|
||||
Ok("f".into())
|
||||
);
|
||||
assert_eq!(
|
||||
md(
|
||||
r#"{"results":[{"success":true,"markdown":{"raw_markdown":"r"}}]}"#,
|
||||
false
|
||||
),
|
||||
Ok("r".into())
|
||||
);
|
||||
assert_eq!(
|
||||
md(r#"{"Results":[{"Success":true,"Markdown":"x"}]}"#, false),
|
||||
Ok("x".into())
|
||||
);
|
||||
assert_eq!(md("null", false), Err("Crawl4AI : échec du crawl".into()));
|
||||
assert_eq!(md("5", false), Err("Crawl4AI : réponse vide".into()));
|
||||
assert_eq!(
|
||||
md(r#"{"results":[{"success":"x"}],"success":"y"}"#, false),
|
||||
Err("Crawl4AI : échec du crawl".into())
|
||||
);
|
||||
assert_eq!(
|
||||
md(r#"{"results":[null,{"success":true}]}"#, false),
|
||||
Err("Crawl4AI : échec du crawl".into())
|
||||
);
|
||||
assert_eq!(
|
||||
md(r#"{"results":[{"success":true,"markdown":null}]}"#, false),
|
||||
Ok(String::new())
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn lignes_normalisees() {
|
||||
assert_eq!(
|
||||
normalize_lines("\n\n a \r\nb \n\n\n\nb\nb\nc\t\n\n"),
|
||||
[" a", "b", "", "b", "c"]
|
||||
);
|
||||
assert!(normalize_lines(" \n\t\n").is_empty());
|
||||
}
|
||||
@@ -149,6 +149,12 @@ pub struct FakeGo {
|
||||
pub projsnaps: Mutex<std::collections::VecDeque<serde_json::Value>>,
|
||||
pub projsnap_refresh: Mutex<Option<serde_json::Value>>,
|
||||
pub projsnap_refreshes: Mutex<Vec<serde_json::Value>>,
|
||||
/// Moteur web intégré (lot 42) : réponses de /internal/v1/web/fetch par
|
||||
/// URL et de /internal/v1/web/search par « requête|limite » ; appels
|
||||
/// reçus (« fetch URL », « search requête|limite »).
|
||||
pub web_fetch: Mutex<HashMap<String, serde_json::Value>>,
|
||||
pub web_search: Mutex<HashMap<String, serde_json::Value>>,
|
||||
pub web_log: Mutex<Vec<String>>,
|
||||
}
|
||||
|
||||
impl FakeGo {
|
||||
@@ -505,6 +511,28 @@ async fn handle(st: Arc<FakeGo>, req: Request<Incoming>) -> Response<Full<Bytes>
|
||||
200,
|
||||
serde_json::json!({ "generating": *st.generating.lock().unwrap() }).to_string(),
|
||||
),
|
||||
"/internal/v1/web/fetch" | "/internal/v1/web/search" => {
|
||||
let fetch = path == "/internal/v1/web/fetch";
|
||||
let body = req.into_body().collect().await.unwrap().to_bytes();
|
||||
let v: serde_json::Value = serde_json::from_slice(&body).unwrap_or_default();
|
||||
let key = if fetch {
|
||||
v["url"].as_str().unwrap_or("").to_string()
|
||||
} else {
|
||||
format!("{}|{}", v["query"].as_str().unwrap_or(""), v["limit"])
|
||||
};
|
||||
st.web_log
|
||||
.lock()
|
||||
.unwrap()
|
||||
.push(format!("{} {key}", if fetch { "fetch" } else { "search" }));
|
||||
let map = if fetch { &st.web_fetch } else { &st.web_search };
|
||||
let out = map
|
||||
.lock()
|
||||
.unwrap()
|
||||
.get(&key)
|
||||
.cloned()
|
||||
.unwrap_or_else(|| serde_json::json!({ "error": format!("inconnu : {key}") }));
|
||||
resp(200, out.to_string())
|
||||
}
|
||||
"/internal/v1/web/reach/reset" => {
|
||||
*st.reach_resets.lock().unwrap() += 1;
|
||||
resp(204, "")
|
||||
|
||||
@@ -671,6 +671,7 @@ async fn tour_complet_par_le_go() {
|
||||
mcp: None,
|
||||
tasks: None,
|
||||
memory: None,
|
||||
web: Default::default(),
|
||||
go_version: Default::default(),
|
||||
});
|
||||
let cfg = std::collections::HashMap::new();
|
||||
@@ -2746,3 +2747,44 @@ async fn outil_memoire_par_la_facade() {
|
||||
.clone();
|
||||
assert_eq!(tool, serde_json::json!("[ok] page 'chat' créée"));
|
||||
}
|
||||
|
||||
/// Lot 42 : un outil web d'un tour de la façade s'exécute chez elle (cache
|
||||
/// des pages compris) ; seule la conversion du moteur intégré est demandée
|
||||
/// au Go, jamais /tool/exec.
|
||||
#[tokio::test]
|
||||
async fn outil_web_par_la_facade() {
|
||||
let call = sse(&[
|
||||
serde_json::json!({"choices":[{"delta":{"tool_calls":[{"index":0,"id":"w1","type":"function","function":{"name":"web_open","arguments":"{\"url\":\"https://exemple.test/doc\"}"}}]}}]}),
|
||||
serde_json::json!({"choices":[{"delta":{},"finish_reason":"tool_calls"}]}),
|
||||
]);
|
||||
let (url, seen) = fake_engine(vec![call, answer("lu")]).await;
|
||||
let (svc, go, _stop, _t) = service_with(Some(&url), |_| {}).await;
|
||||
go.set("config", "WEB_ENGINE", "go");
|
||||
go.web_fetch.lock().unwrap().insert(
|
||||
"https://exemple.test/doc".into(),
|
||||
serde_json::json!({"markdown": "# Doc\n\n## Partie\ntexte"}),
|
||||
);
|
||||
svc.send(serde_json::json!({"message": "lis la doc", "cid": "w"}))
|
||||
.await
|
||||
.unwrap();
|
||||
until_done(&svc, 1).await;
|
||||
assert!(go.tool_log.lock().unwrap().is_empty(), "{:?}", go.tool_log);
|
||||
assert_eq!(
|
||||
*go.web_log.lock().unwrap(),
|
||||
vec!["fetch https://exemple.test/doc".to_string()]
|
||||
);
|
||||
let seen = seen.lock().unwrap().clone();
|
||||
let tool = seen[1].1["messages"]
|
||||
.as_array()
|
||||
.unwrap()
|
||||
.iter()
|
||||
.rfind(|m| m["role"] == "tool")
|
||||
.unwrap()["content"]
|
||||
.clone();
|
||||
assert_eq!(
|
||||
tool,
|
||||
serde_json::json!(
|
||||
"# Ouvert : https://exemple.test/doc\nTotal : 4 lignes, 23 B (23 caractères)\nEn cache 10 min. Utilise web_read ou web_grep pour lire.\n\n## Plan (n° de ligne des titres)\n```\n 1 | Doc\n 3 | Partie\n```"
|
||||
)
|
||||
);
|
||||
}
|
||||
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,291 @@
|
||||
//! Lot 42 : outils web de l'agent (`web_search`, `web_images`, `web_open`,
|
||||
//! `web_read`, `web_grep`) exécutés par la façade.
|
||||
//!
|
||||
//! Le journal tiré du Go (`internal/loki/parity_web_tools_test.go` →
|
||||
//! `tests/parity_web/outils.json`) est rejoué contre le port : mêmes
|
||||
//! réponses du faux Crawl4AI et du faux DuckDuckGo, ce que le moteur intégré
|
||||
//! du Go a tiré des pages servi par le faux Go ; chaque appel doit rendre le
|
||||
//! même résultat et faire les mêmes requêtes (corps envoyés à Crawl4AI,
|
||||
//! en-têtes et cookies envoyés à DuckDuckGo, pages et recherches demandées
|
||||
//! au Go).
|
||||
|
||||
mod common;
|
||||
|
||||
use std::collections::HashMap;
|
||||
use std::convert::Infallible;
|
||||
use std::sync::{Arc, Mutex};
|
||||
use std::time::Duration;
|
||||
|
||||
use bytes::Bytes;
|
||||
use common::fakego;
|
||||
use http_body_util::{BodyExt, Full};
|
||||
use hyper::service::service_fn;
|
||||
use hyper::{Request, Response};
|
||||
use hyper_util::rt::TokioIo;
|
||||
use loki_rs::store::Store;
|
||||
use loki_rs::webtools::Web;
|
||||
use serde_json::{Map, Value, json};
|
||||
use tokio::net::TcpListener;
|
||||
|
||||
fn journal() -> Value {
|
||||
serde_json::from_str(include_str!("parity_web/outils.json")).expect("journal")
|
||||
}
|
||||
|
||||
fn s(v: &Value) -> &str {
|
||||
v.as_str().unwrap_or("")
|
||||
}
|
||||
|
||||
type Log = Arc<Mutex<Vec<Value>>>;
|
||||
|
||||
/// Un petit serveur HTTP : `f(méthode, chemin?requête, en-têtes, corps)`
|
||||
/// rend (statut, en-têtes, corps).
|
||||
async fn serve<F>(f: F) -> String
|
||||
where
|
||||
F: Fn(String, String, hyper::HeaderMap, Vec<u8>) -> (u16, Vec<(String, String)>, String)
|
||||
+ Send
|
||||
+ Sync
|
||||
+ 'static,
|
||||
{
|
||||
let l = TcpListener::bind("127.0.0.1:0").await.unwrap();
|
||||
let addr = l.local_addr().unwrap();
|
||||
let f = Arc::new(f);
|
||||
tokio::spawn(async move {
|
||||
while let Ok((st, _)) = l.accept().await {
|
||||
let f = f.clone();
|
||||
tokio::spawn(async move {
|
||||
let svc = service_fn(move |req: Request<hyper::body::Incoming>| {
|
||||
let f = f.clone();
|
||||
async move {
|
||||
let method = req.method().to_string();
|
||||
let uri = req
|
||||
.uri()
|
||||
.path_and_query()
|
||||
.map(|p| p.to_string())
|
||||
.unwrap_or_default();
|
||||
let headers = req.headers().clone();
|
||||
let body = req.into_body().collect().await.unwrap().to_bytes().to_vec();
|
||||
let (code, hs, out) = f(method, uri, headers, body);
|
||||
let mut r = Response::new(Full::new(Bytes::from(out)));
|
||||
*r.status_mut() = hyper::StatusCode::from_u16(code).unwrap();
|
||||
for (k, v) in hs {
|
||||
r.headers_mut().insert(
|
||||
hyper::header::HeaderName::from_bytes(k.as_bytes()).unwrap(),
|
||||
v.parse().unwrap(),
|
||||
);
|
||||
}
|
||||
Ok::<_, Infallible>(r)
|
||||
}
|
||||
});
|
||||
let _ = hyper::server::conn::http1::Builder::new()
|
||||
.serve_connection(TokioIo::new(st), svc)
|
||||
.await;
|
||||
});
|
||||
}
|
||||
});
|
||||
format!("http://{addr}")
|
||||
}
|
||||
|
||||
fn header(h: &hyper::HeaderMap, k: &str) -> String {
|
||||
h.get(k)
|
||||
.and_then(|v| v.to_str().ok())
|
||||
.unwrap_or("")
|
||||
.to_string()
|
||||
}
|
||||
|
||||
/// Une requête reçue, champs vides omis (comme `omitempty`).
|
||||
fn req(fields: &[(&str, String)]) -> Value {
|
||||
let mut m = Map::new();
|
||||
for (k, v) in fields {
|
||||
if !v.is_empty() {
|
||||
m.insert((*k).into(), json!(v));
|
||||
}
|
||||
}
|
||||
Value::Object(m)
|
||||
}
|
||||
|
||||
/// Le faux Crawl4AI : la réponse rangée pour l'URL demandée (dans l'ordre,
|
||||
/// la dernière reste).
|
||||
async fn fake_crawl(fx: Value, log: Log) -> String {
|
||||
let served: Arc<Mutex<HashMap<String, usize>>> = Default::default();
|
||||
serve(move |method, uri, h, body| {
|
||||
let path = uri.split('?').next().unwrap_or("").to_string();
|
||||
log.lock().unwrap().push(req(&[
|
||||
("path", format!("{method} {path}")),
|
||||
("auth", header(&h, "authorization")),
|
||||
("ctype", header(&h, "content-type")),
|
||||
("body", String::from_utf8_lossy(&body).into_owned()),
|
||||
]));
|
||||
let v: Value = serde_json::from_slice(&body).unwrap_or_default();
|
||||
let target = s(&v["urls"][0]).to_string();
|
||||
let Some(list) = fx[&target].as_array().filter(|l| !l.is_empty()) else {
|
||||
return (404, vec![], format!("inconnu : {target}"));
|
||||
};
|
||||
let mut served = served.lock().unwrap();
|
||||
let n = served.entry(target).or_default();
|
||||
let r = &list[(*n).min(list.len() - 1)];
|
||||
*n += 1;
|
||||
(
|
||||
r["status"].as_u64().unwrap() as u16,
|
||||
vec![],
|
||||
s(&r["body"]).to_string(),
|
||||
)
|
||||
})
|
||||
.await
|
||||
}
|
||||
|
||||
/// Le faux DuckDuckGo des images.
|
||||
async fn fake_ddg(fx: Value, log: Log) -> String {
|
||||
serve(move |_, uri, h, _| {
|
||||
log.lock().unwrap().push(req(&[
|
||||
("path", uri.clone()),
|
||||
("referer", header(&h, "referer")),
|
||||
("accept", header(&h, "accept")),
|
||||
("lang", header(&h, "accept-language")),
|
||||
("ua", header(&h, "user-agent")),
|
||||
("cookie", header(&h, "cookie")),
|
||||
]));
|
||||
let (path, query) = uri.split_once('?').unwrap_or((&uri, ""));
|
||||
let q = url::form_urlencoded::parse(query.as_bytes())
|
||||
.find(|(k, _)| k == "q")
|
||||
.map(|(_, v)| v.into_owned())
|
||||
.unwrap_or_default();
|
||||
let f = &fx[&q];
|
||||
if path == "/i.js" {
|
||||
return (
|
||||
f["api_status"].as_u64().unwrap() as u16,
|
||||
vec![],
|
||||
s(&f["api"]).to_string(),
|
||||
);
|
||||
}
|
||||
let mut hs = vec![];
|
||||
if !s(&f["cookie"]).is_empty() {
|
||||
hs.push(("set-cookie".to_string(), s(&f["cookie"]).to_string()));
|
||||
}
|
||||
(
|
||||
f["page_status"].as_u64().unwrap() as u16,
|
||||
hs,
|
||||
s(&f["page"]).to_string(),
|
||||
)
|
||||
})
|
||||
.await
|
||||
}
|
||||
|
||||
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
||||
async fn outils_web_comme_le_go() {
|
||||
let fx = journal();
|
||||
let tmp = tempfile::tempdir().unwrap();
|
||||
let (_, go, _stop) = fakego::start(tmp.path().to_path_buf()).await;
|
||||
let backend = go.backend.lock().unwrap().unwrap();
|
||||
let store = Store::new(backend, fakego::JETON.into());
|
||||
|
||||
let (crawl_log, img_log): (Log, Log) = Default::default();
|
||||
let crawl = fake_crawl(fx["crawl"].clone(), crawl_log.clone()).await;
|
||||
let ddg = fake_ddg(fx["images"].clone(), img_log.clone()).await;
|
||||
let dead = {
|
||||
let l = std::net::TcpListener::bind("127.0.0.1:0").unwrap();
|
||||
l.local_addr().unwrap().to_string()
|
||||
};
|
||||
let pages = "http://pages.invalid".to_string();
|
||||
let unmask = |v: &str| {
|
||||
v.replace("{{CRAWL}}", &crawl)
|
||||
.replace("{{DDG}}", &ddg)
|
||||
.replace("{{PAGES}}", &pages)
|
||||
.replace("{{DEADADDR}}", &dead)
|
||||
.replace("{{DEAD}}", &format!("http://{dead}"))
|
||||
};
|
||||
let mask = |v: &str| {
|
||||
v.replace(&crawl, "{{CRAWL}}")
|
||||
.replace(&ddg, "{{DDG}}")
|
||||
.replace(&pages, "{{PAGES}}")
|
||||
.replace(&format!("http://{dead}"), "{{DEAD}}")
|
||||
.replace(&dead, "{{DEADADDR}}")
|
||||
};
|
||||
for (k, v) in fx["go_fetch"].as_object().unwrap() {
|
||||
let v: Value = serde_json::from_str(&unmask(&v.to_string())).unwrap();
|
||||
go.web_fetch.lock().unwrap().insert(unmask(k), v);
|
||||
}
|
||||
for (k, v) in fx["go_search"].as_object().unwrap() {
|
||||
go.web_search.lock().unwrap().insert(k.clone(), v.clone());
|
||||
}
|
||||
|
||||
let web = Web::with_ddg_base(&ddg);
|
||||
for c in fx["calls"].as_array().unwrap() {
|
||||
let name = s(&c["name"]);
|
||||
for (k, v) in c["setup"].as_object().into_iter().flatten() {
|
||||
let (bucket, v) = if k == "crawl_key" {
|
||||
("state", s(v).to_string())
|
||||
} else {
|
||||
("config", unmask(s(v)))
|
||||
};
|
||||
if v == "-" {
|
||||
go.del(bucket, k);
|
||||
} else {
|
||||
go.set(bucket, k, &v);
|
||||
}
|
||||
}
|
||||
if let Some(n) = c["advance_s"].as_u64() {
|
||||
web.advance(Duration::from_secs(n));
|
||||
}
|
||||
let args: Map<String, Value> =
|
||||
serde_json::from_str(&unmask(&c["args"].to_string())).unwrap();
|
||||
let got = web
|
||||
.exec(&store, s(&c["tool"]), &args)
|
||||
.await
|
||||
.expect("un outil web");
|
||||
assert_eq!(mask(&got), s(&c["result"]), "{name}");
|
||||
|
||||
let masked = |log: &Log| -> Value {
|
||||
let v = Value::Array(std::mem::take(&mut *log.lock().unwrap()));
|
||||
serde_json::from_str(&mask(&v.to_string())).unwrap()
|
||||
};
|
||||
assert_eq!(masked(&crawl_log), c["crawls"], "{name} : Crawl4AI");
|
||||
// reqwest pose « Accept: */* » quand la requête n'en a pas ; le
|
||||
// client du Go n'en pose aucun (sans effet sur DuckDuckGo).
|
||||
let mut imgs = masked(&img_log);
|
||||
for r in imgs.as_array_mut().into_iter().flatten() {
|
||||
if r["accept"] == "*/*" {
|
||||
r.as_object_mut().unwrap().remove("accept");
|
||||
}
|
||||
}
|
||||
assert_eq!(imgs, c["images"], "{name} : DuckDuckGo");
|
||||
let calls = std::mem::take(&mut *go.web_log.lock().unwrap());
|
||||
let fetched: Vec<String> = calls
|
||||
.iter()
|
||||
.filter_map(|l| l.strip_prefix("fetch "))
|
||||
.map(&mask)
|
||||
.collect();
|
||||
let searched: Vec<String> = calls
|
||||
.iter()
|
||||
.filter_map(|l| l.strip_prefix("search "))
|
||||
.map(|q| q.rsplit_once('|').map_or(q, |(q, _)| q).to_string())
|
||||
.collect();
|
||||
assert_eq!(json!(fetched), c["go_fetch"], "{name} : pages du Go");
|
||||
assert_eq!(json!(searched), c["go_search"], "{name} : recherches du Go");
|
||||
}
|
||||
}
|
||||
|
||||
/// Le Go injoignable : le moteur intégré (par défaut sans configuration)
|
||||
/// le dit ; un autre outil n'est pas servi ici.
|
||||
#[tokio::test]
|
||||
async fn moteur_integre_injoignable() {
|
||||
let dead = std::net::TcpListener::bind("127.0.0.1:0")
|
||||
.unwrap()
|
||||
.local_addr()
|
||||
.unwrap();
|
||||
let store = Store::new(dead, fakego::JETON.into());
|
||||
let web = Web::new();
|
||||
let mut args = Map::new();
|
||||
args.insert("url".into(), json!("https://exemple.test/"));
|
||||
let got = web.exec(&store, "web_open", &args).await.unwrap();
|
||||
assert!(
|
||||
got.starts_with("❌ moteur web intégré injoignable : "),
|
||||
"{got}"
|
||||
);
|
||||
args.insert("query".into(), json!("x"));
|
||||
let got = web.exec(&store, "web_search", &args).await.unwrap();
|
||||
assert!(
|
||||
got.starts_with("❌ Recherche échouée : moteur web intégré injoignable : "),
|
||||
"{got}"
|
||||
);
|
||||
assert_eq!(web.exec(&store, "web_screenshot", &args).await, None);
|
||||
}
|
||||
Reference in new issue
Block a user