mirror of
https://github.com/R0m1k3/CollectFlow.git
synced 2026-10-11 17:26:32 +02:00
perf(qlik): optimize network sync extraction
This commit is contained in:
1 parent
1cb6c2496d
commit
abac5c5ee5
5 files changed
+468
-80
No files matched your search
@@ -0,0 +1,106 @@
|
||||
# Qlik — Performance & tuning de la sync réseau
|
||||
|
||||
Ce document rassemble les variables d'environnement et leviers qui agissent
|
||||
sur la **vitesse** et la **robustesse** de l'extraction Qlik réseau
|
||||
(`POST /api/qlik/sync`, routeur `src/app/api/qlik/sync/route.ts`,
|
||||
extracteur `src/lib/qlik-playwright.ts`, helper de fenêtre
|
||||
`src/lib/qlik-date-range.ts`).
|
||||
|
||||
Toutes ces variables sont **optionnelles** : valeur par défaut sûre.
|
||||
|
||||
> ⚠️ Aucune concurrence / parallélisme Qlik n'a été ajouté — chaque sync
|
||||
> reste **mono-flux** (un seul websocket Engine, une seule fenêtre
|
||||
> temporelle). Les optimisations ci-dessous n'augmentent pas la charge
|
||||
> sur le moteur Qlik ; elles **réduisent** la quantité de travail qu'on
|
||||
> lui demande.
|
||||
|
||||
---
|
||||
|
||||
## Variables d'environnement
|
||||
|
||||
| Nom | Type | Défaut | Bornes | Effet |
|
||||
| --------------------------- | ---- | ------ | ------------------- | -------------------------------------------------------------------------------------- |
|
||||
| `QLIK_SYNC_MONTHS_BACK` | int | `12` | `1`..`12` | Profondeur de la fenêtre temporelle (mois glissants, mois courant exclu). Plus petit = moins de mois scannés = extraction plus rapide. |
|
||||
| `QLIK_SETTLE_MS` | int | `150` | `0`..`2000` | Latence (ms) après chaque mutation de sélection (Date / Article Code) avant l'appel RPC suivant. Protège l'Engine contre les collisions qui provoquent `code:15 Request aborted`. |
|
||||
| `QLIK_CODE15_MAX_ATTEMPTS` | int | `4` | `1`..`8` | Nombre max de retries sur Engine `code:15` par appel RPC, avec backoff `QLIK_SETTLE_MS * attempt`. |
|
||||
| `QLIK_ARTICLE_BATCH_SIZE` | int | `150` | `1`..`500` | Taille du premier lot d'`Article Code` sélectionné par appel `SelectValues`. |
|
||||
| `QLIK_ARTICLE_BATCH_FALLBACKS` | CSV | `75,50` | chaque valeur `1`..`500` | Tailles de repli quand un lot déclenche un `code:15`. Valeurs ordonnées croissant appliquées successivement. |
|
||||
|
||||
> **Format CSV pour `QLIK_ARTICLE_BATCH_FALLBACKS`** : valeurs séparées par
|
||||
> des virgules, ex. `QLIK_ARTICLE_BATCH_FALLBACKS=100,75,50,25`. Les doublons
|
||||
> et valeurs hors bornes sont dédupliqués / clampés automatiquement, puis appliqués dans l'ordre fourni.
|
||||
|
||||
---
|
||||
|
||||
## Optimisations déjà en place (pattern durable)
|
||||
|
||||
Le pipeline reste **mono-connexion** pour ne jamais surcharger le moteur Qlik :
|
||||
|
||||
1. **Fenêtre temporelle bornée** : on ne demande jamais à Qlik de scanner
|
||||
l'historique complet ; uniquement les `QLIK_SYNC_MONTHS_BACK` derniers
|
||||
mois.
|
||||
2. **Découpe mensuelle** : `getMonthRanges()` produit 1 sous-fenêtre par mois
|
||||
calendaire. Chaque cube reste ainsi borné à ~28–31 jours × N codes.
|
||||
3. **Hypercube session object JETABLE** : `fetchCubeForSelection()` crée un
|
||||
`CreateSessionObject` neuf par lot, lit ses pages, puis appelle
|
||||
`DestroySessionObject`. C'est ce qui a éliminé les `code:15 Request aborted`
|
||||
à grande échelle (la réutilisation du même objet sur des centaines de
|
||||
recalculs forçait l'Engine à recalculer en boucle).
|
||||
4. **Fallbacks code 15** : si un lot trop large déclenche un `code:15`, on
|
||||
retente automatiquement avec une taille plus petite (cf.
|
||||
`QLIK_ARTICLE_BATCH_FALLBACKS`) sans tout annuler.
|
||||
|
||||
## Nouveautés (cette itération)
|
||||
|
||||
1. **Fenêtre paramétrable `QLIK_SYNC_MONTHS_BACK`** (1..12, défaut 12). Permet
|
||||
de scoper un sync test à 6 mois au lieu de 12, ou de n'extraire que les 3
|
||||
derniers mois pour un check rapide. Le helper `envMonthsBack()` (dans
|
||||
`src/lib/qlik-date-range.ts`) clamp la valeur et applique le défaut si
|
||||
l'env est mal configurée.
|
||||
2. **Filtre strict des codes centraux** dans `route.ts` (regex `[A-Z0-9][A-Z0-9_-]{1,29}`,
|
||||
longueur 2..30, exclusion de `""`, `"-"`, `"_"`). Réduit les erreurs Engine
|
||||
sur des valeurs parasites et le gaspillage réseau. Le log expose le **nombre**
|
||||
de codes rejetés mais pas les valeurs (hygiène logs).
|
||||
3. **Résumé timing final** : ligne unique `[qlik-pw][summary] rows=… months=…
|
||||
batches=… code15Retries=… fallbackCount=… totalMs=…` émise en fin de sync
|
||||
pour audit/debug sans avoir à parser les logs détaillés par lot.
|
||||
|
||||
## Exemple de `.env.local`
|
||||
|
||||
```env
|
||||
# Sync Qlik : fenêtre par défaut 12 mois ; on peut descendre à 6 pour un
|
||||
# test rapide ou un check hebdo sans recharger tout l'historique.
|
||||
QLIK_SYNC_MONTHS_BACK=12
|
||||
|
||||
# Batch sizes / fallbacks (laisser par défaut sauf contre-indication).
|
||||
QLIK_ARTICLE_BATCH_SIZE=150
|
||||
QLIK_ARTICLE_BATCH_FALLBACKS=75,50
|
||||
QLIK_SETTLE_MS=150
|
||||
QLIK_CODE15_MAX_ATTEMPTS=4
|
||||
```
|
||||
|
||||
## Tests
|
||||
|
||||
```bash
|
||||
# Tests purs Node (aucune dépendance Next.js).
|
||||
npx tsx scripts/test-qlik-date-range.ts
|
||||
```
|
||||
|
||||
Couvre :
|
||||
- repères Qlik (sérial ↔ ISO),
|
||||
- fenêtre 12 mois (référence),
|
||||
- `getMonthRanges()` : 12 mois / fenêtre partielle / année bissextile,
|
||||
- option `monthsBack` : 6 mois + clamp 0 / -5 / 24 / NaN / undefined,
|
||||
- helper `envMonthsBack()` : env absent / vide / valide / hors bornes /
|
||||
non numérique / décimal / espaces.
|
||||
|
||||
## Référence
|
||||
|
||||
- `src/lib/qlik-date-range.ts` — fenêtre, `envMonthsBack`, `getMonthRanges`.
|
||||
- `src/lib/qlik-playwright.ts` — extracteur Playwright, hypercube jetable,
|
||||
fallbacks code 15, `[qlik-pw][summary]` final.
|
||||
- `src/app/api/qlik/sync/route.ts` — route admin, `filterCentralCodes`,
|
||||
applique `QLIK_SYNC_MONTHS_BACK` via `envMonthsBack`.
|
||||
- `scripts/test-qlik-date-range.ts` — tests CLI.
|
||||
- `docs/qlik-integration.md` — doc d'intégration générale (auth NTLM,
|
||||
discovery, hypercube, etc.).
|
||||
@@ -9,6 +9,8 @@
|
||||
* label=2025-06_2026-05, 365 serials)
|
||||
* - exemples supplémentaires (1er du mois, fin d'année, année bissextile)
|
||||
* - erreurs attendues sur entrée mal formée
|
||||
* - option `monthsBack` : 6 mois, bornes (clamp 1..12)
|
||||
* - helper `envMonthsBack` : valeur absente, valide, hors bornes, non numérique
|
||||
*
|
||||
* Aucun framework : assertions manuelles, exit code 0/1.
|
||||
*/
|
||||
@@ -19,7 +21,11 @@ import {
|
||||
buildGridNetworkQlikDateFilter,
|
||||
chunkSerials,
|
||||
getMonthRanges,
|
||||
envMonthsBack,
|
||||
QLIK_DATE_ANCHORS,
|
||||
QLIK_MONTHS_BACK_MIN,
|
||||
QLIK_MONTHS_BACK_MAX,
|
||||
QLIK_MONTHS_BACK_DEFAULT,
|
||||
} from "../src/lib/qlik-date-range";
|
||||
|
||||
let pass = 0;
|
||||
@@ -142,5 +148,47 @@ eq("3 mois (juin, juillet, août)", monthsShort.length, 3);
|
||||
eq("août tronqué à 15j", monthsShort[2].dailySerials.length, 15);
|
||||
eq("août dateFin = 2025-08-15", monthsShort[2].dateFin, "2025-08-15");
|
||||
|
||||
console.log("\n[10] monthsBack=6 (now=2026-06-21) → 2025-12-01 .. 2026-05-31");
|
||||
const f6 = buildGridNetworkQlikDateFilter(new Date("2026-06-21T12:00:00Z"), 6);
|
||||
eq("dateDebut", f6.dateDebut, "2025-12-01");
|
||||
eq("dateFin", f6.dateFin, "2026-05-31");
|
||||
eq("qStart", f6.qStart, 45992);
|
||||
eq("qEnd", f6.qEnd, 46173);
|
||||
eq("label", f6.label, "2025-12_2026-05");
|
||||
eq("dailySerials.length = 182 (déc 2025:31 + jan:31 + fév:28 + mar:31 + avr:30 + mai:31)", f6.dailySerials.length, 182);
|
||||
const months6 = getMonthRanges(f6);
|
||||
eq("getMonthRanges: 6 mois exactement", months6.length, 6);
|
||||
eq("premier mois label", months6[0].label, "2025-12");
|
||||
eq("dernier mois label", months6[5].label, "2026-05");
|
||||
const totalDays6 = months6.reduce((acc, m) => acc + m.dailySerials.length, 0);
|
||||
eq("somme des jours = 182", totalDays6, 182);
|
||||
|
||||
console.log("\n[11] monthsBack — bornes et clamp");
|
||||
eq("monthsBack=1 (now=2026-06-21) → 2026-05-01 .. 2026-05-31 (1 mois complet)",
|
||||
(() => { const x = buildGridNetworkQlikDateFilter(new Date("2026-06-21T12:00:00Z"), 1); return x.dateDebut === "2026-05-01" && x.dateFin === "2026-05-31"; })(),
|
||||
true);
|
||||
eq("monthsBack=0 clampé à 1", (() => { const x = buildGridNetworkQlikDateFilter(new Date("2026-06-21T12:00:00Z"), 0); return x.dateDebut === "2026-05-01"; })(), true);
|
||||
eq("monthsBack=24 clampé à 12 (max)", (() => { const x = buildGridNetworkQlikDateFilter(new Date("2026-06-21T12:00:00Z"), 24); return x.dateDebut === "2025-06-01" && x.dateFin === "2026-05-31"; })(), true);
|
||||
eq("monthsBack=-5 clampé à 1", (() => { const x = buildGridNetworkQlikDateFilter(new Date("2026-06-21T12:00:00Z"), -5); return x.dateDebut === "2026-05-01"; })(), true);
|
||||
eq("monthsBack=NaN ramené au défaut 12", (() => { const x = buildGridNetworkQlikDateFilter(new Date("2026-06-21T12:00:00Z"), Number.NaN); return x.dateDebut === "2025-06-01" && x.dateFin === "2026-05-31"; })(), true);
|
||||
eq("monthsBack=undefined (par défaut) = défaut 12", (() => { const x = buildGridNetworkQlikDateFilter(new Date("2026-06-21T12:00:00Z")); return x.dateDebut === "2025-06-01" && x.dateFin === "2026-05-31"; })(), true);
|
||||
eq("bornes exportées MIN=1", QLIK_MONTHS_BACK_MIN, 1);
|
||||
eq("bornes exportées MAX=12", QLIK_MONTHS_BACK_MAX, 12);
|
||||
eq("défaut exporté = 12", QLIK_MONTHS_BACK_DEFAULT, 12);
|
||||
|
||||
console.log("\n[12] envMonthsBack — lecture env");
|
||||
const PREV = process.env.QLIK_TEST_MONTHS_BACK;
|
||||
delete process.env.QLIK_TEST_MONTHS_BACK;
|
||||
eq("env absent → défaut", envMonthsBack("QLIK_TEST_MONTHS_BACK"), QLIK_MONTHS_BACK_DEFAULT);
|
||||
eq("env vide → défaut", (() => { process.env.QLIK_TEST_MONTHS_BACK = ""; return envMonthsBack("QLIK_TEST_MONTHS_BACK"); })(), QLIK_MONTHS_BACK_DEFAULT);
|
||||
eq("env '6' → 6", (() => { process.env.QLIK_TEST_MONTHS_BACK = "6"; return envMonthsBack("QLIK_TEST_MONTHS_BACK"); })(), 6);
|
||||
eq("env 'abc' → défaut", (() => { process.env.QLIK_TEST_MONTHS_BACK = "abc"; return envMonthsBack("QLIK_TEST_MONTHS_BACK"); })(), QLIK_MONTHS_BACK_DEFAULT);
|
||||
eq("env '24' clampé à MAX=12", (() => { process.env.QLIK_TEST_MONTHS_BACK = "24"; return envMonthsBack("QLIK_TEST_MONTHS_BACK"); })(), QLIK_MONTHS_BACK_MAX);
|
||||
eq("env '-3' clampé à MIN=1", (() => { process.env.QLIK_TEST_MONTHS_BACK = "-3"; return envMonthsBack("QLIK_TEST_MONTHS_BACK"); })(), QLIK_MONTHS_BACK_MIN);
|
||||
eq("env '0' clampé à MIN=1", (() => { process.env.QLIK_TEST_MONTHS_BACK = "0"; return envMonthsBack("QLIK_TEST_MONTHS_BACK"); })(), QLIK_MONTHS_BACK_MIN);
|
||||
eq("env '3.7' troncé à 3", (() => { process.env.QLIK_TEST_MONTHS_BACK = "3.7"; return envMonthsBack("QLIK_TEST_MONTHS_BACK"); })(), 3);
|
||||
eq("env ' 8 ' trimé puis parsé → 8", (() => { process.env.QLIK_TEST_MONTHS_BACK = " 8 "; return envMonthsBack("QLIK_TEST_MONTHS_BACK"); })(), 8);
|
||||
if (PREV === undefined) delete process.env.QLIK_TEST_MONTHS_BACK; else process.env.QLIK_TEST_MONTHS_BACK = PREV;
|
||||
|
||||
console.log(`\n=== ${pass} passés, ${fail} échoués ===`);
|
||||
if (fail > 0) process.exit(1);
|
||||
@@ -3,7 +3,13 @@ import { auth } from "@/lib/auth";
|
||||
import { fetchNetworkMetricsPlaywright } from "@/lib/qlik-playwright";
|
||||
import { upsertNetworkMetrics } from "@/lib/qlik-network-cache";
|
||||
import { pgGetArticlesByFournisseur } from "@/lib/pg-ff-client";
|
||||
import { buildGridNetworkQlikDateFilter } from "@/lib/qlik-date-range";
|
||||
import {
|
||||
buildGridNetworkQlikDateFilter,
|
||||
envMonthsBack,
|
||||
QLIK_MONTHS_BACK_DEFAULT,
|
||||
QLIK_MONTHS_BACK_MAX,
|
||||
QLIK_MONTHS_BACK_MIN,
|
||||
} from "@/lib/qlik-date-range";
|
||||
|
||||
// Tâche d'extraction Qlik potentiellement très longue (hypercube paginé).
|
||||
// On accepte quand même 5 min côté plateforme Next.js, mais on rend la main au
|
||||
@@ -99,6 +105,68 @@ async function requireAdmin(): Promise<NextResponse | null> {
|
||||
return null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Regex de validation d'un code centrale "raisonnable".
|
||||
*
|
||||
* Format attendu d'après l'API FF : préfixe `10000` + 6 chiffres (ex `10000167303`),
|
||||
* donc 11 caractères alphanumériques. Pour rester large et ne pas casser
|
||||
* d'éventuels codes alphanumériques fournisseur (refs internes), on autorise
|
||||
* - longueur 2..30
|
||||
* - jeu `[A-Z0-9_-]`
|
||||
* - premier caractère alphanumérique (pas `_` ni `-`)
|
||||
*
|
||||
* But : éviter d'envoyer à Qlik des chaînes vides, des `-`, des espaces
|
||||
* parasites, ou des entrées manifestement invalides qui déclenchent des
|
||||
* codes d'erreur Engine ou du gaspillage réseau. **Volontairement simple** :
|
||||
* on ne touche pas au mapping ; on filtre seulement ce qui ne ressemble à
|
||||
* aucun code plausible.
|
||||
*/
|
||||
const CENTRAL_CODE_REGEX = /^[A-Z0-9][A-Z0-9_-]{1,29}$/i;
|
||||
|
||||
/**
|
||||
* Filtre strict des codes centraux avant d'envoyer à Qlik.
|
||||
* - trim + drop vides
|
||||
* - drop "-" (placeholder déjà exclu côté agrégation mais on double-check)
|
||||
* - drop doublons (préserve l'ordre, comparaison insensible à la casse)
|
||||
* - drop codes ne matchant pas CENTRAL_CODE_REGEX (longueur / charset)
|
||||
*
|
||||
* Renvoie `{ accepted, rejected, deduped }`. Les logs n'exposent que les
|
||||
* compteurs, jamais les valeurs des codes.
|
||||
*/
|
||||
function filterCentralCodes(rawCodes: Array<string | null | undefined>): {
|
||||
accepted: string[];
|
||||
rejected: number;
|
||||
deduped: number;
|
||||
} {
|
||||
const seen = new Set<string>();
|
||||
const accepted: string[] = [];
|
||||
let rejected = 0;
|
||||
let deduped = 0;
|
||||
for (const raw of rawCodes) {
|
||||
if (raw == null) {
|
||||
rejected++;
|
||||
continue;
|
||||
}
|
||||
const code = String(raw).trim();
|
||||
const key = code.toUpperCase();
|
||||
if (!code || code === "-" || code === "_") {
|
||||
rejected++;
|
||||
continue;
|
||||
}
|
||||
if (seen.has(key)) {
|
||||
deduped++;
|
||||
continue;
|
||||
}
|
||||
if (!CENTRAL_CODE_REGEX.test(code)) {
|
||||
rejected++;
|
||||
continue;
|
||||
}
|
||||
seen.add(key);
|
||||
accepted.push(code);
|
||||
}
|
||||
return { accepted, rejected, deduped };
|
||||
}
|
||||
|
||||
/**
|
||||
* Exécute réellement l'extraction + l'upsert, en mettant à jour le job
|
||||
* au fur et à mesure (compteur fetched, puis upserted, puis finishedAt).
|
||||
@@ -106,30 +174,31 @@ async function requireAdmin(): Promise<NextResponse | null> {
|
||||
async function runJob(job: QlikSyncJob): Promise<void> {
|
||||
try {
|
||||
const articles = await pgGetArticlesByFournisseur(job.fournisseur);
|
||||
const codes = [
|
||||
...new Set(
|
||||
articles
|
||||
.map((a) => (a.codeCentrale ? String(a.codeCentrale).trim() : ""))
|
||||
.filter(Boolean),
|
||||
),
|
||||
];
|
||||
// Filtre strict des codes (trim, dedupe, exclusion vides/`-`/codes trop suspects).
|
||||
const { accepted: codes, rejected: codesRejected, deduped: codesDeduped } = filterCentralCodes(
|
||||
articles.map((a) => a.codeCentrale),
|
||||
);
|
||||
job.requested = codes.length;
|
||||
console.log(
|
||||
`[api/qlik/sync] job=${job.jobId} fournisseur=${job.fournisseur} — ${articles.length} articles, ${codes.length} codes centraux uniques. Échantillon: ${JSON.stringify(codes.slice(0, 5))}`,
|
||||
`[api/qlik/sync] job=${job.jobId} fournisseur=${job.fournisseur} — ${articles.length} articles, ${codes.length} codes centraux retenus, ${codesRejected} rejetés (vide/-/anormaux), ${codesDeduped} doublons supprimés`,
|
||||
);
|
||||
if (codes.length === 0) {
|
||||
job.status = "success";
|
||||
job.message = "Aucun article avec code centrale pour ce fournisseur";
|
||||
job.message = "Aucun code centrale valide pour ce fournisseur";
|
||||
job.finishedAt = new Date().toISOString();
|
||||
return;
|
||||
}
|
||||
|
||||
const dateFilter = buildGridNetworkQlikDateFilter();
|
||||
// Fenêtre temporelle alignée sur la grille, éventuellement raccourcie via
|
||||
// QLIK_SYNC_MONTHS_BACK (1..12, défaut 12). On logue la valeur effective
|
||||
// pour audit.
|
||||
const monthsBack = envMonthsBack("QLIK_SYNC_MONTHS_BACK", QLIK_MONTHS_BACK_DEFAULT);
|
||||
const dateFilter = buildGridNetworkQlikDateFilter(new Date(), monthsBack);
|
||||
job.periode = dateFilter.label;
|
||||
job.dateDebut = dateFilter.dateDebut;
|
||||
job.dateFin = dateFilter.dateFin;
|
||||
console.log(
|
||||
`[api/qlik/sync] job=${job.jobId} → extraction Qlik pour ${codes.length} codes — période ${dateFilter.label} (${dateFilter.dateDebut} → ${dateFilter.dateFin})…`,
|
||||
`[api/qlik/sync] job=${job.jobId} → extraction Qlik pour ${codes.length} codes — fenêtre ${dateFilter.label} (${dateFilter.dateDebut} → ${dateFilter.dateFin}, QLIK_SYNC_MONTHS_BACK=${monthsBack}, bornes ${QLIK_MONTHS_BACK_MIN}..${QLIK_MONTHS_BACK_MAX})…`,
|
||||
);
|
||||
const metrics = await fetchNetworkMetricsPlaywright(codes, undefined, dateFilter);
|
||||
job.fetched = metrics.size;
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
* CollectFlow — Helper de fenêtre temporelle pour Qlik réseau.
|
||||
*
|
||||
* Aligne la timeline d'extraction Qlik réseau sur la timeline de la grille :
|
||||
* - 12 mois complets glissants
|
||||
* - `monthsBack` mois complets glissants (défaut : 12)
|
||||
* - exclut systématiquement le mois courant
|
||||
*
|
||||
* Référence utilisée : `buildLast12MonthsRange()` de `api-ff-client.ts`
|
||||
@@ -33,6 +33,34 @@
|
||||
* Pas de secrets ici. Pure fonction utilitaire testable en CLI Node.
|
||||
*/
|
||||
|
||||
/**
|
||||
* Bornes acceptées pour `monthsBack`. On reste sur 1..12 pour ne pas casser
|
||||
* la sémantique "12 mois glissants alignés sur la grille" : une valeur plus
|
||||
* grande n'apporterait pas de signal réseau supplémentaire (la grille
|
||||
* n'expose elle-même que ~12 mois). En-deçà de 1 mois, la fenêtre serait
|
||||
* vide (mois courant exclu) ou incohérente.
|
||||
*/
|
||||
export const QLIK_MONTHS_BACK_MIN = 1;
|
||||
export const QLIK_MONTHS_BACK_MAX = 12;
|
||||
export const QLIK_MONTHS_BACK_DEFAULT = 12;
|
||||
|
||||
/**
|
||||
* Lit `process.env[name]` et renvoie un entier borné entre `min` et `max`.
|
||||
* Renvoie `defaultValue` si la variable est absente / vide / non numérique.
|
||||
*
|
||||
* Volontairement exportée pour être réutilisée par `qlik-playwright.ts` et
|
||||
* `sync/route.ts` (cohérence avec `envNumber` côté Playwright, mais en pur
|
||||
* Node sans dépendance Next.js).
|
||||
*/
|
||||
export function envMonthsBack(name: string, defaultValue: number = QLIK_MONTHS_BACK_DEFAULT): number {
|
||||
const raw = process.env[name];
|
||||
if (raw == null || raw.trim() === "") return defaultValue;
|
||||
const parsed = Number(raw);
|
||||
if (!Number.isFinite(parsed)) return defaultValue;
|
||||
const clamped = Math.min(QLIK_MONTHS_BACK_MAX, Math.max(QLIK_MONTHS_BACK_MIN, Math.trunc(parsed)));
|
||||
return clamped;
|
||||
}
|
||||
|
||||
/** Epoch Qlik = 1899-12-30 (équivalent Excel/Lotus 1-2-3). */
|
||||
const QLIK_EPOCH_MS = Date.UTC(1899, 11, 30); // 1899-12-30T00:00:00Z
|
||||
|
||||
@@ -91,17 +119,30 @@ export interface QlikDateFilter {
|
||||
}
|
||||
|
||||
/**
|
||||
* Calcule la fenêtre 12 mois complets (excluant mois courant).
|
||||
* Par défaut, utilise `now = new Date()`. Testable via injection de `now`.
|
||||
* Calcule la fenêtre `monthsBack` mois complets (excluant mois courant).
|
||||
* Par défaut, utilise `now = new Date()` et `monthsBack = 12`.
|
||||
*
|
||||
* Logique alignée sur `buildLast12MonthsRange()` de `src/lib/api-ff-client.ts`.
|
||||
* Si la source officielle change, mettre à jour ici aussi (garder en sync).
|
||||
* Logique alignée sur `buildLast12MonthsRange()` de `src/lib/api-ff-client.ts`
|
||||
* (défaut = 12 mois). Si la source officielle change, mettre à jour ici aussi
|
||||
* (garder en sync).
|
||||
*
|
||||
* Bornes : `monthsBack` est clampé dans `[QLIK_MONTHS_BACK_MIN, QLIK_MONTHS_BACK_MAX]`
|
||||
* (1..12). Une valeur invalide (NaN, ≤ 0, > 12) est ramenée aux bornes sans
|
||||
* exception — on veut que la sync reste possible même si l'env est mal
|
||||
* configuré.
|
||||
*/
|
||||
export function buildGridNetworkQlikDateFilter(now: Date = new Date()): QlikDateFilter {
|
||||
export function buildGridNetworkQlikDateFilter(
|
||||
now: Date = new Date(),
|
||||
monthsBack: number = QLIK_MONTHS_BACK_DEFAULT,
|
||||
): QlikDateFilter {
|
||||
const months = Math.min(
|
||||
QLIK_MONTHS_BACK_MAX,
|
||||
Math.max(QLIK_MONTHS_BACK_MIN, Math.trunc(Number.isFinite(monthsBack) ? monthsBack : QLIK_MONTHS_BACK_DEFAULT)),
|
||||
);
|
||||
// Fin : dernier jour du mois précédent le mois courant
|
||||
const endMonth = new Date(now.getFullYear(), now.getMonth(), 0);
|
||||
// Début : 12 mois avant le mois courant (1er du mois)
|
||||
const startMonth = new Date(now.getFullYear(), now.getMonth() - 12, 1);
|
||||
// Début : `months` mois avant le mois courant (1er du mois)
|
||||
const startMonth = new Date(now.getFullYear(), now.getMonth() - months, 1);
|
||||
|
||||
const dateDebut = startMonth.toISOString().slice(0, 10);
|
||||
const dateFin = endMonth.toISOString().slice(0, 10);
|
||||
|
||||
+184
-60
@@ -31,7 +31,47 @@ async function getBrowser(): Promise<Browser> {
|
||||
return browserPromise;
|
||||
}
|
||||
|
||||
interface InPageResult { ok: boolean; rows?: Array<Array<string | number>>; size?: number; error?: string; diag?: Record<string, unknown>; }
|
||||
interface InPageResult {
|
||||
ok: boolean;
|
||||
rows?: Array<Array<string | number>>;
|
||||
size?: number;
|
||||
error?: string;
|
||||
diag?: Record<string, unknown>;
|
||||
// Compteurs agrégés pour le résumé timing final côté Node (cf. qlik-playwright).
|
||||
months?: number;
|
||||
batches?: number;
|
||||
code15Retries?: number;
|
||||
fallbackCount?: number;
|
||||
}
|
||||
|
||||
function envNumber(name: string, defaultValue: number, min: number, max: number): number {
|
||||
const raw = process.env[name];
|
||||
if (raw == null || raw.trim() === "") return defaultValue;
|
||||
const parsed = Number(raw);
|
||||
if (!Number.isFinite(parsed)) return defaultValue;
|
||||
return Math.min(max, Math.max(min, Math.trunc(parsed)));
|
||||
}
|
||||
|
||||
function envNumberCsv(name: string, min: number, max: number): number[] {
|
||||
const raw = process.env[name];
|
||||
if (!raw) return [];
|
||||
return raw
|
||||
.split(",")
|
||||
.map((part) => Number(part.trim()))
|
||||
.filter((value) => Number.isFinite(value))
|
||||
.map((value) => Math.min(max, Math.max(min, Math.trunc(value))));
|
||||
}
|
||||
|
||||
function uniqueOrdered(values: number[]): number[] {
|
||||
const seen = new Set<number>();
|
||||
const out: number[] = [];
|
||||
for (const value of values) {
|
||||
if (seen.has(value)) continue;
|
||||
seen.add(value);
|
||||
out.push(value);
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
export async function fetchNetworkMetricsPlaywright(
|
||||
codeCentraux?: string[],
|
||||
@@ -41,6 +81,8 @@ export async function fetchNetworkMetricsPlaywright(
|
||||
if (!cfg.appNetwork) throw new Error("[qlik-pw] QLIK_APP_NETWORK manquant");
|
||||
if (!cfg.user || !cfg.password) throw new Error("[qlik-pw] identifiants Qlik manquants");
|
||||
|
||||
// Démarre l'horloge pour le résumé timing final (cf. bloc [qlik-pw][summary]).
|
||||
const totalStart = Date.now();
|
||||
console.log(`[qlik-pw] host=${cfg.host} app=${cfg.appNetwork} — ${(codeCentraux ?? []).length} codes à sélectionner`);
|
||||
if (dateFilter) {
|
||||
console.log(`[qlik-pw] filtre Date=${dateFilter.label} (${dateFilter.dateDebut} → ${dateFilter.dateFin}, ${dateFilter.setAnalysis})`);
|
||||
@@ -49,6 +91,12 @@ export async function fetchNetworkMetricsPlaywright(
|
||||
console.log(`[qlik-pw] session NTLM OK, cookie=${sess.cookie.split("=")[0]}=…`);
|
||||
const [cookieName, ...rest] = sess.cookie.split("=");
|
||||
const cookieValue = rest.join("=");
|
||||
const qlikSettleMs = envNumber("QLIK_SETTLE_MS", 150, 0, 2000);
|
||||
const qlikCode15MaxAttempts = envNumber("QLIK_CODE15_MAX_ATTEMPTS", 4, 1, 8);
|
||||
const qlikArticleBatchSize = envNumber("QLIK_ARTICLE_BATCH_SIZE", 150, 1, 500);
|
||||
const csvFallbacks = envNumberCsv("QLIK_ARTICLE_BATCH_FALLBACKS", 1, 500);
|
||||
const qlikArticleBatchFallbacks = uniqueOrdered([qlikArticleBatchSize, ...(csvFallbacks.length ? csvFallbacks : [75, 50]), 75, 50]);
|
||||
console.log(`[qlik-pw] tuning settle=${qlikSettleMs}ms code15Attempts=${qlikCode15MaxAttempts} batchFallbacks=${qlikArticleBatchFallbacks.join(">")}`);
|
||||
|
||||
const browser = await getBrowser();
|
||||
console.log(`[qlik-pw] chromium lancé`);
|
||||
@@ -89,11 +137,14 @@ export async function fetchNetworkMetricsPlaywright(
|
||||
}));
|
||||
|
||||
const result = (await page.evaluate(
|
||||
({ app, dim, mca, mqte, mnb, mcamag, mmarge, codes, token, monthlyPayload, noDateMode }: {
|
||||
({ app, dim, mca, mqte, mnb, mcamag, mmarge, codes, token, monthlyPayload, noDateMode, qlikSettleMs, qlikCode15MaxAttempts, qlikArticleBatchFallbacks }: {
|
||||
app: string; dim: string; mca: string; mqte: string; mnb: string; mcamag: string; mmarge: string;
|
||||
codes: string[]; token: string;
|
||||
monthlyPayload: Array<{ label: string; dateDebut: string; dateFin: string; serials: number[] }>;
|
||||
noDateMode: boolean;
|
||||
qlikSettleMs: number;
|
||||
qlikCode15MaxAttempts: number;
|
||||
qlikArticleBatchFallbacks: number[];
|
||||
}) =>
|
||||
new Promise<InPageResult>((resolve) => {
|
||||
const loc = (window as unknown as { location: Location }).location;
|
||||
@@ -110,7 +161,12 @@ export async function fetchNetworkMetricsPlaywright(
|
||||
ws.onmessage = (ev: MessageEvent) => {
|
||||
const m = JSON.parse(ev.data as string);
|
||||
if (m.method === "OnConnected") return res();
|
||||
if (m.id && pend.has(m.id)) { const x = pend.get(m.id)!; pend.delete(m.id); m.error ? x.rej(new Error(JSON.stringify(m.error))) : x.res(m.result); }
|
||||
if (m.id && pend.has(m.id)) {
|
||||
const x = pend.get(m.id)!;
|
||||
pend.delete(m.id);
|
||||
if (m.error) x.rej(new Error(JSON.stringify(m.error)));
|
||||
else x.res(m.result);
|
||||
}
|
||||
};
|
||||
ws.onerror = () => rej(new Error("ws error (403 ?)"));
|
||||
setTimeout(() => rej(new Error("ws timeout")), 30000);
|
||||
@@ -124,22 +180,34 @@ export async function fetchNetworkMetricsPlaywright(
|
||||
return /"code"\s*:\s*15\b/.test(msg);
|
||||
};
|
||||
const sleep = (ms: number) => new Promise<void>((r) => setTimeout(r, ms));
|
||||
const nowMs = () => (typeof performance !== "undefined" ? performance.now() : Date.now());
|
||||
const timed = async <T>(etape: string, method: string, fn: () => Promise<T>, timings?: Record<string, number>): Promise<{ value: T; ms: number }> => {
|
||||
const start = nowMs();
|
||||
const value = await fn();
|
||||
const ms = Math.round(nowMs() - start);
|
||||
const label = `${etape}:${method}`;
|
||||
if (timings) timings[label] = (timings[label] ?? 0) + ms;
|
||||
console.log(`[qlik-pw][timing] ${label} ms=${ms}`);
|
||||
return { value, ms };
|
||||
};
|
||||
const timingSummary = (timings: Record<string, number>) => JSON.stringify(timings);
|
||||
const rpcWithRetry = async (
|
||||
etape: string,
|
||||
method: string,
|
||||
params: unknown,
|
||||
handle: number,
|
||||
maxAttempts = 4,
|
||||
onCode15Retry?: () => void,
|
||||
): Promise<{ [k: string]: unknown }> => {
|
||||
let lastErr: unknown;
|
||||
for (let attempt = 1; attempt <= maxAttempts; attempt++) {
|
||||
for (let attempt = 1; attempt <= qlikCode15MaxAttempts; attempt++) {
|
||||
try {
|
||||
return await rpc(method, params, handle);
|
||||
} catch (e) {
|
||||
lastErr = e;
|
||||
if (!isEngineCode15(e) || attempt === maxAttempts) throw e;
|
||||
const backoff = 150 * attempt;
|
||||
console.log(`[qlik-pw] retry code15 étape=${etape} méthode=${method} tentative=${attempt}/${maxAttempts} backoff=${backoff}ms`);
|
||||
if (!isEngineCode15(e) || attempt === qlikCode15MaxAttempts) throw e;
|
||||
onCode15Retry?.();
|
||||
const backoff = qlikSettleMs * attempt;
|
||||
console.log(`[qlik-pw] retry code15 étape=${etape} méthode=${method} tentative=${attempt}/${qlikCode15MaxAttempts} backoff=${backoff}ms`);
|
||||
await sleep(backoff);
|
||||
}
|
||||
}
|
||||
@@ -180,6 +248,11 @@ export async function fetchNetworkMetricsPlaywright(
|
||||
// causé par la réutilisation du même objet sur des centaines de recalculs.
|
||||
|
||||
const out: Array<Array<string | number>> = [];
|
||||
// Compteurs agrégés pour le résumé timing final (loggué côté Node après la sync).
|
||||
let aggMonths = 0;
|
||||
let aggBatches = 0;
|
||||
let aggCode15Retries = 0;
|
||||
let aggFallbackCount = 0;
|
||||
|
||||
// Sélectionne Date + Article Code puis pagine le cube. Renvoie le nb de lignes
|
||||
// brutes lues (avant dédoublonnage côté Node).
|
||||
@@ -190,116 +263,155 @@ export async function fetchNetworkMetricsPlaywright(
|
||||
const PAGE = 1400;
|
||||
// Petite latence après chaque mutation de sélection : laisser l'Engine
|
||||
// finaliser l'évaluation de la sélection avant la prochaine requête layout/cube.
|
||||
const SETTLE_MS = 150;
|
||||
const createCubeForBatch = async (): Promise<{ handle: number; id: string }> => {
|
||||
const obj = await rpcWithRetry("createCube", "CreateSessionObject", { qProp: { qInfo: { qType: "cf-net" }, qHyperCubeDef: {
|
||||
const createCubeForBatch = async (timings?: Record<string, number>, onCode15Retry?: () => void): Promise<{ handle: number; id: string }> => {
|
||||
const { value: obj } = await timed("createCube", "CreateSessionObject", () => rpcWithRetry("createCube", "CreateSessionObject", { qProp: { qInfo: { qType: "cf-net" }, qHyperCubeDef: {
|
||||
qDimensions: [{ qLibraryId: dim }],
|
||||
qMeasures: [{ qLibraryId: mca }, { qLibraryId: mqte }, { qLibraryId: mnb }, { qLibraryId: rCamag }, { qLibraryId: rMarge }],
|
||||
qInitialDataFetch: [],
|
||||
} } }, doc);
|
||||
qInitialDataFetch: [{ qTop: 0, qLeft: 0, qWidth: 6, qHeight: PAGE }],
|
||||
} } }, doc, onCode15Retry), timings);
|
||||
const qReturn = obj.qReturn as { qHandle: number; qGenericId?: string; qId?: string };
|
||||
return { handle: qReturn.qHandle, id: String(qReturn.qGenericId ?? qReturn.qId ?? "") };
|
||||
};
|
||||
const destroyCube = async (cube: { handle: number; id: string }) => {
|
||||
const destroyCube = async (cube: { handle: number; id: string }, timings?: Record<string, number>) => {
|
||||
if (!cube.id) return;
|
||||
try {
|
||||
await rpcWithRetry("destroyCube", "DestroySessionObject", { qId: cube.id }, doc);
|
||||
await timed("destroyCube", "DestroySessionObject", () => rpcWithRetry("destroyCube", "DestroySessionObject", { qId: cube.id }, doc), timings);
|
||||
} catch (e) {
|
||||
// Non fatal : l'objet est de toute façon scopé à la session websocket.
|
||||
// Un code 15 ici signifie que l'Engine l'a déjà libéré après un abort.
|
||||
console.log("[qlik-pw] DestroySessionObject ignoré: " + String((e as Error)?.message || e));
|
||||
}
|
||||
};
|
||||
const fetchCubeForSelection = async (sink: Array<Array<string | number>>): Promise<number> => {
|
||||
const cube = await createCubeForBatch();
|
||||
const appendMatrix = (matrix: Array<Array<{ qText?: string; qNum?: number }>>, sink: Array<Array<string | number>>): number => {
|
||||
for (const r of matrix) sink.push([
|
||||
String(r[0]?.qText ?? ""),
|
||||
Number(r[1]?.qNum) || 0,
|
||||
Number(r[2]?.qNum) || 0,
|
||||
Number(r[3]?.qNum) || 0,
|
||||
Number(r[4]?.qNum) || 0,
|
||||
Number(r[5]?.qNum) || 0,
|
||||
]);
|
||||
return matrix.length;
|
||||
};
|
||||
const fetchCubeForSelection = async (sink: Array<Array<string | number>>, timings?: Record<string, number>, onCode15Retry?: () => void): Promise<number> => {
|
||||
const cube = await createCubeForBatch(timings, onCode15Retry);
|
||||
const oh = cube.handle;
|
||||
try {
|
||||
const layout = await rpcWithRetry("getLayout", "GetLayout", {}, oh);
|
||||
const size = ((layout.qLayout as { qHyperCube: { qSize: { qcy: number } } }).qHyperCube).qSize.qcy;
|
||||
const { value: layout } = await timed("layout", "GetLayout", () => rpcWithRetry("layout", "GetLayout", {}, oh, onCode15Retry), timings);
|
||||
const hyperCube = (layout.qLayout as { qHyperCube: { qSize: { qcy: number }; qDataPages?: Array<{ qArea?: { qTop?: number }; qMatrix?: Array<Array<{ qText?: string; qNum?: number }>> }> } }).qHyperCube;
|
||||
const size = hyperCube.qSize.qcy;
|
||||
const firstPage = hyperCube.qDataPages?.find((page) => (page.qArea?.qTop ?? 0) === 0);
|
||||
let got = 0;
|
||||
for (let top = 0; top < size; top += PAGE) {
|
||||
const d = await rpcWithRetry("getHyperCubeData", "GetHyperCubeData", { qPath: "/qHyperCubeDef", qPages: [{ qTop: top, qLeft: 0, qWidth: 6, qHeight: PAGE }] }, oh);
|
||||
let top = 0;
|
||||
if (firstPage?.qMatrix?.length) {
|
||||
got += appendMatrix(firstPage.qMatrix, sink);
|
||||
top = PAGE;
|
||||
}
|
||||
for (; top < size; top += PAGE) {
|
||||
const { value: d } = await timed("getCubeData", "GetHyperCubeData", () => rpcWithRetry("getCubeData", "GetHyperCubeData", { qPath: "/qHyperCubeDef", qPages: [{ qTop: top, qLeft: 0, qWidth: 6, qHeight: PAGE }] }, oh, onCode15Retry), timings);
|
||||
const matrix = (d.qDataPages as Array<{ qMatrix?: Array<Array<{ qText?: string; qNum?: number }>> }>)?.[0]?.qMatrix ?? [];
|
||||
if (!matrix.length) break;
|
||||
for (const r of matrix) sink.push([
|
||||
String(r[0]?.qText ?? ""),
|
||||
Number(r[1]?.qNum) || 0,
|
||||
Number(r[2]?.qNum) || 0,
|
||||
Number(r[3]?.qNum) || 0,
|
||||
Number(r[4]?.qNum) || 0,
|
||||
Number(r[5]?.qNum) || 0,
|
||||
]);
|
||||
got += matrix.length;
|
||||
got += appendMatrix(matrix, sink);
|
||||
if (matrix.length < PAGE) break;
|
||||
}
|
||||
return got;
|
||||
} finally {
|
||||
await destroyCube(cube);
|
||||
await destroyCube(cube, timings);
|
||||
}
|
||||
};
|
||||
|
||||
if (noDateMode) {
|
||||
// Aucun filtre Date : extraction en un seul appel (tous les codes d'un coup).
|
||||
// On sélectionne uniquement Article Code ; Date reste libre.
|
||||
const timings: Record<string, number> = {};
|
||||
let code15Retries = 0;
|
||||
console.log("mode sans dateFilter — sélection de " + codes.length + " codes en une fois");
|
||||
if (codes.length && fh !== -1) {
|
||||
const sel = await rpc("SelectValues", { qFieldValues: codes.map((c) => ({ qText: c })), qToggleMode: false, qSoftLock: true }, fh);
|
||||
const { value: sel } = await timed("selectCode", "SelectValues", () => rpc("SelectValues", { qFieldValues: codes.map((c) => ({ qText: c })), qToggleMode: false, qSoftLock: true }, fh), timings);
|
||||
console.log("SelectValues Article Code → " + JSON.stringify(sel.qReturn));
|
||||
}
|
||||
const got = await fetchCubeForSelection(out);
|
||||
console.log("extraction sans date — " + got + " ligne(s), cumul=" + out.length);
|
||||
const batchRows: Array<Array<string | number>> = [];
|
||||
const { value: got, ms } = await timed("cube", "batch", () => fetchCubeForSelection(batchRows, timings, () => { code15Retries++; }), timings);
|
||||
out.push(...batchRows);
|
||||
aggMonths = 1;
|
||||
aggBatches = 1;
|
||||
aggCode15Retries = code15Retries;
|
||||
console.log("[qlik-pw][timing] noDate rows=" + got + " cumul=" + out.length + " ms=" + ms + " retries15=" + code15Retries + " timings=" + timingSummary(timings));
|
||||
} else {
|
||||
// Itération par mois × lots de codes : pour chaque mois, SelectValues Date
|
||||
// (~30 serials), puis SelectValues Article Code par petits lots. Le cube
|
||||
// reste borné à ~30 jours × 50 codes, sans lire tout le catalogue Qlik.
|
||||
// reste borné à ~30 jours × batch contrôlé, sans lire tout le catalogue Qlik.
|
||||
const total = monthlyPayload.length;
|
||||
const ARTICLE_BATCH = 50;
|
||||
const codeBatches: string[][] = [];
|
||||
if (codes.length) {
|
||||
for (let i = 0; i < codes.length; i += ARTICLE_BATCH) codeBatches.push(codes.slice(i, i + ARTICLE_BATCH));
|
||||
} else {
|
||||
// Comportement historique si aucun code n'est fourni : lire le mois complet.
|
||||
codeBatches.push([]);
|
||||
}
|
||||
console.log("itération par mois — " + total + " mois × " + codeBatches.length + " lot(s) de ≤" + ARTICLE_BATCH + " codes");
|
||||
const fallbackSizes = qlikArticleBatchFallbacks.length ? qlikArticleBatchFallbacks : [150, 75, 50];
|
||||
const fallbackLabel = fallbackSizes.join(">");
|
||||
const estimatedBatches = codes.length ? Math.ceil(codes.length / fallbackSizes[0]) : 1;
|
||||
console.log("itération par mois — " + total + " mois × ~" + estimatedBatches + " lot(s), batchFallbacks=" + fallbackLabel);
|
||||
for (let mi = 0; mi < total; mi++) {
|
||||
const m = monthlyPayload[mi];
|
||||
const prefix = "[qlik-pw] mois " + (mi + 1) + "/" + total + " " + m.label + " (" + m.dateDebut + "→" + m.dateFin + ", " + m.serials.length + "j)";
|
||||
const monthStart = nowMs();
|
||||
const monthTimings: Record<string, number> = {};
|
||||
aggMonths = mi + 1;
|
||||
try {
|
||||
// 1) Sélection Date pour le mois (un seul SelectValues, ~30 valeurs).
|
||||
const dsel = await rpc("SelectValues", {
|
||||
const { value: dsel } = await timed("selectDate", "SelectValues", () => rpc("SelectValues", {
|
||||
qFieldValues: m.serials.map((s) => ({ qNum: s, qText: String(s) })),
|
||||
qToggleMode: false,
|
||||
qSoftLock: true,
|
||||
}, dfh);
|
||||
}, dfh), monthTimings);
|
||||
console.log(prefix + " — SelectValues Date " + m.serials.length + " serials → " + JSON.stringify(dsel.qReturn));
|
||||
// Laisser l'Engine finaliser l'évaluation de la sélection Date
|
||||
// avant la première SelectValues Article Code du 1er lot.
|
||||
await sleep(SETTLE_MS);
|
||||
await sleep(qlikSettleMs);
|
||||
|
||||
// 2) Sélection Article Code par lots contrôlés.
|
||||
for (let bi = 0; bi < codeBatches.length; bi++) {
|
||||
const batch = codeBatches[bi];
|
||||
const batchPrefix = prefix + " — lot " + (bi + 1) + "/" + codeBatches.length + " (" + batch.length + " codes)";
|
||||
let codeIndex = 0;
|
||||
let batchNumber = 0;
|
||||
let fallbackIndex = 0;
|
||||
const totalCodes = codes.length || 0;
|
||||
while (codes.length ? codeIndex < totalCodes : batchNumber === 0) {
|
||||
const batchSize = codes.length ? fallbackSizes[fallbackIndex] : 0;
|
||||
const batch = codes.length ? codes.slice(codeIndex, Math.min(codeIndex + batchSize, totalCodes)) : [];
|
||||
const batchStart = codeIndex;
|
||||
const batchEnd = codeIndex + batch.length;
|
||||
const timings: Record<string, number> = {};
|
||||
let code15Retries = 0;
|
||||
batchNumber++;
|
||||
const batchPrefix = prefix + " — lot " + batchNumber + " [" + batchStart + "," + batchEnd + ") size=" + batch.length;
|
||||
try {
|
||||
if (fh !== -1) {
|
||||
await rpc("Clear", {}, fh);
|
||||
await timed("clearCode", "Clear", () => rpc("Clear", {}, fh), timings);
|
||||
// Laisser l'Engine appliquer le Clear avant la nouvelle sélection.
|
||||
await sleep(SETTLE_MS);
|
||||
const csel = await rpc("SelectValues", {
|
||||
await sleep(qlikSettleMs);
|
||||
const { value: csel } = await timed("selectCode", "SelectValues", () => rpc("SelectValues", {
|
||||
qFieldValues: batch.map((c) => ({ qText: c })),
|
||||
qToggleMode: false,
|
||||
qSoftLock: true,
|
||||
}, fh);
|
||||
}, fh), timings);
|
||||
console.log(batchPrefix + " — SelectValues Article Code → " + JSON.stringify(csel.qReturn));
|
||||
// Laisser l'Engine finaliser l'évaluation de la sélection
|
||||
// avant de créer / interroger le cube (évite code 15 sur le 1er lot).
|
||||
await sleep(SETTLE_MS);
|
||||
await sleep(qlikSettleMs);
|
||||
}
|
||||
const got = await fetchCubeForSelection(out);
|
||||
console.log(batchPrefix + " — " + got + " ligne(s), cumul=" + out.length);
|
||||
const batchRows: Array<Array<string | number>> = [];
|
||||
const { value: got, ms } = await timed("cube", "batch", () => fetchCubeForSelection(batchRows, timings, () => { code15Retries++; }), timings);
|
||||
out.push(...batchRows);
|
||||
aggBatches++;
|
||||
aggCode15Retries += code15Retries;
|
||||
console.log("[qlik-pw][timing] " + batchPrefix + " — rows=" + got + " cumul=" + out.length + " ms=" + ms + " retries15=" + code15Retries + " timings=" + timingSummary(timings));
|
||||
if (!codes.length) break;
|
||||
codeIndex = batchEnd;
|
||||
fallbackIndex = 0;
|
||||
} catch (batchErr) {
|
||||
const msg = String((batchErr as Error)?.message || batchErr);
|
||||
const nextFallbackIndex = fallbackIndex + 1;
|
||||
const nextSize = fallbackSizes[nextFallbackIndex];
|
||||
if (codes.length && isEngineCode15(batchErr) && nextSize && batch.length > nextSize) {
|
||||
console.log("[qlik-pw][timing] " + batchPrefix + " — fallback code15 " + batch.length + "→" + nextSize + " retries15=" + code15Retries + " timings=" + timingSummary(timings));
|
||||
fallbackIndex = nextFallbackIndex;
|
||||
aggFallbackCount++;
|
||||
continue;
|
||||
}
|
||||
console.error(batchPrefix + " — ABORT — " + msg);
|
||||
throw new Error(batchPrefix + " — " + msg);
|
||||
}
|
||||
@@ -312,17 +424,18 @@ export async function fetchNetworkMetricsPlaywright(
|
||||
// 3) Clear Date + Article Code avant le mois suivant (soulage l'Engine).
|
||||
if (mi < total - 1) {
|
||||
try {
|
||||
await rpc("Clear", {}, dfh);
|
||||
if (fh !== -1) await rpc("Clear", {}, fh);
|
||||
await timed("clearDate", "Clear", () => rpc("Clear", {}, dfh), monthTimings);
|
||||
if (fh !== -1) await timed("clearCodeMonth", "Clear", () => rpc("Clear", {}, fh), monthTimings);
|
||||
} catch (clearErr) {
|
||||
console.log(prefix + " — Clear ignoré: " + String((clearErr as Error)?.message || clearErr));
|
||||
}
|
||||
}
|
||||
console.log("[qlik-pw][timing] " + prefix + " — done ms=" + Math.round(nowMs() - monthStart) + " cumul=" + out.length + " timings=" + timingSummary(monthTimings));
|
||||
}
|
||||
}
|
||||
|
||||
ws.close();
|
||||
resolve({ ok: true, rows: out, size: out.length });
|
||||
resolve({ ok: true, rows: out, size: out.length, months: aggMonths, batches: aggBatches, code15Retries: aggCode15Retries, fallbackCount: aggFallbackCount });
|
||||
} catch (e) { try { ws.close(); } catch { /* noop */ } fail(String((e as Error)?.message || e)); }
|
||||
})();
|
||||
}),
|
||||
@@ -338,11 +451,22 @@ export async function fetchNetworkMetricsPlaywright(
|
||||
token: csrfToken,
|
||||
monthlyPayload,
|
||||
noDateMode: !dateFilter,
|
||||
qlikSettleMs,
|
||||
qlikCode15MaxAttempts,
|
||||
qlikArticleBatchFallbacks,
|
||||
},
|
||||
)) as InPageResult;
|
||||
|
||||
if (!result.ok) throw new Error(`[qlik-pw] ${result.error}`);
|
||||
|
||||
// Résumé timing final : synthèse compacte de la sync pour audit/debug.
|
||||
// Les logs détaillés par étape restent en place (timings batch/mois) ;
|
||||
// ici on consolide les compteurs agrégés et la durée totale.
|
||||
const totalMs = Date.now() - totalStart;
|
||||
console.log(
|
||||
`[qlik-pw][summary] rows=${result.size ?? 0} months=${result.months ?? 0} batches=${result.batches ?? 0} code15Retries=${result.code15Retries ?? 0} fallbackCount=${result.fallbackCount ?? 0} totalMs=${totalMs}`,
|
||||
);
|
||||
|
||||
const wanted = codeCentraux ? new Set(codeCentraux) : null;
|
||||
const out = new Map<string, NetworkMetric>();
|
||||
const margeWeight = new Map<string, number>();
|
||||
|
||||
Reference in new issue
Block a user