Michael
091caad347
feat: add web scraping and search services using Playwright with anti-detection mechanisms.
2025-11-29 01:56:25 +01:00
Michael
e6bf46659d
refactor: Improve browser lifecycle management in ScraperService to prevent connection errors
2025-11-29 01:39:43 +01:00
Michael
0439b74bc3
fix: Remove duplicate 'finally' block in scraper_service.py causing SyntaxError
2025-11-29 01:24:34 +01:00
Michael
405756963a
fix: Resolve IndentationError in scraper_service.py caused by loop removal
2025-11-29 01:20:58 +01:00
Michael
aa9d2af5b2
feat: Implement hybrid proxy strategy for Amazon scraping - fallback to Direct IP on last retry
2025-11-29 01:17:03 +01:00
Michael
dd558b0e43
fix: Correct proxy usage in scraper_service.py to handle dict format
2025-11-29 01:07:02 +01:00
Michael
4c410ef48f
fix: Correct Playwright proxy format - use dict with server/username/password instead of URL string
2025-11-29 01:02:41 +01:00
Michael
df2191cdbd
debug: Add proxy initialization logs to verify proxy loading
2025-11-29 00:53:55 +01:00
Michael
8080a97e25
feat: Add same 10 rotating proxies to scraper_service for individual product pages
2025-11-29 00:50:08 +01:00
Michael
3caea8316b
feat: Add Playwright-based web scraping service with advanced stealth techniques and proxy support, including Amazon-specific logic.
2025-11-28 23:53:45 +01:00
Claude
10ec21f4f5
feat: Add proxy support for Amazon scraping
...
- Add AMAZON_PROXY_URL and AMAZON_PROXY_LIST environment variables
- Support single proxy or rotating proxy list (comma-separated)
- Apply proxy only for Amazon requests
- Log proxy usage (hiding credentials)
2025-11-24 08:30:55 +00:00
Claude
e62ff09667
feat: Enhanced stealth JavaScript with comprehensive anti-detection
...
- Multiple webdriver override techniques (property, prototype, getAttribute)
- Realistic navigator.plugins and mimeTypes
- Delete 20+ automation-related window properties (CDP, Selenium, etc.)
- Full chrome object emulation with realistic properties
- WebGL and WebGL2 fingerprinting protection
- Canvas fingerprinting protection with noise injection
- AudioContext fingerprinting protection
- Screen properties spoofing
- iframe contentWindow webdriver hiding
- Battery API mocking
- Hide Playwright/Puppeteer specific objects
2025-11-24 08:25:23 +00:00
Claude
3bae422c38
fix: Add stealth JavaScript injection to bypass Amazon bot detection
...
- Add STEALTH_JS constant with scripts to hide automation fingerprints
- Override navigator.webdriver, plugins, languages, permissions
- Fake WebGL vendor/renderer and hardware concurrency
- Inject stealth script via context.add_init_script() for Amazon pages
- Add bypass_csp=True to allow script injection
- Don't block images for Amazon to avoid detection triggers
- Apply stealth to both direct_search_service and scraper_service
2025-11-24 08:07:49 +00:00
Claude
7243bcfaa1
fix: Improve Amazon scraping to avoid 503 blocking errors
...
- Add User-Agent rotation pool with modern Chrome/Firefox/Edge versions
- Implement Amazon-specific blocking detection ("Toutes nos excuses", etc.)
- Add exponential backoff with jitter for Amazon retries (up to 4 retries)
- Include Sec-Fetch-* headers and proper Referer to appear more legitimate
- Simulate human behavior (mouse movements, random scrolling)
- Add random delays between requests to avoid rate limiting
- Properly differentiate Amazon 503 (blocking) from true product unavailability
- Apply same protections to both search and product scraping services
2025-11-24 07:45:09 +00:00
Michael
38af998a09
feat: add scraper service with URL simplification, Playwright integration, and retry logic for web scraping.
2025-11-23 22:18:11 +01:00
Michael
e551927327
feat: Add direct e-commerce search service using Playwright/Browserless with site-specific configurations.
2025-11-23 16:06:16 +01:00
Michael
8d124bf0f8
feat: Add initial database migration and UI label component, alongside frontend dependency installation.
2025-11-23 14:37:35 +01:00
Claude
3293b5d58a
fix: Improve French price extraction and cookie handling
...
AI Schema:
- Improve extraction prompt for French e-commerce pages
- Add specific instructions for French price format (comma decimal, space thousands)
- Add French stock status terms (Ajouter au panier, Disponible, etc.)
- Fix price normalization to handle French format:
- "12,99" -> 12.99
- "1 234,56" -> 1234.56
- "1.234,56" -> 1234.56
Scraper Service:
- Add comprehensive cookie/popup selectors for French sites:
- E.Leclerc, Auchan, Carrefour (OneTrust, Didomi)
- Cdiscount (#footer_tc_privacy_button_2)
- Boulanger (.bcom-consent-accept-all)
- Gifi, Centrakor (Magento cookie buttons)
- PrestaShop sites (Stokomani, B&M)
- Add common CMP platform selectors
- Add French text-based selectors for cookie acceptance
2025-11-23 08:12:43 +00:00
Claude
89f422744a
fix: Stabilité Browserless v2 + headers anti-détection
...
Browserless (docker-compose.yml):
- Image mise à jour: ghcr.io/browserless/chromium:latest
- Timeout augmenté à 300s (5 minutes)
- shm_size: 2gb (critique pour Chrome)
- Mémoire: 3G limit, 1G reservation
- Nouvelles variables: HEALTH, CHROME_REFRESH_TIME
Scraper Service:
- Ajout headers HTTP réalistes (Accept-Language, Accept)
- Locale fr-FR et timezone Europe/Paris
- Timeout connexion explicite (60s)
- Fermeture propre des ressources (page, context, browser)
- Meilleure gestion des erreurs de fermeture
2025-11-22 10:05:36 +00:00
Claude
b5cd1465ce
fix: Amélioration scraping Amazon + stabilité Browserless
...
Browserless (docker-compose.yml):
- CONNECTION_TIMEOUT augmenté à 180s
- MAX_CONCURRENT_SESSIONS réduit à 3
- Limite mémoire 2G ajoutée
- Args Chrome optimisés (disable-dev-shm-usage, no-sandbox)
Scraper Service:
- Ajout simplification automatique des URLs Amazon (extraction ASIN)
- Suppression des paramètres de tracking (utm_, ref, etc.)
- Système de retries (2 tentatives avec backoff)
- Screenshot pris AVANT extraction texte (évite perte si timeout)
- Popups Amazon ajoutés (#sp-cc-accept, etc.)
- Meilleure gestion des erreurs avec logs détaillés
2025-11-22 09:55:49 +00:00
LogiFlow
afed10abc5
Add files via upload
2025-11-22 10:01:25 +01:00
Michael
Claude
LogiFlow