- Create _find_amazon_product_links() function for Amazon-specific extraction
- Use ASIN-based deduplication for Amazon products
- Bypass all generic filters (_is_non_product_url, _is_navigation_link) for Amazon
- Multiple fallback methods for Amazon product detection:
- data-component-type="s-product-image"
- .s-result-item links
- All /dp/ links in main content
- Clean Amazon URLs by removing tracking parameters
- Extract title from img alt as fallback for Amazon
This separates Amazon's extraction logic from the generic e-commerce filters
that were designed for French sites like Cdiscount, Fnac, etc.
- Add AMAZON_PROXY_URL and AMAZON_PROXY_LIST environment variables
- Support single proxy or rotating proxy list (comma-separated)
- Apply proxy only for Amazon requests
- Log proxy usage (hiding credentials)
- Add debug_enabled field to SearchSite model
- Update schemas and API routes to allow debug toggle
- Add debug toggle button on site cards in Admin UI
- Only dump HTML when debug is enabled for the site (except for critical errors like Amazon blocking)
- Add API endpoints to list/delete debug files per site domain
- Add button to clear debug files for individual sites
- Add User-Agent rotation pool with modern Chrome/Firefox/Edge versions
- Implement Amazon-specific blocking detection ("Toutes nos excuses", etc.)
- Add exponential backoff with jitter for Amazon retries (up to 4 retries)
- Include Sec-Fetch-* headers and proper Referer to appear more legitimate
- Simulate human behavior (mouse movements, random scrolling)
- Add random delays between requests to avoid rate limiting
- Properly differentiate Amazon 503 (blocking) from true product unavailability
- Apply same protections to both search and product scraping services
- Changed debug_dumps path from relative to absolute (/app/debug_dumps)
- Updated direct_search_service.py to use consistent absolute path
- Updated debug.py router to use the same absolute path
- Added debug_dumps directory creation in Dockerfile
- Improved _dump_debug_html with safer filename handling
- Add User model with authentication
- Create JWT-based auth service
- Add login page with admin/admin default credentials
- Create admin panel with user management, settings, and site configuration
- Move settings to admin section (admin only)
- Change search page from multi-site checkboxes to single site selector
- Add logout functionality to layout
- Protect routes with authentication
- E.Leclerc: Fix URL from /cat/recherche~q= to /recherche?q=
(the old URL was showing promotions instead of search results)
- Auchan: Keep correct URL /recherche?text=
- Carrefour: Keep correct URL /s?q=
- Amazon FR/US: Simplify configuration, remove complex anti-bot
headers that were causing more issues than helping
- Simplify browserless search function (remove special Amazon handling)
- Add special handling for Amazon (FR and US):
- Custom User-Agent and HTTP headers mimicking real browser
- Don't block images (Amazon uses them for bot detection)
- Extra wait time and scroll for lazy-loaded content
- Timezone set to Europe/Paris
- Additional Sec-Fetch-* headers for authenticity
- Update Amazon product selectors with more robust patterns:
- a.a-link-normal for direct product links
- [data-asin]:not([data-asin='']) for filtering empty ASINs
- .s-main-slot targeting main search results
- Add &ref=nb_sb_noss parameter to avoid redirects
Michael
Claude