- Add debug_enabled field to SearchSite model
- Update schemas and API routes to allow debug toggle
- Add debug toggle button on site cards in Admin UI
- Only dump HTML when debug is enabled for the site (except for critical errors like Amazon blocking)
- Add API endpoints to list/delete debug files per site domain
- Add button to clear debug files for individual sites
- Add User-Agent rotation pool with modern Chrome/Firefox/Edge versions
- Implement Amazon-specific blocking detection ("Toutes nos excuses", etc.)
- Add exponential backoff with jitter for Amazon retries (up to 4 retries)
- Include Sec-Fetch-* headers and proper Referer to appear more legitimate
- Simulate human behavior (mouse movements, random scrolling)
- Add random delays between requests to avoid rate limiting
- Properly differentiate Amazon 503 (blocking) from true product unavailability
- Apply same protections to both search and product scraping services
- Changed debug_dumps path from relative to absolute (/app/debug_dumps)
- Updated direct_search_service.py to use consistent absolute path
- Updated debug.py router to use the same absolute path
- Added debug_dumps directory creation in Dockerfile
- Improved _dump_debug_html with safer filename handling
- Add User model with authentication
- Create JWT-based auth service
- Add login page with admin/admin default credentials
- Create admin panel with user management, settings, and site configuration
- Move settings to admin section (admin only)
- Change search page from multi-site checkboxes to single site selector
- Add logout functionality to layout
- Protect routes with authentication
- E.Leclerc: Fix URL from /cat/recherche~q= to /recherche?q=
(the old URL was showing promotions instead of search results)
- Auchan: Keep correct URL /recherche?text=
- Carrefour: Keep correct URL /s?q=
- Amazon FR/US: Simplify configuration, remove complex anti-bot
headers that were causing more issues than helping
- Simplify browserless search function (remove special Amazon handling)
- Add special handling for Amazon (FR and US):
- Custom User-Agent and HTTP headers mimicking real browser
- Don't block images (Amazon uses them for bot detection)
- Extra wait time and scroll for lazy-loaded content
- Timezone set to Europe/Paris
- Additional Sec-Fetch-* headers for authenticity
- Update Amazon product selectors with more robust patterns:
- a.a-link-normal for direct product links
- [data-asin]:not([data-asin='']) for filtering empty ASINs
- .s-main-slot targeting main search results
- Add &ref=nb_sb_noss parameter to avoid redirects
- Change default currency from USD to EUR for all French sites
- Update AI extraction prompt to use EUR by default
- Fix E.Leclerc search URL pattern
- Improve product selectors for Auchan, Carrefour, Amazon France
- Add Amazon France language parameter for better results
- Enhance cookie acceptance with site-specific selectors:
- Amazon (#sp-cc-accept)
- E.Leclerc (#onetrust-accept-btn-handler)
- Auchan (#popin_tc_privacy_button_2, #didomi-notice-agree-button)
- Carrefour (#onetrust-accept-btn-handler)
- Cdiscount (#footer_tc_privacy_button_2)
- And more CMP platforms (Didomi, OneTrust, Tarteaucitron, etc.)
- Improve cookie handling with iframe support and retry logic
- Remove ability to add/edit/delete search sites from settings UI
- Configure 16 French e-commerce sites with search URLs:
- Discount: Gifi, Stokomani, B&M, Centrakor, L'Incroyable, Action, La Foir'Fouille
- Grandes Surfaces: E.Leclerc, Auchan, Carrefour
- E-commerce: Amazon France, Cdiscount
- Électronique: Darty, Boulanger, Fnac
- Add automatic database seeding on application startup
- Add /api/search-sites/seed and /api/search-sites/reset endpoints
- Add cookie consent banner handling for sites that require it
- Simplify Settings page to only show site list with toggle for active state
- Add fallback to generic detection when configured selector fails
- Add many more CSS selectors for French e-commerce platforms (PrestaShop, Magento, WooCommerce)
- Add /produits/ pattern for bmstores.fr
- Improve generic container-based link detection
- Add navigation link filter to exclude non-product links
- Improve domain matching in direct_search_service to handle various formats
- Add /discover-all endpoint to discover search URLs for all sites missing one
- Add constants for magic numbers and fix linting issues
- Better domain cleaning in comparison functions
- Move imports to top-level in search_sites router
- Fix line too long issue with input selector
- Define constants for magic numbers (HTTP_OK, MIN_PRODUCTS_TO_MATCH)
- Use 'raise from' pattern for proper exception chaining
- Add search_url_discovery.py service that visits sites and finds search forms
- Automatically discover search_url when a new site is created without one
- Add POST /{site_id}/discover endpoint to force rediscovery
- Discovery runs in background to not block the API response
- Tries multiple methods: form analysis, common URL patterns
- Add search URLs for French discount stores: stokomani, gifi, bmstores,
centrakor, lafoirfouille, tedi, leclerc, carrefour, auchan
- Fix screenshot URL construction in search results
- Include screenshot_url in AI extraction results and error cases
- Use AIService.analyze_image() class method instead of module function
- Handle AIExtractionResponse tuple return type properly
- Add _clean_domain() to strip protocol/www from stored domains
- Fixes "https://https://" URL duplication issue
- Replace httpx with Playwright/Browserless for search pages
- Handles JavaScript-rendered content and bot detection
- Add wait selectors for each site to ensure content is loaded
- Block images and tracking scripts for faster loading
- Sequential site scraping to avoid overloading Browserless
- Remove duckduckgo-search dependency (unreliable results)
- Add direct_search_service.py that scrapes sites' search pages directly
- Add search_url and product_link_selector fields to SearchSite model
- Add beautifulsoup4 and lxml for HTML parsing
- Update schemas with new fields
- Add database migration for new columns
Direct search is more reliable as it queries each site's own search
functionality instead of relying on external search engines.
- Remove SearXNG container and configuration files
- Add duckduckgo-search library for web search
- Create duckduckgo_service.py with async search support
- Update search_service.py to use DuckDuckGo instead of SearXNG
- Fix BROWSERLESS_URL port to 3012
This eliminates the persistent limiter.toml directory issue and
provides a simpler, more reliable search solution.
- Create search_sites table if it doesn't exist
- Add price_selector column only if table exists but column doesn't
- Fixes 400 Bad Request when creating search sites
Claude
Michael