- Add fallback to generic detection when configured selector fails
- Add many more CSS selectors for French e-commerce platforms (PrestaShop, Magento, WooCommerce)
- Add /produits/ pattern for bmstores.fr
- Improve generic container-based link detection
- Add navigation link filter to exclude non-product links
- Improve domain matching in direct_search_service to handle various formats
- Add /discover-all endpoint to discover search URLs for all sites missing one
- Add constants for magic numbers and fix linting issues
- Better domain cleaning in comparison functions
- Add search URLs for French discount stores: stokomani, gifi, bmstores,
centrakor, lafoirfouille, tedi, leclerc, carrefour, auchan
- Fix screenshot URL construction in search results
- Include screenshot_url in AI extraction results and error cases
- Use AIService.analyze_image() class method instead of module function
- Handle AIExtractionResponse tuple return type properly
- Add _clean_domain() to strip protocol/www from stored domains
- Fixes "https://https://" URL duplication issue
- Replace httpx with Playwright/Browserless for search pages
- Handles JavaScript-rendered content and bot detection
- Add wait selectors for each site to ensure content is loaded
- Block images and tracking scripts for faster loading
- Sequential site scraping to avoid overloading Browserless
- Remove duckduckgo-search dependency (unreliable results)
- Add direct_search_service.py that scrapes sites' search pages directly
- Add search_url and product_link_selector fields to SearchSite model
- Add beautifulsoup4 and lxml for HTML parsing
- Update schemas with new fields
- Add database migration for new columns
Direct search is more reliable as it queries each site's own search
functionality instead of relying on external search engines.
Claude