Complete refactoring of the search module for robustness and reliability:
**New Architecture:**
- Modular parser system with specialized parsers for each major site
- Factory pattern for automatic parser selection
- Generic fallback parser for sites without specific implementation
**Specialized Parsers:**
- ✅ Amazon.fr - Robust parser with multiple fallback selectors
- Handles sponsored products, Prime eligibility, ratings, reviews
- Based on proven amazon_scraper_v2.py implementation
- ✅ Cdiscount.com - Optimized for Cdiscount's HTML structure
- ✅ Fnac.com - Handles Fnac's picture elements and article structure
- ✅ Darty.com - Specialized for electronics retailer
- ✅ Boulanger.com - Optimized for Boulanger's product cards
**Key Features:**
- Multiple fallback selectors for each element (title, price, image)
- Proper price parsing for French format (1 234,56 €)
- Rating and review count extraction
- Stock status detection
- Image URL extraction with multiple attribute fallbacks
- Query filtering to ensure relevant results
**Benefits:**
- Much more robust than previous generic parsing
- Easier to maintain (one parser per site)
- Easily extensible (add new parser for new site)
- Better error handling and logging
- Higher success rate for product extraction
**Updated Services:**
- improved_search_service.py now uses ParserFactory
- Legacy _parse_results marked as deprecated
- Conversion layer between ProductResult and SearchResult
This implements the Amazon France model across all major sites
for consistent, reliable product search results.
- Improved image extraction with multiple strategies (src, data-src, srcset, picture elements)
- Filter out placeholder and 1x1 pixel images
- Don't replace real images with non-existent screenshots
- Screenshots would need FastAPI static serving which is unnecessary
- CRITICAL FIX: Load amazon.fr homepage in same context before search
- Preserves cookies/session between homepage and search
- Should fix 2065 bytes issue (blocked requests)
- Homepage popups handled before search
- Load amazon.fr homepage before search to get cookies
- Add 2s delay between homepage and search (human-like)
- Trying to bypass 503 errors by establishing session first
- Each request still creates new context (limitation)
- Disabled proxy (use_proxy=False) for testing
- Previous attempt got 503 from Amazon with proxy
- Will test if direct connection works better
- Can re-enable proxy later if needed
- Created amazon_scraper_v2.py using browserless_service (Playwright)
- Removed dependency on Crawl4AI which was not loading pages correctly
- Use browserless proxy rotation (use_proxy=True)
- More robust selector fallbacks for title, price, rating
- Fixed 2128 bytes issue - now loads full Amazon pages (>100KB)
- Updated router to use new scraper
Previous issue: Crawl4AI only loaded 2128 bytes
Now: Browserless loads complete pages with all products
- Removed proxy_config from BrowserConfig to avoid conflicts
- Browserless service already has integrated proxy rotation
- Will integrate with browserless_service later if needed
- Focus on fixing CSS selectors first
- Add debug logs for each step of product extraction
- Log ASIN, title, link extraction failures
- Save HTML to /tmp/amazon_debug_*.html for inspection
- Enable DEBUG logging level temporarily
- Will help identify why 48 cards found but 0 products extracted
- Changed proxy format from dict to string (http://user:pass@ip:port)
- Updated get_random_proxy() to return Crawl4AI-compatible format
- Replaced deprecated 'proxy' with 'proxy_config' in BrowserConfig
- Fixed test script to handle new proxy format
- Added credential hiding in proxy logging for security
Fixes AttributeError: 'dict' object has no attribute 'strip'
Claude
Michael
LogiFlow