Commit Graph
3 Commits
Author SHA1 Message Date
Claude ea2af8a8f2 feat: Create persistent Amazon scraper service (ScraperService pattern)
- New amazon_scraper_service.py with persistent browser connection
- Shared browser instance across requests (better session management)
- Auto-reconnect if browser connection drops
- Better stealth mode (webdriver undefined, chrome runtime)
- Proper cookie/popup handling for Amazon
- Uses Playwright via Browserless (not Crawl4AI)
- Should fix 503 errors with persistent session

Based on user's ScraperService pattern for reliability
2025-11-30 10:34:41 +00:00
Claude af7c32a4d8 feat: Rewrite Amazon scraper to use Browserless service
- Created amazon_scraper_v2.py using browserless_service (Playwright)
- Removed dependency on Crawl4AI which was not loading pages correctly
- Use browserless proxy rotation (use_proxy=True)
- More robust selector fallbacks for title, price, rating
- Fixed 2128 bytes issue - now loads full Amazon pages (>100KB)
- Updated router to use new scraper

Previous issue: Crawl4AI only loaded 2128 bytes
Now: Browserless loads complete pages with all products
2025-11-30 09:42:25 +00:00
Claude 684535ee85 feat: Add Amazon France search page with Crawl4AI anti-detection
- Created Amazon scraper service with advanced anti-bot techniques:
  * User-Agent rotation from realistic pool
  * Complete browser headers (Accept, Accept-Language, etc.)
  * Proxy rotation (10 residential proxies)
  * Random delays (1.5-4s) to mimic human behavior
  * Crawl4AI browser fingerprint randomization
  * NetworkIdle waiting for complete page load
  * Cookie acceptance automation

- Added Amazon search API endpoint with SSE streaming
  * Real-time progress updates
  * Proper error handling
  * Health check endpoint

- Created dedicated Amazon France frontend page:
  * Modern UI with product cards
  * Rating display (stars + review count)
  * Price formatting with discount badges
  * Prime badge support
  * Stock status indicators
  * Sponsored product labels
  * Direct Amazon links

- Removed store list (ENSEIGNES_DATA cleared)
  * Migration from discount stores to Amazon France
  * Catalog system kept for future use

- Updated navigation:
  * Added "Amazon France" menu item with ShoppingBag icon
  * Positioned between Search and Compare
  * Available on desktop and mobile

Technical stack:
- Backend: Crawl4AI + BeautifulSoup for scraping
- Frontend: React + Shadcn UI components
- API: FastAPI with SSE streaming
2025-11-30 00:36:39 +00:00