feat: Improve Search & Comparateur with persistent browser ScraperService pattern

- Created improved_search_service.py using persistent browser pattern
- Persistent browser connection with auto-reconnect capability
- Proper popup/cookie handling across all sites
- Price extraction with validation (reject unrealistic prices)
- Stock status detection
- Concurrent scraping with semaphore limits (2 sites, 2 products)
- Reuses browser context for better session management
- Screenshot capture for product images
- Updated search router to use improved service
- Added initialization/shutdown in main.py lifespan

Benefits:
✓ More reliable scraping with persistent connections
✓ Better anti-detection (consistent sessions)
✓ Improved price accuracy with validation
✓ Faster performance (reuses browser contexts)
✓ Auto-recovery from connection failures
This commit is contained in:
Claude committed 2025-11-30 10:51:22 +00:00
1 parent 85b19e7680
commit 68db9ad15e
3 files changed
+629 -2

No files matched your search

+6
View File
@@ -18,6 +18,7 @@ from app.services.scheduler_service import scheduled_refresh, scheduler
from app.services import auth_service, search_service, seed_enseignes
from app.services.scheduler import start_scheduler as start_catalog_scheduler, stop_scheduler as stop_catalog_scheduler
from app.services.amazon_scraper_service import amazon_scraper_service
from app.services.improved_search_service import improved_search_service
# Configure logging
logging.basicConfig(
@@ -145,12 +146,17 @@ async def lifespan(app: FastAPI):
logger.info("Initializing Amazon scraper service...")
await amazon_scraper_service.initialize()
# Initialize Improved Search Service (persistent browser for Search/Comparatif)
logger.info("Initializing Improved Search Service...")
await improved_search_service.initialize()
logger.info("Application started")
yield
logger.info("Shutting down schedulers and services...")
scheduler.shutdown(wait=True)
stop_catalog_scheduler()
await amazon_scraper_service.shutdown()
await improved_search_service.shutdown()
logger.info("Application shutdown complete")