Priceflow/implementation_plan.md

Implementation Plan - Catalog Scraper Fix

Problem

The catalog scraper (cataloguemate_scraper.py) fails to retrieve catalogs because the browserless service is likely being blocked or failing to connect (based on logs). However, the target site (cataloguemate.fr) is accessible via simple HTTP requests.

Proposed Changes

1. Modify app/services/cataloguemate_scraper.py

We will implement a fallback mechanism in scrape_catalog_list:

• Primary Method: Continue using browserless_service.get_page_content.
• Fallback Method: If browserless returns no content or fails, use httpx.AsyncClient to fetch the HTML directly.
• Parsing: Use BeautifulSoup to parse the HTML from either source.

[MODIFY] cataloguemate_scraper.py

• Import httpx and random (for user-agents).
• In scrape_catalog_list, add a try/except block or check for empty content.
• If content is missing, call a new helper or inline httpx request with standard headers.

Verification Plan

Automated Verification

Since we cannot run the full app locally (docker/dependencies issues), we will verify by code review and logic.

Manual Verification (User)

1. User triggers the catalog update (via Admin or Scheduler).
2. User checks logs to see if "Fallback to HTTP" message appears.
3. User verifies that catalogs (e.g. B&M, Gifi) appear in the dashboard.