mirror of
https://github.com/R0m1k3/Priceflow.git
synced 2026-10-11 17:29:14 +02:00
34 lines
1.4 KiB
Markdown
34 lines
1.4 KiB
Markdown
# Implementation Plan - Catalog Scraper Fix
|
|
|
|
## Problem
|
|
|
|
The catalog scraper (`cataloguemate_scraper.py`) fails to retrieve catalogs because the `browserless` service is likely being blocked or failing to connect (based on logs). However, the target site (`cataloguemate.fr`) is accessible via simple HTTP requests.
|
|
|
|
## Proposed Changes
|
|
|
|
### 1. Modify `app/services/cataloguemate_scraper.py`
|
|
|
|
We will implement a fallback mechanism in `scrape_catalog_list`:
|
|
|
|
- **Primary Method**: Continue using `browserless_service.get_page_content`.
|
|
- **Fallback Method**: If `browserless` returns no content or fails, use `httpx.AsyncClient` to fetch the HTML directly.
|
|
- **Parsing**: Use `BeautifulSoup` to parse the HTML from either source.
|
|
|
|
#### [MODIFY] [cataloguemate_scraper.py](file:///c:/Users/Michael/VSCODE/Priceflow-1/app/services/cataloguemate_scraper.py)
|
|
|
|
- Import `httpx` and `random` (for user-agents).
|
|
- In `scrape_catalog_list`, add a `try/except` block or check for empty content.
|
|
- If content is missing, call a new helper or inline `httpx` request with standard headers.
|
|
|
|
## Verification Plan
|
|
|
|
### Automated Verification
|
|
|
|
Since we cannot run the full app locally (docker/dependencies issues), we will verify by code review and logic.
|
|
|
|
### Manual Verification (User)
|
|
|
|
1. User triggers the catalog update (via Admin or Scheduler).
|
|
2. User checks logs to see if "Fallback to HTTP" message appears.
|
|
3. User verifies that catalogs (e.g. B&M, Gifi) appear in the dashboard.
|