mirror of
https://github.com/R0m1k3/Priceflow.git
synced 2026-10-20 13:48:16 +02:00
feat: Implement HTTPX fallback in Cataloguemate scraper for enhanced reliability and add associated implementation plan and debug script.
This commit is contained in:
1 parent
17b3009b1d
commit
4bf950c5eb
4 files changed
+179
-27
No files matched your search
@@ -0,0 +1,33 @@
|
||||
# Implementation Plan - Catalog Scraper Fix
|
||||
|
||||
## Problem
|
||||
|
||||
The catalog scraper (`cataloguemate_scraper.py`) fails to retrieve catalogs because the `browserless` service is likely being blocked or failing to connect (based on logs). However, the target site (`cataloguemate.fr`) is accessible via simple HTTP requests.
|
||||
|
||||
## Proposed Changes
|
||||
|
||||
### 1. Modify `app/services/cataloguemate_scraper.py`
|
||||
|
||||
We will implement a fallback mechanism in `scrape_catalog_list`:
|
||||
|
||||
- **Primary Method**: Continue using `browserless_service.get_page_content`.
|
||||
- **Fallback Method**: If `browserless` returns no content or fails, use `httpx.AsyncClient` to fetch the HTML directly.
|
||||
- **Parsing**: Use `BeautifulSoup` to parse the HTML from either source.
|
||||
|
||||
#### [MODIFY] [cataloguemate_scraper.py](file:///c:/Users/Michael/VSCODE/Priceflow-1/app/services/cataloguemate_scraper.py)
|
||||
|
||||
- Import `httpx` and `random` (for user-agents).
|
||||
- In `scrape_catalog_list`, add a `try/except` block or check for empty content.
|
||||
- If content is missing, call a new helper or inline `httpx` request with standard headers.
|
||||
|
||||
## Verification Plan
|
||||
|
||||
### Automated Verification
|
||||
|
||||
Since we cannot run the full app locally (docker/dependencies issues), we will verify by code review and logic.
|
||||
|
||||
### Manual Verification (User)
|
||||
|
||||
1. User triggers the catalog update (via Admin or Scheduler).
|
||||
2. User checks logs to see if "Fallback to HTTP" message appears.
|
||||
3. User verifies that catalogs (e.g. B&M, Gifi) appear in the dashboard.
|
||||
Reference in new issue
Block a user