mirror of
https://github.com/R0m1k3/Priceflow.git
synced 2026-10-19 13:18:12 +02:00
feat: add debug_catalog_images.py to analyze catalog image selectors and update task.md to reflect the new focus on catalog image fixes.
This commit is contained in:
1 parent
d2fe4bf22b
commit
7842eead4a
3 files changed
+29
-19
No files matched your search
+14
-7
@@ -9,16 +9,23 @@
|
||||
- **Prompt Engineering**: Rewrote the system prompt in `ai_schema.py` to explicitly declare the **IMAGE AS THE SOURCE OF TRUTH**.
|
||||
- **Logic Fix**: Disabled the `AIPriceExtractor` (Text-only AI) in `scheduler_service.py` which was short-circuiting the logic before the Vision AI could run.
|
||||
|
||||
## 2. Catalog Scraper Fix
|
||||
## 2. Catalog Scraper Fix (Connectivity)
|
||||
|
||||
**Problem**: Catalogs were not updating because the `browserless` service was failing (likely blocked or network issues), preventing the scraper from loading `cataloguemate.fr`.
|
||||
**Problem**: Catalogs were not updating because the `browserless` service was failing (likely blocked or network issues).
|
||||
|
||||
**Solution**:
|
||||
|
||||
- **HTTP Fallback**: Modified `cataloguemate_scraper.py` to use a robust fallback mechanism.
|
||||
- First attempts to use the secure Browserless browser.
|
||||
- If that fails, it instantly falls back to a standard `httpx` HTTP request, which is often sufficient for static catalog sites.
|
||||
- **HTTP Fallback**: Modified `cataloguemate_scraper.py` to use a robust fallback mechanism (Browserless -> Fallback to HTTPX).
|
||||
|
||||
## 3. Cleanup
|
||||
## 3. Catalog Images Fix (Lazy Loading)
|
||||
|
||||
- Removed 20+ temporary debug/verification scripts (`debug_*.py`, `verify_*.py`) from the root directory to keep the production environment clean.
|
||||
**Problem**: Catalog pages displayed generic icons or placeholders instead of the actual catalog images.
|
||||
**Analysis**: The website uses lazy loading. Structure: `<img src="loader.svg" data-src="REAL_IMAGE_URL" ... />`.
|
||||
**Solution**:
|
||||
|
||||
- **Smart Extraction**: Updated `cataloguemate_scraper.py` to checking `data-src` and `data-original` attributes first.
|
||||
- **Improved Filtering**: Added `loader` to the ignore list and `thumbor` to the whitelist to ensure only high-quality catalog images are selected.
|
||||
|
||||
## 4. Cleanup
|
||||
|
||||
- Removed temporary debug/verification scripts to keep the production environment clean.
|
||||
Reference in new issue
Block a user