feat: add debug_catalog_images.py to analyze catalog image selectors and update task.md to reflect the new focus on catalog image fixes.

This commit is contained in:
Michael committed 2025-12-22 13:49:27 +01:00
1 parent d2fe4bf22b
commit 7842eead4a
3 files changed
+29 -19

No files matched your search

+14 -7
View File
@@ -9,16 +9,23 @@
- **Prompt Engineering**: Rewrote the system prompt in `ai_schema.py` to explicitly declare the **IMAGE AS THE SOURCE OF TRUTH**.
- **Logic Fix**: Disabled the `AIPriceExtractor` (Text-only AI) in `scheduler_service.py` which was short-circuiting the logic before the Vision AI could run.
## 2. Catalog Scraper Fix
## 2. Catalog Scraper Fix (Connectivity)
**Problem**: Catalogs were not updating because the `browserless` service was failing (likely blocked or network issues), preventing the scraper from loading `cataloguemate.fr`.
**Problem**: Catalogs were not updating because the `browserless` service was failing (likely blocked or network issues).
**Solution**:
- **HTTP Fallback**: Modified `cataloguemate_scraper.py` to use a robust fallback mechanism.
- First attempts to use the secure Browserless browser.
- If that fails, it instantly falls back to a standard `httpx` HTTP request, which is often sufficient for static catalog sites.
- **HTTP Fallback**: Modified `cataloguemate_scraper.py` to use a robust fallback mechanism (Browserless -> Fallback to HTTPX).
## 3. Cleanup
## 3. Catalog Images Fix (Lazy Loading)
- Removed 20+ temporary debug/verification scripts (`debug_*.py`, `verify_*.py`) from the root directory to keep the production environment clean.
**Problem**: Catalog pages displayed generic icons or placeholders instead of the actual catalog images.
**Analysis**: The website uses lazy loading. Structure: `<img src="loader.svg" data-src="REAL_IMAGE_URL" ... />`.
**Solution**:
- **Smart Extraction**: Updated `cataloguemate_scraper.py` to checking `data-src` and `data-original` attributes first.
- **Improved Filtering**: Added `loader` to the ignore list and `thumbor` to the whitelist to ensure only high-quality catalog images are selected.
## 4. Cleanup
- Removed temporary debug/verification scripts to keep the production environment clean.