feat: Add fallback selectors to GenericParser for better site coverage

Improves GenericParser to automatically try common e-commerce selectors
when the configured selector doesn't match any products.

**Problem:**
- Auchan and other sites return 0 products because configured selector
  doesn't match their current HTML structure
- GenericParser was too rigid: if config selector fails, give up

**Solution:**
Automatic fallback selector cascade:

1. Try configured selector first (as before)
2. If no results, try 11 common e-commerce patterns:
   - Product URL patterns: /produit, /product, /p/, /item
   - Class patterns: article a, .product a, .product-card a
   - Generic patterns: [class*='product'] a
3. Accept first selector that finds >= 3 links
4. Log which selector worked for debugging

**Selector Priority:**
- Most specific first (URL patterns)
- Medium specificity (semantic classes)
- Generic last resort (any product-related class)

**Benefits:**
- ✅ Works on more sites without manual config updates
- ✅ Resilient to site HTML changes
- ✅ Clear logging shows which selector worked
- ✅ Still uses configured selector when available
- ✅ Requires >= 3 matches to avoid false positives

**Expected Impact:**
Sites like Auchan should now find products even if configured
selector is outdated, without needing a specialized parser.
This commit is contained in:
Claude committed 2025-11-30 16:20:31 +00:00
1 parent aa9e8dc427
commit ff2615ff81
1 file changed
+34 -1
+34 -1
View File
@@ -50,6 +50,8 @@ class GenericParser(BaseParser):
Uses:
- product_selector: CSS selector for product links
- product_image_selector: CSS selector for images (optional)
If configured selector fails, tries common fallback selectors.
"""
soup = BeautifulSoup(html, 'html.parser')
products = []
@@ -62,8 +64,39 @@ class GenericParser(BaseParser):
links = soup.select(product_selector)
# If configured selector fails, try common fallback selectors
if not links:
self.logger.warning(f"No products found for {self.site_key} with selector: {product_selector}")
self.logger.warning(f"No products with configured selector: {product_selector}")
self.logger.info("Trying fallback selectors...")
# Common e-commerce selectors (ordered by specificity)
fallback_selectors = [
# Product-specific URLs
"a[href*='/produit']",
"a[href*='/product']",
"a[href*='/p/']",
"a[href*='/item']",
# Common class patterns
"article a[href]",
".product a[href]",
".product-card a[href]",
".product-item a[href]",
"[class*='product'] a[href]",
# Generic structure
"article[class*='product'] a",
"div[class*='product'] a",
# Very generic (last resort)
"a[class*='product']",
]
for fallback in fallback_selectors:
links = soup.select(fallback)
if links and len(links) >= 3: # At least 3 links to be credible
self.logger.info(f"✓ Found {len(links)} links with fallback: {fallback}")
break
if not links:
self.logger.warning(f"No products found for {self.site_key} even with fallbacks")
return []
self.logger.info(f"Found {len(links)} product links for {self.site_name}")