mirror of
https://github.com/R0m1k3/Priceflow.git
synced 2026-10-11 17:29:14 +02:00
feat: Add fallback selectors to GenericParser for better site coverage
Improves GenericParser to automatically try common e-commerce selectors when the configured selector doesn't match any products. **Problem:** - Auchan and other sites return 0 products because configured selector doesn't match their current HTML structure - GenericParser was too rigid: if config selector fails, give up **Solution:** Automatic fallback selector cascade: 1. Try configured selector first (as before) 2. If no results, try 11 common e-commerce patterns: - Product URL patterns: /produit, /product, /p/, /item - Class patterns: article a, .product a, .product-card a - Generic patterns: [class*='product'] a 3. Accept first selector that finds >= 3 links 4. Log which selector worked for debugging **Selector Priority:** - Most specific first (URL patterns) - Medium specificity (semantic classes) - Generic last resort (any product-related class) **Benefits:** - ✅ Works on more sites without manual config updates - ✅ Resilient to site HTML changes - ✅ Clear logging shows which selector worked - ✅ Still uses configured selector when available - ✅ Requires >= 3 matches to avoid false positives **Expected Impact:** Sites like Auchan should now find products even if configured selector is outdated, without needing a specialized parser.
This commit is contained in:
1 file changed
+34
-1
@@ -50,6 +50,8 @@ class GenericParser(BaseParser):
|
||||
Uses:
|
||||
- product_selector: CSS selector for product links
|
||||
- product_image_selector: CSS selector for images (optional)
|
||||
|
||||
If configured selector fails, tries common fallback selectors.
|
||||
"""
|
||||
soup = BeautifulSoup(html, 'html.parser')
|
||||
products = []
|
||||
@@ -62,8 +64,39 @@ class GenericParser(BaseParser):
|
||||
|
||||
links = soup.select(product_selector)
|
||||
|
||||
# If configured selector fails, try common fallback selectors
|
||||
if not links:
|
||||
self.logger.warning(f"No products found for {self.site_key} with selector: {product_selector}")
|
||||
self.logger.warning(f"No products with configured selector: {product_selector}")
|
||||
self.logger.info("Trying fallback selectors...")
|
||||
|
||||
# Common e-commerce selectors (ordered by specificity)
|
||||
fallback_selectors = [
|
||||
# Product-specific URLs
|
||||
"a[href*='/produit']",
|
||||
"a[href*='/product']",
|
||||
"a[href*='/p/']",
|
||||
"a[href*='/item']",
|
||||
# Common class patterns
|
||||
"article a[href]",
|
||||
".product a[href]",
|
||||
".product-card a[href]",
|
||||
".product-item a[href]",
|
||||
"[class*='product'] a[href]",
|
||||
# Generic structure
|
||||
"article[class*='product'] a",
|
||||
"div[class*='product'] a",
|
||||
# Very generic (last resort)
|
||||
"a[class*='product']",
|
||||
]
|
||||
|
||||
for fallback in fallback_selectors:
|
||||
links = soup.select(fallback)
|
||||
if links and len(links) >= 3: # At least 3 links to be credible
|
||||
self.logger.info(f"✓ Found {len(links)} links with fallback: {fallback}")
|
||||
break
|
||||
|
||||
if not links:
|
||||
self.logger.warning(f"No products found for {self.site_key} even with fallbacks")
|
||||
return []
|
||||
|
||||
self.logger.info(f"Found {len(links)} product links for {self.site_name}")
|
||||
|
||||
Reference in new issue
Block a user