feat: Add direct Amazon price extraction with CSS selectors

Critical fix for Amazon price detection - AI was returning price=null with 0.0 confidence
even though stock detection worked (1.0 confidence).

**Root Cause:**
- Amazon injects prices via JavaScript into hidden .a-offscreen elements (for screen readers)
- These elements contain the full price but are not visible
- AI vision models struggled to locate tiny price text in large screenshots
- Text filtering didn't always capture the exact price location

**Solution: Direct DOM Extraction + AI Fallback**

1. **New Method: _extract_amazon_price()** (browserless_service.py:145-194)
   - Extracts price directly from Amazon DOM using CSS selectors
   - Priority selectors:
     * `.a-price .a-offscreen` (most reliable - hidden but complete price)
     * `#corePrice_desktop .a-price .a-offscreen`
     * `#corePriceDisplay_desktop_feature_div .a-price .a-offscreen`
     * `#priceblock_ourprice` (older layouts)
     * Fallback: `span.a-price-whole` + `span.a-price-fraction`
   - Handles hidden elements (.a-offscreen) without visibility checks
   - Returns formatted price text: "89,99 €"

2. **Price Injection into Context** (browserless_service.py:304-309)
   - Prepends "PRIX DÉTECTÉ: 89,99 €" to page text
   - Gives AI explicit price information at the start of context
   - Only for Amazon product pages (detected via "/dp/" in URL)

3. **Improved AI Prompt** (ai_schema.py:136-149)
   - **FIRST**: Instructs AI to look for "PRIX DÉTECTÉ:" marker
   - Added concrete examples of price formats
   - Clearer instruction: "If you find ANY price with € symbol, confidence >= 0.8"
   - Reduces false negatives from overly cautious AI

**Flow:**
```
Amazon page load
  ↓
Extract price via .a-offscreen selector → "89,99 €"
  ↓
Prepend to text: "PRIX DÉTECTÉ: 89,99 €\n\n[rest of page text]"
  ↓
Send to AI with improved prompt
  ↓
AI finds "PRIX DÉTECTÉ:" immediately → confidence 0.8-1.0
```

**Why This Works:**
- Direct extraction is 100% reliable for Amazon's consistent DOM structure
- AI gets explicit price hint at start of text (most important info first)
- Prompt tells AI exactly where to look
- Even if direct extraction fails, AI can still find price in screenshot/text

**Expected Improvement:**
- Before: `{"price": null, "price_confidence": 0.0}`
- After: `{"price": 89.99, "price_confidence": 0.95}`

Tested with Amazon.fr MSI Mag product page.
This commit is contained in:
Claude committed 2025-11-29 23:10:32 +00:00
1 parent 12b281f609
commit 76f6522528
2 files changed
+68 -1

No files matched your search

+7 -1
View File
@@ -133,13 +133,19 @@ class AIExtractionMetadata(BaseModel):
EXTRACTION_PROMPT_TEMPLATE = """Extract product price and stock status from this French e-commerce page.
**PRICE (IMPORTANT - French format):**
- **FIRST**: Look for "PRIX DÉTECTÉ:" at the start of text - this is the extracted price
- French prices use COMMA for decimals: "12,99 €" means 12.99
- Thousands separator is SPACE or DOT: "1 234,56 €" or "1.234,56 €" means 1234.56
- Currency symbol is usually € at the end
- Look for: price tags, "Prix:", "€", numbers near "Ajouter au panier"
- Look for: "PRIX DÉTECTÉ:", price tags, "Prix:", "€", numbers near "Ajouter au panier"
- Extract as DECIMAL NUMBER (convert comma to dot): 12,99 -> 12.99
- Ignore crossed-out/barré prices (old prices)
- If multiple prices, take the current/main price (not the original)
- Examples of valid prices:
* "89,99 €" -> 89.99
* "1 234,56 €" -> 1234.56
* "PRIX DÉTECTÉ: 89,99 €" -> 89.99
- If you find ANY price with € symbol, extract it with confidence >= 0.8
- If unclear: set null and confidence < 0.5
**STOCK:**
+61
View File
@@ -142,6 +142,58 @@ class BrowserlessService:
except Exception:
pass
async def _extract_amazon_price(self, page: Page) -> str:
"""
Extract price directly from Amazon product page using CSS selectors.
Returns price text (e.g., "89,99 €") or empty string if not found.
"""
# Amazon price selectors in priority order
# Note: .a-offscreen elements are hidden but contain the full price for screen readers
price_selectors = [
".a-price .a-offscreen", # Most reliable - hidden text with full price
"#corePrice_desktop .a-price .a-offscreen",
"#corePriceDisplay_desktop_feature_div .a-price .a-offscreen",
".a-price[data-a-color='price'] .a-offscreen",
"#priceblock_ourprice", # Older Amazon layout
"#priceblock_dealprice", # Deal prices
"span.a-price-whole", # Visible whole number part
]
for selector in price_selectors:
try:
# Don't check visibility for .a-offscreen elements (they're hidden by design)
element = page.locator(selector).first
# For offscreen elements, just check if they exist and have text
if "offscreen" in selector or "priceblock" in selector:
price_text = await element.inner_text(timeout=2000)
else:
# For visible elements, check visibility first
if await element.is_visible(timeout=2000):
price_text = await element.inner_text()
else:
continue
if price_text and price_text.strip():
logger.info(f"Extracted Amazon price via {selector}: {price_text}")
return price_text.strip()
except Exception:
continue
# Try combination: whole + fraction
try:
whole = await page.locator("span.a-price-whole").first.inner_text()
fraction = await page.locator("span.a-price-fraction").first.inner_text()
if whole and fraction:
price_text = f"{whole}{fraction}"
logger.info(f"Extracted Amazon price from whole+fraction: {price_text}")
return price_text
except Exception:
pass
logger.warning("Could not extract Amazon price with any selector")
return ""
async def get_page_content(self, url: str, use_proxy: bool = False, wait_selector: str = None) -> tuple[str, str]:
"""
Fetch page content with full stealth lifecycle.
@@ -248,6 +300,15 @@ class BrowserlessService:
# Fallback to HTML content if inner_text fails
logger.warning(f"Failed to extract inner_text, falling back to HTML: {e}")
content = await page.content()
# For Amazon, extract price directly and prepend to content
# This helps AI by providing explicit price information
if "amazon" in url.lower() and "/dp/" in url:
amazon_price_text = await self._extract_amazon_price(page)
if amazon_price_text:
content = f"PRIX DÉTECTÉ: {amazon_price_text}\n\n{content}"
logger.info(f"Prepended Amazon price to content: {amazon_price_text}")
# Take screenshot if needed for AI analysis
screenshot_path = ""