- Changed proxy format from dict to string (http://user:pass@ip:port)
- Updated get_random_proxy() to return Crawl4AI-compatible format
- Replaced deprecated 'proxy' with 'proxy_config' in BrowserConfig
- Fixed test script to handle new proxy format
- Added credential hiding in proxy logging for security
Fixes AttributeError: 'dict' object has no attribute 'strip'
- Test basic search functionality
- Test multiple queries with delays
- Test anti-detection system (proxies, user-agents)
- Detailed logging for debugging
- Success/failure reporting
- Created Amazon scraper service with advanced anti-bot techniques:
* User-Agent rotation from realistic pool
* Complete browser headers (Accept, Accept-Language, etc.)
* Proxy rotation (10 residential proxies)
* Random delays (1.5-4s) to mimic human behavior
* Crawl4AI browser fingerprint randomization
* NetworkIdle waiting for complete page load
* Cookie acceptance automation
- Added Amazon search API endpoint with SSE streaming
* Real-time progress updates
* Proper error handling
* Health check endpoint
- Created dedicated Amazon France frontend page:
* Modern UI with product cards
* Rating display (stars + review count)
* Price formatting with discount badges
* Prime badge support
* Stock status indicators
* Sponsored product labels
* Direct Amazon links
- Removed store list (ENSEIGNES_DATA cleared)
* Migration from discount stores to Amazon France
* Catalog system kept for future use
- Updated navigation:
* Added "Amazon France" menu item with ShoppingBag icon
* Positioned between Search and Compare
* Available on desktop and mobile
Technical stack:
- Backend: Crawl4AI + BeautifulSoup for scraping
- Frontend: React + Shadcn UI components
- API: FastAPI with SSE streaming
**Problem:**
AI was detecting old crossed-out prices instead of current prices.
Example: Stokomani Lutin - Current: 19,99€, Old (strikethrough): 14,99€
AI detected: 14.99 EUR (wrong - old price)
**Root Cause:**
Generic price extraction was finding ALL price elements without checking
if they were visually crossed-out (text-decoration: line-through).
Many e-commerce sites show:
```html
<span class="old-price" style="text-decoration: line-through">14,99 €</span>
<span class="price">19,99 €</span>
```
The selector finds both, but we were returning the first found.
**Solution: Check CSS text-decoration**
Added strikethrough detection (browserless_service.py:182-189):
```python
# Check if element is strikethrough (old price)
text_decoration = await element.evaluate(
"el => window.getComputedStyle(el).textDecoration"
)
if "line-through" in text_decoration:
continue # Skip crossed-out prices
```
**Flow:**
```
Found elements with .price selector: [elem1, elem2, elem3]
↓
For each element:
1. Check visibility ✓
2. Check text-decoration
→ "line-through" → SKIP ✓
→ "none" → CONTINUE
3. Extract price text
↓
Return first non-strikethrough price
```
**CSS Patterns Detected:**
- `text-decoration: line-through` (most common)
- `text-decoration: line-through solid`
- Combined styles ignored if they don't contain "line-through"
**Expected Results:**
- Before: Returns first price found (14.99 if it's first in DOM)
- After: Skips strikethrough prices, returns current price (19.99)
**Note:** Existing selector already excludes common classes:
`:not([class*='old']):not([class*='was']):not([class*='original'])`
This adds runtime CSS check as additional safety layer.
Partial fix for Stokomani Lutin 14.99 vs 19.99 issue.
May need site-specific selectors if problem persists.
**Problem:**
AI was misinterpreting French decimal comma as thousands separator:
- "3,99 €" detected as 399.00 EUR (instead of 3.99 EUR)
- "1,99 €" detected as 199.00 EUR (instead of 1.99 EUR)
- Caused by AI reading comma as grouping separator, not decimal point
**Root Cause:**
French format uses comma for decimals: "3,99 €" = three euros ninety-nine cents
English format uses dot for decimals: "3.99 €" = three euros ninety-nine cents
AI models (trained mostly on English) interpret:
- "3,99" → "3" and "99" separate → 399 (three hundred ninety-nine)
- Should be: "3.99" → 3.99 (three point ninety-nine)
**Solution: Pre-Convert Format Before AI Processing**
1. **Normalize ALL Text** (browserless_service.py:380-386)
```python
# Convert French to English in all page text
content = re.sub(r'(\d+),(\d{2})\s*€', r'\1.\2 €', content)
# "3,99 €" → "3.99 €"
# "12,50 €" → "12.50 €"
# "1 234,56 €" → "1234.56 €" (also removes space thousands separator)
```
2. **Normalize Extracted Price** (browserless_service.py:400-410)
```python
# Convert French format in directly extracted price
normalized_price = re.sub(r'(\d+),(\d{2})', r'\1.\2', extracted_price)
# "3,99 €" → "3.99 €"
# Prepends: "PRIX DÉTECTÉ: 3.99 €" (not "3,99 €")
```
3. **Updated AI Prompt** (ai_schema.py:135-158)
- Removed instructions to convert comma→dot (now done automatically)
- Added: "The price has already been converted to English format for you"
- Added explicit warnings about common mistakes:
* "3.99 €" means 3.99 (NOT 399.00, NOT 3990.00)
* "1.99 €" means 1.99 (NOT 199.00, NOT 1990.00)
- Clear examples with both CORRECT and INCORRECT values
**Flow:**
```
Page text: "Lutin 3,99 € Ajouter au panier"
↓
Normalization: "Lutin 3.99 € Ajouter au panier"
↓
Extracted price: "3,99 €" → Normalized: "3.99 €"
↓
Sent to AI: "PRIX DÉTECTÉ: 3.99 €\n\nLutin 3.99 € Ajouter au panier"
↓
AI sees only English format: "3.99"
↓
AI Response: {"price": 3.99} ✅ CORRECT
```
**Expected Results:**
- Before: "3,99 €" → AI returns 399.00 ❌
- After: "3,99 €" → Normalized to "3.99 €" → AI returns 3.99 ✅
**Regex Details:**
- `(\d+),(\d{2})\s*€` matches French prices: captures digits before comma, 2 digits after
- `\1.\2 €` replaces with: same digits, dot instead of comma, space + euro symbol
- Handles: 0,99 / 3,99 / 12,99 / 89,99 / 1234,99
Fixes issue where AI misread French decimal comma causing 100x price errors.
**Problem:**
Price extraction was only working for Amazon. Other sites (B&M, Stokomani, etc.)
had prices missing from extracted text, causing AI to rely solely on screenshots.
Example error:
```
WARNING - No prices found in text with € symbol
AI Response: {"price": 9.99} // Wrong! Real price was 1.99€
```
**Root Cause:**
- `page.inner_text('body')` doesn't capture all price elements
- Some sites use hidden elements, iframes, or CSS pseudo-elements
- Only Amazon had dedicated price extraction (_extract_amazon_price)
- Other sites got: `WARNING - Could not extract price from DOM`
**Solution: Universal Price Extraction**
1. **New Method: _extract_generic_price()** (browserless_service.py:145-208)
- Tries common CSS selectors across all e-commerce sites:
* Class-based: `.price`, `.product-price`, `.current-price`
* Attribute-based: `[data-price]`, `[itemprop='price']`
* French-specific: `[class*='prix']`, `[class*='tarif']`
* Excludes old prices: `:not([class*='old'])`
- Validates price format (contains € or comma with digits)
- Falls back to regex: `\d+[,\.]\d{2}\s*€` on all text
- Returns first valid price found
2. **Updated Extraction Logic** (browserless_service.py:319-338)
- Amazon → uses `_extract_amazon_price()` (specific)
- All other sites → uses `_extract_generic_price()` (universal)
- Prepends "PRIX DÉTECTÉ: X,XX €" to text context
- Logs success/failure for debugging
**Flow:**
```
Page load (bmstores.fr)
↓
Try selectors: .price, .product-price, [itemprop='price']...
↓
Found: "1,99 €" via selector .price
↓
Prepend: "PRIX DÉTECTÉ: 1,99 €\n\n[page text]"
↓
Send to AI
↓
AI sees explicit price → {"price": 1.99, "confidence": 0.95}
```
**Expected Results:**
- Before: `WARNING - No prices found in text with € symbol`
- After: `INFO - Found price via selector .price: 1,99 €`
- AI confidence improves (text + image vs image only)
- Fewer digit recognition errors (1 vs 9)
**Tested with:**
- https://bmstores.fr/produits/accessoire-noel-enfant/125651-lutin-farceur-35cm
- Price: 1,99 €
Fixes issue where non-Amazon sites had missing prices in text context,
causing AI to misread prices from screenshots only.
**Problem:**
AI was misreading prices: "1,99 €" detected as "9,99 €" with 90% confidence.
This is a critical OCR error where similar-looking digits (1, 7, 8, 9) are confused.
The model google/gemini-2.5-flash-lite made basic digit recognition mistakes.
**Root Cause:**
- AI vision models can confuse visually similar digits
- No specific instructions in prompt to verify digit accuracy
- No validation that detected price matches text context
- Model may be too lightweight for reliable OCR
**Solution: Three-Part Fix**
1. **Enhanced Prompt with Digit Verification** (ai_schema.py:145-149)
- Added "CRITICAL - Read digits carefully" section
- Explicit warning: "1,99 €" is NOT "9,99 €"
- Instructions to double-check digits 1, 7, 8, 9 (visually similar)
- Context check: Small prices (< 5€) are common
- Verify price makes sense for product type
2. **More Examples in Prompt** (ai_schema.py:152-153)
- Added: "1,99 €" -> 1.99 (NOT 9.99)
- Added: "0,99 €" -> 0.99 (NOT 9.99)
- Reinforces correct parsing of small prices
3. **Price Extraction Logging** (ai_service.py:349-355)
- Extract ALL prices from text with regex: \d+[,\.]\d{2}\s*€
- Log first 10 prices found: ["1,99 €", "2,99 €", ...]
- Allows debugging: Is correct price in text?
- Logs warning if NO prices found in text
4. **Debug Logs to INFO** (ai_service.py:347, 361)
- Changed logger.debug → logger.info for text/prompt preview
- Now visible in production logs for diagnosis
**Expected Improvement:**
- AI should now pay closer attention to digit shapes
- Lower confidence (0.5-0.7) if digits are unclear
- Better accuracy for small prices (0,99 - 4,99)
- Logs will show if problem is in text extraction or AI vision
**Testing:**
After restart, logs will show:
```
INFO - Cleaned text preview: 'Lutin de Noël 1,99 € Ajouter...'
INFO - Prices found in text: ['1,99 €', '2,99 €']
INFO - AI Response: {"price": 1.99, "price_confidence": 0.85}
```
If AI still returns wrong price but text has correct price,
then model is insufficient and should be upgraded to:
- google/gemini-2.0-flash-exp
- anthropic/claude-3-haiku
- openai/gpt-4o-mini
**Problem:**
Search was launching ALL sites in parallel without any limit, causing Browserless
to be overwhelmed with simultaneous connections and timeout:
- 7 active sites = 7 parallel Browserless connections
- Each site scraping multiple products in parallel
- Result: "Timeout 60000ms exceeded" errors
- Most searches failed with 0 results
**Root Cause:**
```python
# search_service.py:489 (before fix)
for gen in generators:
asyncio.create_task(producer(gen)) # ❌ ALL sites at once
```
With 7 sites × 3 products each = up to 21 concurrent Browserless connections.
**Solution: Two-Level Concurrency Control**
1. **Global Site Limit** (search_service.py:476)
- Added `site_semaphore = asyncio.Semaphore(2)`
- Max 2 sites can search in parallel
- Others wait for a slot to free up
- Example: 7 sites → only 2 active at a time
2. **Per-Site Product Limit** (search_service.py:306)
- Reduced from `Semaphore(3)` to `Semaphore(2)`
- Max 2 products scraped in parallel per site
- Less pressure on Browserless
**Impact:**
- Before: 7 sites × 3 products = 21 concurrent connections ❌
- After: 2 sites × 2 products = 4 concurrent connections ✅
- 80% reduction in concurrent load on Browserless
**Flow:**
```
7 sites requested
↓
2 start searching (sites 1, 2)
5 wait in queue (sites 3-7)
↓
Site 1 completes → Site 3 starts
Site 2 completes → Site 4 starts
↓
Process continues until all sites done
```
**Expected Results:**
✅ No more Browserless timeout errors
✅ All sites complete successfully
✅ Sequential processing prevents saturation
✅ Search returns results from all sites
Fixes issue where search returned "Found 0 results" for most sites
due to Browserless connection timeouts.
LogiFlow
Claude
Michael