**Problem:**
Search was launching ALL sites in parallel without any limit, causing Browserless
to be overwhelmed with simultaneous connections and timeout:
- 7 active sites = 7 parallel Browserless connections
- Each site scraping multiple products in parallel
- Result: "Timeout 60000ms exceeded" errors
- Most searches failed with 0 results
**Root Cause:**
```python
# search_service.py:489 (before fix)
for gen in generators:
asyncio.create_task(producer(gen)) # ❌ ALL sites at once
```
With 7 sites × 3 products each = up to 21 concurrent Browserless connections.
**Solution: Two-Level Concurrency Control**
1. **Global Site Limit** (search_service.py:476)
- Added `site_semaphore = asyncio.Semaphore(2)`
- Max 2 sites can search in parallel
- Others wait for a slot to free up
- Example: 7 sites → only 2 active at a time
2. **Per-Site Product Limit** (search_service.py:306)
- Reduced from `Semaphore(3)` to `Semaphore(2)`
- Max 2 products scraped in parallel per site
- Less pressure on Browserless
**Impact:**
- Before: 7 sites × 3 products = 21 concurrent connections ❌
- After: 2 sites × 2 products = 4 concurrent connections ✅
- 80% reduction in concurrent load on Browserless
**Flow:**
```
7 sites requested
↓
2 start searching (sites 1, 2)
5 wait in queue (sites 3-7)
↓
Site 1 completes → Site 3 starts
Site 2 completes → Site 4 starts
↓
Process continues until all sites done
```
**Expected Results:**
✅ No more Browserless timeout errors
✅ All sites complete successfully
✅ Sequential processing prevents saturation
✅ Search returns results from all sites
Fixes issue where search returned "Found 0 results" for most sites
due to Browserless connection timeouts.
**Problem:**
Previous commit changed get_page_content() to always return visible text (inner_text)
instead of HTML source. This broke product search which uses BeautifulSoup to parse
HTML with CSS selectors. Search was returning 0 results for all sites.
**Root Cause:**
- Search service uses: `soup.select(config["product_selector"])` on HTML
- But was receiving plain text instead of HTML structure
- BeautifulSoup couldn't find any product links → 0 results
**Solution: Add extract_text parameter**
Modified `get_page_content()` signature:
```python
async def get_page_content(
url: str,
use_proxy: bool = False,
wait_selector: str = None,
extract_text: bool = False # NEW parameter
) -> tuple[str, str]:
```
**Behavior:**
- `extract_text=False` (DEFAULT): Returns HTML source via `page.content()`
→ Used by search service for BeautifulSoup parsing
- `extract_text=True`: Returns visible text via `page.inner_text('body')`
→ Used by AI monitoring for price extraction
→ Includes Amazon price prepending ("PRIX DÉTECTÉ: ...")
**Changes:**
1. **browserless_service.py** (Lines 197-329)
- Added `extract_text` parameter with default `False`
- Conditional logic: if extract_text, use inner_text + Amazon extraction
- Else: use page.content() (HTML source)
- Preserves backward compatibility (default = HTML)
2. **scheduler_service.py** (Line 113)
- Added `extract_text=True` for AI monitoring
- Ensures AI gets visible text with Amazon prices
3. **search_service.py** (Line 104)
- Added `extract_text=True` for scrape_item() AI analysis
- Search itself uses default (HTML) for BeautifulSoup parsing
**Impact:**
✅ Search now works again (gets HTML for BeautifulSoup)
✅ AI monitoring gets visible text (better extraction)
✅ Amazon price extraction only runs when extract_text=True
✅ Backward compatible (default behavior = HTML)
Fixes issue where all searches returned: "Found 0 results for [site]"
- Add debug_enabled field to SearchSite model
- Update schemas and API routes to allow debug toggle
- Add debug toggle button on site cards in Admin UI
- Only dump HTML when debug is enabled for the site (except for critical errors like Amazon blocking)
- Add API endpoints to list/delete debug files per site domain
- Add button to clear debug files for individual sites
- Remove ability to add/edit/delete search sites from settings UI
- Configure 16 French e-commerce sites with search URLs:
- Discount: Gifi, Stokomani, B&M, Centrakor, L'Incroyable, Action, La Foir'Fouille
- Grandes Surfaces: E.Leclerc, Auchan, Carrefour
- E-commerce: Amazon France, Cdiscount
- Électronique: Darty, Boulanger, Fnac
- Add automatic database seeding on application startup
- Add /api/search-sites/seed and /api/search-sites/reset endpoints
- Add cookie consent banner handling for sites that require it
- Simplify Settings page to only show site list with toggle for active state
- Add search URLs for French discount stores: stokomani, gifi, bmstores,
centrakor, lafoirfouille, tedi, leclerc, carrefour, auchan
- Fix screenshot URL construction in search results
- Include screenshot_url in AI extraction results and error cases
- Use AIService.analyze_image() class method instead of module function
- Handle AIExtractionResponse tuple return type properly
- Add _clean_domain() to strip protocol/www from stored domains
- Fixes "https://https://" URL duplication issue
- Remove duckduckgo-search dependency (unreliable results)
- Add direct_search_service.py that scrapes sites' search pages directly
- Add search_url and product_link_selector fields to SearchSite model
- Add beautifulsoup4 and lxml for HTML parsing
- Update schemas with new fields
- Add database migration for new columns
Direct search is more reliable as it queries each site's own search
functionality instead of relying on external search engines.
- Remove SearXNG container and configuration files
- Add duckduckgo-search library for web search
- Create duckduckgo_service.py with async search support
- Update search_service.py to use DuckDuckGo instead of SearXNG
- Fix BROWSERLESS_URL port to 3012
This eliminates the persistent limiter.toml directory issue and
provides a simpler, more reliable search solution.
This commit adds a comprehensive product search feature that allows users
to search for products across multiple configured e-commerce sites.
Backend changes:
- Add SearXNG service to docker-compose for meta-search capability
- Add SearchSite model and Alembic migration with default sites
- Create searxng_service.py for search queries across sites
- Create light_scraper_service.py for fast HTTP-based extraction
- Create search_service.py orchestrator combining SearXNG + HTTP + Browserless
- Add API endpoints for search (/api/search) with SSE streaming
- Add CRUD endpoints for search sites (/api/search-sites)
Frontend changes:
- Add new /search page with real-time results display
- Create SearchResultCard component for product display
- Add Progress, Checkbox, and Badge UI components
- Update Settings page with search sites management section
- Add navigation link to search page
- Create new modern SVG logo with price-flow gradient design
- Update favicon to SVG format
The search system uses a hybrid approach:
1. SearXNG finds product URLs across configured sites
2. Light HTTP scraper attempts fast extraction
3. Falls back to Browserless for JavaScript-heavy sites
4. Results stream progressively via SSE
Michael
Claude