Claude
f2d45f9566
fix: Improve AI product price extraction for French e-commerce sites
...
Three critical fixes to resolve AI extraction returning null prices with 0.0 confidence:
1. **Add French price/stock keywords** (app/utils/text.py)
- Added € symbol and French price patterns (12,99 €, 1 234,56 €)
- Added French keywords: prix, coût, promotion, réduction
- Added French stock keywords: ajouter au panier, en stock, rupture, épuisé
- Previously only detected $ and English keywords
2. **Enable full-page screenshots** (app/services/browserless_service.py)
- Changed full_page=False to full_page=True
- Now captures entire page instead of viewport-only
- Prevents missing prices located below the fold
3. **Increase image resolution** (app/utils/image.py)
- Increased MAX_IMAGE_SIZE from 1024px to 2048px
- Preserves fine details in price text for better AI recognition
- Improves readability on dense product pages
These changes fix the issue where AI models returned:
{"price": null, "price_confidence": 0.0, "in_stock": null, "in_stock_confidence": 0.0}
Tested with Amazon.fr product monitoring.
2025-11-29 19:07:04 +00:00
Michael
c011f1d6bf
feat: Add Bonial.fr scraper service to fetch promotional catalogs using Playwright.
2025-11-29 19:57:32 +01:00
Michael
c8fcfd7a4d
feat: Add Bonial.fr scraper service for promotional catalogs.
2025-11-29 19:51:34 +01:00
Michael
debc1a973f
feat: implement scheduled catalog scraping service
2025-11-29 19:36:52 +01:00
Michael
b9059aeeda
chore: Add Alembic merge migration for catalog module and notification channels.
2025-11-29 19:32:54 +01:00
Michael
58e6be3f0d
feat: Add initial application structure, core data models, and a new catalog module with Bonial scraping capabilities.
2025-11-29 19:26:56 +01:00
Michael
906c6b483d
feat: add core search configuration including Browserless settings, proxies, user agents, site-specific selectors, and cookie banner handlers.
2025-11-29 18:28:20 +01:00
Michael
b47af1655d
feat: add debug script to display search site configurations and database entries
2025-11-29 17:46:20 +01:00
Michael
5d7f872dae
feat: Add new search service for multi-site product search and item detail scraping.
2025-11-29 17:28:59 +01:00
Michael
74bdcfa63e
feat: introduce search configuration for sites, proxies, user agents, and cookie banners.
2025-11-29 17:22:28 +01:00
Michael
420dd3144a
feat: add NewSearchService to orchestrate multi-site e-commerce searches and scrape item details.
2025-11-29 17:10:20 +01:00
Michael
48d1ab3a05
feat: introduce new search service to orchestrate multi-site product searches and scrape item details.
2025-11-29 14:17:46 +01:00
Michael
30dfb44517
feat: Add new multi-site search service with browserless and AI integration, including site-specific configurations.
2025-11-29 13:52:07 +01:00
Michael
8399b819da
feat: Add centralized search configuration for e-commerce sites and a verification script.
2025-11-29 13:21:45 +01:00
Michael
f976ae1127
feat: Implement web scraping and search system with centralized configuration, browserless integration, and a verification script.
2025-11-29 13:09:53 +01:00
Michael
7f85a9764c
feat: Add Stokomani raw HTML data and new search configuration.
2025-11-29 11:25:49 +01:00
Michael
66a8371665
feat: Add centralized search configuration including site definitions, proxies, user agents, and cookie acceptance selectors.
2025-11-29 11:09:48 +01:00
Michael
0f5c6dc1c6
feat: introduce NewSearchService to orchestrate e-commerce searches and scrape product details.
2025-11-29 11:05:01 +01:00
Michael
97a8e7a951
feat: implement web scraping debugger and consolidate search site configurations.
2025-11-29 10:56:43 +01:00
Michael
bcc3b0900f
feat: Add core search configuration and service for web scraping, including site-specific selectors, proxies, and user agents.
2025-11-29 10:32:01 +01:00
Michael
292eb3d12e
feat: Add NewSearchService for orchestrating e-commerce searches, including site-specific parsing and AI-powered item detail scraping.
2025-11-29 09:59:58 +01:00
Michael
22f8aedaa4
feat: Add centralized search configuration for sites, proxies, and user agents, and create browserless service module.
2025-11-29 09:51:10 +01:00
Michael
f665a000af
feat: add AI service for content extraction and Browserless service for web scraping
2025-11-29 09:46:29 +01:00
Michael
86559ff2d7
feat: implement NewSearchService for orchestrating e-commerce searches, parsing results, and scraping item details.
2025-11-29 09:37:33 +01:00
Michael
867274a9ab
feat: Introduce AI service for data extraction, including dynamic configuration, response parsing, and repair.
2025-11-29 09:27:35 +01:00
Michael
8d903202fb
feat: introduce browserless, scheduler, and search services for advanced web scraping and data collection.
2025-11-29 09:02:34 +01:00
Michael
654145d499
Merge origin/main into antigravity: Resolve conflict by accepting deletion of direct_search_service.py
2025-11-29 08:57:48 +01:00
Michael
6019a80de9
feat: Add core search configuration for site definitions, proxies, and user agents, and introduce a browserless service.
2025-11-29 08:54:05 +01:00
Michael
b6c1fc1cda
feat: Add AI-powered data extraction and direct search services
2025-11-29 02:53:35 +01:00
Michael
a6662e8891
feat: Add direct e-commerce search service using Playwright/Browserless with anti-bot measures and Amazon-specific handling.
2025-11-29 02:38:18 +01:00
Michael
352fc72bfa
feat: Implement direct e-commerce search service using Playwright and Browserless with anti-bot and proxy rotation capabilities.
2025-11-29 02:31:30 +01:00
Michael
fa890f67dc
feat: add direct e-commerce search service using Playwright with anti-bot measures for scraping.
2025-11-29 02:24:20 +01:00
Michael
bad2cfca20
feat: Add direct e-commerce search service using Playwright with anti-bot measures including rotating proxies, user agents, and stealth JavaScript.
2025-11-29 02:17:17 +01:00
Michael
87165a8b9b
feat: Implement direct search service with Playwright/Browserless, including anti-bot measures, proxy rotation, and stealth JavaScript.
2025-11-29 02:07:59 +01:00
Michael
394fa3c118
feat: implement AI service for data extraction with dynamic configuration and robust response handling.
2025-11-29 02:00:42 +01:00
Michael
091caad347
feat: add web scraping and search services using Playwright with anti-detection mechanisms.
2025-11-29 01:56:25 +01:00
Michael
7a73650264
feat: add DirectSearchService for robust e-commerce product search using Playwright/Browserless with advanced bot detection evasion.
2025-11-29 01:52:31 +01:00
Michael
e6bf46659d
refactor: Improve browser lifecycle management in ScraperService to prevent connection errors
2025-11-29 01:39:43 +01:00
Michael
0439b74bc3
fix: Remove duplicate 'finally' block in scraper_service.py causing SyntaxError
2025-11-29 01:24:34 +01:00
Michael
405756963a
fix: Resolve IndentationError in scraper_service.py caused by loop removal
2025-11-29 01:20:58 +01:00
Michael
aa9d2af5b2
feat: Implement hybrid proxy strategy for Amazon scraping - fallback to Direct IP on last retry
2025-11-29 01:17:03 +01:00
Michael
dd558b0e43
fix: Correct proxy usage in scraper_service.py to handle dict format
2025-11-29 01:07:02 +01:00
Michael
4c410ef48f
fix: Correct Playwright proxy format - use dict with server/username/password instead of URL string
2025-11-29 01:02:41 +01:00
Michael
df2191cdbd
debug: Add proxy initialization logs to verify proxy loading
2025-11-29 00:53:55 +01:00
Michael
8080a97e25
feat: Add same 10 rotating proxies to scraper_service for individual product pages
2025-11-29 00:50:08 +01:00
Michael
b7e373d9eb
feat: Add 10 rotating proxies for Amazon to bypass IP bans
2025-11-29 00:47:17 +01:00
Michael
26df52b664
revert: Use Amazon config from commit 698dbaa (15 commits back)
2025-11-29 00:40:27 +01:00
Michael
3c4094d4de
revert: Back to working Amazon configuration from ee8a146 - simpler is better
2025-11-29 00:39:25 +01:00
Michael
c88f52c136
fix: Add ref=nb_sb_noss parameter to Amazon URLs as in working version + increase retry delays
2025-11-29 00:34:09 +01:00
Michael
eb48c4aa33
feat: Add direct search service for e-commerce scraping via Browserless with bot evasion.
2025-11-29 00:30:12 +01:00
Claude
Michael