Michael
f0e1c6668d
feat: Implement new search service for multi-site e-commerce product search and item detail scraping.
2025-12-01 16:56:32 +01:00
Michael
f223dc5a29
feat: Increase Amazon limit to 50 and fix screenshot framing
...
- Increased default Amazon search results limit from 20 to 50 in backend and frontend
- Improved TrackingScraperService to target main product area on Amazon (avoiding reviews)
- Added specific selectors for Amazon product page (#imgTagWrapperId, #productTitle, etc.)
2025-12-01 16:18:35 +01:00
Michael
acfc5c6397
feat: Enhance Amazon stealth with comprehensive anti-detection
...
Added comprehensive stealth techniques to bypass Amazon bot detection:
- Complete HTTP headers (Accept, Accept-Language, Sec-Fetch-*, etc.)
- Extended chrome object with loadTimes, csi, app
- Permissions API override
- Plugins, languages, platform spoofing
- Battery API mocking
This significantly improves bot detection evasion compared to the basic
2-line stealth that was insufficient for Amazon's detection systems.
2025-12-01 13:55:57 +01:00
Michael
2429fa2018
Revert "fix: Remove homepage redirect to avoid Amazon bot detection"
...
This reverts commit 5d3d181f5e
2025-12-01 13:26:05 +01:00
Michael
5d3d181f5e
fix: Remove homepage redirect to avoid Amazon bot detection
...
Amazon detects the homepage-then-search pattern as bot behavior and blocks
with 2065 byte pages. Now navigating directly to search URL to appear more
natural.
This reverts the homepage loading strategy from commit 85b19e7
2025-12-01 13:25:51 +01:00
Michael
95846ec606
fix: Disable ImprovedSearchService auto-init to restore Amazon functionality
...
ImprovedSearchService and AmazonScraperService were both connecting to
Browserless at startup, causing connection conflicts and Amazon blocking
(2065 byte pages).
Now only AmazonScraperService initializes at startup, keeping Browserless
in continuous connection for Amazon. ImprovedSearchService will initialize
on-demand when needed.
This restores Amazon search to working state as it was at commit 85b19e7
2025-12-01 13:21:32 +01:00
Michael
53a05f45e4
fix: Remove auto-initialization of TrackingScraperService to fix Amazon scraper
...
TrackingScraperService now initializes on-demand instead of at startup.
This prevents connection conflicts with AmazonScraperService and ImprovedSearchService
since all services connect to the same Browserless instance.
Fixes: Amazon scraper blocking issue (page too small - 2065 bytes)
2025-12-01 13:05:49 +01:00
Michael
e2a874bf6c
feat: implement TrackingScraperService for robust web scraping with Playwright, including shared browser management and popup handling.
2025-12-01 12:42:14 +01:00
Michael
a792060f08
feat: Add TrackingScraperService with enhanced popup handling for screenshots
...
- Add 47 popup selectors (vs 12 generic before)
- Support French RGPD platforms (Axeptio, Didomi, OneTrust, TarteAuCitron)
- Amazon-specific detection and handling (6 selectors)
- Retry logic with verification pass
- Double popup cleanup before screenshot
- Increased timeouts for slow animations
- New _verify_no_popups() helper function
This resolves popup visibility issues in the Suivi/Dashboard screenshots.
2025-12-01 12:27:51 +01:00
Michael
ffc2f0d632
feat: introduce centralized search configuration for scraping, including site selectors, proxies, and user agents.
2025-12-01 07:40:50 +01:00
Michael
36ae466583
feat: Introduce ImprovedSearchService utilizing Playwright for persistent browser-based e-commerce scraping and price extraction, along with new search configuration.
2025-12-01 01:30:06 +01:00
Michael
b35401a87b
feat: Add improved search and AI price extraction services, update Docker Compose, and introduce benchmark verification script.
2025-12-01 01:20:39 +01:00
Michael
8303e767f7
feat: implement a new persistent browser-based search service for e-commerce scraping.
2025-12-01 01:11:42 +01:00
Michael
8bad965e29
feat: Add ImprovedSearchService for persistent browser-based e-commerce search scraping.
2025-12-01 01:07:22 +01:00
Michael
759a28b2d4
feat: Add ImprovedSearchService for persistent browser-based e-commerce search and scraping.
2025-12-01 01:05:40 +01:00
Michael
4dbb9f6424
feat: add improved search service with persistent browser connection
2025-12-01 01:03:51 +01:00
Michael
93840804a4
feat: implement product search page with site selection, real-time results, and monitoring integration
2025-12-01 01:01:53 +01:00
Michael
39893e8c1e
feat: Add improved search service with persistent browser connection for e-commerce scraping.
2025-12-01 01:00:25 +01:00
Michael
b5c5961618
feat: Implement AI-powered price extraction using Gemma 2 and an improved search service with persistent browser connections for web scraping.
2025-12-01 00:57:05 +01:00
Michael
c159036583
feat: Implement comprehensive search configurations for discount stores and add Gifi-specific search and analysis tools.
2025-12-01 00:36:54 +01:00
Michael
63073d03b5
feat: introduce new search configuration, improved search service, and Carrefour search integration test.
2025-11-30 23:54:48 +01:00
Michael
bf97cd22e7
feat: introduce improved search service with persistent browser and modular parsers
2025-11-30 23:49:34 +01:00
Michael
7cb0e2f94b
feat: add improved search service with persistent browser connection and modular site-specific parsers
2025-11-30 23:45:51 +01:00
Michael
8bf9ddb2d7
feat: implement improved search service using persistent browser connection and modular parsers, alongside a BeautifulSoup check utility.
2025-11-30 23:43:11 +01:00
Michael
87568866e7
feat: introduce improved search service with persistent browser connection and modular site-specific parsers.
2025-11-30 23:40:22 +01:00
Michael
4dd77bd889
feat: add improved search service with persistent browser, modular parsers, and popup handling.
2025-11-30 23:35:40 +01:00
Michael
9855040f3e
feat: Implement improved search service with supporting models, schemas, and API routers, and refactor site verification script.
2025-11-30 23:33:05 +01:00
Michael
4720ed6b93
feat: Implement improved search service with persistent browser and modular site-specific parsers, including Gifi.
2025-11-30 22:21:08 +01:00
Michael
64067347d1
feat: implement new search service with multiple site parsers, search configuration, and supporting inspection/verification scripts.
2025-11-30 22:04:14 +01:00
Michael
95e38287cb
feat: Add new parsers for Auchan, Carrefour, L'Incroyable, and Stokomani, including search configuration and deployment updates.
2025-11-30 22:03:40 +01:00
Michael
b44082aeb4
feat: Implement new product parsers for Stokomani, Auchan, Carrefour, and Gifi, and add a centralized search configuration module.
2025-11-30 20:26:19 +01:00
Michael
e0d73f4d0e
feat: add LaFoirFouille parser to extract product search results from lafoirfouille.fr
2025-11-30 19:09:09 +01:00
Michael
fbd06cfb17
feat: Add initial parser for Auchan.fr to extract product search results.
2025-11-30 19:05:38 +01:00
Michael
078060f155
feat: Add abstract base parser with product result dataclass and implement Stokomani parser for search results.
2025-11-30 19:03:26 +01:00
Michael
ebe685fdf9
feat: add Stokomani parser to extract product search results from stokomani.fr
2025-11-30 18:17:07 +01:00
Michael
b3d5c56707
feat: Add Stokomani search results parser
2025-11-30 18:13:11 +01:00
Michael
c2f042beee
feat: Implement improved search service with centralized configuration and site-specific parsers for various retailers.
2025-11-30 18:05:34 +01:00
Michael
a8bffaec08
feat: Implement initial e-commerce site parsers with a base class and factory for extracting product search results.
2025-11-30 17:42:32 +01:00
Michael
ee13a0bccd
fix: resolve merge conflict in improved_search_service - unified domain matching logic
2025-11-30 17:37:16 +01:00
Michael
3db7db48fa
feat: Add improved search service with persistent browser connection for e-commerce scraping.
2025-11-30 17:32:58 +01:00
Michael
33d9042b10
feat: add search_config module for browserless, proxy, user agent, and site-specific search configurations, and create list_missing_sites script.
2025-11-30 16:57:44 +01:00
Michael
7ba8850838
feat: Implement new search service with comprehensive configuration for various e-commerce sites, proxies, and user agents.
2025-11-30 16:36:51 +01:00
Michael
f20bae52bd
feat: introduce core search configuration for sites, proxies, and user agents, alongside selector diagnosis and HTML dumping utilities.
2025-11-30 16:27:02 +01:00
Michael
79dc4bfafc
feat: Implement centralized search configuration with site definitions, proxy settings, and user agents, and add a debugging script.
2025-11-30 15:58:27 +01:00
Michael
7477d505e5
feat: Add ImprovedSearchService for persistent browser-based web scraping using Playwright.
2025-11-30 15:03:03 +01:00
Michael
510904d899
feat: Add Amazon product scraping service using Playwright and Browserless for persistent browser connections.
2025-11-30 14:56:01 +01:00
Michael
66d6f14d13
Merge branch 'antigravity' of https://github.com/R0m1k3/Priceflow into antigravity
2025-11-30 14:46:22 +01:00
Michael
e9f6c6e0be
feat: Add BrowserlessService for persistent, auto-reconnecting browser operations with enhanced price and content extraction.
2025-11-30 14:46:21 +01:00
Michael
3813173e01
feat: add search_config module centralizing site definitions, proxy settings, user agents, and cookie banner selectors.
2025-11-30 14:14:31 +01:00
Michael
b8a56ca85d
Merge branch 'antigravity' of https://github.com/R0m1k3/Priceflow into antigravity
2025-11-30 13:25:15 +01:00
Michael
c6b9befc9a
feat: introduce BrowserlessService for persistent, auto-reconnecting browser operations and enhanced price extraction
2025-11-30 13:25:13 +01:00
Michael
c93e55519e
Merge main: Resolved conflicts by keeping enhanced browserless service with auto-reconnection + price extraction
2025-11-30 12:49:40 +01:00
Michael
2710388faf
feat: Implement scheduled item checks with AI price extraction, database updates, and price change notifications.
2025-11-30 12:48:30 +01:00
Michael
93d73dbed8
feat: Implement BrowserlessService for robust browser automation with auto-reconnection, popup handling, and price extraction capabilities.
2025-11-30 12:43:22 +01:00
Michael
23afe4bc6c
feat: Add new services for search orchestration, browser automation, and scheduling.
2025-11-30 12:39:26 +01:00
Michael
606fbf1523
feat: Implement Catalogues page for displaying, filtering, searching, and managing promotional catalogues.
2025-11-30 11:32:28 +01:00
Michael
99650a992b
feat: Add Cataloguemate.fr scraper to replace Tiendeo and introduce new Catalogues frontend page.
2025-11-30 11:17:28 +01:00
Michael
03d79e45fb
feat: Introduce Cataloguemate.fr scraper service and associated catalogue administration UI.
2025-11-30 10:49:47 +01:00
Michael
d772e85ae4
feat: Add catalogue and enseigne API endpoints with scraping and scheduling services.
2025-11-30 10:39:16 +01:00
Michael
df8fbded92
feat: add Cataloguemate.fr scraper to replace Tiendeo and utilize BrowserlessService for robust scraping.
2025-11-30 02:47:11 +01:00
Michael
cde1f75d3a
feat: Add Cataloguemate.fr scraper service to replace Tiendeo, utilizing BrowserlessService for robust scraping.
2025-11-30 02:45:24 +01:00
Michael
101e5f0656
feat: add Cataloguemate.fr scraper to replace Tiendeo and utilize BrowserlessService for catalog and page scraping
2025-11-30 02:42:53 +01:00
Michael
efb161a5ef
feat: Add Cataloguemate.fr scraper, replacing Tiendeo and utilizing BrowserlessService for catalog and page scraping.
2025-11-30 02:35:27 +01:00
Michael
11d297764d
feat: Add API endpoints for managing and querying catalogues, enseignes, and scraping statistics.
2025-11-30 02:33:15 +01:00
Michael
ec97822087
feat: Implement Cataloguemate.fr scraper using BrowserlessService to replace Tiendeo and fetch promotional catalogs.
2025-11-30 02:29:58 +01:00
Michael
1176eec95a
feat: add Cataloguemate.fr scraper service for promotional catalogs.
2025-11-30 02:13:42 +01:00
Michael
69cfa4cfa9
feat: add Cataloguemate.fr scraper to replace Tiendeo for improved reliability and simpler structure.
2025-11-30 02:09:24 +01:00
Michael
c6dd114584
feat: Add Cataloguemate.fr scraper service to replace Tiendeo for promotional catalog data.
2025-11-30 02:05:40 +01:00
Michael
be3de83601
feat: add script to debug catalog list and page content extraction for cataloguemate.fr
2025-11-30 02:00:59 +01:00
Michael
af096c7980
feat: Implement Cataloguemate.fr scraper service, add catalogue router and scheduler, and remove Tiendeo scraper verification.
2025-11-30 01:56:56 +01:00
Michael
544825e354
feat: add Tiendeo.fr catalog scraper service using Crawl4AI for promotional catalogs.
2025-11-30 01:33:46 +01:00
Michael
607f47c0f0
feat: add Tiendeo.fr scraper service using Crawl4AI for promotional catalog extraction and parsing.
2025-11-30 01:29:00 +01:00
Michael
2b974aacf9
feat: Implement Tiendeo.fr scraper using Crawl4AI to extract promotional catalogs and their pages.
2025-11-30 01:17:40 +01:00
Michael
2692f76cf0
feat: add Tiendeo.fr scraper service for promotional catalogs using Crawl4AI
2025-11-30 01:12:25 +01:00
Michael
79612690b3
feat: add Tiendeo scraper service with Crawl4AI and update project dependencies and Dockerfile.
2025-11-30 01:06:08 +01:00
Michael
34ac9b343a
feat: add Bonial and Tiendeo catalog scrapers
2025-11-30 00:22:36 +01:00
Michael
a7aefb3e56
feat: add Bonial.fr catalog scraper service using Playwright.
2025-11-29 23:59:50 +01:00
Michael
1fc0c6e69e
feat: Implement Bonial.fr scraper service to extract promotional catalogs using Playwright.
2025-11-29 23:55:44 +01:00
Michael
2f492cfe03
feat: Implement Bonial.fr scraper service using Playwright to extract promotional catalogs.
2025-11-29 23:45:30 +01:00
Michael
468b11c195
feat: Add Bonial.fr scraper service using Playwright to extract promotional catalogs and their details.
2025-11-29 23:33:36 +01:00
Michael
b7f049d414
feat: Implement Bonial.fr scraper service using Playwright to extract promotional catalogs and their details.
2025-11-29 23:25:44 +01:00
Michael
6ed923c1ee
feat: add CatalogueAdmin component to manage and monitor catalogue scraping operations.
2025-11-29 23:17:28 +01:00
Michael
09a647a437
feat: Implement API endpoints for retrieving enseignes, paginated catalogues, detailed catalogue information, catalogue pages, and scraping statistics.
2025-11-29 23:11:28 +01:00
Michael
237e4a2adf
feat: add Vite configuration with React plugin and path aliases.
2025-11-29 23:02:52 +01:00
Michael
5be93653ef
feat: add Vite configuration for React project, including path aliases and build output directory.
2025-11-29 22:59:40 +01:00
Michael
efb5211621
feat: add CatalogueAdmin component to manage catalogue scraping, display statistics, and view logs.
2025-11-29 22:55:09 +01:00
Michael
5c0d5cebf7
feat: Add Catalogues page for displaying, searching, and filtering promotional catalogues by enseigne.
2025-11-29 22:50:44 +01:00
Michael
b224c4988a
feat: implement API router for catalogue and enseigne management with filtering and pagination
2025-11-29 22:39:34 +01:00
Michael
3d3d7c18c1
feat: implement Bonial.fr scraper service for promotional catalogs using Playwright.
2025-11-29 22:31:24 +01:00
Michael
ba932b79ef
feat: add Bonial.fr catalog scraper service using Playwright for promotional catalog data extraction.
2025-11-29 20:35:23 +01:00
Michael
da90ea12f2
feat: implement Bonial.fr catalog scraper using Playwright
2025-11-29 20:04:06 +01:00
Michael
c011f1d6bf
feat: Add Bonial.fr scraper service to fetch promotional catalogs using Playwright.
2025-11-29 19:57:32 +01:00
Michael
c8fcfd7a4d
feat: Add Bonial.fr scraper service for promotional catalogs.
2025-11-29 19:51:34 +01:00
Michael
debc1a973f
feat: implement scheduled catalog scraping service
2025-11-29 19:36:52 +01:00
Michael
1bdeb272d6
fix: Correct alembic upgrade command from head to heads.
2025-11-29 19:34:58 +01:00
Michael
b9059aeeda
chore: Add Alembic merge migration for catalog module and notification channels.
2025-11-29 19:32:54 +01:00
Michael
58e6be3f0d
feat: Add initial application structure, core data models, and a new catalog module with Bonial scraping capabilities.
2025-11-29 19:26:56 +01:00
Michael
906c6b483d
feat: add core search configuration including Browserless settings, proxies, user agents, site-specific selectors, and cookie banner handlers.
2025-11-29 18:28:20 +01:00
Michael
a0a48ced19
feat: add script to verify search service functionality across all configured sites, including Browserless connection and detailed reporting.
2025-11-29 18:15:12 +01:00
Michael
b47af1655d
feat: add debug script to display search site configurations and database entries
2025-11-29 17:46:20 +01:00
Michael