Michael/Priceflow
1
0 Fork 0
mirror of https://github.com/R0m1k3/Priceflow.git synced 2026-10-14 02:36:21 +02:00
Code Issues Packages Projects Releases Wiki Activity
516 Commits 10 Branches 0 Tags
a74ea6e9518156726bdaffbd405229deb44fa74a
Commit Graph
258 Commits
Author SHA1 Message Date
LogiFlow a74ea6e951 Merge pull request #220 from R0m1k3/antigravity 
Antigravity
 2025-12-02 09:37:51 +01:00
Michael 86aae8f628 Merge branch 'origin/antigravity' into antigravity 
Resolved conflict in browserless_service.py by merging Amazon interstitial selectors.
Combined local robust selectors with remote additions for maximum coverage.
 2025-12-02 09:37:31 +01:00
Michael 2b21cd176c feat: add TrackingScraperService for Playwright-based web scraping with popup handling and smart scrolling capabilities. 2025-12-02 09:28:08 +01:00
Michael 5516562cb6 fix: Handle Amazon interstitials and dynamic screenshot updates 
- Added Amazon 'Continue' interstitial selectors to BrowserlessService
- Ported robust popup handling from TrackingScraperService to BrowserlessService
- Updated ItemService to fetch dynamic screenshot URLs from PriceHistory
  (fixes issue where forced updates didn't show new screenshots)
 2025-12-02 09:27:12 +01:00
LogiFlow 468ad432b9 Merge pull request #219 from R0m1k3/antigravity 
feat: Add Playwright script to detect and analyze Amazon "Continuer l…
 2025-12-01 17:27:13 +01:00
Michael 9111aa02b9 feat: Add Playwright script to detect and analyze Amazon "Continuer les achats" popup elements. 2025-12-01 17:26:47 +01:00
LogiFlow e198ab72ab Merge pull request #218 from R0m1k3/antigravity 
feat: implement BrowserlessService for unified, persistent Playwright…
 2025-12-01 17:14:35 +01:00
Michael 9178512465 feat: implement BrowserlessService for unified, persistent Playwright browser management with auto-reconnection and enhanced price extraction. 2025-12-01 17:14:12 +01:00
LogiFlow 8b1267cf44 Merge pull request #217 from R0m1k3/antigravity 
feat: Add automated item checking scheduler service and new dashboard…
 2025-12-01 17:09:29 +01:00
Michael 6eb821c40d feat: Add automated item checking scheduler service and new dashboard item display components. 2025-12-01 17:07:50 +01:00
LogiFlow 25646c99fe Merge pull request #216 from R0m1k3/antigravity 
feat: Implement new search service for multi-site e-commerce product …
 2025-12-01 16:57:11 +01:00
Michael f0e1c6668d feat: Implement new search service for multi-site e-commerce product search and item detail scraping. 2025-12-01 16:56:32 +01:00
LogiFlow f47ceec1bf Merge pull request #215 from R0m1k3/antigravity 
feat: Increase Amazon limit to 50 and fix screenshot framing
 2025-12-01 16:19:45 +01:00
Michael f223dc5a29 feat: Increase Amazon limit to 50 and fix screenshot framing 
- Increased default Amazon search results limit from 20 to 50 in backend and frontend
- Improved TrackingScraperService to target main product area on Amazon (avoiding reviews)
- Added specific selectors for Amazon product page (#imgTagWrapperId, #productTitle, etc.)
 2025-12-01 16:18:35 +01:00
LogiFlow b982fab306 Merge pull request #214 from R0m1k3/antigravity 
feat: Enhance Amazon stealth with comprehensive anti-detection
 2025-12-01 13:57:30 +01:00
Michael acfc5c6397 feat: Enhance Amazon stealth with comprehensive anti-detection 
Added comprehensive stealth techniques to bypass Amazon bot detection:
- Complete HTTP headers (Accept, Accept-Language, Sec-Fetch-*, etc.)
- Extended chrome object with loadTimes, csi, app
- Permissions API override
- Plugins, languages, platform spoofing
- Battery API mocking

This significantly improves bot detection evasion compared to the basic
2-line stealth that was insufficient for Amazon's detection systems.
 2025-12-01 13:55:57 +01:00
Michael 2429fa2018 Revert "fix: Remove homepage redirect to avoid Amazon bot detection" 
This reverts commit 5d3d181f5e.
 2025-12-01 13:26:05 +01:00
Michael 5d3d181f5e fix: Remove homepage redirect to avoid Amazon bot detection 
Amazon detects the homepage-then-search pattern as bot behavior and blocks
with 2065 byte pages. Now navigating directly to search URL to appear more
natural.

This reverts the homepage loading strategy from commit 85b19e7 as Amazon's
bot detection has evolved and now flags this pattern.
 2025-12-01 13:25:51 +01:00
Michael 8dd3082a9a Merge branch 'antigravity' into main - Resolve conflicts 
Resolved conflicts by keeping antigravity improvements:
- tracking_scraper_service.py: 47 popup selectors with RGPD/Amazon support
- main.py: No auto-init of TrackingScraperService (on-demand only)

This brings all popup handling improvements to main branch.
 2025-12-01 13:13:58 +01:00
Michael a792060f08 feat: Add TrackingScraperService with enhanced popup handling for screenshots 
- Add 47 popup selectors (vs 12 generic before)
- Support French RGPD platforms (Axeptio, Didomi, OneTrust, TarteAuCitron)
- Amazon-specific detection and handling (6 selectors)
- Retry logic with verification pass
- Double popup cleanup before screenshot
- Increased timeouts for slow animations
- New _verify_no_popups() helper function

This resolves popup visibility issues in the Suivi/Dashboard screenshots.
 2025-12-01 12:27:51 +01:00
Michael 7592c2781e feat: Implement the core FastAPI application, including database migrations, service orchestration, and a new tracking scraper service. 2025-12-01 09:32:37 +01:00
Michael 36ae466583 feat: Introduce ImprovedSearchService utilizing Playwright for persistent browser-based e-commerce scraping and price extraction, along with new search configuration. 2025-12-01 01:30:06 +01:00
Michael b35401a87b feat: Add improved search and AI price extraction services, update Docker Compose, and introduce benchmark verification script. 2025-12-01 01:20:39 +01:00
Michael 8303e767f7 feat: implement a new persistent browser-based search service for e-commerce scraping. 2025-12-01 01:11:42 +01:00
Michael 8bad965e29 feat: Add ImprovedSearchService for persistent browser-based e-commerce search scraping. 2025-12-01 01:07:22 +01:00
Michael 759a28b2d4 feat: Add ImprovedSearchService for persistent browser-based e-commerce search and scraping. 2025-12-01 01:05:40 +01:00
Michael 4dbb9f6424 feat: add improved search service with persistent browser connection 2025-12-01 01:03:51 +01:00
Michael 39893e8c1e feat: Add improved search service with persistent browser connection for e-commerce scraping. 2025-12-01 01:00:25 +01:00
Michael b5c5961618 feat: Implement AI-powered price extraction using Gemma 2 and an improved search service with persistent browser connections for web scraping. 2025-12-01 00:57:05 +01:00
Michael c159036583 feat: Implement comprehensive search configurations for discount stores and add Gifi-specific search and analysis tools. 2025-12-01 00:36:54 +01:00
Michael 63073d03b5 feat: introduce new search configuration, improved search service, and Carrefour search integration test. 2025-11-30 23:54:48 +01:00
Michael bf97cd22e7 feat: introduce improved search service with persistent browser and modular parsers 2025-11-30 23:49:34 +01:00
Michael 7cb0e2f94b feat: add improved search service with persistent browser connection and modular site-specific parsers 2025-11-30 23:45:51 +01:00
Michael 8bf9ddb2d7 feat: implement improved search service using persistent browser connection and modular parsers, alongside a BeautifulSoup check utility. 2025-11-30 23:43:11 +01:00
Michael 87568866e7 feat: introduce improved search service with persistent browser connection and modular site-specific parsers. 2025-11-30 23:40:22 +01:00
Michael 4dd77bd889 feat: add improved search service with persistent browser, modular parsers, and popup handling. 2025-11-30 23:35:40 +01:00
Michael 9855040f3e feat: Implement improved search service with supporting models, schemas, and API routers, and refactor site verification script. 2025-11-30 23:33:05 +01:00
Michael 4720ed6b93 feat: Implement improved search service with persistent browser and modular site-specific parsers, including Gifi. 2025-11-30 22:21:08 +01:00
Michael 64067347d1 feat: implement new search service with multiple site parsers, search configuration, and supporting inspection/verification scripts. 2025-11-30 22:04:14 +01:00
Michael b44082aeb4 feat: Implement new product parsers for Stokomani, Auchan, Carrefour, and Gifi, and add a centralized search configuration module. 2025-11-30 20:26:19 +01:00
Michael e0d73f4d0e feat: add LaFoirFouille parser to extract product search results from lafoirfouille.fr 2025-11-30 19:09:09 +01:00
Michael fbd06cfb17 feat: Add initial parser for Auchan.fr to extract product search results. 2025-11-30 19:05:38 +01:00
Michael 078060f155 feat: Add abstract base parser with product result dataclass and implement Stokomani parser for search results. 2025-11-30 19:03:26 +01:00
Michael ebe685fdf9 feat: add Stokomani parser to extract product search results from stokomani.fr 2025-11-30 18:17:07 +01:00
Michael b3d5c56707 feat: Add Stokomani search results parser 2025-11-30 18:13:11 +01:00
Michael c2f042beee feat: Implement improved search service with centralized configuration and site-specific parsers for various retailers. 2025-11-30 18:05:34 +01:00
Michael a8bffaec08 feat: Implement initial e-commerce site parsers with a base class and factory for extracting product search results. 2025-11-30 17:42:32 +01:00
Michael ee13a0bccd fix: resolve merge conflict in improved_search_service - unified domain matching logic 2025-11-30 17:37:16 +01:00
Michael 3db7db48fa feat: Add improved search service with persistent browser connection for e-commerce scraping. 2025-11-30 17:32:58 +01:00
Claude 43ea314e41 fix: Improve Fnac image extraction and add Shopify fallbacks 
Fixes two issues:

**1. Fnac - No Images Displayed**
Problem: FnacParser only searched for images in parent container,
missing images nested directly in the link element.

Solution: Two-priority search
- PRIORITY 1: Search in link element first (picture, img)
- PRIORITY 2: Search in parent container
- Added warning log when no image found
- Ensures parent is always defined before use

**2. L'Incroyable - 0 Products (Shopify sites)**
Problem: GenericParser fallbacks didn't include Shopify patterns.
L'Incroyable uses Shopify structure with /products/ URLs.

Solution: Added Shopify-specific fallbacks
- a[href*='/products/'] (Shopify product URLs)
- a.product-card__link (Shopify class)
- a[href*='/collections/'] (Shopify collections)
- li[class*='product'] a (list-based layouts)

**Fallback Order (now 17 selectors):**
1. Product URL patterns (Shopify, generic)
2. Semantic classes (article, .product-card)
3. Shopify-specific selectors
4. Generic structure patterns
5. Last resort wildcards

**Expected Impact:**
- ✅ Fnac images should now display
- ✅ L'Incroyable should find products (Shopify)
- ✅ Better coverage of Shopify-based stores
 2025-11-30 16:25:30 +00:00