Michael/Priceflow
1
0 Fork 0
mirror of https://github.com/R0m1k3/Priceflow.git synced 2026-10-11 17:29:14 +02:00
Code Issues Packages Projects Releases Wiki Activity
536 Commits 10 Branches 0 Tags
4e9f2662bcf910363cc6605729affc9b67999ded
Commit Graph
536 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Michael 4e9f2662bc feat: add Playwright-based scraping service for tracking items. 2025-12-18 11:11:30 +01:00
Michael 5428459f36 feat: Add web scraping service for tracking and AI extraction verification script. 2025-12-18 11:05:09 +01:00
Michael 6fab64b229 feat: Add ScraperService with Playwright for robust web scraping, including browser management, configurable scraping options, and automated popup handling. 2025-12-18 10:56:06 +01:00
Michael 0d68859a5b feat: add Playwright script to scrape gifi.fr product page and extract price information. 2025-12-18 10:42:31 +01:00
Michael 97664a4a16 fix: Improve ItemService screenshot management by prioritizing latest timestamped files and ensuring complete deletion. 2025-12-18 08:50:45 +01:00
Michael 4590273801 docs: revise task.md to address screenshot update caching instead of popup obscuration. 2025-12-18 08:34:02 +01:00
Michael ed7f23d657 chore: Update task plan to focus on removing obscuring popups from product screenshots. 2025-12-18 08:14:56 +01:00
Michael e930f4bbfa feat: Implement main FastAPI application entry point, including database migrations, data seeding, and background task scheduling. 2025-12-02 18:52:41 +01:00
Michael 947152e132 feat: Implement scheduler_service for automated item price and stock tracking, including AI analysis and database updates. 2025-12-02 18:03:32 +01:00
Michael fb89af5aba Resolve merge conflict in scheduler_service.py 2025-12-02 17:52:48 +01:00
Michael 2c0a573a66 feat: Add web scraping service using Playwright and Browserless, and a new scheduler service. 2025-12-02 17:15:28 +01:00
LogiFlow 3c87a61e53 Merge pull request #224 from R0m1k3/antigravity 
Antigravity
 2025-12-02 17:06:32 +01:00
Michael d8b3014021 Resolve merge conflict in browserless_service.py: Keep navigation logic and fix screenshot path 2025-12-02 17:04:49 +01:00
Michael 241970cf70 feat: implement BrowserlessService for persistent browser management with auto-reconnection and enhanced price extraction. 2025-12-02 16:54:40 +01:00
LogiFlow 7a06338313 Merge pull request #223 from R0m1k3/antigravity 
feat: Introduce BrowserlessService for persistent web scraping with e…
 2025-12-02 14:00:33 +01:00
Michael 8a8ef02168 feat: Introduce BrowserlessService for persistent web scraping with enhanced Amazon price extraction, alongside initial scheduler service and task tracking. 2025-12-02 13:59:39 +01:00
LogiFlow 1512b3b967 Merge pull request #222 from R0m1k3/antigravity 
feat: implement persistent browserless service for robust web scrapin…
 2025-12-02 11:46:58 +01:00
Michael 3bbab058c6 feat: implement persistent browserless service for robust web scraping and price extraction with auto-reconnection. 2025-12-02 11:46:32 +01:00
LogiFlow 9d056b525f Merge pull request #221 from R0m1k3/antigravity 
feat: Add French localization file with comprehensive translations fo…
 2025-12-02 09:52:20 +01:00
Michael e7875f35a7 feat: Add French localization file with comprehensive translations for the application UI. 2025-12-02 09:51:56 +01:00
LogiFlow a74ea6e951 Merge pull request #220 from R0m1k3/antigravity 
Antigravity
 2025-12-02 09:37:51 +01:00
Michael 86aae8f628 Merge branch 'origin/antigravity' into antigravity 
Resolved conflict in browserless_service.py by merging Amazon interstitial selectors.
Combined local robust selectors with remote additions for maximum coverage.
 2025-12-02 09:37:31 +01:00
Michael 2b21cd176c feat: add TrackingScraperService for Playwright-based web scraping with popup handling and smart scrolling capabilities. 2025-12-02 09:28:08 +01:00
Michael 5516562cb6 fix: Handle Amazon interstitials and dynamic screenshot updates 
- Added Amazon 'Continue' interstitial selectors to BrowserlessService
- Ported robust popup handling from TrackingScraperService to BrowserlessService
- Updated ItemService to fetch dynamic screenshot URLs from PriceHistory
  (fixes issue where forced updates didn't show new screenshots)
 2025-12-02 09:27:12 +01:00
LogiFlow 468ad432b9 Merge pull request #219 from R0m1k3/antigravity 
feat: Add Playwright script to detect and analyze Amazon "Continuer l…
 2025-12-01 17:27:13 +01:00
Michael 9111aa02b9 feat: Add Playwright script to detect and analyze Amazon "Continuer les achats" popup elements. 2025-12-01 17:26:47 +01:00
LogiFlow e198ab72ab Merge pull request #218 from R0m1k3/antigravity 
feat: implement BrowserlessService for unified, persistent Playwright…
 2025-12-01 17:14:35 +01:00
Michael 9178512465 feat: implement BrowserlessService for unified, persistent Playwright browser management with auto-reconnection and enhanced price extraction. 2025-12-01 17:14:12 +01:00
LogiFlow 8b1267cf44 Merge pull request #217 from R0m1k3/antigravity 
feat: Add automated item checking scheduler service and new dashboard…
 2025-12-01 17:09:29 +01:00
Michael 6eb821c40d feat: Add automated item checking scheduler service and new dashboard item display components. 2025-12-01 17:07:50 +01:00
LogiFlow 25646c99fe Merge pull request #216 from R0m1k3/antigravity 
feat: Implement new search service for multi-site e-commerce product …
 2025-12-01 16:57:11 +01:00
Michael f0e1c6668d feat: Implement new search service for multi-site e-commerce product search and item detail scraping. 2025-12-01 16:56:32 +01:00
LogiFlow f47ceec1bf Merge pull request #215 from R0m1k3/antigravity 
feat: Increase Amazon limit to 50 and fix screenshot framing
 2025-12-01 16:19:45 +01:00
Michael f223dc5a29 feat: Increase Amazon limit to 50 and fix screenshot framing 
- Increased default Amazon search results limit from 20 to 50 in backend and frontend
- Improved TrackingScraperService to target main product area on Amazon (avoiding reviews)
- Added specific selectors for Amazon product page (#imgTagWrapperId, #productTitle, etc.)
 2025-12-01 16:18:35 +01:00
LogiFlow b982fab306 Merge pull request #214 from R0m1k3/antigravity 
feat: Enhance Amazon stealth with comprehensive anti-detection
 2025-12-01 13:57:30 +01:00
Michael acfc5c6397 feat: Enhance Amazon stealth with comprehensive anti-detection 
Added comprehensive stealth techniques to bypass Amazon bot detection:
- Complete HTTP headers (Accept, Accept-Language, Sec-Fetch-*, etc.)
- Extended chrome object with loadTimes, csi, app
- Permissions API override
- Plugins, languages, platform spoofing
- Battery API mocking

This significantly improves bot detection evasion compared to the basic
2-line stealth that was insufficient for Amazon's detection systems.
 2025-12-01 13:55:57 +01:00
LogiFlow aaad0abf90 Merge pull request #213 from R0m1k3/antigravity 
Antigravity
 2025-12-01 13:29:12 +01:00
Michael 2429fa2018 Revert "fix: Remove homepage redirect to avoid Amazon bot detection" 
This reverts commit 5d3d181f5e.
 2025-12-01 13:26:05 +01:00
Michael 5d3d181f5e fix: Remove homepage redirect to avoid Amazon bot detection 
Amazon detects the homepage-then-search pattern as bot behavior and blocks
with 2065 byte pages. Now navigating directly to search URL to appear more
natural.

This reverts the homepage loading strategy from commit 85b19e7 as Amazon's
bot detection has evolved and now flags this pattern.
 2025-12-01 13:25:51 +01:00
LogiFlow 000829078a Merge pull request #212 from R0m1k3/antigravity 
fix: Disable ImprovedSearchService auto-init to restore Amazon functi…
 2025-12-01 13:23:00 +01:00
Michael 95846ec606 fix: Disable ImprovedSearchService auto-init to restore Amazon functionality 
ImprovedSearchService and AmazonScraperService were both connecting to
Browserless at startup, causing connection conflicts and Amazon blocking
(2065 byte pages).

Now only AmazonScraperService initializes at startup, keeping Browserless
in continuous connection for Amazon. ImprovedSearchService will initialize
on-demand when needed.

This restores Amazon search to working state as it was at commit 85b19e7.
 2025-12-01 13:21:32 +01:00
Michael 8dd3082a9a Merge branch 'antigravity' into main - Resolve conflicts 
Resolved conflicts by keeping antigravity improvements:
- tracking_scraper_service.py: 47 popup selectors with RGPD/Amazon support
- main.py: No auto-init of TrackingScraperService (on-demand only)

This brings all popup handling improvements to main branch.
 2025-12-01 13:13:58 +01:00
Michael 53a05f45e4 fix: Remove auto-initialization of TrackingScraperService to fix Amazon scraper 
TrackingScraperService now initializes on-demand instead of at startup.
This prevents connection conflicts with AmazonScraperService and ImprovedSearchService
since all services connect to the same Browserless instance.

Fixes: Amazon scraper blocking issue (page too small - 2065 bytes)
 2025-12-01 13:05:49 +01:00
Michael e2a874bf6c feat: implement TrackingScraperService for robust web scraping with Playwright, including shared browser management and popup handling. 2025-12-01 12:42:14 +01:00
Michael a792060f08 feat: Add TrackingScraperService with enhanced popup handling for screenshots 
- Add 47 popup selectors (vs 12 generic before)
- Support French RGPD platforms (Axeptio, Didomi, OneTrust, TarteAuCitron)
- Amazon-specific detection and handling (6 selectors)
- Retry logic with verification pass
- Double popup cleanup before screenshot
- Increased timeouts for slow animations
- New _verify_no_popups() helper function

This resolves popup visibility issues in the Suivi/Dashboard screenshots.
 2025-12-01 12:27:51 +01:00
Michael 22e15515f6 Merge branch 'main' of https://github.com/R0m1k3/Priceflow 2025-12-01 11:32:14 +01:00
Michael f27a75d9d6 feat: Implement initial application structure with routing, authentication flow, and responsive layout. 2025-12-01 11:32:12 +01:00
LogiFlow 4d11dfe820 Merge pull request #210 from R0m1k3/antigravity 
feat: introduce centralized search configuration for scraping, includ…
 2025-12-01 09:34:21 +01:00
Michael 7592c2781e feat: Implement the core FastAPI application, including database migrations, service orchestration, and a new tracking scraper service. 2025-12-01 09:32:37 +01:00
Michael ffc2f0d632 feat: introduce centralized search configuration for scraping, including site selectors, proxies, and user agents. 2025-12-01 07:40:50 +01:00