Michael/Priceflow
1
0 Fork 0
mirror of https://github.com/R0m1k3/Priceflow.git synced 2026-10-12 01:39:25 +02:00
Code Issues Packages Projects Releases Wiki Activity
166 Commits 10 Branches 0 Tags
698dbaa0293c335fa2c0552fe2c4790f62aaeb25
Commit Graph
166 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Claude 698dbaa029 feat: Add dedicated Amazon extraction without generic filters 
- Create _find_amazon_product_links() function for Amazon-specific extraction
- Use ASIN-based deduplication for Amazon products
- Bypass all generic filters (_is_non_product_url, _is_navigation_link) for Amazon
- Multiple fallback methods for Amazon product detection:
  - data-component-type="s-product-image"
  - .s-result-item links
  - All /dp/ links in main content
- Clean Amazon URLs by removing tracking parameters
- Extract title from img alt as fallback for Amazon

This separates Amazon's extraction logic from the generic e-commerce filters
that were designed for French sites like Cdiscount, Fnac, etc.
 2025-11-24 08:39:11 +00:00
Claude 10ec21f4f5 feat: Add proxy support for Amazon scraping 
- Add AMAZON_PROXY_URL and AMAZON_PROXY_LIST environment variables
- Support single proxy or rotating proxy list (comma-separated)
- Apply proxy only for Amazon requests
- Log proxy usage (hiding credentials)
 2025-11-24 08:30:55 +00:00
Claude e62ff09667 feat: Enhanced stealth JavaScript with comprehensive anti-detection 
- Multiple webdriver override techniques (property, prototype, getAttribute)
- Realistic navigator.plugins and mimeTypes
- Delete 20+ automation-related window properties (CDP, Selenium, etc.)
- Full chrome object emulation with realistic properties
- WebGL and WebGL2 fingerprinting protection
- Canvas fingerprinting protection with noise injection
- AudioContext fingerprinting protection
- Screen properties spoofing
- iframe contentWindow webdriver hiding
- Battery API mocking
- Hide Playwright/Puppeteer specific objects
 2025-11-24 08:25:23 +00:00
Claude 2384fca9ca fix: Make migration idempotent - check if columns exist before adding 2025-11-24 08:17:49 +00:00
Claude 30861acdb8 fix: Correct migration dependency chain to avoid multiple heads 2025-11-24 08:14:39 +00:00
Claude 898e19cf0a fix: Remove duplicate downloadDebugFile function causing build failure 2025-11-24 08:12:05 +00:00
Claude 8a9ec6a6db fix: Add database migration for search_sites missing columns 
Add migration to add debug_enabled, price_selector, search_url, and
product_link_selector columns to the search_sites table.
 2025-11-24 08:11:15 +00:00
Claude 3bae422c38 fix: Add stealth JavaScript injection to bypass Amazon bot detection 
- Add STEALTH_JS constant with scripts to hide automation fingerprints
- Override navigator.webdriver, plugins, languages, permissions
- Fake WebGL vendor/renderer and hardware concurrency
- Inject stealth script via context.add_init_script() for Amazon pages
- Add bypass_csp=True to allow script injection
- Don't block images for Amazon to avoid detection triggers
- Apply stealth to both direct_search_service and scraper_service
 2025-11-24 08:07:49 +00:00
Claude e62117f9b9 feat: Add per-site debug toggle for HTML dump control 
- Add debug_enabled field to SearchSite model
- Update schemas and API routes to allow debug toggle
- Add debug toggle button on site cards in Admin UI
- Only dump HTML when debug is enabled for the site (except for critical errors like Amazon blocking)
- Add API endpoints to list/delete debug files per site domain
- Add button to clear debug files for individual sites
 2025-11-24 08:01:39 +00:00
Claude 7243bcfaa1 fix: Improve Amazon scraping to avoid 503 blocking errors 
- Add User-Agent rotation pool with modern Chrome/Firefox/Edge versions
- Implement Amazon-specific blocking detection ("Toutes nos excuses", etc.)
- Add exponential backoff with jitter for Amazon retries (up to 4 retries)
- Include Sec-Fetch-* headers and proper Referer to appear more legitimate
- Simulate human behavior (mouse movements, random scrolling)
- Add random delays between requests to avoid rate limiting
- Properly differentiate Amazon 503 (blocking) from true product unavailability
- Apply same protections to both search and product scraping services
 2025-11-24 07:45:09 +00:00
LogiFlow 640975aeb6 Merge pull request #66 from R0m1k3/claude/debug-api-search-service-01Xdnq66w9L6D5odAiw99Dsa 
fix: Add missing debug tab rendering in Admin page
 2025-11-24 08:34:10 +01:00
Claude ffeeafa318 fix: Add missing debug tab rendering in Admin page 
The debug tab was defined in the tabs array and had all the state and
functions set up, but was missing the actual JSX rendering block.
This caused the debug dumps to not be displayed in the admin interface.
 2025-11-24 07:32:00 +00:00
LogiFlow df9295ce21 Merge pull request #65 from R0m1k3/claude/debug-api-search-service-01Xdnq66w9L6D5odAiw99Dsa 
fix: Use absolute paths for debug_dumps directory
 2025-11-24 08:28:06 +01:00
Claude a8ff8217b3 fix: Use absolute paths for debug_dumps directory 
- Changed debug_dumps path from relative to absolute (/app/debug_dumps)
- Updated direct_search_service.py to use consistent absolute path
- Updated debug.py router to use the same absolute path
- Added debug_dumps directory creation in Dockerfile
- Improved _dump_debug_html with safer filename handling
 2025-11-24 07:27:45 +00:00
LogiFlow 119e7e3963 Merge pull request #64 from R0m1k3/antigravity 
feat: Implement authentication and authorization with user management…
 2025-11-24 07:40:44 +01:00
Michael 5f4d3afbae feat: Implement authentication and authorization with user management and admin-only debug endpoints. 2025-11-24 07:40:22 +01:00
LogiFlow 7149448f80 Merge pull request #63 from R0m1k3/antigravity 
chore: add .dockerignore to exclude common development and build arti…
 2025-11-24 07:35:08 +01:00
Michael 25b0a824c2 chore: add .dockerignore to exclude common development and build artifacts from Docker builds. 2025-11-24 07:34:45 +01:00
LogiFlow 24430b0383 Merge pull request #62 from R0m1k3/antigravity 
chore: Install frontend dependencies.
 2025-11-24 07:26:59 +01:00
Michael fac4968485 chore: Install frontend dependencies. 2025-11-24 07:25:50 +01:00
LogiFlow de4887eeb7 Merge pull request #61 from R0m1k3/antigravity 
feat: implement direct e-commerce search service with Playwright/Brow…
 2025-11-24 07:18:34 +01:00
Michael e45290350c feat: implement direct e-commerce search service with Playwright/Browserless and add supporting admin page and debug routes. 2025-11-24 07:18:12 +01:00
LogiFlow 7093bee3a4 Merge pull request #60 from R0m1k3/antigravity 
feat: add direct e-commerce search service via Browserless/Playwright…
 2025-11-24 00:48:02 +01:00
Michael df052ae4f7 feat: add direct e-commerce search service via Browserless/Playwright with site-specific configurations. 2025-11-24 00:47:34 +01:00
LogiFlow cab6dcb77d Merge pull request #59 from R0m1k3/antigravity 
feat: implement product search orchestration service with direct, lig…
 2025-11-24 00:28:21 +01:00
Michael 95deddaffb feat: implement product search orchestration service with direct, light scraping, and browserless AI extraction strategies. 2025-11-24 00:28:00 +01:00
LogiFlow 903ff1abae Merge pull request #58 from R0m1k3/antigravity 
feat: Add direct search service for e-commerce sites using Playwright…
 2025-11-24 00:22:50 +01:00
Michael 4ab859114e feat: Add direct search service for e-commerce sites using Playwright/Browserless and configured selectors. 2025-11-24 00:22:28 +01:00
LogiFlow be7c0a313e Merge pull request #57 from R0m1k3/antigravity 
feat: Implement dashboard page for item management and add/edit item …
 2025-11-24 00:10:25 +01:00
Michael a9cb4e4ac5 feat: Implement dashboard page for item management and add/edit item modal. 2025-11-24 00:09:59 +01:00
LogiFlow 97668ca39e Merge pull request #56 from R0m1k3/antigravity 
feat: Add notification service with Apprise integration for email, Di…
 2025-11-23 23:48:09 +01:00
Michael 18d5f3dbf1 feat: Add notification service with Apprise integration for email, Discord, and Mattermost channels. 2025-11-23 23:47:45 +01:00
LogiFlow 18e8d21288 Merge pull request #55 from R0m1k3/antigravity 
feat: Implement a new notification service using Apprise for sending …
 2025-11-23 23:42:20 +01:00
Michael f720a7aa17 feat: Implement a new notification service using Apprise for sending notifications via various channels. 2025-11-23 23:42:00 +01:00
LogiFlow 3552980115 Merge pull request #54 from R0m1k3/antigravity 
feat: Add notification service to send messages via Apprise with supp…
 2025-11-23 23:37:49 +01:00
Michael 7dfc9e1af3 feat: Add notification service to send messages via Apprise with support for various channels. 2025-11-23 23:36:26 +01:00
LogiFlow ebb981b838 Merge pull request #53 from R0m1k3/antigravity 
feat: add comprehensive admin page for user, system settings, notific…
 2025-11-23 23:27:50 +01:00
Michael a26bd343ec feat: add comprehensive admin page for user, system settings, notification, and job management. 2025-11-23 23:24:35 +01:00
LogiFlow e77700d4a7 Merge pull request #52 from R0m1k3/antigravity 
feat: Initialize core application structure with item tracking, price…
 2025-11-23 23:19:05 +01:00
Michael 9637d77cef feat: Initialize core application structure with item tracking, price history, and notification channel management. 2025-11-23 23:17:52 +01:00
LogiFlow ac4a0664ba Merge pull request #51 from R0m1k3/antigravity 
feat: Add Pydantic schemas for items, price history, settings, and se…
 2025-11-23 23:00:58 +01:00
Michael 40225676fe feat: Add Pydantic schemas for items, price history, settings, and search functionality. 2025-11-23 23:00:30 +01:00
LogiFlow bd4f54f820 Merge pull request #50 from R0m1k3/antigravity 
Antigravity
 2025-11-23 22:57:38 +01:00
Michael e1aa76434f Merge branch 'antigravity' of https://github.com/R0m1k3/Priceflow into antigravity 2025-11-23 22:57:13 +01:00
Michael 38f56b8b63 db: drop notification tables and columns, and add an Alembic merge revision. 2025-11-23 22:57:09 +01:00
LogiFlow 8d38857e30 Merge pull request #49 from R0m1k3/antigravity 
Antigravity
 2025-11-23 22:52:56 +01:00
LogiFlow db06518d27 Merge branch 'main' into antigravity 2025-11-23 22:52:50 +01:00
Michael 2cf7f28f06 feat: Add initial Pydantic schemas for item tracking, price history, search sites, and search results. 2025-11-23 22:51:09 +01:00
Michael b5e3b44e5a feat: Implement scheduled item price tracking with AI analysis, core backend services, and a dashboard item management modal. 2025-11-23 22:48:01 +01:00
Michael c9a5f91dd2 feat: Introduce PriceHistoryModal component to display product price history, chart, and statistics. 2025-11-23 22:30:51 +01:00