Michael/Priceflow
1
0 Fork 0
mirror of https://github.com/R0m1k3/Priceflow.git synced 2026-10-13 18:26:20 +02:00
Code Issues Packages Projects Releases Wiki Activity
189 Commits 10 Branches 0 Tags
26df52b6648ff42b4e97c4ecdc5a4ee885a690b8
Commit Graph
189 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Michael 26df52b664 revert: Use Amazon config from commit 698dbaa (15 commits back) 2025-11-29 00:40:27 +01:00
Michael 3c4094d4de revert: Back to working Amazon configuration from ee8a146 - simpler is better 2025-11-29 00:39:25 +01:00
Michael c88f52c136 fix: Add ref=nb_sb_noss parameter to Amazon URLs as in working version + increase retry delays 2025-11-29 00:34:09 +01:00
Michael eb48c4aa33 feat: Add direct search service for e-commerce scraping via Browserless with bot evasion. 2025-11-29 00:30:12 +01:00
Michael 34dc9f6f40 fix: Apply working Amazon anti-blocking strategy from scraper_service - warmup + no image blocking 2025-11-29 00:22:52 +01:00
Michael 46442d5e0c Merge: resolved conflicts in direct_search_service.py 2025-11-29 00:12:28 +01:00
Michael 69aefdbac3 Fix: Resolve merge conflict in direct_search_service.py by confirming random import 2025-11-29 00:06:51 +01:00
Michael 0d102b73d0 feat: Implement direct e-commerce search service using Playwright/Browserless with anti-bot techniques like User-Agent rotation, proxy support, and stealth JS, particularly for Amazon. 2025-11-28 23:59:50 +01:00
Michael b75ad636dd feat: add product search orchestration service with direct, light scraper, and browserless methods 2025-11-28 23:57:51 +01:00
Michael 3caea8316b feat: Add Playwright-based web scraping service with advanced stealth techniques and proxy support, including Amazon-specific logic. 2025-11-28 23:53:45 +01:00
Michael f187e6cb9d feat: Implement direct e-commerce search service with advanced anti-bot stealth using Playwright/Browserless. 2025-11-24 13:10:40 +01:00
Michael ba28f0b711 feat: implement direct e-commerce search service with site-specific configurations using Playwright. 2025-11-24 13:05:21 +01:00
Michael ee8a14668c feat: Implement direct e-commerce search service using Playwright/Browserless to scrape product information from various configured retail websites. 2025-11-24 13:04:03 +01:00
Michael 97039484fa feat: add direct search service for e-commerce sites using Playwright/Browserless with predefined site configurations. 2025-11-24 13:03:47 +01:00
Michael 8053955097 feat: add direct e-commerce search service using Playwright/Browserless with predefined site configurations 2025-11-24 13:01:47 +01:00
LogiFlow 4ed574e814 Merge pull request #74 from R0m1k3/claude/fix-amazon-search-error-01Gnft8Nyjh9KkpKuavHaaPc 
feat: Add dedicated Amazon extraction without generic filters
 2025-11-24 09:40:49 +01:00
Claude 698dbaa029 feat: Add dedicated Amazon extraction without generic filters 
- Create _find_amazon_product_links() function for Amazon-specific extraction
- Use ASIN-based deduplication for Amazon products
- Bypass all generic filters (_is_non_product_url, _is_navigation_link) for Amazon
- Multiple fallback methods for Amazon product detection:
  - data-component-type="s-product-image"
  - .s-result-item links
  - All /dp/ links in main content
- Clean Amazon URLs by removing tracking parameters
- Extract title from img alt as fallback for Amazon

This separates Amazon's extraction logic from the generic e-commerce filters
that were designed for French sites like Cdiscount, Fnac, etc.
 2025-11-24 08:39:11 +00:00
LogiFlow 9cae86d733 Merge pull request #73 from R0m1k3/claude/fix-amazon-search-error-01Gnft8Nyjh9KkpKuavHaaPc 
feat: Add proxy support for Amazon scraping
 2025-11-24 09:31:54 +01:00
Claude 10ec21f4f5 feat: Add proxy support for Amazon scraping 
- Add AMAZON_PROXY_URL and AMAZON_PROXY_LIST environment variables
- Support single proxy or rotating proxy list (comma-separated)
- Apply proxy only for Amazon requests
- Log proxy usage (hiding credentials)
 2025-11-24 08:30:55 +00:00
LogiFlow 4c3b69b577 Merge pull request #72 from R0m1k3/claude/fix-amazon-search-error-01Gnft8Nyjh9KkpKuavHaaPc 
feat: Enhanced stealth JavaScript with comprehensive anti-detection
 2025-11-24 09:26:37 +01:00
Claude e62ff09667 feat: Enhanced stealth JavaScript with comprehensive anti-detection 
- Multiple webdriver override techniques (property, prototype, getAttribute)
- Realistic navigator.plugins and mimeTypes
- Delete 20+ automation-related window properties (CDP, Selenium, etc.)
- Full chrome object emulation with realistic properties
- WebGL and WebGL2 fingerprinting protection
- Canvas fingerprinting protection with noise injection
- AudioContext fingerprinting protection
- Screen properties spoofing
- iframe contentWindow webdriver hiding
- Battery API mocking
- Hide Playwright/Puppeteer specific objects
 2025-11-24 08:25:23 +00:00
LogiFlow 1e1a4465dc Merge pull request #71 from R0m1k3/claude/fix-amazon-search-error-01Gnft8Nyjh9KkpKuavHaaPc 
fix: Make migration idempotent - check if columns exist before adding
 2025-11-24 09:18:42 +01:00
Claude 2384fca9ca fix: Make migration idempotent - check if columns exist before adding 2025-11-24 08:17:49 +00:00
LogiFlow ffbeffcfca Merge pull request #70 from R0m1k3/claude/fix-amazon-search-error-01Gnft8Nyjh9KkpKuavHaaPc 
fix: Correct migration dependency chain to avoid multiple heads
 2025-11-24 09:15:34 +01:00
Claude 30861acdb8 fix: Correct migration dependency chain to avoid multiple heads 2025-11-24 08:14:39 +00:00
LogiFlow ec80a409f8 Merge pull request #69 from R0m1k3/claude/fix-amazon-search-error-01Gnft8Nyjh9KkpKuavHaaPc 
Claude/fix amazon search error 01 gnft8 nyjh9 kkp kuav haa pc
 2025-11-24 09:12:40 +01:00
Claude 898e19cf0a fix: Remove duplicate downloadDebugFile function causing build failure 2025-11-24 08:12:05 +00:00
Claude 8a9ec6a6db fix: Add database migration for search_sites missing columns 
Add migration to add debug_enabled, price_selector, search_url, and
product_link_selector columns to the search_sites table.
 2025-11-24 08:11:15 +00:00
LogiFlow f30479516c Merge pull request #68 from R0m1k3/claude/fix-amazon-search-error-01Gnft8Nyjh9KkpKuavHaaPc 
Claude/fix amazon search error 01 gnft8 nyjh9 kkp kuav haa pc
 2025-11-24 09:10:42 +01:00
Claude 3bae422c38 fix: Add stealth JavaScript injection to bypass Amazon bot detection 
- Add STEALTH_JS constant with scripts to hide automation fingerprints
- Override navigator.webdriver, plugins, languages, permissions
- Fake WebGL vendor/renderer and hardware concurrency
- Inject stealth script via context.add_init_script() for Amazon pages
- Add bypass_csp=True to allow script injection
- Don't block images for Amazon to avoid detection triggers
- Apply stealth to both direct_search_service and scraper_service
 2025-11-24 08:07:49 +00:00
Claude e62117f9b9 feat: Add per-site debug toggle for HTML dump control 
- Add debug_enabled field to SearchSite model
- Update schemas and API routes to allow debug toggle
- Add debug toggle button on site cards in Admin UI
- Only dump HTML when debug is enabled for the site (except for critical errors like Amazon blocking)
- Add API endpoints to list/delete debug files per site domain
- Add button to clear debug files for individual sites
 2025-11-24 08:01:39 +00:00
LogiFlow 6e3492b5cc Merge pull request #67 from R0m1k3/claude/fix-amazon-search-error-01Gnft8Nyjh9KkpKuavHaaPc 
fix: Improve Amazon scraping to avoid 503 blocking errors
 2025-11-24 08:50:20 +01:00
Claude 7243bcfaa1 fix: Improve Amazon scraping to avoid 503 blocking errors 
- Add User-Agent rotation pool with modern Chrome/Firefox/Edge versions
- Implement Amazon-specific blocking detection ("Toutes nos excuses", etc.)
- Add exponential backoff with jitter for Amazon retries (up to 4 retries)
- Include Sec-Fetch-* headers and proper Referer to appear more legitimate
- Simulate human behavior (mouse movements, random scrolling)
- Add random delays between requests to avoid rate limiting
- Properly differentiate Amazon 503 (blocking) from true product unavailability
- Apply same protections to both search and product scraping services
 2025-11-24 07:45:09 +00:00
LogiFlow 640975aeb6 Merge pull request #66 from R0m1k3/claude/debug-api-search-service-01Xdnq66w9L6D5odAiw99Dsa 
fix: Add missing debug tab rendering in Admin page
 2025-11-24 08:34:10 +01:00
Claude ffeeafa318 fix: Add missing debug tab rendering in Admin page 
The debug tab was defined in the tabs array and had all the state and
functions set up, but was missing the actual JSX rendering block.
This caused the debug dumps to not be displayed in the admin interface.
 2025-11-24 07:32:00 +00:00
LogiFlow df9295ce21 Merge pull request #65 from R0m1k3/claude/debug-api-search-service-01Xdnq66w9L6D5odAiw99Dsa 
fix: Use absolute paths for debug_dumps directory
 2025-11-24 08:28:06 +01:00
Claude a8ff8217b3 fix: Use absolute paths for debug_dumps directory 
- Changed debug_dumps path from relative to absolute (/app/debug_dumps)
- Updated direct_search_service.py to use consistent absolute path
- Updated debug.py router to use the same absolute path
- Added debug_dumps directory creation in Dockerfile
- Improved _dump_debug_html with safer filename handling
 2025-11-24 07:27:45 +00:00
LogiFlow 119e7e3963 Merge pull request #64 from R0m1k3/antigravity 
feat: Implement authentication and authorization with user management…
 2025-11-24 07:40:44 +01:00
Michael 5f4d3afbae feat: Implement authentication and authorization with user management and admin-only debug endpoints. 2025-11-24 07:40:22 +01:00
LogiFlow 7149448f80 Merge pull request #63 from R0m1k3/antigravity 
chore: add .dockerignore to exclude common development and build arti…
 2025-11-24 07:35:08 +01:00
Michael 25b0a824c2 chore: add .dockerignore to exclude common development and build artifacts from Docker builds. 2025-11-24 07:34:45 +01:00
LogiFlow 24430b0383 Merge pull request #62 from R0m1k3/antigravity 
chore: Install frontend dependencies.
 2025-11-24 07:26:59 +01:00
Michael fac4968485 chore: Install frontend dependencies. 2025-11-24 07:25:50 +01:00
LogiFlow de4887eeb7 Merge pull request #61 from R0m1k3/antigravity 
feat: implement direct e-commerce search service with Playwright/Brow…
 2025-11-24 07:18:34 +01:00
Michael e45290350c feat: implement direct e-commerce search service with Playwright/Browserless and add supporting admin page and debug routes. 2025-11-24 07:18:12 +01:00
LogiFlow 7093bee3a4 Merge pull request #60 from R0m1k3/antigravity 
feat: add direct e-commerce search service via Browserless/Playwright…
 2025-11-24 00:48:02 +01:00
Michael df052ae4f7 feat: add direct e-commerce search service via Browserless/Playwright with site-specific configurations. 2025-11-24 00:47:34 +01:00
LogiFlow cab6dcb77d Merge pull request #59 from R0m1k3/antigravity 
feat: implement product search orchestration service with direct, lig…
 2025-11-24 00:28:21 +01:00
Michael 95deddaffb feat: implement product search orchestration service with direct, light scraping, and browserless AI extraction strategies. 2025-11-24 00:28:00 +01:00
LogiFlow 903ff1abae Merge pull request #58 from R0m1k3/antigravity 
feat: Add direct search service for e-commerce sites using Playwright…
 2025-11-24 00:22:50 +01:00