Michael/Priceflow
1
0 Fork 0
mirror of https://github.com/R0m1k3/Priceflow.git synced 2026-10-12 01:39:25 +02:00
Code Issues Packages Projects Releases Wiki Activity
204 Commits 10 Branches 0 Tags
6cb145626f5709f86aee122f046c8a32dc586f5d
Commit Graph
204 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
LogiFlow 6cb145626f Merge pull request #78 from R0m1k3/main 
commit
 2025-11-29 02:09:11 +01:00
LogiFlow 326272ae56 Merge pull request #77 from R0m1k3/antigravity 
feat: implement AI service for data extraction with dynamic configura…
 2025-11-29 02:05:07 +01:00
Michael 394fa3c118 feat: implement AI service for data extraction with dynamic configuration and robust response handling. 2025-11-29 02:00:42 +01:00
LogiFlow a4c0431443 Merge pull request #76 from R0m1k3/antigravity 
Antigravity
 2025-11-29 01:56:44 +01:00
Michael 091caad347 feat: add web scraping and search services using Playwright with anti-detection mechanisms. 2025-11-29 01:56:25 +01:00
Michael 7a73650264 feat: add DirectSearchService for robust e-commerce product search using Playwright/Browserless with advanced bot detection evasion. 2025-11-29 01:52:31 +01:00
Michael e6bf46659d refactor: Improve browser lifecycle management in ScraperService to prevent connection errors 2025-11-29 01:39:43 +01:00
Michael 0439b74bc3 fix: Remove duplicate 'finally' block in scraper_service.py causing SyntaxError 2025-11-29 01:24:34 +01:00
Michael 405756963a fix: Resolve IndentationError in scraper_service.py caused by loop removal 2025-11-29 01:20:58 +01:00
Michael aa9d2af5b2 feat: Implement hybrid proxy strategy for Amazon scraping - fallback to Direct IP on last retry 2025-11-29 01:17:03 +01:00
Michael dd558b0e43 fix: Correct proxy usage in scraper_service.py to handle dict format 2025-11-29 01:07:02 +01:00
Michael 4c410ef48f fix: Correct Playwright proxy format - use dict with server/username/password instead of URL string 2025-11-29 01:02:41 +01:00
Michael df2191cdbd debug: Add proxy initialization logs to verify proxy loading 2025-11-29 00:53:55 +01:00
Michael 8080a97e25 feat: Add same 10 rotating proxies to scraper_service for individual product pages 2025-11-29 00:50:08 +01:00
Michael b7e373d9eb feat: Add 10 rotating proxies for Amazon to bypass IP bans 2025-11-29 00:47:17 +01:00
Michael 26df52b664 revert: Use Amazon config from commit 698dbaa (15 commits back) 2025-11-29 00:40:27 +01:00
Michael 3c4094d4de revert: Back to working Amazon configuration from ee8a146 - simpler is better 2025-11-29 00:39:25 +01:00
Michael c88f52c136 fix: Add ref=nb_sb_noss parameter to Amazon URLs as in working version + increase retry delays 2025-11-29 00:34:09 +01:00
Michael eb48c4aa33 feat: Add direct search service for e-commerce scraping via Browserless with bot evasion. 2025-11-29 00:30:12 +01:00
Michael 34dc9f6f40 fix: Apply working Amazon anti-blocking strategy from scraper_service - warmup + no image blocking 2025-11-29 00:22:52 +01:00
Michael 46442d5e0c Merge: resolved conflicts in direct_search_service.py 2025-11-29 00:12:28 +01:00
Michael 69aefdbac3 Fix: Resolve merge conflict in direct_search_service.py by confirming random import 2025-11-29 00:06:51 +01:00
Michael 0d102b73d0 feat: Implement direct e-commerce search service using Playwright/Browserless with anti-bot techniques like User-Agent rotation, proxy support, and stealth JS, particularly for Amazon. 2025-11-28 23:59:50 +01:00
Michael b75ad636dd feat: add product search orchestration service with direct, light scraper, and browserless methods 2025-11-28 23:57:51 +01:00
Michael 3caea8316b feat: Add Playwright-based web scraping service with advanced stealth techniques and proxy support, including Amazon-specific logic. 2025-11-28 23:53:45 +01:00
Michael f187e6cb9d feat: Implement direct e-commerce search service with advanced anti-bot stealth using Playwright/Browserless. 2025-11-24 13:10:40 +01:00
Michael ba28f0b711 feat: implement direct e-commerce search service with site-specific configurations using Playwright. 2025-11-24 13:05:21 +01:00
Michael ee8a14668c feat: Implement direct e-commerce search service using Playwright/Browserless to scrape product information from various configured retail websites. 2025-11-24 13:04:03 +01:00
Michael 97039484fa feat: add direct search service for e-commerce sites using Playwright/Browserless with predefined site configurations. 2025-11-24 13:03:47 +01:00
Michael 8053955097 feat: add direct e-commerce search service using Playwright/Browserless with predefined site configurations 2025-11-24 13:01:47 +01:00
LogiFlow 4ed574e814 Merge pull request #74 from R0m1k3/claude/fix-amazon-search-error-01Gnft8Nyjh9KkpKuavHaaPc 
feat: Add dedicated Amazon extraction without generic filters
 2025-11-24 09:40:49 +01:00
Claude 698dbaa029 feat: Add dedicated Amazon extraction without generic filters 
- Create _find_amazon_product_links() function for Amazon-specific extraction
- Use ASIN-based deduplication for Amazon products
- Bypass all generic filters (_is_non_product_url, _is_navigation_link) for Amazon
- Multiple fallback methods for Amazon product detection:
  - data-component-type="s-product-image"
  - .s-result-item links
  - All /dp/ links in main content
- Clean Amazon URLs by removing tracking parameters
- Extract title from img alt as fallback for Amazon

This separates Amazon's extraction logic from the generic e-commerce filters
that were designed for French sites like Cdiscount, Fnac, etc.
 2025-11-24 08:39:11 +00:00
LogiFlow 9cae86d733 Merge pull request #73 from R0m1k3/claude/fix-amazon-search-error-01Gnft8Nyjh9KkpKuavHaaPc 
feat: Add proxy support for Amazon scraping
 2025-11-24 09:31:54 +01:00
Claude 10ec21f4f5 feat: Add proxy support for Amazon scraping 
- Add AMAZON_PROXY_URL and AMAZON_PROXY_LIST environment variables
- Support single proxy or rotating proxy list (comma-separated)
- Apply proxy only for Amazon requests
- Log proxy usage (hiding credentials)
 2025-11-24 08:30:55 +00:00
LogiFlow 4c3b69b577 Merge pull request #72 from R0m1k3/claude/fix-amazon-search-error-01Gnft8Nyjh9KkpKuavHaaPc 
feat: Enhanced stealth JavaScript with comprehensive anti-detection
 2025-11-24 09:26:37 +01:00
Claude e62ff09667 feat: Enhanced stealth JavaScript with comprehensive anti-detection 
- Multiple webdriver override techniques (property, prototype, getAttribute)
- Realistic navigator.plugins and mimeTypes
- Delete 20+ automation-related window properties (CDP, Selenium, etc.)
- Full chrome object emulation with realistic properties
- WebGL and WebGL2 fingerprinting protection
- Canvas fingerprinting protection with noise injection
- AudioContext fingerprinting protection
- Screen properties spoofing
- iframe contentWindow webdriver hiding
- Battery API mocking
- Hide Playwright/Puppeteer specific objects
 2025-11-24 08:25:23 +00:00
LogiFlow 1e1a4465dc Merge pull request #71 from R0m1k3/claude/fix-amazon-search-error-01Gnft8Nyjh9KkpKuavHaaPc 
fix: Make migration idempotent - check if columns exist before adding
 2025-11-24 09:18:42 +01:00
Claude 2384fca9ca fix: Make migration idempotent - check if columns exist before adding 2025-11-24 08:17:49 +00:00
LogiFlow ffbeffcfca Merge pull request #70 from R0m1k3/claude/fix-amazon-search-error-01Gnft8Nyjh9KkpKuavHaaPc 
fix: Correct migration dependency chain to avoid multiple heads
 2025-11-24 09:15:34 +01:00
Claude 30861acdb8 fix: Correct migration dependency chain to avoid multiple heads 2025-11-24 08:14:39 +00:00
LogiFlow ec80a409f8 Merge pull request #69 from R0m1k3/claude/fix-amazon-search-error-01Gnft8Nyjh9KkpKuavHaaPc 
Claude/fix amazon search error 01 gnft8 nyjh9 kkp kuav haa pc
 2025-11-24 09:12:40 +01:00
Claude 898e19cf0a fix: Remove duplicate downloadDebugFile function causing build failure 2025-11-24 08:12:05 +00:00
Claude 8a9ec6a6db fix: Add database migration for search_sites missing columns 
Add migration to add debug_enabled, price_selector, search_url, and
product_link_selector columns to the search_sites table.
 2025-11-24 08:11:15 +00:00
LogiFlow f30479516c Merge pull request #68 from R0m1k3/claude/fix-amazon-search-error-01Gnft8Nyjh9KkpKuavHaaPc 
Claude/fix amazon search error 01 gnft8 nyjh9 kkp kuav haa pc
 2025-11-24 09:10:42 +01:00
Claude 3bae422c38 fix: Add stealth JavaScript injection to bypass Amazon bot detection 
- Add STEALTH_JS constant with scripts to hide automation fingerprints
- Override navigator.webdriver, plugins, languages, permissions
- Fake WebGL vendor/renderer and hardware concurrency
- Inject stealth script via context.add_init_script() for Amazon pages
- Add bypass_csp=True to allow script injection
- Don't block images for Amazon to avoid detection triggers
- Apply stealth to both direct_search_service and scraper_service
 2025-11-24 08:07:49 +00:00
Claude e62117f9b9 feat: Add per-site debug toggle for HTML dump control 
- Add debug_enabled field to SearchSite model
- Update schemas and API routes to allow debug toggle
- Add debug toggle button on site cards in Admin UI
- Only dump HTML when debug is enabled for the site (except for critical errors like Amazon blocking)
- Add API endpoints to list/delete debug files per site domain
- Add button to clear debug files for individual sites
 2025-11-24 08:01:39 +00:00
LogiFlow 6e3492b5cc Merge pull request #67 from R0m1k3/claude/fix-amazon-search-error-01Gnft8Nyjh9KkpKuavHaaPc 
fix: Improve Amazon scraping to avoid 503 blocking errors
 2025-11-24 08:50:20 +01:00
Claude 7243bcfaa1 fix: Improve Amazon scraping to avoid 503 blocking errors 
- Add User-Agent rotation pool with modern Chrome/Firefox/Edge versions
- Implement Amazon-specific blocking detection ("Toutes nos excuses", etc.)
- Add exponential backoff with jitter for Amazon retries (up to 4 retries)
- Include Sec-Fetch-* headers and proper Referer to appear more legitimate
- Simulate human behavior (mouse movements, random scrolling)
- Add random delays between requests to avoid rate limiting
- Properly differentiate Amazon 503 (blocking) from true product unavailability
- Apply same protections to both search and product scraping services
 2025-11-24 07:45:09 +00:00
LogiFlow 640975aeb6 Merge pull request #66 from R0m1k3/claude/debug-api-search-service-01Xdnq66w9L6D5odAiw99Dsa 
fix: Add missing debug tab rendering in Admin page
 2025-11-24 08:34:10 +01:00
Claude ffeeafa318 fix: Add missing debug tab rendering in Admin page 
The debug tab was defined in the tabs array and had all the state and
functions set up, but was missing the actual JSX rendering block.
This caused the debug dumps to not be displayed in the admin interface.
 2025-11-24 07:32:00 +00:00