Michael/Priceflow
1
0 Fork 0
mirror of https://github.com/R0m1k3/Priceflow.git synced 2026-10-11 17:29:14 +02:00
Code Issues Packages Projects Releases Wiki Activity
209 Commits 10 Branches 0 Tags
352fc72bfaac81950f93472df3bcbca67b21f8be
Commit Graph
97 Commits
Author SHA1 Message Date
Michael 352fc72bfa feat: Implement direct e-commerce search service using Playwright and Browserless with anti-bot and proxy rotation capabilities. 2025-11-29 02:31:30 +01:00
Michael fa890f67dc feat: add direct e-commerce search service using Playwright with anti-bot measures for scraping. 2025-11-29 02:24:20 +01:00
Michael bad2cfca20 feat: Add direct e-commerce search service using Playwright with anti-bot measures including rotating proxies, user agents, and stealth JavaScript. 2025-11-29 02:17:17 +01:00
Michael 87165a8b9b feat: Implement direct search service with Playwright/Browserless, including anti-bot measures, proxy rotation, and stealth JavaScript. 2025-11-29 02:07:59 +01:00
Michael 394fa3c118 feat: implement AI service for data extraction with dynamic configuration and robust response handling. 2025-11-29 02:00:42 +01:00
Michael 091caad347 feat: add web scraping and search services using Playwright with anti-detection mechanisms. 2025-11-29 01:56:25 +01:00
Michael 7a73650264 feat: add DirectSearchService for robust e-commerce product search using Playwright/Browserless with advanced bot detection evasion. 2025-11-29 01:52:31 +01:00
Michael e6bf46659d refactor: Improve browser lifecycle management in ScraperService to prevent connection errors 2025-11-29 01:39:43 +01:00
Michael 0439b74bc3 fix: Remove duplicate 'finally' block in scraper_service.py causing SyntaxError 2025-11-29 01:24:34 +01:00
Michael 405756963a fix: Resolve IndentationError in scraper_service.py caused by loop removal 2025-11-29 01:20:58 +01:00
Michael aa9d2af5b2 feat: Implement hybrid proxy strategy for Amazon scraping - fallback to Direct IP on last retry 2025-11-29 01:17:03 +01:00
Michael dd558b0e43 fix: Correct proxy usage in scraper_service.py to handle dict format 2025-11-29 01:07:02 +01:00
Michael 4c410ef48f fix: Correct Playwright proxy format - use dict with server/username/password instead of URL string 2025-11-29 01:02:41 +01:00
Michael df2191cdbd debug: Add proxy initialization logs to verify proxy loading 2025-11-29 00:53:55 +01:00
Michael 8080a97e25 feat: Add same 10 rotating proxies to scraper_service for individual product pages 2025-11-29 00:50:08 +01:00
Michael b7e373d9eb feat: Add 10 rotating proxies for Amazon to bypass IP bans 2025-11-29 00:47:17 +01:00
Michael 26df52b664 revert: Use Amazon config from commit 698dbaa (15 commits back) 2025-11-29 00:40:27 +01:00
Michael 3c4094d4de revert: Back to working Amazon configuration from ee8a146 - simpler is better 2025-11-29 00:39:25 +01:00
Michael c88f52c136 fix: Add ref=nb_sb_noss parameter to Amazon URLs as in working version + increase retry delays 2025-11-29 00:34:09 +01:00
Michael eb48c4aa33 feat: Add direct search service for e-commerce scraping via Browserless with bot evasion. 2025-11-29 00:30:12 +01:00
Michael 34dc9f6f40 fix: Apply working Amazon anti-blocking strategy from scraper_service - warmup + no image blocking 2025-11-29 00:22:52 +01:00
Michael 46442d5e0c Merge: resolved conflicts in direct_search_service.py 2025-11-29 00:12:28 +01:00
Michael 69aefdbac3 Fix: Resolve merge conflict in direct_search_service.py by confirming random import 2025-11-29 00:06:51 +01:00
Michael 0d102b73d0 feat: Implement direct e-commerce search service using Playwright/Browserless with anti-bot techniques like User-Agent rotation, proxy support, and stealth JS, particularly for Amazon. 2025-11-28 23:59:50 +01:00
Michael b75ad636dd feat: add product search orchestration service with direct, light scraper, and browserless methods 2025-11-28 23:57:51 +01:00
Michael 3caea8316b feat: Add Playwright-based web scraping service with advanced stealth techniques and proxy support, including Amazon-specific logic. 2025-11-28 23:53:45 +01:00
Michael f187e6cb9d feat: Implement direct e-commerce search service with advanced anti-bot stealth using Playwright/Browserless. 2025-11-24 13:10:40 +01:00
Michael ba28f0b711 feat: implement direct e-commerce search service with site-specific configurations using Playwright. 2025-11-24 13:05:21 +01:00
Michael ee8a14668c feat: Implement direct e-commerce search service using Playwright/Browserless to scrape product information from various configured retail websites. 2025-11-24 13:04:03 +01:00
Michael 97039484fa feat: add direct search service for e-commerce sites using Playwright/Browserless with predefined site configurations. 2025-11-24 13:03:47 +01:00
Michael 8053955097 feat: add direct e-commerce search service using Playwright/Browserless with predefined site configurations 2025-11-24 13:01:47 +01:00
Claude 698dbaa029 feat: Add dedicated Amazon extraction without generic filters 
- Create _find_amazon_product_links() function for Amazon-specific extraction
- Use ASIN-based deduplication for Amazon products
- Bypass all generic filters (_is_non_product_url, _is_navigation_link) for Amazon
- Multiple fallback methods for Amazon product detection:
  - data-component-type="s-product-image"
  - .s-result-item links
  - All /dp/ links in main content
- Clean Amazon URLs by removing tracking parameters
- Extract title from img alt as fallback for Amazon

This separates Amazon's extraction logic from the generic e-commerce filters
that were designed for French sites like Cdiscount, Fnac, etc.
 2025-11-24 08:39:11 +00:00
Claude 10ec21f4f5 feat: Add proxy support for Amazon scraping 
- Add AMAZON_PROXY_URL and AMAZON_PROXY_LIST environment variables
- Support single proxy or rotating proxy list (comma-separated)
- Apply proxy only for Amazon requests
- Log proxy usage (hiding credentials)
 2025-11-24 08:30:55 +00:00
Claude e62ff09667 feat: Enhanced stealth JavaScript with comprehensive anti-detection 
- Multiple webdriver override techniques (property, prototype, getAttribute)
- Realistic navigator.plugins and mimeTypes
- Delete 20+ automation-related window properties (CDP, Selenium, etc.)
- Full chrome object emulation with realistic properties
- WebGL and WebGL2 fingerprinting protection
- Canvas fingerprinting protection with noise injection
- AudioContext fingerprinting protection
- Screen properties spoofing
- iframe contentWindow webdriver hiding
- Battery API mocking
- Hide Playwright/Puppeteer specific objects
 2025-11-24 08:25:23 +00:00
Claude 2384fca9ca fix: Make migration idempotent - check if columns exist before adding 2025-11-24 08:17:49 +00:00
Claude 30861acdb8 fix: Correct migration dependency chain to avoid multiple heads 2025-11-24 08:14:39 +00:00
Claude 8a9ec6a6db fix: Add database migration for search_sites missing columns 
Add migration to add debug_enabled, price_selector, search_url, and
product_link_selector columns to the search_sites table.
 2025-11-24 08:11:15 +00:00
Claude 3bae422c38 fix: Add stealth JavaScript injection to bypass Amazon bot detection 
- Add STEALTH_JS constant with scripts to hide automation fingerprints
- Override navigator.webdriver, plugins, languages, permissions
- Fake WebGL vendor/renderer and hardware concurrency
- Inject stealth script via context.add_init_script() for Amazon pages
- Add bypass_csp=True to allow script injection
- Don't block images for Amazon to avoid detection triggers
- Apply stealth to both direct_search_service and scraper_service
 2025-11-24 08:07:49 +00:00
Claude e62117f9b9 feat: Add per-site debug toggle for HTML dump control 
- Add debug_enabled field to SearchSite model
- Update schemas and API routes to allow debug toggle
- Add debug toggle button on site cards in Admin UI
- Only dump HTML when debug is enabled for the site (except for critical errors like Amazon blocking)
- Add API endpoints to list/delete debug files per site domain
- Add button to clear debug files for individual sites
 2025-11-24 08:01:39 +00:00
Claude 7243bcfaa1 fix: Improve Amazon scraping to avoid 503 blocking errors 
- Add User-Agent rotation pool with modern Chrome/Firefox/Edge versions
- Implement Amazon-specific blocking detection ("Toutes nos excuses", etc.)
- Add exponential backoff with jitter for Amazon retries (up to 4 retries)
- Include Sec-Fetch-* headers and proper Referer to appear more legitimate
- Simulate human behavior (mouse movements, random scrolling)
- Add random delays between requests to avoid rate limiting
- Properly differentiate Amazon 503 (blocking) from true product unavailability
- Apply same protections to both search and product scraping services
 2025-11-24 07:45:09 +00:00
Claude a8ff8217b3 fix: Use absolute paths for debug_dumps directory 
- Changed debug_dumps path from relative to absolute (/app/debug_dumps)
- Updated direct_search_service.py to use consistent absolute path
- Updated debug.py router to use the same absolute path
- Added debug_dumps directory creation in Dockerfile
- Improved _dump_debug_html with safer filename handling
 2025-11-24 07:27:45 +00:00
Michael 5f4d3afbae feat: Implement authentication and authorization with user management and admin-only debug endpoints. 2025-11-24 07:40:22 +01:00
Michael e45290350c feat: implement direct e-commerce search service with Playwright/Browserless and add supporting admin page and debug routes. 2025-11-24 07:18:12 +01:00
Michael df052ae4f7 feat: add direct e-commerce search service via Browserless/Playwright with site-specific configurations. 2025-11-24 00:47:34 +01:00
Michael 95deddaffb feat: implement product search orchestration service with direct, light scraping, and browserless AI extraction strategies. 2025-11-24 00:28:00 +01:00
Michael 4ab859114e feat: Add direct search service for e-commerce sites using Playwright/Browserless and configured selectors. 2025-11-24 00:22:28 +01:00
Michael 18d5f3dbf1 feat: Add notification service with Apprise integration for email, Discord, and Mattermost channels. 2025-11-23 23:47:45 +01:00
Michael f720a7aa17 feat: Implement a new notification service using Apprise for sending notifications via various channels. 2025-11-23 23:42:00 +01:00
Michael 7dfc9e1af3 feat: Add notification service to send messages via Apprise with support for various channels. 2025-11-23 23:36:26 +01:00
Michael 9637d77cef feat: Initialize core application structure with item tracking, price history, and notification channel management. 2025-11-23 23:17:52 +01:00