- Use AIService.analyze_image() class method instead of module function
- Handle AIExtractionResponse tuple return type properly
- Add _clean_domain() to strip protocol/www from stored domains
- Fixes "https://https://" URL duplication issue
- Replace httpx with Playwright/Browserless for search pages
- Handles JavaScript-rendered content and bot detection
- Add wait selectors for each site to ensure content is loaded
- Block images and tracking scripts for faster loading
- Sequential site scraping to avoid overloading Browserless
- Remove duckduckgo-search dependency (unreliable results)
- Add direct_search_service.py that scrapes sites' search pages directly
- Add search_url and product_link_selector fields to SearchSite model
- Add beautifulsoup4 and lxml for HTML parsing
- Update schemas with new fields
- Add database migration for new columns
Direct search is more reliable as it queries each site's own search
functionality instead of relying on external search engines.
Claude