- Change default currency from USD to EUR for all French sites
- Update AI extraction prompt to use EUR by default
- Fix E.Leclerc search URL pattern
- Improve product selectors for Auchan, Carrefour, Amazon France
- Add Amazon France language parameter for better results
- Enhance cookie acceptance with site-specific selectors:
- Amazon (#sp-cc-accept)
- E.Leclerc (#onetrust-accept-btn-handler)
- Auchan (#popin_tc_privacy_button_2, #didomi-notice-agree-button)
- Carrefour (#onetrust-accept-btn-handler)
- Cdiscount (#footer_tc_privacy_button_2)
- And more CMP platforms (Didomi, OneTrust, Tarteaucitron, etc.)
- Improve cookie handling with iframe support and retry logic
- Remove ability to add/edit/delete search sites from settings UI
- Configure 16 French e-commerce sites with search URLs:
- Discount: Gifi, Stokomani, B&M, Centrakor, L'Incroyable, Action, La Foir'Fouille
- Grandes Surfaces: E.Leclerc, Auchan, Carrefour
- E-commerce: Amazon France, Cdiscount
- Électronique: Darty, Boulanger, Fnac
- Add automatic database seeding on application startup
- Add /api/search-sites/seed and /api/search-sites/reset endpoints
- Add cookie consent banner handling for sites that require it
- Simplify Settings page to only show site list with toggle for active state
- Add fallback to generic detection when configured selector fails
- Add many more CSS selectors for French e-commerce platforms (PrestaShop, Magento, WooCommerce)
- Add /produits/ pattern for bmstores.fr
- Improve generic container-based link detection
- Add navigation link filter to exclude non-product links
- Improve domain matching in direct_search_service to handle various formats
- Add /discover-all endpoint to discover search URLs for all sites missing one
- Add constants for magic numbers and fix linting issues
- Better domain cleaning in comparison functions
- Move imports to top-level in search_sites router
- Fix line too long issue with input selector
- Define constants for magic numbers (HTTP_OK, MIN_PRODUCTS_TO_MATCH)
- Use 'raise from' pattern for proper exception chaining
- Add search_url_discovery.py service that visits sites and finds search forms
- Automatically discover search_url when a new site is created without one
- Add POST /{site_id}/discover endpoint to force rediscovery
- Discovery runs in background to not block the API response
- Tries multiple methods: form analysis, common URL patterns
- Add search URLs for French discount stores: stokomani, gifi, bmstores,
centrakor, lafoirfouille, tedi, leclerc, carrefour, auchan
- Fix screenshot URL construction in search results
- Include screenshot_url in AI extraction results and error cases
- Use AIService.analyze_image() class method instead of module function
- Handle AIExtractionResponse tuple return type properly
- Add _clean_domain() to strip protocol/www from stored domains
- Fixes "https://https://" URL duplication issue
- Replace httpx with Playwright/Browserless for search pages
- Handles JavaScript-rendered content and bot detection
- Add wait selectors for each site to ensure content is loaded
- Block images and tracking scripts for faster loading
- Sequential site scraping to avoid overloading Browserless
- Remove duckduckgo-search dependency (unreliable results)
- Add direct_search_service.py that scrapes sites' search pages directly
- Add search_url and product_link_selector fields to SearchSite model
- Add beautifulsoup4 and lxml for HTML parsing
- Update schemas with new fields
- Add database migration for new columns
Direct search is more reliable as it queries each site's own search
functionality instead of relying on external search engines.
- Remove SearXNG container and configuration files
- Add duckduckgo-search library for web search
- Create duckduckgo_service.py with async search support
- Update search_service.py to use DuckDuckGo instead of SearXNG
- Fix BROWSERLESS_URL port to 3012
This eliminates the persistent limiter.toml directory issue and
provides a simpler, more reliable search solution.
- Mount ./searxng as /etc/searxng:rw instead of individual files
- Remove custom entrypoint.sh (no longer needed)
- Simplify settings.yml configuration
- Remove entrypoint script approach
- Mount limiter.toml directly as a file
- User must DELETE the SearXNG container in Portainer before redeploying
to clear the cached directory state
- Create entrypoint.sh script that creates limiter.toml before SearXNG starts
- Script removes any existing limiter.toml (file or directory) first
- Then creates proper limiter.toml with bot detection bypass
- Finally executes the original SearXNG entrypoint
- This approach is more reliable than inline commands
- Remove command override (limiter warning is non-blocking)
- Add DNS servers (8.8.8.8, 8.8.4.4, 1.1.1.1) to fix DNS resolution
- The real issue was 'Temporary failure in name resolution' not limiter.toml
SearXNG:
- Use proper command list format instead of shell string
- Use printf with \n for proper newlines in limiter.toml
- Use exec to properly start the entrypoint
Edit Site Modal:
- Add Dialog component import
- Add siteModalOpen and editSiteData state
- Create proper modal with form for editing sites
- Modal opens when clicking edit button
- All site fields editable in modal
SearXNG:
- Use command override to rm -rf limiter.toml before recreating it
- This ensures any directory is removed before creating the file
- Mount only settings.yml, create limiter.toml dynamically
UI:
- Remove opacity-0 from site edit/delete buttons
- Buttons are now always visible (not just on hover)
- Added title attributes for accessibility
- Create search_sites table if it doesn't exist
- Add price_selector column only if table exists but column doesn't
- Fixes 400 Bad Request when creating search sites
- Add run_migrations() function that runs at app startup
- Automatically adds price_selector column to search_sites if missing
- Prevents UndefinedColumn errors on existing databases
Simple and robust solution:
- Mount ./searxng:/etc/searxng:rw (entire directory)
- No entrypoint/command overrides needed
- Both settings.yml and limiter.toml are included
- SearXNG uses its default entrypoint
SearXNG:
- Fix tini not found error by calling entrypoint.sh directly
- Keep limiter.toml creation at container startup
UI Changes:
- Rename "Tableau de bord" to "Suivis produits" in navigation
- Move CSS selector configuration from item modal to Settings page
- Add price_selector field to search sites configuration
- Display CSS selector badge on site cards
Backend:
- Add price_selector column to SearchSite model and schemas
- Update init.sql with search_sites table and migration for existing DBs
- Fix SearXNG limiter.toml directory issue by creating file at container
startup via entrypoint command (rm -rf then recreate)
- Update README to use logo.svg instead of logo.png
- Redesign favicon.svg to match main logo style:
- Same gold-to-green gradient background
- Line gradient from white to green
- Glow effect on chart line
- Decorative dots at data points
- Green arrow indicator
- Add real_ip config to settings.yml to trust proxy headers
- Update limiter.toml to allow all IPs (0.0.0.0/0 and ::/0)
- Re-add limiter.toml volume mount (file now exists)
- Add localhost range (127.0.0.0/8) to pass_ip list
Docker was creating /etc/searxng/limiter.toml as a directory instead of
mounting the file, causing IsADirectoryError. The limiter is already
disabled in settings.yml, so the separate toml file is not needed.
LogiFlow
Claude