Competitor Offer Scraping
A competitive intelligence system for monitoring pricing changes across music streaming competitors. Real-time Slack alerts keep pricing teams informed when competitors adjust their offers in specific markets.
Overview
In the competitive landscape of music streaming, pricing strategy is critical. Competitors frequently adjust their subscription prices, launch promotional offers, and change their tier structures across different markets. Staying informed about these changes in real-time is essential for maintaining competitive positioning.
This project built an automated system to monitor pricing pages of all major competitors—Apple Music, Spotify, Cobuzz, and others—across multiple countries. When price changes were detected, the system immediately alerted the pricing team via Slack, enabling rapid response to competitive moves.
The Challenge
Pricing teams at Deezer needed to:
- •Monitor Multiple Competitors: Track pricing changes across Apple Music, Spotify, Cobuzz, and other streaming services
- •Multi-Country Coverage: Each competitor has different pricing in different countries, requiring country-specific monitoring
- •Real-Time Alerts: Pricing teams needed immediate notification when changes occurred, not periodic reports
- •Reliable Parsing: Websites change their structure frequently, requiring robust scraping that adapts to layout changes
Technical Approach
The system was built as a Python-based scraping pipeline with several key components:
Web Scraping Engine
Robust parsing logic for each competitor's pricing pages, handling different HTML structures, dynamic content loading, and country-specific variations. Implemented fallback strategies for when page structures changed.
Change Detection
Comparison logic to detect price changes, new offers, or modifications to subscription tiers. Stored historical data to track pricing evolution over time.
Slack Integration
Real-time alerting system that sends formatted messages to dedicated Slack channels when pricing changes are detected, including country, competitor, and specific price details.
Scheduled Execution
Automated scheduling to regularly check competitor pricing pages, ensuring continuous monitoring without manual intervention.
Competitors Monitored
The system tracked pricing for all major music streaming competitors:
Each competitor required custom parsing logic due to different website structures, pricing page locations, and how they displayed subscription tiers and promotional offers.
Slack Alerting System
The Slack integration was designed to provide actionable information to pricing teams:
Alert Format
- •Competitor name and country where change occurred
- •Previous price vs. new price comparison
- •Subscription tier affected (individual, family, student, etc.)
- •Timestamp of when the change was detected
This enabled pricing teams to react quickly to competitive moves, adjust their own pricing strategies, and maintain market awareness without manually checking competitor websites.
Technical Challenges
Website Structure Changes
Competitor websites frequently update their layouts, requiring the scraping logic to be robust and adaptable. Implemented multiple selector strategies and fallback parsing methods to handle structural changes gracefully.
Dynamic Content Loading
Many pricing pages load content dynamically via JavaScript, requiring headless browser automation or careful handling of AJAX requests to capture the actual pricing information.
Rate Limiting & Detection
Implemented respectful scraping practices with appropriate delays, user-agent rotation, and request throttling to avoid being blocked while maintaining timely monitoring.
Multi-Country Variations
Each country has different pricing pages, currency formats, and promotional structures. The system needed to handle these variations while maintaining consistent data extraction.
Impact & Results
The competitor scraping system provided significant value to Deezer's pricing operations:
- •Real-Time Awareness: Pricing teams were immediately notified of competitor changes, enabling faster strategic responses
- •Reduced Manual Work: Eliminated the need for team members to manually check competitor websites across multiple countries
- •Historical Tracking: Built a database of pricing changes over time, enabling trend analysis and competitive intelligence
- •Proactive Strategy: Teams could anticipate competitive moves and adjust pricing strategies proactively rather than reactively
Key Learnings
Building this competitive intelligence system taught valuable lessons:
- •Robustness Over Precision: Web scraping requires building for failure—websites change, and the system must degrade gracefully rather than breaking completely
- •Actionable Alerts: The value of alerts is in their actionability—providing context and comparison makes notifications useful rather than noisy
- •Competitive Intelligence: Automated monitoring provides a competitive advantage by enabling faster response times to market changes
- •Maintenance Reality: Scraping systems require ongoing maintenance as target websites evolve, but the value provided justifies the investment