Senior Web Scraper (Scrapy)

About the role

NIQ Digital Shelf is looking for a Senior Web Scraper (Scrapy) in Madrid. The team processes tens of billions of data points from thousands of web and mobile sources and turns them into actionable insights for retailers and brands. You will join a group of 60+ scrapers, engineers, analysts and data experts in the scraping and R&D team, working closely with Customer Success, Operations and Data.

Key responsibilities

• Design and build distributed scraping architectures that scale across thousands of domains.
• Reverse engineer APIs and mobile apps using tools such as Frida, mitmproxy, Charles Proxy and Burp.
• Detect common anti-bot measures (JavaScript checks, browser behavior detection) and build crawlers that adapt to look more human-like.
• Lead technical innovation in scraping: identify edge cases, prototype new approaches and share knowledge with the team.
• Set up anomaly detection and monitoring to spot blockers or failures early.

Required profile

• 5+ years of experience with Python and large-scale scraping projects.
• Deep understanding of anti-bot technologies and how to bypass them, including fingerprinting and WAFs.
• Strong debugging skills across browser, HTTP traffic and device-level interactions.
• Independent thinker who enjoys solving problems no one else has solved.

Required skills

• Python, Scrapy, Requests, httpx, Selenium, Playwright or custom headless browsers.
• Proxies (residential, datacenter, rotating), session management, cookie injection and TLS tweaking.
• Bonus: Celery, Kafka, Linux, AWS, GCP, Kubernetes, Cloud Run, Cloud Functions, Parquet, Avro, MongoDB, InfluxDB, Redis.

What we offer

• Flexible working environment.
• Volunteer time off.
• LinkedIn Learning.
• Employee Assistance Program.
• Friendly, transparent culture, competitive salary and flexible hours.