Senior Software Engineer - Web Scraping & Data Acquisition

About the role

NIQ Digital Shelf is looking for a Senior Software Engineer in Madrid to design and build distributed scraping architectures that scale across thousands of domains. The team of 60+ scrapers, engineers, analysts and data experts works with Customer Success, Operations and Data to deliver high-quality, actionable data at scale. The role sits within the scraping and R&D team, where you can experiment and lead technical innovation.

Key responsibilities

• Design and build distributed scraping architectures that scale across thousands of domains.
• Reverse engineer APIs and mobile apps using tools such as Frida, mitmproxy, Charles Proxy and Burp.
• Detect common anti-bot measures (JavaScript checks, browser behavior detection) and build crawlers that adapt to appear more human-like.
• Lead technical innovation in scraping: identify edge cases, prototype new approaches and share knowledge with the team.
• Set up anomaly detection and monitoring to detect blockers or failures early.

Required profile

• 5+ years of experience with Python and large-scale scraping projects.
• Deep understanding of anti-bot technologies and how to bypass them, including fingerprinting, WAFs and JavaScript challenges.
• Strong debugging skills across browser, HTTP traffic and device-level interactions.
• Independent thinker who enjoys solving problems no one has solved before.

Required skills

• Python, Scrapy, Requests, httpx, Selenium, Playwright or custom headless browser drivers.
• Proxies (residential, datacenter, rotating), session management, cookie injection and TLS tweaking.
• Bonus: structured storage formats (Parquet, Avro), real-time stream processing, distributed task queues (Celery, Kafka), Linux or cloud environments (AWS, GCP), Kubernetes, Cloud Run and Cloud Functions.

What we offer

• Flexible working environment and volunteer time off.
• Competitive salary and flexible hours.
• LinkedIn Learning and Employee Assistance Program (EAP).