Job Description
We are partnering with a fast-growing AI infrastructure company building foundational technology powering the next generation of AI-native search and retrieval systems. The team is building large-scale infrastructure to crawl and index the web, train advanced embedding/retrieval models, and operate high-performance distributed systems at massive scale. They are looking for experienced engineers who enjoy solving deep infrastructure and internet-scale engineering problems. The Role As a Web Crawler Engineer, you will help design and build internet-scale crawling infrastructure capable of processing hundreds of millions of webpages efficiently and reliably. This role sits at the intersection of distributed systems, performance engineering, browser automation, and large-scale data infrastructure. You will work on problems such as: Distributed web crawling at massive scale Intelligent crawl scheduling and prioritisation JavaScript-heavy and dynamic site rendering Anti-bot detection and eva...