Senior Web Crawling Engineer

A stealth, engineering-driven startup with significant funding is looking for a Senior Web Crawling Engineer to design and own the core crawling system that powers their next-generation search platform.

The team builds deeply technical systems in Rust, though strong experience in C++ or another low-level language plus an interest in working with Rust is also welcome.

What You’ll Work On

• Architect and implement large-scale web crawling infrastructure
• Manage crawling pipelines: scheduling, graph traversal, deduplication, and robustness
• Integrate crawled data with downstream indexing and storage systems
• Contribute to high-performance Rust systems used across the product

Requirements

Must-have

• Experience building large-scale web crawlers (millions of URLs or comparable scale)
• Strong background in systems programming (Rust, C++, or similar)
• Ability to design, build, and maintain complex distributed systems
• Solid understanding of HTTP, robots.txt, sitemaps, rate limiting, and crawling best practices

Nice-to-have

• Production Rust experience or a strong desire to work in Rust