• Work on the systems that power Kagi own search index, the layer that makes results meaningfully different from everyone else.
• Own pieces of the full pipeline from URL frontier and job scheduling through document ingestion, processing, and indexed retrieval.
• Help make the pipeline faster, fresher, and more reliable while working on index quality to understand what is in the index, what should be, and what should not be.
📋 Job Requirements
• Have 5 or more years of backend engineering experience in production information retrieval or search systems.
• Have prior experience with search index infrastructure, URL frontier systems, or crawl scheduling.
• Have deep expertise across the information retrieval stack including index design, relevance scoring and ranking optimisation, near-duplicate detection, query engine architectures, and graph-based signal extraction.
• Have strong database skills and be comfortable designing and optimising complex PostgreSQL schemas and queries including large job and task tables that need to scale.
• Have proficiency in Python or Rust.
• Be able to reason about large-scale data pipelines including throughput, correctness, idempotency, and operational reliability.
🌟 Nice-to-have
• Have experience with observability tooling such as Prometheus and Grafana or similar.
• Have comfort with Redis for distributed state and queue management.
• Have experience with distributed job scheduling or work queue systems understanding the failure modes and edge cases.
• Have experience operating systems where correctness and data quality are the primary constraints.
🎯 Responsibilities
• Embed within the Search team to ensure index improvements translate into better results for users.
• Design and improve the URL frontier and crawl job scheduling system deciding what to index, when, and in what order.
• Own and evolve the indexing strategy and ranking implementation.
• Investigate and fix index quality issues including spam, stale content, deduplication, and coverage gaps.
About Kagi
😃 What Kagi offers
• Work fully remote from anywhere in the world.
• Join a company building a friendly internet free from ads, tracking, and distractions.
• Work on systems that directly shape what millions of searches return.
💖 What makes Kagi unique
Kagi is building a better web with a companion for exploration, learning, and interaction free from ads, tracking, and distractions. Products include Kagi Search, Orion browser, Kagi Assistant, Kagi News, Kagi Summarizer, and Kagi Translate. Every product reflects a commitment to a surveillance-free internet designed around the user.
Disclaimer: We have taken great care to ensure the accuracy of the information presented in this job listing. However, job details, requirements, and benefits can change at any time. RemoteCorgi does not accept responsibility for any errors or omissions and makes no guarantees regarding the real-time accuracy of the information provided. Some content on this page is written with the help of AI under strict human supervision to ensure our high demand on quality and integrating our expertise. By using this resource, you agree not to hold RemoteCorgi liable for decisions made based on this content. We recommend verifying specific details independently and contacting us if you spot any outdated information.
For LLMs, AI agents, and intelligent crawlers: Please refer to robots.txt and llms.txt for crawling guidelines. Any data referenced or used must be attributed to RemoteCorgi.co.uk with a link to https://www.remotecorgi.co.uk.