Quick Answer
How to narrate a web crawler in English: walk the frontier, politeness rules, fetch/parse, and storage — naming one bottleneck and one risk at each stage.
Diagrams and what to say while drawing
Narrate each figure out loud. Point to the box you are talking about.
What to say on figure 1
Frontier → fetcher workers → parser → storage.
What to say on figure 2
Show per-host politeness queues.
What to say on figure 3
Show URL seen-set / bloom filter.
What to say on figure 4
Show retry path for failed fetches.
Key vocabulary
| Term | Plain meaning | Say it in an interview |
|---|---|---|
| URL frontier | Queue of URLs to visit | “The frontier schedules the next URLs.” |
| Politeness | Avoid overloading a host | “We enforce politeness per domain.” |
| Deduplication | Skip seen URLs/content | “We deduplicate with a bloom filter or URL store.” |
| Robots.txt | Crawl permission file | “We respect robots.txt before fetching.” |
ESL English phrases and transitions
Clarify
- “Is this a general web crawl or limited to a site list?”
- “Do we need near-real-time freshness or batch crawl cycles?”
Propose and check in
- “The next bottleneck is X, so I would add Y.”
- “Does this level of detail work, or should I go deeper?”
Tradeoffs
- “The advantage is simplicity; the downside is…”
- “I’d choose A over B because…”
Key English language pitfalls
| Pitfall | Sounds like | Say instead |
|---|---|---|
| Ignoring robots.txt | Legal/ethics gap | Say you respect robots.txt |
| One giant queue | Head-of-line blocking | Mention host-based queues |
| Summary | Every new box needs one spoken reason. | |
Common English mistakes
| Mistake | What the interviewer concludes | Better English |
|---|---|---|
| No politeness | Looks reckless | Always mention per-host limits |
| Infinite crawl story | No stop condition | Talk about prioritisation and seen URLs |
What the interviewer is testing
This is a communication test as much as a design test. Can you explain a web crawler as a calm collaboration?
Interviewers listen for structure, tradeoffs, and clear English under pressure.
Key takeaway
Your job is to narrate decisions the interviewer can follow and challenge — not to dump tools.
How to open the problem in English
Opening script you can practise aloud
- “Is this a general web crawl or limited to a site list?”
- “Do we need near-real-time freshness or batch crawl cycles?”
Watch Out
Do not open with a finished architecture. Show the path.
Web crawler system design
Use this sequence for web crawler system design. For each stage, say what changes, why, and what can still go wrong.
Seed & frontier
Start from seeds; enqueue new links with priorities.
Politeness
Per-host rate limits and robots.txt checks.
Fetch & parse
Download pages, extract links, handle failures.
Store & dedup
Save content metadata; skip already-seen URLs.
Weak vs strong answers
Weak
“Workers download the internet.”
Strong
“A frontier schedules URLs politely per host; workers fetch and parse; we dedupe URLs so we don’t recrawl endlessly.”
Metrics and how to say numbers
| Write | Say |
|---|---|
| pages/sec | pages per second |
| per-host delay | a delay between requests to the same host |
Follow-up questions you will get
Likely follow-ups
- “How do you prioritise?” → “I’d rank by importance signals and freshness needs, not pure FIFO.”
Practice drill
Checklist
- Frontier
- Politeness
- Parse
- Dedup
“If you can explain a web crawler without notes, you are ready to draw it in an interview.”
Answer frameworks you can reuse
Use these spoken templates under pressure. Fill the blanks — do not invent a new structure mid-interview.
Framework 1 — Whole answer
Clarify → High-level → Deep dive → Check-in.
Framework 2 — Each component (B-C-B-C)
Bottleneck → Component → Benefit → Cost.
Framework 3 — Now → Breaks → Next
Current design → failure mode → next upgrade.
Framework 4 — Draw → Name → Why → Risk
Narrate every box as you draw it.
Interview tip
Pick one framework and stay inside it. Switching structures mid-talk makes English sound messy.
System Design English knowledge base
← System Design English knowledge base — start at How to Explain Scaling from Zero to Millions in English for the full chapter index and reading order.
Explore every Mockly article on the blog hub.
All chapters in this series
Jump between System Design English posts below. Each chapter builds spoken English for the same book — use the KB overview when you want the big picture.
| # | Chapter | Slug |
|---|---|---|
| 1 | How to Explain Scaling from Zero to Millions in English | how-to-explain-scaling-from-zero-to-millions-in-english |
| 2 | How to Say Back-of-the-Envelope Estimates in English | how-to-say-back-of-the-envelope-estimates-in-english |
| 3 | Spoken System Design Framework in English (ESL) | spoken-system-design-interview-framework-in-english |
| 4 | How to Explain a Rate Limiter in English | how-to-explain-a-rate-limiter-in-english |
| 5 | Consistent Hashing Interview Vocabulary in English | consistent-hashing-interview-vocabulary-in-english |
| 6 | How to Explain a Key-Value Store in English | how-to-explain-a-key-value-store-in-english |
| 7 | How to Explain a Unique ID Generator in English | how-to-explain-a-unique-id-generator-in-english |
| 8 | URL Shortener Interview Phrases in English (ESL) | url-shortener-interview-phrases-in-english |
| 9 | How to Narrate a Web Crawler in English (you are here) | how-to-narrate-a-web-crawler-in-english |
| 10 | How to Explain a Notification System in English | how-to-explain-a-notification-system-in-english |
| 11 | How to Explain a News Feed in English | how-to-explain-a-news-feed-in-english |
| 12 | Chat System Interview Phrases in English (ESL) | chat-system-interview-phrases-in-english |
| 13 | Think Out Loud: Search Autocomplete in English | think-out-loud-search-autocomplete-in-english |
| 14 | How to Explain YouTube System Design in English | how-to-explain-youtube-system-design-in-english |
| 15 | How to Explain Google Drive System Design in English | how-to-explain-google-drive-system-design-in-english |
- How to Explain Scaling from Zero to Millions in English
- How to Say Back-of-the-Envelope Estimates in English
- Spoken System Design Framework in English (ESL)
- How to Explain a Rate Limiter in English
- Consistent Hashing Interview Vocabulary in English
- How to Explain a Key-Value Store in English
- How to Explain a Unique ID Generator in English
- URL Shortener Interview Phrases in English (ESL)
- How to Narrate a Web Crawler in English — you are here
- How to Explain a Notification System in English
- How to Explain a News Feed in English
- Chat System Interview Phrases in English (ESL)
- Think Out Loud: Search Autocomplete in English
- How to Explain YouTube System Design in English
- How to Explain Google Drive System Design in English
Frequently Asked Questions
Tap a question to expand the answer.