Quick Answer
How to narrate a web crawler in English: walk the frontier, politeness rules, fetch/parse, and storage — naming one bottleneck and one risk at each stage.
Diagrams and what to say while drawing
Narrate each figure out loud. Point to the box you are talking about.
What to say on figure 1
Frontier → fetcher workers → parser → storage.
What to say on figure 2
Show per-host politeness queues.
What to say on figure 3
Show URL seen-set / bloom filter.
What to say on figure 4
Show retry path for failed fetches.
Key vocabulary
| Term | Plain meaning | Say it in an interview |
|---|---|---|
| URL frontier | Queue of URLs to visit | “The frontier schedules the next URLs.” |
| Politeness | Avoid overloading a host | “We enforce politeness per domain.” |
| Deduplication | Skip seen URLs/content | “We deduplicate with a bloom filter or URL store.” |
| Robots.txt | Crawl permission file | “We respect robots.txt before fetching.” |
ESL English phrases and transitions
Clarify
- “Is this a general web crawl or limited to a site list?”
- “Do we need near-real-time freshness or batch crawl cycles?”
Propose and check in
- “The next bottleneck is X, so I would add Y.”
- “Does this level of detail work, or should I go deeper?”
Tradeoffs
- “The advantage is simplicity; the downside is…”
- “I’d choose A over B because…”
Key English language pitfalls
| Pitfall | Sounds like | Say instead |
|---|---|---|
| Ignoring robots.txt | Legal/ethics gap | Say you respect robots.txt |
| One giant queue | Head-of-line blocking | Mention host-based queues |
| Summary | Every new box needs one spoken reason. | |
Common English mistakes
| Mistake | What the interviewer concludes | Better English |
|---|---|---|
| No politeness | Looks reckless | Always mention per-host limits |
| Infinite crawl story | No stop condition | Talk about prioritisation and seen URLs |
What the interviewer is testing
This is a communication test as much as a design test. Can you explain a web crawler as a calm collaboration?
Interviewers listen for structure, tradeoffs, and clear English under pressure.
Key takeaway
Your job is to narrate decisions the interviewer can follow and challenge — not to dump tools.
How to open the problem in English
Opening script you can practise aloud
- “Is this a general web crawl or limited to a site list?”
- “Do we need near-real-time freshness or batch crawl cycles?”
Watch Out
Do not open with a finished architecture. Show the path.
Web crawler system design
Use this sequence for web crawler system design. For each stage, say what changes, why, and what can still go wrong.
Seed & frontier
Start from seeds; enqueue new links with priorities.
Politeness
Per-host rate limits and robots.txt checks.
Fetch & parse
Download pages, extract links, handle failures.
Store & dedup
Save content metadata; skip already-seen URLs.
Weak vs strong answers
Weak
“Workers download the internet.”
Strong
“A frontier schedules URLs politely per host; workers fetch and parse; we dedupe URLs so we don’t recrawl endlessly.”
Metrics and how to say numbers
| Write | Say |
|---|---|
| pages/sec | pages per second |
| per-host delay | a delay between requests to the same host |
Follow-up questions you will get
Likely follow-ups
- “How do you prioritise?” → “I’d rank by importance signals and freshness needs, not pure FIFO.”
Practice drill
Checklist
- Frontier
- Politeness
- Parse
- Dedup
“If you can explain a web crawler without notes, you are ready to draw it in an interview.”
Answer frameworks you can reuse
Use these spoken templates under pressure. Fill the blanks — do not invent a new structure mid-interview.
Framework 1 — Whole answer
Clarify → High-level → Deep dive → Check-in.
Framework 2 — Each component (B-C-B-C)
Bottleneck → Component → Benefit → Cost.
Framework 3 — Now → Breaks → Next
Current design → failure mode → next upgrade.
Framework 4 — Draw → Name → Why → Risk
Narrate every box as you draw it.
Interview tip
Pick one framework and stay inside it. Switching structures mid-talk makes English sound messy.
Frequently Asked Questions
Tap a question to expand the answer.