Project P2 — Async Web Scraper Package

Goal

Build an installable async scraper package that can:

• read a list of URLs,
• fetch concurrently with bounded parallelism,
• parse structured fields,
• stream results to JSONL/CSV,
• and expose both library API and CLI usage.

Demo

python -m scraper scrape urls.txt --concurrency 20 --out results.jsonl
scraper scrape urls.txt --concurrency 20 --out results.jsonl

Architecture

scraper/
├── pyproject.toml
├── src/scraper/
│   ├── __init__.py
│   ├── __main__.py
│   ├── cli.py
│   ├── fetch.py
│   ├── parse.py
│   ├── pipeline.py
│   ├── retries.py
│   └── types.py
└── tests/
  ├── test_parse.py
  ├── test_retry.py
  └── test_pipeline.py

Flow: URL source -> async fetch -> parse -> validate -> sink

Implementation blueprint

1) Bounded async fetcher

import asyncio
import aiohttp
 
 
async def fetch_one(session: aiohttp.ClientSession, url: str, sem: asyncio.Semaphore) -> tuple[str, str]:
  async with sem:
    async with session

2) Retry policy wrapper

import asyncio
 
 
async def with_retries(coro_factory, attempts: int = 3, delay: float = 0.5):
  last = None
  for i in range(attempts):
    try:
      return await coro_factory()
    except

3) Streaming sink

import json
 
 
def write_jsonl(path: str, rows):
  with open(path, "w", encoding="utf-8") as f:
    for row in rows:
      f.write(json

Packaging baseline

[project]
name = "async-web-scraper"
version = "0.1.0"
requires-python = ">=3.11"
 
[project.scripts]
scraper = "scraper.__main__:main"

Mandatory package quality:

• src/ layout
• typed public interfaces
• tested parse/retry/pipeline behavior
• README install/run docs

Testing baseline

Core tests expected:

1. parser resilience on malformed HTML
2. retry behavior retries then raises
3. pipeline preserves input/output count contracts
4. CLI args validation and exit codes

Example:

import pytest
 
 
@pytest.mark.asyncio
async def test_retry_succeeds_after_failure():
  calls = {"n": 0}
 
  async def flaky():
    calls["n"] += 1
    if calls["n"] < 2

Release checklist

Before considering P2 complete:

1. pytest -q passes
2. mypy src passes for project modules
3. lint/format checks pass
4. package builds (python -m build) and twine check dist/* passes
5. CLI demo commands work end-to-end
6. TestPyPI publish rehearsal succeeds

ADR notes

1. aiohttp over sync requests: native async concurrency and connection reuse.
2. Semaphore over unbounded task fan-out: controlled resource usage and stability.
3. Streaming sinks over full in-memory aggregation: predictable memory profile.
4. Retry at fetch boundary: isolate transient network failures from parser logic.

Interview defense notes

Be ready to explain:

1. Why async is chosen for this workload.
2. Where cancellation and timeout policy is enforced.
3. How retry policy avoids masking permanent failures.
4. How package boundaries improve testability.
5. What scaling changes you would add next (queueing, distributed workers, observability).

Summary

P2 is the bridge from "I know async syntax" to "I can ship an async product": architecture, operational policy, packaging discipline, and reproducible release workflow.

Self-Check Quiz