How to Build a Reliable List Crawler with Python and Nstproxy
TL;DR
List crawling extracts repeated records across a controlled page sequence. The sequence may use numbered pages, a next link, a cursor, a load-more action, or a supplied URL queue.
BeautifulSoup is the clearest option for a small static list. It gives you direct control, but you must implement requests, pagination, retries, deduplication, checkpoints, and exports.
Scrapy is the stronger Python foundation for recurring multi-page work. Its scheduler, retry middleware, feed exports, statistics, and resumable jobs remove infrastructure code without removing selector maintenance.
Nstproxy Crawl fits when retrieval is the harder problem. It provides bounded discovery, JavaScript rendering, proxy routing, and page outputs while your pipeline keeps responsibility for item identity and validation.
A schema does not guarantee correct data. Production acceptance should also check provenance, duplicate rate, required-field completeness, pagination termination, and unexpected value distributions.
Introduction: list crawling is a state-management problem
List crawling looks like a selector exercise until the first retry, overlapping page, silent template change, or infinite cursor loop. A reliable crawler must know which pages were attempted, which records were accepted, why a record was rejected, and where a restart should resume. This guide tests two Python approaches against a public scraping sandbox, then shows where Nstproxy Crawl can replace the access and rendering layer.
The examples deliberately stop after two pages. That bounded run is enough to verify selectors and control flow without turning a tutorial into an unnecessary full-site collection job.
What is list crawling?
List crawling is the repeated discovery and extraction of similarly shaped records from list pages or a predefined URL queue. Typical targets include product grids, public directories, article indexes, event calendars, and authorized job listings. Each record normally carries both business fields and provenance fields such as , , , , and .
List crawling combines two jobs that should remain separate. Crawling finds the next page or URL; scraping converts each page into records. The distinction is explained further in web scraping versus web crawling.
Pagination determines the queue shape. Numbered pages expose an index, next-link pagination exposes a chain, cursor APIs expose opaque state, and infinite scroll often exposes either a background request or a browser action. The pagination glossary covers these mechanisms in more detail.
Why list crawling fails after a successful demo
List crawling fails in production because a downloaded page is not the same as an accepted record. A response can return an HTTP success status while showing a consent screen, soft error, empty application shell, or repeated page. A parser can also return plausible but wrong text after a layout change.
The first Python test exposed a smaller version of this problem: the server's body declared UTF-8, but the initial Requests decode produced mojibake in the currency and punctuation. Setting the response encoding from detected content corrected the text. That is the kind of defect a “records found” counter misses.
Before crawling, check the site's terms, published policy, and official API options. Google's Robots Exclusion Protocol documentation explains how rules apply to a specific host, protocol, and port. Robots rules govern crawler access signals; they do not grant rights to reuse personal, copyrighted, or restricted data.
Which list crawling method should you choose?
Choose the method by operational responsibility, not by code length.
Bounded site discovery or submitted target handled by the service
JavaScript pages
Requires a separate browser
Requires a rendering integration
Browser rendering is available as a crawl option
Extraction model
CSS/tag parsing you own
CSS/XPath parsing and pipelines you own
Page outputs are managed; exact business records still need validation
Restart and observability
Build checkpoints and metrics
Stats plus persistent-job support
Crawl records and task progress are managed; downstream record state remains yours
Maintenance surface
HTTP, parsing, retries, storage
Selectors, spider rules, pipelines, deployment
Crawl configuration, record parser, schema tests, and vendor integration
Best fit
Small static lists and prototypes
Recurring Python crawlers with custom logic
Dynamic or access-sensitive sites where crawler infrastructure is the bottleneck
The Beautiful Soup documentation defines the library as an HTML/XML parsing tool, not a scheduler or downloader. Scrapy covers more of the crawl lifecycle. Nstproxy Crawl covers retrieval, rendering, bounded discovery, routing, and page-level outputs, but it should not be treated as a substitute for domain validation.
Detailed Tutorial
The tutorial uses https://books.toscrape.com/catalogue/page-1.html, a public sandbox built for scraping practice. Both Python methods were executed with current packages and returned 40 unique records from two list pages.
Method 1: crawl a static list with BeautifulSoup
BeautifulSoup is appropriate when records and the next link are present in the initial HTML and the run is small enough for a single process.
Step 1: install and define the record contract
Install requests and beautifulsoup4. The example keeps raw price text instead of coercing a number because currency parsing is a separate domain rule. It derives a stable item ID from the canonical detail path and keeps the source list URL for traceability.
The Requests documentation recommends explicit timeouts for production requests and distinguishes successful JSON decoding from a successful HTTP response. The same discipline applies to HTML.
Step 2: follow the next link and reject repeated pages
import hashlib, json
from datetime import datetime, timezone
from urllib.parse import urljoin, urlsplit
import requests
from bs4 import BeautifulSoup
next_url ="https://books.toscrape.com/catalogue/page-1.html"max_pages =2session = requests.Session()session.headers["User-Agent"]="Nstproxy-list-crawl-tutorial/1.0"seen_items, seen_pages ={},set()run_time = datetime.now(timezone.utc).isoformat()pages =0while next_url and pages < max_pages: response = session.get(next_url, timeout=(5,20)) response.raise_for_status() response.encoding = response.apparent_encoding
soup = BeautifulSoup(response.text,"html.parser") cards = soup.select("article.product_pod")ifnot cards:raise RuntimeError(f"No product cards found on {response.url}") page_ids =[]for card in cards: link = card.select_one("h3 a[href]") detail_url = urljoin(response.url, link["href"]) parts =[p for p in urlsplit(detail_url).path.split("/")if p] record ={"item_id": parts[-2],"title": link["title"].strip(),"detail_url": detail_url,"list_url": response.url,"price_text": card.select_one("p.price_color").get_text(strip=True),"observed_at": run_time,} seen_items[record["item_id"]]= record
page_ids.append(record["item_id"]) fingerprint = hashlib.sha256("\n".join(sorted(page_ids)).encode()).hexdigest()if fingerprint in seen_pages:raise RuntimeError(f"Repeated page detected at {response.url}") seen_pages.add(fingerprint) pages +=1 next_link = soup.select_one("li.next a[href]") next_url = urljoin(response.url, next_link["href"])if next_link elseNonefor record in seen_items.values():print(json.dumps(record, ensure_ascii=False))print({"pages": pages,"accepted_records":len(seen_items)})
Step 3: interpret the result before scaling
The verified run returned {"pages": 2, "accepted_records": 40}. That proves the selectors and next-link traversal worked for the sampled pages. It does not prove that every later page has the same template, that prices are semantically valid, or that a future run will be identical.
Before increasing max_pages, write records to a table with a uniqueness constraint on item_id, persist the last completed page after a successful commit, and alert if required-field completeness or records-per-page changes sharply.
Method 2: move the queue to Scrapy
Scrapy is appropriate when you want a scheduler, retry middleware, exports, crawl statistics, and a path to persistent jobs while retaining Python selectors and pipelines.
Step 1: express records and pagination as spider output
Install scrapy, save this spider, and keep the two-page guard during validation:
from urllib.parse import urlsplit
import scrapy
classBookListSpider(scrapy.Spider): name ="book_list" start_urls =["https://books.toscrape.com/catalogue/page-1.html"] custom_settings ={"ROBOTSTXT_OBEY":True,"DOWNLOAD_DELAY":0.5,"CONCURRENT_REQUESTS_PER_DOMAIN":2,"RETRY_TIMES":2,"LOG_LEVEL":"INFO",}defparse(self, response): cards = response.css("article.product_pod")ifnot cards: self.logger.error("No product cards found on %s", response.url)returnfor card in cards: detail_url = response.urljoin(card.css("h3 a::attr(href)").get()) parts =[p for p in urlsplit(detail_url).path.split("/")if p]yield{"item_id": parts[-2],"title": card.css("h3 a::attr(title)").get().strip(),"detail_url": detail_url,"list_url": response.url,"price_text": card.css("p.price_color::text").get().strip(),} page =int(response.url.rsplit("-",1)[-1].split(".")[0]) next_href = response.css("li.next a::attr(href)").get()if next_href and page <2:yield response.follow(next_href, callback=self.parse)
Step 2: run an overwrite-safe test export
Run scrapy runspider list_scrapy.py -O books.jl. The verified run used Scrapy 2.13.4, received the two list pages, exported 40 JSON Lines records, and finished normally. The additional request was the site's missing robots.txt, which returned a not-found response; that observation belongs in the run log rather than being silently ignored.
Step 3: add resumability and data-quality gates
For a scheduled crawler, use Scrapy's persistent job directory rather than treating output files as checkpoints. The Scrapy jobs documentation describes pausing and resuming a crawl with JOBDIR and warns that one directory must belong to only one job.
Add an item pipeline that normalizes fields, upserts by item_id, and rejects missing required values. Export spider statistics to monitoring. A finished spider is operationally successful only when accepted-record counts and field distributions also pass.
Take a Quick Look
Test one representative list path in Nstproxy Crawl before expanding the boundary. Compare rendered page completeness, discovered URLs, and downstream accepted records against the Python baseline.
Method 3: Use Nstproxy Crawl for access and rendering
Nstproxy Crawl is appropriate when JavaScript rendering, proxy routing, bounded discovery, or crawler operations consume more effort than the record parser. The current product surface supports a Playground and API workflow, depth and page limits, include/exclude rules, browser rendering, proxy options, and Markdown, HTML, JSON, links, or PDF page outputs. Billing is usage-based per successfully crawled URL; price figures are intentionally omitted because they change. It is a strong fit for teams that want page retrieval and discovery managed while keeping schema logic in their own pipeline. It is not a promise that every page is accessible or that every returned field is correct.
Step 1: submit a representative list path
Open Nstproxy Crawl and begin with one public category or directory URL. Use the Playground before generating API code. The Nstproxy Crawl launch overview provides additional workflow context.
Step 2: set boundaries before enabling recursion
Set a small maximum page count and depth, include only relevant list/detail paths, and exclude carts, accounts, calendars, faceted duplicates, and files. Enable JavaScript only if the required records are absent from raw HTML. These choices limit accidental URL expansion and make a test run explainable.
Step 3: validate page outputs as inputs, not truth
Choose the lightest page format that preserves the fields your parser needs. Then apply the same item_id, provenance, duplicate, and completeness checks used by the Python methods. The structured data glossary explains why a typed shape is useful, but type validity still cannot prove that a value belongs to the correct item.
The live product page was verified for this guide, but an authenticated crawl was not executed because no account credential was available. Treat the Nstproxy method as a documented operational path until your own representative run passes the same acceptance gates.
What most list crawling tutorials leave out
Production list crawling needs explicit terminal states. Every page or URL should end as accepted, permanently rejected, or retryable with a bounded attempt count. “No next link” is not sufficient by itself: also stop on a repeated cursor, repeated page fingerprint, page limit, or deadline.
Schema validation is necessary but incomplete. A model can accept a syntactically valid title taken from the wrong card, a default price copied across every row, or an LLM-extracted attribute unsupported by the page. Store evidence fields, sample raw outputs, and compare distributions between runs.
Use these acceptance metrics:
candidate records versus accepted records;
required-field completeness by field;
duplicate rate after canonicalization;
list-page success and detail-page success separately;
pages that add no unseen item IDs;
freshness lag and crawl usage per accepted record.
Also separate list discovery from detail enrichment. If one detail page fails, retain the discovery record with an enrichment status. Otherwise a partial enrichment failure can erase evidence that the item existed.
Conclusion: choose the ownership boundary you can operate
BeautifulSoup is the clearest learning tool for a bounded static list, while Scrapy is the better Python base when scheduling, retries, exports, and resumability matter. Choose Nstproxy Crawl when browser rendering, proxy routing, and bounded page discovery are the operational bottleneck, then keep stable IDs, validation, checkpoints, and acceptance metrics in your data layer. Start with two representative pages, prove the record contract, and expand only after the crawler can stop and resume predictably.
For recurring crawls that combine several proxy sources and need centralized routing visibility, also evaluate Nstproxy Proxy Manager.
Experience Nstproxy — Start Your Free Trial Today
Run a bounded list-crawl sample, compare it with your Python baseline, and scale only when the accepted records—not merely the page responses—pass review.
No. A list crawler manages page or URL discovery and state, while a scraper extracts fields; practical list-crawling pipelines perform both jobs.
Q: Should I use BeautifulSoup or Scrapy?
Use BeautifulSoup for a small static workflow where you want explicit control, and use Scrapy when scheduling, retries, exports, crawl statistics, and resumable jobs justify a framework.
Q: How should a crawler stop pagination?
Combine signals: no next token, no unseen item IDs, no repeated fingerprint, and enforced page and time limits. One empty response should usually be investigated rather than treated as proof of completion.
Q: Does schema-based extraction eliminate data cleaning?
No. A schema can enforce shape and types, but you still need provenance, canonicalization, deduplication, semantic validation, and drift monitoring.
Q: When should browser rendering be enabled?
Enable browser rendering only when required records or pagination controls are missing from the initial HTML and appear after JavaScript executes.
Q: Is list crawling legal?
Legality depends on the data, jurisdiction, access method, site terms, and intended use. Prefer official APIs when suitable, respect published crawl controls, minimize collection, and obtain legal review for sensitive or commercial use cases.
Lena Zhou
Aug. 11th 2026
110M+ real IPs with 99.9% access success
Blazing-fast average response ~0.5s for high-concurrency tasks
From only $0.1/GB
Get immediate access to premium residential, datacenter, IPv6 and ISP proxy pools.
