MCP + Nstproxy Crawl: Build a Web Search MCP Server for Claude
TL;DR
A web search MCP server needs two tools, not one. A search/discovery tool that returns candidate URLs, and a fetch tool that returns clean page content for the LLM to actually read; most single-tool examples skip the second half.
Nstproxy Crawl fits the fetch half of that pattern. It renders JavaScript-heavy pages and routes requests through Nstproxy's proxy network, which addresses the bot-detection and geo-inconsistency problems a plain HTTP request runs into against a search results page.
Building the server takes two Python packages and about 50 lines of code. This article's version was installed, imported, and run against the real Model Context Protocol (MCP) SDK in the process of writing it, including a live stdio handshake between a test client and the server.
Nstproxy Crawl returns Markdown as a storage reference, not inline text. A working integration has to resolve that reference with a second call before an LLM can read the content -- a detail most quick-start examples skip.
The same server registers with Claude Desktop and Claude Code through the same stdio transport, using claude_desktop_config.json for the former and claude mcp add for the latter.
Each search call bills as one Nstproxy Crawl fetch of a results page, separate from each page-read call. A single user question that triggers a search plus three page reads costs four fetches, not one.
This pattern doesn't replace a purpose-built search API's query understanding or ranking. It's a way to add live web access to an agent using fetch infrastructure you control end to end, including which proxy pool and rendering settings each request uses.
What This Integration Enables
The Model Context Protocol (MCP) is an open-source standard, introduced by Anthropic and now supported across a wide range of clients and servers, for connecting AI applications to external systems -- data sources, tools, and workflows -- through a single, consistent interface. An LLM on its own can't search the web or open a URL; it can only work with what's in its training data and whatever text you paste into the conversation. An MCP server closes that gap by exposing callable tools -- functions with a name, a description, and a typed schema -- that a host application like Claude Desktop or Claude Code can list, call, and feed the results of back into the conversation.
Several hosted, managed web-search MCP servers already exist if you just want search working in five minutes with someone else's infrastructure behind it. Building your own trades that convenience for control: you decide which pages get JavaScript rendering, which proxy pool and geo-target each request uses, how results get cached, and what happens when a target site blocks the request. That control matters most against sites that actively resist plain HTTP scraping, which is most of the modern web including search engine results pages themselves -- see Nstproxy's broader comparison of web-scraping APIs for how Crawl's rendering and proxy layer stacks up against other fetch backends outside of the MCP context specifically.
Nstproxy Crawl is the fetch layer this build uses to fill that gap. It's an AI-oriented web crawling API: give it a URL and it returns clean Markdown, HTML, links, screenshots, or PDF output, with JavaScript rendering and Nstproxy's proxy network handling delivery underneath. It's built for agent tool-use scenarios specifically -- RAG ingestion, monitoring, and the search-then-read pattern this server implements -- rather than being a scraping framework you assemble and maintain yourself. Three things about it matter for an MCP server in particular:
JavaScript rendering -- search-results pages and many modern sites render their content client-side, so a plain HTTP GET returns an empty shell; Crawl executes the page in a real browser first.
Proxy-backed requests -- every fetch routes through Nstproxy's proxy network rather than the calling machine's own IP, which is what keeps a server making repeated search-engine requests from getting blocked after a handful of calls.
Storage-referenced large outputs -- Markdown, HTML, and other large fields come back as a reference token you resolve separately, which keeps the initial response small even when a page's content is long.
Take a Quick Look
If your agent's web requests keep getting blocked or served a JavaScript shell instead of real content, Nstproxy Crawl renders the page in a real browser and routes it through Nstproxy's proxy network before handing your MCP server clean Markdown.
You need Python 3.10 or newer -- the official mcp package requires it -- an MCP-compatible host such as Claude Desktop or Claude Code, and an Nstproxy account with a Crawl API token, available from the Nstproxy Crawl product page after sign-up. Full endpoint and parameter documentation for the underlying API lives in Nstproxy's Crawl docs, useful if you want to add options like screenshots or PDF output beyond what this example uses. You don't need a separate search-engine API key for this build: the web_search tool below fetches a search engine's own results page through Nstproxy Crawl rather than calling a dedicated search API.
Install
Create a virtual environment and install the two packages this server needs: the official MCP Python SDK with its CLI extras, and Nstproxy's Crawl SDK.
Running this in the course of writing this article installed mcp 1.27.0 and nstdata-ai-crawl 0.1.1, the current versions of each package at the time of writing; pip install always resolves to whatever is newest when you run it. No other dependencies are required -- the Crawl SDK ships its own HTTP client internally.
Configure
Save the following as mcp_server.py. It defines two tools: web_search, which fetches a search engine's results page and returns the links on it, and read_page, which fetches any URL and returns its cleaned Markdown content.
import os
from mcp.server.fastmcp import FastMCP
from nstdata_ai_crawl import Format, NstDataClient, ScrapeRequestDto
NSTDATA_API_TOKEN = os.environ["NSTDATA_API_TOKEN"]client = NstDataClient(NSTDATA_API_TOKEN)mcp = FastMCP("nstproxy-web-search")def_extract_markdown(result)->str|None:"""Pull Markdown text out of a scrape result.
Nstproxy Crawl returns Markdown as a storage reference (markdownRef)
rather than inline text, so every result has to be resolved with
read_storage() before an LLM can read it.
"""ifnot result.data ornot result.data.markdownRef:returnNonereturn client.read_storage(result.data.markdownRef).decode("utf-8")@mcp.tool()defweb_search(query:str, max_results:int=5)->str:"""Search the web for a query and return candidate result URLs.
Args:
query: The search query.
max_results: Maximum number of result links to return (default 5).
""" result = client.submit_scrape_task_sync(ScrapeRequestDto( url=f"https://html.duckduckgo.com/html/?q={query}", formats=[Format.LINKS], onlyMainContent=False, timeout=60000,))ifnot result.success ornot result.data:returnf"Search fetch failed: {result.errorMessage or result.errorCode}" links =[ link for link in(result.data.links or[])if"duckduckgo.com"notin link
][:max_results]return"\n".join(f"- {link}"for link in links)if links else"No results found."@mcp.tool()defread_page(url:str)->str:"""Fetch a URL and return its cleaned Markdown content.
Args:
url: The page to fetch and read.
""" result = client.submit_scrape_task_sync(ScrapeRequestDto( url=url, formats=[Format.MARKDOWN], onlyMainContent=True, timeout=60000,))ifnot result.success ornot result.data:returnf"Could not read page: {result.errorMessage or result.errorCode}" markdown = _extract_markdown(result)return markdown or"Page fetched but returned no readable content."if __name__ =="__main__": mcp.run(transport="stdio")
NstDataClient.submit_scrape_task_sync blocks until the fetch completes, which is why both tool functions are plain def rather than async def -- FastMCP supports either. Format.LINKS and Format.MARKDOWN tell Nstproxy Crawl which representations of the page to return; requesting only what each tool needs keeps the response smaller and the storage-reference resolution step cheaper.
Real Handshake
Before wiring this into a live host, verify the server actually speaks MCP correctly, following the same pattern as the official MCP server-building guide. The official SDK's client can talk to a server over the same stdio transport a host uses, so you can test the handshake without Claude Desktop or Claude Code in the loop:
import asyncio
from mcp import ClientSession, StdioServerParameters
from mcp.client.stdio import stdio_client
asyncdefmain(): server_params = StdioServerParameters( command="python3", args=["mcp_server.py"], env={"NSTDATA_API_TOKEN":"dummy-key-for-handshake-test"},)asyncwith stdio_client(server_params)as(read, write):asyncwith ClientSession(read, write)as session:await session.initialize() tools =await session.list_tools()for t in tools.tools:print(f"- {t.name}: required={t.inputSchema.get('required')}")asyncio.run(main())
That output is a real client-server round trip: the client spawned the server as a subprocess, completed the MCP initialize handshake, and received both tools' auto-generated schemas back over tools/list -- no Nstproxy API call was needed for this step, since listing tools doesn't touch the network.
Attach Capabilities
Claude Desktop reads server definitions from claude_desktop_config.json (~/Library/Application Support/Claude/claude_desktop_config.json on macOS, %AppData%\Claude\claude_desktop_config.json on Windows). Add the server under mcpServers:
A ✔ Connected status means Claude Code successfully spawned the server process and completed the MCP handshake with it -- it doesn't confirm the Nstproxy Crawl token is valid, since that call only happens when a tool is actually invoked. If a tool call later fails with an authentication error, the token itself is the first thing to check, not the MCP wiring.
Worked Example
Once the server is attached, a request like "what changed in the MCP spec recently, and summarize the top result" plays out as: Claude calls web_search with a query about MCP spec changes, gets back a handful of candidate URLs, picks one that looks authoritative, and calls read_page on it to pull the actual page content into the conversation before answering. The illustrative shape of that second call's return value looks like this:
{"content":[{"type":"text","text":"# Model Context Protocol Changelog\n\n## 2026-07-28\n\n- ..."}]}
Claude then reads that Markdown text the same way it would read anything else in the conversation, and can cite the source URL it came from since that URL was the argument to read_page in the first place.
Limits
web_search here fetches one search engine's results page and returns raw links; it doesn't rank results, understand query intent, or paginate the way a purpose-built search API would. The "duckduckgo.com" not in link filter is a naive heuristic for excluding the search engine's own navigation links, not a robust results parser -- a production version should filter more precisely or switch to a search provider designed for programmatic use if ranking quality matters. Billing follows Nstproxy Crawl's per-fetch model: every web_search call and every read_page call is a separate billable fetch, so a single user question that triggers one search and three page reads costs four fetches, and the code above doesn't cache repeated lookups. Nstproxy Crawl also caps each request at a 60-second timeout and returns an HTTP 429 with a Retry-After header when a caller exceeds its rate limit; this minimal example doesn't implement retry or backoff, so a production version should add both. Finally, treat this as a tool for an agent doing bounded, on-demand lookups rather than bulk scraping -- respect the target sites' terms of service and robots.txt, and keep request volume proportional to an actual user question rather than running unattended crawls through the same credentials.
Conclusion
A working web search MCP server comes down to two tools -- one that discovers candidate URLs and one that fetches clean content from a specific URL -- wired to a fetch layer that can actually get past what modern websites do to block plain HTTP scraping. Nstproxy Crawl's JavaScript rendering and proxy-backed access cover that fetch layer, the official MCP Python SDK handles the protocol plumbing, and the same server attaches to both Claude Desktop and Claude Code over the same stdio transport. Verify the handshake with a real client before wiring it into a host, watch the per-fetch billing model if your agent searches and reads on every question, and treat the naive link-filtering in this example as a starting point rather than a finished search backend.
An MCP server is a program that exposes callable tools, data, or prompt templates to an AI application over the Model Context Protocol, so the AI can call real functions instead of only generating text.
Q: Do I need to use Nstproxy Crawl specifically, or can I fetch pages myself with plain HTTP requests?
You can fetch pages with any HTTP client, but plain requests typically fail against JavaScript-rendered pages and get blocked by sites with bot detection; Nstproxy Crawl handles both by rendering pages in a real browser and routing requests through its proxy network.
Q: Does this let Claude search the entire web like a search engine?
No. The web_search tool in this example fetches one search engine's results page and returns the links on it; it doesn't index the web itself or rank results the way a dedicated search engine does.
Q: How much does this cost to run?
Cost depends on Nstproxy Crawl's current per-fetch pricing, published on the Crawl product page, multiplied by how many searches and page reads your agent actually performs -- each is a separate billable fetch.
Q: Can I use this server with Claude Code as well as Claude Desktop?
Yes. Both connect to the same stdio server; Claude Desktop reads the server definition from claude_desktop_config.json while Claude Code registers it with claude mcp add.
Q: Is my Nstproxy API token exposed to the LLM?
No. The token lives in the server process's environment variables and is used only inside mcp_server.py to authenticate with Nstproxy Crawl; the LLM only ever sees the tool's return values, never the token itself.
Q: What happens if the target search engine blocks or rate-limits the request?
Nstproxy Crawl returns a non-success result with an error code that the web_search function surfaces as a plain-text failure message; this minimal example doesn't retry automatically, so a production version should add backoff and a fallback query strategy.
Q: Do I need to deploy this server somewhere, or can it run locally?
It can run entirely on your own machine as a local subprocess that Claude Desktop or Claude Code launches over stdio; you only need a public deployment if you're exposing the server to a remote host over HTTP instead.
Marcus Chen
Aug. 24th 2026
110M+ real IPs with 99.9% access success
Blazing-fast average response ~0.5s for high-concurrency tasks
From only $0.1/GB
Get immediate access to premium residential, datacenter, IPv6 and ISP proxy pools.
