Web ingestion

Sitemap crawler for AI agents

Finds and parses every URL in a site’s XML sitemaps.

utilities.sitemap_parser

What it does

Finds and parses every URL in a site’s XML sitemaps. It is one of the Web ingestion tools in the Ahura Utilities catalogue: a ready-made MCP tool that an agent calls with a typed input and gets a structured result back, with no library to install and no sandbox to start.

How agents use it

Agents use sitemap crawler to map a website’s structure before crawling it. Because it is a plain tool call, it slots into a single agent turn or a step in an Ahura Workflow, and large inputs and outputs are passed by reference using a store:// address from Ahura Store.

Interactive simulation

Run sitemap crawler

Press Run to send the call. The dashboard shows what it would use in a partition. Simulated: nothing is actually executed.

ahura mcp · ptn_demo_7f3a

# Map a website’s structure before crawling it.

→ POST /mcp

{
  "jsonrpc": "2.0",
  "id": 1,
  "method": "tools/call",
  "params": {
    "name": "utilities.sitemap_parser",
    "arguments": {
      "source": "https://example.com",
      "output": "memory://crawl/sitemap",
      "ttl": 3600
    }
  }
}

Ready. Press Run.

ptn_demo_7f3a

Partition · N. Virginia · created just now

Simulated

Activeus-eastRetention hold

days left

Run time

0.0s

0 calls

Size

2.3MB

0.00 ACU used

Services in this partition

Utilities
0 calls · 0.00 ACU
Memory
0 calls · 0.00 ACU

Spend by service

Ledger

No usage yet.

Good to know

Billed per call, with the price shown in tools/list. See pricing
Runs inside your partition, so inputs and results stay isolated and follow its retention settings.
Safe to chain: pair it with Ahura Workflows for retries and approvals.

More Web ingestion utilities

All utilities