Site Crawling & Mapping
Site traversal, sitemaps, URL discovery and bulk crawling.
- Endpoints
- 6
- Pricing
- 0 markup
- Domain
- Search & SEO
;;;;----=====++++++++=====----;;;::::····........ .............···········............ ......····:::::;;;;--------------;;;;;:::::······.................. ;;;----===++++++++++++++====---;;;;:::·····......... .................·····.............. ......····::::;;;;;--------------;;;;;:::::·······................ ;;;---===++++++*****++++++===----;;;::::·····...................................·····............. ......····::::;;;;;-------------;;;;;;::::::········............. ;;;---===++++*********+++++===---;;;;::::·····.............................···········........... ......····::::;;;;;------------;;;;;;:::::::···················· ;;;---===++++**********++++====---;;;;::::·······....................··················......... ......····::::;;;;;;;------;;;;;;;;;:::::::···················· :;;;---===+++++*******+++++====----;;;:::::··········........···························........ ......····:::::;;;;;;;;;;;;;;;;;;:::::::::·················::: ::;;---====++++++++++++++++====---;;;;;:::::·····························::::::··········....... ......·····::::::;;;;;;;;;;;;::::::::::················:::::: ::;;;----====++++++++++++=====----;;;;:::::::····················:::::::::::::::::::······...... ......······:::::::::::::::::::::::··················::::::: :::;;;;----=================----;;;;;;:::::::···············::::::::::::::::::::::::::·····...... ......·······::::::::::::::::·······················::::::; ·::::;;;-----===========------;;;;;;::::::::·············::::::::::;;;;;;;;;;;;;;;::::::·····..... .......·············································::::::; ··::::;;;;;----------------;;;;;;::::::::···············:::::::::;;;;;;;;;;;;;;;;;;;;::::·····..... ........························...........·········:::::; ···:::::;;;;;;;-----;;;;;;;;;::::::::···················:::::::;;;;;;;;---------;;;;;;::::·····..... .........···············.......................······::::; ····::::::;;;;;;;;;;;;;;::::::::························:::::::;;;;;;--------------;;;;;::::····..... .............·····................................·····:::: ·····:::::::::::::::::::::::··········..........·········::::::;;;;;;---------------;;;;;:::·····..... .....................................................·····::: ······:::::::::::::::::·········....................·······:::::;;;;;----------------;;;;::::····...... .................................. ........·····: ······::::::::::::··········..........................······:::::;;;;;---------------;;;;;::::····...... ..............·····............... ........···· ······::::::::··········............ ..........······::::;;;;;--------------;;;;;::::·····....... ............·············............ .......··· ·····:::::::··········......... ........·····:::::;;;;;;---------;;;;;;:::::·····........ ...........····················......... ......·· ·:::::::::::········........ .......·····:::::;;;;;;;;;;;;;;;;;;::::::·····........... ............···········:::::··········......... ....... ::::::::::::·······........ .......·····::::::;;;;;;;;;;;;;::::::·······.............................········:::::::::::::::::·······....... ...... :::::::::::::······....... .......······::::::::::::::::::::········...........................·······:::::::;;;;;;;;;;:::::::·····....... ..... ;;;;;;;;::::::·····...... .......·······::::::::::::::·········..........................······::::::;;;;;;;;;;;;;;;;;;;:::::·····...... ..... ---;;;;;;;;::::·····..... ........·························...........................······:::::;;;;;--------------;;;;;:::::····....... ..... --------;;;;::::····...... .........·················...............................······::::;;;;------=========-----;;;;::::·····....... ..... =====-----;;;;:::····..... .................................... ..........·····::::;;;;----===============----;;;;::::····........ ....... +=======----;;;:::····..... ............................ ........····::::;;;----=====++++++++++====----;;;::::·····......... .......... ++++++====---;;;:::····..... ..................... .......····::::;;;---====++++++++++++++====---;;;;::::·····............................. **++++++===---;;;:::···...... ................ ......····:::;;;---===+++++++***++++++====---;;;;::::······.........................·· *****++++===---;;;:::···...... ............. .....····:::;;;---====+++++*****++++++====---;;;;::::········.................······· ******++++===--;;;:::····...... ............ .....····:::;;;---===+++++++**+++++++====----;;;;:::::······························ ******++++===---;;;:::····...... ................ .....···::::;;----===++++++++++++++=====----;;;;::::::························::::: ******++++===---;;;:::·····....... ...................... .....···:::;;;----=====+++++++++======----;;;;;:::::::·················::::::::::: ****+++++===----;;;::::····.......... ............................. ....····:::;;;;----================-----;;;;;:::::::::···········::::::::::::;;;; +++++++====----;;;::::······................................................... .....····::::;;;------=========-------;;;;;;:::::::::········:::::::::::;;;;;;;;; +++=======----;;;:::::······...........................················......... .....····::::;;;;;---------------;;;;;;;:::::::::··········::::::::::;;;;;;;;;-- ======------;;;;:::::······.......................························....... .....·····:::::;;;;;;;;;-;;;;;;;;;;;::::::::················::::::::;;;;;;;----- --------;;;;;;:::::······......................···········::::::::::········....... ......·····::::::;;;;;;;;;;;;;;::::::::······················:::::::;;;;;------- ;;;;;;;;;;::::::·······......................········:::::::::::::::::::······...... ......······::::::::::::::::::::::····························::::::;;;;;------- ;;;;;:::::::········........................·······:::::::::;;;;;;;:::::::······...... .......·······::::::::::::::::···········................········:::::;;;;;------ ::::::::········............................······:::::::;;;;;;;;;;;;;;:::::·····....... .......·········:::::::::::··········........................······:::::;;;;------ ::·········............ ..........······:::::;;;;;;;;;;;;;;;;;;:::::·····....... ........··········::::::···········.............. ............······::::;;;;;---- ·······.......... ........·····:::::;;;;;;;------;;;;;;;:::::·····....... ..........··········::::::··········........... ........·····:::::;;;;;-- ···.......... .......····:::::;;;;;;---------;;;;;;:::::·····.......... ............·········::::::::::·········......... .......·····:::::;;;;; .......... ......·····::::;;;;;;----------;;;;;;:::::······...........................········::::::::::::::::·······........ .......·····::::::;; ....... ......····::::;;;;;;;-------;;;;;;;::::::·······....................·········:::::::::::::::::::::······....... .......·····:::::: ...... ......····:::::;;;;;;;;;;;;;;;;;;;:::::::·········............·········::::::::;;;;;;;;;;;;;::::::·····...... .......······::: ..... ......····::::::;;;;;;;;;;;;;;;;::::::::··························:::::::;;;;;;;;;;;;;;;;;;;;:::::····...... .......······· .... .....·····::::::;;;;;;;;;;;;:::::::::·······················:::::::;;;;;;---------------;;;;;::::····...... .........····
Get started with your agent
Set up https://www.agentrouter.to/SKILL.md, then show me what I can do with Site Crawling & Mapping on AgentRouter.
Endpoints
AnyAPIWebsite Crawl
/web.crawlCrawl a website and get up to 10 of its pages as separate items, each with its URL, title, description, language, HTTP status, clean Markdown text and page metadata such as headings and structured data, ideal for feeding sites into LLMs and search indexes.
$0.0125maximum per call- AnyAPI
Web Map
/web.mapMap an entire website into a clean list of its URLs (with titles and descriptions) in a single call.
$0.001maximum per call 
Context.devCrawl website pages
/crawlCrawl pages starting at a website URL and return their content as Markdown, up to the requested page limit.
$0.003per credit- Context.dev
Site URL discovery
/map_urlsList the URLs of a website (sitemaps and links), optionally filtered by a search term or regex.
$0.003per credit 
Search1APICrawl a site asynchronously
/deepcrawlCrawl multiple pages from a starting URL in an asynchronous job.
$0.02maximum per call- Search1API
List URLs in a sitemap
/sitemapRetrieve a website's sitemap links from a URL to discover the pages it lists.
$0.001maximum per call