网站抓取与结构映射
Site traversal, sitemaps, URL discovery and bulk crawling.
- 工具接口
- 6
- 价格
- 零加价
- 所属领域
- 搜索与网页数据
;;;;----=====++++++++=====----;;;::::····........ .............···········............ ......····:::::;;;;--------------;;;;;:::::······.................. ;;;----===++++++++++++++====---;;;;:::·····......... .................·····.............. ......····::::;;;;;--------------;;;;;:::::·······................ ;;;---===++++++*****++++++===----;;;::::·····...................................·····............. ......····::::;;;;;-------------;;;;;;::::::········............. ;;;---===++++*********+++++===---;;;;::::·····.............................···········........... ......····::::;;;;;------------;;;;;;:::::::···················· ;;;---===++++**********++++====---;;;;::::·······....................··················......... ......····::::;;;;;;;------;;;;;;;;;:::::::···················· :;;;---===+++++*******+++++====----;;;:::::··········........···························........ ......····:::::;;;;;;;;;;;;;;;;;;:::::::::·················::: ::;;---====++++++++++++++++====---;;;;;:::::·····························::::::··········....... ......·····::::::;;;;;;;;;;;;::::::::::················:::::: ::;;;----====++++++++++++=====----;;;;:::::::····················:::::::::::::::::::······...... ......······:::::::::::::::::::::::··················::::::: :::;;;;----=================----;;;;;;:::::::···············::::::::::::::::::::::::::·····...... ......·······::::::::::::::::·······················::::::; ·::::;;;-----===========------;;;;;;::::::::·············::::::::::;;;;;;;;;;;;;;;::::::·····..... .......·············································::::::; ··::::;;;;;----------------;;;;;;::::::::···············:::::::::;;;;;;;;;;;;;;;;;;;;::::·····..... ........························...........·········:::::; ···:::::;;;;;;;-----;;;;;;;;;::::::::···················:::::::;;;;;;;;---------;;;;;;::::·····..... .........···············.......................······::::; ····::::::;;;;;;;;;;;;;;::::::::························:::::::;;;;;;--------------;;;;;::::····..... .............·····................................·····:::: ·····:::::::::::::::::::::::··········..........·········::::::;;;;;;---------------;;;;;:::·····..... .....................................................·····::: ······:::::::::::::::::·········....................·······:::::;;;;;----------------;;;;::::····...... .................................. ........·····: ······::::::::::::··········..........................······:::::;;;;;---------------;;;;;::::····...... ..............·····............... ........···· ······::::::::··········............ ..........······::::;;;;;--------------;;;;;::::·····....... ............·············............ .......··· ·····:::::::··········......... ........·····:::::;;;;;;---------;;;;;;:::::·····........ ...........····················......... ......·· ·:::::::::::········........ .......·····:::::;;;;;;;;;;;;;;;;;;::::::·····........... ............···········:::::··········......... ....... ::::::::::::·······........ .......·····::::::;;;;;;;;;;;;;::::::·······.............................········:::::::::::::::::·······....... ...... :::::::::::::······....... .......······::::::::::::::::::::········...........................·······:::::::;;;;;;;;;;:::::::·····....... ..... ;;;;;;;;::::::·····...... .......·······::::::::::::::·········..........................······::::::;;;;;;;;;;;;;;;;;;;:::::·····...... ..... ---;;;;;;;;::::·····..... ........·························...........................······:::::;;;;;--------------;;;;;:::::····....... ..... --------;;;;::::····...... .........·················...............................······::::;;;;------=========-----;;;;::::·····....... ..... =====-----;;;;:::····..... .................................... ..........·····::::;;;;----===============----;;;;::::····........ ....... +=======----;;;:::····..... ............................ ........····::::;;;----=====++++++++++====----;;;::::·····......... .......... ++++++====---;;;:::····..... ..................... .......····::::;;;---====++++++++++++++====---;;;;::::·····............................. **++++++===---;;;:::···...... ................ ......····:::;;;---===+++++++***++++++====---;;;;::::······.........................·· *****++++===---;;;:::···...... ............. .....····:::;;;---====+++++*****++++++====---;;;;::::········.................······· ******++++===--;;;:::····...... ............ .....····:::;;;---===+++++++**+++++++====----;;;;:::::······························ ******++++===---;;;:::····...... ................ .....···::::;;----===++++++++++++++=====----;;;;::::::························::::: ******++++===---;;;:::·····....... ...................... .....···:::;;;----=====+++++++++======----;;;;;:::::::·················::::::::::: ****+++++===----;;;::::····.......... ............................. ....····:::;;;;----================-----;;;;;:::::::::···········::::::::::::;;;; +++++++====----;;;::::······................................................... .....····::::;;;------=========-------;;;;;;:::::::::········:::::::::::;;;;;;;;; +++=======----;;;:::::······...........................················......... .....····::::;;;;;---------------;;;;;;;:::::::::··········::::::::::;;;;;;;;;-- ======------;;;;:::::······.......................························....... .....·····:::::;;;;;;;;;-;;;;;;;;;;;::::::::················::::::::;;;;;;;----- --------;;;;;;:::::······......................···········::::::::::········....... ......·····::::::;;;;;;;;;;;;;;::::::::······················:::::::;;;;;------- ;;;;;;;;;;::::::·······......................········:::::::::::::::::::······...... ......······::::::::::::::::::::::····························::::::;;;;;------- ;;;;;:::::::········........................·······:::::::::;;;;;;;:::::::······...... .......·······::::::::::::::::···········................········:::::;;;;;------ ::::::::········............................······:::::::;;;;;;;;;;;;;;:::::·····....... .......·········:::::::::::··········........................······:::::;;;;------ ::·········............ ..........······:::::;;;;;;;;;;;;;;;;;;:::::·····....... ........··········::::::···········.............. ............······::::;;;;;---- ·······.......... ........·····:::::;;;;;;;------;;;;;;;:::::·····....... ..........··········::::::··········........... ........·····:::::;;;;;-- ···.......... .......····:::::;;;;;;---------;;;;;;:::::·····.......... ............·········::::::::::·········......... .......·····:::::;;;;; .......... ......·····::::;;;;;;----------;;;;;;:::::······...........................········::::::::::::::::·······........ .......·····::::::;; ....... ......····::::;;;;;;;-------;;;;;;;::::::·······....................·········:::::::::::::::::::::······....... .......·····:::::: ...... ......····:::::;;;;;;;;;;;;;;;;;;;:::::::·········............·········::::::::;;;;;;;;;;;;;::::::·····...... .......······::: ..... ......····::::::;;;;;;;;;;;;;;;;::::::::··························:::::::;;;;;;;;;;;;;;;;;;;;:::::····...... .......······· .... .....·····::::::;;;;;;;;;;;;:::::::::·······················:::::::;;;;;;---------------;;;;;::::····...... .........····
让你的 Agent 开始使用
Set up https://www.agentrouter.to/SKILL.md, then show me what I can do with 网站抓取与结构映射 on AgentRouter.
工具接口
AnyAPI网站抓取
/web.crawl抓取一个网站，最多将其中 10 个页面分别作为结果项返回，包含 URL、标题、描述、语言、HTTP 状态、干净的 Markdown 正文，以及标题层级、结构化数据等页面元数据，适合用于大模型和搜索索引。失败或被跳过的页面单独列出，且不会计费。
$0.0125maximum per call- AnyAPI
网站链接映射
/web.map一次调用将整个网站整理为清晰的 URL 列表，包含标题和描述。
$0.001maximum per call 
Context.dev抓取网站页面
/crawl从网站 URL 开始爬取页面，将内容作为 Markdown 返回，最多爬取指定页数。
$0.003per credit- Context.dev
发现网站链接
/map_urls列出网站的 URL（站点地图和链接），可选按搜索词或正则表达式筛选。
$0.003per credit 
Search1API异步抓取网站
/deepcrawl通过异步任务从起始 URL 爬取多个页面，获取收集到的页面和爬取统计数据；大型响应则返回结果 URL。
$0.02maximum per call- Search1API
列出网站地图中的 URL
/sitemap通过 URL 获取网站的站点地图链接，发现其中列出的页面。
$0.001maximum per call