爬取网站页面
/crawl爬取同一网站内指定数量的页面并返回内容,按请求的页面上限预留最高费用。
- 最近 7 天完成的调用
- 112
- 成功率
- 99.1%
- 执行耗时中位数
- 16.2s
统计不包含取消的调用。执行时间仅统计成功调用。
;;;;----=====++++++++=====----;;;::::····........ .............···········............ ......····:::::;;;;--------------;;;;;:::::······.................. ;;;----===++++++++++++++====---;;;;:::·····......... .................·····.............. ......····::::;;;;;--------------;;;;;:::::·······................ ;;;---===++++++*****++++++===----;;;::::·····...................................·····............. ......····::::;;;;;-------------;;;;;;::::::········............. ;;;---===++++*********+++++===---;;;;::::·····.............................···········........... ......····::::;;;;;------------;;;;;;:::::::···················· ;;;---===++++**********++++====---;;;;::::·······....................··················......... ......····::::;;;;;;;------;;;;;;;;;:::::::···················· :;;;---===+++++*******+++++====----;;;:::::··········........···························........ ......····:::::;;;;;;;;;;;;;;;;;;:::::::::·················::: ::;;---====++++++++++++++++====---;;;;;:::::·····························::::::··········....... ......·····::::::;;;;;;;;;;;;::::::::::················:::::: ::;;;----====++++++++++++=====----;;;;:::::::····················:::::::::::::::::::······...... ......······:::::::::::::::::::::::··················::::::: :::;;;;----=================----;;;;;;:::::::···············::::::::::::::::::::::::::·····...... ......·······::::::::::::::::·······················::::::; ·::::;;;-----===========------;;;;;;::::::::·············::::::::::;;;;;;;;;;;;;;;::::::·····..... .......·············································::::::; ··::::;;;;;----------------;;;;;;::::::::···············:::::::::;;;;;;;;;;;;;;;;;;;;::::·····..... ........························...........·········:::::; ···:::::;;;;;;;-----;;;;;;;;;::::::::···················:::::::;;;;;;;;---------;;;;;;::::·····..... .........···············.......................······::::; ····::::::;;;;;;;;;;;;;;::::::::························:::::::;;;;;;--------------;;;;;::::····..... .............·····................................·····:::: ·····:::::::::::::::::::::::··········..........·········::::::;;;;;;---------------;;;;;:::·····..... .....................................................·····::: ······:::::::::::::::::·········....................·······:::::;;;;;----------------;;;;::::····...... .................................. ........·····: ······::::::::::::··········..........................······:::::;;;;;---------------;;;;;::::····...... ..............·····............... ........···· ······::::::::··········............ ..........······::::;;;;;--------------;;;;;::::·····....... ............·············............ .......··· ·····:::::::··········......... ........·····:::::;;;;;;---------;;;;;;:::::·····........ ...........····················......... ......·· ·:::::::::::········........ .......·····:::::;;;;;;;;;;;;;;;;;;::::::·····........... ............···········:::::··········......... ....... ::::::::::::·······........ .......·····::::::;;;;;;;;;;;;;::::::·······.............................········:::::::::::::::::·······....... ...... :::::::::::::······....... .......······::::::::::::::::::::········...........................·······:::::::;;;;;;;;;;:::::::·····....... ..... ;;;;;;;;::::::·····...... .......·······::::::::::::::·········..........................······::::::;;;;;;;;;;;;;;;;;;;:::::·····...... ..... ---;;;;;;;;::::·····..... ........·························...........................······:::::;;;;;--------------;;;;;:::::····....... ..... --------;;;;::::····...... .........·················...............................······::::;;;;------=========-----;;;;::::·····....... ..... =====-----;;;;:::····..... .................................... ..........·····::::;;;;----===============----;;;;::::····........ ....... +=======----;;;:::····..... ............................ ........····::::;;;----=====++++++++++====----;;;::::·····......... .......... ++++++====---;;;:::····..... ..................... .......····::::;;;---====++++++++++++++====---;;;;::::·····............................. **++++++===---;;;:::···...... ................ ......····:::;;;---===+++++++***++++++====---;;;;::::······.........................·· *****++++===---;;;:::···...... ............. .....····:::;;;---====+++++*****++++++====---;;;;::::········.................······· ******++++===--;;;:::····...... ............ .....····:::;;;---===+++++++**+++++++====----;;;;:::::······························ ******++++===---;;;:::····...... ................ .....···::::;;----===++++++++++++++=====----;;;;::::::························::::: ******++++===---;;;:::·····....... ...................... .....···:::;;;----=====+++++++++======----;;;;;:::::::·················::::::::::: ****+++++===----;;;::::····.......... ............................. ....····:::;;;;----================-----;;;;;:::::::::···········::::::::::::;;;; +++++++====----;;;::::······................................................... .....····::::;;;------=========-------;;;;;;:::::::::········:::::::::::;;;;;;;;; +++=======----;;;:::::······...........................················......... .....····::::;;;;;---------------;;;;;;;:::::::::··········::::::::::;;;;;;;;;-- ======------;;;;:::::······.......................························....... .....·····:::::;;;;;;;;;-;;;;;;;;;;;::::::::················::::::::;;;;;;;----- --------;;;;;;:::::······......................···········::::::::::········....... ......·····::::::;;;;;;;;;;;;;;::::::::······················:::::::;;;;;------- ;;;;;;;;;;::::::·······......................········:::::::::::::::::::······...... ......······::::::::::::::::::::::····························::::::;;;;;------- ;;;;;:::::::········........................·······:::::::::;;;;;;;:::::::······...... .......·······::::::::::::::::···········................········:::::;;;;;------ ::::::::········............................······:::::::;;;;;;;;;;;;;;:::::·····....... .......·········:::::::::::··········........................······:::::;;;;------ ::·········............ ..........······:::::;;;;;;;;;;;;;;;;;;:::::·····....... ........··········::::::···········.............. ............······::::;;;;;---- ·······.......... ........·····:::::;;;;;;;------;;;;;;;:::::·····....... ..........··········::::::··········........... ........·····:::::;;;;;-- ···.......... .......····:::::;;;;;;---------;;;;;;:::::·····.......... ............·········::::::::::·········......... .......·····:::::;;;;; .......... ......·····::::;;;;;;----------;;;;;;:::::······...........................········::::::::::::::::·······........ .......·····::::::;; ....... ......····::::;;;;;;;-------;;;;;;;::::::·······....................·········:::::::::::::::::::::······....... .......·····:::::: ...... ......····:::::;;;;;;;;;;;;;;;;;;;:::::::·········............·········::::::::;;;;;;;;;;;;;::::::·····...... .......······::: ..... ......····::::::;;;;;;;;;;;;;;;;::::::::··························:::::::;;;;;;;;;;;;;;;;;;;;:::::····...... .......······· .... .....·····::::::;;;;;;;;;;;;:::::::::·······················:::::::;;;;;;---------------;;;;;::::····...... .........····
通过你的 Agent 使用
请安装 https://www.agentrouter.to/SKILL.md,然后通过 AgentRouter 使用 Firecrawl 的 /crawl 接口。
输入参数
url必填- 公开的 HTTP(S) URL。不支持 X/Twitter 和 LinkedIn URL,这些地址采用单独的上游费率。
limit选填- 最多爬取的页面数,上限为 10000。提交前会预留全部最高费用。
prompt选填- 描述要爬取的页面,由上游生成路径筛选条件。
maxDiscoveryDepth选填- 从起始 URL 开始的最大链接深度;0 表示只读取起始页。
includePaths选填- 需要包含的路径匹配模式。
excludePaths选填- 需要排除的路径匹配模式。
sitemap选填- 跳过 sitemap、结合 sitemap 与链接发现,或只使用 sitemap。
crawlEntireDomain选填- 爬取整个域名,而不限于起始路径。
allowSubdomains选填- 允许爬取起始域名的子域名。
delay选填- 页面请求之间等待的秒数。
scrapeOptions选填- object
输入示例
{
"url": "https://example.com",
"limit": 1,
"maxDiscoveryDepth": 0
}输出
通过带认证的结果分页获取全部已完成页面。完整输出上限为 12 MiB;超限会失败,不返回不完整的数据。