Web Scrape

/web.scrape

Scrape any web page and get its content back as clean Markdown (or HTML, or raw HTML) plus title and metadata.

Finished calls, last 7 days: 2
Success rate: 100.0%
Median execution time: 858ms

Cancellations excluded. Execution times cover successful calls.

;;;;----=====++++++++=====----;;;::::····........                   .............···········............                             ......····:::::;;;;--------------;;;;;:::::······..................
;;;----===++++++++++++++====---;;;;:::·····.........             .................·····..............                                 ......····::::;;;;;--------------;;;;;:::::·······................
;;;---===++++++*****++++++===----;;;::::·····...................................·····.............                                     ......····::::;;;;;-------------;;;;;;::::::········.............
;;;---===++++*********+++++===---;;;;::::·····.............................···········...........                                       ......····::::;;;;;------------;;;;;;:::::::····················
;;;---===++++**********++++====---;;;;::::·······....................··················.........                                         ......····::::;;;;;;;------;;;;;;;;;:::::::····················
:;;;---===+++++*******+++++====----;;;:::::··········........···························........                                          ......····:::::;;;;;;;;;;;;;;;;;;:::::::::·················:::
::;;---====++++++++++++++++====---;;;;;:::::·····························::::::··········.......                                           ......·····::::::;;;;;;;;;;;;::::::::::················::::::
::;;;----====++++++++++++=====----;;;;:::::::····················:::::::::::::::::::······......                                            ......······:::::::::::::::::::::::··················:::::::
:::;;;;----=================----;;;;;;:::::::···············::::::::::::::::::::::::::·····......                                            ......·······::::::::::::::::·······················::::::;
·::::;;;-----===========------;;;;;;::::::::·············::::::::::;;;;;;;;;;;;;;;::::::·····.....                                           .......·············································::::::;
··::::;;;;;----------------;;;;;;::::::::···············:::::::::;;;;;;;;;;;;;;;;;;;;::::·····.....                                           ........························...........·········:::::;
···:::::;;;;;;;-----;;;;;;;;;::::::::···················:::::::;;;;;;;;---------;;;;;;::::·····.....                                          .........···············.......................······::::;
····::::::;;;;;;;;;;;;;;::::::::························:::::::;;;;;;--------------;;;;;::::····.....                                        .............·····................................·····::::
·····:::::::::::::::::::::::··········..........·········::::::;;;;;;---------------;;;;;:::·····.....                                     .....................................................·····:::
······:::::::::::::::::·········....................·······:::::;;;;;----------------;;;;::::····......                                  ..................................               ........·····:
······::::::::::::··········..........................······:::::;;;;;---------------;;;;;::::····......                              ..............·····...............                    ........····
······::::::::··········............          ..........······::::;;;;;--------------;;;;;::::·····.......                        ............·············............                       .......···
·····:::::::··········.........                   ........·····:::::;;;;;;---------;;;;;;:::::·····........                   ...........····················.........                          ......··
·:::::::::::········........                         .......·····:::::;;;;;;;;;;;;;;;;;;::::::·····...........          ............···········:::::··········.........                          .......
::::::::::::·······........                            .......·····::::::;;;;;;;;;;;;;::::::·······.............................········:::::::::::::::::·······.......                           ......
:::::::::::::······.......                               .......······::::::::::::::::::::········...........................·······:::::::;;;;;;;;;;:::::::·····.......                           .....
;;;;;;;;::::::·····......                                  .......·······::::::::::::::·········..........................······::::::;;;;;;;;;;;;;;;;;;;:::::·····......                          .....
---;;;;;;;;::::·····.....                                   ........·························...........................······:::::;;;;;--------------;;;;;:::::····.......                        .....
--------;;;;::::····......                                    .........·················...............................······::::;;;;------=========-----;;;;::::·····.......                      .....
=====-----;;;;:::····.....                                     ....................................          ..........·····::::;;;;----===============----;;;;::::····........                  .......
+=======----;;;:::····.....                                      ............................                  ........····::::;;;----=====++++++++++====----;;;::::·····.........            ..........
++++++====---;;;:::····.....                                      .....................                         .......····::::;;;---====++++++++++++++====---;;;;::::·····.............................
**++++++===---;;;:::···......                                     ................                                ......····:::;;;---===+++++++***++++++====---;;;;::::······.........................··
*****++++===---;;;:::···......                                    .............                                    .....····:::;;;---====+++++*****++++++====---;;;;::::········.................·······
******++++===--;;;:::····......                                 ............                                        .....····:::;;;---===+++++++**+++++++====----;;;;:::::······························
******++++===---;;;:::····......                            ................                                         .....···::::;;----===++++++++++++++=====----;;;;::::::························:::::
******++++===---;;;:::·····.......                     ......................                                         .....···:::;;;----=====+++++++++======----;;;;;:::::::·················:::::::::::
****+++++===----;;;::::····..........            .............................                                         ....····:::;;;;----================-----;;;;;:::::::::···········::::::::::::;;;;
+++++++====----;;;::::······...................................................                                        .....····::::;;;------=========-------;;;;;;:::::::::········:::::::::::;;;;;;;;;
+++=======----;;;:::::······...........................················.........                                        .....····::::;;;;;---------------;;;;;;;:::::::::··········::::::::::;;;;;;;;;--
======------;;;;:::::······.......................························.......                                       .....·····:::::;;;;;;;;;-;;;;;;;;;;;::::::::················::::::::;;;;;;;-----
--------;;;;;;:::::······......................···········::::::::::········.......                                     ......·····::::::;;;;;;;;;;;;;;::::::::······················:::::::;;;;;-------
;;;;;;;;;;::::::·······......................········:::::::::::::::::::······......                                    ......······::::::::::::::::::::::····························::::::;;;;;-------
;;;;;:::::::········........................·······:::::::::;;;;;;;:::::::······......                                 .......·······::::::::::::::::···········................········:::::;;;;;------
::::::::········............................······:::::::;;;;;;;;;;;;;;:::::·····.......                              .......·········:::::::::::··········........................······:::::;;;;------
::·········............           ..........······:::::;;;;;;;;;;;;;;;;;;:::::·····.......                          ........··········::::::···········..............    ............······::::;;;;;----
·······..........                    ........·····:::::;;;;;;;------;;;;;;;:::::·····.......                    ..........··········::::::··········...........                ........·····:::::;;;;;--
···..........                          .......····:::::;;;;;;---------;;;;;;:::::·····..........           ............·········::::::::::·········.........                      .......·····:::::;;;;;
..........                              ......·····::::;;;;;;----------;;;;;;:::::······...........................········::::::::::::::::·······........                          .......·····::::::;;
.......                                   ......····::::;;;;;;;-------;;;;;;;::::::·······....................·········:::::::::::::::::::::······.......                             .......·····::::::
......                                     ......····:::::;;;;;;;;;;;;;;;;;;;:::::::·········............·········::::::::;;;;;;;;;;;;;::::::·····......                                .......······:::
.....                                       ......····::::::;;;;;;;;;;;;;;;;::::::::··························:::::::;;;;;;;;;;;;;;;;;;;;:::::····......                                  .......·······
....                                          .....·····::::::;;;;;;;;;;;;:::::::::·······················:::::::;;;;;;---------------;;;;;::::····......                                  .........····

Use it with your agent

Set up https://www.agentrouter.to/SKILL.md, then use AnyAPI’s /web.scrape endpoint through AgentRouter.

Inputs

urlRequired: The full web address of the page to scrape, including its domain, for example https://example.com. The https:// prefix is optional.
mobileOptional: When true, render the page with a mobile viewport and user agent instead of desktop. Some sites serve materially different content to mobile.
formatsOptional: Which representations of the page to return. Any combination of: markdown (page content as Markdown), html (the page HTML exactly as the browser received it, including head and script tags). Each requested format is returned under the matching output field. Defaults to both. rawHtml is a deprecated alias of html, returned under a rawHtml field for callers that predate the rename; send html instead.
waitForOptional: Milliseconds to wait for the page to finish rendering before capture. Use this for JavaScript-heavy pages or single-page apps whose content loads after the initial paint. Capped at 15000 to stay within the request timeout. This wait is time you asked us to spend, so your response takes this much longer, and it is excluded from the latency published for this endpoint. If every provider that can wait fails, the page may still be returned, captured without the wait.
blockAdsOptional: When true (upstream default), strip ad and cookie-consent elements before capture. Set false to keep them.
excludeTagsOptional: CSS selectors to drop before capture (for example ["nav", "footer", ".ads"]). Applied after includeTags.
includeTagsOptional: CSS selectors to keep. When set, only content matching these selectors is captured (for example ["article", "main"] or ["#content"]).
requireFieldsOptional: Optional; omit it and routing is unchanged, with the cheapest source serving. Name the output fields this request must be able to return, for example `statusCode`, and it is served only by a source that returns every one of them. Fields you do not name are still returned whenever the serving source has them. This can raise your price: when the cheapest source cannot return a named field, a dearer source serves, and you are quoted and charged its price. A named field can still be absent on a record that genuinely lacks it. Naming a combination that no single source returns together is refused as invalid input, with no charge. A source that reports a field only in some cases, such as only when the website answered with an error, does not count as returning it.
onlyMainContentOptional: When true, return only the main article content, stripping navigation, headers, footers, and other boilerplate. Defaults to false to capture the full page.
preferLatencyUnderMsOptional: Optional; omit it and routing is unchanged, with the cheapest source serving. Prefer sources whose typical response time (median over the trailing 30 days, as published on this endpoint's lane health) is under this many milliseconds; among those, the cheapest serves. This can raise your price: when the cheapest source misses the target, a faster and dearer one serves, and you are quoted and charged its price. If no source is that fast the request is still served, by whichever source offers the best speed for its price - it is never refused for being slow. Sources we have not timed are tried last. This is a preference, not a guarantee: the median describes past requests and is not a ceiling on this one, and it excludes any wait this request itself asks for. On a paginated walk it applies to the first page only: later pages stay with the source that page chose, at the price it was quoted.
sourceOptional: Optional. Source ids to prefer, in order, taken from this endpoint's `lanes[].source.id` in /catalog or /apis. Omit it and the cheapest source serves, with automatic failover. Listed sources are tried first in the order given, then the others, unless `allowFallbacks` is false. A single source with `allowFallbacks` false is served only by that source at its price, quoted and charged exactly, with no failover. The price is that of the dearest source that may serve. An id that does not serve this endpoint is rejected as invalid input with no charge; a listed source that is not serving right now is refused with no charge, so omit `source` to be served by another. On a paginated walk, later pages must include the source that served page one, or omit `source`.
allowFallbacksOptional: Optional, default true. When false, only the sources listed in `source` may serve; the request is refused with no charge if none of them can. When true, the listed sources are tried first and any other source may serve after them, at the normal price.
ignoreSourcesOptional: Optional. Source ids to skip for this request, taken from this endpoint's `lanes[].source.id`. The cheapest remaining source serves and the price is that of the dearest remaining source. An id that does not serve this endpoint, or that is also in `source`, is rejected as invalid input with no charge; skipping every source is rejected the same way.

Example input

{
  "url": "https://example.com",
  "formats": [
    "markdown",
    "html"
  ],
  "onlyMainContent": false
}

Output

AnyAPI output: found (boolean), data (the record) and reason when nothing was found.