[{"data":1,"prerenderedAt":46},["ShallowReactive",2],{"$f267o7ueezjn4b":3},{"href":4,"title":5,"description":6,"kind":7,"mark":8,"planned":9,"contributors":10,"provenance":8,"express":9,"bridge":9,"html":11,"headings":12},"\u002Fdocs\u002Ftools\u002Fweb","Web","The public web, read directly: a page as text, a sitemap as a list, a URL's text or JSON. Nothing to connect.","operates",null,false,[],"\u003Ch2 id=\"what-a-member-can-do\">What a member can do\u003C\u002Fh2>\n\u003Cp>A member can read the public web with three built-in tools. A task names the ones it may use:\u003C\u002Fp>\n\u003Cdiv class=\"prose__table\">\n\u003Ctable>\n\u003Cthead>\n\u003Ctr>\n\u003Cth>Tool\u003C\u002Fth>\n\u003Cth>What it does\u003C\u002Fth>\n\u003C\u002Ftr>\n\u003C\u002Fthead>\n\u003Ctbody>\n\u003Ctr>\n\u003Ctd>\u003Ccode>web.read\u003C\u002Fcode>\u003C\u002Ftd>\n\u003Ctd>Reads one page and returns the words a reader would see, its title, and a hash of exactly those words.\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>\u003Ccode>web.sitemap\u003C\u002Fcode>\u003C\u002Ftd>\n\u003Ctd>Reads a sitemap and returns its addresses as a list, each with the date the site says it last changed.\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>\u003Ccode>web.request\u003C\u002Fcode>\u003C\u002Ftd>\n\u003Ctd>Sends a GET to an address and returns the text or JSON that comes back, with its status.\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003C\u002Ftbody>\n\u003C\u002Ftable>\n\u003C\u002Fdiv>\n\u003Cp>What each takes and returns is in the \u003Ca href=\"\u002Fdocs\u002Ftools\u002Fbuilt-in\u002Fweb\">Web tools reference\u003C\u002Fa>.\u003C\u002Fp>\n\u003Cp>These are plain requests, not a browser. No JavaScript runs and no cookies are sent. A credential is sent only to a\nhost your enterprise attached one to (see below). For a product a member signs in to and clicks through, bind a\n\u003Ca href=\"\u002Fdocs\u002Ftools\u002Fbrowser\">browser\u003C\u002Fa>. For a system with an MCP server, connect it from the \u003Ca href=\"\u002Fdocs\u002Ftools\">Tools\u003C\u002Fa> directory.\u003C\u002Fp>\n\u003Ch3 id=\"a-key-for-an-api\">A key for an API\u003C\u002Fh3>\n\u003Cp>Most APIs worth calling need a key. A person stores it once on \u003Cstrong>Tools\u003C\u002Fstrong>, as a \u003Cstrong>Web credential\u003C\u002Fstrong>: the hosts it is\nfor, the header it goes in, whether it is sent as \u003Ccode>Bearer &lt;key&gt;\u003C\u002Fcode> or as the key alone, and the key itself. It can be\nattached at any layer \u003Cstrong>Tools\u003C\u002Fstrong> offers, with the same rule about who may set it: an enterprise-wide one is the CEO's.\u003C\u002Fp>\n\u003Cp>A task under that layer then calls those hosts as it would any other, and the OS adds the key to each request.\u003C\u002Fp>\n\u003Cul>\n\u003Cli>Only the exact hosts named get it: no subdomain, no parent domain, no wildcard. A request anywhere else carries no\nkey, and neither does a redirect from a named host to another one.\u003C\u002Fli>\n\u003Cli>It is sent over \u003Ccode>https\u003C\u002Fcode> only. A named host reached over plain \u003Ccode>http\u003C\u002Fcode> is refused (\u003Ccode>web_credential_insecure\u003C\u002Fcode>) and\nnever asked.\u003C\u002Fli>\n\u003Cli>The member never reads it, and cannot pass one itself: the headers it may set are still \u003Ccode>accept\u003C\u002Fcode> and\n\u003Ccode>accept-language\u003C\u002Fcode>. If the answer repeats the key, in its body, a header or an address, the key is replaced with\n\u003Ccode>[credential removed]\u003C\u002Fcode> before the member reads it, and the run's record and the logs hold the same.\u003C\u002Fli>\n\u003Cli>A task outside the layer gets nothing, and nothing tells it a key exists.\u003C\u002Fli>\n\u003Cli>Editing it to add a host, or to change the header or how it is sent, needs the key typed again. Removing a host\ndoes not.\u003C\u002Fli>\n\u003Cli>Remove the credential on \u003Cstrong>Tools\u003C\u002Fstrong> and the next request goes without it.\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Cp>The key is a stored value only: no OAuth sign-in, and no automatic rotation.\u003C\u002Fp>\n\u003Ch3 id=\"a-hash-that-changes-when-the-words-do\">A hash that changes when the words do\u003C\u002Fh3>\n\u003Cp>\u003Ccode>web.read\u003C\u002Fcode> returns a \u003Ccode>sha256\u003C\u002Fcode> of the page's visible text. Scripts, styles and tracking snippets are not part of it, so\na site that is redeployed with nothing reworded returns the same hash, and a page with one sentence changed returns a\nnew one. A task can keep the hash to tell whether a page needs reading again, and can pass it to \u003Ccode>memory.train\u003C\u002Fcode> as the\nrevision of what it learned.\u003C\u002Fp>\n\u003Cp>The hash follows the text, not the meaning. A page that prints the time or rotates a banner changes on every read.\u003C\u002Fp>\n\u003Ch3 id=\"public-addresses-only\">Public addresses only\u003C\u002Fh3>\n\u003Cp>A request reaches public addresses, and nothing else. A private, local or internal address is refused\n(\u003Ccode>web_address_refused\u003C\u002Fcode>), including a public name that points at one. Every redirect is checked again before it is\nfollowed, and a redirect to such an address is refused without being requested (\u003Ccode>web_redirect_refused\u003C\u002Fcode>).\u003C\u002Fp>\n\u003Ch3 id=\"whole-or-not-at-all\">Whole, or not at all\u003C\u002Fh3>\n\u003Cp>A page or a response too long for one result is refused, with its size and its hash and none of its content\n(\u003Ccode>web_page_too_large\u003C\u002Fcode>, \u003Ccode>web_body_too_large\u003C\u002Fcode>). A sitemap too long for one result returns how many addresses it has,\n\u003Ccode>complete: false\u003C\u002Fcode>, and no list. A member never receives part of a page as though it were the page.\u003C\u002Fp>\n\u003Cp>A page that answers with an error is refused by \u003Ccode>web.read\u003C\u002Fcode> and \u003Ccode>web.sitemap\u003C\u002Fcode> (\u003Ccode>web_status_not_ok\u003C\u002Fcode>), so an error page\nis not read as content. \u003Ccode>web.request\u003C\u002Fcode> returns the status for the task to judge.\u003C\u002Fp>\n\u003Cdiv class=\"prose__planned\">\n\u003Cp class=\"prose__flag\">Planned\u003C\u002Fp>\n\u003Ch3 id=\"reading-in-parts-and-writing\">Reading in parts, and writing\u003C\u002Fh3>\n\u003Cp>A long page or sitemap will be readable in parts, each carrying the hash of the whole. \u003Ccode>web.request\u003C\u002Fcode> will send\n\u003Ccode>POST\u003C\u002Fcode>, \u003Ccode>PUT\u003C\u002Fcode>, \u003Ccode>PATCH\u003C\u002Fcode> and \u003Ccode>DELETE\u003C\u002Fcode>, waiting for your approval unless the task is kept to reading.\u003C\u002Fp>\n\u003C\u002Fdiv>\n\u003Ch2 id=\"who-connects-it\">Who connects it\u003C\u002Fh2>\n\u003Cp>Nobody, for open addresses. The tools are built in, the same for every enterprise, and a task has them when it names\nthem. For an API that needs a key, a person stores it on \u003Cstrong>Tools\u003C\u002Fstrong> as a web credential, as above.\u003C\u002Fp>\n\u003Ch2 id=\"what-waits-for-you\">What waits for you\u003C\u002Fh2>\n\u003Cp>Nothing a \u003Ccode>web.*\u003C\u002Fcode> tool does waits for you unless the task declares a gate on it. Gate \u003Ccode>web.request\u003C\u002Fcode> and every request\nwaits, with the address it was about to ask for, until you approve it on \u003Cstrong>Gates\u003C\u002Fstrong>. See\n\u003Ca href=\"\u002Fdocs\u002Fcontrol\u002Fgates\">Gates and tool scopes\u003C\u002Fa>.\u003C\u002Fp>\n\u003Ch2 id=\"setup\">Setup\u003C\u002Fh2>\n\u003Col>\n\u003Cli>On the task, name each tool it needs: \u003Ccode>web.read\u003C\u002Fcode>, \u003Ccode>web.sitemap\u003C\u002Fcode>, \u003Ccode>web.request\u003C\u002Fcode>.\u003C\u002Fli>\n\u003Cli>In the task's skill, say which addresses it should read, and what to do with a refusal: skip a page that is too\nlong, and never treat a refusal as the page.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Check it.\u003C\u002Fstrong> Run the task. The run page shows every request, where its redirects ended, and what came back.\u003C\u002Fli>\n\u003C\u002Fol>\n\u003Ch2 id=\"troubleshooting\">Troubleshooting\u003C\u002Fh2>\n\u003Cdiv class=\"prose__table\">\n\u003Ctable>\n\u003Cthead>\n\u003Ctr>\n\u003Cth>What you see\u003C\u002Fth>\n\u003Cth>Why\u003C\u002Fth>\n\u003Cth>What to do\u003C\u002Fth>\n\u003C\u002Ftr>\n\u003C\u002Fthead>\n\u003Ctbody>\n\u003Ctr>\n\u003Ctd>\u003Ccode>web_address_refused\u003C\u002Fcode>\u003C\u002Ftd>\n\u003Ctd>The address is private, local or internal, or its name points at one\u003C\u002Ftd>\n\u003Ctd>Use the public address. A system inside your network is not reachable this way\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>\u003Ccode>web_redirect_refused\u003C\u002Fcode>\u003C\u002Ftd>\n\u003Ctd>A redirect loops, or leads somewhere the tools do not go\u003C\u002Ftd>\n\u003Ctd>Read the address the site redirects to, if it is public\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>\u003Ccode>web_url_invalid\u003C\u002Fcode>\u003C\u002Ftd>\n\u003Ctd>The address is not \u003Ccode>http\u003C\u002Fcode> or \u003Ccode>https\u003C\u002Fcode>, names a port, or carries a username\u003C\u002Ftd>\n\u003Ctd>Pass a plain \u003Ccode>https:\u002F\u002F\u003C\u002Fcode> address\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>\u003Ccode>web_status_not_ok\u003C\u002Fcode>\u003C\u002Ftd>\n\u003Ctd>The page answered with an error status\u003C\u002Ftd>\n\u003Ctd>Check the address. \u003Ccode>web.request\u003C\u002Fcode> shows the status and body\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>\u003Ccode>web_not_html\u003C\u002Fcode>\u003C\u002Ftd>\n\u003Ctd>\u003Ccode>web.read\u003C\u002Fcode> was given something that is not a page\u003C\u002Ftd>\n\u003Ctd>Use \u003Ccode>web.sitemap\u003C\u002Fcode> for a sitemap, \u003Ccode>web.request\u003C\u002Fcode> for JSON or text\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>\u003Ccode>web_page_too_large\u003C\u002Fcode>, \u003Ccode>web_body_too_large\u003C\u002Fcode>\u003C\u002Ftd>\n\u003Ctd>The content does not fit in one result\u003C\u002Ftd>\n\u003Ctd>Have the task skip it. Nothing was returned, so there is nothing to summarise\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>A sitemap returns \u003Ccode>complete: false\u003C\u002Fcode>\u003C\u002Ftd>\n\u003Ctd>The list does not fit in one result\u003C\u002Ftd>\n\u003Ctd>Do not treat missing addresses as removed pages\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>\u003Ccode>web_header_refused\u003C\u002Fcode>\u003C\u002Ftd>\n\u003Ctd>The task set a header other than \u003Ccode>accept\u003C\u002Fcode> or \u003Ccode>accept-language\u003C\u002Fcode>\u003C\u002Ftd>\n\u003Ctd>Remove it. Credentials are never passed by a member: attach one on \u003Cstrong>Tools\u003C\u002Fstrong>\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>\u003Ccode>web_credential_insecure\u003C\u002Fcode>\u003C\u002Ftd>\n\u003Ctd>A host with a web credential was asked for, or redirected to, over plain \u003Ccode>http\u003C\u002Fcode>\u003C\u002Ftd>\n\u003Ctd>Use the \u003Ccode>https\u003C\u002Fcode> address. The key is never sent in clear\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>\u003Ccode>web_credential_unavailable\u003C\u002Fcode>\u003C\u002Ftd>\n\u003Ctd>A web credential on the task's layers could not be read, for example after the secrets key changed. Nothing was requested\u003C\u002Ftd>\n\u003Ctd>Type the key again on \u003Cstrong>Tools\u003C\u002Fstrong> to store it afresh\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>\u003Ccode>[credential removed]\u003C\u002Fcode> in an answer\u003C\u002Ftd>\n\u003Ctd>The service repeated the key back, and it was masked\u003C\u002Ftd>\n\u003Ctd>Nothing: the rest of the answer is as it came\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>A keyed API answers 401\u003C\u002Ftd>\n\u003Ctd>The host called is not one the credential names exactly, or the task is outside its layer\u003C\u002Ftd>\n\u003Ctd>Check the hosts and the layer on \u003Cstrong>Tools\u003C\u002Fstrong>\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>A page reads as nearly empty\u003C\u002Ftd>\n\u003Ctd>The site builds its content with JavaScript\u003C\u002Ftd>\n\u003Ctd>Bind a \u003Ca href=\"\u002Fdocs\u002Ftools\u002Fbrowser\">browser\u003C\u002Fa> for that site\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>The hash changes on every read\u003C\u002Ftd>\n\u003Ctd>The page prints something that changes: a time, a counter, a rotating banner\u003C\u002Ftd>\n\u003Ctd>Compare the text, or read a steadier address\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003C\u002Ftbody>\n\u003C\u002Ftable>\n\u003C\u002Fdiv>\n",[13,17,21,24,27,30,34,37,40,43],{"id":14,"text":15,"level":16,"planned":9},"what-a-member-can-do","What a member can do",2,{"id":18,"text":19,"level":20,"planned":9},"a-key-for-an-api","A key for an API",3,{"id":22,"text":23,"level":20,"planned":9},"a-hash-that-changes-when-the-words-do","A hash that changes when the words do",{"id":25,"text":26,"level":20,"planned":9},"public-addresses-only","Public addresses only",{"id":28,"text":29,"level":20,"planned":9},"whole-or-not-at-all","Whole, or not at all",{"id":31,"text":32,"level":20,"planned":33},"reading-in-parts-and-writing","Reading in parts, and writing",true,{"id":35,"text":36,"level":16,"planned":9},"who-connects-it","Who connects it",{"id":38,"text":39,"level":16,"planned":9},"what-waits-for-you","What waits for you",{"id":41,"text":42,"level":16,"planned":9},"setup","Setup",{"id":44,"text":45,"level":16,"planned":9},"troubleshooting","Troubleshooting",1794928143782]