Web
The public web, read directly: a page as text, a sitemap as a list, a URL's text or JSON. Nothing to connect.
What a member can do
A member can read the public web with three built-in tools. A task names the ones it may use:
| Tool | What it does |
|---|---|
web.read |
Reads one page and returns the words a reader would see, its title, and a hash of exactly those words. |
web.sitemap |
Reads a sitemap and returns its addresses as a list, each with the date the site says it last changed. |
web.request |
Sends a GET to an address and returns the text or JSON that comes back, with its status. |
What each takes and returns is in the Web tools reference.
These are plain requests, not a browser. No JavaScript runs and no cookies are sent. A credential is sent only to a host your enterprise attached one to (see below). For a product a member signs in to and clicks through, bind a browser. For a system with an MCP server, connect it from the Tools directory.
A key for an API
Most APIs worth calling need a key. A person stores it once on Tools, as a Web credential: the hosts it is
for, the header it goes in, whether it is sent as Bearer <key> or as the key alone, and the key itself. It can be
attached at any layer Tools offers, with the same rule about who may set it: an enterprise-wide one is the CEO's.
A task under that layer then calls those hosts as it would any other, and the OS adds the key to each request.
- Only the exact hosts named get it: no subdomain, no parent domain, no wildcard. A request anywhere else carries no key, and neither does a redirect from a named host to another one.
- It is sent over
httpsonly. A named host reached over plainhttpis refused (web_credential_insecure) and never asked. - The member never reads it, and cannot pass one itself: the headers it may set are still
acceptandaccept-language. If the answer repeats the key, in its body, a header or an address, the key is replaced with[credential removed]before the member reads it, and the run's record and the logs hold the same. - A task outside the layer gets nothing, and nothing tells it a key exists.
- Editing it to add a host, or to change the header or how it is sent, needs the key typed again. Removing a host does not.
- Remove the credential on Tools and the next request goes without it.
The key is a stored value only: no OAuth sign-in, and no automatic rotation.
A hash that changes when the words do
web.read returns a sha256 of the page's visible text. Scripts, styles and tracking snippets are not part of it, so
a site that is redeployed with nothing reworded returns the same hash, and a page with one sentence changed returns a
new one. A task can keep the hash to tell whether a page needs reading again, and can pass it to memory.train as the
revision of what it learned.
The hash follows the text, not the meaning. A page that prints the time or rotates a banner changes on every read.
Public addresses only
A request reaches public addresses, and nothing else. A private, local or internal address is refused
(web_address_refused), including a public name that points at one. Every redirect is checked again before it is
followed, and a redirect to such an address is refused without being requested (web_redirect_refused).
Whole, or not at all
A page or a response too long for one result is refused, with its size and its hash and none of its content
(web_page_too_large, web_body_too_large). A sitemap too long for one result returns how many addresses it has,
complete: false, and no list. A member never receives part of a page as though it were the page.
A page that answers with an error is refused by web.read and web.sitemap (web_status_not_ok), so an error page
is not read as content. web.request returns the status for the task to judge.
Planned
Reading in parts, and writing
A long page or sitemap will be readable in parts, each carrying the hash of the whole. web.request will send
POST, PUT, PATCH and DELETE, waiting for your approval unless the task is kept to reading.
Who connects it
Nobody, for open addresses. The tools are built in, the same for every enterprise, and a task has them when it names them. For an API that needs a key, a person stores it on Tools as a web credential, as above.
What waits for you
Nothing a web.* tool does waits for you unless the task declares a gate on it. Gate web.request and every request
waits, with the address it was about to ask for, until you approve it on Gates. See
Gates and tool scopes.
Setup
- On the task, name each tool it needs:
web.read,web.sitemap,web.request. - In the task's skill, say which addresses it should read, and what to do with a refusal: skip a page that is too long, and never treat a refusal as the page.
- Check it. Run the task. The run page shows every request, where its redirects ended, and what came back.
Troubleshooting
| What you see | Why | What to do |
|---|---|---|
web_address_refused |
The address is private, local or internal, or its name points at one | Use the public address. A system inside your network is not reachable this way |
web_redirect_refused |
A redirect loops, or leads somewhere the tools do not go | Read the address the site redirects to, if it is public |
web_url_invalid |
The address is not http or https, names a port, or carries a username |
Pass a plain https:// address |
web_status_not_ok |
The page answered with an error status | Check the address. web.request shows the status and body |
web_not_html |
web.read was given something that is not a page |
Use web.sitemap for a sitemap, web.request for JSON or text |
web_page_too_large, web_body_too_large |
The content does not fit in one result | Have the task skip it. Nothing was returned, so there is nothing to summarise |
A sitemap returns complete: false |
The list does not fit in one result | Do not treat missing addresses as removed pages |
web_header_refused |
The task set a header other than accept or accept-language |
Remove it. Credentials are never passed by a member: attach one on Tools |
web_credential_insecure |
A host with a web credential was asked for, or redirected to, over plain http |
Use the https address. The key is never sent in clear |
web_credential_unavailable |
A web credential on the task's layers could not be read, for example after the secrets key changed. Nothing was requested | Type the key again on Tools to store it afresh |
[credential removed] in an answer |
The service repeated the key back, and it was masked | Nothing: the rest of the answer is as it came |
| A keyed API answers 401 | The host called is not one the credential names exactly, or the task is outside its layer | Check the hosts and the layer on Tools |
| A page reads as nearly empty | The site builds its content with JavaScript | Bind a browser for that site |
| The hash changes on every read | The page prints something that changes: a time, a counter, a rotating banner | Compare the text, or read a steadier address |