# Onto-Reader · About our fetcher
> Onto-Reader/1.0 fetches a page when someone asks Onto for it. Every request it makes, who can trigger it, and the robots.txt that actually stops it.

**Source:** /about-bot
**Extracted:** 2026-09-26T11:50:53.874Z

---
Onto-Reader

## An on-demand fetcher, _not a crawler._

You're here because our user-agent points here. Onto-Reader fetches a page only when someone asks Onto for it, and it reads your robots.txt first. Everything it sends, and what actually stops it, is below.

The string in your logs

`Onto-Reader/1.0 (+https://buildonto.dev/about-bot)`From Vercel · api.buildonto.dev and md.buildonto.dev · [founder@buildonto.dev](mailto:founder@buildonto.dev)

4

Requests at most, per read

15 s

Timeout on the page

1 hr

Cache before it asks again

Every request

### What one read sends you.

In order. A second read of the same URL within the hour is answered from the cache and sends nothing, unless the caller asks for a fresh copy.

1.  01`GET /robots.txt`5 s
    
    First, every time. If your rules for Onto-Reader disallow the page's path, it stops here and never asks for the page.
    
2.  02`GET the page`15 s
    
    The one it came for. Follows up to 5 redirects, each checked again. A 401 or 403 goes back to the caller as WAF\_BLOCKED.
    
3.  03`GET the page · Accept: text/markdown`8 s
    
    Asks whether your site already serves Markdown through the Onto SDK. Affects the score only.
    
4.  04`GET /.onto/index.md`5 s
    
    The same check, by the SDK's file path.
    

It never

*   Runs JavaScript
*   Keeps cookies or submits forms
*   Retries after you refuse it
*   Follows links across your site on its own
*   Keeps the page past an hour's cache

Who sends it

### Someone asked for your page.

There are three ways a request from Onto-Reader starts.

**API and MCP customers**

With a key, 1 credit a URL. The URL goes in that customer's usage history.

**Anyone, at md.buildonto.dev**

No key needed, 10 reads per IP address, ever.

**A site map, when a customer asks**

/v1/map checks robots.txt for the page it starts from, then reads your /sitemap.xml, or that page's links, to list URLs. /v1/batch then reads up to 50 of them, each with its own robots.txt check.

Not Onto-Reader: the free scanner on buildonto.dev fetches your page as a browser and asks for Markdown as GPTBot, the way an AI crawler would. It reads robots.txt first too; when that blocks GPTBot, it sends nothing as GPTBot but still loads the page once as a browser, to check for a firewall.

Control it

### What stops it.

It reads robots.txt on every request, so a change works on the next one. If you've verified a site in your own Onto account, your reads of it skip its robots.txt; nobody else's do.

_Stops it_

#### A group named Onto-Reader

Onto-Reader reads the group that names it before any other, so this stops every fetch and touches no other crawler. The full string from your logs works too; only the name before the slash counts.

User-agent: Onto-Reader
Disallow: /

_Stops it_

#### Block GPTBot, or everyone

With no Onto-Reader group, it follows GPTBot's, then \*. Either one with Disallow: / stops every fetch. The GPTBot rule stops OpenAI's crawler too.

User-agent: GPTBot
Disallow: /

_Stops it there_

#### Path rules

Matched against the path and query it was asked for. The longest matching rule wins and Allow wins a tie, as in RFC 9309; \* and a closing $ work.

User-agent: Onto-Reader
Disallow: /admin/
Allow: /admin/help/

If it did something this page doesn't describe, that's a bug and we want to hear about it: [founder@buildonto.dev](mailto:founder@buildonto.dev).

While you're here

### See what an agent gets from your site.

The same cleaning Onto-Reader does, run on your URL, with a score for how readable it came back.

[Scan your site](/scanner) [How it works](/how-it-works)

---
## Structured Data (JSON-LD)
```json
{
  "@context": "https://schema.org",
  "@graph": [
    {
      "@type": "Organization",
      "@id": "https://buildonto.dev/#org",
      "name": "Onto",
      "url": "https://buildonto.dev",
      "logo": "https://buildonto.dev/icon.png",
      "sameAs": [
        "https://x.com/buildonto",
        "https://github.com/ravixalgorithm/ontosdk"
      ]
    },
    {
      "@type": "WebSite",
      "@id": "https://buildonto.dev/#site",
      "url": "https://buildonto.dev",
      "name": "Onto",
      "publisher": {
        "@id": "https://buildonto.dev/#org"
      }
    },
    {
      "@type": "SoftwareApplication",
      "name": "Onto",
      "applicationCategory": "DeveloperApplication",
      "operatingSystem": "Web",
      "url": "https://buildonto.dev",
      "description": "Onto serves AI agents clean Markdown from your own site, at the same URL, and shows you which agents came. Plus the AIO score, a Read API and an MCP server.",
      "publisher": {
        "@id": "https://buildonto.dev/#org"
      },
      "offers": {
        "@type": "Offer",
        "price": "0",
        "priceCurrency": "USD",
        "description": "Free tier: 1,000 credits / month"
      }
    }
  ]
}
```