# Arbitron robots.txt # # Keep this in sync with the @page routes: `Disallow` matches a URL PREFIX, so an entry that no # longer corresponds to a real route protects nothing while looking like it does. Two did exactly # that until 2026-07-30 — `/scanner` (the page is /opportunities) and `/history` (it is # /trade-history) — leaving both app pages crawlable for months. `scripts/check-public-page-titles.sh` # derives "is this page public?" from this file, so a stale line silently mislabels pages there too. # # Rule of thumb: everything a logged-out visitor is meant to read is Allowed; everything that # redirects to /login, carries a one-time token, or is a UI shell is Disallowed. Crawling a page # that 302s to /login spends crawl budget to index nothing. User-agent: * Allow: / Allow: /learn Allow: /learn/ Allow: /free-tools Allow: /free-tools/ Allow: /fee-rebates Allow: /listings # Per-venue contract specs. Listed explicitly, like /pricing below: check-public-page-titles.sh # derives "is this page public?" from this file, and a page it cannot see is a page whose SERP # title nothing enforces. Allow: /exchanges Allow: /exchanges/ # Both /partners URLs stay Allowed even once Partner:Host moves the programme to # partner.arbitron.app and they start answering 308. A crawler blocked from fetching a URL never # sees the redirect, so the equity never transfers — Allow is what makes the move legible. The # partner host serves its own robots.txt (PartnerSiteFiles), which is where its Disallow rules live. Allow: /partners # /partners/apply is deliberately NOT disallowed: it carries , # and a crawler blocked from fetching the page never reads that noindex. Allowed + noindex is the # combination that actually keeps a form out of the index. Allow: /partners/apply # Both are crawlable and both are deliberate: /pricing is the price list, /terms is the settlement # statement that used to occupy /pricing. Listed explicitly rather than relying on `Allow: /`, # because check-public-page-titles.sh derives "is this page public?" from this file and a page it # cannot see is a page whose SERP title nothing enforces. Allow: /pricing Allow: /terms # The developer portal (ADR 0260): the public face of the API and the MCP server, and the # Markdown twin agents read. The machine endpoints themselves (/api/, /mcp, /connect/) are # below in the Disallow block: a crawler has nothing to index there and an agent with a key # never consults robots.txt for them. Allow: /developers Allow: /developers/ Allow: /llms.txt # The contract document. /api/ is otherwise closed to crawlers (below), and an agent that reads # robots.txt before fetching the OpenAPI document would have been turned away from it. Allow: /api/v1/openapi.json # ── Authenticated app ──────────────────────────────────────────────────────── Disallow: /dashboard Disallow: /admin Disallow: /admin/ Disallow: /opportunities Disallow: /signals Disallow: /strategies Disallow: /strategies/ Disallow: /trade-history # The desk. Behind the login, and the content is one account's private conversation — the one # page on this list where crawling it would be a disclosure rather than a wasted crawl budget. Disallow: /messages Disallow: /listings-calendar Disallow: /news Disallow: /news/ Disallow: /favorites Disallow: /referrals # The partner console. NOT written as `Disallow: /partner` — that is a prefix match and would take # the public /partners pitch with it, the same trap /listings-calendar sets for /listings. The `$` # anchors the bare route; the trailing slash covers everything under it. Disallow: /partner$ Disallow: /partner/ Disallow: /rebates Disallow: /settings Disallow: /profile Disallow: /server Disallow: /exchange-keys Disallow: /blacklist Disallow: /knowledge-base Disallow: /knowledge-base/ Disallow: /portfolio Disallow: /leaderboard # Copy trading: every surface is behind the login, the lead profile included. Disallow: /copy-trading Disallow: /lead/ Disallow: /billing # The in-app price sheet. /pricing (Allowed above) is the public one; this is the same ladder # under the dashboard shell, and it 302s to /login for anyone not signed in. Disallow: /plans Disallow: /share # Legacy routes, kept so anything still linking them stays out of the index. Disallow: /scanner Disallow: /history # ── Auth & one-time-token landings ─────────────────────────────────────────── # Every URL below is reached from an email or a form and is worthless in an index; several carry a # token in the query string, which must never end up in search results. Disallow: /login Disallow: /login-2fa Disallow: /logout Disallow: /register Disallow: /forgot-password Disallow: /reset-password Disallow: /2fa-reset Disallow: /verify-email Disallow: /add-email Disallow: /email-language Disallow: /unsubscribe # Telegram Mini App entry (only reachable from inside Telegram's WebView) and the step-up prompt # that guards sensitive sections for a Mini App session. Disallow: /tg Disallow: /confirm-password # ── Shells, endpoints, non-content ─────────────────────────────────────────── # /listings-calendar above is deliberately more specific than `Allow: /listings`; Google resolves # a conflict by the LONGEST matching path, so the public listings page stays crawlable. Disallow: /not-found Disallow: /Error Disallow: /culture/set Disallow: /api/ Disallow: /hubs/ Disallow: /mcp Disallow: /connect/ Disallow: /.well-known/oauth-protected-resource # ── Facet combinations of the public pages ─────────────────────────────────── # Every view below is noindex or canonicalises to its base URL, but Google has to FETCH a URL to # learn that, and each facet link multiplies the next: the scanner's size × delay × tab × page × # venue, the listings' exchange × sort × dir × time × type × page. On 2026-09-23 URL Inspection # showed Googlebot crawling /arbitrage-scanner/mexc?size=25&delay=500 the same morning while the # coin pages (/funding-rates/coin/*) were last fetched 2026-08-10 and the /funding-rates hub # 2026-08-27 — live rates served six weeks stale. Disallowing the facets costs nothing indexed. # NOT listed on purpose: ?lang= (the RU/UK/FR pages are self-canonical, hreflang-linked and # indexed) and /funding-rates/{venue}?page=N (the path by which Google reaches the coin pages). # The legacy ?exchange= on /arbitrage-scanner stays fetchable so its canonical keeps consolidating. Disallow: /arbitrage-scanner*?*size= Disallow: /arbitrage-scanner*?*delay= Disallow: /arbitrage-scanner*?*tab= Disallow: /arbitrage-scanner*?*page= Disallow: /listings*?*exchange= Disallow: /listings*?*sort= Disallow: /listings*?*dir= Disallow: /listings*?*time= Disallow: /listings*?*type= Disallow: /listings*?*page= Sitemap: https://arbitron.app/sitemap.xml # Machine-readable index for answer engines, including which page owns which claim and how to # cite a rate without dropping its timestamp or its settlement interval. Not part of the robots # spec — some crawlers only discover it from here. # https://arbitron.app/llms.txt