Working with large data from an agent

Summaries first, handles to drill into, depth, fields, cursors, token budgets and the aggregate tools.

The registry holds a growing number of organizations, each mapped down to single values with their sources. An agent should never pull all of that into its context. Every read answers with a summary first and a handle for each thing it summarizes; the agent goes one level deeper only where it needs to. Questions about many organizations (how many, which are top, what differs, what is missing) are answered on the server.

The shape of every response

Reads return one compact JSON object: the node you asked for, a page of what sits under it, and four paging fields. Keys are short and stable; empty fields are left out; a source cited many times is sent once in src and cited by its index in s.

tokens: about how many tokens the response costs (characters / 4).
truncated and next_cursor: present when more is left. Pass the cursor back unchanged; it carries the node, depth, fields and view of the request it continues.
hint: one line saying how to go deeper, e.g. which node to open next.

Keys you will see: h handle, l label, n counts, cov covered/total leaf primitives, conf confidence, vis tier (only when not public), p parameter, v value, u unit, s source index, q quote, d description, items children, more children not shown.

depth and node: walk the map

A company map is company → parts of the business → primitives → components → values. Handles name each level: part:hiring, prim:payroll, comp:3f2a9c1d, val:9c2d41aa (the first eight characters of an id; the full id works too). node opens one; depth (0-4, default 1) says how many levels under it to include.

{
  "name": "get_organization",
  "arguments": {
    "ref": "acme-builders-3f2a9c1d"
  }
}

From the summary to one value with its source is four or five calls of a few hundred tokens each: the summary, node=part:…, node=prim:…, node=comp:…, then node=val:…. The same works on get_company_atlas for your workspaces, and on get_schema, get_mapping_guide, get_glossary, get_construct and compare.

node=prim:estimating, depth=2

{
  "node": {
    "h": "prim:estimating",
    "l": "Estimating",
    "layer": "process",
    "n": {
      "comps": 1,
      "vals": 2
    }
  },
  "items": [
    {
      "h": "comp:3f2a9c1d",
      "l": "In-house estimating team",
      "layer": "people",
      "conf": "inferred_high",
      "n": {
        "vals": 2
      },
      "items": [
        {
          "h": "val:9c2d41aa",
          "l": "Estimating method",
          "p": "estimating_method",
          "v": "estimating software",
          "s": 0
        }
      ]
    }
  ],
  "src": [
    [
      "https://acmebuilders.example/careers/estimator",
      "Estimator job post"
    ]
  ],
  "hint": "Call get_organization ref=acme-builders-3f2a9c1d with node=comp:3f2a9c1d for its 2 values and sub-components.",
  "tokens": 210
}

fields: only what you need

fields is a comma-separated list of keys to keep on each item; h and l always stay. Aliases: values (p, v, u), sources (s, q), counts (n), confidence (conf). With node=comp:… fields=values you get a component's values and nothing else.

Budgets, limits and the size cap

Every response is cut at a token budget: 2000 by default (6000 with view="full"), up to 8000 with budget. limit caps the items per response (up to 200). Overflow is never an error: what does not fit comes back as truncated with a cursor. At least one item is always returned, so a cursor always moves.

On top of that, no read returns more than 64 KB of JSON, whatever the view. A read that would be larger fails with 413 (RESPONSE_TOO_LARGE over REST, Error 413 from a tool) and says to narrow it (filters, node, fields) or page it. This only bites view=v1, which is not paged.

Cursors

Cursors are opaque: pass next_cursor as cursor with the same tool and the same arguments (the workspace, the ref, the filters). Lists over the whole registry (list_organizations, top, aggregate) page by key, not offset, so walking the last page costs the same as the first and rows never repeat or go missing while you walk. Shorter lists (list_industries, get_workspace, list_listings, search) page by position.

Ask the server, not the rows

aggregate: organizations counted by industry, region, size, model, kind, claimed, layer or primitive, with a metric's median and mean per group; or a metric's distribution.
top: the top organizations by a rankings dimension or by revenue, employees or funding, in any scope.
diff: what one organization has that another lacks, then shared primitives whose values differ.
coverage: what is unmapped for an organization, or across an industry (least mapped first).

{
  "name": "aggregate",
  "arguments": {
    "by": "region",
    "industry": "staffing",
    "limit": 3
  }
}

view: compact, full, v1

compact is the default. full adds descriptions, quotes and every field, with a larger budget. v1 returns the response these tools gave before compact envelopes (the whole atlas, schema or list at once, pretty-printed): use it only for code written against that shape, and expect a 413 on large organizations and workspaces. The parameter used to be called format; REST still accepts that name.

Tiers apply first

Compaction only shapes what a read already returned. Private values, the outsider view and what a reader without a key may see are decided before it, the same way as in the browser.

From a shell

The systatlas CLI does the paging for you and prints --json for agents:

systatlas org acme-builders-3f2a9c1d --node part:winning_work --json
systatlas aggregate --by region --industry staffing
systatlas export --industry staffing --csv > staffing.csv

PreviousRead the registry Next systatlas CLI