# AILANG Parse — Agent Reference

> Single-fetch reference for AI agents. Everything you need to call the AILANG Parse hosted service via MCP or REST: OAuth sign-in on the connector, the device authorization flow for headless agents, the tool catalog and REST examples.

If you are an AI agent, this is the page to read. The human-facing docs at [mcp.html](mcp.html) cover the same material with more prose.

---

## What this is

AILANG Parse parses documents **and generates them**. It converts 17 input formats into one consistent block schema (Text, Heading, Table, Image, Audio, Video, List, Section, Change, Link, Comment — 11 variants), and writes that schema back out to 9 output formats (docx, pptx, xlsx, odt, odp, ods, html, md, qmd). See "Generating documents" below — if you are an agent trying to *produce* a document rather than read one, read that section first. Office formats (DOCX, PPTX, XLSX, ODT, ODP, ODS, HTML, Markdown, CSV, EPUB, EML, MBOX) and LaTeX/arXiv papers (TEX, with \\input resolution) are parsed deterministically in milliseconds with no AI calls. PDFs are parsed deterministically by default via `--pdf-backend pdftotext` (poppler `pdftotext -layout`, no AI, highest character recall) — with `--pdf-backend docling` (IBM Docling, local ML layout) and `--pdf-backend liteparse` (run-llama LiteParse, font-size headings) as local alternatives. `--pdf-backend ai` (Gemini/Claude/OpenAI/Ollama) is required for scanned/image-only PDFs with no text layer; the hosted REST/MCP API exposes the AI path. A backend that extracts zero blocks fails loudly (non-zero exit) rather than writing an empty result. Images, audio, and video always go through AI.

Three ways to use it:

1. **Hosted MCP server** — two endpoints with the same tools on the same account:
   - `https://docparse.ailang.sunholo.com/mcp/connect/` — signs in with **OAuth**; the client holds the token. Best for AI clients that can open a browser (Claude, Claude Code, ChatGPT, Codex).
   - `https://docparse.ailang.sunholo.com/mcp/` — authenticated with a `dp_` API key (header preferred); headless agents get one with the device flow.
2. **Hosted REST API** — `https://docparse.ailang.sunholo.com/api/v1/*`. Use this if your agent does not speak MCP.
3. **Local MCP stdio server** — install via `npm i -g @ailang/parse` or `pip install ailang-parse`, then run a stdio MCP server with no auth and no quota. Best for developer machines.

This document focuses on the hosted variants. For local install, see [mcp.html](mcp.html).

---

## Quickest path (hosted MCP)

If your MCP client can run an OAuth sign-in (Claude, Claude Code, ChatGPT, Codex all can), connect to:

```
https://docparse.ailang.sunholo.com/mcp/connect/
```

The first call to a tool that needs an account gets HTTP 401 with a `WWW-Authenticate` header naming the protected-resource metadata. The client then runs OAuth 2.1 (PKCE S256, client ID metadata documents): it opens `https://docparse.ailang.sunholo.com/oauth-login.html`, the user signs in with Google or GitHub and clicks Allow, and the client receives an access token (a `dp_` key labelled `oauth: <app>`, valid 24 hours, renewed with a refresh token). Tools on this endpoint take **no `apiKey` argument**; never ask the user to paste a key.

Call `mcpFormats` first (no sign-in): it lists formats, sample IDs and how to send a document.

If your client cannot run OAuth (a headless agent, a script), use the agent surface `https://docparse.ailang.sunholo.com/mcp/` with a `dp_` key sent as `Authorization: Bearer dp_…` or `X-API-Key: dp_…`. Without a key, a quota-bounded tool there answers `AUTH_REQUIRED`; get a key with the device flow below.

---

## Device authorization flow (RFC 8628) — agent surface `/mcp/` and REST

Not needed on the OAuth connector `/mcp/connect/`. Use it when an agent or script needs a `dp_` key of its own.

The flow has three steps. The agent never sees the human's password — the human approves a short code in their browser, and the agent receives an API key.

### Via MCP tools

**Step 1 — start the flow.** Call `mcpAuth(label: "my-agent")`. The `label` is a human-readable name for the key (it will appear in the user's key list).

The response (returned by the backend `POST /api/v1/auth/device`) contains:

```json
{
  "device_code": "…",
  "user_code": "ABCD-1234",
  "verification_url": "https://www.sunholo.com/docparse/approve.html?code=ABCD-1234",
  "expires_in": 900,
  "interval": 5
}
```

**Step 2 — tell the human.** Print something like:

> Open `https://www.sunholo.com/docparse/approve.html?code=ABCD-1234` in your browser, sign in, and approve. I'll wait.

The `verification_url` already carries the code as a query parameter, so hand the human the whole URL rather than asking them to retype it.

**Step 3 — poll.** Every 5 seconds, call `mcpAuthPoll(deviceCode: "<device_code from step 1>")`. The response will be one of:

- `{"error": {"code": "AUTHORIZATION_PENDING", "retryable": true, …}}` — keep polling every 5s.
- `{"status": "approved", "api_key": "dp_…", "tier": "free"}` — store the API key. Use it for all subsequent `mcpParse`, `mcpConvert`, `mcpAccount` calls.
- `{"status": "expired"}` — the device code expired (15 minutes; `expires_in` is authoritative). Start over with `mcpAuth`.

### Via raw HTTP (for non-MCP agents)

Same two endpoints, called directly:

```bash
# Step 1: start device flow
curl -s -X POST https://docparse.ailang.sunholo.com/api/v1/auth/device \
  -H "Content-Type: application/json" \
  -d '{"label":"my-agent","scope":"parse"}'

# Step 2: human opens verification_url and enters user_code

# Step 3: poll every 5s until approved
curl -s -X POST https://docparse.ailang.sunholo.com/api/v1/auth/device/poll \
  -H "Content-Type: application/json" \
  -d '{"deviceCode":"<device_code from step 1>"}'
```

### After approval

Store the `api_key` (it starts with `dp_`). Send it as a header — `Authorization: Bearer dp_…` or `X-API-Key: dp_…` — on REST calls and on the MCP agent surface, so it never enters the model's context. The `apiKey` tool argument still works where a client cannot set headers.

A working end-to-end shell example lives in [examples/agent_workflow.sh](https://github.com/sunholo-data/ailang-parse/blob/main/examples/agent_workflow.sh).

---

## Tool catalog

`tools/list` on the connector `/mcp/connect/` returns **7** tools: `mcpParse`, `mcpConvert`, `editDocument`, `mcpFormats`, `mcpEstimate`, `mcpAccount` and `submit_feedback`. The agent surface `/mcp/` returns the same plus `mcpAuth` and `mcpAuthPoll`. Large-file upload (`getUploadUrl`) is REST-only (`POST /api/v1/upload/url`, Business tier). Tool results are JSON objects.

**Giving a tool a document** (`mcpParse`, `mcpConvert`, `editDocument`) — pass ONE of:
- `content` + `filename`: the document itself. Text formats (Markdown, HTML, CSV) as plain text; binary files (DOCX, PDF, PPTX, XLSX, ODT, EPUB, …) base64-encoded with `contentEncoding: "base64"`. The filename's extension picks the parser (`notes.md`, `report.docx`).
- an `https://` URL of a publicly reachable document (`filepath` on mcpParse/editDocument, `input` on mcpConvert). Internal and private addresses are refused.
- a `sample_…` ID from `mcpFormats`.

The hosted server never reads a path on the caller's machine.

| Tool | Sign-in? | Purpose | Key params |
|---|---|---|---|
| `mcpFormats` | no | Formats, samples, how to send a document. **Call this first.** | (none) |
| `mcpEstimate` | no | Cost, latency and whether the request counts as AI, before parsing. | `filepath` (sample ID), `outputFormat` |
| `mcpParse` | yes | Parse a document into blocks / markdown / HTML. | `content`+`filename` (+`contentEncoding`) or `filepath`; `outputFormat` |
| `mcpConvert` | yes | Create or convert a document; returns `download_url`. | `content`+`filename` (+`contentEncoding`) or `input`; `outputFormat` |
| `editDocument` | yes | Apply JSON edit deltas, return modified blocks. Office formats only. | `content`+`filename` or `filepath`; `deltas` |
| `mcpAccount` | yes | Plan, usage, keys, request-history setting. | `action` |
| `submit_feedback` | no | Report a bug, missing format or docs gap. | `title`, `body`, `category`, `ailang_version`, `package="sunholo/ailang_parse"` |
| `mcpAuth` | no | `/mcp/` only. Start RFC 8628 device authorization. | `label` |
| `mcpAuthPoll` | no | `/mcp/` only. Poll for device approval. | `deviceCode` |

On `/mcp/connect/` the client's OAuth token authenticates "yes" tools. On `/mcp/` send a `dp_` key as a header (or `apiKey`).

### Tool details

**`mcpParse`**
- `outputFormat`: `"blocks"` (default, structured JSON), `"markdown"`, or `"html"`.
- PDFs are parsed with `pdftotext` (no AI, not an AI request). Images use AI and count as one AI request.

**`mcpConvert`**
- `outputFormat`: one of `docx`, `pptx`, `xlsx`, `odt`, `odp`, `ods`, `html`, `md`, `qmd` (case-insensitive; `markdown`/`htm`/`quarto` are aliases).
- To write a new document, send Markdown as `content` with `filename: "notes.md"`.
- Returns `{"status", "source_format", "target", "filename", "content_type", "encoding", "size_bytes", "content", "download_url", "download_expires_in"}`.
- **`download_url`** is a private link to the generated file, valid for one hour — give it to the user. The file is deleted from storage within a day.
- **`encoding` is load-bearing** if you use `content`: `base64` for the six ZIP container targets (docx, pptx, xlsx, odt, odp, ods), `utf8` for html, md, qmd. Branch on the field, never on the target.

(In *local* stdio mode the same tool writes to disk and returns `{"output": "<path>", "format": "<fmt>"}` instead.)

**`editDocument`**
- `deltas`: a JSON array of edit operations; `""` or `"[]"` round-trips. Returns modified **blocks**, not a file — send their Markdown to `mcpConvert` for a document.

**`mcpFormats()`** — no params, no quota. Input formats with feature flags, output formats, ~26 sample files, and how to send a document. Plans and prices are at `GET /api/v1/pricing`.

**`mcpEstimate(filepath, outputFormat)`** — whether the request counts as AI (`counts_as_ai_request`), expected latency and strategy.

**`mcpAccount(action)`** — `"status"` (default: plan, usage, remaining allowance), `"keys"` (`"usage"` is an alias), `"history"`, `"history_on"`, `"history_off"`.

**`mcpAuth(label)`** / **`mcpAuthPoll(deviceCode)`** — agent surface only; see the device flow above.

---

## Generating documents

If you need to *produce* a document — a report, a deck, a spreadsheet — this is
the section you want.

**Markdown is the format you can write, so it is how you generate a document.**
Write Markdown, then convert it. There is no "create a DOCX" tool and none is
needed: the Markdown parser is a full authoring front end.

```bash
curl -s -X POST https://docparse.ailang.sunholo.com/api/v1/convert \
  -H "X-API-Key: dp_…" -F "filepath=@report.md" -F "target=docx"
```

Or over MCP: `mcpConvert(content: "<the Markdown>", filename: "report.md", outputFormat: "docx")` — then give the user the returned `download_url`.

**What survives Markdown → any of the 9 output formats:**

| Feature | Notes |
|---|---|
| YAML front matter | `title:` / `author:` / `date:` become document properties |
| `**bold**` `*italic*` `` `code` `` `~~strike~~` | real character formatting, not literal asterisks |
| `[links](url)` | real hyperlinks |
| `![images](path.png)` | local paths are read and embedded |
| Fenced code blocks | preserved as code |
| Blockquotes, nested lists, thematic breaks | preserved |
| Tables | including column alignment and colspan |

**What Markdown cannot express:** headers, footers, comments, tracked changes.
These have no Markdown syntax. They survive only when converting *from* a
document that already contains them (e.g. DOCX → DOCX). Do not promise a user a
generated document with a running header — say it needs a source document.

**Targets:** `docx` `pptx` `xlsx` `odt` `odp` `ods` `html` `md` `qmd`. For
`pptx`, headings become slides. For `xlsx`/`ods`, tables become sheets.

**AI generation from a prompt** (`--generate report.docx --prompt "…"`) exists
only in the local `docparse` CLI. It is **not** on the hosted API —
`/api/v1/convert` is deterministic conversion only, and is priced as such. If a
user wants a document authored from a prompt, write the Markdown yourself and
convert it.

**Cost:** one request per conversion, on the same counters as `/parse`. The AI
sub-quota is consumed only when the *source* format needs AI (PDF, images) —
generation itself is pure compute. Output size does not affect the charge.

## REST fallback (for non-MCP agents)

Base URL: `https://docparse.ailang.sunholo.com`

### Discover

```bash
# All 7 tools as machine-readable JSON Schema definitions
curl -s https://docparse.ailang.sunholo.com/api/v1/tools

# Service capabilities (formats, pricing, samples)
curl -s https://docparse.ailang.sunholo.com/api/v1/capabilities

# Pricing tiers (no auth)
curl -s https://docparse.ailang.sunholo.com/api/v1/pricing
```

### Parse a document

```bash
curl -s -X POST https://docparse.ailang.sunholo.com/api/v1/parse \
  -H "Authorization: Bearer dp_…" \
  -H "Content-Type: application/json" \
  -d '{"sample_id":"sample_docx_tables","outputFormat":"markdown"}'
```

To send your own file, upload it as multipart (`-F "filepath=@file.docx"`) or pass an https `sourceUrl`; the server never reads a path you name. Sample IDs come from `mcpFormats` / `/api/v1/capabilities`.

### Generate a document

```bash
# Upload Markdown, get a DOCX back inline (base64 — decode on `encoding`)
curl -s -X POST https://docparse.ailang.sunholo.com/api/v1/convert \
  -H "X-API-Key: dp_…" -F "filepath=@report.md" -F "target=docx" \
  | python3 -c 'import base64,json,sys; d=json.load(sys.stdin); d=json.loads(d["result"]) if isinstance(d.get("result"),str) else d; \
open(d["filename"],"wb").write(base64.b64decode(d["content"]) if d["encoding"]=="base64" else d["content"].encode())'
```

Same four input modes as `/parse`: multipart upload, `sample_id`, `sourceUrl`
(https), `gcsRef` (gs://, Business tier). The parameter is `target`, not
`outputFormat`.

### Estimate before parsing

```bash
curl -s -X POST https://docparse.ailang.sunholo.com/api/v1/estimate \
  -H "Content-Type: application/json" \
  -d '{"filepath":"my-report.pdf","outputFormat":"blocks"}'
```

### Check account

```bash
curl -s -X POST https://docparse.ailang.sunholo.com/me/entitlements \
  -H "Authorization: Bearer dp_…" \
  -H "Content-Type: application/json" \
  -d '{"apiKey":"dp_…"}'
```

---

## Environment variables

These are read by the AILANG `docparse` runtime and the SDKs. They control which deployment mode the tools target.

| Variable | Default | Meaning |
|---|---|---|
| `DOCPARSE_MODE` | `local` | `local` (no auth, parse from FS) or `hosted` (call the API, requires key). |
| `DOCPARSE_API_URL` | `https://docparse.ailang.sunholo.com` | Override for self-hosted deployments. |
| `DOCPARSE_API_KEY` | (unset) | API key for hosted mode. Alternative to passing `apiKey` per call. |

If you connect over remote MCP, the server is already in hosted mode and you set none of these: on `/mcp/connect/` your client signs in with OAuth, on `/mcp/` you send a `dp_` key as a header.

---

## Common error codes

| Code | Meaning | What to do |
|---|---|---|
| `AUTH_REQUIRED` | Quota-bounded tool called on `/mcp/` with no key. | Send a `dp_` key as a header; get one with the device flow, or use `/mcp/connect/` (OAuth). On `/mcp/connect/` the same case is HTTP 401 and the client signs in. |
| `LOCAL_MODE` | Auth/account tool called in local mode. | Local mode has no quota. Just call `mcpParse` directly. |
| `FILE_NOT_FOUND` | `filepath` does not exist (or sample ID unknown). | Check the path, or call `mcpFormats` for valid sample IDs. |
| `UNSUPPORTED_FORMAT` | `outputFormat` not in the supported list. | See `mcpFormats` → `output_formats`. |
| `INVALID_ACTION` | `mcpAccount` called with an unknown action. | Use `status`, `pricing`, `usage`, `history`, or `keys`. |

---

## Pricing tiers (summary)

| Tier | Price | Requests/month | AI parses/month | Daily limit | Max file |
|---|---|---|---|---|---|
| Free | EUR 0 | 1,000 | 50 | 500 | 10 MB |
| Pro | EUR 29/mo | 100,000 | 500 | 5,000 | 25 MB |
| Business | EUR 99/mo | 500,000 | 2,000 | 20,000 | 50 MB |

Pricing is per *document*, not per page — a 200-page PDF and a one-line memo each count as one request. Live pricing comes from `mcpAccount(action: "pricing")` or `GET /api/v1/pricing`.

---

## Review annotations (comments, highlights)

Comments carry author, timestamp, and the span of document text they annotate — not a
detached list. Every format emits the same `CommentBlock` shape, so one consumer handles
review comments across DOCX, XLSX, PPTX and PDF.

```json
{"type": "comment", "author": "Laura", "text": "This indemnity is far too broad.",
 "anchorText": "shall indemnify", "anchorKind": "range", "anchored": true,
 "parentId": "3", "resolved": false}
```

- `anchorText` — the exact text the comment annotates. `anchored: false` with an empty
  `anchorText` means the anchor could not be resolved; it never falls back to nearby text,
  because a confidently misattributed comment is worse than an unanchored one.
- `parentId` — the comment this one replies to, so threads can be reconstructed.
- PDF highlights are anchored by intersecting `/QuadPoints` with word positions, using word
  CENTRE rather than any overlap. Some PDFs store annotations in compressed object streams;
  those return what is visible plus a `mayBeIncomplete` flag rather than silent truncation.

Note: comments are returned on the standard parse path — no extra flag or endpoint.

## Chunking for RAG (v0.27+)

Chunking runs over the parsed block list, not raw text, so boundaries land on real document
structure. It is deterministic and free — no tokenizer, no model call.

Four strategies: `Auto` (picks from the document's composition), `FixedSize` (token budget
+ overlap), `ByStructure` (new chunk per heading), `BySection` (one per slide / sheet /
attachment). There is deliberately no embedding-based strategy — it would need an AI
round-trip, and a chunker that silently costs money is the wrong default.

Guarantees worth relying on:
- **Blocks are never split.** A table stays whole even when it exceeds the budget.
- **Tables keep their cell contents** in chunk text (not the `[table]` placeholder).
- **Every chunk carries provenance** — `headingContext` (nearest heading above it),
  `sectionKind`, `startBlockIndex`/`endBlockIndex`, and `hasTable` — so a retrieved passage
  can be cited with the clause it came from.
- **`tokenCount` is an estimate** (`words × 1.3`). Leave headroom against hard model limits.

See [chunking.html](chunking.html).

## Where to go next

- [chunking.html](chunking.html) — deterministic RAG chunking, four strategies, citation provenance.
- [comments.html](comments.html) — comment extraction across all formats.
- [mcp.html](mcp.html) — full human-readable MCP documentation, including Claude Desktop / Claude Code setup snippets.
- [api.html](api.html) — REST API reference.
- [examples/agent_workflow.sh](https://github.com/sunholo-data/ailang-parse/blob/main/examples/agent_workflow.sh) — complete bash agent lifecycle (discover → auth → estimate → parse → quota check).
- [GitHub repo](https://github.com/sunholo-data/ailang-parse) — source, AILANG modules, SDKs (Python, JS, Go).