Parsers

Authoring loop

1. Inspect representative page content.
2. Draft a complete v2 config.
3. Call POST /v1/parsers/test with an inline config and url (or a supplied body).
4. Fix fields with nulls or zero matches.
5. Create with POST /v1/parsers; execute with scrape or batches.

The dry-run uses the production fetch/extraction path and returns bounded per-field match/null counts, byte sizes, samples, warnings, and a data sample. URL tests bill scrape credits; body-only tests are free and capped at 2 MB.

Config shape

{
  "name": "books-list",
  "type": "html",
  "item_selector": "article.product_pod",
  "fields": [
    { "name": "title", "selector": "h3 a", "attr": "title" },
    {
      "name": "price",
      "selector": ".price_color",
      "type": "float",
      "transform": [{ "op": "regex", "pattern": "([0-9.]+)" }]
    }
  ]
}

Every config has a type and 1–64 uniquely named fields.

Config type	Input	Field locator	List mode
html	HTML	CSS/XPath selector, optional attr	item_selector
json	JSON embedded in HTML	gjson path	item_path
api	Raw JSON response	gjson path	item_path
regex	Any text	regex pattern and capture group	Not available

List mode returns { "items": [...] }; otherwise fields form one object.

Fields and transforms

Field types are string (default), int, float, bool, and date. Date fields need a Go-style layout and can specify a timezone. required, default, and multiple control missing and repeated values.

Transforms run in order: trim (default), no_trim, normalize_ws, regex, replace, map, and resolve_url. A failed conversion becomes null plus a dry-run warning.

CRUD

• GET /v1/parsers lists compact summaries; ?full=1 includes configs.
• POST /v1/parsers creates v2 only. description and config are required.
• GET /v1/parsers/{id} returns the full record.
• PUT /v1/parsers/{id} is a full replacement, not a patch.
• DELETE /v1/parsers/{id} removes it.

owner_id is immutable and rejected. Legacy v1 parsers remain readable/executable but a PUT must retain the original format.

Execute

GET|POST /v1/scrape/{id} fetches a URL. Add async=true for queue execution or a database sink in POST. POST /v1/parse/{id} remains only as a compatibility path for HTML you already have; it is not the authoring/test workflow.