Workflows

A workflow is a linear 1–4 stage pipeline. Stage 1 receives seed URLs; each later stage follows a field extracted by the preceding stage.

Definition

{
  "name": "job-details",
  "description": "Search listings, then scrape each detail page",
  "definition": {
    "stages": [
      {
        "name": "search",
        "description": "Extract result links",
        "parser_id": "p_list…",
        "follow": "detail_url",
        "follow_limit": 500,
        "sink": { "mode": "database", "database": "jobs", "table": "search_results" }
      },
      {
        "name": "detail",
        "description": "Extract full job details",
        "parser_id": "p_detail…",
        "sink": { "mode": "database", "database": "jobs", "table": "job_details", "key": ["_url"] }
      }
    ],
    "dedupe": "per_run"
  }
}

Every workflow and stage needs a description. Non-final stages require follow; the final stage forbids it. follow_limit defaults to 1,000 and is capped at 10,000. dedupe is per_run or per_workflow.

Stages execute as normal fair-queue batches. Partial stage failures can still advance; structural failures (missing parser/sink, no successful jobs, or insufficient credits) fail the run.

Test before running

POST /v1/workflows/{id}/test accepts urls, max_items_per_stage (1–100, default 3), and test_next_stage. It performs a synchronous bounded preflight, creates no runs/batches/tables, and still bills fetch credits. The endpoint is limited to 10 calls/minute.

Run and monitor

• POST /v1/workflows/{id}/run with 1–1,000 seed URLs returns 202 { "run_id": "wr_…" }.
• GET /v1/runs/{id} returns current stage, batch counters, follow diagnostics, warnings, and credits spent.
• GET /v1/workflows/{id}/runs returns recent summaries.
• POST /v1/runs/{id}/cancel cancels the current stage and prevents later stages.

Cancel workflow-stage batches through the run endpoint, not /v1/batches/{id}/cancel.