Documentation

What the service does

Before you spend GPU time curating data toward a target domain, send a sample of the target and a sample of the candidate pool. The engine answers in seconds, on CPU, without touching your model. The corpus never leaves your machines; only samples travel, over HTTPS, processed in memory and never written to disk.

Every verdict carries a confidence tier (high, medium, low), a scope card, notes, a fingerprint of the exact text it was made on, and a forecast_id you use later to report the outcome. Every forecast is timestamped before your first training step. The receipt that ties forecast to outcome is the product.

Quick start

You need Python 3 and your API key from the dashboard. The reference client has no dependencies.

curl -O https://echo-program.com/client/echospace_pilot.py
export ECHOSPACE_URL=https://api.echo-program.com
export ECHOSPACE_API_KEY=es_pilot_...

python3 echospace_pilot.py selftest
python3 echospace_pilot.py forecast --target ./target_docs/ --pool ./pool_docs/ --label first-try \
    --model-params 124000000 --keep 0.25 --language en --regime pretraining --table --out fc_first.json
python3 echospace_pilot.py select-local --forecast fc_first.json --pool ./pool_docs/ --keep 0.25 --out keep.json
python3 echospace_pilot.py outcome --forecast fc_xxxxxxxxxxxx --selected 1.91 --baseline 2.05

• selftest checks the client's hashing against the service's test vectors. Run it once per machine.
• forecast draws seeded random samples of whole documents from both folders, sends them, and prints the verdict. With --table and a SELECT verdict, the saved file also carries the scoring table.
• select-local ranks your whole pool on your own machine from that table and writes keep.json. The pool is never uploaded.
• outcome reports the held-out result of the run under the forecast id. That is the receipt.

keep.json holds the indices of the pool documents to keep, in the order the client read them: files in sorted name order, walking folders recursively, then line order inside .jsonl files.

Documents and sizes

A document is one string. The client reads a folder recursively: every .jsonl line with a text field is one document, and any other file is one document. Over raw HTTP you send lists of strings.

More target data sharpens the reading. A target that arrives as a single document is accepted but noted; send one entry per document. English, whitespace-delimited text is the validated range; text without word boundaries is flagged in script_check.

Authentication

Every call carries the key as a bearer token. Keys are stored hashed on the service and revoked on request; one key per organisation. Send a User-Agent naming your tool; a bare default Python agent is rejected at the edge before it reaches the service.

curl -s https://api.echo-program.com/v1/me -H "Authorization: Bearer $ECHOSPACE_API_KEY" -H "User-Agent: my-pipeline/1.0"

{"key_id": "key_a6326f70", "org": "Acme Research", "daily_quota": 50, "used_last_24h": 2, "forecasts_total": 7}

Forecast

curl -s https://api.echo-program.com/v1/forecast \
  -H "Authorization: Bearer $ECHOSPACE_API_KEY" -H "Content-Type: application/json" -H "User-Agent: my-pipeline/1.0" \
  -d '{"target": ["doc one ...", "doc two ..."], "pool": ["...", "..."], "label": "first-try",
       "context": {"model_params": 124000000, "keep_fraction": 0.25, "language": "en", "regime": "pretraining"}, "table": true}'

{
  "forecast_id": "fc_7e2cc32e34e0",
  "verdict": "SELECT",
  "confidence": "medium",
  "reason": "...",
  "unstable": false,
  "target": {"documents": 6, "chunks": 42, "words": 5376},
  "pool": {"documents": 60, "chunks": 281, "words": 35968, "subsampled": false},
  "notes": [],
  "script_check": {"unspaced_fraction": 0.0},
  "scope": {
    "validated": {"model_size": "...", "keep_fraction": "...", "text": "...", "regime": "...", "selector": "..."},
    "outside_validated_range": [],
    "confidence_before_scope": "medium"
  },
  "fingerprint": {"target_sha256": "...", "pool_sha256": "...", "hasher": "es-h1", "engine": "pilot-0.2"},
  "provenance": {"client": "my-pipeline/1.0", "pool_total_documents": 812000, "target_total_documents": 3100, "sample_seed": 0},
  "engine": "pilot-0.2",
  "elapsed_s": 0.31,
  "scoring_table": {"format": "float32-le-base64", "buckets": 16384, "hasher": "es-h1", "sha256": "...", "values": "..."}
}

• unstable true means the two halves of the target disagree on the call; add target data before acting.
• scoring_table is present only when asked for and the verdict is SELECT; otherwise it is null with a note.
• Keep the response file. select-local needs it, and the fingerprint proves later which text the forecast was made on.

Selection

Two ways to turn a SELECT into a keep list. The local way is the one to use for any pool you would not want to upload.

{"selection_id": "sel_1db5b923d18a", "forecast_id": "fc_7e2cc32e34e0", "forecast_verdict": "SELECT", "mode": "local", "recorded": {"keep_fraction": 0.25, "pool_total_documents": 812000, "kept": 203000, "table_sha256": "...", "keep_sha256": "..."}}

• SELECT returns a targeted keep list (mode: "targeted").
• DO-NOT-SELECT returns a uniform random keep list of the same size (mode: "random"). That is the recommended action, not a fallback.
• NO-FORECAST is refused with 409: the engine will not choose a keep list it cannot stand behind.

{"selection_id": "sel_...", "forecast_id": "fc_...", "verdict": "SELECT", "mode": "targeted", "keep_fraction": 0.25, "pool_documents": 4000, "kept": 1000, "keep_indices": [3, 7, 12, ...], "hasher": "es-h1", "engine": "pilot-0.2", "note": "..."}

The scoring table and the hasher

The table is a plain published importance-weighting selector fitted on your own samples: one float per bigram bucket, sent as float32-le-base64 with its sha256. It says nothing about how the verdict was reached. To score a document, hash its word bigrams into buckets, average the table's values over them, and keep the top fraction of the pool by that score. The reference client does exactly this; the service's own selection agrees with it bit for bit.

{"hasher": "es-h1", "buckets": 16384,
 "rule": "tokens = re.findall(r'\\S+', text.lower()); h = zlib.crc32(token.encode('utf-8')) & 0xFFFFFFFF; bucket = (h[i] * 1000003 + h[i+1]) % buckets",
 "test_vectors": [{"text": "The quick brown fox jumps over the lazy dog.", "word_hashes": [...], "buckets": [...]}, ...]}

If you port the scorer into your own pipeline, run the test vectors before every selection. A hasher that drifts produces a keep list that is silently wrong. The version (es-h1) is stamped on every forecast, table and selection.

Outcome

{"outcome_id": "oc_a1f8bc1852a8", "forecast_id": "fc_...", "forecast_verdict": "SELECT", "forecast_agreed_with_outcome": true, "thanks": "recorded; this is the receipt"}

Where you can, train both arms: the curated keep list and a random keep list of the same size, same model, same budget, and report the held-out metric of each. One arm is still worth reporting.

Other calls

The interactive OpenAPI reference is at https://api.echo-program.com/v1/docs and the schema at https://api.echo-program.com/v1/openapi.json.

Errors

Engine calls run one at a time; a call may wait a few seconds behind another. Retry on network failure is safe: a forecast that did not return was not recorded.

Data handling

• Sent: samples only, over HTTPS, held in memory for the call, never written to disk, never reused.
• Kept: document and word counts, the fingerprint of the sample, the verdict, timestamps, the sampling provenance, and any outcome you report. Never your text. Never the scoring table.
• Aggregated, anonymised forecast and outcome statistics may improve calibration and appear in our publications. Never partner-identifiable, never text.