SparkLang — the Spark programming language

Train and evaluate models with dry-run CI

A language beats another Python SDK because one reviewable .spark file covers train → eval → ship, with dry-run fixtures you can run in CI without API keys.

Offline-first by default: ./spark --dry-run. Real model train / model build jobs when you go live.

Download Spark Playground Getting started

SparkLang at sparklang.dev
MIT license
Source on GitHub
v0.6.3 · Changelog
Installers on this site

Why a language?

Library glue scatters prompts, training scripts, and eval harnesses across repos. A .spark program is one reviewable surface — same file for dry-run CI and live ops.

Typical Python SDK soup

client = OpenAI(...)
resp = client.chat.completions.create(...)
# separate train script, eval notebook,
# CI secrets, ad-hoc shell glue…

One .spark program

model train dataset "data.jsonl" base "base" out "out/job-1" backend "http" -> job
model status "job-1" -> status
ask "Score this adapter on the suite" -> report
print report

# ./spark --dry-run train_eval.spark

The wedge

Dry-run by default

./spark --dry-run uses offline fixtures — no keys, no network. Pass --live only when you want real calls.

./spark --dry-run examples/model_train.spark

Real training jobs

model train / model build submit adapters / checkpoints. Not a markdown plan sold as training.

model train dataset "…" base "…" out "…" backend "http" -> job
model status "job-id" -> status

Ask + eval in the same file

Bind model output, print it, chain classify / extract / pipeline after train — one program for the loop.

ask "Explain gravity in one sentence" -> text
print text

Train → status → eval

Dry-run writes fixture artifact paths (no GPU). Live uses a pluggable backend (http or allowlisted local-yield).

1 Train Submit dataset, base, and output path
2 Status Poll the job and print artifact paths
3 Eval (optional) Ask / classify against the result when you want

# model_train.spark — dry-run first
model train dataset "examples/fixtures/train/dataset.jsonl" base "fixture-base" out "out/train/job-dry-001" backend "http" -> job
model status "job-dry-001" -> status

Analyze / compare / improve are eval helpers. model plan writes an optional markdown plan only — not training. Model training docs · Build a Model.

Also available

Secondary language surfaces — useful, not the homepage thesis.

IDE language ops — IDE docs (ide ask, playbooks, LSP-friendly editing.
embed / retrieve — dry fixtures; live via spark-rag-http when configured.
Voice (demo / gated) — listen / speak; PSTN off by default (dial "+15555550100" placeholder only. Language reference.
Network / browser — capture and browser ops; first-class http get / http post are [roadmap]. Network + web.

Learn Spark

Follow the tutorial path — install, first program, AI primitives, then train.

Hello world

# Dry-run — no network, no API key
./spark --dry-run examples/hello.spark

# Optional live ask — any OpenAI-compatible gateway
export AI_GATEWAY_URL=http://127.0.0.1:4000
./spark --live examples/ask_live.spark

See programming guide and language reference.