# Ephemera — Model inference on a local Ollama daemon (MLX + CLI)

> Self-executing Markdown. The **Ollama binding** of the *inference* intent — serve a model on this
> machine via a long-running daemon (`ollama serve`), invoked through the CLI (`ollama run`) or the local
> REST API (`/api/generate`, `/api/chat`), with an OpenAI-compatible `/v1` surface layered on top. The
> daemon is the source of truth; this file is intent + ledger + audit.

> **Provides / Requires**: **Provides** `inference-endpoint(http://localhost:11434, MODEL)` (+
> OpenAI-compat `/v1`) — consumers discover the served **tag** (`MODEL:latest`) via `GET /api/tags`, never
> assume the bare name. **Requires** nothing mandatory — the daemon and base model are prerequisites this
> plan provisions itself.

---

## 🤖 Director prompt

You are the Director. Execute this plan:
- Observe-before-act; verify each step before advancing
- **No 🔴 gates in this plan** — local inference has no denial-of-wallet, global, or cross-account blast
  radius; the only irreversible act is `ollama rm` (deleting a local model), marked 💥, not 🔴
- Write realized values + verify results back into Live State after each step
- Teardown never removes the **base model** (`BASE_MODEL`) — only the **derived model** (`MODEL`) this
  plan created
- Use only the commands in this plan

> **Candor:** the durable learnings folded into this plan (the reasoning-model empty-response trap, `ollama
> ps` omitting the KV cache, metadata ≠ acceptance, the memory-characterization method in §4) come from a
> dogfooded run against a live daemon elsewhere — proven behavior, not speculation. This contributed copy's
> Live State ships reset to `not-created` (per Ephemera's contribution contract): treat every ✔ row as
> unverified until you re-run it against YOUR OWN daemon. Re-confirm exact flags against `ollama --help` /
> `ollama <cmd> --help` before any run — the live CLI remains the source of truth.

```
Legend  🟢 create · 🟡 config · 🔴 GATE · 💥 destructive · ⏳ wait · ✔ verify
```

## Intent

Serve **text generation** from a local model — no cloud account, no network egress, no per-token bill —
by running the Ollama daemon on this machine, pulling (or already holding) a base model, and layering a
derived model (a `Modelfile` — base + parameters, e.g. an extended `num_ctx`) on top. The unit of work is
a **served tag** reachable at `http://localhost:11434`, callable via `ollama run`, `/api/generate`,
`/api/chat`, or the OpenAI-compatible `/v1/chat/completions`.

**Shared acceptance contract** (every inference binding — Workers AI / AWS Bedrock / GCP Vertex / **Ollama (local)**
— must pass):
1. an inference call returns a well-formed non-empty completion for `text-gen`
2. the call uses the **resolved model id** (determinism — same inputs ⇒ same served tag)
3. **(local-specific)** the endpoint is reachable at the resolved URL — `http://localhost:11434` answers,
   proving the *daemon*, not just the model, is live

### Candor — MLX backend

- As of authoring, the daemon is **Ollama 0.31.x on the MLX backend** — confirm against your live `ollama
  --version` / `ollama --help`; the live CLI is the source of truth. `qwen3.6:27b-mlx` runs on **MLX**, not
  `llama.cpp`. `OLLAMA_KV_CACHE_TYPE`
  and `OLLAMA_FLASH_ATTENTION` are llama.cpp-era knobs that **may be silent no-ops on MLX** — default
  `KV_CACHE=none` and never treat a set env var as proof it took effect. §4 verifies **memory headroom
  (the outcome)**, never `/api/version` liveness, as the signal that a context-size change actually landed.
- `qwen3.6:27b-mlx` emits a `thinking` channel before its final answer. A small `num_predict` can exhaust
  the budget mid-thought and return an **empty** `response` field even though the call "succeeded" —
  that is a **false pass**, not evidence of a working model. Acceptance therefore asserts
  `done_reason=="stop"` **AND** `eval_count>0` **AND** a non-empty **final** answer with an adequate
  `num_predict` budget — never bare `.response` non-empty.
- **Metadata ≠ acceptance.** `ollama show <model>` reporting the right `num_ctx` does **not** prove the
  derived model actually *loads and runs* on MLX at that context size — only the load-and-generate proof
  in §5 does. Treat `ollama show` as a config-intent check, not a functional one.

## Provisioning Inputs

| # | Question | Options (closed enum) | Default | Sets | Gates |
|---|----------|-----------------------|---------|------|-------|
| 1 | Which base model to run? | free-text (a pulled/pullable Ollama model id) | `qwen3.6:27b-mlx` | `BASE_MODEL` | §2 (base-model presence) |
| 2 | Context window for the derived model? | `8192` / `32768` / `65536` / `131072` | `65536` | `NUM_CTX` | §3 (`Modelfile` `num_ctx`) + §4 (memory headroom) |
| 3 | Name of the derived (served) model? | free-text | `qwen3.6-coder-64k` | `MODEL` | §3 (derived-model create) + the served tag consumers discover |
| 4 | KV cache quantization? | `none` / `q8_0` / `q4_0` | `none` | `KV_CACHE` | §1 (daemon env) — **backend-dependent — may no-op on MLX** |
| 5 | How does the daemon start? | `manual` / `login-agent` | `manual` | `AUTOSTART` | §1 (daemon startup method) |

```yaml
# → written into Live State once resolved
resolved_inputs:
  base_model: qwen3.6:27b-mlx
  num_ctx:    65536
  model:      qwen3.6-coder-64k
  kv_cache:   none
  autostart:  manual
  resolved_by: <human who confirmed>
  resolved_at: <timestamp>
```

## Portability ledger note

**Local inference has no 🔴 gate** — there is no denial-of-wallet, no global blast radius, no
cross-account exposure; the machine running the daemon is the only affected surface. The lone
irreversible act is `ollama rm` (deletes a pulled/derived model from local disk), marked 💥. Everything
else — pulling a base model, writing a `Modelfile`, creating a derived model — is free, local, and
**idempotent by content**: Ollama dedupes model layers on re-`create`/re-`pull`, so re-running §2/§3 with
unchanged inputs does not duplicate storage or re-download shared layers. (Mirrors the sibling's "Cloudflare
has no resource-tag API" — a provider-shape asymmetry recorded here, not a gap to work around.)

## 0. Variables

```bash
export OLLAMA_HOST="127.0.0.1:11434"
export BASE_MODEL="qwen3.6:27b-mlx" NUM_CTX="65536" MODEL="qwen3.6-coder-64k" KV_CACHE="none"
```

## Dependency frontier

```
daemon (§1) ─> base-model (§2) ─> derived-model (§3) ─> [§4 memory headroom, §5 acceptance]
```
Non-negotiable edges: the daemon must be reachable before any model op (§2/§3 both call into it); the
base model must be present before the derived model's `Modelfile` (`FROM $BASE_MODEL`) can resolve; the
derived model must exist before §4 (memory headroom) or §5 (load-and-generate acceptance) can run against
it. Teardown reverses this — derived model first, base model untouched (see Director prompt).

## 1. Daemon present & reachable  🟡

> Observe first — reachability is a free read-only check; only install/start if it fails. `ollama serve`
> is not safely re-runnable against an already-bound port, so this branch only fires on absence.

```bash
curl -fs "http://$OLLAMA_HOST/api/version" >/dev/null 2>&1 && echo "daemon already reachable" || {
  command -v ollama >/dev/null 2>&1 || brew install ollama
  if [ "$AUTOSTART" = login-agent ]; then
    brew services start ollama   # writes/loads the launchd plist — daemon survives reboot/login
  else
    ollama serve &                # manual: this-session only, no plist
  fi
}
```

✔ verify (read-only, re-runnable, never gated):
```bash
curl -fs "http://$OLLAMA_HOST/api/version" && echo " <-VERSION-OK"
```
> → Live State: `daemon reachable` = observed version payload, result PASS/FAIL.

## 2. Base model present  🟡⏳

> `ollama pull` is idempotent by content — layers are content-addressed, so a present `BASE_MODEL` no-ops
> instantly; an absent one streams multi-GB. Background the waiter rather than blocking the Director.

```bash
ollama list | grep -qF "$BASE_MODEL" && echo "base model already present" || {
  ollama pull "$BASE_MODEL" &      # 🟡⏳ background — multi-GB download
  wait $!
}
```

✔ verify (read-only, re-runnable):
```bash
ollama list | grep -F "$BASE_MODEL" && echo BASE-PRESENT
```
> → Live State: `base present` = `$BASE_MODEL`, result PASS/FAIL.

## 3. Derived model  🟢

Modelfile — a pure function of the inputs (deterministic; no hidden state):

```
FROM qwen3.6:27b-mlx
PARAMETER num_ctx 65536
```

Render test (confirms the file is a pure function of `BASE_MODEL` + `NUM_CTX` — no hidden state):

```bash
cd <project>   # the directory containing this plan's Modelfile — your project root
export BASE_MODEL="qwen3.6:27b-mlx" NUM_CTX="65536"
printf 'FROM %s\nPARAMETER num_ctx %s\n' "$BASE_MODEL" "$NUM_CTX" | diff - Modelfile && echo RENDER-OK
```

Create the derived model (idempotent — re-running with an unchanged `Modelfile` dedupes via
"using existing layer" lines rather than a fresh build):

```bash
cd <project>   # the directory containing this plan's Modelfile — your project root
ollama create "$MODEL" -f Modelfile
```

✔ verify (metadata — config-intent check, read-only, re-runnable):
```bash
ollama show "$MODEL" --parameters | grep -F 'num_ctx' | grep -F "$NUM_CTX" && echo NUMCTX-OK
ollama show "$MODEL" --modelfile | grep -E '^FROM ' | grep -Fq "$MODEL" && echo FROM-OK
```

> **Metadata is NOT acceptance.** MLX custom-model import has been preview-gated upstream; `ollama show`
> reporting the right `num_ctx` (and a `FROM` line) only proves the *manifest* is well-formed — it does
> not prove the derived model *loads and runs* on the MLX backend. Observed live: `ollama show "$MODEL"
> --modelfile` prints `FROM qwen3.6-coder-64k:latest` — self-referential, because `show` reconstructs a
> Modelfile from the model's own manifest, not the original `Modelfile`'s `FROM qwen3.6:27b-mlx` — a
> concrete illustration of why metadata is a config check, not a functional one. §5 (load-and-generate)
> is the only proof that the model actually runs. The `FROM-OK` grep above is a **coincidental substring
> pass, not a lineage check** — post-create, `ollama show --modelfile` prints this self-referential
> `FROM qwen3.6-coder-64k:latest` line, so `grep -Fq 'qwen3.6'` would pass for *any* tag sharing the
> `qwen3.6` prefix, derived or not. Determinism is actually guaranteed **upstream** of this step: §3's
> render test (`diff` against the deterministic `printf` render, §3 above) plus `ollama create` run from
> that exact verified `Modelfile` — not by this post-create `show`/grep, which only confirms a manifest
> exists.

> → Live State: `model present` = observed `ollama list`/`api/tags` row, result PASS/FAIL; `num_ctx
> correct` = observed value, result PASS/FAIL.

## 4. Memory headroom  🟡

> **The honest signal is OS memory, not `ollama ps`.** `ollama ps` SIZE reports the **weights only** —
> roughly the base model's on-disk size — whether the context is empty or filled to `NUM_CTX`: it does
> **NOT** include the MLX KV cache. So `ollama ps` SIZE is *not* a headroom measure at any fill. The KV
> cache is real but only **OS-visible** (`memory_pressure` / free-% / the runner's RSS). A `/api/version`
> 200 is likewise not this gate's assertion (decorative — proves only that HTTP answers). §4 reasons about
> unified-memory headroom from the OS, and treats the worst-case context fill as a **characterized
> constant you measure once on your own host**, not a per-run stress test.

> **Memory fact (characterize ONCE on your own host — do NOT re-run on drift checks).** `num_ctx` is a
> *ceiling*; MLX allocates the KV cache lazily, so memory grows with the tokens actually used, not with the
> ceiling itself. **On a memory-constrained host, filling near-full context can drive the host into memory
> pressure and even swap (pageouts)** — weights (≈ the base model's on-disk size) + a full-context KV cache
> + ambient apps (OS, a terminal, a browser) can approach or exceed physical RAM on a smaller unified-memory
> (`<TOTAL_RAM>`) machine. **Guidance: run one deliberate near-full-context stress on your own host, record
> the observed memory-pressure result HERE in Live State as your characterized worst case, and do not
> re-run it on every drift check** — a bigger or smaller `<TOTAL_RAM>` will land at a different number, so
> there is no universal figure, only the method. If your characterization comes back tight, the guidance is
> the same regardless of host size: keep the ceiling, use near-full context sparingly, and expect pressure
> near a full fill.

🟡 *Optional* daemon-env knob — adopt ONLY if a live measurement proves it moves resident memory on MLX:

```bash
# NOT adopted by default — see Live State decision note below.
# export OLLAMA_KV_CACHE_TYPE="$KV_CACHE"   # $KV_CACHE=none — MLX runner; llama.cpp-era knob, likely no-op
# export OLLAMA_FLASH_ATTENTION=1           # same candor: unproven on this backend, do not set speculatively
```

`OLLAMA_KV_CACHE_TYPE`/`OLLAMA_FLASH_ATTENTION` are llama.cpp-era env knobs; on the MLX runner (confirm via
`ollama ps` reporting `100% GPU` for your derived model, i.e. MLX, not a llama.cpp GGUF path) they are
unverified and possibly silent no-ops. Per the plan's MLX candor, leave them **unset** by default — confirm
`echo "daemon PATH" | launchctl getenv OLLAMA_KV_CACHE_TYPE` returns nothing on your daemon before trusting
that neither var is set — `KV_CACHE=none` stands, unless a live before/after resident-memory delta on YOUR
backend proves a knob actually moves the number. Absent that delta, leave both unset; this is a deliberate
config no-op, not an oversight.

**Recurring ✔ (cheap, non-stressing — safe on every drift check).** Prove the model *loads and runs* (the
weights floor) and that the host has resting headroom for a large-context run — WITHOUT re-filling `NUM_CTX`:

```bash
# 1. weights-load floor: model loads and answers (tiny budget — does NOT fill the context)
curl -fs http://localhost:11434/api/generate -d '{"model":"'"$MODEL"'","prompt":"hi","stream":false,"options":{"num_predict":8}}' >/dev/null
ollama ps | grep -F "$MODEL" | grep -Fq 'GB' && echo FLOOR-LOADS
# 2. resting OS headroom (the real signal): free memory is healthy BEFORE any big-context run
memory_pressure 2>/dev/null | awk -F: '/free percentage/{gsub(/[ %]/,"",$2); print "FREE_PCT="$2; exit}'
```

Assertions: `FLOOR-LOADS` (the derived model loads and answers — its KV-invisible weights floor ≈ the base
model's on-disk size, `100% GPU`), and `FREE_PCT` is comfortably high at rest (**≥ 40% is a reasonable
resting-headroom guideline**), so a subsequent large-context run has room. **Do NOT re-fill `NUM_CTX` here**
— the near-full-context worst case is the *characterized constant* you record once (Memory-fact note
above), deliberately not re-measured per run. If `FREE_PCT` is low at rest, that is the drift signal (close
apps / unload other models before a big-context run); if you need a large ceiling to fit comfortably, the
only real lever is an **MLX-native** KV-cache reduction — and none is confirmed on this backend, so
`KV_CACHE=none` stands until one is proven live.

> → Live State: `memory headroom` = the floor (loads/runs) + your one-time worst-case-context
> characterization (record the observed memory-pressure result — tight or comfortable — for YOUR host) +
> `KV_CACHE=none` decision; result = **PASS (floor + documented worst-case)** once characterized, with a
> tight-memory caveat if your host's characterization comes back tight.

## 5. Acceptance (load-and-generate)  ✔

> The **shared acceptance contract** member this binding proves locally: (1) a well-formed non-empty
> completion, (2) the resolved model id (`$MODEL`, echoed back in the response), (3) the endpoint answers
> at the resolved URL. All three, reasoning-model-aware.

> **Reasoning-model-aware acceptance (CRITICAL).** `qwen3.6:27b-mlx` emits a `thinking` channel before its
> final answer. A small `num_predict` can exhaust the budget mid-thought and return an **empty** `response`
> even though the call "succeeded" — a false pass. Acceptance asserts `done_reason=="stop"` **AND**
> `eval_count>0` **AND** a non-empty **final** answer, with an **adequate** budget (`num_predict:256`).
> Never assert bare `.response` non-empty with a tiny budget.

Native `/api/generate`:
```bash
curl -fs http://localhost:11434/api/generate \
  -d '{"model":"'"$MODEL"'","prompt":"Reply with exactly: LOADED","stream":false,"options":{"num_predict":256,"temperature":0}}' \
  | python3 -c 'import sys,json; d=json.loads(sys.stdin.read(),strict=False);
assert d.get("done_reason")=="stop", d.get("done_reason");
assert d.get("eval_count",0)>0, "no tokens";
assert d.get("response","").strip(), "empty final answer";
assert d.get("model")=="'"$MODEL"'", d.get("model");
print("GEN-OK:", repr(d["response"].strip()[:40]))'
```

OpenAI-compatible `/v1/chat/completions`:
```bash
curl -fs http://localhost:11434/v1/chat/completions \
  -d '{"model":"'"$MODEL"'","messages":[{"role":"user","content":"Reply with exactly: LOADED"}],"max_tokens":256,"temperature":0}' \
  | python3 -c 'import sys,json; d=json.loads(sys.stdin.read(),strict=False); c=d["choices"][0];
assert c.get("finish_reason")=="stop", c.get("finish_reason");
assert c["message"]["content"].strip(), "empty content";
assert d.get("model")=="'"$MODEL"'", d.get("model");
assert d.get("usage",{}).get("completion_tokens",0)>0, "no tokens";
print("V1-OK:", repr(c["message"]["content"].strip()[:40]))'
```

Served tag (consumers discover this via `GET /api/tags`, never assume the bare `$MODEL` name):
```bash
curl -fs http://localhost:11434/api/tags | python3 -c 'import sys,json; d=json.load(sys.stdin); t=[m["name"] for m in d["models"] if m["name"].startswith("'"$MODEL"'")]; assert t, "not served"; print("SERVED_TAG:", t[0])'
```

> → Live State: `loads & generates` = observed `done_reason`/`eval_count`/final-answer from BOTH surfaces,
> result PASS/FAIL; `SERVED_TAG` = the discovered tag; `MODEL` = the model digest (`ollama list`/`ollama
> show`).

## Live State

```yaml
status:        not-created      # published template - run it to realize state
last_action:   authored — Ollama (local) binding
last_verified: —
resolved_inputs: { base_model: qwen3.6:27b-mlx, num_ctx: 65536, model: qwen3.6-coder-64k, kv_cache: none }
```

| key         | value (filled on apply) |
|-------------|--------------------------|
| ENDPOINT    | `http://localhost:11434` |
| MODEL       | `—` |
| SERVED_TAG  | `—` |

| ✔ check            | expected                                             | observed | result |
|--------------------|-------------------------------------------------------|----------|--------|
| daemon reachable   | `http://localhost:11434` answers                       | — | — |
| base present       | `BASE_MODEL` listed in `ollama list`                   | — | — |
| model present      | `MODEL` listed in `ollama list` / `GET /api/tags`      | — | — |
| num_ctx correct    | `ollama show $MODEL` reports the resolved `NUM_CTX`    | — | — |
| loads & generates  | `done_reason=="stop"` AND `eval_count>0` AND non-empty final answer | — | — |
| memory headroom    | weights floor loads & runs; resting free ≥40%; worst-case context characterized once (not re-run) | — | — |

> §3, §4, and §5 must each be verified live against YOUR OWN daemon before this ships `status: live`. §4
> does **not** treat `ollama ps` SIZE as a headroom signal (it reports weights-only, KV-invisible, at any
> context fill). Judge headroom from the OS: the recurring **floor** (model loads & runs) + resting
> `FREE_PCT` (**≥40%** is a reasonable guideline), plus **your own one-time worst-case-context
> characterization** (record the observed memory-pressure result for your host here — tight or
> comfortable) — never a per-run stress at the full ceiling.
> `OLLAMA_KV_CACHE_TYPE`/`OLLAMA_FLASH_ATTENTION` should stay unset unless a live before/after delta on
> your backend proves a knob moves resident memory — confirm via `launchctl getenv` that neither is set on
> your daemon before trusting `KV_CACHE=none`. §5 covers BOTH `/api/generate` and `/v1/chat/completions` —
> the shared-acceptance-contract members (well-formed completion, resolved model id, endpoint reachability)
> all need to hold before this binding is considered accepted.

## Update (idempotent reconcile)  🟡

> The whole plan is a pure function of `resolved_inputs` (`BASE_MODEL`, `NUM_CTX`, `MODEL`, `KV_CACHE`) —
> re-rendering the `Modelfile` and re-running `ollama create` with unchanged inputs reconciles to the same
> state; it never duplicates layers, storage, or a running load.

```bash
cd <project>   # the directory containing this plan's Modelfile — your project root
export BASE_MODEL="qwen3.6:27b-mlx" NUM_CTX="65536" MODEL="qwen3.6-coder-64k"
printf 'FROM %s\nPARAMETER num_ctx %s\n' "$BASE_MODEL" "$NUM_CTX" > Modelfile   # deterministic re-render — pure function of inputs, no hidden state
ollama create "$MODEL" -f Modelfile        # re-create: dedups existing layers ("using existing layer") — no re-download, no reload into memory
ollama show "$MODEL" --parameters | grep -F 'num_ctx' | grep -F "$NUM_CTX" && echo NUMCTX-OK   # re-assert num_ctx held
```

> Safe to re-run any number of times: the `Modelfile` render is deterministic (same `BASE_MODEL`/`NUM_CTX`
> ⇒ byte-identical file, per §3's `RENDER-OK` diff), and `ollama create` against an unchanged `Modelfile`
> dedupes rather than duplicates — same digest, same manifest, no new layer, no context load. A genuine
> update (not a no-op reconcile) means changing an input (`NUM_CTX`/`BASE_MODEL`) and re-running §3 in
> full; a **changed** `MODEL` digest is the signal something actually moved.
> → Live State: a no-op reconcile leaves `MODEL` digest and `NUMCTX-OK` unchanged; a genuine input change
> updates the `MODEL` digest, `SERVED_TAG`, and `last_verified`.

## Teardown (observe-first, resumable)  💥

> 💥 The only irreversible act in this plan. Never removes `$BASE_MODEL` — it is borrowed/upstream,
> provisioned once and shared by any derived model built from it; only the **derived** `$MODEL` this plan
> created is in scope.

```bash
ollama list | grep -qF "$MODEL" || { echo "$MODEL already absent — no-op"; exit 0; }   # observe-first: absent = nothing to do
ollama rm "$MODEL"                                                                     # 💥 removes ONLY the derived model
```

✔ verify (read-only, re-runnable):
```bash
ollama list | grep -qF "$MODEL" || echo "MODEL-GONE"
ollama list | grep -qF "$BASE_MODEL" && echo "BASE-UNTOUCHED"    # base model must still be present after teardown
```

> Resumable: a crash mid-teardown just leaves `$MODEL` present or absent — re-entry re-observes via
> `ollama list` and either no-ops (already gone) or retries the `rm`; there is no partial state to
> reconcile. → Live State: `status: gone`; clear the realized `MODEL` digest and `SERVED_TAG`.
>
> *This 💥 is authored but deliberately not something to run casually — only invoke it when you actually
> want to remove your derived model; otherwise leave the working model in place.*

## Deliberately not included

- **GPU / cloud hosting** — that is the sibling `inference.cloudflare.md` (Workers AI); this binding is
  local-only by design (no cloud account, no network egress, no per-token bill).
- **Model fine-tuning / training** — out of scope; this plan provisions an existing model with a context
  config (`Modelfile` + `num_ctx`), not a training pipeline.
- **A multi-model router** — one derived model per plan; routing across several served models is a
  separate concern (a proxy/Worker in front of `http://localhost:11434`), not this plan.
- **Endpoint auth** — the endpoint is localhost-bound (`127.0.0.1:11434`); exposing it beyond localhost
  would be a new Provisioning Input (and a new threat model), not a default this plan assumes.