# Ephemera — Model inference on Cloudflare (Workers AI + CLI)

> Self-executing Markdown. The **Cloudflare binding** of the *inference* intent — run AI models at the edge
> via the `AI` binding (or the REST API), optionally through **AI Gateway** for caching/rate-limit/analytics.
> The cloud is the source of truth; this file is intent + ledger + audit.

> **Provides / Requires**: **Provides** `inference-endpoint(WORKER_NAME @ URL)` and, for `MODEL_CLASS=embeddings`,
> the vectors that `vector.cloudflare.md` indexes. **Requires** nothing mandatory (Workers AI is an account
> capability — no resource to provision); optionally **Requires** a zone for `route`/`custom-domain`.

---

## 🤖 Director prompt

You are the Director. Execute this plan:
- Observe-before-act; verify each step before advancing
- Stop at every 🔴 GATE for human "go" — **model inference bills per request/token (denial-of-wallet)**
- Write realized values + verify results back into Live State
- Teardown removes the Worker + optional gateway this plan created; nothing else
- Use only the commands in this plan

> **Candor:** authored, **not yet dogfooded** — confirm model ids against the live Workers AI catalog
> (`wrangler ai models` / the dashboard) and exact flags against `--help`; the live CLI is the source of truth.

```
Legend  🟢 create · 🟡 config · 🔴 GATE (human go) · 💥 destructive (human go) · ⏳ wait · ✔ verify
```

## Intent

Serve model inference — text generation, embeddings, image generation, speech-to-text, translation — from a
Worker, with no GPU to provision and no model to host: Cloudflare runs the model, you call `env.AI.run(model,
inputs)`. Optionally front it with **AI Gateway** (a caching/rate-limiting/observability proxy). The unit of
work is a deployed Worker that exposes an inference route; **embeddings** output feeds a Vectorize index for RAG.

**Shared acceptance contract** (every inference binding — Workers AI / AWS Bedrock / GCP Vertex / Ollama (local) — must pass):
1. an inference call returns a well-formed result for the resolved `MODEL_CLASS`
   (text → non-empty completion · embeddings → a vector of the model's dimension · image → image bytes)
2. the call uses the **resolved model id** (determinism — same inputs ⇒ same model)
3. **(`GATEWAY=ai-gateway`)** the request appears in the gateway's analytics/log (cache + rate-limit active)

## Provisioning Inputs

| # | Question | Options (closed enum) | Default | Sets | Gates |
|---|----------|-----------------------|---------|------|-------|
| 1 | What kind of model? | `text-gen` / `embeddings` / `image-gen` / `asr` / `translation` | `text-gen` | `MODEL_CLASS` | the model id + call shape (§2) |
| 2 | How is it invoked? | `binding` / `rest` | `binding` | `MODE` | §2 (in-Worker `env.AI`) vs §2b (REST + token) |
| 3 | Front with AI Gateway? | `none` / `ai-gateway` | `none` | `GATEWAY` | §1 (gateway) + acceptance 3 |
| 4 | How is the endpoint exposed? | `workers-dev` / `route` / `custom-domain` | `workers-dev` | `ROUTING` | §4 (exposure) |
| 5 | Protect the endpoint? | `none` / `turnstile` / `secret-key` | `none` | `PROTECT` | §5 (auth — denial-of-wallet guard) |
| 6 | Environment | `dev` / `stg` / `uat` / `prod` | `dev` | `ENV` | every resource name (`*-${ENV}`) |

**Why `binding` + AI Gateway matters:** the `AI` binding is the native, token-free path (Workers AI is wired
by binding, like every CF data plane). `rest` is for callers *outside* a Worker (needs an API token).
**AI Gateway** is strongly recommended for anything public: it adds **response caching** (identical prompts
served free), **rate limiting** (the denial-of-wallet guard), and per-request analytics/logs — the missing
controls on a raw, billable inference endpoint.

```yaml
# → written into Live State once resolved
resolved_inputs:
  model_class: text-gen        # text-gen | embeddings | image-gen | asr | translation
  mode:        binding         # binding | rest
  gateway:     none            # none | ai-gateway
  routing:     workers-dev     # workers-dev | route | custom-domain
  protect:     none            # none | turnstile | secret-key
  env:         dev
  resolved_by: <human who confirmed>
  resolved_at: <timestamp>
```

**Resolved model id (a pure function of `MODEL_CLASS`)** — pin one; confirm it exists in the live catalog:

| MODEL_CLASS | Example model id (`@cf/…`) | Output shape |
|-------------|----------------------------|--------------|
| `text-gen`  | `@cf/meta/llama-3.1-8b-instruct` | `{ response: "…" }` |
| `embeddings`| `@cf/baai/bge-base-en-v1.5` | `{ data: [[…768 floats…]] }` |
| `image-gen` | `@cf/black-forest-labs/flux-1-schnell` | PNG bytes |
| `asr`       | `@cf/openai/whisper` | `{ text: "…" }` |
| `translation`| `@cf/meta/m2m100-1.2b` | `{ translated_text: "…" }` |

## Tags & provenance (binding asymmetry)

**Cloudflare has no resource-tag API**, and Workers AI has **no resource at all** (it's an account feature).
Provenance lives in the **Worker** that calls it: naming (`${SVC}-ai-${ENV}`) + `[vars]`
(`MANAGED_BY`/`SOURCE`/`PLAN_VERSION`/`ENVIRONMENT`) + `--message`. An AI Gateway (if created) carries
provenance by its name only.

## 0. Variables

```bash
export ENV="dev"
export MODEL_CLASS="text-gen" MODE="binding" GATEWAY="none" ROUTING="workers-dev" PROTECT="none"
export MODEL_ID="@cf/meta/llama-3.1-8b-instruct"        # the resolved id for MODEL_CLASS (table above)
export SVC="api"
export WORKER_NAME="${SVC}-ai-${ENV}"
export GATEWAY_NAME="${SVC}-gw-${ENV}"                   # only if GATEWAY=ai-gateway
export CONFIG="$PWD/${SVC}-ai.jsonc"
export ACCOUNT_ID="$(command wrangler whoami 2>/dev/null | grep -oE '[0-9a-f]{32}' | head -1)"
# NOTE: `command wrangler` bypasses the wrangler shell-fn so the ambient CLOUDFLARE_API_TOKEN is used.
```

## Dependency frontier

```
(GATEWAY=ai-gateway) AI Gateway (§1) ─┐
worker code (§2) ─────────────────────┼─> wrangler.jsonc (ai binding [+gateway]) ─> 🔴 deploy (§4) ─> ✔ acceptance
protect/turnstile secret (§5) ────────┘
```
Non-negotiable edges: **the gateway exists before the Worker references its id**; **the protect-secret is set
before deploy**; the model id must exist in the live catalog. Teardown reverses (Worker, then gateway).

## 1. AI Gateway  🟡  *(`GATEWAY=ai-gateway` — caching/rate-limit/analytics proxy)*

```bash
command wrangler ai-gateway create "$GATEWAY_NAME" 2>&1 || echo "create via dashboard if CLI lacks the subcommand (confirm live)"
command wrangler ai-gateway get "$GATEWAY_NAME" 2>&1      # ✔ verify
```
> → Live State: GATEWAY_NAME. The binding call then passes `{ gateway: { id: "$GATEWAY_NAME" } }`.

## 2. Worker code — call the model  🟢  *(`MODE=binding`)*

```bash
mkdir -p "$(dirname "$CONFIG")/src"
cat > "$(dirname "$CONFIG")/src/${SVC}-ai.js" <<'JS'
export default {
  async fetch(request, env) {
    const url = new URL(request.url);
    if (url.pathname === '/healthz') return Response.json({ ok: true });
    if (request.method === 'POST' && url.pathname === '/infer') {
      const { input } = await request.json();
      const opts = env.GATEWAY_ID ? { gateway: { id: env.GATEWAY_ID } } : {};
      // shape per MODEL_CLASS:
      //   text-gen:    { prompt: input }            -> { response }
      //   embeddings:  { text: [input] }            -> { data: [[...]] }
      //   translation: { text: input, target_lang } -> { translated_text }
      const out = await env.AI.run(env.MODEL_ID, { prompt: input }, opts);
      return Response.json(out);
    }
    return Response.json({ error: 'not found' }, { status: 404 });
  }
};
JS
test -f "$(dirname "$CONFIG")/src/${SVC}-ai.js" && echo "worker code written"   # ✔ verify
```
> **`MODE=rest` branch (§2b)** — for callers outside a Worker, no binding:
> ```bash
> curl -sS -X POST "https://api.cloudflare.com/client/v4/accounts/$ACCOUNT_ID/ai/run/$MODEL_ID" \
>   -H "Authorization: Bearer $CLOUDFLARE_API_TOKEN" -H 'Content-Type: application/json' -d '{"prompt":"ping"}'
> ```

## 3. Bindings — `${SVC}-ai.jsonc`  🟡

```jsonc
{
  "name": "api-ai-dev",
  "main": "src/api-ai.js",
  "compatibility_date": "2025-04-01",
  "vars": {
    "MANAGED_BY": "ephemera", "SOURCE": "inference.cloudflare.md", "PLAN_VERSION": "2026-06-27", "ENVIRONMENT": "dev",
    "MODEL_ID": "@cf/meta/llama-3.1-8b-instruct",
    "GATEWAY_ID": ""                       // set to GATEWAY_NAME when GATEWAY=ai-gateway
  },
  "ai": { "binding": "AI" }
}
```
```bash
command wrangler deploy --config "$CONFIG" --dry-run 2>&1 | tail -5     # ✔ config valid
```

## 4. Deploy  🔴 (outward-facing — billable inference)

> 🔴 Human go — a public inference endpoint bills per request/token. Confirm `PROTECT` is set for anything
> internet-facing (§5), print the URL, then deploy. Exposure via `routes`/`workers_dev` (see `service.cloudflare.md` §4).

```bash
command wrangler deploy --config "$CONFIG" --message "Source=inference.cloudflare.md PlanVersion=$PLAN_VERSION"
# → Live State: URL; status: live
```

## 5. Protect the endpoint  🟡  *(`PROTECT=turnstile` | `secret-key` — denial-of-wallet guard)*

> A naked, billable inference URL is a wallet-drain target. `secret-key`: require an `Authorization` header
> the handler checks against `wrangler secret put INFER_KEY`. `turnstile`: human-gate public callers (see
> `service.cloudflare.md` §6). AI Gateway rate-limiting (§1) is the complementary control.

```bash
[ "$PROTECT" = secret-key ] && printf '%s' "$INFER_KEY" | command wrangler secret put INFER_KEY --name "$WORKER_NAME"
command wrangler secret list --name "$WORKER_NAME"     # ✔ key present (value never shown)
```

## Acceptance verify  ✔  *(the shared contract)*

```bash
URL="https://api-ai-dev.<subdomain>.workers.dev"   # realized in §4
RESP="$(curl -sS -X POST "$URL/infer" -H 'Content-Type: application/json' -d '{"input":"say hello"}')"
echo "$RESP"
# 1: well-formed for MODEL_CLASS — text-gen: non-empty .response ; embeddings: .data[0] length == model dim
case "$MODEL_CLASS" in
  text-gen)   echo "$RESP" | grep -q '"response"' && echo "1: text OK" ;;
  embeddings) echo "$RESP" | python3 -c 'import sys,json;d=json.load(sys.stdin)["data"][0];print("1: dim",len(d))' ;;
esac
```
> → write Live State: status: live; record the model id used (test 2) + dims; fill verify rows.

## Live State

```yaml
status:        not-created      # published template - run it to realize state
last_action:   authored — Workers AI inference (model-class/mode/gateway/protect knobs)
last_verified: —
resolved_inputs: { model_class: text-gen, mode: binding, gateway: none, routing: workers-dev, protect: none, env: dev }
```

| key          | value (filled on apply) |
|--------------|-------------------------|
| WORKER_NAME  | `${SVC}-ai-${ENV}` |
| MODEL_ID     | `@cf/meta/llama-3.1-8b-instruct` |
| URL          | `—` |
| GATEWAY_NAME | `—` (only if `GATEWAY=ai-gateway`) |

| ✔ check                     | expected                                      | observed | result |
|-----------------------------|-----------------------------------------------|----------|--------|
| inference returns well-formed| per `MODEL_CLASS` (response / vector / bytes)  | —        | — |
| uses resolved model id      | request logged with `$MODEL_ID`                | —        | — |
| gateway active (gateway)    | request appears in gateway analytics           | —        | — |
| endpoint protected (protect)| unauthenticated call `403` (negative)          | —        | — |

> Assert the **negative** when `PROTECT != none`: an unauthenticated `POST /infer` must `403` — an open
> billable endpoint is the failure this plan guards against.

## Update (idempotent reconcile)

- Swap models → edit `MODEL_ID` in §3 `vars`, redeploy (instant; `wrangler rollback` reverts).
- Add/enable the gateway → create it (§1), set `GATEWAY_ID`, redeploy.
- Rotate the protect key → `wrangler secret put INFER_KEY` again.

## Teardown (observe-first, resumable)  💥

> 💥 Human go. Delete the Worker (stops billing on this endpoint), then the gateway if this plan made one.

```bash
command wrangler delete --config "$CONFIG"
[ "$GATEWAY" = ai-gateway ] && command wrangler ai-gateway delete "$GATEWAY_NAME" 2>/dev/null || true
```
```bash
command wrangler deployments list --name "$WORKER_NAME" 2>&1 | grep -qi "not found\|no deployments" && echo "inference gone"  # ✔
```
> → Live State: status: gone; clear realized ids.

---

## Portability ledger — same intent, three bindings

| | AWS (`inference.aws.md`, Bedrock) | Cloudflare (`inference.cloudflare.md`, Workers AI) | GCP (`inference.gcp.md`, Vertex) |
|---|---|---|---|
| Provisioning | model access request + IAM | **nothing** — account feature, `AI` binding | model garden + SA + endpoint |
| Invocation | SDK + IAM creds | `env.AI.run(id, inputs)` — token-free in-Worker | SDK + SA |
| Caching/rate-limit | build it / API GW | **AI Gateway** (built-in) | build it / Apigee |
| Embeddings → vector DB | Bedrock + OpenSearch/pgvector | Workers AI + **Vectorize** (`vector.cloudflare.md`) | Vertex + Vector Search |
| Cold start / GPU | managed, regional | edge, no GPU mgmt | managed, regional |
| Billing guard | IAM + budget alarms | AI Gateway rate-limit + `PROTECT` | quotas + IAM |
| Tags | resource tags (some) | **none** — naming + `[vars]` | labels |

## Deliberately not included

- **Fine-tuning / LoRA hosting** — Workers AI supports some LoRA adapters; uploading/managing them is a
  follow-on, not steady-state inference.
- **Streaming responses (SSE)** — `env.AI.run(..., { stream: true })` returns an SSE stream; the handler
  shape differs (return the stream). A knob, not wired here, to keep the contract a simple round-trip.
- **Batch/async inference** — large jobs belong behind `task-runner.cloudflare.md` (Queue → consumer calls AI).
- **The full model catalog** — only one id per class is pinned; the live catalog (`wrangler ai models`) is truth.