# Ephemera — Self-hosted CI runner on AWS (CodeBuild runner project + aws CLI)

> Self-executing Markdown. The **AWS binding** of the *ci-runner* intent — ephemeral, per-job CI runners
> for **GitHub Actions or GitLab CI/CD**, hosted as an AWS CodeBuild *runner project*: a queued workflow
> job fires a webhook, CodeBuild spins up a one-shot runner container, executes exactly that job, and
> terminates. **Zero idle cost** — no always-on runner EC2/pod to babysit; you pay per build-minute only
> while a job runs. The cloud is the source of truth; this file is intent + write-back ledger + audit.

> **Provides / Requires**: **Requires** `source-connection(CONN_NAME, ConnectionArn)` from
> [`source-connection.aws.md`](./source-connection.aws.md), gated on `ConnectionStatus == AVAILABLE`
> (a `PENDING` connection means the human OAuth handshake hasn't happened — stop and run that plan's §2).
> **Requires** (only when `NETWORK=vpc`) `vpc()` + **private subnets with a NAT route** from
> [`network.aws.md`](./network.aws.md) (`WANT_PRIVATE=yes`). **Provides** `ci-runner(PROJECT_NAME)` —
> consumed *from the git side* by a label in the workflow YAML (§7), not by another plan's CLI.

---

## 🤖 Director prompt

Observe before acting; verify each step before advancing; stop at 🔴/💥 for human go; write realized values
back into Live State. Defining traits of this plan: **(1)** the project's buildspec is a placeholder — for
runner projects CodeBuild **ignores the buildspec** (unless the job label carries `buildspec-override:true`)
and injects its own runner bootstrap; the *workflow YAML in the repo* is the real job definition. **(2)** The
workflow's label **must embed the exact project name** (`codebuild-${PROJECT_NAME}-…`) — a mismatch does not
error, the job **hangs forever** waiting for a runner (§7). **(3)** `create-webhook` on a project that already
has one fails `ResourceAlreadyExistsException` — observe `projects[0].webhook` first, never create blind.
**(4)** Everything here is $0 while idle; the security-consequential moment is §5 (the webhook), which lets
anyone who can push a workflow to `REPO_URL` execute code **under this plan's IAM role** in your account.

> **Candor: PARTIALLY DOGFOODED LIVE 2026-07-12** (us-west-2, ~$0, torn down + absence-verified) on the
> **connection-skipped path** — the handshake was deliberately not run, so the connection stayed `PENDING`.
> Proven live: §1's negative gate (contract 3 — clean STOP on `PENDING`, exit 1, correct remedy), §2 under a
> credential broker (ambient session creds failed IAM `InvalidClientTokenId` exactly as canon says; role
> created via no-session), §4 create + observe-reuse determinism (contract 4) + `"buildspec":""` accepted +
> the deferred-validation finding (see §4 note), §5's exact failure mode on `PENDING`, and the full teardown
> chain with absence checks. One authored bug found + fixed live: the §4 environment-convergence check
> compared text-mode `False` against `false` (see the in-block comment). **Still unrun:** the `AVAILABLE`
> path (webhook success → contract 2), the end-to-end job (contract 5), the gitlab branch, the vpc branch.
> CLI shapes verified against the CodeBuild runner tutorials + API reference 2026-07-11; the migrate-source
> (a realized EKS-Fargate + CodeBuild-runner Terragrunt stack) proved the architecture live — its two
> `null_resource` workarounds (terraform-provider-aws#38572 can't set `source.auth`; webhook create isn't
> idempotent) are **native steps §4/§5 here**.

```
Legend  🟢 create · 🟡 config · 🔴 GATE (human go) · 💥 destructive (human go) · ⏳ wait · ✔ verify
```

## What you need, and why  *(read if CI runners are new to you)*

A hosted runner (GitHub's `ubuntu-latest`, GitLab's shared runners) executes your CI inside *their* cloud —
no AWS credentials unless you export long-lived secrets into the git provider. A **self-hosted runner in
CodeBuild** flips that: the job executes inside **your** AWS account, inheriting the project's IAM role —
no `AWS_SECRET_ACCESS_KEY` in repo secrets, and the job can reach private VPC resources (an EKS/ECS API,
an RDS endpoint) that a hosted runner never could. Unlike a classic self-hosted runner (an ECS service or
EC2 box polling 24/7), a CodeBuild runner project is **event-driven and ephemeral** — idle = $0.

- **`aws` CLI v2, authenticated** (`aws sts get-caller-identity` succeeds).
- **IAM steps under a credential broker** (aws-vault / SSO): §2's role create and teardown's role delete need
  `--no-session` (EPHEMERA.md gotcha); everything else runs on session creds.
- **An applied [`source-connection.aws.md`](./source-connection.aws.md)** with status `AVAILABLE`, same
  region, same provider — discovered in §1, never created here.
- **A repo you administer** at `REPO_URL` (the webhook lands in *its* settings; you add the workflow YAML).
- **(`NETWORK=vpc` only)** an applied [`network.aws.md`](./network.aws.md) run with `WANT_PRIVATE=yes` —
  CodeBuild VPC ENIs get **no public IP**, so they only work in **private subnets with a NAT route**
  (a public subnet silently gives you a runner that can't reach the internet).

## Intent

Stand up an **on-demand, per-job CI runner inside our AWS account** for a git repo's pipeline — GitHub
Actions or GitLab CI/CD via the `PROVIDER` knob — so CI jobs run with IAM-role credentials instead of
exported secrets, can reach VPC-private resources when needed, and cost nothing between jobs. The *intent*
(a self-hosted CI runner) is provider-portable in the **git** dimension (GitHub / GitLab — one plan, one
knob); in the **cloud** dimension it is AWS-shaped: Cloudflare folds CI into Workers Builds (no runner
resource you could bind to an arbitrary repo's workflow), so — like `lambda-layer.aws.md` and
`source-connection.aws.md` — there is no CF sibling binding, and that asymmetry is recorded, not hidden.

**Acceptance contract** (AWS-shaped intent; no sibling binding — see the portability note):
1. `batch-get-projects` returns our project with `source.type == ${SOURCE_TYPE}`, `source.location ==
   ${REPO_URL}`, and `source.auth.type == CODECONNECTIONS` bound to **our** connection ARN.
2. The webhook exists on the project with an `EVENT / WORKFLOW_JOB_QUEUED` filter group (the runner trigger).
3. **Negative (the Requires gate is real).** With the connection **not** `AVAILABLE`, §1 refuses to proceed —
   the runner is never wired to an unauthorized connection.
4. **Determinism.** Re-running `apply` reuses the existing project + webhook (observe-first) — no duplicate
   projects, no `ResourceAlreadyExistsException` crash.
5. **End-to-end (dogfood, human-gated).** A pushed workflow carrying the §7 label produces a build in
   `list-builds-for-project` that reaches `SUCCEEDED` — a real job executed in our account.

## Provisioning Inputs

| # | Question | Options (closed enum) | Default | Sets | Gates |
|---|----------|-----------------------|---------|------|-------|
| 1 | Git provider | `github` / `gitlab` | `github` | `PROVIDER` | source type, §7 label syntax, GitLab scope gotcha |
| 2 | Repo to serve | text — full HTTPS URL | — (required) | `REPO_URL` | §4 source location, §5 webhook target |
| 3 | Runner project name | text — `[A-Za-z0-9][A-Za-z0-9_-]*`, ≤ 40 chars | `${PROVIDER}-runner-${ENV}` | `PROJECT_NAME` | the §7 label **verbatim** |
| 4 | Compute size | `small` / `medium` / `large` | `small` | `COMPUTE` | build $/min (overridable per-job via label) |
| 5 | Docker-in-Docker? | `no` / `yes` | `no` | `DIND` | privileged mode (needed to `docker build` inside a job) |
| 6 | Network | `public` / `vpc` | `public` | `NETWORK` | §1 NAT check, §2 ENI policy, §3 SG, §4 `--<VPC_ID>onfig` |
| 7 | Environment | `dev` / `stg` / `uat` / `prod` | `dev` | `ENV` | name suffix + `Environment` tag |

```yaml
# → written into Live State once resolved (the deterministic input to every step below)
resolved_inputs:
  provider:     github          # github | gitlab
  repo_url:     https://github.com/<owner>/<repo>
  project_name: github-runner-dev
  compute:      small
  dind:         "no"
  network:      public
  env:          dev
  resolved_by:  <human who confirmed>
  resolved_at:  <timestamp>
```

> **Pick `NETWORK=vpc` only when jobs must reach VPC-private resources** (a private EKS endpoint, RDS, an
> internal ALB). It adds the ENI IAM policy, a security group, and a hard dependency on NAT-routed private
> subnets. `public` runners still reach all public AWS APIs and the git provider — most fleets start there.
> **Pick `DIND=yes` only when jobs run `docker build`** — privileged containers are a real grant.

## TAGS — provenance & cost tags

CodeBuild projects tag **on create** with the **lowercase** `key=,value=` list (`tags_lc` — same shape as
ECS); the IAM role takes `Key=,Value=` (`tags_kv`); the VPC-branch SG tags atomically via
`--tag-specifications` (`tags_spec`). Renderers inline in §0 (canonical: `scripts/tags.sh`).

## Live State

```yaml
status:        not-created      # published template - run it to realize state
last_action:   partial dogfood 2026-07-12 — connection-skipped path (PENDING): §1 STOP proven, §2/§4 clean, §5 failed as designed, teardown absence-verified; env-check bug fixed live; gate override for §2-§5 approved by the human (dogfood)
last_verified: 2026-07-12 (see verify table)
resolved_inputs: { provider: github, repo_url: <dogfood: a public repo of the operator>, project_name: github-runner-dev, compute: small, dind: "no", network: public, env: dev }
```

| key           | value (filled on apply) |
|---------------|-------------------------|
| AWS_REGION    | `—` |
| CONN_ARN      | `—` (discovered from source-connection.aws.md, `AVAILABLE`) |
| ROLE_ARN      | `—` (`${PROJECT_NAME}-role`) |
| PROJECT_ARN   | `—` |
| WEBHOOK_URL   | `—` (`payloadUrl` from create-webhook) |
| SG_ID         | `—` (vpc branch only) |
| VPC_ID / SUBNET_IDS | `—` (vpc branch only, discovered from SSM) |

| ✔ check                                   | expected                                                      | observed (2026-07-12 dogfood)                                  | result |
|-------------------------------------------|---------------------------------------------------------------|----------------------------------------------------------------|--------|
| connection AVAILABLE before wiring (neg.) | §1 exits non-zero on `PENDING`/absent                          | STOP on `PENDING`, exit 1, remedy printed                      | **PASS (proven live)** |
| project exists, source+auth correct       | type/location/auth.resource match our inputs                   | GITHUB / repo URL / our conn ARN — all matched                 | **PASS (proven live)** |
| webhook armed                             | filter group `EVENT=WORKFLOW_JOB_QUEUED` present               | not run — `PENDING` conn fails create-webhook (as designed)    | pending-dogfood (needs AVAILABLE) |
| re-apply is deterministic                 | same project ARN, no duplicate / no already-exists crash      | §4 re-run → "reusing project", same ARN                        | **PASS (proven live)** |
| tags present                              | project carries `ManagedBy=ephemera`                           | teardown ownership check read `ephemera` off the live project  | **PASS (proven live)** |
| end-to-end job ran (dogfood)              | a labeled workflow job → build `SUCCEEDED`                     | not run — needs handshake + a workflow push                    | pending-dogfood |

## 0. Variables

```bash
set -euo pipefail
export AWS_REGION="${AWS_REGION:-us-west-2}"
export ENV="${ENV:-dev}"
export PROVIDER="${PROVIDER:-github}"                          # github | gitlab
export REPO_URL="${REPO_URL:?set REPO_URL (full https URL of the repo the runner serves)}"
export PROJECT_NAME="${PROJECT_NAME:-${PROVIDER}-runner-${ENV}}"
export COMPUTE="${COMPUTE:-small}"                             # small | medium | large
export DIND="${DIND:-no}"                                      # yes => privileged containers
export NETWORK="${NETWORK:-public}"                            # public | vpc
ROLE_NAME="${PROJECT_NAME}-role"
IMAGE="${IMAGE:-aws/codebuild/amazonlinux-x86_64-standard:5.0}" # curated runner image; per-job label can override
LOG_GROUP="/aws/codebuild/${PROJECT_NAME}"

case "$PROVIDER" in
  github) SOURCE_TYPE="GITHUB"; PROVIDER_TYPE="GitHub";;
  gitlab) SOURCE_TYPE="GITLAB"; PROVIDER_TYPE="GitLab";;
  *) echo "PROVIDER must be github|gitlab (Bitbucket/Buildkite runners are named omissions)"; exit 1;;
esac
case "$COMPUTE" in
  small)  COMPUTE_TYPE="BUILD_GENERAL1_SMALL";;
  medium) COMPUTE_TYPE="BUILD_GENERAL1_MEDIUM";;
  large)  COMPUTE_TYPE="BUILD_GENERAL1_LARGE";;
  *) echo "COMPUTE must be small|medium|large"; exit 1;;
esac
PRIVILEGED=false; [ "$DIND" = "yes" ] && PRIVILEGED=true
printf '%s' "$PROJECT_NAME" | grep -Eq '^[A-Za-z0-9][A-Za-z0-9_-]{0,39}$' || { echo "PROJECT_NAME must be [A-Za-z0-9][A-Za-z0-9_-]*, ≤40 chars"; exit 1; }
case "$PROVIDER:$REPO_URL" in   # a mismatched host creates a project whose jobs hang, never error
  github:https://github.com/*|gitlab:https://gitlab.com/*) : ;;
  *) echo "REPO_URL host does not match PROVIDER=${PROVIDER}"; exit 1;;
esac
# CONN_NAME must match the applied source-connection.aws.md (its Provisioning Input #2)
export CONN_NAME="${CONN_NAME:-${PROVIDER}-connection}"
printf '%s' "$CONN_NAME" | grep -Eq '^[A-Za-z0-9_-]{1,32}$' || { echo "CONN_NAME must be [A-Za-z0-9_-], ≤32 chars"; exit 1; }

# discover OUR connection: name match is not enough (AWS allows duplicate names) — require ManagedBy=ephemera,
# mirroring source-connection.aws.md's own reuse discipline. Echoes the ARN; empty output = not found.
discover_connection() {
  local arn own
  for arn in $(aws codeconnections list-connections --region "$AWS_REGION" \
      --provider-type-filter "$PROVIDER_TYPE" \
      --query "Connections[?ConnectionName=='${CONN_NAME}'].ConnectionArn" --output text 2>/dev/null); do
    [ -z "$arn" ] || [ "$arn" = "None" ] && continue
    own="$(aws codeconnections list-tags-for-resource --region "$AWS_REGION" --resource-arn "$arn" \
      --query "Tags[?Key=='ManagedBy'].Value | [0]" --output text 2>/dev/null || true)"
    [ "$own" = "ephemera" ] && { printf '%s' "$arn"; return 0; }
  done
  return 0
}

# ── TAGS — resolved once (canonical: scripts/tags.sh) ──
PLAN_SOURCE="ci-runner.aws.md"
PLAN_VERSION="2026-07-11"
TAG_COST_CENTER="${TAG_COST_CENTER:-}"; TAG_OWNER="${TAG_OWNER:-}"
TAGS="$(printf '%s\n' "ManagedBy=ephemera" "Source=${PLAN_SOURCE}" "PlanVersion=${PLAN_VERSION}" \
  "CostCenter=${TAG_COST_CENTER}" "Owner=${TAG_OWNER}" "Environment=${ENV}")"
_tags_list() { printf '%s\n' "$TAGS" "$@" | awk '
  { eq=index($0,"="); if(eq==0) next; k=substr($0,1,eq-1); v=substr($0,eq+1); if(v=="") next;
    val[k]=v; if(!(k in seen)){ order[++n]=k; seen[k]=1 } }
  END { for(i=1;i<=n;i++) print order[i]"="val[order[i]] }'; }
tags_kv()    { _tags_list "$@" | while IFS='=' read -r k v; do printf 'Key=%s,Value=%s ' "$k" "$v"; done; }
tags_lc()    { _tags_list "$@" | while IFS='=' read -r k v; do printf 'key=%s,value=%s ' "$k" "$v"; done; }
tags_brace() { _tags_list "$@" | while IFS='=' read -r k v; do printf '{Key=%s,Value=%s},' "$k" "$v"; done | sed 's/,$//'; }
tags_spec()  { _rt=$1; shift; printf 'ResourceType=%s,Tags=[%s]' "$_rt" "$(tags_brace "$@")"; }
```

## Dependency frontier

```
source-connection.aws.md ──> §1 Requires ✔ (AVAILABLE, or STOP) ─┐
network.aws.md (WANT_PRIVATE=yes) ──> §1 vpc+NAT ✔ (vpc only) ───┤
                                                                 ├─> §2 IAM role 🟢 ─⏳→ §4 project 🟢 ─> §5 webhook 🔴🟢 ─> §6 ✔ ─> §7 usage
                                                (vpc only) §3 SG 🟢 ┘
```
Non-negotiable edges: **the connection must be `AVAILABLE` before the project binds it** (an un-handshaken
connection makes §4 fail or §5 mis-auth); **the role must exist before `create-project` names it** — and IAM
propagation makes that edge eventually-consistent (⏳ retry in §4, the `create-function` lesson); **the
webhook comes last** — it is the step that arms external job dispatch, so everything it exposes must already
be verified. Teardown reverses the arrows (webhook → project → role → SG).

## 1. Requires check — the upstream frontier  ✔  *(creates nothing; may STOP)*

```bash
# 1a — the connection (source-connection.aws.md's Provides): discover by name + ManagedBy tag, gate on AVAILABLE
CONN_ARN="$(discover_connection)"
[ -n "$CONN_ARN" ] || { echo "STOP: no ephemera-managed connection named ${CONN_NAME} — apply source-connection.aws.md first"; exit 1; }
CONN_STATUS="$(aws codeconnections get-connection --region "$AWS_REGION" --connection-arn "$CONN_ARN" \
  --query 'Connection.ConnectionStatus' --output text)"
echo "connection: $CONN_ARN ($CONN_STATUS)"
[ "$CONN_STATUS" = "AVAILABLE" ] || { echo "STOP: connection is ${CONN_STATUS}, not AVAILABLE — complete source-connection.aws.md §2 (human OAuth handshake)"; exit 1; }
```
```bash
# 1b — (NETWORK=vpc only) private subnets exist in SSM AND route to a NAT — CodeBuild ENIs have no public IP
if [ "$NETWORK" = "vpc" ]; then
  VPC_ID="$(aws ssm get-parameter --region "$AWS_REGION" --name /network/vpc --query Parameter.Value --output text 2>/dev/null || true)"
  [ -n "$VPC_ID" ] || { echo "STOP: /network/vpc not in SSM — apply network.aws.md first"; exit 1; }
  SUBNET_IDS="$( { for az in 1a 1b; do aws ssm get-parameter --region "$AWS_REGION" \
    --name "/network/subnet/private/${az}" --query Parameter.Value --output text 2>/dev/null || true; done; } | grep -v '^$' | paste -sd, - || true)"
  [ -n "$SUBNET_IDS" ] || { echo "STOP: no private subnets in SSM — re-run network.aws.md with WANT_PRIVATE=yes"; exit 1; }
  # NAT check reads the subnet's EXPLICIT route-table association (network.aws.md always associates explicitly;
  # a subnet riding the VPC's main table would false-STOP here — that would itself be drift worth surfacing)
  FIRST_SUBNET="${SUBNET_IDS%%,*}"
  RT_NAT="$(aws ec2 describe-route-tables --region "$AWS_REGION" \
    --filters "Name=association.subnet-id,Values=${FIRST_SUBNET}" \
    --query 'RouteTables[0].Routes[?NatGatewayId!=null] | length(@)' --output text 2>/dev/null || echo 0)"
  [ "$RT_NAT" != "0" ] && [ "$RT_NAT" != "None" ] || { echo "STOP: subnet ${FIRST_SUBNET} has no NAT route — a VPC runner there can never reach ${PROVIDER}"; exit 1; }
  echo "vpc ok: $VPC_ID subnets $SUBNET_IDS (NAT-routed)"
fi
```
> → Live State: `CONN_ARN` (+ `VPC_ID`/`SUBNET_IDS` on the vpc branch). This is contract 3's negative: an
> unavailable connection stops the plan here, before anything is created.

## 2. Service role  🟢  *(IAM — `--no-session` under a credential broker)*

The role is what the runner's jobs **become**: `logs` to write build logs, the connection to clone source.
Deliberately **nothing else** — a job that needs ECR/EKS/S3 gets those via an *explicit* extra policy at
compose time (see Composition), never a pre-baked `*`.

```bash
ACCOUNT_ID="$(aws sts get-caller-identity --query Account --output text)"
CONN_ARN="${CONN_ARN:-$(discover_connection)}"   # fresh-shell resume: re-derive (read-only)
[ -n "$CONN_ARN" ] || { echo "STOP: connection not found — run §1"; exit 1; }
if aws iam get-role --role-name "$ROLE_NAME" >/dev/null 2>&1; then
  # adopt a same-named role ONLY if it is ours — never overwrite policies on a borrowed role
  ROLE_OWN="$(aws iam list-role-tags --role-name "$ROLE_NAME" --query "Tags[?Key=='ManagedBy'].Value | [0]" --output text 2>/dev/null || true)"
  [ "$ROLE_OWN" = "ephemera" ] || { echo "STOP: role ${ROLE_NAME} exists but lacks ManagedBy=ephemera — name collision, pick another PROJECT_NAME"; exit 1; }
else
  aws iam create-role --role-name "$ROLE_NAME" --tags $(tags_kv) \
    --assume-role-policy-document '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"Service":"codebuild.amazonaws.com"},"Action":"sts:AssumeRole"}]}'
fi
# connection ARNs come in two spellings (codeconnections / legacy codestar-connections) — allow both actions
CONN_ARN_ALT="$(printf '%s' "$CONN_ARN" | sed 's/:codeconnections:/:codestar-connections:/')"
aws iam put-role-policy --role-name "$ROLE_NAME" --policy-name runner-core --policy-document "{
  \"Version\":\"2012-10-17\",\"Statement\":[
    {\"Effect\":\"Allow\",\"Action\":[\"logs:CreateLogGroup\",\"logs:CreateLogStream\",\"logs:PutLogEvents\"],
     \"Resource\":\"arn:aws:logs:${AWS_REGION}:${ACCOUNT_ID}:log-group:${LOG_GROUP}*\"},
    {\"Effect\":\"Allow\",\"Action\":[\"codeconnections:GetConnection\",\"codeconnections:GetConnectionToken\",\"codeconnections:UseConnection\",
                                      \"codestar-connections:GetConnection\",\"codestar-connections:GetConnectionToken\",\"codestar-connections:UseConnection\"],
     \"Resource\":[\"${CONN_ARN}\",\"${CONN_ARN_ALT}\"]}
  ]}"
if [ "$NETWORK" = "vpc" ]; then
  aws iam put-role-policy --role-name "$ROLE_NAME" --policy-name runner-vpc --policy-document "{
    \"Version\":\"2012-10-17\",\"Statement\":[
      {\"Effect\":\"Allow\",\"Action\":[\"ec2:CreateNetworkInterface\",\"ec2:DescribeDhcpOptions\",\"ec2:DescribeNetworkInterfaces\",
                                        \"ec2:DeleteNetworkInterface\",\"ec2:DescribeSubnets\",\"ec2:DescribeSecurityGroups\",\"ec2:DescribeVpcs\"],
       \"Resource\":\"*\"},
      {\"Effect\":\"Allow\",\"Action\":\"ec2:CreateNetworkInterfacePermission\",
       \"Resource\":\"arn:aws:ec2:${AWS_REGION}:${ACCOUNT_ID}:network-interface/*\",
       \"Condition\":{\"StringEquals\":{\"ec2:AuthorizedService\":\"codebuild.amazonaws.com\"}}}
    ]}"
fi
ROLE_ARN="$(aws iam get-role --role-name "$ROLE_NAME" --query Role.Arn --output text)"
```
```bash
# ✔ role exists, trusts codebuild, carries both policies (vpc branch: two)
aws iam get-role --role-name "$ROLE_NAME" --query Role.AssumeRolePolicyDocument --output json | grep -q codebuild.amazonaws.com \
  && echo "role ok: $ROLE_ARN" || { echo "role trust wrong"; exit 1; }
aws iam list-role-policies --role-name "$ROLE_NAME" --output text
```
> → Live State: `ROLE_ARN`.

## 3. Security group  🟢  *(vpc branch only — egress-only, no ingress)*

```bash
if [ "$NETWORK" = "vpc" ]; then
  SG_ID="$(aws ec2 describe-security-groups --region "$AWS_REGION" \
    --filters "Name=group-name,Values=${PROJECT_NAME}-sg" "Name=vpc-id,Values=${VPC_ID}" \
    --query 'SecurityGroups[0].GroupId' --output text 2>/dev/null || echo None)"
  if [ "$SG_ID" = "None" ] || [ -z "$SG_ID" ]; then
    SG_ID="$(aws ec2 create-security-group --region "$AWS_REGION" --vpc-id "$VPC_ID" \
      --group-name "${PROJECT_NAME}-sg" --description "ci-runner ${PROJECT_NAME} egress" \
      --tag-specifications "$(tags_spec security-group "Name=${PROJECT_NAME}-sg")" \
      --query GroupId --output text)"
  else
    SG_OWN="$(aws ec2 describe-security-groups --region "$AWS_REGION" --group-ids "$SG_ID" \
      --query "SecurityGroups[0].Tags[?Key=='ManagedBy'].Value | [0]" --output text 2>/dev/null || true)"
    [ "$SG_OWN" = "ephemera" ] || { echo "STOP: SG ${PROJECT_NAME}-sg exists but lacks ManagedBy=ephemera — name collision"; exit 1; }
  fi
  echo "sg: $SG_ID"   # default egress-all retained (runner must reach the git provider + AWS APIs); no ingress
fi
```
> → Live State: `SG_ID`. ✔ `describe-security-groups --group-ids "$SG_ID"` shows zero ingress rules.

## 4. The runner project  🟢  *(observe-first; ⏳ IAM-propagation retry)*

The `source.auth` block (type `CODECONNECTIONS` + our ARN) rides **inside `create-project`** — this is the
step the Terraform provider couldn't express (issue #38572) and the migrate-source patched with a
`null_resource` + `aws codebuild update-project`. Here it's just the create. The buildspec is the **empty
string** (the docs' own CLI shape for runner projects — CodeBuild ignores it anyway, Director trait 1;
✔ proven live 2026-07-12: `create-project` accepts `"buildspec":""`).

> ⚠ **Proven live: `create-project` does NOT validate connection status.** It happily binds `auth` to a
> `PENDING` connection and returns success — AWS defers the check to `create-webhook` (§5). So **§1's gate
> is the only thing standing between an apply and a silently half-wired runner**; never skip it on the
> grounds that "AWS will catch it" — it won't until two steps later.

```bash
EXISTING="$(aws codebuild batch-get-projects --region "$AWS_REGION" --names "$PROJECT_NAME" \
  --query 'projects[0].arn' --output text 2>/dev/null || echo None)"
if [ "$EXISTING" != "None" ] && [ -n "$EXISTING" ]; then
  PROJECT_ARN="$EXISTING"; echo "reusing project: $PROJECT_ARN (reconcile via Update)"
else
  VPC_CONFIG=""
  [ "$NETWORK" = "vpc" ] && VPC_CONFIG="{\"vpcId\":\"${VPC_ID}\",\"subnets\":[\"$(printf '%s' "$SUBNET_IDS" | sed 's/,/","/g')\"],\"securityGroupIds\":[\"${SG_ID}\"]}"
  # ⏳ a just-created role may not be assumable yet — retry on the propagation error (up to ~50 s)
  PROJECT_ARN=""
  for i in 1 2 3 4 5; do
    if PROJECT_ARN="$(aws codebuild create-project --region "$AWS_REGION" \
      --name "$PROJECT_NAME" \
      --source "{\"type\":\"${SOURCE_TYPE}\",\"location\":\"${REPO_URL}\",\"gitCloneDepth\":1,\"buildspec\":\"\",\"auth\":{\"type\":\"CODECONNECTIONS\",\"resource\":\"${CONN_ARN}\"}}" \
      --artifacts type=NO_ARTIFACTS \
      --environment "type=LINUX_CONTAINER,image=${IMAGE},computeType=${COMPUTE_TYPE},privilegedMode=${PRIVILEGED}" \
      --service-role "$ROLE_ARN" \
      --timeout-in-minutes 60 --queued-timeout-in-minutes 480 \
      --logs-config "cloudWatchLogs={status=ENABLED,groupName=${LOG_GROUP}}" \
      --tags $(tags_lc) ${VPC_CONFIG:+--<VPC_ID>onfig "$VPC_CONFIG"} \
      --query 'project.arn' --output text 2>/tmp/create-project.err)"; then
      echo "created: $PROJECT_ARN"; break
    fi
    grep -qi 'sts:AssumeRole' /tmp/create-project.err \
      && { echo "IAM still propagating (try $i) — waiting 10s"; sleep 10; } \
      || { cat /tmp/create-project.err; exit 1; }
  done
  [ -n "$PROJECT_ARN" ] || { echo "create-project failed after retries"; cat /tmp/create-project.err; exit 1; }
fi
```
```bash
# ✔ contract 1 — source type/location/auth all bound to OUR connection; environment converged (not drifted)
aws codebuild batch-get-projects --region "$AWS_REGION" --names "$PROJECT_NAME" \
  --query 'projects[0].source.{type:type,location:location,auth:auth.resource}' --output json
aws codebuild batch-get-projects --region "$AWS_REGION" --names "$PROJECT_NAME" \
  --query 'projects[0].source.auth.resource' --output text | grep -q "$CONN_ARN" \
  && echo "auth bound to our connection" || { echo "auth NOT bound — fix with update-project (see Update)"; exit 1; }
# JSON output, NOT text: --output text renders booleans Python-style (`False`) and the compare can never
# match `false` — proven live 2026-07-12 (the authored text-mode check failed on a converged project)
ENV_GOT="$(aws codebuild batch-get-projects --region "$AWS_REGION" --names "$PROJECT_NAME" \
  --query 'projects[0].environment.[computeType,privilegedMode]' --output json | tr -d ' \n[]"')"
[ "$ENV_GOT" = "${COMPUTE_TYPE},${PRIVILEGED}" ] \
  && echo "environment converged (${ENV_GOT})" \
  || { echo "environment drift: got [$ENV_GOT], want [${COMPUTE_TYPE},${PRIVILEGED}] — reconcile via Update"; exit 1; }
```
> → Live State: `PROJECT_ARN`. **vpc branch:** the create above already passed `--<VPC_ID>onfig` as JSON
> (shorthand can't carry the comma-joined subnet list) — verify with `--query 'projects[0].vpcConfig'`.

## 5. Arm the webhook  🔴🟢  *(the authority moment — human go)*

> 🔴 Human go. This step registers a webhook **in the git repo** and starts accepting `WORKFLOW_JOB_QUEUED`
> events: from here on, **anyone who can push a workflow file to `${REPO_URL}` can execute code in this AWS
> account under `${ROLE_NAME}`**. That is the entire point — and a real grant. Confirm the repo's push
> access matches what you'd give the role. Reversible (`delete-webhook`), $0 until a job actually runs.

```bash
# observe first — a second create-webhook on the same project fails ResourceAlreadyExistsException
WEBHOOK_URL="$(aws codebuild batch-get-projects --region "$AWS_REGION" --names "$PROJECT_NAME" \
  --query 'projects[0].webhook.payloadUrl' --output text 2>/dev/null || echo None)"
if [ "$WEBHOOK_URL" != "None" ] && [ -n "$WEBHOOK_URL" ]; then
  echo "webhook already armed: $WEBHOOK_URL"
else
  WEBHOOK_URL="$(aws codebuild create-webhook --region "$AWS_REGION" --project-name "$PROJECT_NAME" \
    --filter-groups '[[{"type":"EVENT","pattern":"WORKFLOW_JOB_QUEUED"}]]' \
    --query 'webhook.payloadUrl' --output text)" || {
      echo "create-webhook failed — a non-AVAILABLE connection fails HERE (proven live:"
      echo "  InvalidInputException: Connection <arn> is unavailable) → finish source-connection.aws.md §2;"
      echo "on the gitlab branch also consider the missing create_runner/manage_runner scopes (gotcha below)"; exit 1; }
  echo "armed: $WEBHOOK_URL"
fi
```
```bash
# ✔ contract 2 — the WORKFLOW_JOB_QUEUED filter is on the project
aws codebuild batch-get-projects --region "$AWS_REGION" --names "$PROJECT_NAME" \
  --query 'projects[0].webhook.filterGroups' --output json | grep -q WORKFLOW_JOB_QUEUED \
  && echo "webhook armed for runner jobs" || { echo "webhook filter wrong"; exit 1; }
```
> → Live State: `WEBHOOK_URL`; `status: live`. Git-side confirmation (optional, manual): GitHub →
> `https://github.com/<owner>/<repo>/settings/hooks` / GitLab → `https://gitlab.com/<ns>/<proj>/-/hooks`
> shows an AWS webhook delivering **Workflow jobs** events.
>
> ⚠ **GitLab scope gotcha (proven by AWS's own docs):** a CodeConnections GitLab connection created *before*
> the runner feature lacks the `create_runner`/`manage_runner` OAuth scopes, and AWS does **not**
> auto-upgrade it. Remedy: in the CodeConnections console, create a **dummy connection** to the same GitLab
> account (triggers re-authorization with the new scopes), after which the existing connection works —
> then delete the dummy.

## 6. Acceptance verify  ✔  *(the contract)*

```bash
# standalone-runnable (the canon `verify` verb): re-derive everything read-only, assume nothing from earlier shells
CONN_ARN="${CONN_ARN:-$(discover_connection)}"
[ -n "$CONN_ARN" ] || { echo "0 FAILED: no ephemera-managed connection ${CONN_NAME}"; exit 1; }
PJSON="$(aws codebuild batch-get-projects --region "$AWS_REGION" --names "$PROJECT_NAME" --output json)"
PROJECT_ARN="${PROJECT_ARN:-$(printf '%s' "$PJSON" | sed -n 's/.*"arn": "\(arn:aws:codebuild[^"]*\)".*/\1/p' | head -1)}"
# 1 — project + source + auth
printf '%s' "$PJSON" | grep -q "\"type\": \"${SOURCE_TYPE}\""      && echo "1a: source type ${SOURCE_TYPE}" || { echo "1a FAILED"; exit 1; }
printf '%s' "$PJSON" | grep -q "\"resource\": \"${CONN_ARN}\""     && echo "1b: auth = our connection"      || { echo "1b FAILED"; exit 1; }
# 2 — webhook filter
printf '%s' "$PJSON" | grep -q '"pattern": "WORKFLOW_JOB_QUEUED"'  && echo "2: runner webhook armed"        || { echo "2 FAILED"; exit 1; }
# 4 — determinism: re-run §4+§5 → same ARN, no already-exists crash (assert against Live State's PROJECT_ARN)
printf '%s' "$PJSON" | grep -q "\"arn\": \"${PROJECT_ARN}\""       && echo "4: project ARN stable"          || { echo "4 FAILED"; exit 1; }
# T — tags (verify-table row, not a numbered contract item)
printf '%s' "$PJSON" | grep -q '"key": "ManagedBy"'                && echo "T: tags ok"                     || { echo "T: project untagged"; exit 1; }
```
> Contract 3 (the negative) is §1's stop; contract 5 (end-to-end) is §7's dogfood row — human-gated, mark
> its verify row `pending-dogfood` until a real job has run. → Live State: fill verify rows.

## 7. Use it — the consumer side lives in the repo  🟡

The runner is consumed by a **label in the workflow YAML**, not by another plan. The label must embed
`${PROJECT_NAME}` **verbatim** — a typo doesn't error, the job *hangs* until its queued-timeout (Director
trait 2). Render per provider:

**GitHub Actions** (`.github/workflows/*.yml`) — the run/attempt IDs let CodeBuild stop the build if the
workflow run is cancelled:

```yaml
jobs:
  my-job:
    runs-on: codebuild-${PROJECT_NAME}-${{ github.run_id }}-${{ github.run_attempt }}
    steps:
      - uses: actions/checkout@v4
      - run: aws sts get-caller-identity   # jobs run AS the service role — no repo secrets
```

**GitLab CI/CD** (`.gitlab-ci.yml`) — note `$CI_JOB_NAME` (not `$CI_JOB_ID`), and tags are a YAML *list*:

```yaml
build-job:
  stage: build
  script:
    - aws sts get-caller-identity
  tags:
    - codebuild-${PROJECT_NAME}-$CI_PROJECT_ID-$CI_PIPELINE_IID-$CI_JOB_NAME
```

Per-job overrides ride extra labels/tags (no plan change): `image:<env-type>-<image>`,
`instance-size:small|medium|large`, `fleet:<name>`, `buildspec-override:true`. Unknown labels are ignored.

```bash
# ✔ contract 5 (dogfood, after pushing a labeled workflow): a build ran and succeeded
aws codebuild list-builds-for-project --region "$AWS_REGION" --project-name "$PROJECT_NAME" \
  --max-items 3 --query 'ids' --output text
# then: aws codebuild batch-get-builds --ids <id> --query 'builds[0].buildStatus'  → SUCCEEDED
```

## Update (idempotent reconcile)  🟡

- **Re-apply** → §1 re-gates, §2 `put-role-policy` overwrites in place, §4/§5 observe-and-reuse. Contract 4.
- **Change `COMPUTE`/`IMAGE`/`DIND`** → `aws codebuild update-project --name "$PROJECT_NAME" --environment
  "type=LINUX_CONTAINER,image=${IMAGE},computeType=${COMPUTE_TYPE},privilegedMode=${PRIVILEGED}"`.
- **Repo moved / auth drifted** (§4's 1b check fails) → `aws codebuild update-project --name "$PROJECT_NAME"
  --source "{\"type\":\"${SOURCE_TYPE}\",\"location\":\"${REPO_URL}\",\"auth\":{\"type\":\"CODECONNECTIONS\",\"resource\":\"${CONN_ARN}\"}}"`.
- **Webhook filters** → `aws codebuild update-webhook --project-name "$PROJECT_NAME" --filter-groups …`.
- **Grant jobs more AWS access** → attach an *additional named* policy to `${ROLE_NAME}` (e.g.
  `runner-ecr-push`) — never widen `runner-core`; the diff stays legible per capability.
- **Change `PROVIDER` or `PROJECT_NAME`** → that's a different runner (the label embeds the name); tear down
  and re-create.

## Teardown — observe-first, resumable  💥

> 💥 Human go. Order matters: webhook → project → role → SG. Ownership-checked by `ManagedBy=ephemera` on
> the project — name-discovery must never delete a same-named project this plan didn't create. IAM deletes
> need `--no-session` under a broker. ⚠ Deleting the webhook removes job dispatch **and** the git-side hook;
> the **CloudWatch log group** (`${LOG_GROUP}`) lingers with your build logs — deleted last, deliberately
> gated on your call (logs may be the audit trail you want to keep).

```bash
# 💥 1 — webhook (disarms external dispatch), then the project (ownership-checked)
if aws codebuild batch-get-projects --region "$AWS_REGION" --names "$PROJECT_NAME" \
     --query 'projects[0].name' --output text 2>/dev/null | grep -q "^${PROJECT_NAME}$"; then
  OWN="$(aws codebuild batch-get-projects --region "$AWS_REGION" --names "$PROJECT_NAME" \
    --query "projects[0].tags[?key=='ManagedBy'].value | [0]" --output text)"
  [ "$OWN" = "ephemera" ] || { echo "project ${PROJECT_NAME} lacks ManagedBy=ephemera — not ours to delete"; exit 1; }
  HOOK="$(aws codebuild batch-get-projects --region "$AWS_REGION" --names "$PROJECT_NAME" \
    --query 'projects[0].webhook.url' --output text 2>/dev/null || echo None)"
  [ "$HOOK" != "None" ] && [ -n "$HOOK" ] && aws codebuild delete-webhook --region "$AWS_REGION" --project-name "$PROJECT_NAME"
  aws codebuild delete-project --region "$AWS_REGION" --name "$PROJECT_NAME"
  echo "project gone"
fi
```
```bash
# 💥 2 — role: ownership check, inline policies off, then delete (--no-session under a broker)
if aws iam get-role --role-name "$ROLE_NAME" >/dev/null 2>&1; then
  ROLE_OWN="$(aws iam list-role-tags --role-name "$ROLE_NAME" --query "Tags[?Key=='ManagedBy'].Value | [0]" --output text 2>/dev/null || true)"
  [ "$ROLE_OWN" = "ephemera" ] || { echo "role ${ROLE_NAME} lacks ManagedBy=ephemera — not ours to delete"; exit 1; }
  for p in $(aws iam list-role-policies --role-name "$ROLE_NAME" --query PolicyNames --output text); do
    aws iam delete-role-policy --role-name "$ROLE_NAME" --policy-name "$p"
  done
  aws iam delete-role --role-name "$ROLE_NAME" && echo "role gone"
fi
```
```bash
# 💥 3 — SG: discovered unconditionally (a fresh teardown shell doesn't know the NETWORK answer) and
# tag-filtered (only OUR SG). Build ENIs from a recent job can linger a few minutes — ⏳ retry until unblocked.
SG_ID="$(aws ec2 describe-security-groups --region "$AWS_REGION" \
  --filters "Name=group-name,Values=${PROJECT_NAME}-sg" "Name=tag:ManagedBy,Values=ephemera" \
  --query 'SecurityGroups[0].GroupId' --output text 2>/dev/null || echo None)"
if [ "$SG_ID" != "None" ] && [ -n "$SG_ID" ]; then
  for i in 1 2 3 4 5 6; do
    aws ec2 delete-security-group --region "$AWS_REGION" --group-id "$SG_ID" 2>/dev/null && { echo "sg gone"; break; }
    echo "sg busy (lingering build ENI) — retry $i in 30s"; sleep 30
  done
fi
```
```bash
# 💥 4 — log group (your build logs — delete only if you don't want the audit trail).
# CodeBuild creates it lazily at the FIRST build — if no job ever ran, absent is the normal outcome (proven live).
aws logs delete-log-group --region "$AWS_REGION" --log-group-name "$LOG_GROUP" 2>/dev/null && echo "logs gone" || echo "log group absent (no build ever ran, or kept)"
```
```bash
# ✔ absence — project, role, (sg), webhook all gone; the connection is NOT ours to touch
aws codebuild batch-get-projects --region "$AWS_REGION" --names "$PROJECT_NAME" \
  --query 'projectsNotFound' --output text | grep -q "$PROJECT_NAME" && echo "✔ project absent" || echo "project STILL PRESENT"
aws iam get-role --role-name "$ROLE_NAME" >/dev/null 2>&1 && echo "role STILL PRESENT" || echo "✔ role absent"
```
> → Live State: `status: gone`, clear realized ids. **The connection stays** — it belongs to
> `source-connection.aws.md` (its teardown, its ledger). The git-side webhook disappears with
> `delete-webhook`; confirm in repo settings if you want the negative proven.

## Composition — how this plugs into the fleet

- **Upstream:** `source-connection.aws.md` **Provides** the connection (§1 discovers by name, gates on
  `AVAILABLE`); `network.aws.md` (`WANT_PRIVATE=yes`) **Provides** the NAT-routed private subnets the vpc
  branch rides.
- **Downstream:** the runner's *jobs* are the consumers. A deploy job that pushes to ECR and rolls a
  cluster needs an **extra named policy** on `${ROLE_NAME}` (Update §) — e.g. `runner-ecr-push`
  (`ecr:GetAuthorizationToken` + repo-scoped push) or `runner-eks-deploy` (cluster `DescribeCluster` + an
  EKS access entry for the role ARN). A deploy-target plan can declare that seam as a **Requires**
  (`ci-runner-role(ROLE_ARN)`) — the role ARN is the hand-off, discovered from the cloud like every other
  edge. The migrate-source ran exactly that loop live (runner → ECR → kubectl → EKS Fargate).
- **The cost A/B this intent exists for:** hosted runners bill per-minute with secrets exported to the git
  provider; an always-on self-hosted box bills 24/7; this binding bills per-minute *inside* your account
  with role credentials. Zero-idle + no-exported-secrets is the pitch — put real numbers in when dogfooded.

## Deliberately not included

- **Bitbucket / Buildkite runners** — CodeBuild also hosts these; the `PROVIDER` enum stays at the
  commissioned pair (github|gitlab). Adding one is a case-branch + label-syntax row, when the need is real.
- **Org/enterprise-wide webhooks** (`ScopeConfiguration: GITHUB_ORGANIZATION` / `GITLAB_GROUP`) — this plan
  serves **one repo** (`REPO_URL`). Fleet-wide runners change the §5 authority analysis (every repo in the
  org can execute as the role) and deserve their own gate design.
- **Self-managed providers** (GitHub Enterprise Server / GitLab self-managed) — need a CodeConnections
  **Host** first; already a named omission in `source-connection.aws.md`.
- **Reserved-capacity fleets & Lambda compute** — cost/latency tuning knobs (`fleet:` label,
  `LINUX_LAMBDA_CONTAINER`) omitted until a real workload wants them; per-job label overrides (§7) cover
  the common cases without plan changes.
- **Job permissions beyond logs + connection** — deliberately the consumer's move (Composition), so the
  role's blast radius is a decision per pipeline, not a default.
- **The PAT / OAuth-app auth route** — CodeBuild also accepts imported source credentials; this plan
  standardizes on CodeConnections because that's the fleet's `Provides` seam (and GitLab runners require
  it anyway).