# Ephemera — Observability on AWS (CloudWatch alarms + SNS + Budgets, aws CLI)

> Self-executing Markdown. The **first binding of a new `observability` intent** — the *struts* for alerting:
> one notification channel + a deterministic alarm per watched resource + a monthly cost guard. You design your
> own alerting by turning watchers on; each watcher is a pattern you can copy for resources this plan doesn't
> know yet. The cloud is the source of truth; this file is intent + write-back ledger + audit trail.
>
> **Provides** `sns-topic(${TOPIC_ARN})` — the alerts channel other things publish into (the
> `docs/recipes/aws-alerts-to-slack.md` recipe composes on it — proven live 2026-07-02; email rides it
> directly). **Requires** only the resources you point
> the `WATCH_*` knobs at — each **discovered, never created here** (a Lambda from
> [`service.aws.md`](./service.aws.md), a queue from [`task-runner.aws.md`](./task-runner.aws.md), a table from
> [`document-db.aws.md`](./document-db.aws.md), an HTTP API — all optional, all borrowed).

## 🤖 Director prompt

Observe before acting; verify each step before advancing; stop at 🔴/💥 for human go; write realized values
back into Live State. `put-metric-alarm` is a **named upsert** — re-running converges by construction — and
every alarm name is a pure function of the knobs, so the watcher set *is* the resolved-inputs record. The one
async human edge: an **email subscription stays `PendingConfirmation` until the human clicks the link** —
that's a ⏳ state, not a failure.

> **Status: DOGFOODED LIVE 2026-07-01 (all four watchers + billing guard + email sub, ~$0).** Ran end-to-end
> against a throwaway fleet (a service.aws.md-shaped Lambda + HTTP API, a queue, a flat table — created,
> watched, torn down): topic + deterministic policy accepted; **the Budgets→SNS handshake validated live**
> (`create-budget` succeeds only if the topic policy admits `budgets.amazonaws.com` — it did, with the
> `aws:SourceAccount` condition); all **five alarms** (incl. the uncertainty-flagged API-`5xx`/`ApiId` shape)
> upserted, wired alarm+ok, and quiet; **the strut test PASSED with the hardened assertion** — forced `ALARM`
> → history showed `Successfully executed action` (time-bounded to the test) → forced back `OK`. The email
> subscription flow behaved exactly as written: `PendingConfirmation` while the human hadn't clicked, publish
> to the topic proven regardless. Teardown (alarms tag-checked → budget fail-closed → topic) + the fleet all
> absence-verified. Honest residuals: **API-`5xx` semantics** (does the ApiId-only series populate?) still
> need real 5xx traffic — config-level proof only; actual email **delivery** needs a confirmed subscription
> (⏳ — and note: a confirmation link clicked *after* teardown fails harmlessly, the topic is gone); a real
> budget breach remains untestable cheaply.

```
Legend  🟢 create · 🟡 config · 🔴 GATE (human go) · 💥 destructive (human go) · ⏳ wait · ✔ verify
```

## What you need, and why  *(read if CloudWatch is new to you)*

- **`aws` CLI v2, authenticated** (`aws sts get-caller-identity` succeeds) + **`python3`**.
- **Run the code blocks under `bash`** (any ≥3.2; see EPHEMERA.md's shell gotcha).
- **Something to watch** — each `WATCH_*` knob names an existing resource (empty = watcher off). Nothing here
  creates workloads; this plan only *observes* them.
- **(email alerts)** an inbox you can open: SNS sends a **confirmation link** and delivers nothing until it's
  clicked (⏳ human step — the closest thing this plan has to a console gate, and it's on your mail client).
- **Cost model:** first 10 alarms free, then ~$0.10/alarm/month; SNS email is free at this scale; **2 budgets
  free**, then ~$0.02/day. Effectively $0 for a small fleet — the guard costs less than what it guards against.
- **Noise model (a decision):** alarms notify on **ALARM and on recovery** (`--ok-actions`) — two emails per
  incident, and the strut test sends both to every confirmed subscriber. Recovery signal is worth it; drop
  `--ok-actions` in §3 if you disagree.

## Intent

Stand up the **alerting struts**: one SNS topic as the notification channel, a **deterministic alarm per
watched resource** — Lambda errors, queue backlog age, table throttles, API 5xx — each wired to the channel,
each a copyable pattern (namespace + metric + dimension + threshold) for anything else you later want watched,
plus an **AWS Budgets monthly cost guard** publishing into the same channel. Budgets is chosen over the classic
billing alarm deliberately: billing metrics live only in us-east-1 behind a one-time console enablement — the
Budgets API has neither gate.

**Acceptance contract** (defined here — the first `observability` binding; future siblings must pass it):
1. the **channel exists** and its subscription state matches the knob (confirmed, or honestly `Pending confirmation`)
2. **every enabled watcher has its alarm**, wired to the channel, in `OK`/`INSUFFICIENT_DATA` (config-level)
3. **an alert actually flows to the topic** — force one alarm to `ALARM` (`set-alarm-state`, the documented
   test lever) and the history records a **successful** SNS action after the test began; then force it back.
   *End-to-end to the topic publish* — delivery beyond the topic needs a confirmed subscription (⏳ §2), and a
   `Failed to execute action` in history is an explicit FAIL (the topic-policy proof)
4. **negative:** a disabled watcher has **no** alarm
5. **(`BILLING_GUARD=budget`)** the budget exists with the declared limit and its notification targets the
   channel *(a real breach is not cheaply testable — config-level, named)*

> **Sibling note (cost guard):** on Cloudflare the billing guard is a **standalone intent** —
> [`cost-guard.cloudflare.md`](./cost-guard.cloudflare.md) (`billing_budget_alert` + per-metric usage
> alerts). This plan's `BILLING_GUARD=budget` *is* the AWS equivalent, folded in because Budgets shares
> the SNS channel — deliberate, not a missing binding. A standalone budgets-only `cost-guard.aws.md`
> would be that intent's AWS sibling if one is ever wanted.

## Provisioning Inputs

| # | Question | Options (closed enum) | Default | Sets | Gates |
|---|----------|-----------------------|---------|------|-------|
| 1 | Email for alerts | address or empty | empty | `EMAIL_ALERT` | §2 (🔴 sends a confirmation email) |
| 2 | Watch a Lambda's errors? | function name or empty | empty | `WATCH_LAMBDA` | §3a |
| 3 | Watch a queue's backlog age? | queue name or empty | empty | `WATCH_QUEUE` | §3b |
| 4 | Watch a table's throttles? | table name or empty | empty | `WATCH_TABLE` | §3c (read + write alarms) |
| 5 | Watch an HTTP API's 5xx? | API id or empty | empty | `WATCH_API` | §3d |
| 6 | Monthly cost guard? | `off` / `budget` | `off` | `BILLING_GUARD` | §4 + acceptance 5 |
| 7 | Budget limit (USD/month) | number | `25` | `BUDGET_LIMIT` | §4 |
| 8 | Alerts noun | text — `[a-z0-9-]` | `ephemera-alerts` | `OBS_NAME` | every resource name |
| 9 | Environment | `dev` / `stg` / `uat` / `prod` | `dev` | `ENV` | names + `Environment` tag |

```yaml
# → written into Live State once resolved (the deterministic input to every step below)
resolved_inputs:
  email_alert:   ""          # empty = topic only (a recipe/Lambda can subscribe later)
  watch_lambda:  ""          # function name or empty
  watch_queue:   ""          # queue name or empty
  watch_table:   ""          # table name or empty
  watch_api:     ""          # HTTP API id or empty
  billing_guard: off         # off | budget
  budget_limit:  25
  obs_name:      ephemera-alerts
  env:           dev
  resolved_by:   <human who confirmed>
  resolved_at:   <timestamp>
```

> **Determinism.** Alarm names are pure functions of `OBS_NAME`+`ENV`+watcher (`${OBS_NAME}-${ENV}-lambda-errors`
> …), and `put-metric-alarm` **upserts by name** — the rare AWS create that is natively idempotent. The topic is
> discover-or-create by name; the topic policy is a **full deterministic document** (we own the topic — no merge
> needed; the budgets statement appears iff the knob is on). The budget upserts by name via delete-and-recreate
> only on explicit Update, never silently.

## Live State

```yaml
status:        not-created      # published template - run it to realize state
last_action:   teardown 2026-07-02 — seam unwired (unsubscribe + remove-permission) → alarm (tag-checked)
               → topic → recipe glue (function/param/role/log-group); all absence-verified (💥 go: Mark)
last_verified: 2026-07-02 composition dogfood (us-west-2, ~$0) — strut test proved CHANNEL-DEEP delivery:
               forced ALARM → SNS → alerts-notifier Lambda → Slack webhook; 🚨+✅ both human-confirmed
               in-channel (notifier 2 invocations / 0 errors). §3's negative checks exercised live this
               run (4 disabled watchers asserted absent — the 2026-07-01 PASS* row is now a clean PASS).

resolved_inputs:
  email_alert:   ""
  watch_lambda:  ""
  watch_queue:   ""
  watch_table:   ""
  watch_api:     ""
  billing_guard: off
  budget_limit:  25
  obs_name:      ephemera-alerts
  env:           dev
realized:
  AWS_REGION:   —
  TOPIC_ARN:    —             # §1
  SUBSCRIPTION: —             # §2 (confirmed | PendingConfirmation | none)
  ALARMS:       —             # §3 (the enabled watcher set, by name)
  BUDGET_NAME:  —             # §4 (iff billing_guard=budget)
```

| ✔ check                            | expected                                              | observed (2026-07-01 dogfood) | result |
|------------------------------------|-------------------------------------------------------|----------|--------|
| channel exists, sub matches knob   | topic present; email sub confirmed/pending per knob   | topic + policy; sub PendingConfirmation (honest ⏳) | PASS |
| enabled watchers alarmed           | one alarm per watcher, action = topic, OK/INSUF       | 5/5 wired (alarm+ok), quiet | PASS |
| alert flows end-to-end (to topic)  | set-alarm-state → Successfully executed action        | success action post-START; recovered OK | PASS |
| disabled watcher has no alarm (neg)| alarm absent for empty knobs                          | n/a this run (all watchers on); §3 check() carries it | PASS* |
| billing guard configured (iff on)  | budget limit + notification → the topic               | 25.0 USD @80% → topic (handshake validated) | PASS |
| tags present                       | topic + alarms carry `ManagedBy=ephemera`             | all tagged (teardown checks relied on it) | PASS |

## TAGS — provenance & cost tags

Tag **on create**: the SNS topic and every alarm take **Key/Value lists** (`tags_kv`). Budgets have no tag API
(a portability note, like Cloudflare's) — provenance rides the deterministic budget name. Watched resources are
**borrowed** — their plans tag them. Renderers inline in §0 (canonical: `scripts/tags.sh`).

## 0. Variables

```bash
set -euo pipefail
export AWS_REGION="${AWS_REGION:-us-west-2}"
export ENV="${ENV:-dev}"
export OBS_NAME="${OBS_NAME:-ephemera-alerts}"
export EMAIL_ALERT="${EMAIL_ALERT:-}"
export WATCH_LAMBDA="${WATCH_LAMBDA:-}"
export WATCH_QUEUE="${WATCH_QUEUE:-}"
export WATCH_TABLE="${WATCH_TABLE:-}"
export WATCH_API="${WATCH_API:-}"
export BILLING_GUARD="${BILLING_GUARD:-off}"   # off | budget
export BUDGET_LIMIT="${BUDGET_LIMIT:-25}"

printf '%s' "$OBS_NAME" | grep -Eq '^[a-z0-9-]+$' || { echo "OBS_NAME must be [a-z0-9-]"; exit 1; }
case "$BILLING_GUARD" in off|budget) ;; *) echo "BILLING_GUARD must be off|budget"; exit 1;; esac
case "$ENV" in dev|stg|uat|prod) ;; *) echo "ENV must be dev|stg|uat|prod"; exit 1;; esac
printf '%s' "$BUDGET_LIMIT" | grep -Eq '^[0-9]+$' || { echo "BUDGET_LIMIT must be a whole USD amount"; exit 1; }

BASE="${OBS_NAME}-${ENV}"
TOPIC_NAME="${BASE}-topic"
BUDGET_NAME="${BASE}-monthly"
ACCOUNT_ID="$(aws sts get-caller-identity --query Account --output text)"

# ── TAGS — resolved once (canonical: scripts/tags.sh) ──
PLAN_SOURCE="observability.aws.md"
PLAN_VERSION="2026-07-01"
TAG_COST_CENTER="${TAG_COST_CENTER:-}"; TAG_OWNER="${TAG_OWNER:-}"
TAGS="$(printf '%s\n' "ManagedBy=ephemera" "Source=${PLAN_SOURCE}" "PlanVersion=${PLAN_VERSION}" \
  "CostCenter=${TAG_COST_CENTER}" "Owner=${TAG_OWNER}" "Environment=${ENV}")"
_tags_list() { printf '%s\n' "$TAGS" "$@" | awk '
  { eq=index($0,"="); if(eq==0) next; k=substr($0,1,eq-1); v=substr($0,eq+1); if(v=="") next;
    val[k]=v; if(!(k in seen)){ order[++n]=k; seen[k]=1 } }
  END { for(i=1;i<=n;i++) print order[i]"="val[order[i]] }'; }
tags_kv() { _tags_list "$@" | while IFS='=' read -r k v; do printf 'Key=%s,Value=%s ' "$k" "$v"; done; }
```

## Requires-discovery (read-only — every watched resource must exist)  ✔

```bash
if [ -n "$WATCH_LAMBDA" ]; then
  aws lambda get-function --region "$AWS_REGION" --function-name "$WATCH_LAMBDA" >/dev/null \
    || { echo "lambda ${WATCH_LAMBDA} not found"; exit 1; }
fi
if [ -n "$WATCH_QUEUE" ]; then
  aws sqs get-queue-url --region "$AWS_REGION" --queue-name "$WATCH_QUEUE" >/dev/null \
    || { echo "queue ${WATCH_QUEUE} not found"; exit 1; }
fi
if [ -n "$WATCH_TABLE" ]; then
  aws dynamodb describe-table --region "$AWS_REGION" --table-name "$WATCH_TABLE" >/dev/null \
    || { echo "table ${WATCH_TABLE} not found"; exit 1; }
fi
if [ -n "$WATCH_API" ]; then
  aws apigatewayv2 get-api --region "$AWS_REGION" --api-id "$WATCH_API" >/dev/null \
    || { echo "http api ${WATCH_API} not found"; exit 1; }
fi
echo "watched resources discovered"
```
> → Live State: the watched names (discovered, NOT created).

## Dependency frontier

```
§1 topic 🟢 (policy per knobs) ─┬─ §2 email sub 🔴⏳ (confirmation link — human)
                                ├─ §3 watchers 🟡 (one upsert-alarm per enabled knob; targets discovered above)
                                └─ §4 budget 🟡 (iff knob — its notification needs the topic ARN + policy)
                                                    all ─> §5 ✔ acceptance (incl. the forced-alarm strut test)
```

Non-negotiable edges: **everything hangs off the topic ARN** (alarm actions, budget subscriber — topic first);
the **budgets topic-policy statement must exist before `create-budget`** (Budgets validates it can publish —
the §1 policy carries it iff the knob is on). Watchers are independent of each other. Teardown reverses.

## 1. The channel — SNS topic  🟢  *(discover-or-create; policy is ours, built whole)*

```bash
# observe — create-topic is idempotent BY NAME for identical attributes, but we observe anyway for the tags path
TOPIC_ARN="arn:aws:sns:${AWS_REGION}:${ACCOUNT_ID}:${TOPIC_NAME}"
if aws sns get-topic-attributes --region "$AWS_REGION" --topic-arn "$TOPIC_ARN" >/dev/null 2>&1; then
  # ownership check BEFORE adopting — the policy write below must never clobber a foreign same-named topic
  TT="$(aws sns list-tags-for-resource --region "$AWS_REGION" --resource-arn "$TOPIC_ARN" --output json 2>&1 || true)"
  printf '%s' "$TT" | grep -q '"ephemera"' && echo "topic exists — reuse (ours)" \
    || { echo "🔴 topic ${TOPIC_NAME} exists but is NOT tagged ManagedBy=ephemera — adopting is a human decision; stop."; exit 1; }
else
  TOPIC_ARN="$(aws sns create-topic --region "$AWS_REGION" --name "$TOPIC_NAME" \
    --tags $(tags_kv) --query 'TopicArn' --output text)"
fi
# policy — full deterministic document (the topic is OURS — the adoption gate above proved it): a
# default-equivalent owner statement (minus Add/RemovePermission — this plan manages policy declaratively),
# plus the budgets publisher iff the knob is on (CloudWatch alarms in-account ride the owner statement)
POL="$(TA="$TOPIC_ARN" ACCT="$ACCOUNT_ID" BG="$BILLING_GUARD" python3 - <<'PY'
import json, os
sts = [{
  "Sid": "OwnerFullControl", "Effect": "Allow", "Principal": {"AWS": "*"},
  "Action": ["SNS:Publish", "SNS:Subscribe", "SNS:GetTopicAttributes", "SNS:DeleteTopic",
             "SNS:SetTopicAttributes", "SNS:ListSubscriptionsByTopic", "SNS:Receive"],
  "Resource": os.environ["TA"],
  "Condition": {"StringEquals": {"AWS:SourceOwner": os.environ["ACCT"]}}
}]
if os.environ["BG"] == "budget":
    sts.append({"Sid": "AllowBudgets", "Effect": "Allow",
                "Principal": {"Service": "budgets.amazonaws.com"},
                "Action": "SNS:Publish", "Resource": os.environ["TA"],
                "Condition": {"StringEquals": {"aws:SourceAccount": os.environ["ACCT"]}}})
print(json.dumps({"Version": "2012-10-17", "Statement": sts}))
PY
)"
aws sns set-topic-attributes --region "$AWS_REGION" --topic-arn "$TOPIC_ARN" \
  --attribute-name Policy --attribute-value "$POL"
echo "TOPIC_ARN=${TOPIC_ARN}"
```
```bash
# ✔ topic present + policy carries the budgets statement iff the knob is on (asserted)
GOTPOL="$(aws sns get-topic-attributes --region "$AWS_REGION" --topic-arn "$TOPIC_ARN" \
  --query 'Attributes.Policy' --output text)"
if [ "$BILLING_GUARD" = "budget" ]; then
  printf '%s' "$GOTPOL" | grep -q 'AllowBudgets' && echo "policy: budgets publisher on" || { echo "missing AllowBudgets"; exit 1; }
else
  printf '%s' "$GOTPOL" | grep -q 'AllowBudgets' && { echo "AllowBudgets present but knob off"; exit 1; } || echo "policy: no budgets statement (matches knob)"
fi
```
> → Live State: `TOPIC_ARN`; `status: creating`.

## 2. Email subscription — iff `EMAIL_ALERT`  🔴⏳  *(outward-facing: it emails a human)*

> 🔴 Human go — this sends a confirmation email to `EMAIL_ALERT`. ⏳ The subscription delivers **nothing**
> until the human clicks the link; `PendingConfirmation` is an honest intermediate state, not an error.
> ⚠ Re-calling `subscribe` on a pending address **re-sends the confirmation mail** — the observe-first guard
> below is what prevents that; the `PendingConfirmation` sentinel is non-empty, so the guard holds.

```bash
if [ -n "$EMAIL_ALERT" ]; then
  SUBS="$(aws sns list-subscriptions-by-topic --region "$AWS_REGION" --topic-arn "$TOPIC_ARN" \
    --query "Subscriptions[?Endpoint=='${EMAIL_ALERT}'].SubscriptionArn | [0]" --output text)"
  if [ "$SUBS" = "None" ] || [ -z "$SUBS" ]; then
    aws sns subscribe --region "$AWS_REGION" --topic-arn "$TOPIC_ARN" \
      --protocol email --notification-endpoint "$EMAIL_ALERT" >/dev/null
    echo "confirmation email sent to ${EMAIL_ALERT} — ⏳ click the link; verify below reports the state"
  fi
fi
```
```bash
# ✔ subscription state per knob, ASSERTED (PendingConfirmation = PASS-with-⏳; absent = FAIL)
if [ -n "$EMAIL_ALERT" ]; then
  SARN="$(aws sns list-subscriptions-by-topic --region "$AWS_REGION" --topic-arn "$TOPIC_ARN" \
    --query "Subscriptions[?Endpoint=='${EMAIL_ALERT}'].SubscriptionArn | [0]" --output text)"
  case "$SARN" in
    None|"") echo "subscription MISSING for ${EMAIL_ALERT}"; exit 1 ;;
    PendingConfirmation) echo "subscription: PendingConfirmation (⏳ click the link)" ;;
    *) echo "subscription: confirmed (${SARN##*:})" ;;
  esac
fi
```
> → Live State: `SUBSCRIPTION` (`confirmed` when the ARN is real; `PendingConfirmation` while it isn't).

## 3. Watchers — one deterministic alarm per enabled knob  🟢  *(the struts; copy the pattern for anything else)*

> Every alarm: 5-minute period, `treat-missing-data notBreaching` (an idle resource is not an incident),
> action = the topic, **named upsert** (`put-metric-alarm` converges by name — re-runs are free). The four
> below are patterns: *namespace + metric + dimension + threshold*. Watching something new = one more block
> in this shape.

```bash
alarm() {  # alarm <name> <namespace> <metric> <dimName> <dimValue> <stat> <threshold>
  aws cloudwatch put-metric-alarm --region "$AWS_REGION" \
    --alarm-name "$1" --namespace "$2" --metric-name "$3" \
    --dimensions Name="$4",Value="$5" \
    --statistic "$6" --period 300 --evaluation-periods 1 \
    --threshold "$7" --comparison-operator GreaterThanOrEqualToThreshold \
    --treat-missing-data notBreaching \
    --alarm-actions "$TOPIC_ARN" --ok-actions "$TOPIC_ARN" \
    --tags $(tags_kv)
  echo "alarm upserted: $1"
}
[ -n "$WATCH_LAMBDA" ] && alarm "${BASE}-lambda-errors"        AWS/Lambda   Errors                        FunctionName "$WATCH_LAMBDA" Sum 1
[ -n "$WATCH_QUEUE" ]  && alarm "${BASE}-queue-age"            AWS/SQS      ApproximateAgeOfOldestMessage QueueName    "$WATCH_QUEUE"  Maximum 300
if [ -n "$WATCH_TABLE" ]; then
  alarm "${BASE}-table-read-throttles"  AWS/DynamoDB ReadThrottleEvents  TableName "$WATCH_TABLE" Sum 1
  alarm "${BASE}-table-write-throttles" AWS/DynamoDB WriteThrottleEvents TableName "$WATCH_TABLE" Sum 1
fi
[ -n "$WATCH_API" ] && alarm "${BASE}-api-5xx" AWS/ApiGateway 5xx ApiId "$WATCH_API" Sum 1
echo "watchers applied"
```
```bash
# ✔ every enabled watcher has its alarm wired to the topic; every DISABLED watcher has none (the negative)
check() {  # check <name> <enabled: yes|no>
  FOUND="$(aws cloudwatch describe-alarms --region "$AWS_REGION" --alarm-names "$1" \
    --query 'length(MetricAlarms)' --output text)"
  if [ "$2" = "yes" ]; then
    [ "$FOUND" = "1" ] || { echo "MISSING alarm $1"; exit 1; }
    ACT="$(aws cloudwatch describe-alarms --region "$AWS_REGION" --alarm-names "$1" \
      --query 'MetricAlarms[0].[AlarmActions[0],OKActions[0],StateValue]' --output text)"
    printf '%s' "$ACT" | grep -qF "$TOPIC_ARN" || { echo "alarm $1 not wired to the topic"; exit 1; }
    printf '%s' "$ACT" | grep -Eq '(OK|INSUFFICIENT_DATA)$' || { echo "alarm $1 already ALARMING — the watched resource has a live problem"; exit 1; }
    echo "alarm $1: present, wired (alarm+ok actions), quiet"
  else
    [ "$FOUND" = "0" ] || { echo "alarm $1 EXISTS but its watcher is off"; exit 1; }
    echo "alarm $1: absent (matches knob)"
  fi
}
check "${BASE}-lambda-errors"         "$([ -n "$WATCH_LAMBDA" ] && echo yes || echo no)"
check "${BASE}-queue-age"             "$([ -n "$WATCH_QUEUE" ]  && echo yes || echo no)"
check "${BASE}-table-read-throttles"  "$([ -n "$WATCH_TABLE" ]  && echo yes || echo no)"
check "${BASE}-table-write-throttles" "$([ -n "$WATCH_TABLE" ]  && echo yes || echo no)"
check "${BASE}-api-5xx"               "$([ -n "$WATCH_API" ]    && echo yes || echo no)"
```
> → Live State: `ALARMS` (the enabled set, by name).

## 4. Billing guard — iff `BILLING_GUARD=budget`  🟢  *(Budgets, not the us-east-1 console-gated classic)*

```bash
if [ "$BILLING_GUARD" = "budget" ]; then
  # observe — budgets are named; recreate only on explicit Update, never silently
  if aws budgets describe-budget --account-id "$ACCOUNT_ID" --budget-name "$BUDGET_NAME" >/dev/null 2>&1; then
    echo "budget ${BUDGET_NAME} exists — reuse"
  else
    BUDGET="$(LIM="$BUDGET_LIMIT" NAME="$BUDGET_NAME" python3 -c 'import json,os;print(json.dumps({"BudgetName":os.environ["NAME"],"BudgetLimit":{"Amount":os.environ["LIM"],"Unit":"USD"},"TimeUnit":"MONTHLY","BudgetType":"COST"}))')"
    NOTIF="$(TA="$TOPIC_ARN" python3 -c 'import json,os;print(json.dumps([{"Notification":{"NotificationType":"ACTUAL","ComparisonOperator":"GREATER_THAN","Threshold":80,"ThresholdType":"PERCENTAGE"},"Subscribers":[{"SubscriptionType":"SNS","Address":os.environ["TA"]}]}]))')"
    aws budgets create-budget --account-id "$ACCOUNT_ID" \
      --budget "$BUDGET" --notifications-with-subscribers "$NOTIF"
  fi
fi
```
```bash
# ✔ budget limit + notification per knob (config-level — a real breach is not cheaply testable, named)
if [ "$BILLING_GUARD" = "budget" ]; then
  AMT="$(aws budgets describe-budget --account-id "$ACCOUNT_ID" --budget-name "$BUDGET_NAME" \
    --query 'Budget.BudgetLimit.Amount' --output text)"
  printf '%s' "$AMT" | grep -Eq "^${BUDGET_LIMIT}(\.0+)?$" && echo "budget: ${AMT} USD/month" || { echo "budget limit ${AMT} ≠ ${BUDGET_LIMIT}"; exit 1; }
  aws budgets describe-notifications-for-budget --account-id "$ACCOUNT_ID" --budget-name "$BUDGET_NAME" \
    --query 'Notifications[0].Threshold' --output text
fi
```
> → Live State: `BUDGET_NAME` (+ threshold 80%).

## 5a. Acceptance — read-only checks  ✔  *(safe on every drift pass)*

```bash
# tags (drift) — topic + first enabled alarm
TEST_ALARM=""
[ -n "$WATCH_LAMBDA" ] && TEST_ALARM="${BASE}-lambda-errors"
[ -z "$TEST_ALARM" ] && [ -n "$WATCH_QUEUE" ] && TEST_ALARM="${BASE}-queue-age"
[ -z "$TEST_ALARM" ] && [ -n "$WATCH_TABLE" ] && TEST_ALARM="${BASE}-table-read-throttles"
[ -z "$TEST_ALARM" ] && [ -n "$WATCH_API" ]   && TEST_ALARM="${BASE}-api-5xx"
TT="$(aws sns list-tags-for-resource --region "$AWS_REGION" --resource-arn "$TOPIC_ARN" --output json)"
printf '%s' "$TT" | grep -q '"ManagedBy"' && echo "topic tags ok" || { echo "topic untagged"; exit 1; }
if [ -n "$TEST_ALARM" ]; then
  AA="$(aws cloudwatch describe-alarms --region "$AWS_REGION" --alarm-names "$TEST_ALARM" \
    --query 'MetricAlarms[0].AlarmArn' --output text)"
  AT="$(aws cloudwatch list-tags-for-resource --region "$AWS_REGION" --resource-arn "$AA" --output json)"
  printf '%s' "$AT" | grep -q '"ManagedBy"' && echo "alarm tags ok" || { echo "alarm untagged"; exit 1; }
fi
```

## 5b. The strut test — an alert actually flows  🟡  *(MUTATES + notifies — acceptance/on-demand, NEVER a drift `verify`)*

> 🟡 This forces an alarm state and **really publishes to the channel** (two notifications to every confirmed
> subscriber: ALARM, then the OK recovery). That is its purpose. It proves the publish **to the topic** —
> delivery beyond the topic needs the ⏳ confirmed subscription. A drift `verify` run stops at §5a.

```bash
if [ -n "$TEST_ALARM" ]; then
  START="$(date -u +%Y-%m-%dT%H:%M:%SZ)"   # time-bound the history query to THIS test (stale successes must not pass)
  aws cloudwatch set-alarm-state --region "$AWS_REGION" --alarm-name "$TEST_ALARM" \
    --state-value ALARM --state-reason "ephemera acceptance strut test"
  ST="$(aws cloudwatch describe-alarms --region "$AWS_REGION" --alarm-names "$TEST_ALARM" \
    --query 'MetricAlarms[0].StateValue' --output text)"
  [ "$ST" = "ALARM" ] && echo "3a: state forced to ALARM" || { echo "3a: state is ${ST}"; exit 1; }
  # poll history (eventually consistent) for a SUCCESSFUL action after START; a failed action = explicit FAIL
  OK3=no
  for i in 1 2 3 4 5 6 7 8 9 10 11 12; do
    H="$(aws cloudwatch describe-alarm-history --region "$AWS_REGION" --alarm-name "$TEST_ALARM" \
      --history-item-type Action --start-date "$START" --max-records 10 --output json)"
    printf '%s' "$H" | grep -q 'Failed to execute action' && { echo "3b: ACTION FAILED — topic policy?"; exit 1; }
    if printf '%s' "$H" | grep -q 'Successfully executed action'; then OK3=yes; break; fi
    sleep 5
  done
  [ "$OK3" = yes ] && echo "3b: SNS action executed successfully (post-${START})" || { echo "3b: no successful action within 60s"; exit 1; }
  aws cloudwatch set-alarm-state --region "$AWS_REGION" --alarm-name "$TEST_ALARM" \
    --state-value OK --state-reason "ephemera acceptance strut test complete"
  echo "3: strut test complete — the channel carried a real alert (to the topic)"
else
  echo "3: skipped (no watcher enabled — channel-only apply)"
fi
```
> → Live State: fill the verify rows, `last_verified`, `status: live`.

## Update (idempotent reconcile)  🟡

- **Toggle a watcher** → on: re-run §3 (upsert). Off: `delete-alarms --alarm-names <name>` — the named-upsert
  model means §3 never deletes; removal is explicit.
- **Change a threshold** → edit the `alarm` call, re-run (upsert converges).
- **Change `BUDGET_LIMIT`** → `aws budgets update-budget` with the regenerated document (explicit, never silent).
- **Change `EMAIL_ALERT`** → subscribe the new address (its own confirmation), then unsubscribe the old ARN.
- **Watch something new** → copy a §3 pattern block: namespace + metric + dimension + threshold. That's the strut.
- ⚠ **`put-metric-alarm` ignores `--tags` on update** (documented) — re-runs converge *config*, not tags. If an
  alarm's tags drift, reconcile explicitly: `aws cloudwatch tag-resource --resource-arn <alarm-arn> --tags $(tags_kv)`.

## Teardown — observe-first, resumable  💥

> 💥 Human go. Deletes only what this plan created — alarms (ownership-checked by tag), the budget, the topic
> (subscriptions die with it). Watched resources are **borrowed and untouched** — removing observability never
> touches the observed.

```bash
# alarms — check the tag on each of OUR deterministic names before deleting
for A in "${BASE}-lambda-errors" "${BASE}-queue-age" "${BASE}-table-read-throttles" "${BASE}-table-write-throttles" "${BASE}-api-5xx"; do
  N="$(aws cloudwatch describe-alarms --region "$AWS_REGION" --alarm-names "$A" --query 'length(MetricAlarms)' --output text)"
  if [ "$N" = "1" ]; then
    AA="$(aws cloudwatch describe-alarms --region "$AWS_REGION" --alarm-names "$A" --query 'MetricAlarms[0].AlarmArn' --output text)"
    AT="$(aws cloudwatch list-tags-for-resource --region "$AWS_REGION" --resource-arn "$AA" --output json 2>&1 || true)"
    printf '%s' "$AT" | grep -q '"ManagedBy"' || { echo "alarm ${A} lacks our tag — not ours"; exit 1; }
    aws cloudwatch delete-alarms --region "$AWS_REGION" --alarm-names "$A"
    echo "deleted ${A}"
  fi
done
# budget (iff it exists — named, provenance rides the name since Budgets has no tags). A DELETE failure
# must fail loud — only the absent case is a no-op.
if aws budgets describe-budget --account-id "$ACCOUNT_ID" --budget-name "$BUDGET_NAME" >/dev/null 2>&1; then
  aws budgets delete-budget --account-id "$ACCOUNT_ID" --budget-name "$BUDGET_NAME"
  echo "budget deleted"
fi
# topic (ownership-checked; deleting it drops all subscriptions incl. pending)
TOPIC_ARN="arn:aws:sns:${AWS_REGION}:${ACCOUNT_ID}:${TOPIC_NAME}"
if aws sns get-topic-attributes --region "$AWS_REGION" --topic-arn "$TOPIC_ARN" >/dev/null 2>&1; then
  TT="$(aws sns list-tags-for-resource --region "$AWS_REGION" --resource-arn "$TOPIC_ARN" --output json 2>&1 || true)"
  printf '%s' "$TT" | grep -q '"ManagedBy"' || { echo "topic lacks our tag — not ours"; exit 1; }
  aws sns delete-topic --region "$AWS_REGION" --topic-arn "$TOPIC_ARN"
fi
```
```bash
# ✔ teardown verify — capture, then assert absence (pipefail-safe)
OUT="$(aws sns get-topic-attributes --region "$AWS_REGION" --topic-arn "$TOPIC_ARN" 2>&1 || true)"
printf '%s' "$OUT" | grep -qi 'NotFound' && echo "topic gone" || { echo "topic STILL EXISTS"; exit 1; }
LEFT="$(aws cloudwatch describe-alarms --region "$AWS_REGION" --alarm-name-prefix "${BASE}-" \
  --query 'length(MetricAlarms)' --output text)"
[ "$LEFT" = "0" ] && echo "alarms gone" || { echo "${LEFT} alarms remain"; exit 1; }
OUT="$(aws budgets describe-budget --account-id "$ACCOUNT_ID" --budget-name "$BUDGET_NAME" 2>&1 || true)"
printf '%s' "$OUT" | grep -qi 'NotFound' && echo "budget gone" || { echo "budget STILL EXISTS (knob-off also yields NotFound — this is real)"; exit 1; }
```
> → Live State: `status: gone`, clear realized ids.

## Composition — how this plugs into the fleet

`sns-topic(${TOPIC_ARN})` is the fleet's alert bus: email rides it directly (§2); the
**`docs/recipes/aws-alerts-to-slack.md` recipe** (proven live 2026-07-02) composes
it with a [`service.aws.md`](./service.aws.md)-shaped subscriber and
[`messaging.slack.md`](./messaging.slack.md)'s webhook — deliberately a *recipe*, not a plan, because the
primitives are all here and the wiring is three small steps (see *Deliberately not included*). The watchers
point at the fleet's own plans: [`service.aws.md`](./service.aws.md)'s function,
[`task-runner.aws.md`](./task-runner.aws.md)'s queue, [`document-db.aws.md`](./document-db.aws.md)'s table —
observability composes on everything and creates none of it.

## Deliberately not included

- **SNS→Lambda→Slack delivery** — a composition of existing plans, not a new capability: the topic (here) + a
  subscriber function ([`service.aws.md`](./service.aws.md)'s shape) + the webhook
  ([`messaging.slack.md`](./messaging.slack.md)). Lives as the golden-path recipe
  `docs/recipes/aws-alerts-to-slack.md` — the decision of 2026-07-01, **proven live 2026-07-02** (an agent
  composed it from the markdown alone; channel-deep delivery human-confirmed). (AWS Chatbot is the managed
  alternative; its Slack workspace authorization is console-gated — named so it's a choice.)
- **The classic billing alarm** (CloudWatch `EstimatedCharges`) — billing metrics exist only in us-east-1
  behind a one-time "Receive Billing Alerts" console enablement; Budgets does the same job with neither gate.
- **Dashboards** — presentation, and the first paid tier hides there ($3/dashboard/month after three); alarms
  are the struts, dashboards are décor. Add deliberately if wanted.
- **Composite alarms, anomaly detection, metric math, Logs Insights queries** — real power tools, each a
  deliberate step past "the struts"; the watcher pattern extends to all of them when a need is named.
- **Cross-account/organization budgets and alerting** — a trust topology decision, out of scope for a
  single-account fleet.