# Recipe — "wire my alarms to Slack" (CloudWatch → SNS → Lambda → webhook)

> A **golden path** for AWS alerting that lands in a Slack channel: every CloudWatch alarm
> [`observability.aws.md`](../../observability.aws.md) manages — and anything else that publishes to its
> topic — arrives as a formatted Slack message, ~$0/month at small-fleet scale. This is **composition
> without includes** — each plan stays standalone; this doc sequences them and carries the one piece of
> wiring no plan owns (the SNS-subscriber Lambda). The machine-readable edges live in
> [`catalog.json`](../../services/ephemera-releases/catalog.json).
>
> **Proven live 2026-07-02** (the fleet's first agent-composition dogfood): an agent handed exactly the
> three plans below wired the whole chain; the only human touches were the two Slack OAuth gates and the
> final teardown go. The strut test's forced `ALARM` → 🚨 message and its `OK` recovery → ✅ message were
> both received in-channel.

## What you end up with

Any alarm transition (and anything else published to the alerts topic) appears in a Slack channel you
chose, formatted (`🚨 *alarm-name* → *ALARM*` / `✅ … → *OK*`), within seconds. The shape:

```
messaging.slack ──Provides slack-channel(webhook)──► [Keychain] ──(operator copies once)──► [SSM SecureString]
  (MODE=incoming-webhook, two 🔴 OAuth clicks)                                                     │
                                                                                                   ▼
observability.aws ──Provides sns-topic(TOPIC_ARN)──► glue (this recipe): alerts-notifier Lambda ──► Slack
  (alarms + the strut test)                           subscribes to the topic, reads the webhook
                                                      from SSM at runtime, POSTs the message
```

## ⚠️ The key idea: the webhook crosses machines via SSM — never a plan, never an env-var value

The webhook URL is a **secret** (anyone holding it can post to your channel). It travels:
Slack consent screen → macOS **Keychain** (`messaging.slack.md`'s discipline) → **SSM SecureString**
(one transient copy at apply time, value never echoed) → read **at runtime** by the notifier
(`service.aws.md`'s SECRETS=ssm pattern: the function env carries the parameter **name** only). No plan,
ledger, or function config ever contains the URL — and the negative is checkable
(`get-function-configuration` must not contain `hooks.slack.com`).

## The order

| step | plan | knobs | Provides → | Requires |
|------|------|-------|-----------|----------|
| 1 | `messaging.slack.md` | `MODE=incoming-webhook`, `DELIVERY=manual-paste` | `slack-channel(webhook)` → Keychain item `slack-webhook-${ENV}` | a Slack workspace; **🔴 create-app + 🔴 install consent** |
| 2 | *glue (below)* | `SVC=alerts-notifier` | the subscriber: `${SVC}-${ENV}` Lambda, webhook in SSM | step 1's Keychain item; IAM (⚠ `--no-session` under a broker) |
| 3 | `observability.aws.md` | `WATCH_*` per your fleet (see twist below), `EMAIL_ALERT=""` | `sns-topic(TOPIC_ARN)` + alarms | watched resources exist |
| 4 | *glue (below)* | — | topic → Lambda subscription + scoped invoke permission | steps 2 + 3 |

**The self-watching twist (what the dogfood ran):** point `WATCH_LAMBDA` at the notifier itself
(`alerts-notifier-${ENV}`). Zero extra fleet for a demo, and in production it means notifier failures
alarm too. Honest residual: if the notifier is *fully* broken, the message *about* its breakage can't
reach Slack either — a confirmed `EMAIL_ALERT` subscription (`observability.aws.md` §2 ⏳) is the
independent backstop.

## Step 1 — the Slack webhook (🔴 ×2, click-path proven live)

Resolved inputs: `MODE=incoming-webhook`, `APP_NAME=ephemera-alerts-${ENV}`, Keychain item
`slack-webhook-${ENV}` @ account `ephemera-slack-${ENV}`. The manifest (two live corrections already
folded into `messaging.slack.md`: `features.bot_user` **is required** — `incoming-webhook` is a bot
scope and Slack rejects bot scopes without a bot user; and the paste dialog **defaults to its JSON
tab**, so hand a human JSON):

```json
{
  "display_information": { "name": "ephemera-alerts-dev",
    "description": "AWS CloudWatch alarms to Slack (Ephemera aws-alerts-to-slack recipe)" },
  "features": { "bot_user": { "display_name": "ephemera-alerts" } },
  "oauth_config": { "scopes": { "bot": ["incoming-webhook"] } },
  "settings": { "incoming_webhooks": { "incoming_webhooks_enabled": true },
    "org_deploy_enabled": false, "token_rotation_enabled": false }
}
```

Click-path: `https://api.slack.com/apps?new_app=1` → **From a manifest** → pick workspace → paste the
JSON above (JSON tab) → **Next** → review shows *"Post messages to specific channels in Slack"* →
**Create** → sidebar **Incoming Webhooks** → **Add New Webhook to Workspace** → pick the alerts channel
→ **Allow** → copy the `https://hooks.slack.com/services/…` URL → store it (value at the hidden prompt,
never on the command line):

```bash
security add-generic-password -s "slack-webhook-${ENV}" -a "ephemera-slack-${ENV}" -w
```

## Step 2 — the glue Lambda (🟢, no gate: no public endpoint exists in this recipe)

`service.aws.md`'s shape minus exposure — the function is invocable by SNS only. Region + `ENV` as your
fleet; TAGS per the movement (`Source=docs/recipes/aws-alerts-to-slack.md`). Run blocks under `bash`.

```bash
set -euo pipefail
export AWS_REGION="${AWS_REGION:-us-west-2}" ENV="${ENV:-dev}" SVC="alerts-notifier"
FN="${SVC}-${ENV}"; ROLE="${FN}-exec"; SECRET_PARAM="/${SVC}/${ENV}/slack-webhook"
ACCOUNT_ID="$(aws sts get-caller-identity --query Account --output text)"
TAGS_KV="Key=ManagedBy,Value=ephemera Key=Source,Value=docs/recipes/aws-alerts-to-slack.md Key=Environment,Value=${ENV}"

# 2a 🟢 webhook → SSM SecureString (observe first; value read from Keychain, never echoed)
if ! aws ssm get-parameter --region "$AWS_REGION" --name "$SECRET_PARAM" >/dev/null 2>&1; then
  WEBHOOK="$(security find-generic-password -s "slack-webhook-${ENV}" -a "ephemera-slack-${ENV}" -w)"
  printf '%s' "$WEBHOOK" | grep -q '^https://hooks.slack.com/services/' || { echo "webhook shape unexpected"; exit 1; }
  aws ssm put-parameter --region "$AWS_REGION" --name "$SECRET_PARAM" --type SecureString \
    --value "$WEBHOOK" --tags $TAGS_KV
fi
aws ssm get-parameter --region "$AWS_REGION" --name "$SECRET_PARAM" \
  --query 'Parameter.Type' --output text | grep -q SecureString && echo "✔ param SecureString"

# 2b 🟢 execution role — IAM: under a credential broker run these via e.g. `aws-vault exec <profile> --no-session --`
cat > /tmp/ephemera-trust.json <<'JSON'
{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"Service":"lambda.amazonaws.com"},"Action":"sts:AssumeRole"}]}
JSON
ROLE_ARN="$(aws iam get-role --role-name "$ROLE" --query 'Role.Arn' --output text 2>/dev/null)" || \
ROLE_ARN="$(aws iam create-role --role-name "$ROLE" --assume-role-policy-document file:///tmp/ephemera-trust.json \
  --tags $TAGS_KV --query 'Role.Arn' --output text)"
aws iam attach-role-policy --role-name "$ROLE" --policy-arn arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
aws iam put-role-policy --role-name "$ROLE" --policy-name "${FN}-bindings" --policy-document \
  "{\"Version\":\"2012-10-17\",\"Statement\":[{\"Sid\":\"ReadWebhook\",\"Effect\":\"Allow\",\"Action\":\"ssm:GetParameter\",\"Resource\":\"arn:aws:ssm:${AWS_REGION}:${ACCOUNT_ID}:parameter${SECRET_PARAM}\"}]}"
aws iam get-role-policy --role-name "$ROLE" --policy-name "${FN}-bindings" --query PolicyName --output text

# 2c 🟢 the function (handler generated here — the artifact is part of the recipe)
BUILD="$(mktemp -d)"   # scratch build dir is fine — nothing later resolves paths relative to it
cat > "$BUILD/lambda_function.py" <<'PY'
import json, os, urllib.request

_WEBHOOK = None

def _webhook():
    global _WEBHOOK
    if _WEBHOOK is None:
        import boto3
        p = os.environ["WEBHOOK_PARAM"]
        _WEBHOOK = boto3.client("ssm").get_parameter(Name=p, WithDecryption=True)["Parameter"]["Value"]
    return _WEBHOOK

def handler(event, context):
    for rec in event.get("Records", []):
        sns = rec.get("Sns", {})
        raw = sns.get("Message", "")
        try:
            m = json.loads(raw)
            state = m.get("NewStateValue", "?")
            emoji = {"ALARM": ":rotating_light:", "OK": ":white_check_mark:"}.get(state, ":bell:")
            text = "%s *%s* -> *%s*\n%s" % (emoji, m.get("AlarmName", "?"), state, m.get("NewStateReason", ""))
        except (ValueError, TypeError):
            text = ":bell: %s\n```%s```" % (sns.get("Subject") or "SNS notification", raw[:1000])
        req = urllib.request.Request(_webhook(), data=json.dumps({"text": text}).encode(),
                                     headers={"Content-Type": "application/json"})
        with urllib.request.urlopen(req, timeout=10) as r:
            if r.status != 200:
                raise RuntimeError("slack webhook returned %s" % r.status)
    return {"ok": True}
PY
( cd "$BUILD" && zip -q -X fn.zip lambda_function.py )
if aws lambda get-function --region "$AWS_REGION" --function-name "$FN" >/dev/null 2>&1; then
  aws lambda update-function-code --region "$AWS_REGION" --function-name "$FN" --zip-file fileb://"$BUILD/fn.zip" >/dev/null
  aws lambda wait function-updated --region "$AWS_REGION" --function-name "$FN"
else
  for i in 1 2 3 4 5; do   # ⏳ fresh-role IAM propagation (the proven idiom; fired live in the dogfood fleet)
    aws lambda create-function --region "$AWS_REGION" --function-name "$FN" \
      --runtime python3.12 --role "$ROLE_ARN" --handler lambda_function.handler \
      --zip-file fileb://"$BUILD/fn.zip" --timeout 15 --memory-size 128 \
      --environment "{\"Variables\":{\"WEBHOOK_PARAM\":\"${SECRET_PARAM}\"}}" \
      --tags "{\"ManagedBy\":\"ephemera\",\"Source\":\"docs/recipes/aws-alerts-to-slack.md\",\"Environment\":\"${ENV}\"}" \
      >/dev/null 2>&1 && break
    [ "$i" = 5 ] && { echo "create-function kept failing"; exit 1; }
    echo "role not yet assumable — retry ${i}/5"; sleep 5
  done
  aws lambda wait function-active --region "$AWS_REGION" --function-name "$FN"
fi
# ✔ env carries the parameter NAME only — the negative that keeps the webhook a secret
aws lambda get-function-configuration --region "$AWS_REGION" --function-name "$FN" \
  --query 'Environment.Variables' --output json | grep -q 'hooks.slack.com' \
  && { echo "WEBHOOK VALUE IN CONFIG"; exit 1; } || echo "✔ function config clean (name only)"
```

## Step 3 — the channel + alarms

Apply [`observability.aws.md`](../../observability.aws.md) as written — `WATCH_LAMBDA=alerts-notifier-${ENV}`
(the twist) plus whatever your fleet needs (`WATCH_QUEUE`/`WATCH_TABLE`/`WATCH_API`); `EMAIL_ALERT` empty or
a real inbox for the backstop. Its §1 verify, §3 positive+negative checks, and §5a tag drift all apply
unchanged.

## Step 4 — the seam: subscribe the notifier to the topic

```bash
TOPIC_ARN="arn:aws:sns:${AWS_REGION}:${ACCOUNT_ID}:${OBS_NAME:-ephemera-alerts}-${ENV}-topic"
FN_ARN="$(aws lambda get-function --region "$AWS_REGION" --function-name "$FN" --query 'Configuration.FunctionArn' --output text)"
# permission BEFORE subscribe, scoped to exactly this topic (never --principal sns.amazonaws.com unscoped)
aws lambda get-policy --region "$AWS_REGION" --function-name "$FN" --query Policy --output text 2>/dev/null \
  | grep -q 'sns-ephemera-alerts' || \
  aws lambda add-permission --region "$AWS_REGION" --function-name "$FN" \
    --statement-id sns-ephemera-alerts --action lambda:InvokeFunction \
    --principal sns.amazonaws.com --source-arn "$TOPIC_ARN" >/dev/null
SUB="$(aws sns list-subscriptions-by-topic --region "$AWS_REGION" --topic-arn "$TOPIC_ARN" \
  --query "Subscriptions[?Endpoint=='${FN_ARN}'].SubscriptionArn | [0]" --output text)"
{ [ "$SUB" = "None" ] || [ -z "$SUB" ]; } && SUB="$(aws sns subscribe --region "$AWS_REGION" \
  --topic-arn "$TOPIC_ARN" --protocol lambda --notification-endpoint "$FN_ARN" \
  --query SubscriptionArn --output text)"
# ✔ lambda subscriptions confirm instantly — a real ARN, not PendingConfirmation (unlike email)
printf '%s' "$SUB" | grep -q '^arn:aws:sns:' && echo "✔ subscription live"
```

## Acceptance — the strut test IS the end-to-end proof

Run `observability.aws.md` §5b (force `ALARM` → assert *Successfully executed action*, time-bounded →
force `OK`). With this recipe wired, that test stops being topic-deep and becomes **channel-deep**: a 🚨
message then a ✅ message appear in the Slack channel. Assert the machine half too — the notifier ran
clean for both (Slack accepted both posts, since the handler raises on non-200):

```bash
SINCE=$(( ($(date +%s) - 300) * 1000 ))
EV="$(aws logs filter-log-events --region "$AWS_REGION" --log-group-name "/aws/lambda/${FN}" --start-time "$SINCE" --output json)"
[ "$(printf '%s' "$EV" | grep -c '"START RequestId')" -ge 2 ] && \
! printf '%s' "$EV" | grep -qE '\[ERROR\]|Task timed out|Traceback' && echo "✔ notifier: ≥2 invocations, 0 errors"
```

The human half — "I can see both messages in the channel" — is the recipe's real contract; get the go
recorded before calling it live.

## Candor — gates, noise, cost

- **Gates:** the two Slack OAuth clicks (step 1) are the only 🔴s; the notifier has **no public endpoint**
  (SNS-invoke only, permission scoped to one topic ARN), so `service.aws.md`'s exposure 🔴 never applies.
  Teardown is the usual 💥.
- **Noise:** `observability.aws.md`'s two-notifications-per-incident decision (ALARM + recovery) now lands
  in a channel humans read — that's the point, but it doubles per-incident volume; drop `--ok-actions`
  there if your channel disagrees.
- **Cost:** ~$0 — alarms under the 10-free tier, SNS→Lambda delivery free, invocations negligible, one SSM
  standard parameter free.
- **Delivery semantics:** SNS retries a failing Lambda subscriber (async, ~3 attempts) then **drops** —
  a DLQ on the notifier is the named omission if alert loss is unacceptable.
- **AWS Chatbot** is the managed alternative (no glue Lambda) — its Slack workspace authorization is
  console-gated and coarser-grained; named so it's a choice, not an oversight.

## Teardown (💥 — reverse order, each plan's own section)

1. **Seam:** `aws sns unsubscribe` the notifier's subscription; `aws lambda remove-permission
   --statement-id sns-ephemera-alerts`.
2. **observability.aws.md** teardown as written (alarms tag-checked → budget → topic).
3. **Glue:** delete function → SSM param → role (detach managed + delete inline first; IAM ⇒
   `--no-session` under a broker) — each ownership-checked via `ManagedBy=ephemera`.
4. **messaging.slack.md** teardown (delete the app in its App Config UI — the webhook dies with it) +
   `security delete-generic-password -s "slack-webhook-${ENV}" -a "ephemera-slack-${ENV}"`.

Keep-it-live is equally legitimate — steps 1–4 of this recipe are a permanent alerting spine; re-run
`observability.aws.md` §3 to grow the watcher set as the fleet grows.

## Deliberately not a meta-plan

Same stance as [`local-slack-agent.md`](./local-slack-agent.md): this is a doc sequencing three standalone
plans plus ~40 lines of glue no plan owns. If the notifier Lambda earns standalone life (multiple channels,
message routing, severity filtering), it graduates into a plan; until then the recipe is the lighter answer.