Provisioning a queue or env var across BeWith environments

Overview

Four deployed environments on two platforms that keep their queues and env vars in different places, plus two ArgoCD branches inside the one EKS config repo. Several infrastructure repos look authoritative and are dead.

The failure to prevent: covering the greppable half — Terraform and main — and shipping. The ECS half lives in no repo, and IR staging reads a different branch. Both are silent when you miss them.

Read references/environments-and-repos.md first. It carries the environment→platform→file map, which branch each ArgoCD environment reads, which repos are live with the evidence, and the commands to re-verify. Two of the six infra repos were deprecated weeks apart, one of them mid-2026, so do not answer "where does this live" from memory.

For how these infra repos relate to the app/service repos they deploy (which repo owns which service, what's deprecated org-wide), see the repo-map skill.

Checklist — seven places, not two

• IAC-Terraform environments/dev/infrastructure/9.sqs/9.N.<name>/ — backend.tf, locals.tf, providers.tf, sqs.tf. Copy the nearest sibling; 9.2.sms is the plain FIFO one. This is IR staging (dev_euw1, eu-west-1). Find the next free N by listing the directory — gh api repos/bewith-dev/IAC-Terraform/contents/environments/dev/infrastructure/9.sqs --jq '.[].name' — don't guess past the sibling numbers you've seen.
• IAC-Terraform environments/production/…/9.sqs/9.N.<name>/ — same four files, prod_use1 / us-east-1 / bewith-prod-use1-terraform-iac state bucket. This is US production.
• Kubernetes-deployments on main — conf-values-prod-nls.yaml only. That is what US production reads; leave the conf-values-staging-nls.yaml copy on main alone, editing it changes nothing and just collides with the staging PR.
• Kubernetes-deployments on staging, a second PR — conf-values-staging-nls.yaml. This copy, not the one on main, is what IR staging reads.
• sandbox/localstack/init-sqs.sh — the docker-compose stack everyone develops against. This only creates the queue; each service's own local .env (outside sandbox) has to set its AWS_*_QUEUE_URL to http://127.0.0.1:4566/000000000000/<queue-name> itself — sandbox/docker-compose.yml documents that URL pattern but doesn't set the var for you.
• ECS FR staging — create the queue in eu-central-1 and add the variable to two S3 env files, backend-service-libs/dev/services/.env and sqs-consumer/dev/.env. No repo will remind you.
• ECS FR production — same again, same AWS account, the prod/ copies of both files.

Never add the queue to pulumi/. That project was deleted from main on 2026-08-03 as superseded by IAC-Terraform; it survives on the staging branch and still attracts dependabot PRs, which is exactly why it keeps getting mistaken for the right place. PR #411/#412 — the same PR pair references/environments-and-repos.md cites as the correct two-PR pattern for wiring an EKS env var — is this incident: it wired the Helm side correctly but declared the queue itself under pulumi/infra/config/{prod,staging}.json, which nothing applies. The queue was never created and production erred at ~720/hour until Terraform got it. Getting the two conf-values PRs right is necessary but not sufficient — the queue still has to exist in Terraform.

Getting the queue itself right

• FIFO or standard follows the producer. MessageGroupId is valid only on a FIFO queue, so a producer that sets it forces fifo_queue = true and a .fifo name suffix.
• content_based_deduplication = true is required, not decorative, whenever the producer sets no MessageDeduplicationId — SQS rejects every send otherwise. notifications-handler.sendMessageIdToSQS sets none.
• Read the sibling's name; never derive one. Live names include bewith_dev-whatsapp_queue.fifo (hyphen), notificaitonsCampaigns.fifo (typo, in production) and bewith_service_communication_users (no env prefix). The prefix is bewith_dev_ in eu-west-1 and localstack, bewith_prod_ in us-east-1 — and eu-central-1 follows a different convention entirely: camelCase plus a DEV/PROD suffix (smsQueueDEV.fifo, notificaitonsCampaignsProd.fifo, casing inconsistent between queues). The same logical queue therefore has two unrelated names.
• Copy email or sms, not whatsapp or notifications-campaigns. The whatsapp queues carry a DLQ and a 60s visibility timeout because the provider rate-limits; notifications-campaigns raises MaximumMessageSize to 1MB for batch payloads. Email and sms are the plain per-recipient dispatch queues: their sqs.tf sets nothing beyond fifo_queue and content_based_deduplication, so every other attribute is a bare AWS default — 4-day retention, 30s visibility, 256KB, no DLQ.
• A DLQ is the exception, used only by 9.10.notifications-campaigns and whatsapp. Add one when something would actually drain it.
• Retention diverges between platforms — don't copy the eu-central-1 figure into Terraform. eu-central-1's queues were created by hand with MessageRetentionPeriod 1209600 (14 days); the Terraform siblings above set nothing and get the 4-day AWS default instead. Match the platform you are in rather than trying to reconcile them mid-task.

Applying it

Terraform dispatches, it doesn't push-trigger: both workflows are workflow_dispatch only, so merging the PR applies nothing — you dispatch dev, then dispatch prod separately (prod is Hard-Floor: a human dispatches it). EKS needs a kubectl rollout restart after ArgoCD syncs — the ConfigMap has no checksum annotation, so synced pods still run the old env until restarted. ECS has no IaC at all: copy a live sibling's attributes with get-queue-attributes rather than composing from scratch, and back up each S3 env file before editing it.

Read references/applying-the-change.md for the exact workflow names, inputs, gh/aws/kubectl commands, and the S3 env-file backup sequence — every command above needs the literal syntax from that file, not a reconstruction from memory.

Verification, per environment

1. The queue exists: aws sqs get-queue-url --queue-name <name> --region <region>.
2. The edit landed: re-download each S3 env file and diff it against what you uploaded. There is no PR to review it for you.
3. The variable reached the process: kubectl exec … printenv on EKS; a task started from a revision that references the env file on ECS.
4. sqs-consumer logs start pulling from <name>Queue <PROCESSOR> at boot — it names every queue it polls.

Common mistakes

sqs-consumer is one codebase deployed unchanged to every platform, so its SqsService behaves identically whether the missing var is an EKS ConfigMap key or an ECS env file line: it throws at construction if any queue URL it expects is absent, taking down every queue it polls, not just the new one.

Mistake	What happens
Terraform + main only	IR staging never gets the variable; SqsService throws at construction and the whole consumer stays down.
EKS half only	Both FR environments silently lack it, and FR production is a live ECS environment deploying weekly.
Merged the ConfigMap and moved on	ArgoCD reports synced; pods run the old env until a rollout restart.
Added the URL to backend-service-libs only	The producer can send and nothing consumes. sqs-consumer keeps its own copy of every queue URL in its own env file, and its SqsService throws at construction when one is missing — so the miss shows up as the whole consumer down, not as one idle queue.
Composed the ECS queue's attributes from scratch	It drifts from its siblings — a different visibility timeout or retention than the same queue has in every other environment, with no Terraform to reveal it. Read a sibling with get-queue-attributes and mirror it.
FIFO queue without content-based dedup	Every send fails: "The queue should either have ContentBasedDeduplication enabled or MessageDeduplicationId provided".
Queue name derived from a convention	It does not exist. The live names are inconsistent per environment.
Declared the queue in pulumi/	Nothing applies it; the URL points at a queue that was never created.
Trusted a stale local clone	pulumi/ still looks alive in checkouts older than 2026-08-03. Fetch, and read origin/main.
Forgot sandbox/localstack/init-sqs.sh	Every developer's consumer refuses to boot after they pull.