Design Doc: Integration Lifecycle Management

DraftAuthor: @relbnsClickUp: CU-86ca6745p

Discussion document

This Design Doc is a proposal for a design review, not a final decision. It captures the design rationale for the Integration Lifecycle Management feature so the approach can be reviewed before the build starts. Background, problem statement, and goals are carried in the Appendix (they predate a PRD — see related_prd).

TL;DR

Operators today cannot add or edit an integration without an engineer doing a manual MongoDB write in each environment. This design delivers operator self-service: an editable source URL from the Support Tool, a single unambiguous mode replacing the overlapping status / disableCron flags, and auto-seeding of new integrations on merge. It is deliberately a v1.5 bridge built on the existing two-collection layout (organizations-providers-config + organizations-api-mapping) — the data model stays strict (IDs fixed up-front in the template, sync insert-only) so nothing has to be unwound when the unified integrations-config collection lands (CU-86ca338hc, months out). The auth backbone (system tokens) is already merged in #2769.

1. Approach & Rationale

All three capabilities sit on one shared spine: an OrganizationProviderModeEnum (off / manual / auto) on the config schema, replacing status / disableCron, with cron/manual ingestion gated by mode and a back-compatibility normalization for rows that still carry the old fields. Two mode origins, two defaults: a newly synced row is inserted off (dormant until Support activates it), while an existing row that has no mode is normalized by its old flags — a running integration becomes auto, disableCron === true becomes manual, archived becomes off (see §11). The pending_setup enum value that exists in the current branch is removed: IDs are always known at sync time, so there is no incomplete/pending state.

(a) Editable source URL. Paste a new URL, preview it against the live provider, replace it, or roll back to the spec original. Backed by an originalRequestParams snapshot taken on first edit, preview/replace/rollback endpoints with SSRF guards, and an Edit-URL dialog in the FE.
(b) Auto-sync of new integrations. On merge to develop / master, a GitHub workflow seeds new integrations into mongo through an internal-gateway endpoint that inserts only if the row does not exist. The body is the existing, locally-tested spec files copied verbatim; a small per-integration template.json supplies only the per-stage IDs, overlaid onto the copied spec, inserted in mode: off. An environment with no IDs is skipped. Insert-only — updates to an already-synced integration do not propagate (deferred to CU-86ca4vtq4).
(c) System-token auth. A system-tokens collection in auth-service and a SystemTokenGuard in @libs/guards with scope-based access (integration-sync). Tokens are minted by bewith ci system-token create. The auth backbone for (b). Already merged in #2769.

Editable IDs removed. An earlier cut made org / community / leader IDs editable from the dashboard (with a Run-check). That is dropped entirely — mechanism and terminology: IDs are fixed up-front in template.json and copied to the right environment at sync time, so there is no in-dashboard ID editor, no Run-check, and no IDs support endpoint. Correcting an ID on a live row stays a manual document edit until the unified collection lands.

Two adjacent repos. The migration (§11) is authored in the migrations-tool repo (not backend-services/scripts/), and registering a new integration so it actually runs on a schedule requires a PR to the sqs-consumer repo — today a manual, easily-forgotten step that §8 addresses.

Why this shape. v1.5 keeps the data model strict so the orchestration moves cleanly onto the unified IntegrationConfigService later; writes go through validated service code (not direct-to-mongo from CI); and the skip-on-update is made loud rather than silent. Alternatives are summarized in §14.

2. Affected Repos

Repo	What changes
`backend-services`	Shared `mode` enum (drop `pending_setup`) + run-gating + back-compat normalization; editable-URL snapshot field + preview/replace/rollback endpoints with SSRF guards; templates-sync internal-gateway endpoint + service; GitHub sync workflows. The `system-tokens` collection + `SystemTokenGuard` in auth-service (an app in this monorepo) is already merged in `#2769`.
`migrations-tool`	One Mongo migration normalizing `status` / `disableCron` → `mode` on `organizations-providers-config` (existing running rows → `auto`). Authored here, not in `backend-services/scripts/`.
`sqs-consumer`	Per-integration cron registration (`cronjobs.config.ts` entry + `cronjobs.service.ts` `addCronJob`) so a synced integration actually runs on schedule. The integration-creation skill opens this PR automatically (§8) — interim, until full automation.
`support-tool-frontend`	Edit URL dialog, and a Mode dropdown (color-coded current state) in the integrations dashboard.

3. Architecture

Two choices are visible above. The internal-gateway endpoint + system token (rather than the GitHub workflow writing to mongo directly) keeps prod DB credentials out of CI and routes every write through validated service code. The existing two collections are kept (rather than introducing the unified collection now) so v1.5 ships without blocking on the migration; the read/write logic is structured so it lifts onto the unified model later.

4. Data Model

No new domain collection is introduced; the change is to the existing organizations-providers-config schema (the mode field) plus an originalRequestParams snapshot for URL editing (a). The system-tokens collection (c) is already merged.

Collection	Field changes	Notes / constraints
`organizations-providers-config`	Add `mode: OrganizationProviderModeEnum` (`off` \| `manual` \| `auto` — `pending_setup` dropped); remove (after normalization) `status` + `disableCron`. Add `originalRequestParams` (snapshot of the spec original, taken on first URL edit).	Unset `mode` on an existing row normalizes by old flags — running → `auto`, `disableCron === true` → `manual`, `archived` → `off`. A newly synced row is `off`. Org / community / leader IDs remain required (non-null), set from the template.
`organizations-api-mapping`	No structural change.	`defaultValues.createdBy` carries the leader ID (overlaid by sync per stage).
`system-tokens` (auth-service)	Already merged (`#2769`) — token hash + scope (`integration-sync`).	Minted via `bewith ci system-token create`.

Mode lifecycle (run-gating is derived from this field):

Migrations / backfills required: Yes — a one-time migration (authored in the migrations-tool repo, not backend-services/scripts/) normalizes existing rows from status / disableCron into mode (running → auto), plus runtime back-compat normalization for any row read before/around the migration. See §10 and §11.

5. API / Contracts

Method + Path	Purpose	Request	Response
`POST /internal/integrations/sync-templates`	Seed new integrations (insert-only)	Bearer system-token (`scope=integration-sync`); spec files + per-stage `template.json` IDs per folder	`{ inserted, skipped, warnings }` summary
Mode-update support endpoint (internal-gateway)	Set an existing row's `mode` (`off` / `manual` / `auto`) from the dashboard dropdown	existing config row id + new `mode`	updated row
Editable-URL endpoints: preview / replace / rollback (internal-gateway)	Preview a new source URL against the live provider, replace it, or roll back to the `originalRequestParams` snapshot	row id + candidate URL	preview payload / updated row

Exact route paths and request/response DTOs for the mode-update and editable-URL endpoints are pinned at the build stage (see §15). The sync RPC is SYNC_INTEGRATION_TEMPLATES.

Naming (review comment): the service method that does the deep validation and moves a single template into the collection must be named for that intent — e.g. validateAndInsertIntegration — not a generic syncOne. The generic-to-specific boundary is where the name must become explicit.

Auth scopes / roles per endpoint:

Endpoint	Auth
`POST /internal/integrations/sync-templates`	Bearer system-token, `SystemTokenGuard`, scope `integration-sync`
Mode-update / editable-URL (Support Tool)	Internal-gateway operator auth (existing Support Tool access)

6. Frontend & Components

Two surfaces in the Support Tool integrations dashboard (support-tool-frontend):

Component	Reuse / new	States handled
Edit URL dialog	dialog + form (preview / replace / rollback)	default / previewing / preview-ok / preview-error / replaced / rolled-back / error
Mode dropdown	inline dropdown on the integrations row, color-coded by current state (`off` / `manual` / `auto`) — replaces the earlier three-button cell to save row space	each mode + switching + error

Design-system anchoring: the Support Tool is not yet mapped to a COMPONENTS.md catalog the way the design-system-backed frontends are, and there is no PRD §7.2 to anchor to. The formal component mapping is resolved at the build stage by frontend-developer; for this backend-led v1.5 it is not the gating concern.

7. Observability Instrumentation

No PRD §8.1 analytics events are defined for this v1.5 — not relevant as product analytics. Operational signal instead:

Signal	Where it fires	Payload
Per-integration sync outcome	templates-sync service → workflow	inserted / skipped — already exists, not updated / skipped — env IDs missing / warnings; surfaced as a PR comment that @-mentions the author + an Actions job summary
Structured logs	sync insert, URL preview/replace/rollback, mode change	unique message + context (`organizationId`, `communityId`, provider) per the repo error-logging rules

Run-tracking / health columns (lastRunAt / lastRunStatus) are deferred — removed from #2720, to be re-introduced on the unified model (CU-86ca338hc).

8. Integration Points & Dependencies

Auto-sync: PR merge → workflow → endpoint → mongo

Auto-sync workflow model — the sync runs in layered scopes, behind a path filter so it never fires on unrelated backend changes:

Trigger — paths: ['specs/integrations/**'] on both events (the whole folder, not just template.json, so editing a body spec still produces a signal).
On a PR (pull_request) — runs dry-run and posts the preview comment (what would be inserted / skipped-as-existing), pre-merge.
On merge (push to develop / master) — runs the real apply. The prod apply is gated behind a GitHub Environment with a required approver (see §11).
Processing is send-all — the workflow sends one HTTP POST carrying every integration's template; the backend sweeps them and inserts the missing ones (idempotent and self-healing; a sync that failed once is retried on the next run). Insert-only caps the write surface to genuinely-new rows. It is one request, not N — the cost is N existence-checks in mongo, not a burst of HTTP calls. Open (review comment): quantify N at today's scale and weigh send-all vs diff-scoped processing — see §15 Q3.
The PR comment is diff-scoped to the folders changed in that PR (and @-mentions the author), so the signal stays relevant even though processing is send-all.

Rationale: keeping the insert-if-not-exists rule in the backend (explicit, unit-tested) is less fragile than computing a correct git diff in CI for the write path; the diff is used only to scope the human-facing signal.

Registering the integration to actually run (sqs-consumer). Seeding the row (mode: off) does not make it run — a separate PR to the sqs-consumer repo adds the cron entry (cronjobs.config.ts) + addCronJob wiring (cronjobs.service.ts), keyed by the prod organizationId. Today this is a manual step that gets forgotten; the integration-creation skill (city-events-api-integration) is extended to open this PR + a ClickUp ticket automatically — interim, until the unified model automates it.

Dependency	Why	Failure-mode handling
GitHub Actions	trigger sync on merge / PR	dry-run on PR; apply on merge; prod behind approver
internal-gateway + `SystemTokenGuard`	validated write path	reject on missing/invalid token or scope
auth-service `system-tokens`	token verification	per-env tokens (separate mongos); rotation runbook
MongoDB (providers-config + api-mapping)	the integration rows; per-env via the workflow base URL (prod → US, staging → IL)	insert-only; reconcile on half-applied insert (§9)
Live provider endpoint	URL preview (a)	SSRF host guard; reject plain `http:`
`sqs-consumer` repo	registers the cron so a synced integration runs on schedule	skill opens the PR automatically (interim); keyed by prod `organizationId`

Cross-repo coordination / merge order — see the 5-PR split in §13. backend-services ships the mode foundation and endpoints first; support-tool-frontend consumes them.

9. Failure Modes & Error Handling

Scenario	Technical handling
SSRF on URL edit	Preview guards the host; the replace/rollback write paths reuse the same host check and reject plain `http:`.
Half-applied insert across two collections (config inserts, api-mapping fails → dangling row)	A transaction, or an idempotent reconcile on the next send-all sweep. Resolution path in §15.
Sync: environment has no IDs in `template.json`	Skip the env with a warning — never insert an incomplete row. Surfaced in the sync summary (not silent).
Sync: row already exists	Skip — not updated (`CU-86ca4vtq4`). Surfaced as an author-tagged PR comment so the dev knows the spec edit did not propagate (see Appendix B).
Unknown / unset `mode` on read	Normalize by old flags — running → `auto`, `disableCron === true` → `manual`, `archived` → `off` (preserves the row's prior running state; a newly synced row is `off`).

10. Breaking Changes

mode replaces status + disableCron (and the pending_setup enum value is dropped). Requires the normalization migration (§11, authored in migrations-tool) and runtime back-compat for rows read before the migration completes. No external API contract breaks; this is an internal schema/runtime change coordinated across backend-services + migrations-tool.

11. Rollout

Feature flag: the mode field itself is the runtime gate — a newly synced integration is off (dormant) until Support activates it; an existing migrated row keeps its prior state (running → auto). No separate flag needed.
Migration / backfill: one-time migration authored in the migrations-tool repo (Mongo script under src/scripts/) normalizing status / disableCron → mode (running → auto), plus runtime normalization for the transition window. Run per env via the tool's MONGODB_URI.
Regions / per-env write isolation: each sync workflow targets its own *_INTERNAL_GATEWAY_BASE_URL (+ token) → the internal-gateway of that env → that env's Mongo. So prod writes to US, staging to IL purely via the base URL — no region logic in code. Verify once in GitHub Environments that the develop / prod base-URL variables point at the right regions (the values live in repo settings, not the code).
Phased rollout: sync prod apply is gated behind a GitHub Environment with a required approver — a write to the prod collection is one approved click, not an automatic side-effect. develop applies automatically.
Rollback plan: set the row mode to off to stop ingestion without a code deploy; the URL edit keeps originalRequestParams so a source change rolls back in one action.

12. Testing Strategy

Unit: insert-if-not-exists rule (inserted / skipped-exists / skipped-missing-IDs); mode normalization (status/disableCron → mode; running → auto, disableCron === true → manual, archived → off); run-gating per mode; SSRF host check (rejects plain http: and disallowed hosts); template-overlay (per-stage IDs onto verbatim spec). The migration itself is unit-tested in the migrations-tool repo.
Integration / E2E: sync endpoint end-to-end through SystemTokenGuard; editable-URL preview/replace/rollback against a stub provider; half-applied-insert reconcile.
Manual / QA: Support Tool surfaces (Edit URL dialog, Mode dropdown) across all states; prod-approver gate on the sync workflow; the author-tagged PR comment renders correctly.

13. Task Breakdown

The umbrella task is CU-86ca6745p. Every PR is reviewed by Baruch, in the read/approve order below — small, single-purpose PRs that build on each other. The earlier draft #2720 (backend) + #342 (FE) are kept as draft source-of-copy, not merged (see the PR-plan companion doc).

Order	Task	Repo	Effort	Risk	Depends on	CU-id
1	Mode foundation — enum (drop `pending_setup`) + run-gating + back-compat normalization	`backend-services`	M	Med	—	`CU-86ca67470`
2	Mode migration — normalize `status` / `disableCron` → `mode` (running → `auto`)	`migrations-tool`	S	Med	#1	`CU-86ca67479` (repurposed)
3	Editable URL + SSRF — snapshot field, preview/replace/rollback endpoints	`backend-services`	M	Med	#1	`CU-86ca6747d`
4	Auto-sync (insert-only) + templates-sync endpoint — incl. clear sync-fn naming, prod-approver gate	`backend-services`	L	High	#1	`CU-86ca6747p`
5	FE — Edit URL dialog + Mode dropdown (color-coded)	`support-tool-frontend`	M	Low	#1, #3, #4	`CU-86ca6747x`
6	Creation-skill: auto-open `sqs-consumer` PR + ClickUp ticket (interim)	skill + `sqs-consumer`	S	Low	—	TBD

Dropped: Editable IDs (was PR 2) — removed per the review; IDs are fixed in the template. Its CU-86ca67479 is repurposed for the migration (task #2).

Already merged / out of band, review re-assigned to Baruch: system-token auth (Aviad's task — #2769 merged + any follow-up); per-env token provisioning (CU-86ca5f7mw).

Effort scale (rough): XS = under 1 day · S = 1-3 days · M = up to 1 week · L = 1-2 weeks · XL = up to 1 month.

14. Risks & Alternatives Considered

Risk: SSRF on URL edit → Mitigation: shared host guard across preview + write paths; reject plain http:.
Risk: Half-applied insert across two collections → Mitigation: transaction or idempotent reconcile on the next send-all sweep.
Risk: Single shared CI token per env → Mitigation: per-env tokens (separate mongos), scope limiting, rotation runbook (bewith ci system-token rotate).
Risk: Dev expects a spec edit to propagate, but sync is insert-only → Mitigation: author-tagged PR comment stating exactly how to apply a change (see Appendix B).
Alternative: Nullable IDs + a pending_setup seed state + a dashboard editable-IDs flow → Rejected / removed: per-stage IDs are known at seed time (they live in template.json), so nullability buys nothing and would hold incomplete docs + index edge-cases in prod. Chosen instead: IDs required, set from the template; env without IDs is skipped. The pending_setup enum value and the editable-IDs flow are both removed from the code.
Alternative: Wait for the unified integrations-config collection (CU-86ca338hc) → Rejected: months out; Ops needs self-service now. v1.5 keeps the model strict so logic moves cleanly to IntegrationConfigService later.
Alternative: Keep manual mongo inserts → Rejected: slow, serialized on R&D, error-prone, staging/prod drift.
Alternative: Direct-to-mongo from the GitHub workflow using DB-URI secrets → Rejected: prod DB credentials in CI writing straight to prod is too high-risk. Replaced by system-token + internal-gateway endpoint (writes go through validated service code).
Alternative: Build update-existing / smart-merge now → Deferred (CU-86ca4vtq4): conflict semantics deserve their own design; insert-only is safe and small.
Alternative: Diff-scoped sync (process only the templates changed in the PR) → Kept send-all for now: it is self-healing (a once-failed sync retries on the next run) and the write surface is already capped by insert-only. Revisit if the integration count makes the per-sweep existence-checks costly (§15 Q3).

15. Open Questions for the Build Stage

#	Question	Resolution path
1	Transaction vs idempotent reconcile for the two-collection insert	spike during task #4
2	Exact route paths + DTOs for the editable-URL + mode-update endpoints	pin during tasks #3 / #4
3	Send-all vs diff-scoped sync — quantify per-sweep request / mongo-lookup volume at today's integration count; decide whether diff-scoping is worth the CI-diff fragility	analyze during task #4
4	Support Tool component mapping (no `COMPONENTS.md` today)	frontend-developer during task #5
5	Per-env token provisioning rollout	`CU-86ca5f7mw`

16. Definition of Done

Feature-level DoD — checked once across the whole feature. PRD-anchored items are marked n/a because this v1.5 was authored as a design rationale ahead of a PRD (see related_prd).

Appendix — additional context

Content carried from the design rationale that has no direct home in the template above.

A. Background, problem statement & goals

Current state. An integration is defined by two MongoDB documents in the integrations service: a row in organizations-providers-config (which provider, which org/community, run config) and a row in organizations-api-mapping (how the provider payload maps to our event shape). Today both are checked into the repo as spec JSON under specs/integrations/<name>/ (organizations-providers-config.json, organizations-api-mapping.json). Putting one live is manual: a developer merges the spec PR, the service deploys, then someone hand-inserts the two documents in staging, verifies, then again in production. Editing an existing integration (a wrong org/community/leader ID, a changed source URL) is the same — a manual mongo write, per environment, by an engineer. There is no operator-facing path. Runtime control is split across two overlapping flags: status (active/archived, the per-provider kill-switch from CU-86c9yaevr) and disableCron (so Support can choose whether the cron runs). Together they answer "is this running, and how?" ambiguously.

Problem.

Ops cannot self-serve — every add or edit needs an engineer with prod mongo access; slow, serialized on R&D, error-prone (wrong ID, forgotten community, prod/staging drift).
Two overlapping flags make "is this integration running?" ambiguous.
The long-term cure is a single unified integrations-config collection (CU-86ca338hc) merging both documents — months out.

This feature is v1.5: it delivers operator self-service and removes the manual mongo step on the existing two-collection layout, keeping the data model strict so nothing unwinds when the unified collection lands.

Goals.

Operators add and edit integrations from the Support Tool — no manual mongo writes, no per-environment hand-inserts.
One unambiguous mode replacing status + disableCron, with run-gating derived from it.
New integrations seed themselves automatically on merge, fully formed (real IDs, not placeholders), in a controlled non-running state.
Do all of the above without blocking on the unified-collection migration.

B. The role of `template.json` and developer awareness

The spec files are the source of record for an integration's body — organizations-providers-config.json and organizations-api-mapping.json, exactly as authored and tested locally. The sync copies them verbatim. template.json is a small per-integration overlay holding only the per-stage IDs (staging / prod → organizationId, defaultCommunityId, leaderId); the sync substitutes those onto the copied spec (config organizationId / defaultCommunityId, mapping defaultValues.createdBy) so the only thing that differs per environment is the IDs. This keeps "what was tested" identical to "what lands". (New convention — no template.json files exist yet.)

The sync is insert-only and one-directional (repo → mongo, new rows only). It does not update an existing row and never writes back to the repo. This creates an awareness risk: a developer who edits a spec file in a PR may expect the change to propagate, but it will be skipped. To make that explicit, the workflow posts the per-integration outcome (inserted / skipped — already exists, not updated / warnings) as a PR comment that @-mentions the PR author, plus an Actions job summary. The message states how to apply a change to a live integration: fields editable in the Support Tool (source URL, mode) → change them there; any other field (org / community / leader IDs, mapping, request/response params, default values, etc.) → a manual edit of the document in the collection, since those are not editable in the Support Tool. Auto-applying spec changes with conflict handling is the deferred CU-86ca4vtq4 work.

C. Settled decisions

Default mode = off. A new or unknown-mode integration stays dormant; Support activates it deliberately. No auto-run on seed.
Updates are not auto-applied in v1.5 (insert-only). A spec edit in a PR produces an author-tagged preview comment, not a live update; editable fields go through the Support Tool and other fields via a manual document edit. Auto-applying updates with a conflict gate is the deferred CU-86ca4vtq4 work.

D. Relationship to other tracks

Lives alongside CU-86ca4gc61 (the in-flight #2720 + support-tool #342), which is left untouched.
DAL / orchestration split out of OrganizationsProvidersConfigService — deferred to CU-86ca4tyn9.
Unified integrations-config collection — CU-86ca338hc (the long-term cure this v1.5 bridges to).
Run-tracking / health columns (lastRunAt / lastRunStatus) — removed from #2720, to be re-introduced on the unified model.

TL;DR​

1. Approach & Rationale​

2. Affected Repos​

3. Architecture​

4. Data Model​

5. API / Contracts​

6. Frontend & Components​

7. Observability Instrumentation​

8. Integration Points & Dependencies​

9. Failure Modes & Error Handling​

10. Breaking Changes​

11. Rollout​

12. Testing Strategy​

13. Task Breakdown​

14. Risks & Alternatives Considered​

15. Open Questions for the Build Stage​

16. Definition of Done​

Appendix — additional context​

A. Background, problem statement & goals​

B. The role of template.json and developer awareness​

C. Settled decisions​

D. Relationship to other tracks​