Skip to main content
🗓️ Schedule discussionOpens Google Calendar — add attendees there.

Bounded-by-construction pagination primitive + lint enforcement

OpenAuthor: @relbnsOwner: openEffort: L

Problem​

A production bug came from a data migration that loaded a whole collection into memory and iterated it one document at a time instead of batching. It passed local tests on a tiny dataset and only failed at production volume. The deeper, recurring root cause is structural: nothing guarantees a list read is bounded. Pagination is a documented convention ([pagination.md], CLAUDE.md), not an enforced mechanism. Verified against backend-services on 2026-06-17:

  • The shared primitive getManyAndCount (libs/helpers/get-many-and-count.ts) is not bounded by default: 
    if (limit !== undefined) { queryBuilder = queryBuilder.limit(limit); }
    const records = await queryBuilder.exec();
    Omit limit and it returns the entire collection.
  • It is used 27 times, versus 236 raw multi-document .find( calls in *.service.ts. The raw, unguarded find is the de-facto default — there is no single path.
  • The name even collides: several services define their own getManyAndCount methods (some TypeORM FindManyOptions), and CLAUDE.md already bans "generic getMany" — so there is existing intent to lean on.
  • A lint rule cannot cleanly detect "this filtered find returns too many rows": a 5-row find({ orgId }) is mechanically indistinguishable from a 5M-row one without volume knowledge. So "detect the bad pattern" is inherently noisy; "require a safe primitive" is deterministic.

Proposal​

Flip the approach from detect the bad pattern (heuristic, noisy) to require a safe primitive (deterministic — the pit of success). Make the one shared list-read path safe by default and route every list read through it.

  1. Harden the primitive — bounded by construction. getManyAndCount defaults and clamps limit to PaginationEnum.MAX_PAGE_SIZE (already defined as 200). Even if a caller forgets limit, the worst case is one capped page — never the whole collection. Genuine full scans (export, backfill) take an explicit, greppable/lintable opt-out ({ unbounded: true }, or a sibling cursor/stream helper) so the intent is loud rather than accidental.
  2. Enforce the single path with a lint rule (in @bewith-dev/eslint-plugin): a raw multi-document .find( awaited in a service without .limit()/.cursor() is flagged → "route through getManyAndCount or add an explicit bound." This is low-false-positive because the fix is uniform and the default-cap makes even unflagged cases safe. Ship warn-first (the eqeqeq pattern), promote to error per repo once clean.

This is the make-it-impossible layer beneath the already-built audit layer: the scalability-reviewer agent + scale-review policy (bewith-docs#225) and the three AST lint rules no-unbounded-find / no-collection-load-in-memory / require-batched-backfill (eslint-plugin#52). Those catch the unambiguous mechanical cases and provide the judgment layer; this proposal removes the foot-gun entirely for the common list-read path.

Why now / Why us​

The incident already happened, and the audit layer (agent + policy + lint) shipped as PRs in the same effort — but all of it is detection. Detection still lets the next unbounded list reach review and rely on a human noticing. We own the shared helper and the eslint-plugin, so we can close the loop at the source: a primitive that is safe by default is the only thing that makes "every list is bounded" true rather than aspirational. Doing it now, while the context is fresh and the audit layer is landing, is cheaper than re-learning it from the next incident.

Sketch of approach​

  • Phase 1 — harden the primitive (backend-services). Add default + clamp of limit to MAX_PAGE_SIZE in getManyAndCount; add the explicit full-scan opt-out (cursor/stream helper or { unbounded: true }). Audit the 27 current callers first — some may rely on "no limit = everything"; mark the legitimate full-scans with the opt-out. This is a behavioral change to a shared helper → database + architect sign-off.
  • Phase 2 — lint enforcement (eslint-plugin). Add the rule that steers a raw unbounded multi-doc .find( in a service to the primitive. Ship warn-tier; it drives the migration of the 236 raw finds incrementally.
  • Phase 3 — promote to error. Once a repo's violations are cleared, flip the rule to error there.
  • Naming cleanup (opportunistic): reconcile the per-service getManyAndCount methods with CLAUDE.md's "no generic getMany" rule so the sanctioned primitive is unambiguous.

Open questions​

  • Default cap value: MAX_PAGE_SIZE (200) for the clamp, or a lower default page size with 200 as the hard ceiling?
  • Opt-out shape: an options flag ({ unbounded: true }) vs. a separate named helper (streamAll / cursor) — which reads more clearly and lints more reliably?
  • Does the same hardening belong on the Aurora/TypeORM list path, or is this Mongo-first?
  • Effort is L and it changes a shared helper across every list endpoint — should this graduate to an RFC rather than stay a proposal?
  • Scope of the Phase-2 lint rule: services only, or also repositories/gateways?
  • bewith-docs#225 — scalability-reviewer agent + scale-review policy (the audit layer, planning side)
  • eslint-plugin#52 — no-unbounded-find / no-collection-load-in-memory / require-batched-backfill (the audit layer, code side)
  • backend-services rules: pagination.md, service-methods.md, coding-principles-from-pr-reviews.md
  • backend-services libs/helpers/get-many-and-count.ts, libs/helpers/pagination.helper.ts, libs/enums/pagination.enum.ts

Triage notes​

(none yet)