Skip to main content

obs-logs

Search and analyze production logs to investigate errors, volume, and patterns. Vendor-neutral by mandate, so the skill's method (filter, aggregate, time-range, discover attributes) is what matters and the backend is swappable. The backend is Grafana Loki (+ CloudWatch) โ€” the only one; do not propose adding another. Owned by observability; used by identify-bug.

Whatever the backend, log lines are Winston structured JSON with consistent fields (service, level, status, a correlation id) and low-cardinality labels (env/platform/cluster/service/level/status); ids/urls/errors live in the body. Query on the labels, drill into the body.

Access โ€” how to connectโ€‹

Credentials live in 1Password; read them at use-time (the op CLI reads items even if op whoami says "not signed in"). Never echo the secret values โ€” use them only inside the command (headers/vars).

Grafana โ€” the working backend. Query it first.

  • MCP (preferred): a grafana MCP server (docker run grafana/mcp-grafana -t stdio, GRAFANA_URL=https://platformobs.grafana.net, service-account token) exposes Loki log queries + dashboards. It is registered at user scope, so a new session in any repo loads its tools (MCP loads at session start โ€” a session already running when it was added won't have it). Token: op://MCP-Exposed/Grafana Loki/credential.
  • Direct API (fallback, no MCP): query Grafana Cloud Loki over HTTPS with the service-account token as a Bearer header โ€” LogQL via /loki/api/v1/query_range. Read the token in-process, never echo it. Network egress needs dangerouslyDisableSandbox on the Bash tool. 
    GRAFANA_TOKEN=$(op read "op://MCP-Exposed/Grafana Loki/credential")
    curl -sG "https://platformobs.grafana.net/loki/api/v1/query_range" \
      --data-urlencode 'query={env="production",service="<svc>"} |= "error"' \
      --data-urlencode "start=$(date -u -v-1H +%s)000000000" \
      --data-urlencode "limit=25" \
      -H "Authorization: Bearer $GRAFANA_TOKEN"
    (Confirm the exact Loki datasource/route for the stack with devops-infra if the gateway path differs.)
  • The legacy 1Password "Grafana Staging IR" / "Grafana Prod US" items are UI logins, not API tokens โ€” use the service-account token above.

Which environment's logs live where (confirm current coverage with devops-infra / Infrastructure & Environments ยง8): Grafana Loki + CloudWatch (via loki-cloudwatch-forwarder) covers the stack. Clusters seen in Loki: bewith-prod-cluster, cluster-prod, bewith-dev-cluster, cluster-dev. If a service or window looks uncovered, that is a gap to fix at source โ€” there is no second backend to fall back to.

Tool choiceโ€‹

GoalUse
Raw logs, see structure, pattern discoverya search query (recent entries; cluster by pattern)
Counts, GROUP BY, top-N, rates, time seriesan analyze/aggregate query (do NOT page raw logs to count)

On Grafana Loki that's LogQL โ€” a search is {env="production",service="โ€ฆ"} |= "โ€ฆ" (raw lines; add | json to parse the body), and a count/aggregate is count_over_time({โ€ฆ}[5m]) or sum by (service) (count_over_time({โ€ฆ}[5m])). The decision is the same whatever the backend: search to see, aggregate to count. Don't page raw logs just to count. Keep queries free of PII/secrets.

Query method (backend-agnostic)โ€‹

  • Filter on labels: service:<name>, status:error, env:production (Loki: {env="production",service="โ€ฆ"}).
  • Severity: errors+warns; critical/alert/emergency for the worst.
  • Boolean + exclude: combine services; exclude noise (-status:info, -service:*php*).
  • Wildcards: service:*Controller, service:php*.
  • Time range: start relative (now-1h, now-1d, now-7d); narrow/widen around the incident; use absolute (ISO/Unix-ms) for a known window.
  • Discover attributes: pull all fields on a sample first to learn the custom attributes before filtering on them.

Workflowโ€‹

  1. Frame the question โ€” an error to find, a volume/rate to measure, or a pattern to cluster.
  2. Pick search vs aggregate per the table.
  3. Start narrow (one service, status:error, short window) โ†’ widen as needed.
  4. Report โ€” the query used, the finding (count/pattern/sample lines, redacted of PII), and the likely culprit service + the next drill-down. Feed bug investigations back to identify-bug.

Cross-referencesโ€‹

  • observability โ€” owns the vendor-neutral obs stack + what to emit; this skill reads it.
  • identify-bug โ€” uses this skill when a bug involves production/backend errors.
  • devops-infra โ€” owns the Loki/CloudWatch provisioning + loki-cloudwatch-forwarder.
  • Source: the cursor-rules production-logs skill, generalized to vendor-neutral and retargeted at Grafana.