obs-logs
Search and analyze production logs to investigate errors, volume, and patterns. Vendor-neutral by mandate, so the skill's method (filter, aggregate, time-range, discover attributes) is what matters and the backend is swappable. The backend is Grafana Loki (+ CloudWatch) โ the only one; do not propose adding another. Owned by observability; used by identify-bug.
Whatever the backend, log lines are Winston structured JSON with consistent fields (service, level, status, a correlation id) and low-cardinality labels (
env/platform/cluster/service/level/status); ids/urls/errors live in the body. Query on the labels, drill into the body.
Access โ how to connectโ
Credentials live in 1Password; read them at use-time (the op CLI reads items even if op whoami says "not signed in"). Never echo the secret values โ use them only inside the command (headers/vars).
Grafana โ the working backend. Query it first.
- MCP (preferred): a
grafanaMCP server (the nativemcp-grafanabinary,GRAFANA_URL=https://platformobs.grafana.net, service-account token) exposes Loki log queries + dashboards. Run it as the binary, not thegrafana/mcp-grafanacontainer - the container form dies at session start whenever the Docker daemon happens to be down, and the daemon is not needed for anything else here. Install it once via the sentinel-triage runbook. It is registered at user scope, so a new session in any repo loads its tools (MCP loads at session start โ a session already running when it was added won't have it). Token:op://MCP-Exposed/Grafana Loki/credential. - Direct API (fallback, no MCP): query Grafana Cloud Loki over HTTPS with the service-account token as a Bearer header โ LogQL via
/loki/api/v1/query_range. Read the token in-process, never echo it. Network egress needsdangerouslyDisableSandboxon the Bash tool.(Confirm the exact Loki datasource/route for the stack withGRAFANA_TOKEN=$(op read "op://MCP-Exposed/Grafana Loki/credential")curl -sG "https://platformobs.grafana.net/loki/api/v1/query_range" \--data-urlencode 'query={env="production",service="<svc>"} |= "error"' \--data-urlencode "start=$(date -u -v-1H +%s)000000000" \--data-urlencode "limit=25" \-H "Authorization: Bearer $GRAFANA_TOKEN"devops-infraif the gateway path differs.) - The legacy 1Password "Grafana Staging IR" / "Grafana Prod US" items are UI logins, not API tokens โ use the service-account token above.
Which environment's logs live where (confirm current coverage with devops-infra / Infrastructure & Environments ยง8): Grafana Loki + CloudWatch (via loki-cloudwatch-forwarder) covers the stack. Clusters seen in Loki: bewith-prod-cluster, cluster-prod, bewith-dev-cluster, cluster-dev. If a service or window looks uncovered, that is a gap to fix at source โ there is no second backend to fall back to.
Tool choiceโ
| Goal | Use |
|---|---|
| Raw logs, see structure, pattern discovery | a search query (recent entries; cluster by pattern) |
| Counts, GROUP BY, top-N, rates, time series | an analyze/aggregate query (do NOT page raw logs to count) |
On Grafana Loki that's LogQL โ a search is {env="production",service="โฆ"} |= "โฆ" (raw lines; add | json to parse the body), and a count/aggregate is count_over_time({โฆ}[5m]) or sum by (service) (count_over_time({โฆ}[5m])). The decision is the same whatever the backend: search to see, aggregate to count. Don't page raw logs just to count. Keep queries free of PII/secrets.
Query method (backend-agnostic)โ
- Filter on labels:
service:<name>,status:error,env:production(Loki:{env="production",service="โฆ"}). - Severity: errors+warns; critical/alert/emergency for the worst.
- Boolean + exclude: combine services; exclude noise (
-status:info,-service:*php*). - Wildcards:
service:*Controller,service:php*. - Time range: start relative (
now-1h,now-1d,now-7d); narrow/widen around the incident; use absolute (ISO/Unix-ms) for a known window. - Discover attributes: pull all fields on a sample first to learn the custom attributes before filtering on them.
Workflowโ
- Frame the question โ an error to find, a volume/rate to measure, or a pattern to cluster.
- Pick search vs aggregate per the table.
- Start narrow (one service, status:error, short window) โ widen as needed.
- Report โ the query used, the finding (count/pattern/sample lines, redacted of PII), and the likely culprit service + the next drill-down. Feed bug investigations back to
identify-bug.
Cross-referencesโ
observabilityโ owns the vendor-neutral obs stack + what to emit; this skill reads it.identify-bugโ uses this skill when a bug involves production/backend errors.devops-infraโ owns the Loki/CloudWatch provisioning +loki-cloudwatch-forwarder.- Source: the cursor-rules production-logs skill, generalized to vendor-neutral and retargeted at Grafana.