obs-logs

Search and analyze production logs to investigate errors, volume, and patterns. Vendor-neutral by mandate, so the skill's method (filter, aggregate, time-range, discover attributes) is what matters and the backend is swappable. The org has moved to Grafana — query Grafana Loki (+ CloudWatch) first. Datadog (EU) is legacy: it still holds historical data and is being decommissioned, so reach for it only as a fallback for a service or window Grafana does not yet cover. Owned by observability; used by identify-bug.

Whatever the backend, log lines are Winston structured JSON with consistent fields (service, level, status, a correlation id) and low-cardinality labels (env/platform/cluster/service/level/status); ids/urls/errors live in the body. Query on the labels, drill into the body.

Access — how to connect

Credentials live in 1Password; read them at use-time (the op CLI reads items even if op whoami says "not signed in"). Never echo the secret values — use them only inside the command (headers/vars).

Grafana — the working backend. Query it first.

• MCP (preferred): a grafana MCP server (docker run grafana/mcp-grafana -t stdio, GRAFANA_URL=https://platformobs.grafana.net, service-account token) exposes Loki log queries + dashboards. It is registered at user scope, so a new session in any repo loads its tools (MCP loads at session start — a session already running when it was added won't have it). Token: op://MCP-Exposed/Grafana Loki/credential.
• Direct API (fallback, no MCP): query Grafana Cloud Loki over HTTPS with the service-account token as a Bearer header — LogQL via /loki/api/v1/query_range. Read the token in-process, never echo it. Network egress needs dangerouslyDisableSandbox on the Bash tool.

  GRAFANA_TOKEN=$(op read "op://MCP-Exposed/Grafana Loki/credential")
  curl -sG "https://platformobs.grafana.net/loki/api/v1/query_range" \
    --data-urlencode 'query={env="production",service="<svc>"} |= "error"' \
    --data-urlencode "start=$(date -u -v-1H +%s)000000000" \
    --data-urlencode "limit=25" \
    -H "Authorization: Bearer $GRAFANA_TOKEN"

  (Confirm the exact Loki datasource/route for the stack with devops-infra if the gateway path differs.)
• The legacy 1Password "Grafana Staging IR" / "Grafana Prod US" items are UI logins, not API tokens — use the service-account token above.

Datadog (EU) — legacy fallback only. Historical data, and any service not yet on Grafana.

• Site is datadoghq.eu (EU, not .com); API host https://api.datadoghq.eu.
• Keys: op read "op://RnD-developers/Datadog API/credential" (API key) and op read "op://RnD-developers/Datadog Application Key/credential" (App key). Reads require BOTH headers — DD-API-KEY and DD-APPLICATION-KEY. Use the direct API v2; egress needs dangerouslyDisableSandbox.

  DD_API=$(op read "op://RnD-developers/Datadog API/credential")
  DD_APP=$(op read "op://RnD-developers/Datadog Application Key/credential")
  curl -sG "https://api.datadoghq.eu/api/v2/logs/events" \
    --data-urlencode "filter[query]=service:<svc> status:error" \
    --data-urlencode "filter[from]=now-1h" --data-urlencode "filter[to]=now" \
    --data-urlencode "page[limit]=25" \
    -H "DD-API-KEY: $DD_API" -H "DD-APPLICATION-KEY: $DD_APP"

  Aggregations (counts/group-by/timeseries): POST https://api.datadoghq.eu/api/v2/logs/analytics/aggregate.

Which environment's logs live where (confirm current coverage with devops-infra / Infrastructure & Environments §8): the target is Grafana Loki + CloudWatch (via loki-cloudwatch-forwarder) covering the stack. Datadog historically carried EKS (international) and is being retired as Loki coverage completes. When unsure which store holds a given service/window, check Grafana first, then Datadog.

Tool choice

Goal	Use
Raw logs, see structure, pattern discovery	a search query (recent entries; cluster by pattern)
Counts, GROUP BY, top-N, rates, time series	an analyze/aggregate query (do NOT page raw logs to count)

On Grafana Loki that's LogQL — a search is {env="production",service="…"} |= "…" (raw lines; add | json to parse the body), and a count/aggregate is count_over_time({…}[5m]) or sum by (service) (count_over_time({…}[5m])). On Datadog (legacy) it's GET /api/v2/logs/events (search) vs POST /api/v2/logs/analytics/aggregate (aggregate). The decision is the same: search to see, aggregate to count. Don't page raw logs just to count. Keep queries free of PII/secrets.

Query method (backend-agnostic)

• Filter on labels: service:<name>, status:error, env:production (Loki {env="production",service="…"}; Datadog @environment:production).
• Severity: errors+warns; critical/alert/emergency for the worst.
• Boolean + exclude: combine services; exclude noise (-status:info, -service:*php*).
• Wildcards: service:*Controller, service:php*.
• Time range: start relative (now-1h, now-1d, now-7d); narrow/widen around the incident; use absolute (ISO/Unix-ms) for a known window.
• Discover attributes: pull all fields on a sample first to learn the custom attributes before filtering on them.

Workflow

1. Frame the question — an error to find, a volume/rate to measure, or a pattern to cluster.
2. Pick search vs aggregate per the table.
3. Start narrow (one service, status:error, short window) → widen as needed.
4. Report — the query used, the finding (count/pattern/sample lines, redacted of PII), and the likely culprit service + the next drill-down. Feed bug investigations back to identify-bug.

Cross-references

• observability — owns the vendor-neutral obs stack + what to emit; this skill reads it.
• identify-bug — uses this skill when a bug involves production/backend errors.
• devops-infra — owns the Loki/CloudWatch provisioning + loki-cloudwatch-forwarder.
• Source: cursor-rules bewith-datadog-logs skill, generalized to vendor-neutral; flipped to Grafana-first as the migration off Datadog completed.