Client Logging Guidelines

Status: Recommendation (non-binding) Audience: Teams building applications on the wbsp-platform that deploy to AWS EKS and Lambda Applies to: Application log output destined for the LogWatch aggregation & anomaly-detection platform

Purpose

These guidelines describe how applications should emit log output so that it can be ingested, parsed, correlated, and analysed by LogWatch with no per-application custom parsing. They are written to be referenced from a project's constitution as a recommended principle, not a mandate. Section §9 — When Deviation Is Acceptable defines exactly when a project may diverge.

The single most important outcome is this:

Emit machine-readable, structured logs whose fields map cleanly onto the OpenTelemetry Logs Data Model — either as OTLP, or as JSON on stdout.

LogWatch's anomaly detection (statistical baselines + LogBERT) is only as good as the structure of the data it receives. Unstructured, free-text logs degrade detection quality for every tenant sharing a service baseline, so structure is a shared responsibility rather than a per-team preference.

How LogWatch Ingests Logs

LogWatch accepts four ingestion paths. They are listed in order of preference:

Principle 1 — Prefer OpenTelemetry. Applications SHOULD export logs via OTLP, or emit structured JSON on stdout that a collector converts to OTLP. Free-text and syslog are acceptable fallbacks only when the source cannot do either (§9).

Whichever path you choose, the data ends up normalised to the OpenTelemetry Logs Data Model. Aligning your field names to that model up front means nothing is lost in translation.

1 — Preferred Format: Structured JSON Mapped to the OTel Logs Data Model

Log records SHOULD be JSON objects, one record per line (newline-delimited JSON / JSON Lines), written to stdout (or exported via OTLP, which uses the same field semantics). Each record SHOULD populate the following fields. Names follow the OpenTelemetry Logs Data Model and Semantic Conventions so they require zero remapping.

Minimal compliant record

{
  "timestamp": "2026-06-03T11:22:33.123Z",
  "severity_text": "ERROR",
  "severity_number": 17,
  "body": "payment authorization failed",
  "service.name": "checkout-api",
  "service.version": "2.4.1",
  "deployment.environment": "production",
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "span_id": "00f067aa0ba902b7",
  "attributes": {
    "order.id": "ord_8842",
    "payment.provider": "stripe",
    "http.response.status_code": 402
  }
}

Principle 2 — One line, one event, valid JSON. Each log record SHOULD be a single self-contained JSON object on one line. Never split one event across multiple lines (the notable exception — stack traces — is handled in §8).

Severity mapping (OTel severity_number)

2 — The Golden Rule for EKS and Lambda: Log to stdout/stderr

Principle 3 — Don't manage log files or shippers inside the app. In both EKS and Lambda, applications SHOULD write logs to stdout/stderr and let the platform's collector forward them. Applications SHOULD NOT write to log files on disk, rotate files, or open direct network connections to LogWatch from business code.

This is the Twelve-Factor "logs as event streams" principle, and it is also Kubernetes and Lambda best practice. The collection layer is provided by the platform, not the application:

• EKS: a node-level collector (Fluent Bit and/or the OpenTelemetry Collector running as a DaemonSet) tails container stdout/stderr, enriches each record with Kubernetes resource attributes (k8s.pod.name, k8s.namespace.name, k8s.deployment.name, service.name), and forwards via OTLP to LogWatch. Your job is simply to emit clean JSON; the collector adds the infrastructure context.
• Lambda: writes to stdout/stderr land in the function's CloudWatch log group. Enable JSON-structured Lambda logging (the platform sets the log format to JSON) so the runtime's own fields are structured, and emit your application logs as JSON too. The ADOT (AWS Distro for OpenTelemetry) Lambda layer can additionally export logs/traces via OTLP and auto-injects trace_id/span_id.

Direct OTLP export from the application (Principle 1, path 1) is also fully supported and is the right choice for long-running services that already initialise an OpenTelemetry SDK for tracing — reuse the same SDK for logs.

3 — Put Variables in Attributes, Not in the Message

Principle 4 — Stable message, structured detail. The body SHOULD be a stable template; dynamic values SHOULD be attributes.

Anomaly detection groups logs by message template. Interpolating IDs, counts, and durations into the message text creates near-infinite unique "messages" and destroys grouping.

# Avoid — every request is a unique string
log.info(f"user {user_id} fetched {count} orders in {ms}ms")

# Prefer — stable body, variable data as attributes
log.info("user fetched orders", user_id=user_id, order_count=count, duration_ms=ms)

Attribute conventions:

• Use OpenTelemetry Semantic Conventions names where one exists (http.request.method, http.response.status_code, db.system, url.path, client.address, error.type, …). Don't invent a new name for a standard concept.
• Use lower.dotted.namespacing for custom attributes (order.id, payment.provider).
• Keep attribute value types stable — a field that is sometimes a number and sometimes a string fragments the schema. Pick one type per key.

4 — Recommended Libraries by Language

All of the libraries below produce structured JSON and have first-class OpenTelemetry integration. Preferred is the default recommendation; alternatives are acceptable.

Language notes

• Go — Since Go 1.21, log/slog is the ecosystem-standard structured frontend; start there and only reach for zerolog/zap if profiling shows logging is a hot path (zero-allocation). Use slog.JSONHandler and the otelslog bridge for trace context.
• Node / TypeScript — pino is fast (5–8× Winston), logs JSON to stdout by design, and treats transport/routing as an infrastructure concern — exactly the model in Principle 3. Do not use pino-pretty or colourised output in production.
• Python — structlog's processor pipeline gives ergonomic structured logging while reusing stdlib logging handlers for output; add the OTel LoggingHandler to attach trace context. Avoid bare print() for anything you want searchable.
• Java — The OpenTelemetry Java Agent is the lowest-effort path: it auto-captures Logback/Log4j2 and injects trace_id/span_id with no code change. Otherwise emit JSON via logstash-logback-encoder or Log4j2 JsonTemplateLayout.
• Rust — tracing is the de-facto standard; configure a tracing-subscriber JSON formatting layer for output and tracing-opentelemetry for OTLP. Put filtering layers before expensive layers, and always use a batch exporter, not the simple one.

Principle 5 — Use the platform's OpenTelemetry SDK/agent rather than a bespoke logger. If a project already runs an OpenTelemetry SDK or auto-instrumentation agent for tracing, route logs through the same pipeline instead of adding a second mechanism.

5 — What Not To Do

These patterns break ingestion or anomaly detection and SHOULD be avoided:

• ❌ Pretty-printed / colourised / multi-line output in production (ANSI codes, box-drawing).
• ❌ Free-text-only logs with no severity and no structure.
• ❌ Interpolating IDs/counts/timestamps into the message body (Principle 4).
• ❌ Writing to log files on disk in containers/functions (Principle 3).
• ❌ Logging secrets, credentials, full PII, tokens, card numbers (see §6).
• ❌ Mixing types for the same attribute key across records.
• ❌ Local-timezone timestamps. Always UTC.
• ❌ One logical event spread across several lines (except stack traces, §8).

6 — Sensitive Data

Principle 6 — Don't log secrets; assume redaction is a backstop, not a license.

LogWatch applies configurable, per-tenant redaction at ingestion time (built-in patterns for emails, credit cards, bearer tokens, plus custom rules), and redacted data never reaches storage. This is a safety net — applications SHOULD still avoid logging secrets, credentials, access tokens, passwords, full PANs, or unnecessary PII in the first place. Log identifiers (user.id, order.id) rather than personal data, and never log Authorization headers or raw request bodies that may contain credentials.

7 — Trace Correlation

When a log is emitted inside a request/trace context, the record SHOULD carry the W3C trace_id (32 hex chars) and span_id (16 hex chars). This lets LogWatch pivot between a log line and its distributed trace, which is central to root-cause narratives.

• If you use an OpenTelemetry SDK/agent (Java agent, ADOT, otelslog, instrumentation-pino, tracing-opentelemetry), trace context is injected automatically — prefer this.
• Propagate context across service boundaries using W3C traceparent headers.
• If injecting manually, read the active span context and add trace_id/span_id as top-level fields.

8 — Errors, Exceptions, and Stack Traces

• Log exceptions at ERROR (or FATAL for process-ending faults) with error.type, error.message, and the stack trace in error.stack_trace (or exception.stacktrace per OTel conventions).
• A multi-line stack trace SHOULD travel inside a single JSON record (as one string field), so it remains one event. If your runtime prints multi-line traces to stderr, ensure the collector's multiline parser is configured — or, better, serialise the trace into the JSON record yourself.

9 — When Deviation Is Acceptable

These are recommendations. A project MAY deviate, and remain constitution-compliant, in the following cases. Where a project deviates, it SHOULD record the reason in its own constitution or an ADR.

1. The runtime or platform genuinely cannot emit structured JSON. Off-the-shelf components, network appliances, and some legacy daemons only speak syslog or plain text. Use the syslog (RFC 5424) or generic-JSON path and accept reduced structure.
2. A vendored third-party component whose log format you don't control. Forward as-is; let LogWatch's parsers do best-effort extraction.
3. Local development. Pretty-printed, human-friendly console output is encouraged for developer ergonomics — gate it behind an environment flag so production stays JSON.
4. Extreme hot paths where even structured logging is too costly. Prefer sampling, metrics, or trace events over dropping structure entirely; document the trade-off.
5. A different but equivalent semantic model. If a team standardises on field names that still map cleanly and losslessly onto the OTel Logs Data Model, that satisfies the intent of these guidelines.

Deviation is not justified merely by "we already have a logger" or "JSON is verbose." The cost of non-conformance is borne by the shared anomaly-detection baselines, not just the originating team.

10 — Quick Conformance Checklist

A service conforms to these guidelines when:

• Logs are emitted to stdout/stderr (or exported via OTLP), not to files.
• Each record is a single-line JSON object (or OTLP log record).
• Every record has severity_text and a body.
• service.name is set (resource attribute), plus service.version and deployment.environment where available.
• Timestamps are RFC 3339 in UTC (or omitted, deferring to server time).
• Dynamic values are attributes, not interpolated into body.
• trace_id / span_id are present inside request contexts.
• No secrets/credentials/PII are logged.
• Production output has no ANSI colour / pretty-printing.
• The logging library is one of the recommended per-language options (§4), wired to the project's OpenTelemetry SDK/agent where one exists.

Suggested Constitution Wording

Projects can reference these guidelines from their constitution with a principle such as:

Observability — Structured Logging (RECOMMENDED). Services SHOULD emit structured JSON logs to stdout/stderr (or export via OTLP) using fields that map onto the OpenTelemetry Logs Data Model, per the platform Client Logging Guidelines. Services SHOULD include service.name, severity, and trace context, and SHOULD NOT log secrets. Deviation is permitted for sources that cannot emit structured output, and MUST be documented. See user-docs/client-guidelines.md.

References

• OpenTelemetry — Logs Data Model & Logs signal
• OpenTelemetry — Specification status (logs stable across SDKs)
• OpenTelemetry — Semantic Conventions
• The Twelve-Factor App — Logs
• Choosing a Go Logging Library in 2026 · Dash0
• How to Set Up Structured Logging in Go with OpenTelemetry
• Pino vs Winston in 2026
• How to Add Structured Logging to Node.js APIs with Pino 9 + OpenTelemetry (2026)
• Choosing a Python Logging Library in 2026 · Dash0
• Bridging Python's Logging Module to OpenTelemetry · Dash0
• How to Structure Logs Properly in Rust with tracing and OpenTelemetry
• Emit contextualized JSON logs with Java · OpenTelemetry / Grafana
• OpenTelemetry Logback logging [Java] · Uptrace
• AWS Distro for OpenTelemetry — Lambda
• OpenTelemetry Collector vs Fluent Bit
• Types of logging in Amazon EKS — AWS Prescriptive Guidance