LogWatch Technical Reference Guide

Version: 0.1.0 | Last Updated: May 2026

This document covers the technical details of LogWatch: architecture, data model, API reference, configuration, performance tuning, and operational concerns. For a conceptual overview, see user-guide-overview.md.

Table of Contents

1. Architecture Overview
2. Technology Stack
3. Data Model
4. API Reference
5. Ingestion Pipeline
6. Anomaly Detection Engine
7. Natural-Language Query System
8. Root-Cause Analysis Pipeline
9. Alert Evaluation Engine
10. Multi-Tenancy and Security
11. Configuration Reference
12. Performance Tuning
13. Monitoring LogWatch Itself
14. Troubleshooting

Architecture Overview

LogWatch uses a hybrid storage architecture with four application components:

                        ┌───────────────────────────────────────┐
                        │           LogWatch Platform           │
  Ingestion Sources     │                                       │
  ─────────────────     │  ┌──────────┐    ┌────────────────┐   │
  OTel Collector ───────┤  │ Fastify  │───>│   ClickHouse   │   │
  Fluent Bit     ───────┤  │   API    │    │  (telemetry)   │   │
  Syslog         ───────┤  │ Server   │    └────────────────┘   │
  HTTP JSON      ───────┤  └────┬─────┘                         │
  CI/CD Webhooks  ──────┤       │          ┌────────────────┐   │
                        │   ┌───▼────┐     │ Anomaly Engine │   │
                        │   │ Redis  │────>│   (Python/     │   │
                        │   │(buffer)│     │    FastAPI)    │   │
                        │   └────────┘     └───────┬────────┘   │
                        │                          │            │
                        │  ┌──────────┐    ┌───────▼────────┐   │
                        │  │ Next.js  │    │  PostgreSQL    │   │
                        │  │  Web UI  │    │ (operational)  │   │
                        │  └──────────┘    └────────────────┘   │
                        └───────────────────────────────────────┘

Component Responsibilities

Storage Engines

Technology Stack

API Server (apps/api)

Runtime:       Node.js 20+ (LTS)
Framework:     Fastify 4.x
Auth:          NextAuth.js (OAuth 2.0 / OIDC), JWT, API key auth
DB Clients:    @clickhouse/client, pg (node-postgres), ioredis
WebSocket:     @fastify/websocket (live tail)
Validation:    Zod (request/response schemas)
HTTP:          OTLP/HTTP, REST/JSON, Syslog/TCP

Anomaly Engine (apps/anomaly-engine)

Runtime:       Python 3.11+
Framework:     FastAPI + Uvicorn
ML:            PyTorch, HuggingFace Transformers (LogBERT)
Statistics:    NumPy, SciPy (isolation forest, z-score)
Queue:         Redis consumer (BullMQ-compatible)
DB:            asyncpg (PostgreSQL), clickhouse-connect (ClickHouse)

Web UI (apps/web)

Framework:     Next.js 15 (App Router)
Language:      TypeScript
Styling:       Tailwind CSS
Charts:        Recharts
State:         React Query (TanStack Query)
Auth:          NextAuth.js

Shared Packages

Data Model

Storage Split

• ClickHouse stores high-volume, append-only telemetry (6 tables)
• PostgreSQL stores operational data requiring transactions (17 tables)
• Cross-engine references use value-based lookups (UUIDs stored as TEXT)

ClickHouse Tables

otel_logs

Primary log storage. Partitioned by (tenant_id, toDate(timestamp)).

Key indexes: Bloom filter on trace_id and span_id; token BF on body for full-text search; set index on severity_text.

Sort order: (tenant_id, resource_fingerprint, severity_number, timestamp) — optimized for service + severity + time range queries.

TTL: 30 DAY DELETE (configurable per tenant).

otel_traces

Distributed trace span storage. Partitioned by (tenant_id, toDate(start_time)).

TTL: 14 DAY DELETE.

metrics_time_series, metrics_samples, metrics_samples_1h

Derived metrics from logs and traces. metrics_samples_1h is populated by a materialized view that rolls up raw samples into hourly aggregations (min, max, avg, count, sum).

PostgreSQL Tables (Key Entities)

Tenant Isolation

• PostgreSQL: Row-Level Security (RLS) policies on all tenant-scoped tables. The API sets app.current_tenant per request via SET LOCAL.
• ClickHouse: Partition-by-tenant + application-layer WHERE tenant_id = ? on every query. No native RLS in ClickHouse; the API enforces isolation.

API Reference

Base URL: http://localhost:4000/api/v1

Authentication

All API requests require authentication via one of:

• API Key: X-API-Key: lw_abcdef... header
• JWT Bearer Token: Authorization: Bearer <token> header (obtained via OAuth flow)

Ingestion Endpoints

POST /api/v1/ingest/otlp

Accept OTLP log and trace data (protobuf or JSON encoding).

Content-Type: application/x-protobuf  OR  application/json
X-API-Key: lw_...

Returns 200 OK on success, 429 Too Many Requests under backpressure, 503 Service Unavailable if storage is down.

POST /api/v1/ingest/json

Accept logs in a simplified JSON format.

{
  "logs": [
    {
      "timestamp": "2026-05-11T14:30:00.000Z",
      "severity": "ERROR",
      "body": "Connection refused: redis-primary:6379",
      "service": "payment-service",
      "attributes": {
        "request_id": "req-abc-123",
        "user_id": "user-456"
      }
    }
  ]
}

POST /api/v1/ingest/syslog

Accept syslog messages (RFC 5424 and RFC 3164). TCP connection, newline-delimited.

POST /api/v1/deployments

Register a deployment event from CI/CD.

{
  "service": "payment-service",
  "version": "v2.3.1",
  "commit_sha": "a1b2c3d4e5f6...",
  "deployer": "ci-pipeline",
  "changelog": "Changed Redis connection pool size"
}

Query Endpoints

GET /api/v1/logs

Search log records.

Response:

{
  "records": [...],
  "total": 1247,
  "cursor": "eyJ0...",
  "took_ms": 42
}

GET /api/v1/logs/tail

WebSocket endpoint for live tail streaming.

Connect: ws://localhost:4000/api/v1/logs/tail?service=payment-service&severity=ERROR

Each message is a JSON log record.

GET /api/v1/traces/:traceId

Retrieve all spans for a specific trace.

Response: Array of span objects with timeline data, sorted by start_time.

GET /api/v1/traces/:traceId/logs

Retrieve all log records correlated with a specific trace.

GET /api/v1/anomalies

List detected anomalies.

GET /api/v1/anomalies/:id

Get anomaly detail including sample logs and correlated traces.

POST /api/v1/anomalies/:id/feedback

Submit feedback on an anomaly.

{
  "feedback": "false_positive",
  "notes": "This is expected behaviour during the weekly batch run"
}

POST /api/v1/ai/query

Submit a natural-language log query.

{
  "question": "What caused the spike in 5xx errors between 2pm and 3pm UTC yesterday?"
}

Response:

{
  "question": "...",
  "generated_query": "SELECT ... FROM otel_logs ...",
  "results": [...],
  "summary": "Found 247 error entries...",
  "took_ms": 1230
}

GET /api/v1/anomalies/:id/rca

Get the root-cause analysis report for an anomaly.

Management Endpoints

CRUD: /api/v1/alert-rules

CRUD: /api/v1/notification-channels

CRUD: /api/v1/redaction-rules

CRUD: /api/v1/sampling-rules

Standard REST (GET list, GET by ID, POST create, PUT update, DELETE).

GET /api/v1/services

List auto-discovered services with dependency data.

GET /api/v1/services/:id/dependencies

Get the dependency graph for a specific service.

GET /api/v1/cost/usage

Get per-service ingestion volume and cost data.

GET /api/v1/cost/recommendations

Get AI-generated sampling recommendations.

GET /api/v1/health

Health check for load balancer probes.

{
  "status": "ok",
  "clickhouse": "ok",
  "postgres": "ok",
  "redis": "ok",
  "anomaly_engine": "ok"
}

Ingestion Pipeline

Flow

Client → API Server → Validation → Redaction → Redis Buffer → ClickHouse Batch Insert
                                                     ↓
                                              Anomaly Engine
                                              (async scoring)

1. Receive: API server accepts log records via OTLP, JSON, or syslog
2. Validate: Normalize to internal data model (fill defaults for missing fields per FR-044)
3. Redact: Apply tenant redaction rules to body and attributes
4. Buffer: Write to Redis stream for burst absorption
5. Batch insert: Worker drains Redis buffer and writes to ClickHouse in batches (configurable batch size, default 10,000 records or 1 second, whichever comes first)
6. Async scoring: Anomaly engine consumes from Redis to score records against baselines

Backpressure

When Redis memory hits the configured limit (maxmemory 256mb, policy noeviction):

1. Redis returns OOM on write attempts
2. API server catches this and returns 429 Too Many Requests with Retry-After header
3. OTLP-compatible clients automatically retry with exponential backoff
4. No log data is silently dropped

Normalization Rules

Anomaly Detection Engine

Detection Pipeline

Redis Stream → Feature Extraction → Model Inference → Scoring → PostgreSQL (anomaly record)
                                                                        ↓
                                                                 Alert Evaluator

Models

Statistical Baseline (always active)

• Metrics tracked: log volume per minute, error rate, p50/p95/p99 latency
• Technique: Z-score against rolling 7-day baseline with hourly and day-of-week decomposition
• Threshold: Z-score > 3.0 triggers anomaly candidate
• Used during: Learning period (first 7 days) and as a first-pass filter

LogBERT (after 7-day baseline)

• Model: BERT-base fine-tuned on log template patterns
• Input: Log body text, tokenized into log template components
• Output: Anomaly probability score (0.0 - 1.0)
• Training: Continuous fine-tuning on per-service data; retrained weekly
• GPU: Optional but recommended. CPU inference is supported but 10x slower.

LLM Semantic Analysis (for high-severity candidates)

• Model: Claude API (claude-sonnet-4-6 by default)
• Input: Anomaly candidate context: sample logs, baseline statistics, recent deployment history, service dependencies
• Output: Severity classification, human-readable explanation, false-positive probability
• Rate limiting: Max 100 Claude API calls per hour per tenant (configurable)

Severity Classification

Feedback Loop

User feedback on anomalies (helpful, not_helpful, false_positive) is incorporated:

1. false_positive entries are added to the service's suppression list
2. LogBERT is retrained with corrected labels at next weekly cycle
3. Statistical baseline thresholds are adjusted for recurring false positives

Natural-Language Query System

Flow

User Question → Claude API (translation) → ClickHouse SQL → Execute → Claude API (summarization) → Response

1. User submits plain-English question via API or UI
2. System prompt provides the ClickHouse schema, available tables, and column types
3. Claude translates the question to a ClickHouse SQL query
4. Query is validated (read-only, tenant-scoped, time-bounded)
5. Query is executed against ClickHouse
6. Results are passed back to Claude for plain-English summarization
7. Response includes: original question, generated SQL, results, and summary

Security Constraints

• Generated queries are restricted to SELECT only (no DDL/DML)
• WHERE tenant_id = ? is always injected (cannot be overridden by the LLM)
• Query execution timeout: 30 seconds
• Result set limit: 10,000 rows

Root-Cause Analysis Pipeline

Trigger

RCA reports are generated automatically when:

• An anomaly is classified as critical severity
• An anomaly is manually requested by a user ("Generate Report" button)

Evidence Gathering

1. Log patterns: Anomalous log records and their diff from baseline patterns
2. Correlated traces: Traces containing spans from the affected service during the anomaly window
3. Recent deployments: Deployment events for the affected service and its dependencies within the preceding 24 hours
4. Service dependencies: Upstream and downstream services, their error rates and latency during the window
5. Baseline comparison: Current vs. historical metric values

Report Generation

All evidence is assembled into a structured context and sent to Claude API with a system prompt instructing it to:

• Identify the most probable root cause
• Assign a confidence score
• Cite specific evidence for each claim
• Structure the output as a narrative suitable for an incident review

Alert Evaluation Engine

Evaluation Loop

The alert evaluator runs as a background worker in the API server:

1. Every evaluation_interval_seconds (default 60s), evaluate all enabled alert rules
2. For each rule, execute the rule condition against current data
3. If the condition matches:
   • Check cooldown: skip if last fired within cooldown_minutes
   • Check mute window: record but don't notify if mute_until is in the future
   • Create an alert_event record
   • Send notifications to all configured channels
4. Notification delivery is async with retry (3 attempts, exponential backoff)

Rule Types

Notification Payloads

Slack:

{
  "text": "LogWatch Alert: Critical anomaly detected",
  "blocks": [
    {
      "type": "section",
      "text": {
        "type": "mrkdwn",
        "text": "*Novel NullPointerException in checkout-service*\nSeverity: critical (0.89)\nDetected: 2026-05-11 14:33 UTC\n<https://logwatch.example.com/anomalies/abc-123|View in LogWatch>"
      }
    }
  ]
}

PagerDuty: Events API v2 trigger event with severity mapping (critical → critical, warning → warning, info → info).

Multi-Tenancy and Security

Tenant Isolation Model

Authentication Methods

RBAC Permissions Matrix

Data Redaction

Redaction runs at ingestion time (before data is written to ClickHouse). Once redacted, original values are not recoverable.

Built-in patterns:

• Credit card numbers (Luhn-valid 13-19 digit sequences)
• Email addresses
• Bearer tokens (Bearer [A-Za-z0-9._-]+)

Custom patterns: any valid regular expression, applied to body, attributes, or both.

Configuration Reference

Environment Variables

Retention Configuration

Set per tenant in tenants.retention_config (JSON):

{
  "logs_days": 30,
  "traces_days": 14,
  "metrics_days": 90
}

ClickHouse enforces TTL based on these values. Changing retention does not retroactively delete data — it takes effect for newly expired data.

Ingestion Configuration

Performance Tuning

ClickHouse

• Memory: Minimum 4 GB RAM for ClickHouse. For 100K logs/sec, recommend 16 GB+.
• Disk: SSD required. ClickHouse compression typically achieves 10:1 on log data.
• Storage estimate: 100K logs/sec * 1 KB avg * 86,400 sec/day = ~8.6 GB/day raw, ~0.86 GB/day compressed.
• Index granularity: Default 8192 is optimal for most workloads. Reduce to 1024 for very selective queries on small datasets.

PostgreSQL

• Connections: Set max_connections to at least (API replicas * 20) + 10.
• Shared buffers: 25% of available RAM (standard PostgreSQL tuning).
• RLS overhead: Negligible for properly indexed tables. Ensure tenant_id is the leading column in all frequently queried indexes.

Redis

• Memory: 256 MB handles ~500K buffered records. Increase for higher burst tolerance.
• Policy: noeviction for ingestion buffer (triggers backpressure rather than data loss).

API Server

• Node.js cluster: Run one process per CPU core using cluster module or PM2.
• Concurrency: Fastify handles ~50K req/sec per process on modern hardware.
• Kubernetes: 2+ replicas for availability. HPA based on CPU utilization (target 70%).

Anomaly Engine

• GPU: NVIDIA GPU with CUDA recommended for LogBERT inference. Falls back to CPU (10x slower).
• Memory: 4 GB minimum. LogBERT model requires ~1.5 GB.
• Scaling: Increase replicas for higher ingestion rates. Each replica processes independently from Redis.

Monitoring LogWatch Itself

LogWatch exposes Prometheus-compatible metrics at /metrics on the API server:

Health Endpoint

GET /health returns status of all dependencies. Use as Kubernetes liveness/readiness probe.

Troubleshooting

Logs not appearing in search

1. Check API health: curl http://localhost:4000/health
2. Check Redis buffer: redis-cli info memory — if used_memory is near maxmemory, backpressure is active
3. Check ClickHouse: clickhouse-client -q "SELECT count() FROM otel_logs WHERE timestamp > now() - INTERVAL 5 MINUTE"
4. Check ingestion endpoint response: should be 200 OK, not 429 or 503

High false-positive rate

1. Ensure the service has 7+ days of baseline data (check anomaly_baselines table)
2. Submit false_positive feedback on incorrect detections
3. Check anomaly_models for the service — status should be ready
4. Review hourly_pattern and dow_pattern in baselines for completeness

Slow search queries

1. Check ClickHouse system.query_log for slow queries
2. Ensure queries include time range filters (partition pruning)
3. Verify bloom filter indexes are present: SHOW CREATE TABLE otel_logs
4. For large result sets, use pagination (cursor parameter)

Alert notifications not arriving

1. Verify notification channel is configured and verified
2. Check alert_events table for the rule — status should be firing
3. Check cooldown: last_fired_at + cooldown_minutes may not have elapsed
4. Check mute window: mute_until may be in the future
5. Check notification_results in alert_events for delivery errors

Anomaly engine not processing

1. Check engine health: curl http://localhost:8001/health
2. Check Redis connection from the engine
3. Check Python logs: docker compose logs anomaly-engine
4. Verify ANOMALY_ENGINE_URL in API server config points to the correct address