#008

Log Aggregation & Anomaly Detection

ServiceWebsite

Unified log ingestion with AI-powered anomaly detection and alerting

Project Variant:
Dark factory developmentPublic
Candidate
2
Dark factory
3
Guided development
4
Raw
5
Custom development
6
Alpha
7
Beta
8
Production

Collaborators

philcal (project owner)

philip-callender

How do I use this software?

This software runs wherever suits you — you just need somewhere to host it. Pick the option that fits your team:

OptionWhat it means
Self-hostingSet up the environment and run it yourself, on your own infrastructure.
Tooltwist hostingTooltwist can host and run it for you.
Other providersFind a host in the provider directory — or, if you already have a support company, we're happy to give them the tools to deploy the application for you.
Licensing

This variant is open source — you're free to use it and modify it at no cost. Hosting and support arrangements are provided separately and are not covered by this licence.

Who can help me?

Plenty of people can help you get the most from this software — browse the provider directory. Some providers can host it for you, others can customise it to your needs, and others again offer technical support and a helpdesk.

Tooltwist can host and customise the software for you, and Twist Teams provides technical support services.

Already have a support provider? We're happy to give them the tools to fully support the platform.

Not sure who to ask? Feel free to .

How can I help?

If you would like to help develop or test this project, go to the Collaborators tab (after you log in) and request to join. Your help will be appreciated!

Read me first

If you have just been handed LogWatch and you are not sure what it is, start here. This page is written in plain language — no code, no jargon, nothing you need a technical background to follow. The other documents in this folder go deeper once you know whether this is the right tool for you.


1. Who this is for

LogWatch is for the people who keep online services running.

If your company has a website, a mobile app, or any software that customers use, then somewhere behind it are dozens of separate programs talking to each other. When a customer says "I couldn't check out", someone has to work out which of those programs went wrong, and why. That someone is usually called a site reliability engineer, a DevOps engineer, or simply "whoever is on call tonight".

Those people are the primary audience. LogWatch is built for how they actually work: under time pressure, often at an awkward hour, with a customer waiting.

Two other groups get value from it. Engineering managers use it to see whether services are healthy and what the monitoring is costing. Developers use it when they are adding a new piece of software and want to check it is reporting properly.


2. What it does

Every piece of software writes a running diary of what it is doing. One line might say "customer 4471 added a jacket to their basket". Another might say "payment provider did not answer in time". These lines are called logs, and a busy company produces millions of them a day.

LogWatch does four things with them.

It collects them. All the logs, from every program, into one place — instead of scattered across dozens of machines nobody can search.

It searches them. Type a word and get every matching line from anywhere in the system, in about a second.

It notices problems on its own. This is the part that saves the most time. LogWatch learns what a normal day looks like for each program, and tells you when something departs from it — before a customer complains.

It tells the right people. When something matters, it sends a message to Slack, to a paging service, or to email.


3. The domain: what "observability" means

The field LogWatch belongs to is called observability. The word describes a simple idea: can you tell what is happening inside your systems without taking them apart?

It helps to know why this became hard. Software used to be one large program on one computer. When it broke, you looked in one place. Modern systems are built from many small programs — one handles the shopping basket, one handles payment, one sends email. Each may run on many machines and be replaced several times a week.

That design is good for building software quickly. It is bad for working out what went wrong, because a single customer's checkout might pass through eight programs, and the failure could be in any of them — or in how they interact.

Three kinds of information help answer that, and LogWatch handles all three:

  • Logs — the running diary, described above.
  • Traces — the path one customer request took through all those programs, and how long each step took. Invaluable when something is slow rather than broken.
  • Anomalies — departures from normal that were noticed automatically rather than by someone watching a screen.

A few terms you will meet:

TermWhat it means
ServiceOne of those small programs. "Checkout service", "payment service".
SeverityHow serious a log line is: from routine detail up to a fatal failure.
TraceThe full journey of one request across every service it touched.
AnomalySomething unusual, found automatically.
AlertA message sent to a person because something needs attention.
BaselineWhat "normal" looks like for a service — which varies by hour and by day.
IngestionThe act of receiving logs. You are usually billed by how much you ingest.
RetentionHow long data is kept before it is deleted.
RedactionRemoving sensitive data — card numbers, passwords — before storing anything.
TenantOne organisation's separate, walled-off space within the system.

4. Where it fits

LogWatch does not replace the software you already run. It sits alongside it and listens.

Who typically uses it. Any organisation running its own online services and large enough that no single person can hold the whole system in their head — usually somewhere between a handful and a few hundred engineers. It suits teams who find the big commercial monitoring products expensive, or who want to keep their data on their own infrastructure.

What it connects to. Your services send their logs to LogWatch using OpenTelemetry, an open industry standard that most modern software supports already. If some of your systems are older, LogWatch also accepts the formats they use, so you do not have to modernise everything first.

Going the other way, LogWatch sends alerts into the tools your team already watches: Slack, PagerDuty, email, or any system that accepts a web notification.

It is multi-tenant, meaning one installation can serve several separate organisations or divisions whose data never mixes.


5. First run

When you open LogWatch for the first time, you will be asked to sign in. There is no separate password to remember — it uses your organisation's existing sign-in.

You will land on the Log Explorer. If you are looking at the preview or demo version, it is already full of realistic sample data for a fictional retailer called Acme Retail Group, so you can explore straight away.

Do this first: pick any log line and click it. It opens to show everything recorded with that event, including a link to the full journey of that request. That single action is the core of the product; everything else supports it.

Then visit Anomalies in the left-hand menu. You will see problems LogWatch found by itself. Open the one about the checkout error rate — it shows not just that something broke, but a plausible explanation of why, assembled automatically. If that is useful to you, LogWatch is likely to be useful to you.


6. Setting up your own environment

Four steps, roughly in this order. Your engineers will do most of it; the Installation Guide and Integration Guide have the detail.

Step one — install it. Either on your own servers or on the WBSP platform. Fresh installations are prepared using the process in Production Setup, which creates your organisation's space and your first administrator account.

Step two — check the redaction rules. Before pointing any real system at it. LogWatch arrives with rules that strip card numbers, security tokens, and email addresses out of logs as they arrive. They are switched on from the start, deliberately — so sensitive data is protected before your first log is stored, not after somebody notices. Add rules for anything specific to your business.

Step three — connect your services. Point them at LogWatch one at a time. Start with a small, non-critical one so you can confirm data is arriving before committing everything. Each service appears in the Service Registry automatically once it starts reporting; there is no list to maintain by hand.

Step four — set up alerting. Add a notification channel — your team's Slack channel is the usual starting point — and then switch on the starter alert rules that came with the system. They ship switched off on purpose, because an alert with nowhere to go is worse than no alert at all.

Give it a week before tuning anything. LogWatch needs to observe normal behaviour before it can recognise abnormal behaviour.


7. Day to day

Most days, nobody opens LogWatch at all — which is the point. It runs quietly and speaks up when needed.

When an alert arrives, the usual path is: read the alert, open the anomaly it points at, read the automatic explanation, follow the sample logs or the trace to confirm it, then fix the underlying problem. Mark the anomaly resolved, and say whether the explanation was helpful. That last step is not bureaucracy — the feedback improves what you are shown next time.

When a customer reports a problem, start at the Log Explorer. Narrow to the time it happened, search for the customer's order or account, then open the matching log line and follow its trace.

When you are deploying something new, keep the Anomalies screen open for an hour afterwards. LogWatch knows when deployments happened and will connect a new problem to a recent release if the timing lines up — which is how the example in the demo was diagnosed.

Once a week or so is enough for the rest: a glance at Anomalies for lower-priority items, and a look at the Cost screen to see whether ingestion is drifting upwards.


8. Ongoing care

A handful of things need attention over the longer term.

Monthly: review costs. The Cost screen shows what each service contributes and suggests where sampling — keeping a representative portion rather than every line — would save money without losing what you rely on. One misbehaving service can quietly double a bill.

Monthly: check retention. Data is deleted automatically once it reaches the age you set. Confirm those settings still match what your business and your regulators require.

Quarterly: review who has access. People join and leave. The audit log records administrative changes if you need to demonstrate that.

Quarterly: revisit alert rules. Alerts that fire constantly get ignored, and an ignored alert is worse than none. If a rule cries wolf, tighten or retire it.

Ongoing: keep redaction current. New features log new things. When your services start recording a new kind of sensitive value, add a rule for it.

Occasionally: take a backup. Before a major upgrade, and on whatever routine your organisation requires. Backups are encrypted with a passphrase you choose — store that passphrase somewhere safe, because without it a backup cannot be restored.


9. Who can do what

Three levels of access:

RoleCan do
ViewerLook at everything — logs, traces, anomalies, alerts, costs. Change nothing.
EditorAll of the above, plus manage alert rules and notification channels, and resolve anomalies.
AdminAll of the above, plus manage users, API keys, redaction rules, and system settings.

Most people should be viewers or editors. Keep the number of administrators small.


10. About the sample data

The preview and demo versions arrive populated with a fictional online retailer — fifteen services, a week of activity, and three worked incidents including the checkout failure used throughout the demonstration.

None of it is real. It exists so you can judge the product with something on the screen instead of empty tables. When you install LogWatch properly for your own organisation, you start from empty and it fills with your own data as your services begin reporting. The sample data is never mixed with yours.


11. Common questions

Will this slow my systems down? No. Services hand their logs off and carry on; they do not wait.

What if LogWatch is unavailable? Your systems keep running. You lose visibility for that period, not service.

Do I have to connect everything at once? No, and you should not. Start with one service, confirm it works, then expand.

Why is a screen empty? Almost always because nothing has been sent yet, or because you are looking at a time range with no activity. Widen the time range first.

Why did I get an alert about something that turned out to be fine? Early on this is normal — the system is still learning what is normal for you. Mark it a false positive; that is exactly the feedback it needs.

It found a problem but the explanation is wrong. Mark it "not helpful". The explanation is a strong starting hypothesis, not a verdict, and your correction improves later ones.

Can I get my data out? Yes. It is your data, and there is a full export and backup facility.

Does it cost money to run? The software is yours. You pay for the computers and storage it runs on, which is what the Cost screen helps you keep in hand.


12. The other documents here

DocumentWhat it coversWho it is for
User Guide — OverviewUsing each screen in detailEveryone
User Guide — TechnicalHow it works internallyEngineers
Installation GuideGetting it runningWhoever installs it
Integration GuideConnecting your servicesEngineers
Production SetupPreparing a real environment for first useWhoever operates it
Client GuidelinesDevelopment standards used to build itDevelopers

A reasonable order: this page, then the User Guide Overview, then the Installation and Integration guides when you are ready to set it up.


13. Where to get help

The community at wbsp.ai is the place to go. You can ask questions, suggest improvements, and find people who customise applications like this one for a living.

One thing worth understanding about what you have: LogWatch is AI-generated core software. Everything described here works, but it is deliberately plain — no company branding, no bespoke styling. It is meant to be taken and shaped to fit your organisation. If you have domain knowledge and some familiarity with AI tools you can do that yourself; if not, the community can connect you with someone who will. It is also worth browsing what others have already built, as one of those versions may be closer to what you need than the plain one.