Logs Are a Product, Not a Byproduct
What building Traycer taught me about treating observability as something people actually read.
Traycer started as an internal annoyance. We had logs everywhere and answers nowhere. A request would fail, and the trail would be scattered across four services, each with its own idea of what a 'correlation id' was. The data existed. The product did not.
That distinction - data versus product - is the thing I keep coming back to. A log line is data. A log a human can read at 2am is a product.
Nobody is the user, so everyone is
Logs are the rare interface with no named owner. They are written by whoever added the statement, read by whoever is on call, and designed by no one. The result is predictable: a stream of console.log that tells you that something happened, never what or why.
The purpose of a log is to answer the question you will have later, not the one you have now.
What a log owes the reader
- A subject. What is this line about? The entity id, not the process id.
- A verb. What happened, in plain language and past tense.
- A consequence. What the system did next, or will do next.
- A way to correlate. One id that threads the whole request together.
If a line cannot answer those four questions, it is noise with a timestamp.
The part that surprised me
The hardest work was not storage or search. It was vocabulary. Once the team agreed on a small set of fields and what each one meant, the logs became readable almost by accident. The schema did the design.
A modern tooling architecture and open standard for creating and storing logs based on 4-point cardinality.
Observability is usually framed as a tooling problem. I think it is a writing problem wearing a costume. The teams with the best incidents are rarely the ones with the fanciest dashboards. They are the ones whose logs read like sentences.