Section 1 — Why Centralization: How Distributed Logs Fail
Keeping each host's logs on its own disk works right up until it stops working. The moment an estate grows past a handful of machines, nobody holds the whole picture, and the evidence needed during an incident is scattered, partial, and out of order. This section names the five problems that make distributed logs fail and shows how centralization answers each one.
A Scenario That Clarifies Everything
Imagine a company with 200 servers. Each one writes its own logs to local files under /var/log, and the operations team follows a simple rule: SSH to the affected machine and read the log. That rule is excellent for one server and a disaster for 200.
One morning the CEO learns that customer data is being sold on an online forum. The security team is asked to investigate, and the first question is obvious: where did the attacker get in? The analyst has to connect to 200 servers, collect the log files by hand, and try to rebuild the timeline. Five obstacles get in the way.
- Three servers restarted last week and took their logs with them.
- The firewall keeps only 7 days of logs, and the attack probably started two weeks ago.
- One server's clock is 40 minutes slow, which scrambles the order of events.
- Every host logs in its own format, so comparing them against each other takes hours.
- The sheer volume makes a manual search across those logs take weeks.
After three days of work the analyst can offer only one sentence: probably a web server, but not certain. That is not good enough for the CEO, the customers, or the regulator. The company does not know what data left, who took it, or how to stop it happening again.
When a host stops sending logs, the outside world looks exactly like a day on which nothing happened. Missing data and an absence of incidents are indistinguishable from the outside, and that ambiguity is what makes distributed logging dangerous.
This is not a hypothetical. It is what happens in organizations that keep their logs scattered, and it is the clearest statement of the centralization argument: scattered logs do not exist at the moment you need them.
Five fundamental problems sit behind that failure: scale, time, format, data lifespan, and human capacity. Each one is a different kind of limit with a different answer, and the five sections that follow take them one at a time.
Scale
The first problem is the simplest and also the most fundamental: volume. One server needs nothing more than its local log. Ten servers can be collected with a script. A hundred servers break that script, and a thousand servers turn collection from a manual chore into an engineering problem.
Why? Because collecting logs from a thousand servers creates three challenges that a single server never had.
First, volume. A thousand servers each produce several gigabytes of logs per day, and the daily total reaches terabytes. That volume cannot be stored and searched in plain text files. You need a specialised database that can find millions of records in seconds.
Second, variety. A thousand servers means a thousand configurations. Some send to Syslog, some write to files, some write to a database. Some use UTF-8 and some Latin-1, and some record local time while others record UTC. Without a normalisation layer, that variety makes the data unusable.
Third, confidence. When you collect logs from a thousand servers, how do you know they all actually sent some? If a critical host breaks because of a configuration error and never ships, you simply do not see that host, and the blindness can stay hidden for months.
The Canadian Centre for Cyber Security is explicit about this: organizations should regularly verify that every critical source really is forwarding its logs. A silent source is an invisible security hole.
Centralization answers all three challenges. A server built for volume also normalises the variety, and it can monitor which sources have stopped reporting.
Time
The second problem is subtler and usually ignored: clocks. Every device keeps its own clock, and those clocks are never perfectly synchronised. A few seconds of drift is normal, and a few minutes of drift is not unusual in a large network.
Rebuilding an incident without an accurate ordering is impossible. Suppose the firewall reports a suspicious connection at 9:45 and the web server reports receiving the same connection at 10:00. Which happened first? If the firewall clock runs 15 minutes fast, the real order is reversed, and the analyst has no way of knowing.
The Canadian Centre for Cyber Security treats this as a serious risk and warns that even small clock differences can make it impossible to reconstruct the sequence of an attack. The answer is to synchronise every device against a single time source, usually NTP, and that synchronisation is only trustworthy when a central authority monitors it.
Centralization solves this in two ways. First, the central server can act as the time authority and pull every device into line with itself. Second, even when synchronisation is imperfect, the central server records the arrival time of every message so the analyst can separate the event time reported by the device from the receipt time recorded centrally. That distinction is what post-incident analysis runs on.
Format
The third problem looks simple and is in practice one of the most expensive: a shared language. Every device logs in its own format.
A firewall might write: 2026-10-19 14:32:05 DENY TCP 10.0.1.5:443 -> 10.0.2.7:52341
A Linux server might write: Oct 19 14:32:05 webserver sshd[1234]: Failed password for root from 10.0.1.5 port 52341
A web application might write: {"timestamp":"2026-10-19T14:32:05Z","level":"WARN","user":"ali","action":"login_failed","ip":"10.0.1.5"}
All three describe the same event, a failed login attempt from a given address. But the three formats share nothing, so an analyst cannot put them side by side with a single search. Each one has to be understood on its own, and that is not something anyone can do across millions of records.
Centralization solves this with normalisation: converting every format into one structure. After normalisation, all three logs become a single record with the same fields, namely time, source, destination, event type, and result. One search now returns all three together.
The Canadian Centre for Cyber Security names normalisation as a core benefit of a central log architecture, because it is what makes correlation across sources possible. Without normalisation there is no correlation, and without correlation, multi-stage attacks cannot be detected.
Data Lifespan
The fourth problem is about survival. Local logs are fragile, and three things can destroy them.
First, the restart. If a server reboots and its logs live in /var/log under the default configuration, part of the log can be lost. Systems backed by tmpfs, which is how many modern distributions handle /var/log/journal, lose everything on every restart.
Second, the full disk. Logs grow. If rotation is not configured correctly the disk fills, and then either the server stops or old logs get deleted. Either way, data is gone.
Third, the server itself. If a server disappears because of hardware failure or an attack, its logs disappear with it. Worse, the logs you need most are the ones that explain the failure, and those are exactly the ones that are lost.
This is the bitter paradox of local logging: the logs vanish precisely when you need them. In the scenario above, three restarted servers destroyed exactly the records the analyst needed to answer the CEO's question.
Centralization fixes this by separating the place where logs are produced from the place where they are kept. Logs are stored on the central server, independent of the fate of the host that produced them. If that host fails, its logs survive, and if its disk fills, the earlier logs are already safe centrally.
That separation is a foundational principle of reliable system design: no system should be the only witness to itself. A server that may be compromised must not be the sole keeper of the logs recording that compromise. Centralization is what provides that independence.
Human Capacity
The fifth problem is about people. Even if every earlier problem is solved, including volume, time, format, and data lifespan, one remains: a human analyst cannot review millions of events a day by hand.
A mid-sized network produces millions of events daily, and perhaps a few dozen of them genuinely matter. But how does the analyst know which ones? Review everything and it takes years. Review nothing and attacks get missed.
Centralization fixes this with automation. The central server can hold rules that separate important events from noise, such as raising an alert when one user has 10 failed logins in 5 minutes. Such a rule filters millions of events and delivers a few dozen real ones to a human being. This is exactly what a SIEM does, and it is explained in detail in Section 4.
Without centralization, automation is not possible. The data is scattered across a hundred different places, and nobody can run a single rule over all of it at once.
Why a Simple Script Is Not Enough
Someone will suggest writing a cron script that copies every host's logs to a central server each night and declare the problem solved. It sounds reasonable, but in practice it runs into five serious limits.
First, latency. A cron script runs once a night, so logs arrive 24 hours late. An attack at 3am stays unknown until the following evening, and in security, 24 hours of delay is a disaster.
Second, volume. Copying several terabytes from 200 servers every night saturates network bandwidth and takes hours. If the network drops halfway through, the copy starts over.
Third, format. A cron script copies logs exactly as they are, with no normalisation at all. What you end up with is a large warehouse of inconsistent files that are still impossible to search.
Fourth, confidence. A cron script cannot tell you whether every host actually sent its logs. If a host breaks because of a configuration error, the script raises no warning whatsoever. You believe you have everything while sitting on an invisible hole.
Fifth, scalability. A cron script works for 10 servers. At 1000 you are writing and maintaining dozens of parallel scripts, which in practice becomes a full engineering project without a single benefit of a purpose-built system.
The real answer is a designed architecture rather than a cron script: transport, collection, processing, storage, and analysis layers, each with its own job. Section 3 covers that architecture in detail.
The Single Source of Truth
Every problem above reduces to one foundational concept: the single source of truth. Whenever data lives in more than one place, three questions appear. Which copy is correct? Which copy is newer? And do you have all the copies at all?
A single source of truth is one agreed location where the reference copy lives. For logs, that is a central server that collects everything and becomes the basis for every analysis and decision. When an analyst asks what happened yesterday, they look in one place, not in 200.
The Canadian Centre for Cyber Security makes the same point in its guidance: a log architecture must provide a unified view of all events so that distributed detection and post-incident analysis become possible. Without that single view, every analysis runs on a fragment of data and its result cannot be trusted.
This is not only an architectural idea but an operational requirement. The architecture below answers all five problems at once: scattered sources converge on one collector, formats and clocks are unified, stored data is separated from the host that produced it, and rules do the filtering that a human cannot.
Summary
Five fundamental problems make distributed logs fail in large networks:
| Problem | What happens | What centralization does |
|---|---|---|
| Scale | Collecting logs from hundreds of servers becomes manual and impossible | One collection point designed for volume |
| Time | Clock drift scrambles the order of events | A single time reference plus receipt timestamps |
| Format | Every device speaks its own language | Normalisation into one structure |
| Data lifespan | Local logs die with a restart or a full disk | Separation of production from storage |
| Human capacity | An analyst cannot review millions of events by hand | Automation and intelligent filtering |
Centralization answers all five problems. It is not a single tool, though; it is a whole family of tools, each supplying part of that answer. The next section introduces that family: The Log Server Family.
Previous: Section 0 — Introduction · Next: Section 2 — The Log Server Family · All sections