Section 0 — Introduction
Today's network is not yesterday's network. A decade ago, an organization with ten servers and one firewall was considered a large network.
The same organization today runs dozens of virtual machines, several Kubernetes clusters, cloud services, Internet of Things devices, and hundreds of remote users. Each of those components produces a log. Every log is a piece of a puzzle that nobody can see in full on their own.
An earlier guide, The Philosophy of Logs, explained why event logging was invented and what need it answered. It left a more fundamental question unanswered, and this document answers exactly that question.
Why This Guide Was Written
Two earlier guides were two separate halves. One covered the philosophy of logs, and the other covered the family of central log servers. In practice those halves form one continuous chain.
You cannot discuss a SIEM without understanding why scattered logs fail. You cannot design a central log architecture without knowing what each layer is responsible for. And you cannot choose a tool without knowing where that tool sits in the chain.
When a network grows from one server to a thousand, where do the logs go? If every device keeps its own logs, nobody holds the complete picture at the moment an incident happens.
This document completes that chain. It starts with why centralization is necessary, moves on to which tools exist, and then explains how they work at the technical level. From there it digs into the SIEM as the analytical brain, states the operational and privacy challenges honestly, works through three complete scenarios, and ends with an executable design checklist.
Who This Guide Is For
This guide has two audiences, and it tries to be readable by both without dropping down to the level of either.
The first audience is the engineer and the specialist: someone who has worked with rsyslog, Elastic Stack, Wazuh, or Splunk, or who is designing a log architecture for an organization right now. That reader gets enough technical detail: protocols, queue types, the data lifecycle, correlation rules, and architectural decisions. Nothing here assumes the reader already knows everything, so every key concept carries a short explanation the first time it appears.
The second audience is the newcomer or the reader with no background: someone who has just entered infrastructure or security, or a manager trying to understand what their team is talking about. That reader gets a different entry point: every section opens with an analogy or an everyday example, and technical terms are explained in parentheses or in a short sentence. The goal is not to simplify the guide but to make it enterable, so a newcomer can start at the beginning and climb step by step without getting lost on the way.
How to Read This Guide
The guide is designed to be read linearly, and the order of its sections is not arbitrary. Each section builds on the one before it.
If you are in a hurry, or you only need one part of the material, this short roadmap will help:
- If you only want the "why": read Section 1 (The Philosophy of Centralization), then Section 7 (Summary and Design Principles). Those two give you the full argument without the technical detail.
- If you want to choose a tool: read Section 2 (The Family of Central Log Servers) and Section 4 (SIEM in Depth). The comparison table in Section 2 tells you which tool fits which goal.
- If you are an engineer who wants to design: Section 3 (Technical Architecture) and Section 7 (Design Principles) are the backbone of your work. Do not skip Section 5 (Challenges), because that is exactly where many designs fail.
- If you want to see it work in practice: read Section 6 (Scenarios). Its three complete scenarios connect every earlier section in real situations.
Section Preview
Section 1 — The Philosophy of Centralization: Why scattered logs fail. This section shows that centralization is not a matter of taste but the answer to five fundamental problems: scale, time, format, data lifetime, and human capacity. It opens with a short scenario about a 200-server network without centralization, and shows why even a simple script cannot solve those problems.
Section 2 — The Family of Central Log Servers: a complete map of the tools. This section introduces every member of the family by name: the Syslog server, the Log Management System (LMS), SIEM, SOAR, XDR, and supporting components such as the Log Shipper and the Relay. For each one it states the role, the limitation, and the primary audience. It closes with a comprehensive comparison table and a reference three-tier architecture.
Section 3 — Technical Architecture: the log pipeline from production to analysis. This section explains, at the technical level, how the pipeline actually works.
The path runs from the production layer (operating system, application, network, cloud) through transport (Push versus Pull, agent versus agentless, protocols) and collection (Relay, Aggregator, disk buffer). Processing (Parse, Normalize, Enrich, Route) and storage (index-based versus column-based, the data lifecycle) come next, and the pipeline ends in analysis and response. Two hidden but essential columns are introduced as well: a single time source (NTP) and the correlation ID.
Section 4 — SIEM in Depth: the analytical brain. This section explains in detail what a SIEM does that the rest of the family does not: event correlation, a rule-based approach versus a statistical one versus machine learning, and the concept of a use case. It also examines real correlation rule examples (Brute Force, Impossible Travel, Lateral Movement, Data Exfiltration), alert management and alert fatigue, the role of threat intelligence, and the honest limitations.
Section 5 — Operational Challenges and Privacy. This section says what vendor advertising does not: data volume and cost, data quality, the security of the log system itself (what happens if the SIEM is hacked?), privacy and data masking, and compliance with standards such as PCI-DSS and HIPAA. The importance of NTP and the maintenance of rules are covered as well. Its key sentence: "A SIEM is worse than no SIEM if the team is not ready."
Section 6 — Complete Scenarios and Problem Solving. Three complete scenarios are presented, each with an initial state, events scattered across every layer, what each tool sees, the breaking point, and the architectural solution.
The first scenario is a gradual intrusion, from Insider to Lateral Movement to Exfiltration. The second is a network outage and data survival. The third is post-incident analysis and compliance.
Section 7 — Summary and Design Principles. This section turns everything into an executable checklist: ten principles for designing a central log architecture, the key decisions an architect has to make, and the common mistakes. It then offers a gradual implementation roadmap for an organization that starts from zero.
A Small Contract with the Reader
Three promises run through this guide, and it is worth knowing them up front.
First, no technical term is left unexplained. The first time a term appears, it is explained in parentheses or in a short sentence. If it appears again later, the guide assumes you have already learned it.
Second, no technical claim is left without an example. Every abstract concept comes with a concrete example. When we talk about event correlation, for instance, a real example follows immediately to show what that correlation actually looks like in practice.
Third, limitations are never hidden. This guide is not vendor advertising. Wherever a tool has a limitation, it is stated, and wherever an approach fails, it is stated too. The aim is that after reading the guide you can make a decision, not merely be impressed.
Summary
- This guide begins where you are now, with why centralization is necessary, and ends at an executable design checklist in its final section.
- It has two audiences: engineers who already work with tools such as rsyslog, Elastic Stack, Wazuh, or Splunk, and newcomers to infrastructure and security.
- Every section carries enough technical detail for an engineer and a short plain-language explanation for a newcomer, without dropping down to either audience's level.
- If you are short on time, the roadmap above points you straight at the sections that answer your goal without the rest.
- The guide keeps three promises: every term is explained, every claim is illustrated, and every limitation is stated, so that you can decide rather than merely be persuaded.