Section 3 — Technical Architecture: The Log Pipeline from Production to Analysis
Knowing the names of the tools is not the same as knowing how data actually moves. This section follows a single log message from the moment it is created to the moment a human or a machine decides something, across six layers that each own exactly one job.
The previous section introduced the family of central log servers. This one is about the mechanics between them: what each piece does, what it guarantees, and what it does not. The goal is that an engineer can design the architecture and a newcomer can see where each piece sits in the picture.
The Pipeline as a Single Concept
The log pipeline is the complete path of one message from birth to destination. In rsyslog that path is described by three concepts, namely input, ruleset, and action. The system syslog-ng names the same three things sources, log paths, and destinations.
That resemblance is not a coincidence. Every mature log tool follows the same triad because it is the natural answer to one question: where does the data come from, what is done to it, and where does it go.
A real pipeline in a large network is not three stages. It is at least six distinct layers, each with its own responsibility, and the rest of this section walks through all six from the bottom to the top.
Production
Everything starts with an event. An application throws an error, a user signs in, a firewall drops a packet, a Linux kernel reboots. Each of those happens somewhere else, and each producer writes in its own dialect.
Operating system. The Linux kernel keeps its messages in a ring buffer readable through /dev/kmsg, and the systemd service maintains a journal of its own. Both sources produce the records you rely on when judging whether a machine is healthy.
Applications. A web server such as Nginx or Apache, an SSH server, a database, or one of your own services. Each writes logs its own way: some to a file, some to Syslog, some to stdout.
Network. Routers, switches, and firewalls normally speak Syslog and send to a central server.
Cloud and containers. Modern estates add cloud services such as AWS CloudWatch or Azure Monitor, plus container output arriving through stdout and stderr.
Every source that produces logs has to be visible in the pipeline design. The Canadian Centre for Cyber Security recommends keeping a complete inventory of critical sources and checking regularly whether each one really is forwarding.
Transport
Data has to reach the destination, and this layer carries the decisions that determine how far you can trust what arrives.
Push or pull. In a push model the source, or an agent running on it, sends logs toward the destination. In a pull model the destination goes to the source and collects. The choice between them depends on your network topology and your security policy.
Agent or agentless. An agent is a small program installed on each server that reads the logs and forwards them. Its advantage is local control and local processing. Agentless means the device ships its own logs, such as a firewall sending Syslog. The advantage there is simplicity, at the cost of less control over format and schedule.
Protocols. The protocol you choose sets the reliability of everything downstream, because each one buys a different guarantee.
| Protocol | Delivery guarantee | Where it fits |
|---|---|---|
| UDP | None | Simple syslog where occasional loss is accepted |
| TCP | Connection level only | General transport over a reliable network |
| TLS | Connection level, encrypted | Any hop that crosses an untrusted network |
| RELP | Per-message acknowledgement | Paths where a lost message is unacceptable |
| HTTP | Standard request and response | Shipping to Elasticsearch or Splunk HEC |
| Kafka | Durable and replayable | Very high volume that needs replay |
If you use UDP, understand that you are gambling. Nothing guarantees delivery, and when a receive buffer fills the message is dropped without a sound, so UDP is the wrong choice for security and audit logs.
Collection and Buffering
Once logs leave the source they have to land somewhere, and this layer does two jobs: aggregation and buffering.
Aggregation. A central server, or several of them, that receives logs from many sources. It can be a plain rsyslog instance or an entire Elasticsearch cluster. Its main role is to receive the data and hold it temporarily before processing.
Relay. In a large network an intermediate tier sits between the sources and the final destination. A relay collects from many sources and forwards onward. Its value is that it limits access across network boundaries, because the sources reach only the relay, and it spreads the load. rsyslog was built for this role and works across multiple hops: application, node collector, relay, SIEM.
Buffering. This is the most important part of the layer, and it answers the key question here: if the destination is slow or the network drops, where do the logs go? Without a buffer they are discarded. rsyslog uses disk-assisted queues, where data is buffered in memory and spills to disk when needed. The effect is that a slow destination such as an Elasticsearch under load cannot lose the logs meant for other destinations.
In syslog-ng the same idea is expressed as flow control: a destination that cannot keep up makes the source stop reading, so the buffer never overflows.
A buffer without monitoring is dangerous. If the queue fills and nobody notices, logs are dropped silently, and watching the queue fill rate is part of this layer rather than an optional extra.
Processing
Raw data is almost always unusable, and this layer turns it into something searchable and analysable. Four operations do that work.
Parsing means extracting fields from raw text. A single SSH line like the one below carries four facts that any search needs:
Failed password for root from 10.0.1.5 port 52341
Parsed, the same event becomes structured fields:
action=failed_login user=root src_ip=10.0.1.5 src_port=52341
In rsyslog, modules such as mmjsonparse and mmnormalize do this work. In the Elastic Stack, Logstash or the Elasticsearch ingest pipeline take the role.
Normalization means turning different formats into one structure. A firewall may write the time one way and a Linux server another:
firewall: 2026-10-19 14:32:05
linux: Oct 19 14:32:05
result: 2026-10-19T14:32:05Z (ISO 8601)
Without this step, comparison and correlation across sources are simply not possible.
Enrichment means adding information that is not in the original log but matters for analysis. Given an IP address, enrichment can attach geolocation, the organization that owns it, or the asset record behind it. The Google SecOps documentation shows how geolocation and file metadata are attached to events so the analyst gets more context. In rsyslog the same thing is done with custom modules or by calling an API.
Routing means deciding where each message goes. A security log such as a failed login belongs in the SIEM. A performance log such as response time belongs in a performance analytics system. A debug log, which is bulky, can be dropped or sent to cheap archive.
In rsyslog this is done with if/then rules inside a ruleset, and one message can be fanned out to several destinations in parallel.
Smart routing lowers the bill. rsyslog was designed to reduce data before it reaches expensive systems such as a SIEM, and removing noise at the edge cuts bandwidth, CPU, and storage cost in every layer above.
Storage
Processed data has to be stored somewhere it can be searched, and the choice of storage sets your cost and your speed directly.
Index-oriented storage. Elasticsearch and OpenSearch keep data in an index so full-text search stays fast. This is excellent for security analysis and ad-hoc search, but storing the data costs more.
Column-oriented storage. ClickHouse is optimized for aggregations such as failed logins per hour. It compresses better and runs cheaper on bulk data, while full-text search inside it stays more limited.
Raw file storage. Sometimes logs are kept only for archive and compliance and never need fast search. A compressed file or object storage such as S3 is then the cheapest option.
Data lifecycle. Log data loses value as it ages. Today's log is vital for live diagnosis, last month's matters for post-incident analysis, and last year's matters only for compliance. A mature architecture moves data between tiers by age, and Elasticsearch does this with Index Lifecycle Management (ILM):
Hot fast SSD, full-text search
Warm slower disk, full-text search
Cold cheap storage, limited search
Frozen searchable snapshot on low-cost object storage (S3)
A common pattern keeps one day in Hot and everything else in Frozen, which cuts cost by up to 16 times. ILM manages the standard Hot/Warm/Cold/Delete phases and can automate rollover by time or index size.
Retention, the period you keep data, is a legal and economic decision rather than a purely technical one. PCI-DSS requires at least one year of log retention with three months immediately available, and HIPAA and SOC 2 carry requirements of their own, so storage design has to start from them.
Analytics and Response
The last layer is where data becomes a decision, and it has two halves: analysis and response.
Analysis covers searching the data, visual dashboards such as error rate trend charts, and event correlation, which is the subject of SIEM In Depth. A search might be as simple as every failed login for one user in the past 24 hours. Analysis can be manual, performed by an analyst, or automatic, performed by rules.
Response is what happens once analysis reaches a conclusion. An alert to the security team, a ticket opened, or in mature systems an automated action such as locking the account or blocking the IP. This is the layer where SOAR operates.
Analysis without response is incomplete. If the system detects an attack in progress and nothing is done about it, the detection is worth zero, so the pipeline design must state up front which action each kind of alert triggers.
Two Hidden Pillars
Two elements run through every layer above. They look small, but without them the whole pipeline is worthless.
One clock, through NTP. If server clocks are not synchronised, the order of events falls apart during analysis. MyCERT, Malaysia's national incident response centre, warns explicitly that when time is not synchronised across a network's computers, matching actions across different log files becomes difficult.
Even during an intrusion, wrong clocks make it almost impossible to correlate logged activity with external actions. NTP, the Network Time Protocol, synchronises every device against a single time source, and in a log architecture it is a prerequisite rather than an option.
Treat NTP as part of the pipeline, not as an operating system detail. A central log server can also act as the time authority for the estate and monitor whether strays stay in line, which is the same centralization argument applied to time.
One correlation ID. A single user request may pass through ten services. Each writes its own log, and without a shared identifier the analyst cannot tell that the ten belong together. A correlation ID, carried in the request header and written into every log, closes that gap. In distributed systems it plays the role of the tracking number in a postal service, because without it every parcel is a separate story.
The Whole Pipeline in One View
The six layers sit together below, from the sources that produce events at the bottom to the people and systems that act on the data at the top.
Summary
The log pipeline is not a bag of separate parts; it is one interconnected system, and a weak layer weakens everything above it. Unreliable transport loses data, incomplete processing makes analysis impossible, storage without a lifecycle explodes in cost, and unsynchronised clocks make correlation meaningless.
| Layer | What it does | What breaks without it |
|---|---|---|
| Production | Creates the events | Sources nobody inventoried stay invisible |
| Transport | Moves data with a chosen guarantee | Loss that nobody notices |
| Collection and buffering | Gathers and holds data through outages | Logs dropped when the destination slows |
| Processing | Parses, normalizes, enriches, routes | Searches that quietly return wrong answers |
| Storage | Keeps data searchable by age tier | Cost that grows without limit |
| Analytics and response | Turns findings into action | Detection with no consequence |
Good architecture sees these six layers together rather than apart. A SIEM is only the analytics and response layer, and without the five layers beneath it, it has nothing to analyse. The next section opens that layer up.
Previous: Section 2 — The Log Server Family · Next: Section 4 — SIEM In Depth · All sections