Section 2 — The Log Server Family: A Map of the Tools
In the previous section we saw why scattered logs fail in large networks. But centralization is not a single tool; it is a whole family of tools, each supplying part of the answer. Not knowing that family is how an organization ends up buying the wrong tool for the wrong problem.
This family has five principal members, plus a handful of supporting tools the architecture cannot run without. This section takes each member in turn and states what it does, what it deliberately does not do, and where it sits in the architecture.
Why "Log Server" Is a Misleading Phrase
When someone says "we have a log server", that single sentence can point at ten different things. It might be a plain rsyslog writing network messages into files, or an Elasticsearch cluster indexing millions of log records. It might be an enterprise SIEM correlating threats, or some combination of all three.
That ambiguity is dangerous in practice. An organization that believes it has a log server may in fact have nothing but a file warehouse, and at the moment of an incident it discovers it has no search tool and no analysis tool. The opposite case is just as common: a company that bought an expensive SIEM, only to find that nobody configured the correlation rules, leaving nothing but a useless dashboard.
The fix is to know the family and know what each member contributes. The five principal members are introduced below, one at a time, with their names, roles, and boundaries.
Syslog Server
The simplest and oldest member of the family. A Syslog server is a program that receives Syslog messages from network devices, stores them, and makes them searchable. Those messages usually come from routers, switches, firewalls, and Linux servers that speak the standard Syslog protocol.
What it does: collect, store, and display messages. Some versions can even receive SNMP Trap alongside them. For a network operations (NetOps) team that wants to know why a switch keeps rebooting, this tool is enough and appropriate.
What it does not do: advanced security analysis, event correlation, or automated response. The most a good Syslog server does is send an alert email when several failed logins are recorded.
Real-world examples: rsyslog, which the earlier writing on its architecture covered, syslog-ng, and Kiwi Syslog Server. These tools are usually open source or inexpensive, and they install on every Linux distribution.
A Syslog server is a pipe, not a brain. If you only need collection it is sufficient, and if you need analysis you have to move a layer higher.
Log Management System (LMS)
One step above a raw Syslog server. A log management system gathers data from far more diverse sources than Syslog alone, normalises and indexes it, and makes fast search across enormous volumes possible. Its primary goal is making data usable, not necessarily analysing it for security.
What it does: collection from multiple sources such as files, APIs, and agents; format normalisation; long-term storage; fast search; visual dashboards; and performance analysis and troubleshooting. It answers the question "what happened?", not the question "is this dangerous?".
What it does not do: advanced security analysis. A log management system can show that a user logged in at 3am from a new IP address, but it cannot say that this login is part of an attack chain. For that, you need a SIEM.
Real-world examples: Elastic Stack, previously called ELK, which is Elasticsearch plus Logstash plus Kibana plus Beats, is one of the most popular open-source options. Beats collects the logs, Logstash processes them, Elasticsearch indexes and stores them, and Kibana displays them. Graylog is another option. Loki, from Grafana, takes a lighter approach: it indexes only the labels and keeps the log text compressed in cheap storage.
Elastic Stack is not a SIEM out of the box. It is a platform you can build SIEM-like capabilities on, but the correlation rules, the alerts, and the detection logic are yours to write, and the free version ships no built-in correlation engine. Tools such as Elastalert close part of that gap.
SIEM (Security Information and Event Management)
Now we reach the SIEM, which settles into its own place in the architecture. A SIEM is a log management system with one extra layer added on top: a security analysis layer. The key difference lives entirely in that extra layer, namely event correlation and threat detection.
What it does: everything a log management system does, plus correlation rules (for example "failed login, then access to the finance folder, then abnormal data volume"), matching against a database of known threats, prioritised security alerts, and in advanced editions automated response. It answers the question "is this dangerous?".
The fundamental difference from a log management system: a log management system says "user X logged in from a new IP address". A SIEM combines that with other data and says "user X, who had never previously logged in from that IP, signed in at 3am, then tried to raise their own privileges, and after that read 500 records from the database. This combination matches the pattern of an internal intrusion." That combination is correlation.
Real-world examples: Splunk, which is both a log manager and a SIEM, Wazuh, which is open source and focused on intrusion detection and compliance, and IBM QRadar or Microsoft Sentinel at the enterprise level. A SIEM is built for a security team (SecOps) and a security operations centre (SOC), not for the day-to-day troubleshooting of one service.
This tool is expensive and complex, and if it is not configured properly it floods the security team with false positives until the team is exhausted. A SIEM with no correlation rules configured is just an expensive log management system.
SOAR (Security Orchestration, Automation and Response)
One more layer, sitting on top of the SIEM. A SOAR does not collect logs itself, or very little of them; it uses the data the SIEM produced in order to automate the response to a threat. If the SIEM says "this is an attack", the SOAR can automatically update the firewall, lock the user account, or open a ticket for the support team.
Difference from the SIEM: the SIEM detects and the SOAR responds. A SOAR runs a playbook, which is a set of predefined actions executed in response to a type of incident. The SIEM is an analytical tool and the SOAR is an operational one.
Real-world examples: Palo Alto Cortex XSOAR, and Shuffle, which is open source. These tools suit organizations whose volume of security incidents has outgrown what their team can handle manually.
This tool is expensive and complex, and it needs a mature team to design and maintain its playbooks. For an organization that has only just started, a SOAR is too early.
XDR (Extended Detection and Response)
A newer generation whose focus is unifying security data from scattered sources, namely the endpoint, the network, email, and cloud, and analysing it automatically. XDR evolved from EDR, the Endpoint Detection and Response: EDR focused on the endpoint alone, while XDR extends the scope to the network, the server, the cloud, and beyond.
Difference from the SIEM: the SIEM focuses on "collect everything" and then analyses it. XDR focuses on deliberately collecting the data that has the highest detection value, and it uses the endpoint as its correlation anchor. An XDR is usually presented as a complement to a SIEM rather than a replacement: the SIEM is the data store and the correlation engine, and the XDR is the analysis and faster-response layer on top of it.
Real-world examples: CrowdStrike Falcon, Cisco XDR, and Microsoft Defender XDR. These tools earn their place in large networks that already run several separate security products and want one integrated view across all of them.
This tool is usually tied to a single vendor's ecosystem and does not offer the SIEM's flexibility for custom rules. If your organization runs security tools from several different vendors, an XDR may not give you the complete picture.
Supporting Members: Pipes and Bridges
Beyond the principal members, a few supporting tools play the part of pipes, and the architecture does not work without them.
Log shipper: a lightweight agent that runs on every server, reads the log files, and sends them to the central system. Its responsibilities are reading the file, preliminary processing of the format, adding metadata such as the host name and the environment, buffering when the destination is unreachable, and forwarding to the central system. Examples are Filebeat for ELK, Fluentd, Fluent Bit, Vector, and Promtail for Loki.
Log relay: an intermediate server that gathers logs from several sources and forwards them to the final destination. A relay is especially useful when you want to move logs between different networks while restricting access between those networks. It can also do extra processing such as filtering and transforming the data.
Log aggregator: a central service that receives the logs sent to it. This can be a simple rsyslog server, or an Elasticsearch cluster, or Loki.
These tools are not SIEMs themselves and perform no security analysis; they only move data around. Without them, none of the principal members of the family has an input.
The Comprehensive Comparison Table
The table below puts the five principal members and the supporting tools side by side, so that the difference between collecting, searching, detecting, responding, and transporting stays visible at a glance.
| Type | Main goal | What it collects | Primary audience | Example | Does it do security analysis? |
|---|---|---|---|---|---|
| Syslog Server | Concentrate network messages | Syslog from routers, switches, firewalls | NetOps | rsyslog, syslog-ng, Kiwi | No (storage and display only) |
| Log Management (LMS) | Search and analyse data | Logs from any source (file, API, Agent) | DevOps, SRE, compliance | Elastic Stack, Graylog, Loki | No (unless you add it yourself) |
| SIEM | Security threat detection | Security logs and correlation | SecOps, SOC | Splunk, Wazuh, QRadar, Sentinel | Yes (correlation and alerting) |
| SOAR | Automating the response | Data from the SIEM | SOC (incident response) | Cortex XSOAR, Shuffle | No (it only responds) |
| XDR | Unified security visibility | Endpoint, network, and cloud | Security team | CrowdStrike, Cisco XDR | Yes (analysis and response) |
| Log Shipper | Transporting the data | Local log files | Everyone | Filebeat, Fluentd, Vector | No (it only transports) |
Reference Architecture: Stacked, Not Competing
A common mistake is to believe that you have to choose just one of these tools. In practice, these tools stack on top of one another in layers. A reference architecture has three layers.
Layer 1, collection and transport: on every server, a log shipper such as Filebeat or rsyslog collects the logs and sends them to a central server. If the network is large, one or more relays sit in between so that the load is spread out and access between networks stays restricted.
Layer 2, storage and search: a log management system such as Elastic Stack or Loki receives the data, normalises it, indexes it, and stores it. This layer answers the question "what happened?" and is used for troubleshooting, performance analysis, and compliance.
Layer 3, security analysis and response: a SIEM such as Wazuh or Splunk feeds on that same data and correlates threats. If the team is mature, a SOAR automates the simple responses. If the organization runs several security tools, an XDR can provide an integrated view across the endpoint and the network.
Every layer has its own job, and removing one weakens the layer above it. Without Layer 1, Layer 2 has no data. Without Layer 2, Layer 3 cannot search. Without Layer 3, the data exists but no security analysis does.
An Honest Warning: Choose the Tool for the Problem, Not the Reverse
This section needs one honest warning at the end. Many organizations buy a SIEM because "everyone else buys one", without knowing which problem they want to solve. The result is an expensive tool that nobody uses.
Before choosing a tool, answer these questions:
- Do you only want to collect and search logs? A log management system is enough, and a SIEM is not required.
- Do you need security threat detection? A SIEM is required, but only if you have an analyst team that configures the correlation rules.
- Is your incident volume beyond what the team can handle manually? Consider a SOAR.
- Do you run several different security tools and want an integrated view? An XDR may suit you, but weigh the limitations of its ecosystem.
The answers to those questions choose the tool, not the other way round. These tools stack rather than compete, so you do not have to buy them all at once; choose the one your problem asks for.
The next section takes these same layers apart in technical detail, from the point where a log is produced to the point where the final analysis is produced.
Summary
- "Log server" is one name for several different tools, so know whether you are holding a file warehouse or a security analytics platform before you buy anything.
- The family has five principal members: the Syslog server, the log management system, the SIEM, the SOAR, and the XDR.
- These tools stack on top of one another rather than competing, and removing any layer weakens the layer above it.
- The log management system answers "what happened?", and the SIEM answers "is this dangerous?"; only the second one correlates events into an attack chain.
- Choose the tool for the problem, not the reverse; buying an expensive platform without a defined problem produces cost and nothing else.
Previous: Section 1 — Why Centralization · Next: Section 3 — Technical Architecture · All sections