Section 5 — Operational Challenges and Privacy: What Separates Theory from Practice
The previous section described the SIEM as the analytical brain of the platform. A SIEM on paper and a SIEM in production are different systems, and the gap between them is where these projects fail. This section is about that gap, meaning the challenges that appear only once real traffic, real attackers, and real invoices arrive.
Five foundational challenges face every central log architect sooner or later, namely data volume and storage cost, data quality, the security of the logging system itself, privacy and compliance, and time. A sixth follows at the end of this section and is discussed far less often, namely people and continuous tuning. None of them are highlighted in a vendor brochure, yet all of them decide whether a central logging project succeeds or dies.
Data Volume and Storage Cost
The first challenge is the simple one: logs grow. A medium-sized network produces anywhere from a few hundred gigabytes to a few terabytes of logs per day. Keeping all of it for a year costs more than most organisations ever budget for.
The bill is not only about raw storage. Indexed storage costs more than a flat file, because an index exists to make search fast, and Elasticsearch is built that way. Keep every record in the Hot tier on fast SSD and the cost climbs steeply with the volume.
Mature architectures therefore use a data lifecycle. Fresh data stays in Hot, older data moves down to Warm and Cold, and the oldest records end up in Frozen, which is cheap object storage.
Designing that lifecycle is an economic and legal decision, not just a technical one. The PCI-DSS standard explicitly requires at least 12 months of log retention, with the most recent 3 months immediately available. You cannot delete everything after one month, and keeping everything forever has no end.
The practical answer is a retention window based on risk. Recent data stays fast and searchable, older data moves to archive, and anything with no legal or operational requirement gets deleted.
A university study found that fewer than 5 percent of records older than 90 days are ever queried. Most of what you keep is never used in practice, but that 5 percent becomes vital the moment an investigation reaches for it.
Before buying any tool, answer one question: what do we keep, for how long, and why. Without a clear answer, the storage bill grows toward a limit nobody in the project can point to.
Data Quality
The second challenge is subtler and is usually discovered late: the data you have may not be what you think it is. Three different failure shapes show up in practice.
Silent sources. A firewall, a server, or a critical application may stop sending logs because of a configuration error. The central system raises no alarm, because technically everything works and only the data fails to arrive.
The Canadian Centre for Cyber Security stresses that organisations should regularly verify whether every critical source really is forwarding, since a silent source is an invisible security hole. The answer is source health monitoring: give every critical source a time threshold and alert when nothing arrives.
A practical threshold looks like this: alert if no log has arrived from a given server in the last 10 minutes. Source health has to be monitored continuously rather than verified once during installation.
Broken formats. An application may ship a new release that changes its log format. When the parsing rule no longer matches the new layout, fields are no longer extracted and the log stays raw text. The SIEM still sees the record but cannot analyse it.
Watch the parse success rate, which is the share of logs whose fields are extracted correctly. When that percentage drops, an application changed its format without telling you, and every rule that reads those fields is silently broken.
Meaningless data. A large share of logs is noise: debug messages, duplicates, and events with no analytical value. Noise raises storage cost and lowers the signal during analysis.
The answer is to filter at the edge, before the data reaches the central system. rsyslog was designed for exactly this job, reducing data before it reaches the expensive systems downstream.
Security of the Logging System Itself
The third challenge is a paradox: a system built to monitor security is itself a target. If an attacker reaches the SIEM, they can delete the evidence, suppress the alerts, or mine the log data for other weak points.
The SIEM becomes a high-value target, which is a single central point that sees everything and loses everything if it is compromised. Three practical consequences follow from that risk.
The SIEM is one of the most attractive targets in the estate. It holds every security signal in one place, so compromising it destroys the evidence, silences the alerts, and hands the attacker a map of your defences. Protect the logging platform itself like the most sensitive system you run.
Access control. Access to the SIEM should follow the principle of least privilege. Not every analyst needs every dataset. A study showed that restricting raw log access to fewer than 20 percent of SOC staff cut the likelihood of accidental data disclosure by about 64 percent.
Separating sensitive data. If personal data and security events are stored in one place, a single intrusion exposes both at once. The answer is to keep sensitive fields in a separate store and replace them with tokens in the main system, which sharply reduces the volume of data leaked in a breach.
Data integrity. If an attacker can modify logs, the evidence loses its credibility. The answer is immutable storage, in which a record cannot be changed once it is written.
Store security logs immutably and restrict who may delete anything at all. An attacker with write access to the evidence destroys its value entirely, and no later investigation can prove what actually happened.
Privacy and Compliance
The fourth challenge is the most complex and the most legal: logs contain personal data. A login log holds a username, a web access log holds an IP address, and an application log may hold an email address or a phone number.
Privacy law protects this material, and GDPR in Europe is the best known example of it. Even an IP address counts as personal data under some GDPR interpretations. That means privacy has to enter the design of a central log system at the start rather than as a later thought, and it rests on three key principles.
Data minimisation. Collect only the data you genuinely need. If a field is not essential for security analysis, do not collect it. Article 5.1.c of the GDPR requires exactly this, and in practice it means reviewing log fields regularly and removing those with no analytical value.
Redaction. Sensitive fields such as passwords, credit card numbers, or health data have to be masked before storage. A study showed that encrypting selected sensitive fields instead of the whole record keeps the security and markedly improves search performance. The figures were up to 54 percent better throughput and up to 42 percent lower query latency.
GDPR data minimisation is a collection rule rather than a storage rule: a field that is never collected cannot leak in a breach. Combine it with redaction at the source, because masking only the fields you already hold leaves the rest of the record exposed.
Role-based access control. Not everyone should see every log. A security analyst may need authentication logs and no access to the logs of a financial application. Access design should follow need to know rather than convenience.
Compliance. Different standards set different log retention requirements. PCI-DSS requires at least 12 months of retention with 3 months of immediate access. HIPAA is stricter for health data, and SOC 2 focuses on security controls. A log system has to be designed against these requirements from the start, because changing it after deployment is expensive and slow.
With a cloud SIEM or a managed service, the vendor is the data processor and you remain the controller. You have to know where the data is stored, who can reach it, and what deletion commitments the contract carries at the end of the term.
Time (NTP)
The fifth challenge looks small and is foundational: the clock. If server clocks are not synchronised, the order of events falls apart during analysis.
ISO 27001 control 8.17 explicitly requires the clocks of all information-processing systems to be synchronised to a single time source. The control looks simple, yet almost every other aspect of security depends on it.
Here is what a one-minute skew does to an investigation:
firewall suspicious connection detected 10:02:15
IDS same traffic observed 10:02:45
app server unexpected login 10:03:00
The order of those three lines is the whole investigation. If the clocks of these systems differ by even one minute, the analyst cannot establish what happened first, and a coordinated attack dissolves into unrelated noise.
The fix is NTP, the Network Time Protocol, which synchronises every device against a single time source. NTP itself has to be secured as well, because an attacker who can move the clock can corrupt the evidence everything else depends on.
NTP is a prerequisite rather than an option. An architecture without it is built on shifting sand, and the failure surfaces much later as correlation that quietly returns the wrong answer.
Restrict NTP traffic to your authorised time servers, authenticate it where the protocol allows, and monitor it. An attacker who shifts time backwards can bury an intrusion outside every retention window you rely on.
People and Continuous Tuning
One challenge is missing from every brochure and belongs here: a SIEM without skilled people is only an expensive display. Correlation rules have to be written, tested, and updated. False alerts have to be reduced, and new threats demand new patterns.
This is not a one-off project but a continuous process. An organisation that buys a SIEM without assigning a team to tune it ends up with a system full of false alerts that exhausts the analysts and buries the real ones.
This is the alert fatigue described in Section 4. The technical fixes, namely grouping, scoring, and enrichment, only work when a human team stands behind them and keeps tuning and reviewing the rules. Without that team the SIEM becomes a sleeping prince, expensive, beautiful, and useless.
Do not buy a SIEM without a team to run it. A SIEM nobody tunes does not fail loudly, it fails quietly, and it burns analyst trust until somebody eventually switches the alerts off.
Summary
These challenges are not obstacles that get solved later. They are part of the design, and a central log architect has to answer them from the first day:
- What do we keep, for how long, and why? (cost and compliance)
- How do we know that every critical source really is forwarding? (data quality)
- What happens if the SIEM is hacked? (security of the system itself)
- What personal data lives in the logs, and how is it protected? (privacy)
- Are the clocks of all systems synchronised? (NTP)
- Who tunes and maintains the rules? (people)
The answers separate a central logging system that works in production from one that only looks good on paper. They are also the subject of the next section, which puts every one of these challenges into complete scenarios.
Previous: Section 4 — SIEM In Depth · Next: Section 6 — Full Scenarios and Problem Solving · All sections