Section 6 — Full Scenarios and Problem Solving: When All the Layers Meet

The previous section closed with the challenges that decide whether central logging survives production. This section puts every earlier piece into a real situation, because a log architecture is not a set of components but something that either holds under pressure or does not.

This section runs three complete scenarios. Each one starts from an initial state, follows the events across every layer, shows what each tool sees and what it misses, exposes the breaking point, and then gives the architectural fix. By the end of the third scenario you should see the whole picture rather than isolated pieces.

A Slow Breach: From Phishing to Data Exfiltration

Initial State

A company runs 300 servers, 20 firewalls, and 5,000 users, protected by a security team of four. It runs a SIEM, an Elastic Stack log management system, and a Filebeat agent on every server. Firewalls forward their logs over Syslog to a relay that delivers them to the central system, and NTP keeps every clock in step.

Events Across the Layers

Four timestamps tell the whole story.

10:15 PM, production layer, phishing. A member of the sales department receives an email that appears to come from the human resources service and asks for a password "update". The employee clicks the link and types the password in. The attacker now holds the credential, and no system logs this event because the phishing happened on a site outside the organisation.

10:20 PM, production layer, login. The attacker signs in from an external IP address never seen before at this company, using the employee's credentials. Active Directory, the authentication server, records the successful sign-in: event_id=4624, user=ali, src_ip=185.x.x.x, logon_type=3.

10:35 PM, production layer, lateral movement. The attacker uses the ali account to reach a file server. The file server records the connection: user=ali, src_ip=185.x.x.x, target=fileserver-03, share=finance$.

10:50 PM, production layer, exfiltration. The attacker compresses 2,000 financial files from the file server into a single archive and uploads it to a cloud file-sharing service. The firewall records the outbound traffic: src=fileserver-03, dst=185.x.x.x, bytes=450MB, proto=HTTPS.

10:20  Active Directory  event_id=4624, user=ali, src_ip=185.x.x.x, logon_type=3
10:35  fileserver-03     user=ali, src_ip=185.x.x.x, target=fileserver-03, share=finance$
10:50  firewall          src=fileserver-03, dst=185.x.x.x, bytes=450MB, proto=HTTPS

What Each Tool Sees

The log management system. Elastic Stack sees all three events, but it sees them separately. A search for user=ali returns three results: a sign-in from a new IP address, access to the finance folder, and a 450-megabyte upload. None of them looks like an attack on its own. A new IP can be a business trip, and finance folder access can be part of the job.

The SIEM. Here the difference appears. The SIEM holds a correlation rule that reads: alert if a user signs in from a new IP, then reaches a folder they do not normally access, then uploads an unusual volume of data. It joins the three events inside a 30-minute window and raises a high-priority alert naming ali as a probable data exfiltration.

The Breaking Point

The alert is raised at 10:55 PM, and the four-person security team is off duty at that hour. The alert sits in the queue until morning. The next morning the analyst reads it and finds the data already gone, uploaded hours earlier.

Danger

Five hours separate the alert from the response, and five hours is exactly what the attacker needed to exfiltrate 2,000 financial files. A detection nobody is awake to receive is not a control.

The Architectural Fix

Three architectural lessons come out of this scenario.

First, the alert must reach the right channel. A high-priority alert should not sit only in a SIEM dashboard. It must go to a messaging system such as PagerDuty or a Telegram or Slack channel that can wake the team at 3 AM. Designing the escalation path is part of SIEM architecture, not a later thought.

Tip

Write the escalation path down before the first alert ever fires: who gets woken, in what order, and what happens when nobody acknowledges. An escalation path that exists only in someone's head is not an escalation path.

Second, the response must be automated. With a SOAR platform, the organisation could define a playbook: when a high-priority exfiltration alert fires, lock the user account, terminate its active sessions, and block the destination IP at the firewall. That response runs in seconds and stops the attacker even while the security team is asleep.

Third, phishing is a gap in the pipeline. Phishing was logged by no system because it happened on an external site. To close that gap, the organisation must also collect Email Gateway logs and web proxy logs, so it can tell that a user clicked a suspicious link. The log pipeline is only complete when it covers every relevant source rather than just servers.

A Network Outage and Data Survival

Initial State

A company operates two data centres, one in Tehran and one in Isfahan. The Tehran servers forward their logs to a central server in Isfahan over an MPLS link with limited bandwidth. The central server runs a Disk-Assisted queue that buffers incoming data on disk when the link fails.

Events Across the Layers

02:00, the link fails. The MPLS link between Tehran and Isfahan goes down because a fibre cable has been cut on the route. The repair estimate is six hours.

02:00 to 08:00, production layer, business as usual. The Tehran servers keep working. A web server in Tehran starts throwing repeated errors, a user makes failed login attempts at 03:00, and a firewall blocks suspicious traffic. All of it is logged, and none of it can reach Isfahan.

02:00 to 08:00, collection layer, buffering. A Filebeat or rsyslog agent on each Tehran server collects the logs, and when the link fails the agent cannot reach its destination. A local queue on the agent holds the data on disk. The central server in Isfahan runs its own Disk-Assisted queue for everything arriving from other sources, such as the Isfahan firewalls.

08:00, the link returns. The MPLS link is repaired and the Tehran agents start draining their local queues. Six hours of data reaches the central server in Isfahan in order, and the central server drains its own Disk-Assisted queue. No log is lost.

What Would Have Happened Without Buffering

Without a buffer on the Tehran agents, every new log line would either sit in memory until something overwrote it or be dropped on the spot. Six hours of data would be lost.

That six-hour window would hold the web server errors, the failed login attempts, and the suspicious firewall traffic. The next morning the security team could not say what happened, and the operations team could not say how the web server behaved.

Worse than the lost data is the loss of trust. Once the security team knows that every network outage silently costs six hours of data, it can no longer trust its own analysis. An attack may have happened inside that window, and nobody would ever know.

Warning

A Disk-Assisted queue discards data without a sound once it fills up. Nothing about it shows on the dashboard, so a team that never watches the queue depth can believe it has full coverage while evidence quietly evaporates.

The Architectural Fix

Three architectural lessons come out of this scenario.

First, buffer at several layers. Buffering only on the central server is not enough, because the failure can happen at any point along the path. The agent needs its own buffer too. A mature architecture buffers both at the edge, in the agent, and in the centre, in the relay or aggregator.

Second, the queue must be monitored. A Disk-Assisted queue drops data silently once it fills. Monitoring the queue depth and the discard rate is therefore part of the architecture rather than an optional extra.

Third, queue capacity must be designed. A Disk-Assisted queue has a finite capacity, for example 2 gigabytes, and an outage that outlasts that capacity costs data. The design starts from the longest tolerable outage: to survive 24 hours offline, the queue must hold 24 hours of data. This is an economic decision, because more capacity means more disk.

Post-Incident Analysis and Compliance

Initial State

A financial services company discovers that its customer data is for sale on a black market. The attack probably started weeks earlier and nobody noticed. The security team has to establish what happened, which data was affected, and what action the regulator requires.

Events Across the Layers (Reconstructing the Past)

The production layer. Web server, database, Active Directory, firewall, and Email Gateway logs. All of them have been stored in the central system for the last 90 days.

The storage layer. The last 30 days of data sit in the Hot tier on SSD and answer searches quickly. Data from 30 to 90 days ago sits in the Warm tier on slower disk, and anything older than 90 days is archived in Frozen on object storage.

The analysis layer. The team defines a search window covering all suspicious activity in the last 90 days, and the SIEM builds a timeline:

• Day 1: a phishing email reaches the user reza. The Email Gateway logged it, but the SIEM marked it low priority because thousands of phishing messages are blocked every day.
• Day 2: reza signs in from a new IP address in a neighbouring country. Active Directory logged it, and the SIEM tagged it as "login from a new IP", but at low priority because business travel is common.
• Day 3 to 10: reza gradually gains access to more folders and downloads small files. The SIEM tags the pattern as unusual access, but because it falls inside working hours it receives medium priority.
• Day 15: a large 2-gigabyte upload goes to an external cloud service. The firewall logged it, and the SIEM tagged it as data exfiltration and raised a high-priority alert. Nobody worked the alert.

What the Post-Incident Analysis Shows

Using the correlated logs, the security team can reconstruct the attack exactly:

Stage	Day	The log that holds the evidence
Entry point	1	The Email Gateway holds the phishing email
Stolen credential	2	Active Directory holds the sign-in
Lateral movement	3 to 10	The file server holds the folder access
Exfiltration	15	The firewall holds the outbound traffic

Every step is reconstructable from the last 90 days of logs. The company can state precisely which data was affected, namely 2,000 financial files, which customers are involved, and what action the regulator requires.

The Breaking Points

The breaking point in this scenario is failing to respond to the alert. The SIEM raised it on day 15 and nobody handled it. With a response procedure in place, for example a rule that every high-priority alert is reviewed within one hour, the attack would have stopped on day 15 rather than weeks later.

The second breaking point is noise reduction done badly. If the Email Gateway blocks thousands of phishing emails a day and every one of them becomes an alert in the SIEM, the real alert disappears into the noise. A well-designed SIEM separates events that look alike: a blocked phishing email is not the same thing as a clicked phishing email.

Security

An unhandled high-priority alert is worse than no alert at all. It tells the team they are protected while the one detection that mattered sits unread in a queue, and by then the data is already for sale.

The Architectural Fix

Three architectural lessons come out of this scenario.

First, retention must be designed from the compliance requirement. If the company kept logs for only 30 days, it could not reconstruct an attack that started weeks earlier. PCI-DSS requires at least 12 months of retention, and that requirement exists for exactly these cases.

Second, the data lifecycle must stay searchable. Older data lives in Frozen but remains searchable, only more slowly. If old data were merely archived and could not be searched, post-incident analysis would be impossible.

Third, the response procedure must be defined. A SIEM without a response procedure is only an observer. The team must know who receives each high-priority alert, inside and outside working hours, and what action follows. That procedure is part of the architecture rather than a later thought.

Summary

Three scenarios, one shared lesson: a central log architecture works only when all of its layers are seen together.

• A slow breach shows that a log platform can hold every relevant event and still miss the attack, because correlation and a staffed escalation path were missing. The five-hour gap between alert and response cost 2,000 financial files.
• A network outage shows that buffering at the edge decides whether six hours of logs survive, and that an unmonitored queue loses data in silence even when nothing visibly fails.
• Post-incident analysis shows that retention, a searchable lifecycle, and a defined response procedure are what turn a detected attack into a provable one, and that the PCI-DSS 12-month minimum exists for precisely this case.

Centralised logging is not a set of separate parts. It is one chain, and a weak link anywhere in it weakens the whole. Designing that chain from the first step is the subject of the next section.

Previous: Section 5 — Operational Challenges and Privacy · Next: Section 7 — Summary and Design Principles · All sections