Smart Logging Guide for Rocky Linux

This document is written for infrastructure engineers who want to stop memorizing dozens of commands and instead translate their thoughts directly into commands. If you have ever received a call at 11 PM asking "Why was Nginx down between 3 and 5 PM?" and you knew exactly what you wanted but not what to type, this guide is for you.

Where Is the Problem?

There is a deep gap between what is in your head and what you need to type. The problem is usually not that you don't know journalctl or grep — it is that you don't know how to turn your mental question into a precise command.

Tip

The problem is not just finding a command. You need to know which log to inspect, which time range to select, and what information to extract from a large volume of output.

What Does This Guide Change?

Instead of memorizing commands, you will learn this pattern:

Mental question → Log source → Right filter → Linux command → Result

Once you internalize this pattern, you no longer need to search for a ready-made command for every new problem. You just build the right question and translate it step by step. This document acts as a mental translator — not a dictionary to memorize, but a way of thinking that keeps you from forgetting commands.

Scene One: "Give Me the Nginx Log from 3 to 5"

Let's solve that real scenario step by step — not by memorizing a command, but by breaking down the thought and translating each piece.

The Chunking Technique

The human brain cannot hold a long string of characters all at once. But if you break it into meaningful chunks, each chunk becomes like a familiar word. This is the Chunking technique from cognitive psychology.

Your thought has four words, and each word becomes exactly one piece of the command:

Your thought	Command part	Meaning
log	journalctl	Tool to view systemd logs
Nginx	-u nginx	Filter by service name
from 3 o'clock	--since "15:00"	Start of time range
to 5 o'clock	--until "17:00"	End of time range

The final result is exactly what was in your head:

journalctl -u nginx --since "15:00" --until "17:00"

Tip

This is like ordering food. You don't say "Please prepare the chemical combination of bread, cheese, and vegetables at 200 degrees for 12 minutes." You say: "Give me a veggie pizza." Logging works the same way — you just need to know the menu items (the command chunks).

Info

Mental Exercise — Try It Right Now

Before moving on, solve this scenario in your head:

"Give me the SSH service log from yesterday until now"

Thought	Command
log	journalctl
SSH	-u sshd
from yesterday	--since "yesterday"
until now	not needed — the default is "until now"

journalctl -u sshd --since "yesterday"

If you could build this command without looking at the answer, congratulations — your brain is starting to think like a terminal.

Know Your Brain — Why You Forget Commands

Before going further, we need to understand why learning Linux commands is hard. When you know the enemy, victory becomes easier.

The Three Enemies of Your Memory

• Cognitive load: Your brain holds only 7±2 items in short-term memory at once. A command like journalctl -u nginx --since --until -p err -o json has more than 7 chunks.
• No semantic connection: When you memorize --since, it has no semantic link to the thought "from such-and-such time." It is just a string of characters that gets erased.
• No repetition: The Ebbinghaus forgetting curve shows that without repetition, 70% of information is lost within the first 24 hours.

Our Three Weapons

We use three cognitive psychology techniques:

Technique	Plain explanation	Use in logging
Chunking	Breaking large information into small meaningful chunks	Each thought = one command chunk
Mental Model	Building a mental map of how the system works	Understanding "where logs come from and where they go"
Pattern Recognition	Learning patterns instead of memorizing	5 golden patterns = 90% of needs

Where Are the Logs? — Mental Map of Rocky Linux

Before anything else, you need to know where logs come from, where they are stored, and how to reach them. On Rocky Linux, every service produces logs, and these logs are stored in two parallel paths.

The Golden Rule: Where Do I Look?

This is the simplest rule. Memorize it:

Your thought	Where to look	Tool
"I want the log of service X"	Journal	journalctl -u X
"I want the service's own file log"	/var/log/	cat, grep, tail
"I want the kernel log"	Both	dmesg or journalctl -k
"I want the user login log"	Specific file	last, lastb, lastlog
"I want the security (audit) log"	Specific file	ausearch, aureport
"I want the Docker log"	Docker engine	docker logs

Tip

Imagine your city has two libraries: a digital one (Journal) that works with a search engine, and a traditional one (log files) where you have to walk the shelves yourself. Most of the time, the digital library is enough.

The Right Path to Finding a Log

You don't need to memorize the path of every log file. First, decide what kind of information you are after: a service, the kernel, user logins, or security events.

If the service is managed by systemd, journalctl is usually the best starting point, because you can filter by service, time, severity level, and even PID. For applications that keep their own logs, check /var/log/. For kernel, driver, or hardware issues, dmesg and journalctl -k give you a better view.

The core principle is this: first identify the type of log, then choose the storage location and the right tool.

The Five Golden Patterns — 90% of Your Needs

Research shows that 90 percent of an infrastructure engineer's daily logging tasks can be handled with just 5 patterns.

#	Mental question	Command
1	"Give me the log of such-and-such service"	journalctl -u service-name
2	"From such-and-such time to such-and-such time"	journalctl --since "..." --until "..."
3	"Only errors / warnings"	journalctl -p err
4	"Show me live"	journalctl -f -u service-name
5	"Combine the above"	journalctl -u nginx -p err --since "yesterday"

The Formula for Combining Patterns

Now look at the general formula:

journalctl  +  [what?]  +  [when?]  +  [how important?]  +  [how to display?]

journalctl -u service --since "start" --until "end" -p level -o format

Each part is optional. You don't need to write all of them:

# Only "what"
journalctl -u nginx

# "what" + "when"
journalctl -u nginx --since "today"

# "what" + "when" + "how important"
journalctl -u nginx --since "today" -p err

# "what" + "live"
journalctl -f -u nginx

Tip

systemd severity levels from low to high: debug, info, notice, warning, err, crit, alert, emerg. When you pass -p err, all levels above err are also included.

Complete Translation Table: Thought → Command

This table is the most important part of this document. Every time a thought forms in your head, look here.

"What?" — Service Name

Your thought	Command
Give me the Nginx log	journalctl -u nginx
Give me the SSH log	journalctl -u sshd
Give me the firewall log	journalctl -u firewalld
Give me the Docker log	journalctl -u docker
Give me the PostgreSQL log	journalctl -u postgresql
Give me the MariaDB/MySQL log	journalctl -u mariadb
Give me the Apache log	journalctl -u httpd
Give me the Cron log	journalctl -u crond
Give me the DNS log	journalctl -u named
Give me the NTP/time log	journalctl -u chronyd
Give me the network log	journalctl -u NetworkManager
Give me the email log	journalctl -u postfix
Give me the everything log	journalctl
Give me the kernel log	journalctl -k

Tip

Don't know the service name? systemctl list-units --type=service shows a list of all services.

"When?" — Time Range

Your thought	Command
Since today	--since "today"
Since yesterday	--since "yesterday"
Last one hour	--since "1 hour ago"
Last 30 minutes	--since "30 min ago"
Since 3 o'clock	--since "15:00"
From 3 to 5 o'clock	--since "15:00" --until "17:00"
Specific date	--since "2024-06-15"
Specific date and time	--since "2024-06-15 09:00"
Since this boot	-b
Since the previous boot	-b -1

"How Important?" — Severity Level

Your thought	Command	Explanation
Only critical	-p crit	System at risk of collapse
Only errors	-p err	Important errors
Errors and above	-p err	Includes crit, alert, emerg
Only warnings	-p warning	Potential problems
Everything from warning upward	-p warning	Includes err, crit, alert, emerg

"How to Display?" — Output Format

Your thought	Command	Explanation
Show normal	default	Standard format
Show short	-o short	Date + message
Show JSON	-o json-pretty	For machine processing
No pager	--no-pager	Good for piping
Last N lines	-n 50	Like tail
Show live	-f	Like tail -f
Reverse (newest first)	-r	Reverse order

```mdx
## Real-World Scenes — Practical Practice

Now let's solve **real scenarios** using this same method. Each scene is a real situation you have probably faced.

### Scene 2: "Why Did the Server Restart?"

You came in this morning and the server had restarted. You want to know what happened last night:

```bash
# Errors from the previous boot (last night)
journalctl -b -1 -p err

# If you want to know why the kernel restarted
journalctl -b -1 -k -p err

# Restart history
last reboot

Scene 3: "Who SSH'd into the Server?"

# Successful logins today
journalctl -u sshd --since "today" | grep "Accepted"

# Failed logins today
journalctl -u sshd --since "today" | grep "Failed"

# Or simpler: last logins
last -10

# Failed logins
lastb -10

# Or directly from the file
grep "Failed password" /var/log/secure

Scene 4: "The Disk Is Full — Why?"

# Size of journal logs
journalctl --disk-usage

# Size of files in /var/log
du -sh /var/log/* | sort -rh | head -15

# Delete logs older than 7 days
journalctl --vacuum-time=7d

# Limit size to 500 MB
journalctl --vacuum-size=500M

# Force log file rotation
logrotate -f /etc/logrotate.conf

Warning

journalctl --vacuum-time and --vacuum-size permanently delete logs. Before running them, make sure you don't need them. In production environments, it is better to keep a backup copy on a remote server first.

Scene 5: "The Firewall Is Blocking Something"

# Firewall log
journalctl -u firewalld --since "1 hour ago"

# Dropped/rejected packets in the kernel
journalctl -k --since "1 hour ago" | grep -i "drop\|reject\|block"

# Or directly from dmesg
dmesg -T | grep -i "drop\|reject"

# fail2ban log (is an IP banned?)
fail2ban-client status
fail2ban-client status sshd

Scene 6: "A Docker Container Crashed"

# Container log (last 100 lines)
docker logs web-app --tail 100

# Live container log
docker logs web-app -f

# Log from a specific time
docker logs web-app --since "1h"

# The Docker service's own log
journalctl -u docker -p err --since "today"

# Check state and exit reason
docker inspect web-app | jq '.[0].State'

Scene 7: "The Server Is Slow — What's Eating Resources?"

# All errors in the last hour
journalctl -p err --since "1 hour ago" --no-pager

# Kernel errors (OOM Killer, I/O errors)
dmesg -T | grep -i "oom\|error\|fail\|kill"

# Check for OOM
journalctl -k | grep -i "oom\|out of memory"

# Disk I/O errors
dmesg -T | grep -i "i/o error\|read error\|write error"

Scene 8: "The Service Won't Start"

# First: see what systemctl itself says
systemctl status nginx

# Second: recent service logs
journalctl -u nginx -n 50 --no-pager

# Third: only errors
journalctl -u nginx -p err -n 20

# Fourth: Nginx's own log
tail -50 /var/log/nginx/error.log

# Fifth: maybe the port is occupied?
ss -tlnp | grep ":80\|:443"

Tip

Memorize the order of these five steps: service status → service log → filtered errors → dedicated log → system resources. This pattern almost always leads you to the root cause.

Beyond journalctl — Classic Files

journalctl is very powerful, but not all logs live there. Some services also keep their own separate log files.

Map of Important Log Files

File path	What's inside	Your thought
/var/log/messages	General system log	"Something strange happened but I don't know what"
/var/log/pvm/messages	General PVM log	"What happened in the PVM hypervisor"
/var/log/secure	Authentication log (SSH, sudo, login)	"Who logged in?"
/var/log/audit/audit.log	SELinux and audit security log	"Did SELinux block it?"
/var/log/nginx/error.log	Nginx errors	"The site is throwing errors"
/var/log/nginx/access.log	HTTP requests	"How many hit it? From where?"
/var/log/cron	Cron job execution	"Did the cron job run or not?"
/var/log/maillog	Email log (postfix)	"Did the email go out or not?"
/var/log/boot.log	System boot log	"Was there a problem at boot?"
/var/log/firewalld	Firewall log	"What did the firewall do?"
/var/log/dnf.log	Package install/update	"What was installed recently?"

File Search Tools

Whenever you think "search the file and find such-and-such word":

# Simple search
grep "word" /path/to/file

# Case-insensitive search
grep -i "error" /var/log/messages

# Search with line numbers
grep -n "Failed" /var/log/secure

# Search inside compressed files (.gz)
zgrep "error" /var/log/messages-20240615.gz

# Last 50 lines
tail -50 /var/log/messages

# Follow live
tail -f /var/log/nginx/error.log

# Last 50 lines then live
tail -50f /var/log/secure

Tip

grep is like Ctrl+F in a browser — but for files. tail -f is like putting a live camera on the file and watching every new line as it is written.

Smart Search — The Art of Filtering

Sometimes logs are thousands of lines long and you are looking for a needle in a haystack. This is where filtering tools help you.

"Only Lines That Contain a Certain Word"

# Thought: "logs that contain the word error"
journalctl -u nginx --since "today" | grep "error"

# Thought: "logs that contain error or fail"
journalctl -u nginx --since "today" | grep -iE "error|fail"

# Thought: "logs that do NOT contain info" (inverse)
journalctl -u nginx --since "today" | grep -v "info"

"How Many Times Did It Happen?"

# Thought: "how many failed logins today?"
journalctl -u sshd --since "today" | grep "Failed" | wc -l

# Thought: "from which IPs did failed logins come?"
grep "Failed password" /var/log/secure | awk '{print $(NF-3)}' | sort | uniq -c | sort -rn | head -20

# Thought: "how many people hit the site?"
awk '{print $1}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head -20

"Filter by Time"

# Thought: "only logs between 3 and 5 o'clock from the file"
awk '/15:00:/,/17:00:/' /var/log/messages

# Thought: "only today's logs, Jun 15"
grep "Jun 15" /var/log/messages

"Sort and Summarize the Output"

# Thought: "group errors so I can see which one is most frequent"
journalctl -p err --since "today" --no-pager | awk '{print $5}' | sort | uniq -c | sort -rn

# Thought: "save the result to a file"
journalctl -u nginx --since "today" -p err > /tmp/nginx-errors.log

Info

Security Logs — "Who Did What?"

Audit (System Auditing)

The Linux Audit system records every sensitive action. When you think "who changed this file?" or "who ran sudo?":

# All security events today
ausearch -ts today

# Who ran sudo?
ausearch -ts today -m USER_CMD

# Failed logins
ausearch -ts today -m USER_LOGIN --success no

# Security report summary
aureport --summary

# Login report
aureport -l

# Failed authentication report
aureport --failed

SELinux — "Did SELinux Block It?"

# Did SELinux block anything?
ausearch -m AVC -ts today

# Summary of SELinux blocks
ausearch -m AVC -ts today | aureport -a

# Or directly from the file
grep "denied" /var/log/audit/audit.log | tail -20

Fail2Ban — "Who Got Banned?"

# Overall fail2ban status
fail2ban-client status

# Which IPs got banned in SSH?
fail2ban-client status sshd

# fail2ban operational log
journalctl -u fail2ban --since "today"

# Unban an IP
fail2ban-client set sshd unbanip 1.2.3.4

Kernel and Boot Logs

dmesg — "Is the Hardware Having Issues?"

When you think the problem is the hardware or a driver:

# Kernel log with human-readable time
dmesg -T

# Only errors and warnings
dmesg -T -l err,warn

# Is it a disk problem?
dmesg -T | grep -i "error\|fail\|i/o"

# Is it a memory problem?
dmesg -T | grep -i "oom\|memory"

# Is it a network problem?
dmesg -T | grep -i "eth\|link\|network"

# Live kernel log
dmesg -Tw

Boot — "Was There a Problem During Startup?"

# Current boot log
journalctl -b

# Previous boot log
journalctl -b -1

# Boot errors
journalctl -b -p err

# List of boots
journalctl --list-boots

# How long did it take to come up
systemd-analyze

# Which service slowed down the boot
systemd-analyze blame | head -20

Powerful Combinations — Pipe Mastery

One of the most powerful concepts in Linux is the pipe (|). By combining commands, you can answer any question.

# "Give me the 10 IPs with the most failed logins"
grep "Failed password" /var/log/secure | awk '{print $(NF-3)}' | sort | uniq -c | sort -rn | head -10

# "How many times did HTTP status 500 come in today? Broken down by URL"
awk '$9 == 500 {print $7}' /var/log/nginx/access.log | sort | uniq -c | sort -rn

# "Show Nginx error logs live and filter only 502"
tail -f /var/log/nginx/error.log | grep "502"

# "Give errors from all services today and group them"
journalctl -p err --since "today" --no-pager | awk '{print $5}' | sort | uniq -c | sort -rn | head

Common Troubleshooting

"journalctl Returns Empty Output!"

# Check: is the journal running?
systemctl status systemd-journald

# Check: is the log persistent?
ls -la /var/log/journal/

# If the directory does not exist → logs are wiped after reboot
sudo mkdir -p /var/log/journal
sudo systemd-tmpfiles --create --prefix /var/log/journal
sudo systemctl restart systemd-journald

Warning

By default on some Rocky Linux versions, the journal is stored only in /run/log/journal/ (volatile memory). If /var/log/journal/ does not exist, all logs are wiped after a restart.

"The Log Times Are Wrong!"

# Check timezone
timedatectl

# Set timezone to Tehran
sudo timedatectl set-timezone Asia/Tehran

# Check clock synchronization
chronyc tracking

"There Are Too Many Logs — I Can't Read Them!"

# Solution 1: only errors
journalctl -u nginx -p err

# Solution 2: last N lines
journalctl -u nginx -n 30

# Solution 3: reverse (newest first)
journalctl -u nginx -r -n 30

# Solution 4: JSON output for processing
journalctl -u nginx -o json-pretty -n 5

# Solution 5: save to a file
journalctl -u nginx --since "today" > /tmp/nginx.log
less /tmp/nginx.log

"I Get Permission Denied!"

Some logs can only be read by root:

# Run directly with sudo
sudo journalctl -u nginx

# Or add the user to the systemd-journal group
sudo usermod -aG systemd-journal $USER
# Then log out and log back in

Security

Membership in the systemd-journal group grants the user read access to all system logs. Grant this group only to users who genuinely need it — logs may contain sensitive data such as tokens or passwords in URLs.

Mental Cheat Sheet — Quick Reference

Print this table and stick it next to your monitor.

Your thought	Command
"Give me the service X log"	journalctl -u X
"From 3 to 5 o'clock"	--since "15:00" --until "17:00"
"Since yesterday"	--since "yesterday"
"Last hour"	--since "1 hour ago"
"Only errors"	-p err
"Only warnings and above"	-p warning
"Show live"	-f
"Last N lines"	-n 50
"Kernel log"	journalctl -k or dmesg -T
"Previous boot log"	journalctl -b -1
"Who logged in?"	last · lastb · lastlog
"Docker container log"	docker logs container-name
"Security log"	ausearch -ts today
"Search for word X"	| grep "X"
"How big are the logs?"	journalctl --disk-usage
"Delete old logs"	journalctl --vacuum-time=7d