Logs & Debugging

Learn how Linux systems record events, how to inspect logs, and how to troubleshoot real problems using professional debugging techniques.

Difficulty: Beginner
Estimated reading time: 75 min

Introduction

At some point, every Linux system eventually breaks.

Maybe:

• a website stops responding
• Docker containers fail
• a service crashes
• SSH stops working
• disk space fills up
• the server becomes slow
• networking suddenly fails

When this happens, beginners often panic.

But experienced Linux administrators usually do one thing first:

Check the logs.

Logs are one of the most important parts of Linux.

They explain:

• what happened
• when it happened
• which process caused it
• why something failed

Understanding logs and debugging is critical for:

• Linux administration
• DevOps
• Docker
• Kubernetes
• cloud infrastructure
• production servers

In real infrastructure work:

Reading logs is a daily activity.

What Are Logs?

Understanding Logs

A log is simply:

A recorded event.

Applications and services constantly generate logs.

Examples:

Logs create a historical record of system activity.

Why Logs Matter

Without logs:

• debugging would be nearly impossible
• server failures would become guesswork
• security incidents would be difficult to investigate

Logs help answer questions like:

• Why did the service crash?
• Why can users not log in?
• Why is the server slow?
• Which process consumed memory?
• Why did deployment fail?

Linux Logging Philosophy

Linux Is Extremely Transparent

Linux systems expose enormous amounts of information.

Almost everything can be logged:

• authentication
• networking
• kernel events
• services
• applications
• hardware
• containers

This transparency is one reason Linux dominates servers.

Traditional Linux Logs

The /var/log Directory

Most Linux logs are stored in:

/var/log

This is one of the most important directories in Linux.

Explore:

cd /var/log
ls

Common log files:

Reading Log Files

Simple example:

cat /var/log/syslog

Usually output is enormous.

More practical:

less /var/log/syslog

Navigation inside less:

Understanding Log Structure

Example Log Entry

May 13 14:32:11 server nginx[1234]: Started worker process

Breakdown:

Understanding log structure becomes easier with practice.

journalctl and systemd Logs

Modern Linux Logging

Modern Linux systems commonly use:

systemd-journald

Logs are managed through:

journalctl

This is one of the most important Linux debugging tools.

Viewing Entire Journal

journalctl

This displays system logs chronologically.

Usually huge.

Recent Logs

journalctl -n 50

Shows last 50 lines.

Very common during troubleshooting.

Live Log Monitoring

Real-time logs:

journalctl -f

Similar to:

tail -f

This continuously streams new log entries.

Service-Specific Logs

Logs for a Specific Service

Example:

journalctl -u nginx

This filters logs only for Nginx.

Very important during debugging.

Real-Time Service Logs

journalctl -u docker -f

Useful for:

• Docker debugging
• application monitoring
• deployment troubleshooting

Boot Logs

Show logs from current boot:

journalctl -b

Previous boot:

journalctl -b -1

Very useful after crashes or failed boots.

Understanding dmesg

What Is dmesg?

dmesg displays kernel messages.

Example:

dmesg

Kernel logs include:

• hardware detection
• drivers
• disks
• USB devices
• memory issues

Common dmesg Usage

View recent kernel messages:

dmesg | tail

Useful after:

• plugging hardware
• disk failures
• kernel errors

The tail Command

Reading End of Files

Most useful log command:

tail

Example:

tail /var/log/syslog

Shows latest lines only.

Live Monitoring

One of the most important Linux debugging commands:

tail -f /var/log/syslog

This streams logs live.

Very common in production systems.

Searching Logs

grep

Search logs:

grep ERROR app.log

Case insensitive:

grep -i error app.log

Combining Commands

Example:

journalctl -u nginx | grep failed

This is classic Linux workflow:

small tools combined together

Understanding Log Levels

Common Log Levels

Applications categorize messages.

Understanding severity helps prioritize problems.

Real-World Debugging Workflow

Example: Website Is Down

Imagine users cannot access your website.

Step 1 — Check Service Status

systemctl status nginx

Maybe service failed.

Step 2 — Inspect Logs

journalctl -u nginx

Potential output:

configuration syntax error

Step 3 — Validate Configuration

nginx -t

Find exact config problem.

Step 4 — Restart Service

sudo systemctl restart nginx

Step 5 — Verify Port Listening

ss -tuln

Check whether port 80 exists.

This is real Linux troubleshooting.

Debugging Processes

Check Running Processes

ps aux

Search process:

ps aux | grep nginx

Monitor Resources

top

or:

htop

Look for:

• CPU spikes
• memory exhaustion
• frozen processes

Debugging Networking

Connectivity Tests

ping google.com

DNS Problems

nslookup example.com

Test HTTP Response

curl localhost:3000

Extremely useful for backend debugging.

Check Listening Ports

ss -tuln

This often instantly reveals problems.

Debugging Disk Space Problems

Check Filesystem Usage

df -h

Find Large Directories

du -sh /*

Disk exhaustion is one of the most common Linux problems.

Understanding Exit Codes

Every Command Returns Status

Check status:

echo $?

Very useful in scripts and debugging automation.

Reading Logs Like a Professional

Important Mindset

Beginners often:

• skim logs randomly
• panic at huge output

Experienced admins instead:

• isolate the problem
• narrow scope
• search systematically

Good debugging is often:

Controlled investigation.

Common Beginner Mistakes

Ignoring Logs

Many beginners restart services repeatedly without reading logs.

This wastes time.

Logs usually contain the answer.

Reading Entire Huge Logs

Bad approach:

cat giant.log

Better:

tail
grep
less
journalctl -n

Ignoring Timestamps

Always verify:

• when issue started
• whether logs match timeline

Forgetting Permissions

Some logs require root access:

sudo journalctl

Docker and Logs

Container Logs

Docker captures:

stdout + stderr

View logs:

docker logs container-name

Live logs:

docker logs -f container-name

Understanding Linux streams makes this behavior much easier to understand.

Why Logging Matters for DevOps

Modern infrastructure depends heavily on logs.

Examples:

Large infrastructures generate enormous amounts of logs continuously.

Observability and Modern Infrastructure

Logging is part of a larger concept called:

Observability.

Modern systems rely on:

• logs
• metrics
• tracing
• monitoring

to understand infrastructure behavior.

Logs are often the first layer of visibility.

Linux Philosophy and Debugging

Linux embraces transparency.

Instead of hiding system internals:

Linux exposes:

• logs
• processes
• services
• networking
• kernel messages

This makes Linux extremely powerful for troubleshooting.

The Bigger Picture

Once you understand logs and debugging, Linux starts feeling much less mysterious.

You begin understanding:

• how systems fail
• how services communicate
• how infrastructure behaves internally
• how production troubleshooting works

Debugging is one of the most important real-world Linux skills.

In professional environments:

The ability to investigate problems calmly is incredibly valuable.

What Comes Next

In the next chapter, we will explore:

• disks and partitions
• filesystems
• mounting drives
• storage management
• disk usage
• formatting devices
• Linux disk management

This is where Linux starts interacting directly with physical storage.