Pipes, Redirections & Streams

Learn how Linux commands communicate, how input and output streams work, and why pipes are one of the most powerful concepts in Unix systems.

Difficulty: Beginner
Estimated reading time: 50 min

Introduction

One of the most powerful Linux ideas is this:

Small tools working together.

Unlike many operating systems that rely heavily on large graphical applications, Linux was designed around:

• simple commands
• reusable tools
• text streams
• composability

This design philosophy is one reason Linux became so dominant in:

• servers
• DevOps
• cloud infrastructure
• automation
• scripting

At the center of this philosophy are:

• streams
• redirections
• pipes

Once you understand these concepts, the Linux terminal becomes dramatically more powerful.

Everything Is a Stream

Most Linux programs communicate using streams.

A stream is simply:

A flow of data.

Imagine water flowing through pipes.

Linux commands work similarly.

Data flows:

• into programs
• through programs
• out of programs

The Three Standard Streams

Every Linux process automatically receives three default streams.

Usually written as:

stdin
stdout
stderr

These streams are one of the foundations of Unix design.

Standard Output (stdout)

What Is stdout?

Standard output is where normal program output goes.

Example:

echo "hello"

Output:

hello

This text was written to:

stdout

By default, stdout appears in the terminal.

Redirecting stdout

You can redirect output into a file using:

>

Example:

echo "hello" > file.txt

Instead of displaying in the terminal:

hello

gets written into:

file.txt

Overwriting Files

Important:

>

overwrites existing files.

Example:

echo "new content" > file.txt

Old content disappears.

This is a very common beginner mistake.

Appending Instead of Overwriting

Use:

>>

Example:

echo "another line" >> file.txt

This appends content instead of replacing it.

Real Example

date >> logs.txt

Every execution adds another timestamp.

Useful for:

• logs
• automation
• scripting

Standard Error (stderr)

What Is stderr?

Errors are separated from normal output.

Example:

ls missing-file

Output:

ls: cannot access 'missing-file'

This message comes from:

stderr

not stdout.

Why Separate Errors Exist

Separating stdout and stderr is extremely important.

Imagine automation:

program > output.txt

You may want:

• normal data saved
• errors still visible

Linux allows this separation cleanly.

Redirecting stderr

Redirect stderr:

2> errors.txt

Example:

ls missing-file 2> errors.txt

Now the error message goes into:

errors.txt

instead of the terminal.

Redirecting Both stdout and stderr

Example:

command > output.txt 2>&1

Meaning:

Now both outputs go into the same file.

Discarding Output

Linux includes a special file:

/dev/null

Anything written there disappears.

Example:

command > /dev/null

Suppress all normal output.

Suppress errors too:

command > /dev/null 2>&1

Very common in scripts and cron jobs.

Standard Input (stdin)

What Is stdin?

stdin provides input into a program.

By default:

Keyboard → stdin

Example:

cat

Type:

hello

The terminal echoes the input.

Stop with:

Ctrl + C

Redirecting stdin

Use:

<

Example:

cat < file.txt

This feeds file contents into the command.

Instead of keyboard input:

file.txt → stdin

Understanding Pipes

What Is a Pipe?

Pipes are one of the greatest Unix inventions.

Pipe symbol:

|

A pipe sends:

stdout of one command
        ↓
stdin of another command

Simple Pipe Example

ls | less

Flow:

ls output
    ↓
less input

This allows scrolling large outputs.

Why Pipes Are Powerful

Pipes allow commands to become building blocks.

Example workflow:

Command A
    ↓
Command B
    ↓
Command C

This modular design is deeply connected to Unix philosophy.

Real Example

cat logs.txt | grep ERROR

Flow:

logs.txt
    ↓
cat
    ↓
grep filters lines

Output:

Only ERROR lines

grep

One of the most important Linux commands.

Search text:

grep hello file.txt

Search pipeline output:

ps aux | grep nginx

This searches running processes.

Very common in Linux administration.

Combining Multiple Pipes

Example:

cat access.log | grep 500 | less

Flow:

log file
    ↓
filter errors
    ↓
view interactively

This demonstrates Linux composability.

Another Real Example

Count running Docker processes:

ps aux | grep docker | wc -l

Breakdown:

This style of chaining commands is extremely common.

Useful Commands for Pipes

sort

Sort lines alphabetically.

Example:

cat names.txt | sort

uniq

Remove duplicate lines.

Example:

cat names.txt | sort | uniq

wc

Count data.

Example:

wc -l file.txt

Count lines from pipe:

cat logs.txt | wc -l

head

Display first lines:

cat logs.txt | head

tail

Display last lines:

cat logs.txt | tail

Live monitoring:

tail -f app.log

Extremely important in DevOps workflows.

tee Command

What Does tee Do?

tee writes output to:

• terminal
• file

simultaneously.

Example:

echo "hello" | tee output.txt

Output appears:

• on screen
• inside file

Very useful during debugging.

Real-World DevOps Examples

Monitoring Logs

tail -f /var/log/nginx/access.log

Filtering Errors

tail -f app.log | grep ERROR

Counting Failed Requests

cat access.log | grep 500 | wc -l

Finding Running Containers

docker ps | grep nginx

Saving API Response

curl https://api.example.com > response.json

Suppressing Noise

apt update > /dev/null 2>&1

Useful in automation scripts.

Pipes and Unix Philosophy

The Unix Philosophy

Unix systems were designed around this principle:

Do one thing well.

Instead of giant monolithic programs:

Linux combines small reusable tools.

Examples:

Pipes connect these tools together.

This philosophy heavily influenced:

• Linux
• DevOps
• Docker
• cloud-native systems

Why Streams Matter So Much

Streams Are Everywhere

Streams are not limited to terminals.

Examples:

Understanding streams helps explain modern infrastructure.

Docker Example

When running:

docker logs container

you are viewing:

stdout + stderr

from containerized processes.

Containers heavily rely on Linux stream behavior.

Common Beginner Mistakes

Accidentally Overwriting Files

Dangerous:

>

Safer append:

>>

Forgetting stderr

Example:

command > output.txt

Errors still appear on screen.

Because stderr was not redirected.

Incorrect Pipe Assumptions

Pipes transfer:

stdout only

not stderr by default.

This often confuses beginners.

The Bigger Picture

Once you understand streams and pipes, Linux starts feeling dramatically more powerful.

You begin understanding:

• how commands cooperate
• how automation works
• how logs flow
• how Docker captures output
• how CI/CD pipelines process data

Pipes are one of the reasons Unix systems remained influential for decades.

They make Linux feel less like a collection of commands and more like:

A programmable data-processing system.

What Comes Next

In the next chapter, we will explore:

• Bash scripting
• variables
• conditions
• loops
• functions
• automation
• writing reusable shell scripts

This is where Linux transforms from a command-line operating system into an automation platform.