Linux RAID

Guide: Managing Software RAID 5 with mdadm in Linux

This guide covers how to create a RAID 5 array using three disk partitions (/dev/sda2, /dev/sdb2, /dev/sdc2), format and mount it, and simulate a drive failure and recovery.

Part 1: Core RAID Theory & Interview Questions

1. RAID Levels Comparison Cheat Sheet

2. Common Scenario Questions & Expected Answers

Q: “We have a highly transactional database server with extreme write requirements. Which RAID level should we choose?”

• Answer: RAID 10. RAID 5 and RAID 6 suffer from parity calculation overheads (the “write penalty”), which significantly chokes transactional database performance. RAID 10 offers striping performance combined with pure mirror safety, making it the industry standard for databases.

Q: “What is the difference between Hardware RAID and Software RAID?”

• Hardware RAID: Uses a dedicated physical PCIe controller card. It has its own onboard processor and dedicated RAM cache (often protected by a battery backup unit). The Linux kernel has no idea a RAID exists; it sees a single block device. It is fast and unburdens the server’s CPU, but it is expensive and relies on proprietary hardware.
• Software RAID (mdadm): Managed entirely by the Linux kernel driver using the server’s primary host CPU and RAM. It is free, highly flexible, and drive-agnostic. However, it consumes a small fraction of the host processor’s cycles during heavy compute tasks like parity generation. Při použití kódu buďte obezřetní.Můžeš si tento návod uložit pro budoucí přípravu. Chceš si v praxi nasimulovat ten Hot Spare scénář, nebo se přesuneme k zápisu do /etc/fstab, ať máš celý dnešní diskový lab kompletně dotažený?Odpovědi od umělé inteligence můžou obsahovat chyby. Další informace

Part 2: Creating the RAID 5 Array

Step 1: Create the RAID 5 device

Combine your three partitions into a new single RAID device named /dev/md0.

sudo mdadm --create --verbose /dev/md0 --level=5 --raid-devices=3 /dev/sda2 /dev/sdb2 /dev/sdc2

(If prompted to continue creating the array, type y and press Enter).

Step 2: Monitor the initial synchronization

Linux will immediately start syncing the drives in the background. You can check the real-time progress using:

cat /proc/mdstat

To view detailed technical information about the array’s health and active devices, use:

sudo mdadm --detail /dev/md0

Step 3: Create a File System and Mount the Array

Once created, the array acts like a standard block device. Format it with ext4 and mount it to your system.

sudo mkfs.ext4 /dev/md0
sudo mkdir -p /mnt/my_raid
sudo mount /dev/md0 /mnt/my_raid

Note: Since RAID 5 uses one drive worth of space for parity data, running df -h will show a total usable capacity of roughly 2 GB (3 disky - 1 disk).

Part 3: Testing Fault Tolerance (Simulating Drive Failure)

To fully understand how RAID 5 protects your data, you can intentionally break and repair the array without losing any files.

Step 4: Mark a drive as failed (Broken)

Simulate a sudden hardware failure on one of the drives (e.g., /dev/sdc2):

sudo mdadm /dev/md0 --fail /dev/sdc2

If you check cat /proc/mdstat or df -h now, you will notice the array status is marked as degraded, but your files in /mnt/my_raid/ remain fully accessible.

Step 5: Fyzicky/Logicky remove the failed drive

Before replacing a broken drive, you must logically remove it from the active active array structure:

sudo mdadm /dev/md0 --remove /dev/sdc2

Step 6: Insert a new drive (Repairing the array)

Once the “new” replacement drive is ready, add it back to the array. The rebuild process will trigger automatically.

sudo mdadm /dev/md0 --add /dev/sdc2

You can watch the array rebuild itself back to 100% health by running cat /proc/mdstat again.

Part 4: Advanced RAID Operations (The “Interview Flex” Commands)

1. Saving the RAID Configuration (Persistence)

By default, a newly created software RAID might not assemble correctly or under the same name after a reboot. To make it persistent.

# Generate the configuration and append it to the standard mdadm config file
sudo mdadm --detail --scan | sudo tee -a /etc/mdadm.conf

Note: On some distributions (like RHEL/AlmaLinux), you might also need to update the initramfs (sudo dracut -f) so the kernel detects the array early during the boot sequence.

2. Configuring a “Hot Spare” Drive

A Hot Spare is a standby drive connected to the array that does nothing until an active drive fails. When a failure occurs, the array automatically pulls the spare in and starts rebuilding without human intervention.

# Add a 4th partition/drive to your existing RAID 5 array as a spare
sudo mdadm /dev/md0 --add /dev/sdd2

If you run cat /proc/mdstat, the spare drive will be marked with an [S] flag.

3. Managing Rebuild Speed Limits

During a RAID rebuild, disk I/O hits 100%, which can degrade server performance for production users. You can inspect and throttle the minimum and maximum rebuild speeds through the sysctl interface:

# Check current rebuild speed limits (in KiB/s)
cat /proc/sys/dev/raid/speed_limit_min
cat /proc/sys/dev/raid/speed_limit_max

# Dynamically increase the minimum speed to force a faster rebuild
echo 50000 | sudo tee /proc/sys/dev/raid/speed_limit_min

Guide: Automating Mounts with /etc/fstab & Interview Tips

The /etc/fstab (File System Table) file configuration determines how storage devices, RAID arrays, and LVM logical volumes are automatically mounted during the system boot sequence.

Part 1: Syntax and Structure

To edit the file, always open it as root (e.g., sudo nano /etc/fstab). Every entry requires exactly 6 columns separated by spaces or tabs:

# [1. Device/UUID]              [2. Mountpoint]     [3. FS Type]  [4. Options]             [5. Dump] [6. Pass]
/dev/md0                        /mnt/my_raid        ext4          defaults                  0         2
/dev/testgroup/backup           /mnt/vgroup_backup  ext4          defaults,noatime,nofail   0         2

Column Breakdown:

1. Device Identifier: The path to the block device (/dev/md0), LVM path (/dev/testgroup/backup), or filesystem UUID (UUID=...).
2. Mountpoint: The target directory where the files will be accessible. Note: This directory must already exist.
3. Filesystem Type: The underlying format type (e.g., ext4, xfs, vfat, ntfs).
4. Mount Options: Configuration flags that dictate device behavior (comma-separated, no spaces).
5. Dump Flag: Used by the ancient dump backup utility. Always set to 0 on modern systems.
6. FSCK Pass: Dictates the order of filesystem integrity checks at boot.
   • 0 = Disable checking (used for network shares, swaps, or CD-ROMs).
   • 1 = High priority (reserved exclusively for the root / partition).
   • 2 = Low priority (used for all secondary data drives, LVMs, and RAIDs).

Part 2: The Golden Rule (Testing Before Rebooting)

Critical Interview Tip: If you make a single typo or write a non-existent device path in /etc/fstab, the Linux kernel will freeze during the next boot sequence and drop into Emergency Mode.

Never reboot a production server immediately after editing fstab. Always run the following test first:

sudo mount -a

• What it does: Reads /etc/fstab and attempts to mount all currently unmounted entries.
• How it helps you look like a pro: If this command runs silently without errors, your syntax is verified, and the server is guaranteed to boot safely.

Part 3: Advanced Mount Options (Interview Differentiators)

When interviewers ask how to optimize performance or ensure stability, mention that you replace defaults with these specific parameters:

• nofail
  • Purpose: Prevents the OS boot process from crashing if the drive is missing.
  • Use Case: Highly critical for external storage, secondary RAIDs, or network drives. If the hardware component isn’t available, Linux skips it and proceeds to start the operating system normally.
• noatime
  • Purpose: Disables writing access-time metadata (atime) every single time a file is read.
  • Use Case: Dramatically boosts storage I/O performance on heavy database nodes, web servers, and SSD drives by cutting unnecessary write wear.
• ro (Read-Only)
  • Purpose: Mounts the filesystem with strict write protections.
  • Use Case: Used for cold data archives, system backups, or secure compliance shares where data modification must be prevented.

Part 6: Useful Commands Cheat Sheet

1. Stopping the Array

sudo umount /mnt/my_raid && sudo mdadm --stop /dev/md0

• What it does: Unmounts the file system and deactivates the RAID array.
• Why use it: Safely shuts down the array before physically removing drives or changing configurations. It does not delete data.

2. Assembling the Array

sudo mdadm --assemble --scan

• What it does: Scans all drives for RAID metadata and rebuilds/re-starts existing arrays automatically.
• Why use it: Used when moving RAID drives to a completely new server or recovering an offline array after a reboot.

3. Wiping RAID Metadata

sudo mdadm --zero-superblock /dev/sda2

• What it does: Overwrites the RAID identification header (superblock) on the drive with zeroes.
• Why use it: Essential when decommissioning an array. If you don’t zero the superblock, Linux will still recognize the drive as part of a broken RAID, causing conflicts when you try to reuse it.

Production Guide: Automated RAID Monitoring with Custom Alert Scripts

This guide explains how to set up robust, production-ready email alerts for Linux Software RAID (mdadm). It specifically addresses a common real-world issue where public SMTP servers (like Gmail or Seznam) reject default system emails due to invalid headers (root@localhost), and solves it using a custom Bash wrapper script.

The Problem with Default mdadm Alerts

When you simply add MAILADDR to /etc/mdadm.conf, the mdadm daemon sends emails using raw local addresses (e.g., From: root or [email protected]). Modern SMTP relays strictly block these emails with errors such as:

• 451 4.4.1 error when enqueuing the message
• 501 5.1.7 invalid sender address format

The Solution: Using the PROGRAM Directive

Instead of letting mdadm format the mail natively, we configure it to trigger a custom Bash script via the PROGRAM keyword. This script catches the alert parameters and fires a fully compliant email via the standard mail command wrapped by msmtp.

Step 1: Configure the SMTP Client (msmtp)

Ensure your /etc/msmtprc file is properly mapped to a valid outbound email server using a secure Application Password (not your primary password).

# /etc/msmtprc Configuration Example
defaults
auth             on
tls              on
tls_starttls     on
tls_trust_file   /etc/pki/tls/certs/ca-bundle.crt

account          default
host             smtp.seznam.cz
port             587
from             [email protected]
user             [email protected]
password         your_16_character_app_password

Verify that the system wide sendmail alias is linked straight to msmtp:

sudo ln -sf /usr/bin/msmtp /usr/sbin/sendmail

Step 2: Create the Custom Alert Script

Create a lightweight handler script that intercepts system events and structures clean MIME headers.

sudo vim /usr/sbin/raid-alert.sh

Paste the following script layout (make sure to update the recipient address):

#!/bin/bash
# Arguments passed by mdadm: $1 = Event Name, $2 = Failed Device Node
EVENT=$1
DEVICE=$2

echo "RAID Alert: Event '$EVENT' was detected on storage device '$DEVICE'!" | \
mail -s "RAID Alert: $EVENT on $DEVICE" [email protected]

Save and exit (:wq), then apply executable privileges to the file:

sudo chmod +x /usr/sbin/raid-alert.sh

Step 3: Configure mdadm.conf to Use the Script

Open your RAID configuration file:

sudo vim /etc/mdadm.conf

Important: Remove any existing MAILADDR lines so they don’t trigger concurrent failures. Instead, use the PROGRAM directive pointing to your script:

# /etc/mdadm.conf
ARRAY /dev/md0 metadata=1.2 UUID=387f9130:4dd1f16d:c89baf2b:c4be4b12
PROGRAM /usr/sbin/raid-alert.sh

Step 4: Apply Changes and Run the Test Poplach

Restart the monitoring service to load the new automation policy:

sudo systemctl restart mdmonitor

Force a simulated disk catastrophe event on the active array to verify end-to-end functionality:

sudo mdadm --monitor --test --oneshot /dev/md0

Expected Outcome

The terminal will execute silently without errors. Within seconds, your external mailbox will receive three distinct notifications capturing the full state change log of the monitoring daemon init sequence:

1. RAID Alert: TestMessage on /dev/md0 (The forced manual trigger validation).
2. RAID Alert: NewArray on /dev/md0 (The monitor discovering the array on state reload).
3. RAID Alert: DeviceDisappeared on /dev/md/md0 (Kernel path resolution logging).