What are the core concepts covered in Advanced Bash Scripting, Streams & Cron Automation?

Bash strict mode: set -euo pipefail for bulletproof automation scripts; I/O Redirection: stdin (0), stdout (1), stderr (2), and piping ( | ); Pattern filtering and stream processing using grep, sed, and awk; Scheduling recurring background tasks using system crontab syntax

Intermediate 24 min readModule: Module 6: Bash Scripting, Streams & Cron Job Automation

Advanced Bash Scripting, Streams & Cron Automation

Write robust, production-grade bash shell scripts using strict mode (set -euo pipefail), pipeline stream processing with awk/sed/grep, and scheduled cron jobs.

What You Will Learn in This Lesson

Bash strict mode: set -euo pipefail for bulletproof automation scripts
I/O Redirection: stdin (0), stdout (1), stderr (2), and piping ( | )
Pattern filtering and stream processing using grep, sed, and awk
Scheduling recurring background tasks using system crontab syntax

Introduction & Core Concept

Bash (Bourne Again SHell) is the standard command-line shell and scripting language on Linux. Writing robust shell scripts enables engineers to automate server backups, database maintenance, container health checks, and CI/CD pipelines.

WHY DOES THIS MATTER IN THE REAL WORLD?

A poorly written shell script that fails silently or ignores errors can delete production files or leave servers in corrupt states. Employing strict mode, proper exit codes, and automated crontab scheduling ensures reliable, repeatable infrastructure automation.

Syntax & Structure

bash

#!/usr/bin/env bash
set -euo pipefail
command 2>&1 | tee output.log
crontab -e
0 2 * * * /usr/local/bin/backup.sh

Production Automated Backup Script with Strict Mode

bash

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
#!/usr/bin/env bash
# Production Database & Directory Backup Automation Script
# Strict Mode: Exit immediately on error, unset variables, or pipeline failure
set -euo pipefail
 
# Configuration Variables
BACKUP_SRC="/var/www/html"
BACKUP_DEST="/var/backups/site"
TIMESTAMP=$(date +"%Y%m%d_%H%M%S")
ARCHIVE_NAME="site_backup_${TIMESTAMP}.tar.gz"
RETENTION_DAYS=7
 
# Ensure destination directory exists
mkdir -p "${BACKUP_DEST}"
 
echo "[$(date)] Starting backup of ${BACKUP_SRC}..."
 
# Create compressed tar archive
tar -czf "${BACKUP_DEST}/${ARCHIVE_NAME}" -C "${BACKUP_SRC}" .
 
# Verify archive was created and calculate size
ARCHIVE_SIZE=$(du -h "${BACKUP_DEST}/${ARCHIVE_NAME}" | awk '{print $1}')
echo "[$(date)] Backup completed successfully: ${ARCHIVE_NAME} (${ARCHIVE_SIZE})"
 
# Prune backups older than retention policy
echo "[$(date)] Pruning backups older than ${RETENTION_DAYS} days..."
find "${BACKUP_DEST}" -name "site_backup_*.tar.gz" -type f -mtime +${RETENTION_DAYS} -delete
 
echo "[$(date)] Maintenance job finished cleanly."

Line-by-Line Technical Breakdown

1Crontab Expression Anatomy: A standard cron expression consists of 5 fields: `minute (0-59) hour (0-23) day-of-month (1-31) month (1-12) day-of-week (0-7)`.

2Common Cron Examples: `*/15 * * * *` (every 15 minutes), `0 3 * * *` (every day at 3:00 AM), `0 0 * * 0` (every Sunday at midnight).

Try It Yourself (Interactive Editor)

Modify the code in real-time and click Run to test live browser output and console logs.

Intelligent Code Runner & Live Sandbox[BASH]

BASH SOURCE EDITOR

Interactive Live Code

#!/usr/bin/env bash
# Production Database & Directory Backup Automation Script
# Strict Mode: Exit immediately on error, unset variables, or pipeline failure
set -euo pipefail

# Configuration Variables
BACKUP_SRC="/var/www/html"
BACKUP_DEST="/var/backups/site"
TIMESTAMP=$(date +"%Y%m%d_%H%M%S")
ARCHIVE_NAME="site_backup_${TIMESTAMP}.tar.gz"
RETENTION_DAYS=7

# Ensure destination directory exists
mkdir -p "${BACKUP_DEST}"

echo "[$(date)] Starting backup of ${BACKUP_SRC}..."

# Create compressed tar archive
tar -czf "${BACKUP_DEST}/${ARCHIVE_NAME}" -C "${BACKUP_SRC}" .

# Verify archive was created and calculate size
ARCHIVE_SIZE=$(du -h "${BACKUP_DEST}/${ARCHIVE_NAME}" | awk '{print $1}')
echo "[$(date)] Backup completed successfully: ${ARCHIVE_NAME} (${ARCHIVE_SIZE})"

# Prune backups older than retention policy
echo "[$(date)] Pruning backups older than ${RETENTION_DAYS} days..."
find "${BACKUP_DEST}" -name "site_backup_*.tar.gz" -type f -mtime +${RETENTION_DAYS} -delete

echo "[$(date)] Maintenance job finished cleanly."

>=== KWAS Production Deployment Pipeline ===

>Running set -euo pipefail checks: OK

>Containers healthy, 0 errors logged.

>Deployment completed successfully.

Common Mistakes & How to Avoid Them

#1: Relying on interactive user $PATH variables inside cron jobs.

Cron executes with a minimal default PATH (/usr/bin:/bin). Always specify full absolute paths to executables and redirect output to a log file.

Incorrect / Antipattern

0 2 * * * backup.sh

Correct / Professional Solution

0 2 * * * /usr/local/bin/backup.sh >> /var/log/backup.log 2>&1

Industry Best Practices & Professional Standards

Always include set -euo pipefail at the top of all production bash scripts.
Quote all variable references (`"${VAR}"`) to prevent word splitting and globbing bugs.
Redirect cron job stdout and stderr to a log file (`>> /var/log/job.log 2>&1`) for auditability.

Lesson Summary & Core Takeaways

Strict mode (`set -euo pipefail`) prevents silent script failures.
Stream redirection (`2>&1`) combines stderr and stdout for unified logging.
Cron coordinates automated recurring tasks with minimal system overhead.