Your Progress

Phase 0: Overview
Phase 1: Linux
Phase 2: Bash
Phase 3: Processes
Phase 4: Network
Phase 5: Storage
Phase 6: Monitor
0% Complete

Linux SysAdmin Deep Dive: Building a Production-Ready Server from Scratch

From blank Ubuntu VM to fully monitored, hardened, Dockerized server

This project documents my journey of setting up an Ubuntu Server 24.04 ARM64 virtual machine on UTM (MacBook Air), then systematically learning and applying every fundamental Linux sysadmin skill: filesystem navigation, user management, bash scripting, process control, networking, security hardening, disk management, monitoring, and Docker. Every phase includes real scripts I wrote, real terminal output, and real lessons learned.

Before vs. After This Project

✗Afraid of the terminal

✓Comfortable navigating any Linux system

✗No idea how permissions work

✓Can audit and fix file permissions confidently

✗Copy-paste bash commands

✓Write production-ready bash scripts from scratch

✗No understanding of systemd

✓Create and manage custom systemd services

✗Server security is a mystery

✓Hardened SSH, firewall rules, fail2ban setup

✗No disk management knowledge

✓LVM, inode monitoring, swap management

✗No monitoring or alerting

✓Automated metrics collection and threshold alerts

✗Docker is just a buzzword

✓Build images, run containers, port mapping

Technologies Used

🐧

Ubuntu Server

24.04 LTS ARM64

📜

Bash

Shell scripting

🔐

SSH

Secure remote access

🛡

UFW

Firewall management

⚙️

systemd

Service management

🐳

Docker

Containerization

⏰

cron

Task scheduling

💾

LVM

Logical Volume Manager

Lab Environment

  • OS: Ubuntu Server 24.04 LTS (ARM64)
  • Hypervisor: UTM on MacBook Air (Apple Silicon)
  • Connection: SSH from macOS terminal into the VM
  • Username: charith
  • IP Address: 192.168.64.3
Connecting to the Server
$▊
💡

Key Takeaway

The entire project runs on a headless Ubuntu Server VM with no GUI. Everything is done through SSH and the command line. This is how real servers work in production — there is no desktop environment, just a terminal.

Phase 1: Linux Foundations

Before you can manage a Linux server, you need to understand its foundations: the filesystem hierarchy, user and group management, file permissions, and the package management system. This phase explores all of these through hands-on scripts and exercises.

Exploring the Linux Filesystem
▶️

Click the Play button above to start!

Watch the commands execute step by step

🏙️
Think of it this way...
Think of the Linux filesystem like a city — /etc is the city hall (all the configuration), /var/log is the security cameras (all the logs), /home is the residential area, and /tmp is a whiteboard that gets erased each night.
💡

Key Takeaway

Permissions matter more than you think. chmod changes file permissions, chown changes file ownership. SUID (Set User ID) lets a file run as its owner — that is why /usr/bin/passwd can modify /etc/shadow even when run by a normal user. SGID works the same way but for group permissions. Always audit SUID binaries on production servers.

⚠️

Warning

Never use chmod 777 on any file. This gives read, write, and execute permissions to everyone on the system. It is the equivalent of leaving your front door wide open with a sign that says "come on in." If you see 777 permissions in production, treat it as a security incident.
🤯

Did You Know?

The Linux filesystem starts at / (called "root"). Unlike Windows, there are no drive letters like C: or D:. Everything, including external drives and network shares, is mounted somewhere under this single tree. The command mount shows you where every device is attached.

Phase 2: Bash Scripting

Bash scripting is the duct tape of system administration. It lets you automate repetitive tasks, build monitoring tools, and chain commands together into powerful workflows. In this phase, I moved beyond running individual commands and started building real tools — a log analyzer, a backup system, and a live system monitor.

Bash Basics: Variables, Loops, Functions
▶️

Click the Play button above to start!

Watch the commands execute step by step

📖
Think of it this way...
A bash script is like a recipe — you write the steps once, and the chef (Linux) follows them exactly every time. Variables are your ingredients, functions are cooking techniques, and pipes are like passing a dish from one chef to the next for additional preparation.
💡

Key Takeaway

Always start your scripts with set -euo pipefail. This is your safety net. -e exits on any error, -u treats unset variables as errors, and -o pipefail catches failures in piped commands. Without this, a script can silently fail halfway through and keep running, causing real damage on a production server.

🤯

Did You Know?

The Unix pipeline concept (cmd1 | cmd2 | cmd3) is one of the most powerful ideas in computing. Each command does one small thing well, and the pipe connects them. For example, grep "error" syslog | sort | uniq -c | sort -rn | head -5 finds errors, sorts them, counts unique entries, sorts by count, and shows the top 5 — all in a single line. This "small tools connected by pipes" philosophy is the Unix Way.

Phase 3: Process & Service Management

Everything running on a Linux system is a process. Understanding how processes work, how to inspect them, how to manage them with systemd, and how to troubleshoot stuck or zombie processes is essential for any sysadmin. This phase covers the entire lifecycle: from PID 1 (systemd) to creating your own custom services.

👨‍🍳
Think of it this way...
A process is like a chef cooking — the program is the recipe book on the shelf, but a process is a chef actively cooking from that recipe. PID is the chef's employee badge number. You can have multiple chefs cooking from the same recipe (multiple processes from the same program), and each has their own badge number.
Inspecting Processes and Services
▶️

Click the Play button above to start!

Watch the commands execute step by step

💡

Key Takeaway

kill vs kill -9: know the difference. By default, kill sends SIGTERM (signal 15), which politely asks a process to shut down and lets it clean up (save files, close connections, release locks). kill -9 sends SIGKILL, which immediately terminates the process with no chance to clean up. Always try SIGTERM first. Only use SIGKILL as a last resort when a process is truly unresponsive.

🤯

Did You Know?

A zombie process is a process that has finished running but still has an entry in the process table because its parent has not yet read its exit status (via the wait() system call). You can spot them with ps aux | grep Z. A few zombies are harmless, but thousands of them can exhaust the PID table and prevent new processes from starting. The fix is usually to kill or restart the parent process.
🏷️
Think of it this way...
Think of PID 1 (systemd) as the general manager of a hotel. Every other process is a guest or staff member. When systemd starts, it reads its checklist (unit files) and launches all the services the hotel needs: the front desk (SSH), the security guards (UFW), the janitors (cron jobs). If any of them crash, systemd can restart them automatically — that is what Restart=always does in a unit file.

Phase 4: Networking & Security

A server connected to a network is a target. This phase covers hardening the server against attacks: configuring the UFW firewall, locking down SSH, running a comprehensive security audit, and understanding fail2ban. Security is not a one-time task — it is a continuous process of auditing, hardening, and monitoring.

Setting Up the UFW Firewall
▶️

Click the Play button above to start!

Watch the commands execute step by step

⚠️

Warning

Always allow port 22 (SSH) BEFORE enabling the firewall. Otherwise you lock yourself out of your own server. If you are connected remotely via SSH and you enable UFW without allowing port 22, your connection will drop immediately and you will have no way to get back in unless you have physical console access to the machine.
SSH Hardening Configuration
▶️

Click the Play button above to start!

Watch the commands execute step by step

💡

Key Takeaway

SSH hardening checklist: (1) Disable root login with PermitRootLogin no. (2) Limit auth attempts with MaxAuthTries 3. (3) Disable password auth with PasswordAuthentication no. (4) Require SSH keys with PubkeyAuthentication yes. These four settings eliminate the vast majority of SSH-based attacks.

🤯

Did You Know?

fail2ban is a tool that automatically bans IP addresses after repeated failed login attempts. It monitors log files (like /var/log/auth.log) in real time and adds temporary firewall rules to block offending IPs. The default configuration bans an IP for 10 minutes after 5 failed SSH attempts. On internet-facing servers, fail2ban blocks thousands of brute-force attempts per day — it is one of the first things you should install on any public server.
🏰
Think of it this way...
Think of server security like a medieval castle. The firewall (UFW) is the outer wall — it controls who can even approach. SSH hardening is the gatehouse — checking credentials before letting anyone in. fail2ban is the guard who remembers faces and bans troublemakers. And security audits are the castle inspector who walks around checking for cracks in the walls.

Phase 5: Storage & Disk Management

Running out of disk space at 3 AM is every sysadmin's nightmare. This phase covers LVM (Logical Volume Manager) for flexible disk management, inode monitoring, swap configuration, and building an automated disk monitoring script that catches problems before they become outages.

🏠
Think of it this way...
LVM is like having adjustable rooms in your house — instead of fixed walls (partitions), you can move walls and make rooms bigger without rebuilding the entire house. With traditional partitions, if your /home partition fills up, you are stuck even if /var has plenty of free space. With LVM, you can dynamically grow or shrink volumes without downtime.
Inspecting Disk Usage and LVM
▶️

Click the Play button above to start!

Watch the commands execute step by step

💡

Key Takeaway

Inode exhaustion is a silent killer. Every file and directory on a Linux filesystem uses one inode. You can run out of inodes even when you have plenty of disk space — this typically happens when an application creates millions of tiny files (like mail queues or session files). The command df -i shows inode usage. If IUse% hits 100%, no new files can be created even if the disk is 50% empty.

🤯

Did You Know?

When you run rm on a large file, the disk space is not immediately freed if any process still has the file open. Linux only releases the space when the last file descriptor is closed. This is why df sometimes shows a disk as full even after you have deleted files. The fix is to restart the process that had the file open, or use lsof +L1 to find deleted files still held open.
📋
Think of it this way...
Think of inodes like entries in a library catalog. Each book (file) needs a catalog entry that records its title, location, and checkout status. You can have empty shelves (free disk space) but if you run out of catalog cards (inodes), you cannot add any new books. The df -i command is like checking how many blank catalog cards remain.

Phase 6: Monitoring, Alerting & Docker

The final phase ties everything together. First, we build an automated monitoring and alerting system that collects metrics via cron and triggers alerts when thresholds are breached. Then, we explore Docker to containerize applications on our server. This phase represents the transition from managing a server to operating one.

Monitoring & Alerting

Docker Containerization

📦
Think of it this way...
A Docker container is like a shipping container — it packages everything your app needs (code, libraries, config) into a standardized box that runs the same way on any machine. Just like how a shipping container can be loaded onto any truck, ship, or train, a Docker container runs identically on your laptop, a staging server, or a production cluster.
Docker in Action
▶️

Click the Play button above to start!

Watch the commands execute step by step

💡

Key Takeaway

Port mapping: -p 8080:80 means "connect port 8080 on the host to port 80 inside the container." The format is always HOST_PORT:CONTAINER_PORT. When you visit http://server:8080, the traffic enters port 8080 on the host machine, Docker forwards it to port 80 inside the container, and Nginx (listening on port 80 inside the container) handles the request.

🤯

Did You Know?

Docker images use a layered filesystem. Each instruction in a Dockerfile (FROM, RUN, COPY) creates a new read-only layer. When you change one line and rebuild, Docker only rebuilds from that layer onward — unchanged layers are cached. This is why you should put rarely-changing instructions (like installing dependencies) before frequently-changing ones (like copying your code). Smart layer ordering can reduce build times from minutes to seconds.

What I Learned The Hard Way

1.

Always test firewall rules before enabling UFW

I locked myself out of my own VM once by enabling UFW without allowing SSH first. Had to access the VM console through UTM to fix it. On a remote cloud server, this mistake could mean a support ticket and hours of downtime.

2.

rm -rf / protection exists for a reason

Modern Linux distributions have safeguards against rm -rf /, but rm -rf /* (with the asterisk) bypasses them. I learned to always double-check destructive commands, use rm -i for interactive confirmation, and never run anything as root unless absolutely necessary.

3.

Bash scripts fail silently without set -euo pipefail

I had a backup script that was silently failing for weeks because a directory did not exist. Without set -e, the script continued past the error and reported success. The backups were empty. Now every script starts with set -euo pipefail.

4.

Log files can fill up a disk faster than you think

A misconfigured service was logging every request at DEBUG level, generating 2 GB of logs per hour. The disk filled up overnight. I learned to always configure log rotation (logrotate) and monitor /var/log sizes.

5.

Docker containers are ephemeral by default

I stored important data inside a container, then ran docker rm and lost everything. Containers are disposable — any data that needs to persist must be stored in Docker volumes or bind mounts, not inside the container filesystem.

Final Quiz: Test Your Knowledge

10 questions covering every phase of the project. Score 80% or higher to celebrate!

Linux SysAdmin Deep Dive — Final Quiz

1 / 10

What directory stores all system configuration files on Linux?

Review Flashcards

15 flashcards to help you memorize key Linux sysadmin concepts. Click a card to flip it.

Linux SysAdmin Key Concepts

Click any card to flip and reveal the definition.

What does /etc store?

System configuration files — SSH config, user accounts, network settings, firewall rules, and more.

What is the difference between chmod and chown?

chmod changes file permissions (who can read/write/execute). chown changes file ownership (which user and group own the file).

What does set -euo pipefail do?

-e: exit on error. -u: error on undefined variables. -o pipefail: catch failures in piped commands. The essential safety net for bash scripts.

What is PID 1?

systemd (or init) — the first userspace process started by the kernel. It is the parent of all other processes and manages all services.

SIGTERM vs SIGKILL

SIGTERM (15): politely asks a process to stop, allows cleanup. SIGKILL (9): forcefully terminates with no cleanup. Always try SIGTERM first.

What is a zombie process?

A process that has finished but its parent hasn't read its exit status via wait(). It still occupies a PID table entry. Fix: restart the parent.

What are the 4 SSH hardening essentials?

1. PermitRootLogin no 2. MaxAuthTries 3 3. PasswordAuthentication no 4. PubkeyAuthentication yes

What is fail2ban?

A tool that monitors log files for failed login attempts and automatically bans offending IPs via firewall rules. Essential for public servers.

What are inodes?

Metadata structures for files — they store permissions, ownership, timestamps, and disk block locations. You can run out of inodes even with free disk space.

What is LVM?

Logical Volume Manager — allows dynamic resizing of disk partitions without downtime. Think of it as adjustable walls in a house.

What does rsync --link-dest do?

Creates hard links to unchanged files during backups, making each backup appear complete while only using disk space for changed files.

What is the cron format?

minute hour day-of-month month day-of-week command Example: */5 * * * * means every 5 minutes.

What does -p 8080:80 mean in Docker?

Maps host port 8080 to container port 80. Format: HOST_PORT:CONTAINER_PORT. Traffic on 8080 gets forwarded to the container's port 80.

Docker image vs container

Image: a read-only template/blueprint. Container: a running instance of an image. You can run many containers from one image.

Why does df still show a disk as full after rm?

If a process still has the deleted file open, Linux won't free the space until the file descriptor is closed. Use lsof +L1 to find such files.

View Source Code

All scripts, configuration files, and documentation from this project are available on GitHub. Explore the complete source code, try the scripts on your own VM, and follow along with each phase.

View on GitHub