Linux System Administration

Deep dive into Linux system boot management, service management, log management, system monitoring, and performance optimization

Back to Tutorial List

1. System Administration Overview

System administration is the core responsibility of Linux operation and maintenance. It involves multiple aspects such as system boot, service management, log monitoring, and performance optimization. Effective system administration ensures the stable operation, security, and performance optimization of Linux systems.

1.1 Main Responsibilities of System Administration

  • System Boot managementment: configuration system boot process, service auto-start, and boot levels.
  • Service managementment: Start, stop, restart, and monitor system services.
  • Log managementment: Collect, analyze, and archive system logs.
  • System Monitoring: Monitor system resource using, performance indicators, and abnormal conditions.
  • Backup and Recovery: Develop backup strategies, execute backup operations, and recover systems.
  • Software Package managementment: Install, update, uninstall, and manage software packages.
  • System Updates and Upgrades: Update system patches and upgrade system versions.
  • Performance Optimization: optimization system configuration and improve system performance.
  • Security managementment: configuration firewalls, user permissions, and security policies.

1.2 System Administration Tools

  • systemd: Modern Linux system initialization and service manager.
  • systemctl: systemd command-line tool for managing services.
  • journalctl: View and manage systemd logs.
  • top/htop: Real-time system monitoring tools.
  • vmstat: Virtual memory statistics tool.
  • iostat: Input/output statistics tool.
  • sar: System activity reporting tool.
  • rsyslog: System log management service.
  • logrotate: Log rotation tool.
  • cron: Scheduled task scheduler.

2. System Boot managementment

System boot management is an important part of Linux system administration, involving the system boot process, initialization system, and boot level configuration.

2.1 Linux System Boot Process

  1. BIOS/UEFI Initialization: Hardware self-check, load boot device.
  2. Boot Loader: GRUB2 loads kernel and initial RAM disk.
  3. Kernel Initialization: Load kernel modules, mount root filesystem.
  4. Initialization System: systemd starts system services.
  5. User Login: Display login interface, user logs into system.

2.2 systemd Overview

systemd is the initialization system and service manager for modern Linux systems. It replaces the traditional SysV init system, providing faster boot speed and more powerful features.

2.2.1 Main Features of systemd

  • Parallel Startup: Start services in parallel to improve boot speed.
  • Dependency managementment: Start services based on dependencies.
  • Service Monitoring: Monitor service status, automatically restart failed services.
  • Log managementment: Integrated log management functionality.
  • Idempotency: Support multiple executions of the same command without side effects.
  • Resource Control: Control service resource using through cgroups.

2.3 Managing System Boot

# View system boot time
systemd-analyze

# View service startup times
systemd-analyzeblame

# View critical chain of startup process
systemd-analyze critical-chain

# View system runlevel
systemctl get-default

# Set system runlevel
systemctl set-default multi-user.target  # Multi-user mode (no GUI)
systemctl set-default graphical.target  # Graphical interface mode

# View startup items list
systemctl list-unit-files --type=service | grep enabled

# View system status
systemctl status

# Reboot system
systemctl reboot

# Shutdown system
systemctl poweroff

# Suspend system
systemctl suspend

# Hibernate system
systemctl hibernate

3. System Service managementment

Service management is one of the core tasks of Linux system administration, involving starting, stopping, restarting, and monitoring system services.

3.1 Viewing Service Status

# View all service status
systemctl status

# View specific service status
systemctl status sshd

# View all enabled services
systemctl list-unit-files --type=service | grep enabled

# View all run services
systemctl list-units --type=service | grep run

# View service dependencies
systemctl list-dependencies sshd

# View service reverse dependencies
systemctl list-dependencies --reverse sshd

3.2 Managing Services

# Start service
systemctl start sshd

# Stop service
systemctl stop sshd

# Restart service
systemctl restart sshd

# Reload service configuration
systemctl reload sshd

# Enable service (auto-start on boot)
systemctl enable sshd

# Disable service (prevent auto-start on boot)
systemctl disable sshd

# Check if service is enabled
systemctl is-enabled sshd

# Check if service is active
systemctl is-active sshd

# Mask service (prevent manual or automatic start)
systemctl mask sshd

# Unmask service
systemctl unmask sshd

3.3 Service Configuration Files

systemd service configuration files are typically located in the following directories:

  • /lib/systemd/system/: System default service configuration files.
  • /etc/systemd/system/: User custom service configuration files, with higher priority.

3.3.1 Service Configuration File Example

# /etc/systemd/system/my-service.service
[Unit]
Description=My Custom Service
After=network.target

[Service]
Type=simple
ExecStart=/usr/local/bin/my-service
Restart=always
User=nobody
Group=nobody

[Install]
WantedBy=multi-user.target

3.3.2 Reloading Service Configuration

# After modifying service configuration file, reload configuration
systemctl daemon-reload

# Restart service to apply new configuration
systemctl restart my-service

4. System Log managementment

System logs are important records of Linux system operation status, containing key information such as system startup, service operation, and error messages. Effective log management is crucial for system monitoring, troubleshooting, and security auditing.

4.1 Log System Overview

  • systemd-journald: systemd's logging service, stored in memory and persistent store.
  • rsyslog: Traditional system logging service, forwards logs to files or remote servers.
  • logrotate: Log rotation tool for managing log file size and retention time.

4.2 Using journalctl to View Logs

# View all logs
journalctl

# View latest logs
journalctl -n

# View logs in real-time
journalctl -f

# View logs for specific service
journalctl -u sshd

# View logs within specific time range
journalctl --since "2023-01-01" --until "2023-01-02"

# View logs with error level and above
journalctl -p err..emerg

# View boot logs
journalctl -b

# View logs for specific process
journalctl _PID=1234

# View logs for specific user
journalctl _UID=1000

# Output logs in JSON format
journalctl -o json

# View log disk using
journalctl --disk-using

# Clean logs (keep 2 days)
journalctl --vacuum-time=2d

# Clean logs (keep 100MB)
journalctl --vacuum-size=100M

4.3 Using rsyslog to management Logs

# View rsyslog configuration file
cat /etc/rsyslog.conf

# View rsyslog status
systemctl status rsyslog

# Start rsyslog
systemctl start rsyslog

# Common log file locations
/var/log/messages  # System message logs
/var/log/auth.log  # Authentication logs (Debian/Ubuntu)
/var/log/secure    # Authentication logs (CentOS/RHEL)
/var/log/daemon.log  # Daemon logs
/var/log/cron.log  # Cron job logs
/var/log/kern.log  # Kernel logs
/var/log/boot.log  # Boot logs
/var/log/mail.log  # Mail logs

4.4 Configuring logrotate

# View logrotate configuration file
cat /etc/logrotate.conf

# View application-specific logrotate configurations
ls /etc/logrotate.d/

# logrotate configuration example
/var/log/myapp.log {
    daily
    rotate 7
    compress
    delaycompress
    missingok
    notifempty
    create 644 root root
    postrotate
        systemctl reload myapp
    endscript
}

5. systemmonitor

systemmonitor is Linuxsystemmanagement important 组成部分, 它可以helpingmanagement员及时发现systemexception, performance瓶颈 and securityissues.

5.1 常用monitortool

5.1.1 top and htop

# usingtop查看systemstatus
top

# usinghtop查看systemstatus (更友 good  界面) 
htop

# 按CPUusing率sort
top -o %CPU

# 按memoryusing率sort
top -o %MEM

# 查看specificprocess
top -p 1234

5.1.2 vmstat

# 查看虚拟memorystatus
vmstat

# 每2秒刷 new 一次, 共刷 new 5次
vmstat 2 5

# 查看详细information
vmstat -a

# 查看memoryusingcircumstances
vmstat -s

5.1.3 iostat

# 查看I/Ostatisticsinformation
iostat

# 每2秒刷 new 一次, 共刷 new 5次
iostat 2 5

# 查看详细 diskI/Oinformation
iostat -x

# 查看specificdisk I/Oinformation
iostat -x /dev/sda

5.1.4 sar

# installationsar (sysstatpackage) 
sudo apt install sysstat  # Debian/Ubuntu
sudo yum install sysstat  # CentOS/RHEL

# 查看CPUusingcircumstances
sar -u

# 每2秒刷 new 一次, 共刷 new 5次
sar -u 2 5

# 查看memoryusingcircumstances
sar -r

# 查看diskI/Ocircumstances
sar -b

# 查看networkstatisticsinformation
sar -n DEV

# 查看historystatisticsdata
sar -f /var/log/sysstat/saXX

5.1.5 netstat and ss

# 查看network连接
netstat -tuln
ss -tuln

# 查看所 has network连接
netstat -ant
ss -ant

# 查看specific端口 连接
netstat -tuln | grep 80
ss -tuln | grep 80

# 查看network连接statusstatistics
netstat -s
ss -s

5.2 systemresourcemonitor

# 查看CPUinformation
lscpu
cat /proc/cpuinfo

# 查看memoryinformation
free -h
cat /proc/meminfo

# 查看disk空间
df -h

# 查看diskpartition
fdisk -l

# 查看diskI/Operformance
iotop

# 查看networkinterfaceinformation
ifconfig
ip addr show

# 查看networktraffic
tcpdump
tshark

# 查看loadcircumstances
uptime
w

6. System Backup and Recovery

System backup is an important part of Linux system administration, allowing quick system recovery in case of failures, reducing data loss and downtime.

6.1 Backup Strategies

  • Full Backup: Backup the entire system or data.
  • Incremental Backup: Only backup data changed since last backup.
  • Differential Backup: Only backup data changed since last full backup.
  • Cold Backup: Backup while system is shutdown.
  • Hot Backup: Backup while system is run.

6.2 Common Backup Tools

6.2.1 rsync

# Use rsync to backup directory
rsync -avz /source/directory /backup/directory

# Remote backup
rsync -avz /source/directory user@remote:/backup/directory

# Incremental backup
rsync -avz --link-dest=/backup/previous /source/directory /backup/current

# Exclude certain files during backup
rsync -avz --exclude='*.tmp' --exclude='log/' /source/directory /backup/directory

# Check rsync version
rsync --version

6.2.2 tar

# Create compressed backup file
tar -czvf backup.tar.gz /source/directory

# Extract backup file
tar -xzvf backup.tar.gz -C /destination/directory

# Create incremental backup
tar -czvf backup1.tar.gz /source/directory
tar -czvf backup2.tar.gz --listed-incremental=backup.snar /source/directory

# View backup file contents
tar -tzvf backup.tar.gz

6.2.3 dd

# Backup entire disk
dd if=/dev/sda of=/dev/sdb bs=4M status=progress

# Create disk image
dd if=/dev/sda of=/backup/sda.img bs=4M status=progress

# Restore disk from image
dd if=/backup/sda.img of=/dev/sda bs=4M status=progress

# Backup MBR
dd if=/dev/sda of=/backup/mbr.img bs=512 count=1

# Restore MBR
dd if=/backup/mbr.img of=/dev/sda bs=512 count=1

6.3 Automated Backup

# Create backup script
cat > /root/backup.sh << 'EOF'
#!/bin/bash

# Set backup directory
BACKUP_DIR="/backup/$(date +%Y-%m-%d)"

# Create backup directory
mkdir -p $BACKUP_DIR

# Backup system configuration
rsync -avz /etc/ $BACKUP_DIR/etc/

# Backup user data
rsync -avz /home/ $BACKUP_DIR/home/

# Backup database
mysqldump -u root -ppassword --all-databases > $BACKUP_DIR/databases.sql

# Compress backup
 tar -czvf $BACKUP_DIR.tar.gz $BACKUP_DIR

# Delete temporary directory
rm -rf $BACKUP_DIR

# Keep only last 7 days of backups
find /backup -name "*.tar.gz" -mtime +7 -delete
EOF

# Set script execution permission
chmod +x /root/backup.sh

# Add to crontab (execute at 2 AM daily)
echo "0 2 * * * /root/backup.sh" >> /etc/crontab

# Restart crontab service
systemctl restart cron

7. Software Package managementment

Software package management is an important part of Linux system administration, involving software installation, updating, uninstallation, and management.

7.1 Package managementment System Overview

  • Debian/Ubuntu: Uses dpkg and apt package management system.
  • CentOS/RHEL: Uses rpm and yum/dnf package management system.
  • Arch Linux: Uses pacman package management system.
  • Universal: Uses snap, flatpak and other universal package management systems.

7.2 Debian/Ubuntu Package managementment

# Update package list
apt update

# Upgrade all packages
apt upgrade

# Install package
apt install package-name

# Uninstall package (keep configuration)
apt remove package-name

# completely uninstall package (remove configuration)
apt purge package-name

# Search package
apt search package-name

# View package information
apt show package-name

# View installed packages
apt list --installed

# Clean cache
apt clean
apt autoclean

# Automatically remove unnecessary packages
apt autoremove

7.3 CentOS/RHEL Package managementment

# Update package list
yum check-update

# Upgrade all packages
yum update

# Install package
yum install package-name

# Uninstall package
yum remove package-name

# Search package
yum search package-name

# View package information
yum info package-name

# View installed packages
yum list installed

# Clean cache
yum clean all

# Automatically remove unnecessary packages
yum autoremove

# Use dnf (new generation package manager)
dnf check-update
dnf update
dnf install package-name
dnf remove package-name

7.4 Source Code Installation

# Download source code package
wget https://example.com/package-1.0.tar.gz

# Extract source code package
tar -xzvf package-1.0.tar.gz

# Enter source code directory
cd package-1.0

# configuration
./configure --prefix=/usr/local/package

# Compile
make

# Install
make install

# Uninstall
make uninstall

8. System Updates and Upgrades

System updates and upgrades are important tasks in Linux system administration, which can fix system vulnerabilities, update system components, and improve system performance.

8.1 System Updates

8.1.1 Debian/Ubuntu System Updates

# Update package list
apt update

# Upgrade all packages
apt upgrade

# Upgrade system (including kernel)
apt full-upgrade

# Automatically remove unnecessary packages
apt autoremove

# Clean cache
apt clean

8.1.2 CentOS/RHEL System Updates

# Update all packages
yum update

# Or use dnf
dnf update

# Update only security patches
yum update --security

# Automatically remove unnecessary packages
yum autoremove

# Clean cache
yum clean all

8.2 System Upgrades

8.2.1 Debian System Upgrade

# Edit source configuration file
nano /etc/apt/sources.list

# Change sources to new version (e.g., from buster to bullseye)
# Replace all buster with bullseye

# Update package list
apt update

# Upgrade system
apt upgrade
apt full-upgrade

# Clean unnecessary packages
apt autoremove
apt clean

8.2.2 Ubuntu System Upgrade

# Check system updates
apt update && apt upgrade

# Use do-release-upgrade to upgrade system
do-release-upgrade

# Upgrade to development version
do-release-upgrade -d

8.2.3 CentOS System Upgrade

# Backup system

# Install upgrade tool
yum install centos-release-upgrade

# Run upgrade tool
centos-upgrade-tool-cli --network 8

# Or upgrade from CentOS 7 to CentOS 8
# Use CentOS 8 official upgrade tool

9. System Performance Optimization

System performance optimization is an important task in Linux system administration, which can improve system response speed, reduce resource consumption, and enhance user experience.

9.1 System Boot Optimization

# View boot time consumption
systemd-analyzeblame

# Disable unnecessary services
systemctl disable service-name

# optimization GRUB boot parameters
nano /etc/default/grub
# Modify GRUB_CMDLINE_LINUX line, add the following parameters
# GRUB_CMDLINE_LINUX="quiet splash elevator=deadline"

# Update GRUB
grub-mkconfig -o /boot/grub/grub.cfg

9.2 Memory Optimization

# View memory using
free -h

# Clean page cache
sync; echo 1 > /proc/sys/vm/drop_caches

# Clean page cache, directory entries, and inodes
sync; echo 3 > /proc/sys/vm/drop_caches

# Adjust swap using policy
sysctl vm.swappiness=10
# Make permanent
nano /etc/sysctl.conf
# Add: vm.swappiness=10

sysctl -p

9.3 Disk I/O Optimization

# View disk I/O scheduler
cat /sys/block/sda/queue/scheduler

# Temporarily set scheduler
echo deadline > /sys/block/sda/queue/scheduler

# Permanently set scheduler
nano /etc/default/grub
# Add: GRUB_CMDLINE_LINUX="elevator=deadline"
grub-mkconfig -o /boot/grub/grub.cfg

# optimization filesystem mount options
nano /etc/fstab
# Add noatime,nodiratime options
# /dev/sda1 / ext4 defaults,noatime,nodiratime 0 1

# Remount filesystem
mount -o remount /

9.4 Network Optimization

# Adjust TCP buffer size
sysctl -w net.core.rmem_max=16777216
sysctl -w net.core.wmem_max=16777216
sysctl -w net.ipv4.tcp_rmem='4096 87380 16777216'
sysctl -w net.ipv4.tcp_wmem='4096 65536 16777216'

# Enable TCP congestion control algorithms
sysctl -w net.ipv4.tcp_congestion_control=cubic

# Enable SYN cookies (prevent SYN flood attacks)
sysctl -w net.ipv4.tcp_syncookies=1

# Make permanent
nano /etc/sysctl.conf
# Add the above parameters
sysctl -p

10. System Administration Practice

10.1 Case Objective

configuration system administration tasks for Linux servers, including service management, log management, system monitoring, and backup strategies.

10.2 Implementation Steps

10.2.1 Service managementment Configuration

# View currently run services
systemctl list-units --type=service | grep run

# Disable unnecessary services
systemctl disable bluetooth
ssystemctl disable cups
ssystemctl disable avahi-daemon

# Enable necessary services
systemctl enable sshd
systemctl enable fail2ban
systemctl enable firewall

# Restart services to apply configuration
systemctl restart sshd

10.2.2 Log managementment Configuration

# configuration rsyslog
nano /etc/rsyslog.conf
# Add the following content
# *.info;mail.none;authpriv.none;cron.none /var/log/messages
# authpriv.* /var/log/secure
# mail.* -/var/log/maillog
# cron.* /var/log/cron
# *.emerg :omusrmsg:*
# uucp,news.crit /var/log/spooler
# local7.* /var/log/boot.log

# Restart rsyslog service
systemctl restart rsyslog

# configuration logrotate
nano /etc/logrotate.d/custom
# Add the following content
/var/log/myapp.log {
    daily
    rotate 7
    compress
    delaycompress
    missingok
    notifempty
    create 644 root root
}

# Manually execute logrotate
logrotate -f /etc/logrotate.conf

10.2.3 System Monitoring Configuration

# Install monitoring tools
apt install htop iotop nethogs sysstat  # Debian/Ubuntu
yum install htop iotop nethogs sysstat  # CentOS/RHEL

# configuration sysstat
nano /etc/sysstat/sysstat
# Change ENABLED="false" to ENABLED="true"

# Restart sysstat service
systemctl restart sysstat

# View system load
uptime

# View CPU using
top

# View memory using
free -h

# View disk using
df -h

# View network connections
ss -tuln

10.2.4 Backup Strategy Configuration

# Create backup directories
mkdir -p /backup/{daily,weekly,monthly}

# Create backup script
nano /root/backup.sh
#!/bin/bash

# Variable definitions
DATE=$(date +%Y-%m-%d)
DST_DIR=/backup/daily/$DATE

# Create backup directory
mkdir -p $DST_DIR

# Backup system configuration
rsync -avz /etc $DST_DIR/

# Backup user data
rsync -avz /home $DST_DIR/

# Backup web data
if [ -d /var/www ]; then
    rsync -avz /var/www $DST_DIR/
fi

# Backup database
if command -v mysqldump &> /dev/null; then
    mysqldump -u root -ppassword --all-databases > $DST_DIR/databases.sql
fi

# Compress backup
tar -czvf $DST_DIR.tar.gz $DST_DIR
rm -rf $DST_DIR

# Clean backups older than 7 days
find /backup/daily -name "*.tar.gz" -mtime +7 -delete

# Set script execution permission
chmod +x /root/backup.sh

# Add to crontab
crontab -e
# Add the following line (execute at 1 AM daily)
0 1 * * * /root/backup.sh

# Manually execute backup test
/root/backup.sh

11. Interactive Exercises

Exercise 1: Service managementment

Perform the following operations:

  • 1. View currently run services on the system.
  • 2. Disable unnecessary services (such as bluetooth, cups, etc.).
  • 3. Enable necessary services (such as sshd, fail2ban, etc.).
  • 4. Restart the sshd service and verify its status.
  • 5. Set the sshd service to start automatically on boot.

Exercise 2: Log managementment

Perform the following operations:

  • 1. Use journalctl to view system logs.
  • 2. View logs for the sshd service.
  • 3. View system logs in real-time.
  • 4. configuration rsyslog to output logs from a specific service to a separate file.
  • 5. configuration logrotate to manage log file size and retention time.

Exercise 3: System Monitoring

Perform the following operations:

  • 1. Use top/htop to view system status.
  • 2. Use vmstat to view virtual memory using.
  • 3. Use iostat to view disk I/O status.
  • 4. Use ss to view network connection status.
  • 5. Install and configure sysstat to view system performance historical data.

Exercise 4: System Backup

Perform the following operations:

  • 1. Use rsync to backup the system configuration directory.
  • 2. Use tar to create a compressed backup file.
  • 3. Create an automated backup script including system configuration, user data, and database.
  • 4. Add the backup script to crontab for scheduled execution.
  • 5. Test the backup recovery process.

Recommended Tutorials

Linux Basics Linux File System Linux Command Line Linux Users and Permissions Linux Process managementment Linux Network Configuration Linux Server Configuration Linux Security Linux advanced topics View complete Tutorial List