Nginxlogmanagementtutorial

logmanagementoverview

for what需要logmanagement？

log is Nginxserver important 组成部分, 它记录了server runstatus, requestprocessingcircumstances and errorinformation. throughlogmanagement, 可以monitorserver runstatus, analysis访问模式, 排查error and securityissues, for server optimization and maintenanceproviding important 依据.

log class型

访问log (access_log) : 记录客户端 requestinformation, includingrequestmethod, URL, status码, response时间etc.
errorlog (error_log) : 记录server errorinformation, includingconfigurationerror, run时erroretc.
otherlog: such as rewrite log, upstream logetc., 用于specificfunctions debug and monitor

logmanagement 目标

monitorserverstatus: throughlogUnderstandserver runstatus and performance
analysis访问模式: Understanduser 访问behavior and 偏 good
排查error: througherrorlog fast 速定位 and 解决issues
securityaudit: 检测 and analysis潜 in security威胁
optimizationperformance: 根据loganalysis结果optimizationserverconfiguration
compliance要求: 满足某些行业 compliance性要求

访问logconfiguration

访问log 作用

访问log记录了客户端 for server 所 has request, includingrequest时间, requestmethod, requestURL, status码, response big small , response时间etc.information. throughanalysis访问log, 可以Understand网站访问circumstances, userbehavior and serverperformance.

1. basic访问logconfiguration

# basic访问logconfiguration
http {
    # 定义log格式
    log_format main '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';
    
    # 全局访问logconfiguration
    access_log /var/log/nginx/access.log main;
    
    server {
        listen 80;
        server_name example.com;
        
        # server级别 访问logconfiguration
        access_log /var/log/nginx/example.access.log main;
        
        location / {
            root /var/www/example.com;
            index index.html;
        }
    }
}

2. 自定义log格式

# 自定义log格式
http {
    # 标准log格式
    log_format main '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';
    
    # package含response时间 log格式
    log_format extended '$remote_addr - $remote_user [$time_local] "$request" '
                          '$status $body_bytes_sent "$http_referer" '
                          '"$http_user_agent" "$http_x_forwarded_for" '
                          'rt=$request_time ut=$upstream_response_time cs=$upstream_cache_status';
    
    # package含request体 big  small  log格式
    log_format detailed '$remote_addr - $remote_user [$time_local] "$request" '
                          '$status $body_bytes_sent $request_length "$http_referer" '
                          '"$http_user_agent" "$http_x_forwarded_for" '
                          'rt=$request_time ut=$upstream_response_time';
    
    # using自定义log格式
    access_log /var/log/nginx/access.log extended;
    
    server {
        listen 80;
        server_name example.com;
        
        access_log /var/log/nginx/example.access.log detailed;
        
        location / {
            root /var/www/example.com;
            index index.html;
        }
    }
}

3. log格式variable说明

$remote_addr: 客户端IP地址
$remote_user: 客户端user名 (such as果启用了authentication)
$time_local: 本地时间
$request: 完整 request行 (method, URL, protocol)
$status: HTTPstatus码
$body_bytes_sent: 发送给客户端 response体 big small
$http_referer: HTTP Referer头
$http_user_agent: HTTP User-Agent头
$http_x_forwarded_for: HTTP X-Forwarded-For头
$request_time: requestprocessing时间 (秒)
$upstream_response_time: on 游serverresponse时间 (秒)
$upstream_cache_status: on 游cachestatus
$request_length: request long 度 (includingrequest行, 头部 and request体)

4. 访问logconfigurationexample

# 按虚拟主机configuration访问log
http {
    log_format main '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';
    
    server {
        listen 80;
        server_name example.com;
        
        access_log /var/log/nginx/example.access.log main;
        error_log /var/log/nginx/example.error.log warn;
        
        root /var/www/example.com;
        index index.html;
        
        location / {
            try_files $uri $uri/ =404;
        }
    }
    
    server {
        listen 80;
        server_name api.example.com;
        
        access_log /var/log/nginx/api.access.log main;
        error_log /var/log/nginx/api.error.log warn;
        
        location / {
            proxy_pass http://backend;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        }
    }
}

errorlogconfiguration

errorlog 作用

errorlog记录了server errorinformation, includingconfigurationerror, run时error, 连接erroretc.. througherrorlog, 可以 fast 速定位 and 解决server issues, 确保server 正常run.

1. basicerrorlogconfiguration

# basicerrorlogconfiguration
http {
    # 全局errorlogconfiguration
    error_log /var/log/nginx/error.log warn;
    
    server {
        listen 80;
        server_name example.com;
        
        # server级别 errorlogconfiguration
        error_log /var/log/nginx/example.error.log error;
        
        root /var/www/example.com;
        index index.html;
        
        location / {
            try_files $uri $uri/ =404;
        }
    }
}

2. errorlog级别

Nginxerrorlogsupport以 under 级别 ( from low to high ) :

debug: 详细 debuginformation, 用于Development and debug
info: 一般information, 用于Understandserver runstatus
notice: 注意information, 可能需要关注但不 is error
warn: warninginformation, 表示可能存 in issues
error: errorinformation, 表示server遇 to issues
crit: 严重errorinformation, 表示server遇 to 严重issues
alert: 警报information, 表示需要立即采取行动 issues
emerg: 紧急information, 表示server无法继续run issues

建议: in produceenvironmentin, 建议using warn or error 级别, 避免logfile过 big .

3. errorlogconfigurationexample

# 按environmentconfigurationerrorlog
http {
    # Developmentenvironment: 详细log
    error_log /var/log/nginx/error.log debug;
    
    # produceenvironment: 只记录warning and error
    # error_log /var/log/nginx/error.log warn;
    
    server {
        listen 80;
        server_name example.com;
        
        error_log /var/log/nginx/example.error.log error;
        
        root /var/www/example.com;
        index index.html;
        
        location / {
            try_files $uri $uri/ =404;
        }
    }
    
    server {
        listen 80;
        server_name api.example.com;
        
        # APIserver: 更详细 errorlog
        error_log /var/log/nginx/api.error.log info;
        
        location / {
            proxy_pass http://backend;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        }
    }
}

loganalysistool

for what需要loganalysistool？

随着网站traffic 增 long , logfile会变得越来越 big , 手动analysislog变得非常 difficult . using专业 loganalysistool, 可以更 high 效地analysislogdata, 提取 has value information, for server optimization and maintenanceproviding依据.

1. 常用 loganalysistool

awk/sed/grep: Linux in 置 commands行tool, 用于 simple loganalysis
GoAccess: 实时 Webloganalysistool, providing直观 visualization界面
ELK Stack: Elasticsearch, Logstash, Kibana 组合, 用于large-scaleloganalysis
Graylog: 集in式logmanagement平台, support实时搜索 and analysis
Splunk: 商业loganalysis平台, functions强 big 但收费
AWStats: 免费 loganalysistool, providing详细访问statistics
Webalizer: 另一个免费 loganalysistool, providingbasic 访问statistics

2. usingGoAccessanalysislog

# installationGoAccess (Ubuntu/Debian) 
sudo apt update
sudo apt install goaccess

# installationGoAccess (CentOS/RHEL) 
sudo yum install epel-release
sudo yum install goaccess

# analysis访问log并生成HTML报告
goaccess /var/log/nginx/access.log -o /var/www/html/report.html --log-format=COMBINED

# 实时analysislog
goaccess /var/log/nginx/access.log -o /var/www/html/report.html --log-format=COMBINED --real-time-html

# analysis压缩 logfile
gunzip -c /var/log/nginx/access.log.gz | goaccess -o /var/www/html/report.html --log-format=COMBINED

3. usingELK Stackanalysislog

# installationELK Stack (usingDocker) 
docker pull docker.elastic.co/elasticsearch/elasticsearch:7.14.0
docker pull docker.elastic.co/kibana/kibana:7.14.0
docker pull docker.elastic.co/logstash/logstash:7.14.0

# configurationLogstashprocessingNginxlog
# /etc/logstash/conf.d/nginx.conf
input {
  file {
    path => "/var/log/nginx/access.log"
    start_position => "beginning"
    sincedb_path => "/dev/null"
  }
}

filter {
  grok {
    match => {
      "message" => "%{IPORHOST:remote_addr} - %{DATA:remote_user} \[%{HTTPDATE:time_local}\] \"%{WORD:method} %{URIPATHPARAM:uri} HTTP/%{NUMBER:http_version}\" %{NUMBER:status} %{NUMBER:body_bytes_sent} \"%{DATA:http_referer}\" \"%{DATA:http_user_agent}\""
    }
  }
  date {
    match => ["time_local", "dd/MMM/yyyy:HH:mm:ss Z"]
    target => "@timestamp"
  }
}

output {
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "nginx-access-%{+YYYY.MM.dd}"
  }
}

# 启动ELK Stack
docker-compose up -d

4. simple loganalysiscommands

# 查看访问量最 high  IP
awk '{print $1}' /var/log/nginx/access.log | sort | uniq -c | sort -nr | head -10

# 查看访问量最 high  URL
awk '{print $7}' /var/log/nginx/access.log | sort | uniq -c | sort -nr | head -10

# 查看status码statistics
awk '{print $9}' /var/log/nginx/access.log | sort | uniq -c | sort -nr

# 查看response时间较 long  request
awk '{if($NF>1) print $0}' /var/log/nginx/access.log | sort -kNF -nr | head -10

# 查看specificstatus码 request
grep " 404 " /var/log/nginx/access.log | head -20

# 查看specificIP request
grep "192.168.1.100" /var/log/nginx/access.log | head -20

# 查看specific时间段 request
grep "10/Oct/2023:14:" /var/log/nginx/access.log | head -20

log轮转configuration

for what需要log轮转？

随着时间推移, logfile会变得越来越 big , 占用 big 量 disk空间, 并且会影响loganalysis efficiency. log轮转 is amanagementlogfile method, 它会定期将logfilerename, 压缩 and delete, 以保持logfile 合理 big small .

1. usinglogrotateforlog轮转

big many 数Linuxsystem都 in 置了 logrotate tool, 用于managementlogfile 轮转.

# Nginxlog轮转configuration
# /etc/logrotate.d/nginx
/var/log/nginx/*.log {
    daily
    missingok
    rotate 14
    compress
    delaycompress
    notifempty
    create 0640 nginx nginx
    sharedscripts
    postrotate
        if [ -f /var/run/nginx.pid ]; then
            kill -USR1 $(cat /var/run/nginx.pid)
        fi
    endscript
}

2. logrotateconfigurationparameter说明

daily: 每天轮转一次
weekly: 每周轮转一次
monthly: 每月轮转一次
missingok: such as果logfile不存 in , ignoreerror
rotate 14: 保留14个轮转file
compress: 压缩轮转 after logfile
delaycompress: latency压缩, 压缩 before 一天 logfile
notifempty: such as果logfile for 空, 不for轮转
create 0640 nginx nginx: creation new logfile, 设置permission for 0640, 所 has 者 for nginx:nginx
sharedscripts: in 所 has logfile轮转 after 执行一次脚本
postrotate: in log轮转 after 执行脚本
kill -USR1 $(cat /var/run/nginx.pid): 向Nginx发送USR1信号, 重 new 打开logfile

3. 手动执行log轮转

# 手动执行log轮转
sudo logrotate /etc/logrotate.d/nginx

# 强制执行log轮转 (即使条件不满足) 
sudo logrotate -f /etc/logrotate.d/nginx

# testlog轮转configuration
sudo logrotate -d /etc/logrotate.d/nginx

4. log轮转 best practices

根据log 生成速度, 选择合适轮转频率 (daily/weekly/monthly)
设置合理保留时间, 避免占用过 many disk空间
启用压缩, reducinglogfile big small
using delaycompress, 确保loganalysistoolable to读取最 new logfile
定期checklogfile big small and disk空间usingcircumstances
考虑将logfilestore in 单独 partition, 避免logfile占用过 many systempartition空间

logsecurity and performance

1. logsecurity

# logsecurityconfiguration
http {
    # 设置logfilepermission
    #  in logrotateconfigurationin设置: create 0640 nginx nginx
    
    # 避免记录敏感information
    log_format main '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';
    
    # 不记录request体in 敏感information
    # 注意: 默认circumstances under , Nginx不会记录request体 in 容
    
    # 限制logfile 访问permission
sudo chmod 0640 /var/log/nginx/*.log
sudo chown nginx:nginx /var/log/nginx/*.log

    # encryptionstorelog (such as果需要) 
    # 可以usingfilesystemencryption or 专门 logencryptiontool
}

2. logperformanceoptimization

# logperformanceoptimization
http {
    # 关闭不需要 访问log
    location /images/ {
        access_log off;
        root /var/www/example.com;
    }
    
    # using缓冲写入log
    access_log /var/log/nginx/access.log main buffer=32k;
    
    # asynchronous写入log (Nginx 1.7.10+) 
    access_log /var/log/nginx/access.log main buffer=32k flush=1m;
    
    # 合理设置log格式, 避免记录不必要 information
    log_format minimal '$remote_addr - [$time_local] "$request" $status $body_bytes_sent';
    
    #  for 静态fileusing简化 log格式
    location /static/ {
        access_log /var/log/nginx/static.access.log minimal;
        root /var/www/example.com;
    }
}

3. logperformanceoptimization建议

for 静态file and 频繁访问 resource关闭 or using简化访问log
using缓冲写入log, reducingI/Ooperation
usingasynchronous写入log (Nginx 1.7.10+) , improvingperformance
合理设置log格式, 只记录必要 information
定期clean or 归档 old logfile
考虑using专门 logserver, 将logstore in 远程server on

实践case: configuration完整 logmanagementsystem

fake设我们需要 for 一个produceenvironment Nginxserverconfiguration完整 logmanagementsystem, 要求:

记录详细访问log and errorlog
implementationlog轮转, 保留30天 log
usingGoAccessfor实时loganalysis
optimizationlogperformance, reducing for server 影响
确保log security性

# 1. Nginxlogconfiguration
# /etc/nginx/nginx.conf
http {
    # 定义详细 log格式
    log_format main '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for" '
                      'rt=$request_time ut=$upstream_response_time';
    
    # 定义简化 log格式
    log_format minimal '$remote_addr - [$time_local] "$request" $status $body_bytes_sent';
    
    # 全局logconfiguration
    access_log /var/log/nginx/access.log main buffer=32k flush=1m;
    error_log /var/log/nginx/error.log warn;
    
    server {
        listen 80;
        server_name example.com;
        
        # server级别 logconfiguration
        access_log /var/log/nginx/example.access.log main buffer=32k flush=1m;
        error_log /var/log/nginx/example.error.log error;
        
        root /var/www/example.com;
        index index.html;
        
        # 静态fileusing简化 log格式
        location /static/ {
            access_log /var/log/nginx/static.access.log minimal buffer=32k flush=1m;
        }
        
        # graph片 and graph标关闭访问log
        location ~* \.(jpg|jpeg|png|gif|ico|svg)$ {
            access_log off;
        }
        
        location / {
            try_files $uri $uri/ =404;
        }
    }
}

# 2. log轮转configuration
# /etc/logrotate.d/nginx
/var/log/nginx/*.log {
    daily
    missingok
    rotate 30
    compress
    delaycompress
    notifempty
    create 0640 nginx nginx
    sharedscripts
    postrotate
        if [ -f /var/run/nginx.pid ]; then
            kill -USR1 $(cat /var/run/nginx.pid)
        fi
    endscript
}

# 3. GoAccessconfiguration
# installationGoAccess
sudo apt update
sudo apt install goaccess

# creationloganalysis脚本
# /usr/local/bin/analyze-nginx-logs.sh
#!/bin/bash
mkdir -p /var/www/html/reports
chown nginx:nginx /var/www/html/reports
goaccess /var/log/nginx/access.log -o /var/www/html/reports/access.html --log-format=COMBINED --real-time-html

# 设置定时task
# /etc/crontab
0 * * * * root /usr/local/bin/analyze-nginx-logs.sh

# 4. logsecurityconfiguration
sudo chmod 0640 /var/log/nginx/*.log
sudo chown nginx:nginx /var/log/nginx/*.log

# 5. monitorlogfile big  small 
# /usr/local/bin/check-log-size.sh
#!/bin/bash
LOG_DIR="/var/log/nginx"
MAX_SIZE=104857600  # 100MB

for log_file in "$LOG_DIR"/*.log; do
    if [ -f "$log_file" ]; then
        SIZE=$(stat -c %s "$log_file")
        if [ "$SIZE" -gt "$MAX_SIZE" ]; then
            echo "Warning: Log file $log_file is too large: $(($SIZE / 1024 / 1024))MB"
        fi
    fi
done

# 设置定时task
# /etc/crontab
*/30 * * * * root /usr/local/bin/check-log-size.sh

这个configurationpackage含了以 under logmanagement措施:

定义了详细访问log格式, package含request时间, response时间etc.information
for 静态fileusing简化 log格式, reducinglog量
for graph片 and graph标关闭访问log, 进一步reducinglog量
using缓冲 and asynchronous写入log, improvingperformance
configurationlog轮转, 保留30天 log, 并自动压缩
usingGoAccessfor实时loganalysis, 生成visualization报告
设置合理 logfilepermission, 确保security性
monitorlogfile big small , 避免logfile过 big

互动练习

1. 以 under 哪些 is Nginxlogmanagement important 措施？ ( many 选)

A. configuration详细访问log格式

B. implementationlog轮转, 避免logfile过 big

C. usingloganalysistool, analysis访问模式 and error

D. optimizationlogperformance, reducing for server 影响

E. 确保log security性, 保护敏感information

2. 以 under 哪个指令用于configuration访问log？

A. access_log

B. error_log

C. log_format

D. log_rotate

3. 实践练习: configurationNginxlogmanagementsystem

请configuration一个Nginxlogmanagementsystem, 要求:

定义详细访问log格式, package含request时间, response时间etc.information
for 静态fileusing简化 log格式
configurationlog轮转, 保留14天 log, 并自动压缩
usingGoAccessforloganalysis, 生成HTML报告
设置合理 logfilepermission, 确保security性

summarized and 展望

through本tutorial, 您已经Learning了Nginxlogmanagement coreknowledge, includinglogconfiguration, log格式, loganalysis and log轮转etc.. logmanagement is Nginxservermanagement important 组成部分, 它可以:

monitorserver runstatus and performance
analysisuser 访问behavior and 偏 good
fast 速定位 and 解决server error
检测 and analysis潜 in security威胁
for server optimization and maintenanceproviding important 依据

in after 续 tutorialin, 我们将深入LearningNginx advanced主题, includingadvancedfunctions and 企业级application, helping您全面MasterNginx usingtechniques.