WEB SERVER LOG TRIAGE
A triage reference for Apache httpd and Nginx access and error logs: where the logs are, how to read the format, and read-only greps that surface scanners, injection attempts, brute force, web shells, large downloads and log tampering. It is for detection and investigation, NOT for attacking web servers. Commands assume bash, GNU coreutils and awk on the server or on a copy of its logs; paths are given for Debian/Ubuntu and RHEL/Rocky/Alma. An access log shows the REQUEST and the status code, not whether an attack worked. Confirm every hit with the response size, the error log, the application logs and the files on disk. All addresses below (192.0.2.x, 198.51.100.x, 203.0.113.x, example.com) are placeholders.
1. PRESERVE THE LOGS FIRST#
Copy logs and config off the host before logrotate deletes old files or anyone "cleans up". Default retention: Debian keeps 14 daily files for Apache and Nginx; RHEL keeps 4 weekly httpd files and 10 daily nginx files. Inspect: ls -la --time-style=full-iso /var/log/apache2 /var/log/nginx # Debian ls -la --time-style=full-iso /var/log/httpd /var/log/nginx # RHEL Collect (to evidence storage, not the web root): E=/mnt/evidence/web-$(hostname)-$(date -u +%Y%m%dT%H%M%SZ).tar.gz tar -czf "$E" /var/log/apache2 /var/log/nginx /etc/apache2 /etc/nginx sha256sum "$E" | tee -a /mnt/evidence/SHA256SUMS # RHEL: /var/log/httpd and /etc/httpd instead of the apache2 paths Notes: keep the config with the logs: it defines the log format. Every access log line carries its own UTC offset ([16/Oct/2026:13:00:01 +0000]). Work on the copy from here on.
2. FIND EVERY LOG FILE#
Default files:
Debian Apache: /var/log/apache2/access.log, error.log,
other_vhosts_access.log (vhosts without their own log)
RHEL httpd: /var/log/httpd/access_log, error_log (underscores;
/etc/httpd/logs is a link to it)
Nginx (both): /var/log/nginx/access.log, error.log
Inspect:
apache2ctl -S # Debian: vhosts and the main ErrorLog
httpd -S # RHEL ("apachectl -S" is refused there)
apache2ctl -t -D DUMP_INCLUDES 2>/dev/null | awk '/\//{print $NF}' \
| xargs grep -Hn -E '^\s*(CustomLog|GlobalLog|ErrorLog|TransferLog)'
# RHEL: the same line with httpd instead of apache2ctl
nginx -T 2>/dev/null | grep -nE 'access_log|error_log|log_format'
Notes: a CustomLog inside a <VirtualHost> sends that host's requests ONLY
to its own file. "grep -r" skips the symlinks in sites-enabled; use the
DUMP_INCLUDES list above or grep -R.
Flags: access_log off or a log sent to /dev/null; CustomLog with env=!...
(some requests deliberately not logged); a vhost with no log at all; logs
written to a pipe or a path outside /var/log.
3. READ THE FORMAT#
Apache "combined" (mod_log_config):
%h %l %u %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i"
Debian's combined uses %O (bytes sent INCLUDING headers) instead of %b.
Nginx "combined" (ngx_http_log_module):
$remote_addr - $remote_user [$time_local] "$request" $status
$body_bytes_sent "$http_referer" "$http_user_agent"
RHEL's nginx "main" format adds "$http_x_forwarded_for" at the end.
Fields for awk (combined, default space split):
$1 client $4 time $6 "METHOD $7 URL $9 status $10 bytes
awk -F'"': $2 request line, $4 referer, $6 user agent
Inspect:
L=/var/log/nginx/access.log # or a file from section 2
head -3 "$L" # check the format before trusting awk
Notes: Apache logs " and \ as \" and \\, whitespace as \n or \t, other
non-printable bytes as \xhh; Nginx logs ", \ and bytes below 32 or above
126 as \xXX. URL-encoding (%2e) is logged as sent, so grep both forms.
Response time is not in combined: add %D (microseconds) to Apache or
$request_time (seconds, millisecond resolution) to Nginx.
4. ROTATED LOGS AND TIME WINDOWS#
Inspect: ls -tr /var/log/nginx/access.log* # oldest first, by mtime zcat -f $(ls -tr /var/log/nginx/access.log*) > all-access.log L=all-access.log # every day, oldest first zgrep -h '16/Oct/2026:1[3-5]:' /var/log/nginx/access.log* > window.log zgrep -c ' 404 ' /var/log/nginx/access.log* # per file, gz or plain Notes: zcat -f and zgrep read plain and .gz files alike. Debian rotates to access.log.1, access.log.2.gz ...; RHEL uses date suffixes (access_log-20261011) and does not compress httpd logs by default. Write all-access.log to your analysis box, not the web server. The sections below use "$L"; point it at window.log to stay in 13:00-15:59.
5. FIRST LOOK: WHO, WHAT, WHEN#
Inspect:
awk '{print $1}' "$L" | sort | uniq -c | sort -rn | head # clients
awk '{print $9}' "$L" | sort | uniq -c | sort -rn # statuses
awk '{print $7}' "$L" | sort | uniq -c | sort -rn | head -20 # URLs
awk -F'"' '{print $6}' "$L" | sort | uniq -c | sort -rn | head
# requests per minute, busiest first:
awk '{print substr($4, 2, 17)}' "$L" | uniq -c | sort -rn | head
# one client's story, in order:
awk '$1 == "203.0.113.88" {print $4, $6, $7, $9, $10}' "$L"
Flags: one address far above the rest; a burst minute; a user agent that
names a tool; a status mix that changes (404s, then 200s).
6. SCANNERS AND PROBES (ATT&CK T1595.002, T1595.003)#
Inspect:
awk '$9 == 404 {print $1}' "$L" | sort | uniq -c | sort -rn | head
# well-known probe paths, and what each client got back:
P='"[A-Z]+ /(\.env|\.git/|\.aws/|wp-login\.php|xmlrpc\.php'
P="$P|phpmyadmin|server-status|[^ ]*\.(bak|old|sql|zip)[ ?])"
grep -E "$P" "$L" | awk '{print $1, $9, $7}' | sort | uniq -c \
| sort -rn | head
# tool names in the user agent (it is set by the client, so a hint only):
awk -F'"' '{print $6}' "$L" \
| grep -iE 'zgrab|nmap|nikto|sqlmap|masscan|nuclei|gobuster|wpscan' \
| sort | uniq -c | sort -rn
Flags: what matters is a 200 (or a large size) on a sensitive path such as
/.env, /.git/config or a backup file; hundreds of 404s from one address in
minutes is background noise on any public server unless something hit.
7. INJECTION AND TRAVERSAL ATTEMPTS (ATT&CK T1190)#
Inspect (grep the whole line: payloads also arrive in the user agent and
the referer):
grep -iE '\.\./|%2e%2e|\.\.%2f|%252e' "$L" # path traversal
grep -iE 'union(%20|\+|\s|/\*.*\*/)+(all(%20|\+|\s)+)?select' "$L"
grep -iE 'sleep(\(|%28)|benchmark(\(|%28)' "$L" # time-based SQLi
grep -iE '\$\{jndi:|%24%7bjndi' "$L" # Log4Shell
grep -iE '<script|%3cscript' "$L" # reflected XSS
# for each source found, what it got back:
grep -iE '\.\./|%2e%2e|\.\.%2f|%252e|union|jndi|<script|%3cscript' "$L" \
| awk '{print $1, $9, $10}' | sort | uniq -c
Flags: a 200 whose size matches the target file (a traversal to
/etc/passwd answered with a few KB); the same address then requesting
new URLs; 500 errors on the attempts (the input reached code). Expect false
positives from search boxes and from URLs that legitimately contain "..".
8. BRUTE FORCE ON LOGIN PAGES (ATT&CK T1110.001, T1110.003, T1110.004)#
Inspect:
# POSTs to login URLs, per client and status:
awk '$6 == "\"POST" && $7 ~ /login|signin|auth|wp-login/ {print $1, $9}' \
"$L" | sort | uniq -c | sort -rn | head
# per minute, to see the rate:
awk '$6 == "\"POST" && $7 ~ /login/ {print substr($4, 2, 17)}' "$L" \
| uniq -c | sort -rn | head
Flags: one address with hundreds of failures then a different status
(many apps answer 200 to a failed login and 302 to a good one: learn your
app's pattern first); many addresses with one or two attempts each
(spraying or credential stuffing). The access log does not show the user
name in a POST body: take it from the application log.
9. WEB SHELLS (ATT&CK T1505.003)#
Inspect the logs:
# scripts in upload or static folders:
W='"(GET|POST) /([^ ]*/)?(uploads?|images?|img|files|static|tmp)/'
W="$W[^ ]*\.(php[0-9]?|phtml|phar|jsp|aspx?)[ ?]"
grep -E "$W" "$L" | awk '{print $1, $6, $7, $9}' | sort | uniq -c
# POST targets, least requested first:
awk '$6 == "\"POST" {sub(/\?.*/, "", $7); print $7}' "$L" \
| sort | uniq -c | sort -n | head
# scripts requested by only ONE client address:
awk '{u = $7; sub(/\?.*/, "", u)}
u ~ /\.(php[0-9]?|phtml|phar|jsp|aspx?)$/ {print u, $1}' "$L" \
| sort -u | awk '{n[$1]++; ip[$1] = $2}
END {for (u in n) if (n[u] == 1) print u, ip[u]}'
Inspect the host (root, read-only):
find /var/www -type f \( -name '*.php*' -o -name '*.phtml' \
-o -name '*.phar' \) -path '*upload*' -ls
find /var/www -type f -newermt '2026-10-15' -ls # changed since
ps -u www-data -o pid,ppid,lstart,args --forest # RHEL: apache, nginx
ss -tnp | grep -E 'apache2|httpd|nginx|php-fpm' # live connections
Flags: a .php file in an upload folder; repeated POSTs to one script by
one address, often with Referer "-"; a script nobody else ever requested;
web root files newer than your last deploy; sh, bash, python, perl, nc or
curl as a child of the web server user; outbound connections from it.
10. LARGE RESPONSES AND EXFILTRATION (ATT&CK T1041)#
Inspect:
# biggest single responses:
awk '$10 ~ /^[0-9]+$/ {print $10, $1, $9, $7}' "$L" | sort -rn | head
# bytes per client address:
awk '$10 ~ /^[0-9]+$/ {b[$1] += $10}
END {for (i in b) printf "%.0f %s\n", b[i], i}' "$L" | sort -rn | head
Flags: archives, dumps or backups (.sql, .tar.gz, .zip, .bak) served from
an upload folder or the web root; one address far above its normal
volume; a big 200 for a URL that should not exist. Apache %b and Nginx
$body_bytes_sent exclude headers; Apache %O includes them.
11. 5XX SPIKES AND THE ERROR LOG#
Inspect:
awk '$9 ~ /^5/ {print substr($4, 2, 17), $9}' "$L" | uniq -c \
| sort -rn | head # 5xx per minute and code
grep 'Oct 16 16:0' /var/log/apache2/error.log | head # Apache window
grep '^2026/10/16 16:0' /var/log/nginx/error.log | head # Nginx window
grep -E 'AH00052|uploads/' /var/log/apache2/error.log
# RHEL: /var/log/httpd/error_log
Error log line formats:
Apache: [Fri Oct 16 16:02:04.000001 2026] [module:level] [pid N:tid N]
[client 203.0.113.5:51514] message
Nginx: 2026/10/16 16:02:04 [error] 1234#1234: *5 message, client: ...
Flags: PHP warnings or errors that name a file in an upload folder (a
script running where none should); AH00052 "child pid ... exit signal
Segmentation fault" after odd requests; errors from an address you already
flagged. Most 5xx spikes are outages (backend down, AH00957 "attempt to
connect ... failed"): check the deploy and service history first.
12. PROXIES, CDNS AND X-FORWARDED-FOR#
Behind a load balancer or CDN, the first field is the PROXY unless the
server rewrites it (tested on Apache 2.4 and Nginx 1.26):
- Apache mod_remoteip: with a trusted RemoteIPInternalProxy or
RemoteIPTrustedProxy, %h and %a log the address from the header and
%{c}a logs the real peer. A header from an untrusted peer is ignored.
- Nginx realip (set_real_ip_from + real_ip_header): $remote_addr becomes
the last address in the header; $realip_remote_addr keeps the peer.
- Without these modules, X-Forwarded-For is just text the client sent.
Inspect:
apache2ctl -M 2>/dev/null | grep remoteip # RHEL: httpd -M
nginx -T 2>/dev/null | grep -nE 'real_ip|set_real_ip_from'
Flags: an X-Forwarded-For value in a log line from a peer that is not
your proxy (spoofed); a trusted-proxy range wider than your real proxies.
Log %{c}a or $realip_remote_addr next to the client address.
13. TAMPERING AND GAPS (ATT&CK T1685.006)#
Inspect:
awk '{print substr($4, 2, 14)}' "$L" | uniq -c # lines per hour
find /var/log/apache2 /var/log/nginx -type l -ls # links (/dev/null?)
ls -la --time-style=full-iso /var/log/nginx/ # sizes and mtimes
Flags: whole hours missing on a busy site (the list above skips them); a
current log much smaller than its rotated neighbours; a log replaced by a
link to /dev/null; changed owner or mode; mtime older than the newest line.
Compare with any copy shipped off the host (syslog, SIEM, CDN logs): an
attacker on the box can edit local files, not the copy you already sent.
QUICK TRIAGE ORDER#
1. Copy logs and config off the host and hash them (section 1). 2. Find every log file and the real format (sections 2 and 3). 3. Join the rotated files and cut the incident window (section 4). 4. Busiest clients, statuses, URLs and minutes (section 5). 5. Probes and injection attempts that got a 200 or a big size (6 and 7). 6. Login brute force and its last status per address (section 8). 7. Web shells in the logs AND on disk; web user processes (section 9). 8. Large downloads and per-address volume (section 10). 9. Error log for the same minutes; proxy settings; gaps (sections 11-13). Formats and variables checked against the Apache httpd 2.4 documentation (mod_log_config, log files, mod_remoteip) and the Nginx documentation (ngx_http_log_module, ngx_http_core_module, ngx_http_realip_module); technique IDs per MITRE ATT&CK Enterprise v20. Default paths, formats and rotation read from Debian 13 (Apache 2.4.69, Nginx 1.26.3) and Rocky Linux 9 (httpd 2.4.62, Nginx 1.20.1); every command run on synthetic logs.
WEB SERVER LOG TRIAGE
=====================
A triage reference for Apache httpd and Nginx access and error logs: where
the logs are, how to read the format, and read-only greps that surface
scanners, injection attempts, brute force, web shells, large downloads and
log tampering. It is for detection and investigation, NOT for attacking web
servers. Commands assume bash, GNU coreutils and awk on the server or on a
copy of its logs; paths are given for Debian/Ubuntu and RHEL/Rocky/Alma.
An access log shows the REQUEST and the status code, not whether an attack
worked. Confirm every hit with the response size, the error log, the
application logs and the files on disk. All addresses below (192.0.2.x,
198.51.100.x, 203.0.113.x, example.com) are placeholders.
1. PRESERVE THE LOGS FIRST
--------------------------
Copy logs and config off the host before logrotate deletes old files or
anyone "cleans up". Default retention: Debian keeps 14 daily files for
Apache and Nginx; RHEL keeps 4 weekly httpd files and 10 daily nginx files.
Inspect:
ls -la --time-style=full-iso /var/log/apache2 /var/log/nginx # Debian
ls -la --time-style=full-iso /var/log/httpd /var/log/nginx # RHEL
Collect (to evidence storage, not the web root):
E=/mnt/evidence/web-$(hostname)-$(date -u +%Y%m%dT%H%M%SZ).tar.gz
tar -czf "$E" /var/log/apache2 /var/log/nginx /etc/apache2 /etc/nginx
sha256sum "$E" | tee -a /mnt/evidence/SHA256SUMS
# RHEL: /var/log/httpd and /etc/httpd instead of the apache2 paths
Notes: keep the config with the logs: it defines the log format. Every
access log line carries its own UTC offset ([16/Oct/2026:13:00:01 +0000]).
Work on the copy from here on.
2. FIND EVERY LOG FILE
----------------------
Default files:
Debian Apache: /var/log/apache2/access.log, error.log,
other_vhosts_access.log (vhosts without their own log)
RHEL httpd: /var/log/httpd/access_log, error_log (underscores;
/etc/httpd/logs is a link to it)
Nginx (both): /var/log/nginx/access.log, error.log
Inspect:
apache2ctl -S # Debian: vhosts and the main ErrorLog
httpd -S # RHEL ("apachectl -S" is refused there)
apache2ctl -t -D DUMP_INCLUDES 2>/dev/null | awk '/\//{print $NF}' \
| xargs grep -Hn -E '^\s*(CustomLog|GlobalLog|ErrorLog|TransferLog)'
# RHEL: the same line with httpd instead of apache2ctl
nginx -T 2>/dev/null | grep -nE 'access_log|error_log|log_format'
Notes: a CustomLog inside a <VirtualHost> sends that host's requests ONLY
to its own file. "grep -r" skips the symlinks in sites-enabled; use the
DUMP_INCLUDES list above or grep -R.
Flags: access_log off or a log sent to /dev/null; CustomLog with env=!...
(some requests deliberately not logged); a vhost with no log at all; logs
written to a pipe or a path outside /var/log.
3. READ THE FORMAT
------------------
Apache "combined" (mod_log_config):
%h %l %u %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i"
Debian's combined uses %O (bytes sent INCLUDING headers) instead of %b.
Nginx "combined" (ngx_http_log_module):
$remote_addr - $remote_user [$time_local] "$request" $status
$body_bytes_sent "$http_referer" "$http_user_agent"
RHEL's nginx "main" format adds "$http_x_forwarded_for" at the end.
Fields for awk (combined, default space split):
$1 client $4 time $6 "METHOD $7 URL $9 status $10 bytes
awk -F'"': $2 request line, $4 referer, $6 user agent
Inspect:
L=/var/log/nginx/access.log # or a file from section 2
head -3 "$L" # check the format before trusting awk
Notes: Apache logs " and \ as \" and \\, whitespace as \n or \t, other
non-printable bytes as \xhh; Nginx logs ", \ and bytes below 32 or above
126 as \xXX. URL-encoding (%2e) is logged as sent, so grep both forms.
Response time is not in combined: add %D (microseconds) to Apache or
$request_time (seconds, millisecond resolution) to Nginx.
4. ROTATED LOGS AND TIME WINDOWS
--------------------------------
Inspect:
ls -tr /var/log/nginx/access.log* # oldest first, by mtime
zcat -f $(ls -tr /var/log/nginx/access.log*) > all-access.log
L=all-access.log # every day, oldest first
zgrep -h '16/Oct/2026:1[3-5]:' /var/log/nginx/access.log* > window.log
zgrep -c ' 404 ' /var/log/nginx/access.log* # per file, gz or plain
Notes: zcat -f and zgrep read plain and .gz files alike. Debian rotates to
access.log.1, access.log.2.gz ...; RHEL uses date suffixes
(access_log-20261011) and does not compress httpd logs by default.
Write all-access.log to your analysis box, not the web server. The
sections below use "$L"; point it at window.log to stay in 13:00-15:59.
5. FIRST LOOK: WHO, WHAT, WHEN
------------------------------
Inspect:
awk '{print $1}' "$L" | sort | uniq -c | sort -rn | head # clients
awk '{print $9}' "$L" | sort | uniq -c | sort -rn # statuses
awk '{print $7}' "$L" | sort | uniq -c | sort -rn | head -20 # URLs
awk -F'"' '{print $6}' "$L" | sort | uniq -c | sort -rn | head
# requests per minute, busiest first:
awk '{print substr($4, 2, 17)}' "$L" | uniq -c | sort -rn | head
# one client's story, in order:
awk '$1 == "203.0.113.88" {print $4, $6, $7, $9, $10}' "$L"
Flags: one address far above the rest; a burst minute; a user agent that
names a tool; a status mix that changes (404s, then 200s).
6. SCANNERS AND PROBES (ATT&CK T1595.002, T1595.003)
-----------------------------------------------------
Inspect:
awk '$9 == 404 {print $1}' "$L" | sort | uniq -c | sort -rn | head
# well-known probe paths, and what each client got back:
P='"[A-Z]+ /(\.env|\.git/|\.aws/|wp-login\.php|xmlrpc\.php'
P="$P|phpmyadmin|server-status|[^ ]*\.(bak|old|sql|zip)[ ?])"
grep -E "$P" "$L" | awk '{print $1, $9, $7}' | sort | uniq -c \
| sort -rn | head
# tool names in the user agent (it is set by the client, so a hint only):
awk -F'"' '{print $6}' "$L" \
| grep -iE 'zgrab|nmap|nikto|sqlmap|masscan|nuclei|gobuster|wpscan' \
| sort | uniq -c | sort -rn
Flags: what matters is a 200 (or a large size) on a sensitive path such as
/.env, /.git/config or a backup file; hundreds of 404s from one address in
minutes is background noise on any public server unless something hit.
7. INJECTION AND TRAVERSAL ATTEMPTS (ATT&CK T1190)
---------------------------------------------------
Inspect (grep the whole line: payloads also arrive in the user agent and
the referer):
grep -iE '\.\./|%2e%2e|\.\.%2f|%252e' "$L" # path traversal
grep -iE 'union(%20|\+|\s|/\*.*\*/)+(all(%20|\+|\s)+)?select' "$L"
grep -iE 'sleep(\(|%28)|benchmark(\(|%28)' "$L" # time-based SQLi
grep -iE '\$\{jndi:|%24%7bjndi' "$L" # Log4Shell
grep -iE '<script|%3cscript' "$L" # reflected XSS
# for each source found, what it got back:
grep -iE '\.\./|%2e%2e|\.\.%2f|%252e|union|jndi|<script|%3cscript' "$L" \
| awk '{print $1, $9, $10}' | sort | uniq -c
Flags: a 200 whose size matches the target file (a traversal to
/etc/passwd answered with a few KB); the same address then requesting
new URLs; 500 errors on the attempts (the input reached code). Expect false
positives from search boxes and from URLs that legitimately contain "..".
8. BRUTE FORCE ON LOGIN PAGES (ATT&CK T1110.001, T1110.003, T1110.004)
-----------------------------------------------------------------------
Inspect:
# POSTs to login URLs, per client and status:
awk '$6 == "\"POST" && $7 ~ /login|signin|auth|wp-login/ {print $1, $9}' \
"$L" | sort | uniq -c | sort -rn | head
# per minute, to see the rate:
awk '$6 == "\"POST" && $7 ~ /login/ {print substr($4, 2, 17)}' "$L" \
| uniq -c | sort -rn | head
Flags: one address with hundreds of failures then a different status
(many apps answer 200 to a failed login and 302 to a good one: learn your
app's pattern first); many addresses with one or two attempts each
(spraying or credential stuffing). The access log does not show the user
name in a POST body: take it from the application log.
9. WEB SHELLS (ATT&CK T1505.003)
---------------------------------
Inspect the logs:
# scripts in upload or static folders:
W='"(GET|POST) /([^ ]*/)?(uploads?|images?|img|files|static|tmp)/'
W="$W[^ ]*\.(php[0-9]?|phtml|phar|jsp|aspx?)[ ?]"
grep -E "$W" "$L" | awk '{print $1, $6, $7, $9}' | sort | uniq -c
# POST targets, least requested first:
awk '$6 == "\"POST" {sub(/\?.*/, "", $7); print $7}' "$L" \
| sort | uniq -c | sort -n | head
# scripts requested by only ONE client address:
awk '{u = $7; sub(/\?.*/, "", u)}
u ~ /\.(php[0-9]?|phtml|phar|jsp|aspx?)$/ {print u, $1}' "$L" \
| sort -u | awk '{n[$1]++; ip[$1] = $2}
END {for (u in n) if (n[u] == 1) print u, ip[u]}'
Inspect the host (root, read-only):
find /var/www -type f \( -name '*.php*' -o -name '*.phtml' \
-o -name '*.phar' \) -path '*upload*' -ls
find /var/www -type f -newermt '2026-10-15' -ls # changed since
ps -u www-data -o pid,ppid,lstart,args --forest # RHEL: apache, nginx
ss -tnp | grep -E 'apache2|httpd|nginx|php-fpm' # live connections
Flags: a .php file in an upload folder; repeated POSTs to one script by
one address, often with Referer "-"; a script nobody else ever requested;
web root files newer than your last deploy; sh, bash, python, perl, nc or
curl as a child of the web server user; outbound connections from it.
10. LARGE RESPONSES AND EXFILTRATION (ATT&CK T1041)
----------------------------------------------------
Inspect:
# biggest single responses:
awk '$10 ~ /^[0-9]+$/ {print $10, $1, $9, $7}' "$L" | sort -rn | head
# bytes per client address:
awk '$10 ~ /^[0-9]+$/ {b[$1] += $10}
END {for (i in b) printf "%.0f %s\n", b[i], i}' "$L" | sort -rn | head
Flags: archives, dumps or backups (.sql, .tar.gz, .zip, .bak) served from
an upload folder or the web root; one address far above its normal
volume; a big 200 for a URL that should not exist. Apache %b and Nginx
$body_bytes_sent exclude headers; Apache %O includes them.
11. 5XX SPIKES AND THE ERROR LOG
--------------------------------
Inspect:
awk '$9 ~ /^5/ {print substr($4, 2, 17), $9}' "$L" | uniq -c \
| sort -rn | head # 5xx per minute and code
grep 'Oct 16 16:0' /var/log/apache2/error.log | head # Apache window
grep '^2026/10/16 16:0' /var/log/nginx/error.log | head # Nginx window
grep -E 'AH00052|uploads/' /var/log/apache2/error.log
# RHEL: /var/log/httpd/error_log
Error log line formats:
Apache: [Fri Oct 16 16:02:04.000001 2026] [module:level] [pid N:tid N]
[client 203.0.113.5:51514] message
Nginx: 2026/10/16 16:02:04 [error] 1234#1234: *5 message, client: ...
Flags: PHP warnings or errors that name a file in an upload folder (a
script running where none should); AH00052 "child pid ... exit signal
Segmentation fault" after odd requests; errors from an address you already
flagged. Most 5xx spikes are outages (backend down, AH00957 "attempt to
connect ... failed"): check the deploy and service history first.
12. PROXIES, CDNS AND X-FORWARDED-FOR
-------------------------------------
Behind a load balancer or CDN, the first field is the PROXY unless the
server rewrites it (tested on Apache 2.4 and Nginx 1.26):
- Apache mod_remoteip: with a trusted RemoteIPInternalProxy or
RemoteIPTrustedProxy, %h and %a log the address from the header and
%{c}a logs the real peer. A header from an untrusted peer is ignored.
- Nginx realip (set_real_ip_from + real_ip_header): $remote_addr becomes
the last address in the header; $realip_remote_addr keeps the peer.
- Without these modules, X-Forwarded-For is just text the client sent.
Inspect:
apache2ctl -M 2>/dev/null | grep remoteip # RHEL: httpd -M
nginx -T 2>/dev/null | grep -nE 'real_ip|set_real_ip_from'
Flags: an X-Forwarded-For value in a log line from a peer that is not
your proxy (spoofed); a trusted-proxy range wider than your real proxies.
Log %{c}a or $realip_remote_addr next to the client address.
13. TAMPERING AND GAPS (ATT&CK T1685.006)
------------------------------------------
Inspect:
awk '{print substr($4, 2, 14)}' "$L" | uniq -c # lines per hour
find /var/log/apache2 /var/log/nginx -type l -ls # links (/dev/null?)
ls -la --time-style=full-iso /var/log/nginx/ # sizes and mtimes
Flags: whole hours missing on a busy site (the list above skips them); a
current log much smaller than its rotated neighbours; a log replaced by a
link to /dev/null; changed owner or mode; mtime older than the newest line.
Compare with any copy shipped off the host (syslog, SIEM, CDN logs): an
attacker on the box can edit local files, not the copy you already sent.
QUICK TRIAGE ORDER
------------------
1. Copy logs and config off the host and hash them (section 1).
2. Find every log file and the real format (sections 2 and 3).
3. Join the rotated files and cut the incident window (section 4).
4. Busiest clients, statuses, URLs and minutes (section 5).
5. Probes and injection attempts that got a 200 or a big size (6 and 7).
6. Login brute force and its last status per address (section 8).
7. Web shells in the logs AND on disk; web user processes (section 9).
8. Large downloads and per-address volume (section 10).
9. Error log for the same minutes; proxy settings; gaps (sections 11-13).
Formats and variables checked against the Apache httpd 2.4 documentation
(mod_log_config, log files, mod_remoteip) and the Nginx documentation
(ngx_http_log_module, ngx_http_core_module, ngx_http_realip_module);
technique IDs per MITRE ATT&CK Enterprise v20. Default paths, formats and
rotation read from Debian 13 (Apache 2.4.69, Nginx 1.26.3) and Rocky Linux
9 (httpd 2.4.62, Nginx 1.20.1); every command run on synthetic logs.
Defensive reference on CyberRamen. Offensive / red-team sheets live on OffensiveRamen.com.