Featured Nginx ngxtop real time metrics for Nginx

Folks may have heard of ngxtop for real time metrics for Nginx lebinh/ngxtop · GitHub ?

Here's how I installed ngxtop for Centmin Mod Nginx based servers.

For CentOS 6, to install ngxtop type:

Code (Text):

yum -y install python-pip; pip install ngxtop sqlite-devel

For CentOS 7, to install ngxtop type:

Code (Text):

yum -y install python-pip
pip install --upgrade pip
pip install ngxtop

By default ngxtop only reports new connections (live) not previously logged connections. So if you have a relatively idle server you may not see live stats. Also Centmin Mod by default buffers a certain amount of access log entries in memory (256KB) before flushing and writing to disk for performance reasons. This means you may not get live request entries in your access logs until you do an nginx restart or reload or until that memory buffer is full (256KB) or flush time is reached (5 minutes).

Code (Text):

access_log /home/nginx/domains/domain.com/log/access.log combined buffer=256k flush=5m;

To check live stats for Centmin Mod Nginx hosted domains pass the location of the domain vhost's access.log type:

Code (Text):

ngxtop -l /home/nginx/domains/yourdomain.com/log/access.log

To check previously logged data use the --no-follow option

Code (Text):

service nginx reload
ngxtop -l /home/nginx/domains/yourdomain.com/log/access.log --no-follow

More commands outlined at lebinh/ngxtop · GitHub - just remember to pass the Centmin Mod Nginx accesslog file with the commands (-l /home/nginx/domains/yourdomain.com/log/access.log).

For example:

Code (Text):

service nginx reload
ngxtop -l /home/nginx/domains/yourdomain.com/log/access.log --no-follow top remote_addr

print status >400 for requests and referer

Code (Text):

service nginx reload
ngxtop -l /home/nginx/domains/yourdomain.com/log/access.log --no-follow -i 'status >= 400' print request status http_referer

print the request, user agent and remote ip for realtime checks

Code (Text):

service nginx reload
ngxtop -l /home/nginx/domains/yourdomain.com/log/access.log print request http_user_agent remote_addr

To check all access logs including logrotated gzipped logs you would need to change the defined access log to this pattern to include both rotated gzipped logs and non-gzip access logs using cat and zcat. In below examples change domain=yourdomain.com to your domain name.

i.e. print top 10 requests where status code = 200

Code (Text):

service nginx reload
domain=yourdomain.com
zcat -f /home/nginx/domains/$domain/log/access.log* | ngxtop --no-follow top request -i 'status == 200' -n10

i.e. print top 10 requests where status code = 200 for April 2018 only (Apr/2018)

Code (Text):

service nginx reload
domain=yourdomain.com
zcat -f /home/nginx/domains/$domain/log/access.log* | grep 'Apr/2018' | ngxtop --no-follow top request -i 'status == 200' -n10

i.e. print only today's top 10 where status code = 200 using grep filter on grep "$(date +"%d/%b/%Y")" i.e. 06/Apr/2018

Code (Text):

service nginx reload
domain=yourdomain.com
zcat -f /home/nginx/domains/$domain/log/access.log* | grep "$(date +"%d/%b/%Y")" | ngxtop --no-follow top request -i 'status == 200' -n10

i.e. print top 10 requests where status code = 444 for April 2018 only (Apr/2018)

Code (Text):

service nginx reload
domain=yourdomain.com
zcat -f /home/nginx/domains/$domain/log/access.log* | grep 'Apr/2018' | ngxtop --no-follow top request -i 'status == 444' -n10

i.e. print top 10 requests where status code = 503 for April 2018 only (Apr/2018)

Code (Text):

service nginx reload
domain=yourdomain.com
zcat -f /home/nginx/domains/$domain/log/access.log* | grep 'Apr/2018' | ngxtop --no-follow top request -i 'status == 503' -n10

i.e. print top 10 requests where status code = 500 for April 2018 only (Apr/2018)

Code (Text):

service nginx reload
domain=yourdomain.com
zcat -f /home/nginx/domains/$domain/log/access.log* | grep 'Apr/2018' | ngxtop --no-follow top request -i 'status == 500' -n10

i.e. print top 10 user agent where status code = 200

Code (Text):

service nginx reload
domain=yourdomain.com
zcat -f /home/nginx/domains/$domain/log/access.log* | ngxtop --no-follow top http_user_agent -i 'status == 200' -n10

---

i.e. print top 10 user agent where status code = 200 for April 2018 only (Apr/2018)

Code (Text):

service nginx reload
domain=yourdomain.com
zcat -f /home/nginx/domains/$domain/log/access.log* | grep 'Apr/2018' | ngxtop --no-follow top http_user_agent -i 'status == 200' -n10

i.e. print top 10 user agent where status code = 444 for April 2018 only (Apr/2018)

Code (Text):

service nginx reload
domain=yourdomain.com
zcat -f /home/nginx/domains/$domain/log/access.log* | grep 'Apr/2018' | ngxtop --no-follow top http_user_agent -i 'status == 444' -n10

i.e. print top 10 user agent where status code = 503 for April 2018 only (Apr/2018)

Code (Text):

service nginx reload
domain=yourdomain.com
zcat -f /home/nginx/domains/$domain/log/access.log* | grep 'Apr/2018' | ngxtop --no-follow top http_user_agent -i 'status == 503' -n10

i.e. print top 10 user agent where status code = 500 for April 2018 only (Apr/2018)

Code (Text):

service nginx reload
domain=yourdomain.com
zcat -f /home/nginx/domains/$domain/log/access.log* | grep 'Apr/2018' | ngxtop --no-follow top http_user_agent -i 'status == 500' -n10

i.e. print top 10 HTTP status codes

Code (Text):

service nginx reload
domain=yourdomain.com
zcat -f /home/nginx/domains/$domain/log/access.log* | ngxtop --no-follow top status -n10

i.e. print top 10 HTTP status codes for April 2018 only (Apr/2018)

Code (Text):

service nginx reload
domain=yourdomain.com
zcat -f /home/nginx/domains/$domain/log/access.log* | grep 'Apr/2018' | ngxtop --no-follow top status -n10

Average body bytes sent of 200 responses of requested path begin with '/wp-content'

Code (Text):

service nginx reload
domain=yourdomain.com
zcat -f /home/nginx/domains/$domain/log/access.log* | ngxtop --no-follow avg bytes_sent --filter 'status == 200 and request_path.startswith("/wp-content")'

example

Code (Text):

service nginx reload
domain=yourdomain.com
zcat -f /home/nginx/domains/$domain/log/access.log* | ngxtop --no-follow avg bytes_sent --filter 'status == 200 and request_path.startswith("/wp-content")'
running for 1 seconds, 439 records processed: 757.45 req/sec

average ['bytes_sent']
|   avg(bytes_sent) |
|-------------------|
|         60281.207 |

Average body bytes sent of 200 responses of requested path begin with '/wp-content' for April 2018 only

Code (Text):

service nginx reload
domain=yourdomain.com
zcat -f /home/nginx/domains/$domain/log/access.log* | grep 'Apr/2018' | ngxtop --no-follow avg bytes_sent --filter 'status == 200 and request_path.startswith("/wp-content")'

example

Code (Text):

service nginx reload
domain=yourdomain.com
zcat -f /home/nginx/domains/$domain/log/access.log* | grep 'Apr/2018' | ngxtop --no-follow avg bytes_sent --filter 'status == 200 and request_path.startswith("/wp-content")'
running for 0 seconds, 176 records processed: 872.65 req/sec

average ['bytes_sent']
|   avg(bytes_sent) |
|-------------------|
|         52312.716 |

Top 10 requests with highest total bytes sent

Code (Text):

service nginx reload
domain=yourdomain.com
zcat -f /home/nginx/domains/$domain/log/access.log* | ngxtop --no-follow --order-by 'avg(bytes_sent) * count'

Top 10 requests with highest total bytes sent for April 2018 only

Code (Text):

service nginx reload
domain=yourdomain.com
zcat -f /home/nginx/domains/$domain/log/access.log* | grep 'Apr/2018' | ngxtop --no-follow --order-by 'avg(bytes_sent) * count'

 

command you used is for live stats, as mentioned above

 

is really live as in same moment.. so if little traffic, use the --no-follow option

 

you installed python 2.7 addon ? as it seems ngxtop doesn't work with python 2.7 yet https://github.com/lebinh/ngxtop/issues/17

 

sweet nice work

 

yeah expected if your forum has an active number of externally referenced images by members

 

Added another example command to first post

print the request, user agent and remote ip for realtime checks

Code:

ngxtop -l /home/nginx/domains/yourdomain.com/log/access.log print request http_user_agent remote_addr

 

nope command line only I believe

 

might want to check/ask at Issues · lebinh/ngxtop · GitHub and see if the issue has come up before

also updated install info for CentOS7 i don't think sqlite-devel is correct

 

yup change the access.log buffer and flush time for your vhost