Featured Nginx ngxtop real time metrics for Nginx

negative · Dec 13, 2015

Also, i want to check / filter specific file in request_path. Is it possible?

For example, when i run the
Code:
ngxtop -l /home/nginx/domains/domain.com/log/access.log
Command Line shows these output line by line
request_path | count | avg_bytes_sent | 2xx | 3xx | 4xx | 5xx |

I want to show only if request_path like '/search'

Are there any parameter or can i use any wildcard like %search% ? Thanks @eva2000

eva2000 · Dec 13, 2015

might want to ask at Issues · lebinh/ngxtop · GitHub

eva2000 · Mar 1, 2016

Posting more examples of ngxtop usage Blocking bad or aggressive bots | Centmin Mod Community

Example siege benchmark with 'GetWeb' user agent to test my own forum's new bot blocking and rate limiting config

444 count was 30 entries

Code (Text):

cat /home/nginx/domains/community.centminmod.com/log/access.log | grep 'GetWeb' | ngxtop --no-follow
running for 0 seconds, 30 records processed: 26318.58 req/sec

Summary:
|   count |   avg_bytes_sent |   2xx |   3xx |   4xx |   5xx |
|---------+------------------+-------+-------+-------+-------|
|      30 |            0.000 |     0 |     0 |    30 |     0 |

Detailed:
| request_path   |   count |   avg_bytes_sent |   2xx |   3xx |   4xx |   5xx |
|----------------+---------+------------------+-------+-------+-------+-------|
| /              |      30 |            0.000 |     0 |     0 |    30 |     0 |

group by remote_addr acount

Code (Text):

cat /home/nginx/domains/community.centminmod.com/log/access.log | grep 'GetWeb' | ngxtop --no-follow --group-by remote_addr
running for 0 seconds, 30 records processed: 33078.11 req/sec

Summary:
|   count |   avg_bytes_sent |   2xx |   3xx |   4xx |   5xx |
|---------+------------------+-------+-------+-------+-------|
|      30 |            0.000 |     0 |     0 |    30 |     0 |

Detailed:
| remote_addr     |   count |   avg_bytes_sent |   2xx |   3xx |   4xx |   5xx |
|-----------------+---------+------------------+-------+-------+-------+-------|
| IPADDR |      30 |            0.000 |     0 |     0 |    30 |     0 |

group by user agent without filtering on GetWeb but all user agenst

Code (Text):

cat /home/nginx/domains/community.centminmod.com/log/access.log | ngxtop --no-follow --group-by http_user_agent
running for 41 seconds, 557981 records processed: 13740.10 req/sec

Summary:
|   count |   avg_bytes_sent |    2xx |   3xx |    4xx |   5xx |
|---------+------------------+--------+-------+--------+-------|
|  557981 |        19773.468 | 275904 | 30752 | 251250 |    68 |

Detailed:
| http_user_agent                                                                                                                |   count |   avg_bytes_sent |   2xx |   3xx |    4xx |   5xx |
|--------------------------------------------------------------------------------------------------------------------------------+---------+------------------+-------+-------+--------+-------|
| Amazon Route 53 Health Check Service; ref:2a00cab0-xxxx-4e4d-bb32-30d1d58xxxxx; report http://amzn.to/1vsZADi                  |  243810 |          176.998 |     0 |     0 | 243807 |     0 |
| Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Q312461; SV1; .NET CLR 1.0.3705; .NET CLR 1.1.4322)                         |   99117 |        77823.952 | 98846 |     0 |    264 |     7 |
| NewRelicPinger/1.0 (652248)                                                                                                    |   16034 |         9245.900 | 16032 |     0 |      2 |     0 |
| Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)                                                       |   15758 |        11262.793 | 10130 |  5388 |    239 |     1 |

Filter for just Feb 29th access.log via grep for all http_user_agents

Code (Text):

cat /home/nginx/domains/community.centminmod.com/log/access.log | grep '29/Feb' | ngxtop --no-follow --group-by http_user_agent
running for 8 seconds, 105938 records processed: 13751.72 req/sec

Summary:
|   count |   avg_bytes_sent |   2xx |   3xx |   4xx |   5xx |
|---------+------------------+-------+-------+-------+-------|
|  105938 |        22324.371 | 52923 |  5451 | 47563 |     0 |

Detailed:
| http_user_agent                                                                                                                |   count |   avg_bytes_sent |   2xx |   3xx |   4xx |   5xx |
|--------------------------------------------------------------------------------------------------------------------------------+---------+------------------+-------+-------+-------+-------|
| Amazon Route 53 Health Check Service; ref:2a00cab0-xxxx-4e4d-bb32-30d1d58xxxxx; report http://amzn.to/1vsZADi                  |   45075 |          176.996 |     0 |     0 | 45074 |     0 |
| Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Q312461; SV1; .NET CLR 1.0.3705; .NET CLR 1.1.4322)                         |   22568 |        78140.910 | 22568 |     0 |     0 |     0 |
| NewRelicPinger/1.0 (652248)                                                                                                    |    2942 |         7230.017 |  2942 |     0 |     0 |     0 |

Filter for Feb 29th access.log via grep for 444 status codes and print request path, status code and http_user_agent logged - you can see how well the above bot rate limiting and blocking is doing

Code (Text):

cat /home/nginx/domains/community.centminmod.com/log/access.log | grep '29/Feb' | ngxtop --no-follow -i 'status == 444' print request status http_user_agent
running for 8 seconds, 670 records processed: 81.26 req/sec

request, status, http_user_agent:
| request                                                                                                            |   status | http_user_agent                                                                         |
|--------------------------------------------------------------------------------------------------------------------+----------+-----------------------------------------------------------------------------------------|
| GET / HTTP/1.1                                                                                                     |      444 | GetWeb                                                                                  |
| GET / HTTP/1.1                                                                                                     |      444 | Mozilla/5.0 (compatible; GrapeshotCrawler/2.0; +http://www.grapeshot.co.uk/crawler.php) |
| GET / HTTP/1.1                                                                                                     |      444 | Mozilla/5.0 (compatible; linkdexbot/2.0; +http://www.linkdex.com/bots/)                 |
| GET / HTTP/1.1                                                                                                     |      444 | Mozilla/5.0 (compatible; proximic; +http://www.proximic.com/info/spider.php)            |
| GET / HTTP/1.1                                                                                                     |      444 | libwww-perl/5.833                                                                       |
| GET /account/alerts HTTP/1.1                                                                                       |      444 | Mozilla/5.0 (compatible; proximic; +http://www.proximic.com/info/spider.php)            |
| GET /categories/centmin-mod.7/ HTTP/1.1                                                                            |      444 | Mozilla/5.0 (compatible; proximic; +http://www.proximic.com/info/spider.php)            |
| GET /find-new/300035/posts HTTP/1.1                                                                                |      444 | Mozilla/5.0 (compatible; proximic; +http://www.proximic.com/info/spider.php)            |
| GET /find-new/300043/posts HTTP/1.1                                                                                |      444 | Mozilla/5.0 (compatible; proximic; +http://www.proximic.com/info/spider.php)            |
| GET /find-new/300145/posts HTTP/1.1                                                                                |      444 | Mozilla/5.0 (compatible; proximic; +http://www.proximic.com/info/spider.php)            |
| GET /forums/beta-release-code.9 HTTP/1.1                                                                           |      444 | Mozilla/5.0 (compatible; proximic; +http://www.proximic.com/info/spider.php)            |
| GET /forums/beta-release-code.9/ HTTP/1.1                                                                          |      444 | Mozilla/5.0 (compatible; proximic; +http://www.proximic.com/info/spider.php)            |
| GET /forums/bug-reports.12/?prefix_id=29&order=post_date HTTP/1.0                                                  |      444 | Mozilla/5.0 (compatible; Qwantify/2.2w; +https://www.qwant.com/)/*

For March 1 check out 444 status codes

Code (Text):

cat /home/nginx/domains/community.centminmod.com/log/access.log | grep '01/Mar' | ngxtop --no-follow -i 'status == 444' print request status http_user_agent
running for 0 seconds, 17 records processed: 46.12 req/sec

request, status, http_user_agent:
| request                                                     |   status | http_user_agent                                                              |
|-------------------------------------------------------------+----------+------------------------------------------------------------------------------|
| GET / HTTP/1.1                                              |      444 | libwww-perl/5.833                                                            |
| GET /threads/nginx-failed-to-install-upgrade.1798/ HTTP/1.1 |      444 | Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)      |
| GET /threads/php-7-0-3-is-available.6001/ HTTP/1.1          |      444 | Mozilla/5.0 (compatible; proximic; +http://www.proximic.com/info/spider.php) |

Check only Feb 29 for Googlebot via grep of access.log and pipe through to ngxtop and print http_user_agent and status codes

Code (Text):

cat /home/nginx/domains/community.centminmod.com/log/access.log | grep '29/Feb' | grep 'Googlebot' | ngxtop --no-follow print http_user_agent status    
running for 0 seconds, 2598 records processed: 15727.07 req/sec

http_user_agent, status:
| http_user_agent                                                                                                                                                                                    |   status |
|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------|
| DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)                                                                                               |      200 |
| Googlebot-Image/1.0                                                                                                                                                                                |      200 |
| Googlebot-Image/1.0                                                                                                                                                                                |      304 |
| Googlebot-Image/1.0                                                                                                                                                                                |      404 |
| Googlebot/2.1; +http://www.google.com/bot.html)                                                                                                                                                    |      200 |
| Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)                                                                                                                           |      200 |
| Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)                                                                                                                           |      301 |
| Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)                                                                                                                           |      303 |
| Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)                                                                                                                           |      304 |
| Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)                                                                                                                           |      307 |
| Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)                                                                                                                           |      403 |
| Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)                                                                                                                           |      404 |
| Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12F70 Safari/600.1.4 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) |      200 |
| Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12F70 Safari/600.1.4 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) |      301 |
| SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)                          |      200 |

Same filter as before but switch out status for remote_addr for Googlebot count for Feb 29

Code (Text):

cat /home/nginx/domains/community.centminmod.com/log/access.log | grep '29/Feb' | grep 'Googlebot' | ngxtop --no-follow print http_user_agent remote_addr
running for 0 seconds, 2598 records processed: 17187.30 req/sec

http_user_agent, remote_addr:
| http_user_agent                                                                                                                                                                                    | remote_addr    |
|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------|
| DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)                                                                                               | 66.249.79.61   |
| DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)                                                                                               | 66.249.79.63   |
| DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)                                                                                               | 66.249.79.65   |
| Googlebot-Image/1.0                                                                                                                                                                                | 192.240.110.42 |
| Googlebot-Image/1.0                                                                                                                                                                                | 209.58.130.199 |
| Googlebot-Image/1.0                                                                                                                                                                                | 66.249.79.61   |
| Googlebot-Image/1.0                                                                                                                                                                                | 66.249.79.63   |
| Googlebot-Image/1.0                                                                                                                                                                                | 66.249.79.65   |
| Googlebot/2.1; +http://www.google.com/bot.html)                                                                                                                                                    | 162.216.19.183 |
| Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)                                                                                                                           | 127.0.0.1      |
| Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)                                                                                                                           | 192.240.110.42 |
| Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)                                                                                                                           | 209.58.130.199 |
| Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)                                                                                                                           | 66.249.64.125  |
| Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)                                                                                                                           | 66.249.64.2    |
| Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)                                                                                                                           | 66.249.66.179  |
| Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)                                                                                                                           | 66.249.66.182  |
| Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)                                                                                                                           | 66.249.66.185  |
| Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)                                                                                                                           | 66.249.66.44   |
| Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)                                                                                                                           | 66.249.79.223  |
| Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)                                                                                                                           | 66.249.79.230  |
| Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)                                                                                                                           | 66.249.79.237  |
| Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)                                                                                                                           | 66.249.79.61   |
| Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)                                                                                                                           | 66.249.79.63   |
| Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)                                                                                                                           | 66.249.79.65   |
| Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12F70 Safari/600.1.4 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) | 66.249.66.179  |
| Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12F70 Safari/600.1.4 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) | 66.249.66.185  |
| Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12F70 Safari/600.1.4 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) | 66.249.66.47   |
| Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12F70 Safari/600.1.4 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) | 66.249.79.61   |
| Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12F70 Safari/600.1.4 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) | 66.249.79.63   |
| Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12F70 Safari/600.1.4 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) | 66.249.79.65   |
| SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)                          | 66.249.79.61   |

Building on previous command to group by remote_addr for filtered Googlebot on Feb 29th

Code (Text):

cat /home/nginx/domains/community.centminmod.com/log/access.log | grep '29/Feb' | grep 'Googlebot' | ngxtop --no-follow --group-by remote_addr
running for 0 seconds, 2598 records processed: 16824.80 req/sec

Summary:
|   count |   avg_bytes_sent |   2xx |   3xx |   4xx |   5xx |
|---------+------------------+-------+-------+-------+-------|
|    2598 |        12314.505 |  1670 |   881 |    47 |     0 |

Detailed:
| remote_addr    |   count |   avg_bytes_sent |   2xx |   3xx |   4xx |   5xx |
|----------------+---------+------------------+-------+-------+-------+-------|
| 66.249.79.61   |    1231 |        14078.539 |   852 |   363 |    16 |     0 |
| 66.249.79.223  |     525 |         8911.514 |   292 |   232 |     1 |     0 |
| 66.249.79.63   |     311 |        13264.614 |   208 |    95 |     8 |     0 |
| 66.249.66.179  |     230 |         8986.509 |   119 |   102 |     9 |     0 |
| 66.249.79.65   |     191 |        14356.131 |   139 |    50 |     2 |     0 |
| 66.249.79.230  |      34 |         8199.324 |    18 |    16 |     0 |     0 |
| 66.249.66.182  |      29 |         9298.276 |    16 |     9 |     4 |     0 |
| 66.249.66.185  |      19 |         5710.632 |     8 |     6 |     5 |     0 |
| 66.249.79.237  |      16 |         7718.500 |     9 |     7 |     0 |     0 |
| 209.58.130.199 |       3 |          795.000 |     1 |     0 |     2 |     0 |

Filter for Feb 29th only on several bots you want to check for via egrep case insensitive and group count by http_user_agent. Googlebot, Baidu, bingbot, ahrefsbot, yandex and msnbot

Code (Text):

cat /home/nginx/domains/community.centminmod.com/log/access.log | grep '29/Feb' | egrep -i 'Googlebot|Baidu|bingbot|Ahrefsbot|yandex|msnbot' | ngxtop --no-follow --group-by http_user_agent
running for 1 seconds, 8609 records processed: 16183.12 req/sec

Summary:
|   count |   avg_bytes_sent |   2xx |   3xx |   4xx |   5xx |
|---------+------------------+-------+-------+-------+-------|
|    8609 |        11377.074 |  5210 |  3150 |   249 |     0 |

Detailed:
| http_user_agent                                                                                                                                                                                    |   count |   avg_bytes_sent |   2xx |   3xx |   4xx |   5xx |
|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+------------------+-------+-------+-------+-------|
| Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)                                                                                                                            |    2496 |        10524.571 |  1668 |   770 |    58 |     0 |
| Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)                                                                                                                           |    2430 |        11587.994 |  1524 |   860 |    46 |     0 |
| Mozilla/5.0 (compatible; AhrefsBot/5.0; +http://ahrefs.com/robot/)                                                                                                                                 |    1180 |         5371.688 |   339 |   743 |    98 |     0 |
| Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)                                                                                                                |    1117 |        15648.950 |   695 |   379 |    43 |     0 |
| Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)                                                                                                                                   |    1053 |        11357.065 |   721 |   329 |     3 |     0 |
| Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12F70 Safari/600.1.4 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) |     122 |        27010.811 |   121 |     1 |     0 |     0 |
| Mozilla/5.0 (compatible; YandexImages/3.0; +http://yandex.com/bots)                                                                                                                                |      91 |        27978.187 |    47 |    44 |     0 |     0 |
| Googlebot-Image/1.0                                                                                                                                                                                |      33 |         5185.970 |    12 |    20 |     1 |     0 |
| Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.116 Safari/537.36                                                                          |      14 |         4820.214 |    14 |     0 |     0 |     0 |
| Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.116 Safari/537.36                                                                                     |       9 |         7903.889 |     9 |     0 |     0 |     0 |

BamaStangGuy · Mar 20, 2018

Is this still working? None of my sites are showing information. Has something changed with the log files that I need to adjust for? ChristianForums.com is no longer reporting any information from ngxtop.

Access Log setting: access_log /home/nginx/domains/christianforums.com/log/access.log main_ext buffer=256k flush=60m;

eva2000 · Mar 20, 2018

using proper --no-follow flag as per 1st post Nginx - ngxtop real time metrics for Nginx ? ngxtop without that flag displays live stats but centmin mod buffers log writes for performance so you won't see live ngxtop stats for the duration of the buffer paramters - in your case buffer needs to fill be 256kb first or flushed to disk logs every 60 mins

example

Code (Text):

ngxtop -l /home/nginx/domains/magento.domain.com/log/access.log --no-follow
running for 0 seconds, 270 records processed: 6154.38 req/sec

Summary:
|   count |   avg_bytes_sent |   2xx |   3xx |   4xx |   5xx |
|---------+------------------+-------+-------+-------+-------|
|     270 |         5097.341 |   269 |     1 |     0 |     0 |

Detailed:
| request_path                                                                                              |   count |   avg_bytes_sent |   2xx |   3xx |   4xx |   5xx |
|-----------------------------------------------------------------------------------------------------------+---------+------------------+-------+-------+-------+-------|
| /customer/section/load/                                                                                   |       2 |         9207.000 |     2 |     0 |     0 |     0 |
| /                                                                                                         |       1 |         6746.000 |     1 |     0 |     0 |     0 |
| /admin_auIkmz4lFi/admin/index/index/key/1207d0682c84f45d9eb6fea9dae0d209e31ff4767dc496bd3b43ecf4972b97b3/ |       1 |            0.000 |     0 |     1 |     0 |     0 |
| /admin_auIkmz4lFi/admin/index/index/key/d370b44501ca1365a5790a583737e800ef6c61c53eecac181d93ffba6e4572e3/ |       1 |         2357.000 |     1 |     0 |     0 |     0 |
| /static/version1521122361/_cache/merged/9de8797d0c615276fc794278a0af8df2.css                              |       1 |        43735.000 |     1 |     0 |     0 |     0 |
| /static/version1521122361/_cache/merged/c0b5661e7cfc4d3e29c8fc8495e4d56e.js                               |       1 |        25048.000 |     1 |     0 |     0 |     0 |
| /static/version1521122361/_cache/merged/ddce5b0a41e6dbae4b6f458053932955.js                               |       1 |        24335.000 |     1 |     0 |     0 |     0 |
| /static/version1521122361/_cache/merged/e6a662994948b56fa4f39e1f141aaed3.css                              |       1 |        75792.000 |     1 |     0 |     0 |     0 |
| /static/version1521122361/adminhtml/Magento/backend/en_US/FormData.js                                     |       1 |         1661.000 |     1 |     0 |     0 |     0 |
| /static/version1521122361/adminhtml/Magento/backend/en_US/Magento_Catalog/catalog/product.js              |       1 |          924.000 |     1 |     0 |     0 |     0 |

BamaStangGuy · Apr 4, 2018

eva2000 said: ↑

using proper --no-follow flag as per 1st post Nginx - ngxtop real time metrics for Nginx ? ngxtop without that flag displays live stats but centmin mod buffers log writes for performance so you won't see live ngxtop stats for the duration of the buffer paramters - in your case buffer needs to fill be 256kb first or flushed to disk logs every 60 mins

example

Code (Text):

ngxtop -l /home/nginx/domains/magento.domain.com/log/access.log --no-follow
running for 0 seconds, 270 records processed: 6154.38 req/sec

Summary:
|   count |   avg_bytes_sent |   2xx |   3xx |   4xx |   5xx |
|---------+------------------+-------+-------+-------+-------|
|     270 |         5097.341 |   269 |     1 |     0 |     0 |

Detailed:
| request_path                                                                                              |   count |   avg_bytes_sent |   2xx |   3xx |   4xx |   5xx |
|-----------------------------------------------------------------------------------------------------------+---------+------------------+-------+-------+-------+-------|
| /customer/section/load/                                                                                   |       2 |         9207.000 |     2 |     0 |     0 |     0 |
| /                                                                                                         |       1 |         6746.000 |     1 |     0 |     0 |     0 |
| /admin_auIkmz4lFi/admin/index/index/key/1207d0682c84f45d9eb6fea9dae0d209e31ff4767dc496bd3b43ecf4972b97b3/ |       1 |            0.000 |     0 |     1 |     0 |     0 |
| /admin_auIkmz4lFi/admin/index/index/key/d370b44501ca1365a5790a583737e800ef6c61c53eecac181d93ffba6e4572e3/ |       1 |         2357.000 |     1 |     0 |     0 |     0 |
| /static/version1521122361/_cache/merged/9de8797d0c615276fc794278a0af8df2.css                              |       1 |        43735.000 |     1 |     0 |     0 |     0 |
| /static/version1521122361/_cache/merged/c0b5661e7cfc4d3e29c8fc8495e4d56e.js                               |       1 |        25048.000 |     1 |     0 |     0 |     0 |
| /static/version1521122361/_cache/merged/ddce5b0a41e6dbae4b6f458053932955.js                               |       1 |        24335.000 |     1 |     0 |     0 |     0 |
| /static/version1521122361/_cache/merged/e6a662994948b56fa4f39e1f141aaed3.css                              |       1 |        75792.000 |     1 |     0 |     0 |     0 |
| /static/version1521122361/adminhtml/Magento/backend/en_US/FormData.js                                     |       1 |         1661.000 |     1 |     0 |     0 |     0 |
| /static/version1521122361/adminhtml/Magento/backend/en_US/Magento_Catalog/catalog/product.js              |       1 |          924.000 |     1 |     0 |     0 |     0 |

Click to expand...

Coming back to this:

This returns results:

Code:

grep ' 444 ' /home/nginx/domains/christianforums.com/log/access.log

This does not:

Code:

grep ' 444 ' /home/nginx/domains/christianforums.com/log/access.log | ngxtop --no-follow

ngxtop is simply not working at all for me but I can grep 444 and it return every instance.

BamaStangGuy · Apr 4, 2018

I have just decided to go with goaccess.

eva2000 · Apr 4, 2018

strange then make sure nginx logrotation didn't just happen as then the current access log would of been rotated and new log might not have entries

Code (Text):

ls -lah /home/nginx/domains/domain1.com/log/access.log*
-rw-r--r-- 1 nginx nginx    0 Apr  3 05:13 /home/nginx/domains/domain1.com/log/access.log
-rw-r--r-- 1 nginx nginx 512K Apr  3 04:40 /home/nginx/domains/domain1.com/log/access.log-20180403

against rotated log with missing 444 entry

Code (Text):

grep ' 444 ' /home/nginx/domains/domain1.com/log/access.log | ngxtop --no-follow
running for 0 seconds, 0 records processed: 0.00 req/sec

Summary:
|   count | avg_bytes_sent   |   2xx |   3xx |   4xx |   5xx |
|---------+------------------+-------+-------+-------+-------|
|       0 |                  |     0 |     0 |     0 |     0 |

Detailed:
| request_path   | count   | avg_bytes_sent   | 2xx   | 3xx   | 4xx   | 5xx   |
|----------------+---------+------------------+-------+-------+-------+-------|

change access.log* wildcard entry to look through all logs included rotated uncompressed one

Code (Text):

grep ' 444 ' /home/nginx/domains/domain1.com/log/access.log* | ngxtop --no-follow
running for 0 seconds, 1 records processed: 1101.45 req/sec

Summary:
|   count |   avg_bytes_sent |   2xx |   3xx |   4xx |   5xx |
|---------+------------------+-------+-------+-------+-------|
|       1 |            0.000 |     0 |     0 |     1 |     0 |

Detailed:
| request_path   |   count |   avg_bytes_sent |   2xx |   3xx |   4xx |   5xx |
|----------------+---------+------------------+-------+-------+-------+-------|
| /              |       1 |            0.000 |     0 |     0 |     1 |     0 |

If you had compressed .log.gz logged you can use zgrep instead of grep

eva2000 · Apr 4, 2018

Also @BamaStangGuy try native ngxtop status method
Code (Text):
cat /home/nginx/domains/domain1.com/log/access.log | ngxtop --no-follow -i 'status == 444'

eva2000 · Apr 6, 2018

Updated 1st post with more examples and info about buffered access logs

To check all access logs including logrotated gzipped logs you would need to change the defined access log to this pattern to include both rotated gzipped logs and non-gzip access logs using zcat. In below examples change domain=yourdomain.com to your domain name.

i.e. print top 10 requests where status code = 200
Code (Text):
service nginx reload
domain=yourdomain.com
zcat -f /home/nginx/domains/$domain/log/access.log* | ngxtop --no-follow top request -i 'status == 200' -n10
i.e. print top 10 requests where status code = 200 for April 2018 only (Apr/2018)
Code (Text):
service nginx reload
domain=yourdomain.com
zcat -f /home/nginx/domains/$domain/log/access.log* | grep 'Apr/2018' | ngxtop --no-follow top request -i 'status == 200' -n10
i.e. print only today's top 10 where status code = 200 using grep filter on grep "$(date +"%d/%b/%Y")" i.e. 06/Apr/2018
Code (Text):
service nginx reload
domain=yourdomain.com
zcat -f /home/nginx/domains/$domain/log/access.log* | grep "$(date +"%d/%b/%Y")" | ngxtop --no-follow top request -i 'status == 200' -n10
i.e. print top 10 requests where status code = 444 for April 2018 only (Apr/2018)
Code (Text):
service nginx reload
domain=yourdomain.com
zcat -f /home/nginx/domains/$domain/log/access.log* | grep 'Apr/2018' | ngxtop --no-follow top request -i 'status == 444' -n10
i.e. print top 10 requests where status code = 503 for April 2018 only (Apr/2018)
Code (Text):
service nginx reload
domain=yourdomain.com
zcat -f /home/nginx/domains/$domain/log/access.log* | grep 'Apr/2018' | ngxtop --no-follow top request -i 'status == 503' -n10
i.e. print top 10 requests where status code = 500 for April 2018 only (Apr/2018)
Code (Text):
service nginx reload
domain=yourdomain.com
zcat -f /home/nginx/domains/$domain/log/access.log* | grep 'Apr/2018' | ngxtop --no-follow top request -i 'status == 500' -n10
i.e. print top 10 user agent where status code = 200
Code (Text):
service nginx reload
domain=yourdomain.com
zcat -f /home/nginx/domains/$domain/log/access.log* | ngxtop --no-follow top http_user_agent -i 'status == 200' -n10
i.e. print top 10 user agent where status code = 200 for April 2018 only (Apr/2018)
Code (Text):
service nginx reload
domain=yourdomain.com
zcat -f /home/nginx/domains/$domain/log/access.log* | grep 'Apr/2018' | ngxtop --no-follow top http_user_agent -i 'status == 200' -n10
i.e. print top 10 user agent where status code = 444 for April 2018 only (Apr/2018)
Code (Text):
service nginx reload
domain=yourdomain.com
zcat -f /home/nginx/domains/$domain/log/access.log* | grep 'Apr/2018' | ngxtop --no-follow top http_user_agent -i 'status == 444' -n10
i.e. print top 10 user agent where status code = 503 for April 2018 only (Apr/2018)
Code (Text):
service nginx reload
domain=yourdomain.com
zcat -f /home/nginx/domains/$domain/log/access.log* | grep 'Apr/2018' | ngxtop --no-follow top http_user_agent -i 'status == 503' -n10
i.e. print top 10 user agent where status code = 500 for April 2018 only (Apr/2018)
Code (Text):
service nginx reload
domain=yourdomain.com
zcat -f /home/nginx/domains/$domain/log/access.log* | grep 'Apr/2018' | ngxtop --no-follow top http_user_agent -i 'status == 500' -n10
i.e. print top 10 HTTP status codes
Code (Text):
service nginx reload
domain=yourdomain.com
zcat -f /home/nginx/domains/$domain/log/access.log* | ngxtop --no-follow top status -n10
i.e. print top 10 HTTP status codes for April 2018 only (Apr/2018)
Code (Text):
service nginx reload
domain=yourdomain.com
zcat -f /home/nginx/domains/$domain/log/access.log* | grep 'Apr/2018' | ngxtop --no-follow top status -n10
Average body bytes sent of 200 responses of requested path begin with '/wp-content'
Code (Text):
service nginx reload
domain=yourdomain.com
zcat -f /home/nginx/domains/$domain/log/access.log* | ngxtop --no-follow avg bytes_sent --filter 'status == 200 and request_path.startswith("/wp-content")'
example
Code (Text):
service nginx reload
domain=yourdomain.com
zcat -f /home/nginx/domains/$domain/log/access.log* | ngxtop --no-follow avg bytes_sent --filter 'status == 200 and request_path.startswith("/wp-content")'
running for 1 seconds, 439 records processed: 757.45 req/sec

average ['bytes_sent']
|   avg(bytes_sent) |
|-------------------|
|         60281.207 |
Average body bytes sent of 200 responses of requested path begin with '/wp-content' for April 2018 only
Code (Text):
service nginx reload
domain=yourdomain.com
zcat -f /home/nginx/domains/$domain/log/access.log* | grep 'Apr/2018' | ngxtop --no-follow avg bytes_sent --filter 'status == 200 and request_path.startswith("/wp-content")'
example
Code (Text):
service nginx reload
domain=yourdomain.com
zcat -f /home/nginx/domains/$domain/log/access.log* | grep 'Apr/2018' | ngxtop --no-follow avg bytes_sent --filter 'status == 200 and request_path.startswith("/wp-content")'
running for 0 seconds, 176 records processed: 872.65 req/sec

average ['bytes_sent']
|   avg(bytes_sent) |
|-------------------|
|         52312.716 |
Top 10 requests with highest total bytes sent
Code (Text):
service nginx reload
domain=yourdomain.com
zcat -f /home/nginx/domains/$domain/log/access.log* | ngxtop --no-follow --order-by 'avg(bytes_sent) * count'
Top 10 requests with highest total bytes sent for April 2018 only
Code (Text):
service nginx reload
domain=yourdomain.com
zcat -f /home/nginx/domains/$domain/log/access.log* | grep 'Apr/2018' | ngxtop --no-follow --order-by 'avg(bytes_sent) * count'

eva2000 · Apr 6, 2018

Example checking all 503 http status codes for all access logs where centmin.sh menu option 22 auto installed wordpress installs by default have nginx rate limiting on wp-login, xmlrpc.php requests which will return 503 http status code if requests hit nginx rate limit threshold.

Code (Text):

service nginx reload
domain=yourdomain.com
zcat -f /home/nginx/domains/$domain/log/access.log* | ngxtop --no-follow top request -i 'status == 503' -n10
running for 36 seconds, 71095 records processed: 1986.74 req/sec

top request
| request                                                      |   count |
|--------------------------------------------------------------+---------|
| POST /wp-login.php HTTP/1.0                                  |   70205 |
| POST /wp-login.php HTTP/1.1                                  |     849 |
| GET /xmlrpc.php HTTP/1.1                                     |      25 |
| POST http://domain.com.com/wp-login.php/ HTTP/1.1            |      10 |
| GET /admin/wp-login.php HTTP/1.1                             |       1 |
| GET /docs/wp-login.php HTTP/1.0                              |       1 |
| GET /site/wp-login.php HTTP/1.1                              |       1 |
| GET /wp-login.php HTTP/1.1                                   |       1 |
| GET /wp/wp-login.php HTTP/1.1                                |       1 |
| POST /xmlrpc.php HTTP/1.1                                    |       1 |

filter for Apr/2018 only

Code (Text):

service nginx reload
domain=yourdomain.com
zcat -f /home/nginx/domains/$domain/log/access.log* | grep 'Apr/2018' | ngxtop --no-follow top request -i 'status == 503' -n10

filter for today only

Code (Text):

service nginx reload
domain=yourdomain.com
zcat -f /home/nginx/domains/$domain/log/access.log* | grep "$(date +"%d/%b/%Y")" | ngxtop --no-follow top request -i 'status == 503' -n10

Top http user agent strings for http 503 status codes

Code (Text):

service nginx reload
domain=yourdomain.com
zcat -f /home/nginx/domains/$domain/log/access.log* | ngxtop --no-follow top http_user_agent -i 'status == 503' -n10                                  
running for 36 seconds, 71095 records processed: 1986.32 req/sec

top http_user_agent
| http_user_agent                                                                                                           |   count |
|---------------------------------------------------------------------------------------------------------------------------+---------|
| -                                                                                                                         |   70205 |
| Mozilla/4.0 (compatible; MSIE 9.0; Windows NT 6.1; 125LA; .NET CLR 2.0.50727; .NET CLR 3.0.04506.648; .NET CLR 3.5.21022) |     730 |
| Mozilla/5.0 (Windows NT 6.1; WOW64; rv:30.0) Gecko/20100101 Firefox/30.0                                                  |     108 |
| Mozilla/5.0 (unknown-x86_64-linux-gnu) Siege/4.0.2                                                                        |      25 |
| Mozilla/5.0 (Windows; U; Windows NT 5.1; ru; rv:1.9.0.2) Gecko/2008091620 Firefox/3.0.2                                   |      10 |
| Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)                                                  |      10 |
| Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko                                                      |       4 |
| Mozilla/4.0 (compatible; Synapse)                                                                                         |       1 |
| Mozilla/5.0 (Linux; U; Android 2.2) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1                 |       1 |
| Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:50.0) Gecko/20100101 Firefox/50.0                                            |       1 |

Top http user agent strings for http 444 status codes

Code (Text):

service nginx reload
domain=yourdomain.com
zcat -f /home/nginx/domains/$domain/log/access.log* | ngxtop --no-follow top http_user_agent -i 'status == 444' -n10   
running for 35 seconds, 242 records processed: 6.99 req/sec

top http_user_agent
| http_user_agent                                                                                                                 |   count |
|---------------------------------------------------------------------------------------------------------------------------------+---------|
| Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.65 Safari/537.36 OPR/26.0.1656.24   |      46 |
| Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36 OPR/52.0.2871.40 |      37 |
| Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.111 Safari/537.36 OPR/27.0.1689.69  |      31 |
| Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.65 Safari/537.36 OPR/26.0.1656.32   |      28 |
| Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36 OPR/26.0.1656.60   |      28 |
| Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36 OPR/26.0.1656.60   |      25 |
| Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36                    |      14 |
| Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.91 Safari/537.36 OPR/27.0.1689.54   |      14 |
| NYU Internet Census (https://scan.lol; research@scan.lol)                                                                       |       7 |
| -                                                                                                                               |       3 |

eva2000 · Apr 8, 2018

whipped up a ngx-top.sh script for testing daily, monthly and previous month parsing of access log stats

ran against vanilla.domain.com test vhost for previous month's remaining logs which are left after logrotation

Code (Text):

time ./ngx-top.sh profile-month-last vanilla.domain.com

----------------------------------------------------------------------------------------
Processing Last Month Mar/2018 stats for /home/nginx/domains/vanilla.domain.com/log/access.log logs
----------------------------------------------------------------------------------------

----------------------------------------------------------------------------------------
Top Visitor IPs
----------------------------------------------------------------------------------------
/usr/bin/zcat -f /home/nginx/domains/vanilla.domain.com/log/access.log-20180330.gz /home/nginx/domains/vanilla.domain.com/log/access.log-20180331.gz | fgrep "Mar/2018" | ngxtop --no-follow --group-by remote_addr
running for 167 seconds, 8683978 records processed: 51978.92 req/sec

Summary:
|   count |   avg_bytes_sent |     2xx |   3xx |   4xx |   5xx |
|---------+------------------+---------+-------+-------+-------|
| 8683978 |         8258.521 | 8683800 |     5 |   166 |     7 |

Detailed:
| remote_addr   |   count |   avg_bytes_sent |     2xx |   3xx |   4xx |   5xx |
|---------------+---------+------------------+---------+-------+-------+-------|
| 209.xx.xx.xx  | 5837255 |         6835.846 | 5837227 |     0 |    26 |     2 |
| 127.0.0.1     |  218526 |        11173.546 |  218509 |     0 |    17 |     0 |
| 127.0.0.2     |  218220 |        11177.518 |  218220 |     0 |     0 |     0 |
| 127.0.0.4     |  206519 |        11177.557 |  206519 |     0 |     0 |     0 |
| 127.0.0.6     |  205778 |        11177.561 |  205778 |     0 |     0 |     0 |
| 127.0.0.3     |  204013 |        11177.455 |  204013 |     0 |     0 |     0 |
| 127.0.0.5     |  201698 |        11177.480 |  201698 |     0 |     0 |     0 |
| 127.0.0.7     |  198381 |        11177.338 |  198381 |     0 |     0 |     0 |
| 127.0.0.9     |  197683 |        11177.809 |  197683 |     0 |     0 |     0 |
| 127.0.0.8     |  196498 |        11177.531 |  196498 |     0 |     0 |     0 |

----------------------------------------------------------------------------------------
Top Status Codes
----------------------------------------------------------------------------------------
/usr/bin/zcat -f /home/nginx/domains/vanilla.domain.com/log/access.log-20180330.gz /home/nginx/domains/vanilla.domain.com/log/access.log-20180331.gz | fgrep "Mar/2018" | ngxtop --no-follow top status
running for 145 seconds, 8683978 records processed: 59770.65 req/sec

top status
|   status |   count |
|----------+---------|
|      200 | 8683800 |
|      404 |     128 |
|      499 |      25 |
|      401 |      12 |
|      502 |       7 |
|      302 |       5 |
|      403 |       1 |

----------------------------------------------------------------------------------------
Top 5 Requests with HTTP 200 Status Code
----------------------------------------------------------------------------------------
/usr/bin/zcat -f /home/nginx/domains/vanilla.domain.com/log/access.log-20180330.gz /home/nginx/domains/vanilla.domain.com/log/access.log-20180331.gz | fgrep "Mar/2018" | ngxtop --no-follow top request -i 'status == 200' -n5
running for 297 seconds, 8683800 records processed: 29257.94 req/sec

top request
| request                                    |   count |
|--------------------------------------------+---------|
| GET / HTTP/1.1                             | 5512568 |
| GET / HTTP/1.0                             | 3170764 |
| POST /settings/analyticstick.json HTTP/1.1 |     191 |
| POST /notifications/inform HTTP/1.1        |      85 |
| HEAD / HTTP/1.1                            |      70 |

----------------------------------------------------------------------------------------
Top 5 Requests with HTTP 444 Status Code
----------------------------------------------------------------------------------------
/usr/bin/zcat -f /home/nginx/domains/vanilla.domain.com/log/access.log-20180330.gz /home/nginx/domains/vanilla.domain.com/log/access.log-20180331.gz | fgrep "Mar/2018" | ngxtop --no-follow top request -i 'status == 444' -n5
running for 198 seconds, 0 records processed: 0.00 req/sec

top request
| request   | count   |
|-----------+---------|

----------------------------------------------------------------------------------------
Top 5 Requests with HTTP >=500 Status Code
----------------------------------------------------------------------------------------
/usr/bin/zcat -f /home/nginx/domains/vanilla.domain.com/log/access.log-20180330.gz /home/nginx/domains/vanilla.domain.com/log/access.log-20180331.gz | fgrep "Mar/2018" | ngxtop --no-follow top request -i 'status >= 500' -n5
running for 204 seconds, 7 records processed: 0.03 req/sec

top request
| request                                                   |   count |
|-----------------------------------------------------------+---------|
| GET /applications/dashboard/design/style.css.map HTTP/1.1 |       2 |
| HEAD / HTTP/1.1                                           |       2 |
| POST /settings/analyticstick.json HTTP/1.1                |       2 |
| POST /notifications/inform HTTP/1.1                       |       1 |

----------------------------------------------------------------------------------------
Top 5 HTTP User Agents with HTTP 200 Status Code
----------------------------------------------------------------------------------------
/usr/bin/zcat -f /home/nginx/domains/vanilla.domain.com/log/access.log-20180330.gz /home/nginx/domains/vanilla.domain.com/log/access.log-20180331.gz | fgrep "Mar/2018" | ngxtop --no-follow top http_user_agent -i 'status == 200' -n5
running for 290 seconds, 8683800 records processed: 29902.33 req/sec

top http_user_agent
| http_user_agent                                                                                                                 |   count |
|---------------------------------------------------------------------------------------------------------------------------------+---------|
| -                                                                                                                               | 5500965 |
| ApacheBench/2.3                                                                                                                 | 3170768 |
| Mozilla/5.0 (pc-x86_64-linux-gnu) Siege/4.0.4                                                                                   |   11459 |
| Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.162 Safari/537.36 OPR/52.0.2871.30 |     442 |
| curl/7.29.0                                                                                                                     |      83 |

----------------------------------------------------------------------------------------
Top 5 HTTP User Agents with HTTP 444 Status Code
----------------------------------------------------------------------------------------
/usr/bin/zcat -f /home/nginx/domains/vanilla.domain.com/log/access.log-20180330.gz /home/nginx/domains/vanilla.domain.com/log/access.log-20180331.gz | fgrep "Mar/2018" | ngxtop --no-follow top http_user_agent -i 'status == 444' -n5
running for 193 seconds, 0 records processed: 0.00 req/sec

top http_user_agent
| http_user_agent   | count   |
|-------------------+---------|

----------------------------------------------------------------------------------------
Top 5 HTTP User Agents with HTTP >=500 Status Code
----------------------------------------------------------------------------------------
/usr/bin/zcat -f /home/nginx/domains/vanilla.domain.com/log/access.log-20180330.gz /home/nginx/domains/vanilla.domain.com/log/access.log-20180331.gz | fgrep "Mar/2018" | ngxtop --no-follow top http_user_agent -i 'status >= 500' -n5
running for 201 seconds, 7 records processed: 0.03 req/sec

top http_user_agent
| http_user_agent                                                                                                                 |   count |
|---------------------------------------------------------------------------------------------------------------------------------+---------|
| Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.162 Safari/537.36 OPR/52.0.2871.30 |       5 |
| curl/7.29.0                                                                                                                     |       2 |

----------------------------------------------------------------------------------------

real    29m12.734s
user    29m12.590s
sys     0m24.762s

negative · Oct 25, 2018

@eva2000 I can't see any result when i type
ngxtop -l /home/nginx/domains/domain.com/log/access.log

I look the nginx.conf and there is access_log off parameter but i see the high sizes access logs under of the domain log directories? So it logs the access i think but i can't see anything on ngxtop command.

P.S : amplify nginx shows all process without any problem already.

eva2000 · Oct 25, 2018

negative said: ↑

@eva2000 I can't see any result when i type
ngxtop -l /home/nginx/domains/domain.com/log/access.log

I look the nginx.conf and there is access_log off parameter but i see the high sizes access logs under of the domain log directories? So it logs the access i think but i can't see anything on ngxtop command.

P.S : amplify nginx shows all process without any problem already.
Click to expand...

re-read first post Nginx - ngxtop real time metrics for Nginx as Centmin Mod by default buffers a certain amount of access log entries in memory (256KB) before flushing and writing to disk for performance reasons as such to check previously logged data use the --no-follow option. You can't check live ngxtop data while Centmin Mod buffers access logs, so need to disable buffering to see live ngxtop data.

negative · Oct 25, 2018

eva2000 said: ↑

re-read first post Nginx - ngxtop real time metrics for Nginx as Centmin Mod by default buffers a certain amount of access log entries in memory (256KB) before flushing and writing to disk for performance reasons as such to check previously logged data use the --no-follow option. You can't check live ngxtop data while Centmin Mod buffers access logs, so need to disable buffering to see live ngxtop data.
Click to expand...

No result even using
ngxtop -l /home/nginx/domains/domain.com/log/access.log --no-follow

Cursor is just waiting in next line. Nothing shows.

eva2000 · Oct 25, 2018

how long you waiting ? could be processing very large logs

you can use cat and grep filters to reduce size of access logs that ngxtop processes example for limiting log data to day/month at Nginx - ngxtop real time metrics for Nginx

negative · Oct 25, 2018

eva2000 said: ↑

how long you waiting ? could be processing very large logs
Click to expand...

Log size about 200MB and i wait about 30sec and cancelled with ctrl+c
Should i wait more ?

eva2000 · Oct 25, 2018

you can use cat and grep filters to reduce size of access logs that ngxtop processes example for limiting log data to day/month at Nginx - ngxtop real time metrics for Nginx

negative · Oct 25, 2018

I wait 87 sec and results are Still no output interesting.

Code (Text):

[16:45][root@server ~]# ngxtop -l /home/nginx/domains/domain.com/log/access.log --no-follow
running for 87 seconds, 0 records processed: 0.00 req/sec

Summary:
|   count | avg_bytes_sent   |   2xx |   3xx |   4xx |   5xx |
|---------+------------------+-------+-------+-------+-------|
|       0 |                  |     0 |     0 |     0 |     0 |

Detailed:
| request_path   | count   | avg_bytes_sent   | 2xx   | 3xx   | 4xx   | 5xx   |
|----------------+---------+------------------+-------+-------+-------+-------|

eva2000 · Oct 25, 2018

what's output for these commands
Code (Text):
ls -lAh /home/nginx/domains/domain.com/log/
oh if you used nginx amplify and modified log format, ngxtop won't be able to read the logs as format is different

Log in / Register

Forums

Join the community today

Featured Nginx ngxtop real time metrics for Nginx

negative Active Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

BamaStangGuy Active Member

eva2000 Administrator Staff Member

BamaStangGuy Active Member

BamaStangGuy Active Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

negative Active Member

eva2000 Administrator Staff Member

negative Active Member

eva2000 Administrator Staff Member

negative Active Member

eva2000 Administrator Staff Member

negative Active Member

eva2000 Administrator Staff Member

.

Log in / Register

Useful Searches

Join the community today

Featured Nginx ngxtop real time metrics for Nginx

negative Active Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

BamaStangGuy Active Member

eva2000 Administrator Staff Member

BamaStangGuy Active Member

BamaStangGuy Active Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

negative Active Member

eva2000 Administrator Staff Member

negative Active Member

eva2000 Administrator Staff Member

negative Active Member

eva2000 Administrator Staff Member

negative Active Member

eva2000 Administrator Staff Member

.