Discover Centmin Mod today
Register Now

Featured Nginx ngxtop real time metrics for Nginx

Discussion in 'Nginx, PHP-FPM & MariaDB MySQL' started by eva2000, Jun 5, 2014.

  1. eva2000

    eva2000 Administrator Staff Member

    30,160
    6,784
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,135
    Local Time:
    8:31 PM
    Nginx 1.13.x
    MariaDB 5.5
    Folks may have heard of ngxtop for real time metrics for Nginx lebinh/ngxtop · GitHub ?

    Here's how I installed ngxtop for Centmin Mod Nginx based servers.

    For CentOS 6, to install ngxtop type:
    Code:
    yum -y install python-pip; pip install ngxtop sqlite-devel
    
    For CentOS 7, to install ngxtop type:
    Code:
    yum -y install python-pip
    pip install --upgrade pip
    pip install ngxtop
    
    By default ngxtop only reports new connections (live) not previously logged connections. So if you have a relatively idle server you may not see live stats.

    To check live stats for Centmin Mod Nginx hosted domains pass the location of the domain vhost's access.log type:
    Code:
    ngxtop -l /home/nginx/domains/yourdomain.com/log/access.log
    
    To check previously logged data use the --no-follow option
    Code:
    ngxtop -l /home/nginx/domains/yourdomain.com/log/access.log --no-follow
    
    More commands outlined at lebinh/ngxtop · GitHub - just remember to pass the Centmin Mod Nginx accesslog file with the commands (-l /home/nginx/domains/yourdomain.com/log/access.log).

    For example:
    Code:
    ngxtop -l /home/nginx/domains/yourdomain.com/log/access.log --no-follow top remote_addr
    
    print status >400 for requests and referer
    Code:
    ngxtop -l /home/nginx/domains/yourdomain.com/log/access.log --no-follow -i 'status >= 400' print request status http_referer
    
    print the request, user agent and remote ip for realtime checks
    Code:
    ngxtop -l /home/nginx/domains/yourdomain.com/log/access.log print request http_user_agent remote_addr
     
    Last edited: Dec 10, 2015
    • Like Like x 1
    • Informative Informative x 1
  2. pamamolf

    pamamolf Well-Known Member

    2,722
    242
    63
    May 31, 2014
    Ratings:
    +433
    Local Time:
    1:31 PM
    Nginx-1.13.x
    MariaDB 10.1.x
    I just install it and run :

    ngxtop -l /home/nginx/domains/yourdomain.com/log/access.log

    It shows 0 as it is my test forum..... Then i try to surf on the forum and do some posts and topics and checking it is still 0 .... ?
     
  3. eva2000

    eva2000 Administrator Staff Member

    30,160
    6,784
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,135
    Local Time:
    8:31 PM
    Nginx 1.13.x
    MariaDB 5.5
    command you used is for live stats, as mentioned above

     
  4. pamamolf

    pamamolf Well-Known Member

    2,722
    242
    63
    May 31, 2014
    Ratings:
    +433
    Local Time:
    1:31 PM
    Nginx-1.13.x
    MariaDB 10.1.x
    but i am connecting to nginx after running the command.....

    Or i must wait to end the session (and how long? ) to try?
     
  5. eva2000

    eva2000 Administrator Staff Member

    30,160
    6,784
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,135
    Local Time:
    8:31 PM
    Nginx 1.13.x
    MariaDB 5.5
    is really live as in same moment.. so if little traffic, use the --no-follow option
     
  6. dorobo

    dorobo Active Member

    420
    104
    43
    Jun 6, 2014
    Ratings:
    +161
    Local Time:
    6:31 PM
    latest
    latest
    I got this

    Code:
    # ngxtop -l /home/nginx/domains/xxxxxx.com/log/access.log --no-follow
    Traceback (most recent call last):
      File "/usr/local/bin/ngxtop", line 7, in <module>
        from ngxtop.ngxtop import main
      File "/usr/local/lib/python2.7/site-packages/ngxtop/ngxtop.py", line 64, in <module>
        import sqlite3
      File "/usr/local/lib/python2.7/sqlite3/__init__.py", line 24, in <module>
        from dbapi2 import *
      File "/usr/local/lib/python2.7/sqlite3/dbapi2.py", line 27, in <module>
        from _sqlite3 import *
    ImportError: No module named _sqlite3
     
  7. eva2000

    eva2000 Administrator Staff Member

    30,160
    6,784
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,135
    Local Time:
    8:31 PM
    Nginx 1.13.x
    MariaDB 5.5
  8. dorobo

    dorobo Active Member

    420
    104
    43
    Jun 6, 2014
    Ratings:
    +161
    Local Time:
    6:31 PM
    latest
    latest
    fixed it by

    $ yum -y install sqlite-devel

    then

    $ cd /svr-setup/Python-2.7.7
    $ ./configure --prefix=/usr/local --with-threads --enable-unicode=ucs4 --enable-shared LDFLAGS="-Wl,-rpath /usr/local/lib"
    $ make
    $ make altinstall
     
    • Like Like x 1
  9. eva2000

    eva2000 Administrator Staff Member

    30,160
    6,784
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,135
    Local Time:
    8:31 PM
    Nginx 1.13.x
    MariaDB 5.5
    sweet nice work (y)
     
  10. dorobo

    dorobo Active Member

    420
    104
    43
    Jun 6, 2014
    Ratings:
    +161
    Local Time:
    6:31 PM
    latest
    latest
    thanks to stackoverflow :D
     
    • Like Like x 1
  11. BamaStangGuy

    BamaStangGuy Active Member

    470
    137
    43
    May 25, 2014
    Ratings:
    +180
    Local Time:
    5:31 AM
    Interesting results. This is a pretty active site on SSL and proxy.php is getting a lot of work.

    Code:
    running for 59 seconds, 543990 records processed: 9158.22 req/sec
    
    Summary:
    |   count |   avg_bytes_sent |    2xx |    3xx |   4xx |   5xx |
    |---------+------------------+--------+--------+-------+-------|
    |  543990 |        37692.300 | 322380 | 206372 | 15127 |   111 |
    
    Detailed:
    | request_path         |   count |   avg_bytes_sent |   2xx |   3xx |   4xx |   5xx |
    |----------------------+---------+------------------+-------+-------+-------+-------|
    | /proxy.php           |   56343 |        79050.931 | 36878 | 19359 |   101 |     5 |
    | /css.php             |   19700 |        10742.102 | 19449 |   182 |    69 |     0 |
    | /deferred.php        |    6273 |           51.169 |  6265 |     0 |     8 |     0 |
    | /mobiquo/mobiquo.php |    3553 |         2098.095 |  3518 |     0 |     1 |    34 |
    | /                    |    3000 |        25747.176 |  2961 |     0 |    39 |     0 |
    | /goto/post           |    2851 |           10.381 |     0 |  2848 |     3 |     0 |
    | /forums/             |    2574 |        23106.085 |  2558 |     0 |    16 |     0 |
    | /login/login         |    2560 |        15899.627 |  2301 |   205 |    54 |     0 |
    | /login/              |    1961 |        11193.751 |  1960 |     0 |     1 |     0 |
    | /register/register   |    1133 |        18284.578 |  1078 |     0 |    55 |     0 |
     
    • Like Like x 1
  12. eva2000

    eva2000 Administrator Staff Member

    30,160
    6,784
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,135
    Local Time:
    8:31 PM
    Nginx 1.13.x
    MariaDB 5.5
    yeah expected if your forum has an active number of externally referenced images by members :)
     
  13. BamaStangGuy

    BamaStangGuy Active Member

    470
    137
    43
    May 25, 2014
    Ratings:
    +180
    Local Time:
    5:31 AM
    Taigachat likes requests as well.

    Code:
    running for 20 seconds, 238090 records processed: 11907.53 req/sec
    
    Summary:
    |   count |   avg_bytes_sent |    2xx |   3xx |   4xx |   5xx |
    |---------+------------------+--------+-------+-------+-------|
    |  238090 |        14652.663 | 204208 | 26781 |  6866 |   235 |
    
    Detailed:
    | request_path         |   count |   avg_bytes_sent |   2xx |   3xx |   4xx |   5xx |
    |----------------------+---------+------------------+-------+-------+-------+-------|
    | /taigachat/list.json |   90003 |          467.810 | 89968 |     0 |    35 |     0 |
    | /css.php             |   14478 |        11891.153 | 14264 |   208 |     6 |     0 |
    | /                    |   11739 |        58538.345 | 11261 |     0 |   478 |     0 |
    | /mobiquo/mobiquo.php |    9647 |         2386.218 |  9441 |     0 |     0 |   206 |
    | /deferred.php        |    5990 |           48.007 |  5973 |     0 |    17 |     0 |
    | /forums/             |    3623 |        33778.260 |  3289 |     0 |   334 |     0 |
    | /proxy.php           |    2045 |        77919.337 |  1422 |   623 |     0 |     0 |
    | /login/login         |    1655 |        15721.532 |  1131 |   332 |   192 |     0 |
    | /login/              |    1474 |         5307.315 |   926 |     0 |   548 |     0 |
    | /blog.php            |    1437 |            4.083 |     0 |  1434 |     3 |     0 |
     
    • Like Like x 2
  14. eva2000

    eva2000 Administrator Staff Member

    30,160
    6,784
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,135
    Local Time:
    8:31 PM
    Nginx 1.13.x
    MariaDB 5.5
    Added another example command to first post

    print the request, user agent and remote ip for realtime checks
    Code:
    ngxtop -l /home/nginx/domains/yourdomain.com/log/access.log print request http_user_agent remote_addr
     
  15. RoldanLT

    RoldanLT Well-Known Member

    3,901
    949
    113
    May 25, 2014
    Phillipines
    Ratings:
    +1,298
    Local Time:
    6:31 PM
    1.11
    10.2
    I really hope this is possible to view on the web :).
    No need to login to terminal.
     
  16. eva2000

    eva2000 Administrator Staff Member

    30,160
    6,784
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,135
    Local Time:
    8:31 PM
    Nginx 1.13.x
    MariaDB 5.5
    nope command line only I believe
     
  17. negative

    negative Member

    216
    21
    18
    Apr 11, 2015
    Ratings:
    +49
    Local Time:
    1:31 PM
    1.9.10
    10.1.11
    Hello @eva2000

    I run the that command via ssh
    Code:
    yum -y install python-pip; pip install ngxtop sqlite-devel
    And it says:

    Code:
    Installed:
      python-pip.noarch 0:7.1.0-1.el7                                                                                                                          
    
    Complete!
    /usr/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
      InsecurePlatformWarning
    You are using pip version 7.1.0, however version 7.1.2 is available.
    You should consider upgrading via the 'pip install --upgrade pip' command.
    Collecting ngxtop
    /usr/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
      InsecurePlatformWarning
      Downloading ngxtop-0.0.2-py2.py3-none-any.whl
    Collecting sqlite-devel
      [B]Could not find a version that satisfies the requirement sqlite-devel (from versions: )
    No matching distribution found for sqlite-devel[/B]
    Than i run
    pip install --upgrade pip

    And it upgraded to new version.
    successfully uninstalled pip-7.1.0
    Successfully installed pip-7.1.2

    But still, if i run the "pip install ngxtop sqlite-devel" command, still i see same errors
     
  18. eva2000

    eva2000 Administrator Staff Member

    30,160
    6,784
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,135
    Local Time:
    8:31 PM
    Nginx 1.13.x
    MariaDB 5.5
    might want to check/ask at Issues · lebinh/ngxtop · GitHub and see if the issue has come up before

    also updated install info for CentOS7 i don't think sqlite-devel is correct
     
    Last edited: Dec 10, 2015
  19. negative

    negative Member

    216
    21
    18
    Apr 11, 2015
    Ratings:
    +49
    Local Time:
    1:31 PM
    1.9.10
    10.1.11
    Okay i have fixed the problem.

    I can't see realtime stats exactly, because i think access.log is delaying while saving records in your standart nginx configuration right ?
    Thanks
     
  20. eva2000

    eva2000 Administrator Staff Member

    30,160
    6,784
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,135
    Local Time:
    8:31 PM
    Nginx 1.13.x
    MariaDB 5.5
    yup change the access.log buffer and flush time for your vhost