Join the community today
Become a Member

Beta Branch Centmin Mod Nginx live vhost traffic statistics preview & discussion

Discussion in 'Beta release code' started by eva2000, May 28, 2015.

  1. eva2000

    eva2000 Administrator Staff Member

    30,632
    6,862
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,308
    Local Time:
    11:48 AM
    Nginx 1.13.x
    MariaDB 5.5
    Dedicated discussion thread for Nginx Vhost Traffic Statistics module that I plan to add to Centmin Mod .08 beta.

    How Nginx Vhost Traffic Stats Module Integrates into Centmin Mod Nginx



    Module: https://github.com/vozlt/nginx-module-vts

    On nginx upgrades within .08 beta which has nginx vhost traffic stats module added, you will get 3 new include files for nginx.conf, vhost.conf and your individual nginx vhost domain.com.conf files. On upgrades only nginx.conf and vhost.conf get the new include files automatically populated for existing files. The 3rd include file for nginx vhost domain.com.conf will need manually adding for existing installs (for new fresh installs, they are added automatically during centmin.sh menu option 2 add vhost stage).

    in nginx.conf
    Code:
    http {
    include /usr/local/nginx/conf/vts_http.conf;
    contents = Sets parameters for a shared memory zone that will keep states for various keys. The cache is shared between all worker processes
    Code:
    vhost_traffic_status_zone;
    in vhost.conf
    Code:
    include /usr/local/nginx/conf/vts_mainserver.conf;
    contents = ip restricted by default enables vhost_traffic_status = on or off for each individual nginx vhost and display in html format (other option is in json format) to be displayed at yourmainhostname/vhost_status or yourmainhostname/vhost_status.html
    Code:
    location /vhost_status {
        allow 127.0.0.1;
        #allow YOURIPADDRESS;
        deny all;
        vhost_traffic_status on;
        vhost_traffic_status_display;
        vhost_traffic_status_display_format html;
    }
    
    location = /vhost_status.html {
        allow 127.0.0.1;
        #allow YOURIPADDRESS;
        deny all;
    }
    
    for individual nginx domain.com.conf vhosts will need to add this manually for existing servers that upgrade (for new fresh installs, already added)
    Code:
    include /usr/local/nginx/conf/vts_server.conf;
    contents = enables vhost_traffic_status = on or off for each individual nginx vhost to be displayed at yourmainhostname/vhost_status
    Code:
                vhost_traffic_status on;
    local virtualbox testing with bit of siege benchmarking :D

    at yourmainhostname/vhost_status

    vhost_status_02.png

    or yourmainhostname/vhost_status.html for custom CSS colour version

    vhost_status_blue_02.png
     
    Last edited: Jun 1, 2015
    • Like Like x 2
  2. eva2000

    eva2000 Administrator Staff Member

    30,632
    6,862
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,308
    Local Time:
    11:48 AM
    Nginx 1.13.x
    MariaDB 5.5
    • Like Like x 1
  3. eva2000

    eva2000 Administrator Staff Member

    30,632
    6,862
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,308
    Local Time:
    11:48 AM
    Nginx 1.13.x
    MariaDB 5.5
    If you do not have a static IP address and can not use IP restriction in, then you'll need to password protect that directory

    Step 1. create htpasswd file /usr/local/nginx/conf/vts_htpasswd with user and password of your choosing with below command where USERNAME and PASSWORD are of your own choosing.via the custom password generator at /usr/local/nginx/conf/htpasswd.sh

    If you're backing up your nginx vhosts maybe better to place /usr/local/nginx/conf/vts_htpasswd file at something like /home/nginx/domains/yourdomain.com/private/vts_htpasswd

    At /usr/local/nginx/conf/vts_htpasswd type in SSH window the command below
    Code:
    /usr/local/nginx/conf/htpasswd.sh create /usr/local/nginx/conf/vts_htpasswd USERNAME PASSWORD
    
    or

    At /home/nginx/domains/yourdomain.com/private/vts_htpasswd replacing yourdomain.com with your domain type in SSH window the command below
    Code:
    /usr/local/nginx/conf/htpasswd.sh create /home/nginx/domains/yourdomain.com/private/vts_htpasswd USERNAME PASSWORD
    
    Step 2. edit /usr/local/nginx/conf/vts_mainserver.conf

    Then contents of /usr/local/nginx/conf/vts_mainserver.conf becomes

    Code:
            location /vhost_status {
                auth_basic "Private";
                auth_basic_user_file /usr/local/nginx/conf/vts_htpasswd;
                vhost_traffic_status on;
                vhost_traffic_status_display;
                vhost_traffic_status_display_format html;
            }
    
           location = /vhost_status.html {
                auth_basic "Private";
                auth_basic_user_file /usr/local/nginx/conf/vts_htpasswd;
                allow 127.0.0.1;
               #allow YOURIPADDRESS;
              deny all;
          }
    
    or

    Code:
            location /vhost_status {
                auth_basic "Private";
                auth_basic_user_file  /home/nginx/domains/yourdomain.com/private/vts_htpasswd;
                vhost_traffic_status on;
                vhost_traffic_status_display;
                vhost_traffic_status_display_format html;
            }
    
           location = /vhost_status.html {
                auth_basic "Private";
                auth_basic_user_file  /home/nginx/domains/yourdomain.com/private/vts_htpasswd;
                allow 127.0.0.1;
               #allow YOURIPADDRESS;
              deny all;
          }
    
    Step 3. restart Nginx and PHP services via command shortcut

    Code:
    nprestart
    
     
    Last edited: Jun 1, 2015
  4. RoldanLT

    RoldanLT Well-Known Member

    3,940
    962
    113
    May 25, 2014
    Phillipines
    Ratings:
    +1,315
    Local Time:
    9:48 AM
    1.11
    10.2
    I turn off the entire log feature to improve the performance a bit, so can't use this for now :).
     
  5. vozlt

    vozlt New Member

    6
    3
    3
    May 29, 2015
    Ratings:
    +4
    Local Time:
    10:48 AM
    The working of the module doesn't matter at all whether the access_log directive "on" or "off".
    Again, this module works well on "access_log off".
     
    • Like Like x 2
  6. RoldanLT

    RoldanLT Well-Known Member

    3,940
    962
    113
    May 25, 2014
    Phillipines
    Ratings:
    +1,315
    Local Time:
    9:48 AM
    1.11
    10.2
    Aw, OK.
    Thanks for the info!
     
  7. eva2000

    eva2000 Administrator Staff Member

    30,632
    6,862
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,308
    Local Time:
    11:48 AM
    Nginx 1.13.x
    MariaDB 5.5
    thanks @vozlt for the info and awesome add on you wrote - assuming you're author vozlt/nginx-module-vts · GitHub :)
     
  8. eva2000

    eva2000 Administrator Staff Member

    30,632
    6,862
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,308
    Local Time:
    11:48 AM
    Nginx 1.13.x
    MariaDB 5.5
  9. vozlt

    vozlt New Member

    6
    3
    3
    May 29, 2015
    Ratings:
    +4
    Local Time:
    10:48 AM
    To customize after the module installed
    1. You need to change the {{uri}} string to your status uri in status.template.html as follows:

      shell> vi share/status.template.html

      var vtsStatusURI = "yourStatusUri/format/json", vtsUpdateInterval = 1000;

    2. And then, customizing and copy status.template.html to server root directory as follows:

      shell> cp share/status.template.html /usr/share/nginx/html/status.html

    3. Configure nginx.conf

      server {
      server_name YOURDOMAIN;
      root /usr/share/nginx/html;

      # Redirect requests for / to /status.html
      location = / {
      return 301 /status.html;
      }

      location = /status.html {}

      # Everything beginning /status (except for /status.html) is
      # processed by the status handler
      location /status {
      vhost_traffic_status_display;
      vhost_traffic_status_display_format json;
      }
      }

    4. Access to your html.

      http://YOURDOMAIN/status.html
    To customize before the module installed
    1. Modify share/status.template.html (Do not change {{uri}} string)

    2. Recreate the ngx_http_vhost_traffic_status_module_html.h as follows:

      shell> cd util
      shell> ./tplToDefine.sh status.template.html > ../src/ngx_http_vhost_traffic_status_module_html.h

    3. Add the module to the build configuration by adding
      --add-module=/path/to/nginx-module-vts

    4. Build the nginx binary.

    5. Install the nginx binary.
     
    Last edited: Jun 1, 2015
    • Informative Informative x 1
  10. eva2000

    eva2000 Administrator Staff Member

    30,632
    6,862
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,308
    Local Time:
    11:48 AM
    Nginx 1.13.x
    MariaDB 5.5
    Thanks seems path to fileToHex.pl is relative as tried but can't find fileToHex.pl

    Code:
    util/tplToDefine.sh share/status.template.html > src/ngx_http_vhost_traffic_status_module_html.h
    Can't open perl script "fileToHex.pl": No such file or directory
    
    but fine with running it within util directory

    Code:
    ./tplToDefine.sh ../share/status.template.html > ../src/ngx_http_vhost_traffic_status_module_html.h
    
    more verbose look
    Code:
    bash -x tplToDefine.sh ../share/status.template.html > ../src/ngx_http_vhost_traffic_status_module_html.h
    + PATH=/sbin:/usr/sbin:/bin:/usr/bin
    + export PATH
    + template=../share/status.template.html
    + '[' -z ../share/status.template.html ']'
    ++ date +%s
    + tmp=../share/status.template.html.1433144033
    + cp -af ../share/status.template.html ../share/status.template.html.1433144033
    + '[' -f ../share/status.template.html.1433144033 ']'
    + perl -p -i -e s/%/%%/g ../share/status.template.html.1433144033
    + perl -p -i -e 's/{{uri}}/%V/g' ../share/status.template.html.1433144033
    + echo '#define NGX_HTTP_VHOST_TRAFFIC_STATUS_HTML_DATA \'
    + perl fileToHex.pl ../share/status.template.html.1433144033
    + rm -f ../share/status.template.html.1433144033
    Tried the before install method but sill old colors show up after nginx recompile - is it because i use url vhost_status and not status for url ?

    just changing the CSS in share/status.template.html

    Code:
            <style>
                body {
                    background:  white;
                    font-family: Helvetica, Arial, sans-serif;
                }
    
                h1 {
                    margin: .5em 0 0 0;
                }
    
                h2 {
                    margin: .8em 0 .3em 0;
                }
    
                h3 {
                    margin: .5em 0 .3em 0;
                }
    
                table {
                    font-size: .8em;
                    margin: .5em 0;
                    border-collapse: collapse;
                    border-bottom: 1px #43a6df solid;
                }
    
                thead th {
                    font-size: 1em;
                    background: #43a6df;
                    padding: .1em .3em;
                    border: .2em solid #FFF;
                }
    
                tbody tr.odd {
                    background: #F5F5F5;
                }
    
                tbody th {
                    text-align: left;
                }
    
                tbody td {
                    height: 1.2em;
                    text-align: right;
                }
    
                tbody td.key {
                    font-size: 1em;
                    background: #43a6df;
                    padding: .1em .3em;
                    border: .2em solid #FFF;
                }
            </style>
    
    while before install doesn't work, after install works :)

    View attachment 1607
     
    Last edited: Jun 1, 2015
  11. vozlt

    vozlt New Member

    6
    3
    3
    May 29, 2015
    Ratings:
    +4
    Local Time:
    10:48 AM
    First of all, make sure the modified status.template.html works well before built in code. (at To customize after the module installed).
     
  12. eva2000

    eva2000 Administrator Staff Member

    30,632
    6,862
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,308
    Local Time:
    11:48 AM
    Nginx 1.13.x
    MariaDB 5.5
    I believe it should as all I am doing is sed replace on the status.template.html file

    Code:
    cd /svr-setup/nginx-module-vts
    sed -i 's|#DED|#43a6df|g' /svr-setup/nginx-module-vts/share/status.template.html
    sed -i 's|#DED|#43a6df|g' /svr-setup/nginx-module-vts/share/status.compress.html
    cd util
    ./tplToDefine.sh ../share/status.template.html > ../src/ngx_http_vhost_traffic_status_module_html.h
    
    then recompile nginx with --add-module=../nginx-module-vts

    FYI on the /vhost_status page which still shows the old #DED colours, it's pulling the compressed html status template in source code still if that helps. So i tried this too but doesn't work either
    Code:
    ./tplToDefine.sh ../share/status.compress.html > ../src/ngx_http_vhost_traffic_status_module_html.h
    still shows old #DED colours

    for working after install, i just copy /svr-setup/nginx-module-vts/share/status.template.html to /status.html and that works

     
    Last edited: Jun 1, 2015
  13. vozlt

    vozlt New Member

    6
    3
    3
    May 29, 2015
    Ratings:
    +4
    Local Time:
    10:48 AM
    The status.template.html and status.compress.html is same content file.
    The status.compress.html is just html compressed file of status.template.html.
    The best way to built would be customizing before install and then built in code as follows:

    1. copy status.template.html to serverRoot/status.html

    2. vi serverRoot/status.html

    Change {{uri}} to yourStatusUri

    3. nginx.conf configure
    server {
    server_name YOURDOMAIN;
    root /usr/share/nginx/html;

    # Redirect requests for / to /status.html
    location = / {
    return 301 /status.html;
    }

    location = /status.html {}

    # Everything beginning /status (except for /status.html) is
    # processed by the status handler
    location /status {
    vhost_traffic_status_display;
    vhost_traffic_status_display_format json;
    }
    }

    4. customizing serverRoot/status.html & testing

    5. vi serverRoot/status.html

    Change yourStatusUri to {{uri}}

    6. built in code
    cd util
    ./tplToDefine.sh serverRoot/status.html > ../src/ngx_http_vhost_traffic_status_module_html.h

    7. compile
     
  14. eva2000

    eva2000 Administrator Staff Member

    30,632
    6,862
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,308
    Local Time:
    11:48 AM
    Nginx 1.13.x
    MariaDB 5.5
    thanks @vozlt for explanation :) oh for code you can wrap it in [CODE][/CODE] tags too on the forums :)

    Guess it would be harder to do this way if I am automating this in an installer as I would have to know the uri that the end user uses for their Centmin Mod Nginx install before hand. I suppose I could try it from the default main host name's nginx vhost.conf file - but would also need to know their url/path if they decide to use something other than /status or /status.html i.e. /vhost_status or /vhost_status.html
    Code:
    awk -F " " '/server_name/ {print $2}' /usr/local/nginx/conf/conf.d/virtual.conf | sed 's|;||'
    mainhostname.domain.com
    then replace {{uri}} with that mainhostname.domain.com/vhost_status

    But i guess if i set it to /vhost_status or /vhost_status.html not many folks will change it later
     
  15. eva2000

    eva2000 Administrator Staff Member

    30,632
    6,862
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,308
    Local Time:
    11:48 AM
    Nginx 1.13.x
    MariaDB 5.5
    Okay did it the method @vozlt outlined with custom vhost_status.html template at /usr/local/nginx/html/vhost_status.html

    changed include file at /usr/local/nginx/conf/vts_mainserver.conf to

    Code:
    location /vhost_status {
        allow 127.0.0.1;
        #allow YOURIPADDRESS;
        deny all;
        vhost_traffic_status on;
        vhost_traffic_status_display;
        vhost_traffic_status_display_format html;
    }
    
    location = /vhost_status.html {
        allow 127.0.0.1;
        #allow YOURIPADDRESS;
        deny all;
    }
    committed changes update vts_mainserver.conf include file and nginx-module-vts custom C… · centminmod/centminmod@5570e3c · GitHub

    vhost_status_blue_02.png
     
    Last edited: Jun 1, 2015
  16. eva2000

    eva2000 Administrator Staff Member

    30,632
    6,862
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,308
    Local Time:
    11:48 AM
    Nginx 1.13.x
    MariaDB 5.5
  17. RoldanLT

    RoldanLT Well-Known Member

    3,940
    962
    113
    May 25, 2014
    Phillipines
    Ratings:
    +1,315
    Local Time:
    9:48 AM
    1.11
    10.2
    I have NGINX_VHOSTSTATS=n
    On initial install, I got this:
     
    • Like Like x 1
  18. eva2000

    eva2000 Administrator Staff Member

    30,632
    6,862
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,308
    Local Time:
    11:48 AM
    Nginx 1.13.x
    MariaDB 5.5
    is that during install or at end of install and you try nginx restart ?

    at end of install, if you do nginx restart any errors ? that's all that matters if at end of install and nginx restart has no errors
     
  19. RoldanLT

    RoldanLT Well-Known Member

    3,940
    962
    113
    May 25, 2014
    Phillipines
    Ratings:
    +1,315
    Local Time:
    9:48 AM
    1.11
    10.2
    It happens at the end of initial install.
    Even ngxrestart still display the same error.

    I have to comment it out on nginx.conf and demodomain.conf
     
    • Like Like x 1
  20. eva2000

    eva2000 Administrator Staff Member

    30,632
    6,862
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,308
    Local Time:
    11:48 AM
    Nginx 1.13.x
    MariaDB 5.5
    • Like Like x 1