Nginx Nginx - Perfect install for netdata (system monitor) - Step by step

First a little intro about netdata, it's a live feed system monitor that can monitor multiple servers. Have a look at the GitHub here at netdata/netdata for a nice introduction of netdata.

To set it up is effortless. It installs missing packages and sets a cronjob top keep it up to date. Only big negative is that it's accessible on your server ip and port by default. But you can block it in the configuration file so that you only can access it on your static IP. The link below will install netdata with the most options.

Code (Text):

bash <(curl -Ss https://my-netdata.io/kickstart.sh)

to install the default installer, just hit Y to download missing YUM packages
and keep pressing enter to confirm your freshly install.

--- Check KSM (kernel memory deduper) ---

Memory de-duplication instructions

If have kernel has memory de-duper (called Kernel Same-page Merging,
or KSM) available, but it is not currently enabled.

To enable it run:

Code (Text):

echo 1 >/sys/kernel/mm/ksm/run
    echo 1000 >/sys/kernel/mm/ksm/sleep_millisecs

This did not work, due to permission issue so

Code (Text):

echo 1 >/sys/kernel/mm/ksm/run
    cd /sys/kernel/mm/ksm/
    nano sleep_millisecs
    and change the default 20 to 1000

If you enable it, you will save 40-60% of netdata memory.

--- Check version.txt ---
--- Check apps.plugin ---
--- Copy uninstaller ---
--- Basic netdata instructions ---

netdata by default listens on all IPs on port 19999, so you can access it with:

Code (Text):

http://yourserverip:19999/

To stop netdata run:

Code (Text):

systemctl stop netdata

To start netdata run:

Code (Text):

systemctl start netdata

Uninstall script copied to: /usr/libexec/netdata-uninstaller.sh
--- Installing new netdata-updater in cron ---
Update script is located at /etc/cron.daily/netdata-updater

netdata-updater works from cron. It will trigger an email from cron
only if it fails (it should not print anything when it can update netdata).

At the moment our netdata is unprotected and you can access it on the http://serverip:19999/ or http://serverx.domain.com:19999/

---

This is far from ideal, so we want to set it up with some protection.

TEXT HOW TO TO PLACE

Enable Nginx stats
Enable Nginx stub_status module

Check nginx -V
it should have the 'stub_status' module on the list.

On out Centmin instaltions whe have to check /usr/local/nginx/conf/conf.d/virtual.conf that it has this entry

Code (Text):

location /nginx_status {
        stub_status on;
        access_log   off;
        allow 127.0.0.1;
        #allow youripaddress;
        deny all;
        }

After you restarted nginx you can check if it works with

Code (Text):

curl http://127.0.0.1/nginx_status
Active connections: 2
server accepts handled requests
 264 264 6682
Reading: 0 Writing: 1 Waiting: 1

Aferward check and add to /etc/netdata/python.d/nginx.conf

Code (Text):

localhost:
   name : 'local'
   url  : 'http://localhost/nginx_status'

You can afterward check the new Nginx stats in Netdata
on the 'nginx local' on the right, and see Nginx active connections, requests, status and the connection rate.
Protect access

Edit nano /etc/netdata/netdata.conf

Code (Text):

[web]
       # x-frame-options response header =
       allow connections from = localhost mystaticIP
       # allow dashboard from = localhost *

php-fpm monitoring

Enable the php status page. On a default centminmod install you just need to uncomment:

Code (Text):

#include /usr/local/nginx/conf/phpstatus.conf;

from /usr/local/nginx/conf/conf.d/virtual.conf

Code (Text):

nprestart

Create (if it doesn't exist already) a file phpfpm.conf in /etc/netdata/python.d with the following contents:

Code (Text):

local:
  url     : 'http://localhost/phpstatus'

mariadb
Create a user for netdata to use to access the database

Code (Text):

mysql -u root -p
MariaDB [(none)]> CREATE USER 'netdata'@'localhost';
MariaDB [(none)]> GRANT USAGE on *.* to 'netdata'@'localhost';
MariaDB [(none)]> FLUSH PRIVILEGES;
MariaDB [(none)]> \q

Install the required python libraries

Code (Text):

pip install mysqlclient

Create mysql.conf in /etc/netdata/python.d/

Code (Text):

local:
  'my.cnf'   : '/etc/my.cnf'
  user     : 'netdata'

Redis
creating redis.conf in /etc/netdata/python.d/ with the following contents:

Code (Text):

socket:
  name     : 'local'
  socket   : '/var/lib/redis/redis.sock'

localhost:
  name     : 'local'
  host     : 'localhost'
  port     : 6379

Enable elasticsearch
Add elasticsearch.conf in /etc/netdata/python.d with the following contents:

Code (Text):

local:
  host               : 'localhost'    # Elasticsearch server ip address or hostname
  port               : '9200'         # Port on which elasticsearch listens

Now restart the netdata service using the systemctl command.

Code (Text):

systemctl restart netdata

You can afterward check the new Nginx stats in Netdata
on the 'nginx local' on the right, and see Nginx active connections, requests, status and the connection rate.


TODO LIST

• Layout of this post
• Add on
• Endless --> New ideas please!

Added

• Corrected small typo in Nginx Monitoring(08/10/2019)
• Nginx monitoring (26/06/2019)
• elasticsearch (15/08/2019)
• Redis (15/08/2019)
• phpfpm (15/08/2019)
• Protection (15/08/2019)
• mariadb (15/08/2019)

This is a quickly setup to get netdata working.

Special thanks to @jair for the contributions.

As I pretend to actualize this post frequently, any feedback, suggestions, new add-ons are welcome. It will help other users so please don't hesitate and POST.

I will keep this post up to date with any feedback given on post below this one. So always refer to this first post to get it with the latest updates on fresh installs.

 

Just noticed that I was not running the latest update and that I'm unable to edit my first post... To bad.
I love to use nightly builds for testing and getting the latest new stuff, but be aware it has a risk.



So ran the updater with the

Code (Text):

bash <(curl -Ss https://my-netdata.io/kickstart.sh) --auto-update

This will create a daily cronjob to keep the install up to date.

 

Well, followed some instructions that I found on GitHub to place it in the server part, but that also does not work, it gives the directive error.

I remember I had this before and @eva2000 told me to use a conf file of nginx, but guess what happened, my memory can't recall that file.

So, still struggling to set up the htpasswd on sub.domain.com that is installed as a normal domain with option . Getting error 403, that means that nginx understood it, but can't handle the instruction.

For security reason I edit the vhost file and enable the password and when I want to see it myself I disable it, but it's not a real world situation, just a quick workaround

 

Did that already, deleted the old hash, and used a username/password that was generated with option 22 to be sure that it should work. But no go. Opened an issue on GitHub and it's on the roadmap to create a separate login page.

So, waiting for that. I don't look every 5 minutes to the netdata, so the workaround is oke for me now.

 

Noticed you saw my post but just to give it another try my current vhost. Maybe you discover something strange.

Code (Text):

#x# HTTPS-DEFAULT
 upstream backend {
    # the netdata server
    server 127.0.0.1:19999;
    keepalive 64;
}

server {
    # nginx listens to this
    listen 443 ssl http2;

    # the virtual host name of this
    server_name $vhost;

    include /usr/local/nginx/conf/ssl/$vhost/$vhost.crt.key.conf;
    include /usr/local/nginx/conf/ssl_include.conf;
    #limit_req zone=xwplogin burst=1 nodelay;
    #auth_basic "Private Access";
    #auth_basic_user_file /home/nginx/$vhost/htpasswd_netdata;
    #include /usr/local/nginx/conf/php-wpsc.conf;


  


    location / {
        #limit_req zone=xwplogin burst=1 nodelay;
        auth_basic "Private Access";
        auth_basic_user_file /home/nginx/$vhost/htpasswd_netdata;
        #include /usr/local/nginx/conf/php-wpsc.conf;
        proxy_set_header X-Forwarded-Host $host;
        proxy_set_header X-Forwarded-Server $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_pass http://backend;
        proxy_http_version 1.1;
        proxy_pass_request_headers on;
        proxy_set_header Connection "keep-alive";
        proxy_store off;
  
        }

  # cloudflare authenticated origin pull cert community.centminmod.com/threads/13847/
  #ssl_client_certificate /usr/local/nginx/conf/ssl/cloudflare/$vhost/origin.crt;
  #ssl_verify_client on;
  http2_max_field_size 16k;
  http2_max_header_size 32k;
  # mozilla recommended
  ssl_ciphers TLS13-AES-128-GCM-SHA256:TLS13-AES-256-GCM-SHA384:TLS13-CHACHA20-POLY1305-SHA256:TLS13-AES-128-CCM-8-SHA256:TLS13-AES-128-CCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS;
  ssl_prefer_server_ciphers   on;
  #add_header Alternate-Protocol  443:npn-spdy/3;

  # before enabling HSTS line below read centminmod.com/nginx_domain_dns_setup.html#hsts
  #add_header Strict-Transport-Security "max-age=31536000; includeSubdomains;";
  #add_header X-Frame-Options SAMEORIGIN;
  add_header X-Xss-Protection "1; mode=block" always;
  add_header X-Content-Type-Options "nosniff" always;
  #add_header Referrer-Policy "strict-origin-when-cross-origin";
  #spdy_headers_comp 5;
  ssl_buffer_size 1369;
  ssl_session_tickets on;
 
  # enable ocsp stapling
  resolver 8.8.8.8 8.8.4.4 1.1.1.1 1.0.0.1 valid=10m;
  resolver_timeout 10s;
  ssl_stapling on;
  ssl_stapling_verify on;

  # limit_conn limit_per_ip 16;
  # ssi  on;

  access_log /home/nginx/domains/$vhost/log/access.log combined buffer=256k flush=5m;
  error_log /home/nginx/domains/$vhost/log/error.log;

  include /usr/local/nginx/conf/autoprotect/$vhost/autoprotect-$vhost.conf;
  root /home/nginx/domains/$vhost/public;
  # uncomment cloudflare.conf include if using cloudflare for
  # server and/or vhost site
  #include /usr/local/nginx/conf/cloudflare.conf;
  include /usr/local/nginx/conf/503include-main.conf;

#  include /usr/local/nginx/conf/503include-only.conf;

# block common exploits, sql injections etc
#include /usr/local/nginx/conf/block.conf;

  # Enables directory listings when index file not found
  #autoindex  on;

  # Shows file listing times as local time
  #autoindex_localtime on;

#  include /usr/local/nginx/conf/pre-staticfiles-local-$vhost.conf;
#  include /usr/local/nginx/conf/pre-staticfiles-global.conf;
#  include /usr/local/nginx/conf/staticfiles.conf;
   include /usr/local/nginx/conf/php.conf;
 
  include /usr/local/nginx/conf/drop.conf;
  #include /usr/local/nginx/conf/errorpage.conf;
  include /usr/local/nginx/conf/vts_server.conf;
}

 

To clarify, when NOT using private windows it gives the 403 directly, so named the second time when using it in a Private Window.

I remember I had the same kind of issue when regeneration the password, don't remember if I got 401 or 403. But the only way that I could resolve it was to delete the vhost and recreate it.

Will give that a try (when time allows it)

 

EckyBrazzz was thinking.... Let's beat CMM and install it with an option of 22 and delete the database, all wp-content, FTP, etc.

Result:


Great passwd is working, but without javascript

Edit: upstream backend not working, can access( yep, with java) on the netdata http://serverip:/19999

csf did't open that port

Code (Text):

# Allow incoming TCP ports
TCP_IN = "20,21,667,25,53,80,110,111,143,161,443,465,587,993,995,1110,1186,1194,2049,81,9418,30001:50011"

# Allow outgoing TCP ports
TCP_OUT = "2525,465,111,2049,1110,1194,9418,20,21,22,25,53,80,110,113,443,587,993,995,8080"

# Allow incoming UDP ports
UDP_IN = "67,68,111,2049,1110,33434:33534,20,21,53"

# Allow outgoing UDP ports
# To allow outgoing traceroute add 33434:33523 to this list
UDP_OUT = "67,68,111,2049,1110,33434:33534,20,21,53,113,123,8080"

Reinventing the wheel made easy....@eva2000

Excluding #include /usr/local/nginx/conf/staticfiles.conf; resolved the 403 error.

Current vhost.conf

Code (Text):

#x# HTTPS-DEFAULT
 {
   # the netdata server
   server 127.0.0.1:19999;
   keepalive 64;
}

 server {
  listen 443 ssl http2;
  server_name s$vhost www.s$vhost;

  include /usr/local/nginx/conf/ssl/s$vhost/s$vhost.crt.key.conf;
  include /usr/local/nginx/conf/ssl_include.conf;

  # cloudflare authenticated origin pull cert community.centminmod.com/threads/13847/
  #ssl_client_certificate /usr/local/nginx/conf/ssl/cloudflare/s$vhost/origin.crt;
  #ssl_verify_client on;
  http2_max_field_size 16k;
  http2_max_header_size 32k;
  # mozilla recommended
  ssl_ciphers TLS13-AES-128-GCM-SHA256:TLS13-AES-256-GCM-SHA384:TLS13-CHACHA20-POLY1305-SHA256:TLS13-AES-128-CCM-8-SHA256:TLS13-AES-128-CCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS;
  ssl_prefer_server_ciphers   on;
  #add_header Alternate-Protocol  443:npn-spdy/3;

  # before enabling HSTS line below read centminmod.com/nginx_domain_dns_setup.html#hsts
  #add_header Strict-Transport-Security "max-age=31536000; includeSubdomains;";
  #add_header X-Frame-Options SAMEORIGIN;
  add_header X-Xss-Protection "1; mode=block" always;
  add_header X-Content-Type-Options "nosniff" always;
  #add_header Referrer-Policy "strict-origin-when-cross-origin";
  #spdy_headers_comp 5;
  ssl_buffer_size 1369;
  ssl_session_tickets on;
 
  # enable ocsp stapling
  resolver 8.8.8.8 8.8.4.4 1.1.1.1 1.0.0.1 valid=10m;
  resolver_timeout 10s;
  ssl_stapling on;
  ssl_stapling_verify on;

  # limit_conn limit_per_ip 16;
  # ssi  on;

  access_log /home/nginx/domains/s$vhost/log/access.log combined buffer=256k flush=5m;
  error_log /home/nginx/domains/s$vhost/log/error.log;

  #include /usr/local/nginx/conf/autoprotect/s$vhost/autoprotect-s$vhost.conf;
  root /home/nginx/domains/s$vhost/public;
  # uncomment cloudflare.conf include if using cloudflare for
  # server and/or vhost site
  #include /usr/local/nginx/conf/cloudflare.conf;
  include /usr/local/nginx/conf/503include-main.conf;


  # Enables directory listings when index file not found
  #autoindex  on;


location ~* / {
    limit_req zone=xwplogin burst=1 nodelay;
    #limit_conn xwpconlimit 30;
    auth_basic "Private";
    auth_basic_user_file /home/nginx/domains/s$vhost/htpasswd_wplogin;
    #include /usr/local/nginx/conf/php-wpsc.conf;
    include /usr/local/nginx/conf/503include-only.conf;
    proxy_set_header X-Forwarded-Host $host;
    proxy_set_header X-Forwarded-Server $host;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_pass http://backend;
    proxy_http_version 1.1;
    proxy_pass_request_headers on;
    proxy_set_header Connection "keep-alive";
    proxy_store off;


    }


}

 

Well, taking back the fact that I can connect op port 19999, did some changes in /etc/netdata config and tested with several VPN's

Code (Text):

[web]
        # x-frame-options response header =
        allow connections from = localhost mystaticIP
        # allow dashboard from = localhost *

 

@eva2000 You marked one of my previous posts as interesting. I guess it's the part about "csf didn't open that port".

And that part is worrying me also. I did not open port 19999 and still are able to access it.

 

The new version v1.15.0 Changelog - Netdata Documentation has a lot of bug fixes and improved memory usage. It`s even more light weight than before. I did not find that on the bugfixes but on the forum
Let the cron do it`s job and wait to see the result !

 

Want to edit the first post, but the forum won't allow it, didn't know that, so it's going to be a new post like "install-netdata-system-monitor-on-cmm - PART 2" The same that @eva2000 does here https://community.centminmod.com/posts/51900/ with the private testing of stuff/addons. Only I have to reinvent the same thing again, while the solution is almost ready (suggestion @eva2000 )

But for the meanwhile, I keep posting new things here until I fixed the password and the access to domain issue so that PART 2 has all the things at once.