Beta Branch addons/netdata.sh - new system monitor addon

Centmin Mod has addons/linfo.sh addon for basic system monitoring already. However, I came across an much better system monitoring application called netdata. It's so much easier to install and configure than munin so will be much easier to automate the installation and configuration via new addons/netdata.sh addon for Centmin Mod 123.09beta01+ and higher.

• More info on netdata at Home · firehol/netdata Wiki · GitHub
• Memory Requirements · firehol/netdata Wiki · GitHub
• Performance · firehol/netdata Wiki · GitHub
• Internal Plugins · firehol/netdata Wiki · GitHub
  
• External Plugins · firehol/netdata Wiki · GitHub

Still working on the installer addon and testing it internally for now. But here's screenshots from local Virtualbox test install on CentOS 7.2 based Centmin Mod 123.09beta01 LEMP stack.

Enjoy

Note netdata does live monitoring !

---

 

And when Nginx is properly configured for netdata monitoring

 

Wow netdata author, Costa has other developed tools which maybe useful for Centmin Mod LEMP stack installs too - from centminmod · Issue #214 · firehol/netdata · GitHub

including anti-ddos related tools !

 

wow netdata has custom dashboard support too Custom Dashboards · firehol/netdata Wiki · GitHub

 

Digging into netdata reveals even more awesome features. Generating moment in time snapsnot system data via netdata api can output stats in json, csv, html and google chart formats generate snapshot statistics to html file ? · Issue #217 · firehol/netdata · GitHub !

List of available charts

Code (Text):

cat /etc/netdata/netdata.conf | grep "^\[" | cut -d '[' -f 2 | cut -d ']' -f 1 | grep -v ^global$ | grep -v "^plugin" | sort -u
apps.cpu
apps.cpu_system
apps.cpu_user
apps.files
apps.lreads
apps.lwrites
apps.major_faults
apps.mem
apps.minor_faults
apps.pipes
apps.preads
apps.processes
apps.pwrites
apps.sockets
apps.threads
cpu.cpu0
cpu.cpu0_interrupts
cpu.cpu0_softirqs
cpu.cpu1
cpu.cpu1_interrupts
cpu.cpu1_softirqs
cpu.cpu2
cpu.cpu2_interrupts
cpu.cpu2_softirqs
cpu.cpu3
cpu.cpu3_interrupts
cpu.cpu3_softirqs
cpu.cpu4
cpu.cpu4_interrupts
cpu.cpu4_softirqs
cpu.cpu5
cpu.cpu5_interrupts
cpu.cpu5_softirqs
cpu.cpu6
cpu.cpu6_interrupts
cpu.cpu6_softirqs
cpu.cpu7
cpu.cpu7_interrupts
cpu.cpu7_softirqs
disk_avgsz.md1
disk_avgsz.md2
disk_avgsz.md4
disk_avgsz.sda
disk_avgsz.sdb
disk_avgsz.sdc
disk_await.sda
disk_await.sdb
disk_await.sdc
disk_backlog.sda
disk_backlog.sdb
disk_backlog.sdc
disk_iotime.sda
disk_iotime.sdb
disk_iotime.sdc
disk.md1
disk.md2
disk.md4
disk_mops.sda
disk_mops.sdb
disk_mops.sdc
disk_ops.md1
disk_ops.md2
disk_ops.md4
disk_ops.sda
disk_ops.sdb
disk_ops.sdc
disk_qops.sda
disk_qops.sdb
disk_qops.sdc
disk.sda
disk.sdb
disk.sdc
disk_svctm.sda
disk_svctm.sdb
disk_svctm.sdc
disk_util.sda
disk_util.sdb
disk_util.sdc
example.random
example.random2
groups.cpu
groups.cpu_system
groups.cpu_user
groups.files
groups.lreads
groups.lwrites
groups.major_faults
groups.mem
groups.minor_faults
groups.pipes
groups.preads
groups.processes
groups.pwrites
groups.sockets
groups.threads
ipv4.bcast
ipv4.bcastpkts
ipv4.errors
ipv4.fragsin
ipv4.fragsout
ipv4.packets
ipv4.tcperrors
ipv4.tcphandshake
ipv4.tcppackets
ipv4.tcpsock
ipv4.udperrors
ipv4.udppackets
ipv6.ect
ipv6.errors
ipv6.icmp
ipv6.icmpmldv2
ipv6.icmpneighbor
ipv6.icmptypes
ipv6.mcast
ipv6.mcastpkts
ipv6.packets
ipv6.udppackets
mem.committed
mem.kernel
mem.pgfaults
mem.slab
mem.writeback
mysql_local.binlog_cache
mysql_local.binlog_stmt_cache
mysql_local.connection_errors
mysql_local.connections
mysql_local.handlers
mysql_local.innodb_cur_row_lock
mysql_local.innodb_io
mysql_local.innodb_io_ops
mysql_local.innodb_io_pending_ops
mysql_local.innodb_log
mysql_local.innodb_os_log
mysql_local.innodb_os_log_io
mysql_local.innodb_rows
mysql_local.join_issues
mysql_local.net
mysql_local.queries
mysql_local.sort_issues
mysql_local.table_locks
mysql_local.thread_cache_misses
mysql_local.threads
mysql_local.tmp
netdata.apps_cpu
netdata.apps_files
netdata.clients
netdata.net
netdata.plugin_chartsd_example
netdata.plugin_chartsd_mysql
netdata.plugin_chartsd_postfix
netdata.plugin_chartsd_sensors
netdata.plugin_proc_cpu
netdata.plugin_tc_cpu
netdata.plugin_tc_time
netdata.requests
netdata.server_cpu
net.eth0
netfilter.conntrack_changes
netfilter.conntrack_errors
netfilter.conntrack_expect
netfilter.conntrack_new
netfilter.conntrack_search
netfilter.conntrack_sockets
net_packets.eth0
postfix.qemails
postfix.qsize
sensors.temp_coretemp_0_platform_coretemp_0
system.active_processes
system.cpu
system.ctxt
system.entropy
system.forks
system.idlejitter
system.interrupts
system.intr
system.io
system.ipv4
system.ipv6
system.load
system.processes
system.ram
system.softirqs
system.swap
system.swapio
users.cpu
users.cpu_system
users.cpu_user
users.files
users.lreads
users.lwrites
users.major_faults
users.mem
users.minor_faults
users.pipes
users.preads
users.processes
users.pwrites
users.sockets
users.threads

Centmin Mod addons/netdata.sh will now also include a option to invoke netdata api to generate csv formatted snapshots via

Code (Text):

addons/netdata.sh snapshot

You'll be able to define which charts you want snapshot option to generate for as well via CHARTS variable

Code (Text):

CHARTS='system.cpu system.load system.ram system.io system.swap system.processes system.intr system.entropy nginx.connections nginx.requests nginx.connections_status nginx.connect_rate netdata.server_cpu netdata.clients netdata.requests'

 

Optimising Netdata Memory Usage Profiles

netdata memory usage will need auto tweaking in addons/netdata.sh script to auto detect server's memory and cpu resources and auto optimise both the netdata history entries limit and update frequency to ensure the server has enough free memory to service netdata's requirements. You can see discussion at history = 3600 changes & effects ? · Issue #221 · firehol/netdata · GitHub

Currently, i scripted addons/netdata.sh to adjust history and update frequency to between 8hrs, 12hrs or 24hrs and 1, 2 or 4 second intervals depending on the detected system memory free/available.

For example, if you have less than 512MB available memory free/available, then adjust netdata.conf update interval from 1 to 4 seconds and history to 7200 entries (7200 x 4 = 28800 = 8hrs). Based on 2000 dimensions memory usage would be estimated at 7200 x 4 bytes x 2000 dimensions = ~55 MB of memory required. If you have 2.6.32+ kernel with memory dedup support, addons/netdata.sh will auto enable memory dedup at kernel level which can save around 40-55% memory usage. So that 55 MB memory required would be reduced to around 25-33 MB for a server which has less than 512MB free memory detected.

Currently, for test system I have set to 1 second interval updates and 86,400 history entries = ~465.8 MB

Code (Text):

smem -tuk
User     Count     Swap      USS      PSS      RSS 
nobody       1        0   656.0K   719.0K     1.2M 
dbus         1        0   880.0K   957.0K     1.8M 
ntp          1        0  1004.0K     1.1M     2.3M 
polkitd      1        0     4.9M     5.7M     8.2M 
postfix      4        0     5.5M     6.7M    16.8M 
redis        6        0    14.9M    15.4M    21.4M 
netdata      3        0    73.6M    75.1M   469.5M 
mysql        1        0    75.4M    75.7M    78.0M 
root        25        0   100.2M   128.5M   214.9M 
nginx       14        0   242.8M   277.8M   472.1M 
---------------------------------------------------
            57        0   519.8M   587.7M     1.3G

 

Netdata + Nginx Reverse Proxy Configuration

For addons/netdata.sh addon it will auto configure the netdata installation behind a front end Centmin Mod Nginx reverse proxy in a directory off the main hostname of the server i.e. host.domain.com/netdata which will be http authentication password protected with optional IP address restricted. The netdata.conf will be configured to bind to 127.0.0.1 so default port 19999 will not be public net accessible and only the host.domain.com/netdata url will be accessiable.

You can see now that my test server's netdata instance is behind Centmin Mod Nginx reverse proxy, that netdata reported stats for Nginx have more activity as expected

 

addons/netdata.sh logrotation support

Netdata has it's own logging at /var/log/netdata. The addons/netdata.sh addon will also configure and setup system logrotation to ensure the log sizes don't get out of control

Code (Text):

ls -lah /var/log/netdata/
total 4.2M
drwxrwxr-x   2 netdata netdata   67 Apr 12 19:41 .
drwxr-xr-x. 14 root    root    4.0K Apr 12 19:41 ..
-rw-r--r--   1 root    root    3.5M Apr 15 14:45 access.log
-rw-r--r--   1 root    root       0 Apr 12 19:41 debug.log
-rw-r--r--   1 root    root    696K Apr 15 15:09 error.log
-rw-r--r--   1 netdata netdata    0 Apr 12 19:41 .keep

 

wow netdata continues to improve with shell integration too prometheus support & shell scripts integration with netdata by ktsaou · Pull Request #1504 · firehol/netdata · GitHub

 

addons/netdata.sh part 2 preview continued addons/netdata.sh - new system monitor addon - part 2