Discover Centmin Mod today
Register Now

Beta Branch addons/netdata.sh - new system monitor addon

Discussion in 'Beta release code' started by eva2000, Apr 13, 2016.

Thread Status:
Not open for further replies.
  1. eva2000

    eva2000 Administrator Staff Member

    30,165
    6,784
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,136
    Local Time:
    5:44 AM
    Nginx 1.13.x
    MariaDB 5.5
    Centmin Mod has addons/linfo.sh addon for basic system monitoring already. However, I came across an much better system monitoring application called netdata. It's so much easier to install and configure than munin so will be much easier to automate the installation and configuration via new addons/netdata.sh addon for Centmin Mod 123.09beta01+ and higher.

    Still working on the installer addon and testing it internally for now. But here's screenshots from local Virtualbox test install on CentOS 7.2 based Centmin Mod 123.09beta01 LEMP stack.

    Enjoy :D

    Note netdata does live monitoring !

    netdata-localinstall-00.png
    netdata-localinstall-01.png
    netdata-localinstall-02.png
    netdata-localinstall-03.png
    netdata-localinstall-04.png
    netdata-localinstall-05.png
    netdata-localinstall-06.png
    netdata-localinstall-07.png
    netdata-localinstall-08.png
    netdata-localinstall-09.png
    netdata-localinstall-010.png
    netdata-localinstall-011.png
    netdata-localinstall-012.png
    netdata-localinstall-013.png
    netdata-localinstall-014.png
    netdata-localinstall-015.png
    netdata-localinstall-016.png
    netdata-localinstall-017.png
    netdata-localinstall-018.png
    netdata-localinstall-019.png
    netdata-localinstall-020.png
    netdata-localinstall-021.png
    netdata-localinstall-022.png
    netdata-localinstall-023.png
    netdata-localinstall-024.png
    netdata-localinstall-025.png
    netdata-localinstall-026.png
    netdata-localinstall-027.png
    netdata-localinstall-028.png
    netdata-localinstall-029.png
    netdata-localinstall-030.png
    netdata-localinstall-031.png
     
    Last edited: Apr 13, 2016
    • Winner Winner x 4
    • Like Like x 1
  2. eva2000

    eva2000 Administrator Staff Member

    30,165
    6,784
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,136
    Local Time:
    5:44 AM
    Nginx 1.13.x
    MariaDB 5.5
    And when Nginx is properly configured for netdata monitoring :D

    upload_2016-4-13_6-17-37.png
     
    • Like Like x 3
  3. eva2000

    eva2000 Administrator Staff Member

    30,165
    6,784
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,136
    Local Time:
    5:44 AM
    Nginx 1.13.x
    MariaDB 5.5
    Wow netdata author, Costa has other developed tools which maybe useful for Centmin Mod LEMP stack installs too - from centminmod · Issue #214 · firehol/netdata · GitHub

    including anti-ddos related tools !
    upload_2016-4-13_8-18-35.png
     
    • Agree Agree x 1
    • Useful Useful x 1
  4. eva2000

    eva2000 Administrator Staff Member

    30,165
    6,784
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,136
    Local Time:
    5:44 AM
    Nginx 1.13.x
    MariaDB 5.5
    wow netdata has custom dashboard support too Custom Dashboards · firehol/netdata Wiki · GitHub

     
    • Like Like x 1
  5. eva2000

    eva2000 Administrator Staff Member

    30,165
    6,784
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,136
    Local Time:
    5:44 AM
    Nginx 1.13.x
    MariaDB 5.5
    Digging into netdata reveals even more awesome features. Generating moment in time snapsnot system data via netdata api can output stats in json, csv, html and google chart formats generate snapshot statistics to html file ? · Issue #217 · firehol/netdata · GitHub !

    List of available charts
    Code (Text):
    cat /etc/netdata/netdata.conf | grep "^\[" | cut -d '[' -f 2 | cut -d ']' -f 1 | grep -v ^global$ | grep -v "^plugin" | sort -u
    apps.cpu
    apps.cpu_system
    apps.cpu_user
    apps.files
    apps.lreads
    apps.lwrites
    apps.major_faults
    apps.mem
    apps.minor_faults
    apps.pipes
    apps.preads
    apps.processes
    apps.pwrites
    apps.sockets
    apps.threads
    cpu.cpu0
    cpu.cpu0_interrupts
    cpu.cpu0_softirqs
    cpu.cpu1
    cpu.cpu1_interrupts
    cpu.cpu1_softirqs
    cpu.cpu2
    cpu.cpu2_interrupts
    cpu.cpu2_softirqs
    cpu.cpu3
    cpu.cpu3_interrupts
    cpu.cpu3_softirqs
    cpu.cpu4
    cpu.cpu4_interrupts
    cpu.cpu4_softirqs
    cpu.cpu5
    cpu.cpu5_interrupts
    cpu.cpu5_softirqs
    cpu.cpu6
    cpu.cpu6_interrupts
    cpu.cpu6_softirqs
    cpu.cpu7
    cpu.cpu7_interrupts
    cpu.cpu7_softirqs
    disk_avgsz.md1
    disk_avgsz.md2
    disk_avgsz.md4
    disk_avgsz.sda
    disk_avgsz.sdb
    disk_avgsz.sdc
    disk_await.sda
    disk_await.sdb
    disk_await.sdc
    disk_backlog.sda
    disk_backlog.sdb
    disk_backlog.sdc
    disk_iotime.sda
    disk_iotime.sdb
    disk_iotime.sdc
    disk.md1
    disk.md2
    disk.md4
    disk_mops.sda
    disk_mops.sdb
    disk_mops.sdc
    disk_ops.md1
    disk_ops.md2
    disk_ops.md4
    disk_ops.sda
    disk_ops.sdb
    disk_ops.sdc
    disk_qops.sda
    disk_qops.sdb
    disk_qops.sdc
    disk.sda
    disk.sdb
    disk.sdc
    disk_svctm.sda
    disk_svctm.sdb
    disk_svctm.sdc
    disk_util.sda
    disk_util.sdb
    disk_util.sdc
    example.random
    example.random2
    groups.cpu
    groups.cpu_system
    groups.cpu_user
    groups.files
    groups.lreads
    groups.lwrites
    groups.major_faults
    groups.mem
    groups.minor_faults
    groups.pipes
    groups.preads
    groups.processes
    groups.pwrites
    groups.sockets
    groups.threads
    ipv4.bcast
    ipv4.bcastpkts
    ipv4.errors
    ipv4.fragsin
    ipv4.fragsout
    ipv4.packets
    ipv4.tcperrors
    ipv4.tcphandshake
    ipv4.tcppackets
    ipv4.tcpsock
    ipv4.udperrors
    ipv4.udppackets
    ipv6.ect
    ipv6.errors
    ipv6.icmp
    ipv6.icmpmldv2
    ipv6.icmpneighbor
    ipv6.icmptypes
    ipv6.mcast
    ipv6.mcastpkts
    ipv6.packets
    ipv6.udppackets
    mem.committed
    mem.kernel
    mem.pgfaults
    mem.slab
    mem.writeback
    mysql_local.binlog_cache
    mysql_local.binlog_stmt_cache
    mysql_local.connection_errors
    mysql_local.connections
    mysql_local.handlers
    mysql_local.innodb_cur_row_lock
    mysql_local.innodb_io
    mysql_local.innodb_io_ops
    mysql_local.innodb_io_pending_ops
    mysql_local.innodb_log
    mysql_local.innodb_os_log
    mysql_local.innodb_os_log_io
    mysql_local.innodb_rows
    mysql_local.join_issues
    mysql_local.net
    mysql_local.queries
    mysql_local.sort_issues
    mysql_local.table_locks
    mysql_local.thread_cache_misses
    mysql_local.threads
    mysql_local.tmp
    netdata.apps_cpu
    netdata.apps_files
    netdata.clients
    netdata.net
    netdata.plugin_chartsd_example
    netdata.plugin_chartsd_mysql
    netdata.plugin_chartsd_postfix
    netdata.plugin_chartsd_sensors
    netdata.plugin_proc_cpu
    netdata.plugin_tc_cpu
    netdata.plugin_tc_time
    netdata.requests
    netdata.server_cpu
    net.eth0
    netfilter.conntrack_changes
    netfilter.conntrack_errors
    netfilter.conntrack_expect
    netfilter.conntrack_new
    netfilter.conntrack_search
    netfilter.conntrack_sockets
    net_packets.eth0
    postfix.qemails
    postfix.qsize
    sensors.temp_coretemp_0_platform_coretemp_0
    system.active_processes
    system.cpu
    system.ctxt
    system.entropy
    system.forks
    system.idlejitter
    system.interrupts
    system.intr
    system.io
    system.ipv4
    system.ipv6
    system.load
    system.processes
    system.ram
    system.softirqs
    system.swap
    system.swapio
    users.cpu
    users.cpu_system
    users.cpu_user
    users.files
    users.lreads
    users.lwrites
    users.major_faults
    users.mem
    users.minor_faults
    users.pipes
    users.preads
    users.processes
    users.pwrites
    users.sockets
    users.threads


    Centmin Mod addons/netdata.sh will now also include a option to invoke netdata api to generate csv formatted snapshots via
    Code (Text):
    addons/netdata.sh snapshot


    You'll be able to define which charts you want snapshot option to generate for as well via CHARTS variable
    Code (Text):
    CHARTS='system.cpu system.load system.ram system.io system.swap system.processes system.intr system.entropy nginx.connections nginx.requests nginx.connections_status nginx.connect_rate netdata.server_cpu netdata.clients netdata.requests'
     
    Last edited: Apr 14, 2016
    • Like Like x 1
  6. eva2000

    eva2000 Administrator Staff Member

    30,165
    6,784
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,136
    Local Time:
    5:44 AM
    Nginx 1.13.x
    MariaDB 5.5

    Optimising Netdata Memory Usage Profiles



    netdata memory usage will need auto tweaking in addons/netdata.sh script to auto detect server's memory and cpu resources and auto optimise both the netdata history entries limit and update frequency to ensure the server has enough free memory to service netdata's requirements. You can see discussion at history = 3600 changes & effects ? · Issue #221 · firehol/netdata · GitHub

    Currently, i scripted addons/netdata.sh to adjust history and update frequency to between 8hrs, 12hrs or 24hrs and 1, 2 or 4 second intervals depending on the detected system memory free/available.

    For example, if you have less than 512MB available memory free/available, then adjust netdata.conf update interval from 1 to 4 seconds and history to 7200 entries (7200 x 4 = 28800 = 8hrs). Based on 2000 dimensions memory usage would be estimated at 7200 x 4 bytes x 2000 dimensions = ~55 MB of memory required. If you have 2.6.32+ kernel with memory dedup support, addons/netdata.sh will auto enable memory dedup at kernel level which can save around 40-55% memory usage. So that 55 MB memory required would be reduced to around 25-33 MB for a server which has less than 512MB free memory detected.

    Currently, for test system I have set to 1 second interval updates and 86,400 history entries = ~465.8 MB

    Code (Text):
    smem -tuk
    User     Count     Swap      USS      PSS      RSS 
    nobody       1        0   656.0K   719.0K     1.2M 
    dbus         1        0   880.0K   957.0K     1.8M 
    ntp          1        0  1004.0K     1.1M     2.3M 
    polkitd      1        0     4.9M     5.7M     8.2M 
    postfix      4        0     5.5M     6.7M    16.8M 
    redis        6        0    14.9M    15.4M    21.4M 
    netdata      3        0    73.6M    75.1M   469.5M 
    mysql        1        0    75.4M    75.7M    78.0M 
    root        25        0   100.2M   128.5M   214.9M 
    nginx       14        0   242.8M   277.8M   472.1M 
    ---------------------------------------------------
                57        0   519.8M   587.7M     1.3G
    


    upload_2016-4-14_6-9-30.png
     
    Last edited: Apr 14, 2016
    • Winner Winner x 2
  7. eva2000

    eva2000 Administrator Staff Member

    30,165
    6,784
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,136
    Local Time:
    5:44 AM
    Nginx 1.13.x
    MariaDB 5.5

    Netdata + Nginx Reverse Proxy Configuration



    For addons/netdata.sh addon it will auto configure the netdata installation behind a front end Centmin Mod Nginx reverse proxy in a directory off the main hostname of the server i.e. host.domain.com/netdata which will be http authentication password protected with optional IP address restricted. The netdata.conf will be configured to bind to 127.0.0.1 so default port 19999 will not be public net accessible and only the host.domain.com/netdata url will be accessiable.

    You can see now that my test server's netdata instance is behind Centmin Mod Nginx reverse proxy, that netdata reported stats for Nginx have more activity as expected :)

    upload_2016-4-14_11-27-18.png
     
    Last edited: Apr 14, 2016
    • Winner Winner x 2
    • Useful Useful x 1
  8. eva2000

    eva2000 Administrator Staff Member

    30,165
    6,784
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,136
    Local Time:
    5:44 AM
    Nginx 1.13.x
    MariaDB 5.5

    addons/netdata.sh logrotation support



    Netdata has it's own logging at /var/log/netdata. The addons/netdata.sh addon will also configure and setup system logrotation to ensure the log sizes don't get out of control :)

    Code (Text):
    ls -lah /var/log/netdata/
    total 4.2M
    drwxrwxr-x   2 netdata netdata   67 Apr 12 19:41 .
    drwxr-xr-x. 14 root    root    4.0K Apr 12 19:41 ..
    -rw-r--r--   1 root    root    3.5M Apr 15 14:45 access.log
    -rw-r--r--   1 root    root       0 Apr 12 19:41 debug.log
    -rw-r--r--   1 root    root    696K Apr 15 15:09 error.log
    -rw-r--r--   1 netdata netdata    0 Apr 12 19:41 .keep
     
    • Like Like x 6
  9. eva2000

    eva2000 Administrator Staff Member

    30,165
    6,784
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,136
    Local Time:
    5:44 AM
    Nginx 1.13.x
    MariaDB 5.5
    • Informative Informative x 1
  10. eva2000

    eva2000 Administrator Staff Member

    30,165
    6,784
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +10,136
    Local Time:
    5:44 AM
    Nginx 1.13.x
    MariaDB 5.5
    • Like Like x 2
Thread Status:
Not open for further replies.