Welcome to Centmin Mod Community
Become a Member

Sysadmin Glances IOW/s

Discussion in 'System Administration' started by upgrade81, Mar 7, 2018.

Tags:
  1. upgrade81

    upgrade81 Premium Member Premium Member

    133
    6
    18
    Sep 5, 2016
    Italy
    Ratings:
    +8
    Local Time:
    5:00 AM
    1.13.8
    10
    Hello, it is recently that I use VM I have always used dedicated.

    Done because I see the% of iowait a little strange (10 15% ssd disks) using top2 (glances)

    Also I can not understand (as photos) what is IOW/S related to each process and why report so high redis?

    grabilla.g20888.png
     
  2. eva2000

    eva2000 Administrator Staff Member

    36,040
    7,906
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,188
    Local Time:
    1:00 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    glances has overhead itself when running to verify try using sar -d and pidstat -d outlined in 20 Useful Commands of 'Sysstat' Utilities (mpstat, pidstat, iostat and sar) for Linux Performance Monitoring to double check

    i.e. check process id for redis-server for 10 runs at 1 sec interval for disk stats + cpu and memory
    Code (Text):
    pidstat -durh -p $(pidof redis-server) 1 10
    


    but yes redis by default has disk backed persistence as redis server isn't just used for caching but for persistent storage like mysql as well. See these threads

    in redis.conf
    Code (Text):
       ################################ SNAPSHOTTING  ################################
       #
       # Save the DB on disk:
       #
       #   save <seconds> <changes>
       #
       #   Will save the DB if both the given number of seconds and the given
       #   number of write operations against the DB occurred.
       #
       #   In the example below the behaviour will be to save:
       #   after 900 sec (15 min) if at least 1 key changed
       #   after 300 sec (5 min) if at least 10 keys changed
       #   after 60 sec if at least 10000 keys changed
       #
       #   Note: you can disable saving at all commenting all the "save" lines.
       #
       #   It is also possible to remove all the previously configured save
       #   points by adding a save directive with a single empty string argument
       #   like in the following example:
       #
       #   save ""
      
       save 900 1
       save 300 10
       save 60 10000
    
     
    Last edited: Mar 7, 2018
  3. eva2000

    eva2000 Administrator Staff Member

    36,040
    7,906
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,188
    Local Time:
    1:00 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    Also what are you using redis for ? just caching or storing too ?
     
  4. upgrade81

    upgrade81 Premium Member Premium Member

    133
    6
    18
    Sep 5, 2016
    Italy
    Ratings:
    +8
    Local Time:
    5:00 AM
    1.13.8
    10
    Thanks for the reply

    on this vm I use it for wordpress caching (nginx level) and object cache.

    here is the result of pidstat, in a time with low traffic

    Code (Text):
    03/07/2018      _x86_64_        (2 CPU)
    
    #      Time   UID       PID    %usr %system  %guest    %CPU   CPU  minflt/s  majflt/s     VSZ    RSS   %MEM   kB_rd/s   kB_wr/s kB_ccwr/s  Command
     1520383732   994       840    0.00    0.00    0.00    0.00     0      9.00      0.00 1711820 314548   1.91      0.00      4.00      0.00  redis-server
    
    #      Time   UID       PID    %usr %system  %guest    %CPU   CPU  minflt/s  majflt/s     VSZ    RSS   %MEM   kB_rd/s   kB_wr/s kB_ccwr/s  Command
     1520383733   994       840    0.00    1.00    0.00    1.00     0    469.00      0.00 1711820 314548   1.91      0.00    328.00      0.00  redis-server
    
    #      Time   UID       PID    %usr %system  %guest    %CPU   CPU  minflt/s  majflt/s     VSZ    RSS   %MEM   kB_rd/s   kB_wr/s kB_ccwr/s  Command
     1520383734   994       840    0.00    3.00    0.00    3.00     0     44.00      0.00 1711820 314548   1.91      0.00  33268.00      0.00  redis-server
    
    #      Time   UID       PID    %usr %system  %guest    %CPU   CPU  minflt/s  majflt/s     VSZ    RSS   %MEM   kB_rd/s   kB_wr/s kB_ccwr/s  Command
     1520383735   994       840    0.00    3.00    0.00    3.00     0     15.00      0.00 1711820 314548   1.91      0.00  22600.00      0.00  redis-server
    
    #      Time   UID       PID    %usr %system  %guest    %CPU   CPU  minflt/s  majflt/s     VSZ    RSS   %MEM   kB_rd/s   kB_wr/s kB_ccwr/s  Command
     1520383736   994       840    3.00    3.00    0.00    6.00     0    441.00      0.00 1711820 314780   1.92      0.00  44120.00      0.00  redis-server
    
    #      Time   UID       PID    %usr %system  %guest    %CPU   CPU  minflt/s  majflt/s     VSZ    RSS   %MEM   kB_rd/s   kB_wr/s kB_ccwr/s  Command
     1520383737   994       840    0.00    0.00    0.00    0.00     0      3.00      0.00 1711820 314780   1.92      0.00      0.00      0.00  redis-server
    
    #      Time   UID       PID    %usr %system  %guest    %CPU   CPU  minflt/s  majflt/s     VSZ    RSS   %MEM   kB_rd/s   kB_wr/s kB_ccwr/s  Command
     1520383738   994       840    0.00    1.00    0.00    1.00     0     26.00      0.00 1711820 314780   1.92      0.00      8.00      0.00  redis-server
    
    #      Time   UID       PID    %usr %system  %guest    %CPU   CPU  minflt/s  majflt/s     VSZ    RSS   %MEM   kB_rd/s   kB_wr/s kB_ccwr/s  Command
     1520383739   994       840    0.00    3.00    0.00    3.00     0     51.00      0.00 1711820 314872   1.92      0.00  33336.00      0.00  redis-server
    
    #      Time   UID       PID    %usr %system  %guest    %CPU   CPU  minflt/s  majflt/s     VSZ    RSS   %MEM   kB_rd/s   kB_wr/s kB_ccwr/s  Command
     1520383740   994       840    0.00    1.00    0.00    1.00     0     16.00      0.00 1711820 314872   1.92      0.00      4.00      0.00  redis-server
    
    #      Time   UID       PID    %usr %system  %guest    %CPU   CPU  minflt/s  majflt/s     VSZ    RSS   %MEM   kB_rd/s   kB_wr/s kB_ccwr/s  Command
     1520383741   994       840    4.00    0.00    0.00    4.00     1     74.00      0.00 1711820 314904   1.92      0.00  33344.00      0.00  redis-server
    


    Here is my redisinfo.

    Code:
    redis-cli 4.0.8
    --------------------------------------------------------------------------
    redis-cli info
    # Server
    redis_version:4.0.8
    redis_git_sha1:00000000
    redis_git_dirty:0
    redis_build_id:32fcc4a764b07c42
    redis_mode:standalone
    os:Linux 4.15.3-1.el7.elrepo.x86_64 x86_64
    arch_bits:64
    multiplexing_api:epoll
    atomicvar_api:atomic-builtin
    gcc_version:4.8.5
    process_id:840
    run_id:0018c4ab3acf7435dff07ddfdb2ecef5e8bd3be2
    tcp_port:6379
    uptime_in_seconds:1138361
    uptime_in_days:13
    hz:10
    lru_clock:10433585
    executable:/usr/bin/redis-server
    config_file:/etc/redis.conf
    
    # Clients
    connected_clients:6
    client_longest_output_list:0
    client_biggest_input_buf:0
    blocked_clients:0
    
    # Memory
    used_memory:282991400
    used_memory_human:269.88M
    used_memory_rss:313245696
    used_memory_rss_human:298.73M
    used_memory_peak:935529888
    used_memory_peak_human:892.19M
    used_memory_peak_perc:30.25%
    used_memory_overhead:6176432
    used_memory_startup:761608
    used_memory_dataset:276814968
    used_memory_dataset_perc:98.08%
    total_system_memory:16821084160
    total_system_memory_human:15.67G
    used_memory_lua:436224
    used_memory_lua_human:426.00K
    maxmemory:326107136
    maxmemory_human:311.00M
    maxmemory_policy:allkeys-lru
    mem_fragmentation_ratio:1.11
    mem_allocator:jemalloc-3.6.0
    active_defrag_running:0
    lazyfree_pending_objects:0
    
    # Persistence
    loading:0
    rdb_changes_since_last_save:4949
    rdb_bgsave_in_progress:0
    rdb_last_save_time:1520382820
    rdb_last_bgsave_status:ok
    rdb_last_bgsave_time_sec:1
    rdb_current_bgsave_time_sec:-1
    rdb_last_cow_size:10559488
    aof_enabled:1
    aof_rewrite_in_progress:0
    aof_rewrite_scheduled:0
    aof_last_rewrite_time_sec:2
    aof_current_rewrite_time_sec:-1
    aof_last_bgrewrite_status:ok
    aof_last_write_status:ok
    aof_last_cow_size:50216960
    aof_current_size:322936761
    aof_base_size:288875076
    aof_pending_rewrite:0
    aof_buffer_length:0
    aof_rewrite_buffer_length:0
    aof_pending_bio_fsync:0
    aof_delayed_fsync:53
    
    # Stats
    total_connections_received:1092317
    total_commands_processed:90661434
    instantaneous_ops_per_sec:69
    total_net_input_bytes:17937074490349
    total_net_output_bytes:950471106900
    instantaneous_input_kbps:20791.25
    instantaneous_output_kbps:960.71
    rejected_connections:0
    sync_full:0
    sync_partial_ok:0
    sync_partial_err:0
    expired_keys:1733603
    evicted_keys:788473
    keyspace_hits:35576096
    keyspace_misses:30644414
    pubsub_channels:0
    pubsub_patterns:0
    latest_fork_usec:4999
    migrate_cached_sockets:0
    slave_expires_tracked_keys:0
    active_defrag_hits:0
    active_defrag_misses:0
    active_defrag_key_hits:0
    active_defrag_key_misses:0
    
    # Replication
    role:master
    connected_slaves:0
    master_replid:c475813f5bc5000efc585c46e1e70ffbdc574fe8
    master_replid2:0000000000000000000000000000000000000000
    master_repl_offset:0
    second_repl_offset:-1
    repl_backlog_active:0
    repl_backlog_size:1048576
    repl_backlog_first_byte_offset:0
    repl_backlog_histlen:0
    
    # CPU
    used_cpu_sys:29126.48
    used_cpu_user:20478.17
    used_cpu_sys_children:14892.86
    used_cpu_user_children:9008.40
    
    # Cluster
    cluster_enabled:0
    
    # Keyspace
    db0:keys=5257,expires=0,avg_ttl=0
    db7:keys=45436,expires=45436,avg_ttl=29021885
    
    --------------------------------------------------------------------------
    redis-cli --intrinsic-latency 10
    Max latency so far: 3 microseconds.
    Max latency so far: 16 microseconds.
    Max latency so far: 77 microseconds.
    Max latency so far: 139 microseconds.
    Max latency so far: 2018 microseconds.
    Max latency so far: 2232 microseconds.
    Max latency so far: 2408 microseconds.
    
    3342355 total runs (avg latency: 2.9919 microseconds / 2991.90 nanoseconds per run).
    Worst run took 805x longer than the average latency.
    
    --------------------------------------------------------------------------
    vmstat 1 10
    procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
     r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
     1  1      0 1198884 263408 10646972    0    0    31 14356   16   13  4  3 89  4  0
     0  0      0 1668700 263408 10180472    0    0     0 120480 1419  645  3  9 63 25  0
     0  0      0 1636112 263408 10213792    0    0    12    16  886 1298  4  3 93  0  0
     0  1      0 1602920 263408 10247120    0    0    48 66616 1304 2408  4  3 84  9  0
     0  0      0 1602668 263408 10247056    0    0     0  7612  528  483  0  1 94  6  0
     0  0      0 1538592 263408 10313712    0    0    24 30680 2315 5296 12  6 79  3  0
     0  0      0 1538628 263408 10313652    0    0     0 36116  422  320  0  1 95  5  0
     0  0      0 1467768 263408 10380116    0    0    20  5824 2457 4812 10  6 82  2  0
     0  0      0 1433880 263408 10413420    0    0     8 65280 1309 2416  4  4 86  7  0
     0  0      0 1433508 263408 10413428    0    0     0    52  718  477  1  1 99  0  0
    --------------------------------------------------------------------------
    
    smem -kU redis
      PID User     Command                         Swap      USS      PSS      RSS
      840 redis    /usr/bin/redis-server 127.0        0   296.5M   296.6M   299.0M
    --------------------------------------------------------------------------
    
    --------------------------------------------------------------------------
    smem -kmU redis
    Map                                       PIDs   AVGPSS      PSS
    [vdso]                                       1        0        0
    [vsyscall]                                   1        0        0
    [vvar]                                       1        0        0
    /usr/lib64/libdl-2.17.so                     1     8.0K     8.0K
    /usr/lib64/ld-2.17.so                        1    11.0K    11.0K
    /usr/lib64/libpthread-2.17.so                1    11.0K    11.0K
    [stack]                                      1    16.0K    16.0K
    /usr/lib64/libm-2.17.so                      1    20.0K    20.0K
    /usr/lib64/libjemalloc.so.1                  1    59.0K    59.0K
    /usr/lib/locale/locale-archive               1    73.0K    73.0K
    /usr/lib64/libc-2.17.so                      1    74.0K    74.0K
    /usr/bin/redis-server                        1   832.0K   832.0K
    <anonymous>                                  1   295.6M   295.6M
    --------------------------------------------------------------------------
    
    --------------------------------------------------------------------------
    smem -tpk
      PID User     Command                         Swap      USS      PSS      RSS
      876 root     /sbin/agetty --noclear tty1        0   128.0K   180.0K     1.6M
      479 root     /sbin/rngd -f                      0   180.0K   225.0K     1.6M
      496 root     /usr/sbin/irqbalance --fore        0   268.0K   470.0K     2.8M
      451 root     /sbin/auditd                       0   508.0K   585.0K     2.3M
      489 chrony   /usr/sbin/chronyd                  0   612.0K   735.0K     3.1M
    20169 root     /bin/bash ./redisinfo.sh           0   252.0K   802.0K     3.1M
      503 root     /usr/sbin/crond -n                 0   736.0K   883.0K     3.0M
      414 root     /usr/lib/systemd/systemd-ud        0   864.0K   947.0K     2.9M
      486 dbus     /bin/dbus-daemon --system -        0   908.0K  1017.0K     3.0M
      501 root     /usr/lib/systemd/systemd-lo        0     1.1M     1.2M     3.1M
     1257 root     /usr/libexec/postfix/master        0     1.2M     1.5M     4.6M
     1260 postfix  qmgr -l -t unix -u                 0     1.3M     1.9M     6.5M
     7815 postfix  pickup -l -t unix -u               0     1.4M     1.9M     6.5M
     2323 root     /usr/sbin/sshd -D                  0  1008.0K     2.0M     7.6M
     4695 root     pure-ftpd (SERVER)                 0     1.7M     2.2M     5.5M
    18363 root     -bash                              0     1.8M     2.3M     4.8M
        1 root     /usr/lib/systemd/systemd --        0     2.4M     2.5M     4.6M
    29467 postfix  tlsmgr -l -t unix -u               0     2.4M     2.9M     7.8M
    18361 root     sshd: [email protected]/0                   0     2.0M     3.0M     8.5M
      482 root     /usr/sbin/haveged -w 4067 -        0     3.1M     3.1M     4.6M
      495 root     /usr/sbin/NetworkManager --        0     2.4M     3.4M     9.1M
     3144 root     php-fpm: master process (/u        0     1.2M     3.8M    11.7M
      499 polkitd  /usr/lib/polkit-1/polkitd -        0     5.7M     6.4M    10.4M
    20241 root     python /usr/bin/smem -tpk          0     8.2M     8.4M    10.6M
      845 root     /usr/bin/python -Es /usr/sb        0    10.7M    11.8M    17.2M
      497 root     /usr/sbin/rsyslogd -n              0     3.7M    17.8M    55.2M
    30879 root     lfd - sleepin                      0    25.0M    25.0M    27.1M
      389 root     /usr/lib/systemd/systemd-jo        0     7.2M    27.1M    74.8M
     3132 root     nginx: master process /usr/        0   992.0K    29.7M    88.0M
     3134 nginx    nginx: worker process              0    28.4M    59.5M   121.6M
     3133 nginx    nginx: worker process              0    31.3M    62.4M   124.4M
    19057 nginx    php-fpm: pool www                  0    27.7M    81.5M   142.2M
    19043 nginx    php-fpm: pool www                  0    30.4M    84.2M   144.9M
      840 redis    /usr/bin/redis-server 127.0        0   296.5M   296.7M   299.0M
     1163 mysql    /usr/sbin/mysqld                   0     3.5G     3.5G     3.5G
    -------------------------------------------------------------------------------
       35 8                                           0     4.0G     4.2G     4.7G
    --------------------------------------------------------------------------
    
    --------------------------------------------------------------------------
    free -mlt
                  total        used        free      shared  buff/cache   available
    Mem:          16041        4215        1399         280       10426       11153
    Low:          16041       14642        1399
    High:             0           0           0
    Swap:          1022           0        1022
    Total:        17064        4215        2422
    

    Thanks
     
  5. eva2000

    eva2000 Administrator Staff Member

    36,040
    7,906
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,188
    Local Time:
    1:00 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    looks good at low traffic what does glances report ?
     
  6. upgrade81

    upgrade81 Premium Member Premium Member

    133
    6
    18
    Sep 5, 2016
    Italy
    Ratings:
    +8
    Local Time:
    5:00 AM
    1.13.8
    10
    there is half of the traffic compared to the previous screenshots

    grabilla.g57648.png
     
  7. upgrade81

    upgrade81 Premium Member Premium Member

    133
    6
    18
    Sep 5, 2016
    Italy
    Ratings:
    +8
    Local Time:
    5:00 AM
    1.13.8
    10
    I have not figured out why there are so many php processes despite being wordpress cached. I have to investigate the template and plugin
    Bah
     
  8. eva2000

    eva2000 Administrator Staff Member

    36,040
    7,906
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,188
    Local Time:
    1:00 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    if you are using php-fpm with dynamic or static process manager instead of default ondemand then idle/spare php-fpm processes are normal. But glances itself has overhead while running so better to use native tools like top, sar, pidstat, vmstat, iostat etc to figure out usage instead of relying on glances.

    you have to figure out what php-fpm and redis are doing. That is left up to you to figure out.

    However, there's many linux tools and scripts that can help you figure out what was causing the load issues and when.

    Tools and commands you will want to read up on and learn for basic system admin tasks and troubleshooting.
     
    • Like Like x 1
..