Discover Centmin Mod today
Register Now

Sysdig

Discussion in 'System Administration' started by eva2000, Jul 18, 2015.

Tags:
  1. eva2000

    eva2000 Administrator Staff Member

    29,016
    6,585
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +9,775
    Local Time:
    12:38 PM
    Nginx 1.13.x
    MariaDB 5.5
    Just came across Sysdig and thought I'd share with folks :)

    Sysdig Install



    Installing Sysdig on Centmin Mod LEMP stack is easy as alot of required packages are already installed by Centmin Mod.

    Code:
    rpm --import https://s3.amazonaws.com/download.draios.com/DRAIOS-GPG-KEY.public
    curl -s -o /etc/yum.repos.d/draios.repo http://download.draios.com/stable/rpm/draios.repo
    yum -y install sysdig

    Sysdig Usage



    More examples at Sysdig | Wiki | Sysdig Examples.

    Basic Command List

    Capture all the events from the live system and print them to screen
    Code:
    sysdig
    
    Capture all the events from the live system and save them to disk
    Code:
    sysdig -qw dumpfile.scap
    
    Read events from a file and print them to screen
    Code:
    sysdig -r dumpfile.scap
    
    Print all the open system calls invoked by cat
    Code:
    sysdig proc.name=cat and evt.type=open
    
    Print the name of the files opened by cat
    Code:
    ./sysdig -p"%evt.arg.name" proc.name=cat and evt.type=open
    
    List the available chisels
    Code:
    ./sysdig -cl
    
    Run the spy_ip chisel for the 192.168.1.157 IP address:
    Code:
    sysdig –c spy_ip 192.168.1.157
    
    See the top processes in terms of CPU usage
    Code:
    sysdig -c topprocs_cpu
    
    See the top processes for CPU 0
    Code:
    sysdig -c topprocs_cpu evt.cpu=0
    
    Observe the standard output of a process
    Code:
    sysdig -s4096 -A -c stdout proc.name=cat
    
    See the files where most time has been spent
    Code:
    sysdig -c topfiles_time
    
    See the files where apache spent most time
    Code:
    sysdig -c topfiles_time proc.name=httpd
    
    See the top processes in terms of I/O errors
    Code:
    sysdig -c topprocs_errors
    
    See the top files in terms of I/O errors
    Code:
    sysdig -c topfiles_errors
    
    See all the failed disk I/O calls
    Code:
    sysdig fd.type=file and evt.failed=true
    
    See all the failed file opens by httpd
    Code:
    sysdig "proc.name=httpd and evt.type=open and evt.failed=true"
    
    See the system calls where most time has been spent
    Code:
    sysdig -c topscalls_time
    
    See the top system calls returning errors
    Code:
    sysdig -c topscalls "evt.failed=true"
    
    snoop failed file opens as they occur
    Code:
    sysdig -p "%12user.name %6proc.pid %12proc.name %3fd.num %fd.typechar %fd.name" evt.type=open and evt.failed=true
    
    Print the file I/O calls that have a latency greater than 1ms:
    Code:
    sysdig -c fileslower 1
    
    Dump system activity to file, so that sysdig can be used to process it later.
    Code:
    sysdig -w trace.scap
    
    View the top network connections for a single container.
    Code:
    sysdig -pc -c topconns container.name=wordpress1
    
    See the files where apache spends the most
    time doing I/O.
    Code:
    sysdig -c topfiles_time proc.name=httpd
    
    Show all the interactive commands executed inside a given container.
    Code:
    sysdig -pc -c spy_users container.name=wordpress1
    
    Show every time a file is opened under /etc.
    Code:
    sysdig evt.type=open and fd.name contains /etc
    
    Sample install output
    Code:
    yum -y install sysdig
    Loaded plugins: fastestmirror, priorities
    draios                                                                                                  | 2.9 kB  00:00:00
    draios/x86_64/primary_db                                                                                | 8.6 kB  00:00:00
    Loading mirror speeds from cached hostfile
    * base: mirror.optus.net
    * epel: mirror.overthewire.com.au
    * extras: mirror.aarnet.edu.au
    * rpmforge: mirror.ventraip.net.au
    * updates: mirror.aarnet.edu.au
    149 packages excluded due to repository priority protections
    Resolving Dependencies
    --> Running transaction check
    ---> Package sysdig.x86_64 0:0.1.101-1 will be installed
    --> Processing Dependency: dkms for package: sysdig-0.1.101-1.x86_64
    --> Running transaction check
    ---> Package dkms.noarch 0:2.2.0.3-30.git.7c3e7c5.el7 will be installed
    --> Finished Dependency Resolution
    
    Dependencies Resolved
    
    ===============================================================================================================================
    Package                  Arch                     Version                                      Repository                Size
    ===============================================================================================================================
    Installing:
    sysdig                   x86_64                   0.1.101-1                                    draios                   1.3 M
    Installing for dependencies:
    dkms                     noarch                   2.2.0.3-30.git.7c3e7c5.el7                   epel                      77 k
    
    Transaction Summary
    ===============================================================================================================================
    Install  1 Package (+1 Dependent package)
    
    Total download size: 1.3 M
    Installed size: 4.5 M
    Downloading packages:
    (1/2): dkms-2.2.0.3-30.git.7c3e7c5.el7.noarch.rpm                                                       |  77 kB  00:00:00
    (2/2): sysdig-0.1.101-x86_64.rpm                                                                        | 1.3 MB  00:00:06
    -------------------------------------------------------------------------------------------------------------------------------
    Total                                                                                          212 kB/s | 1.3 MB  00:00:06
    Running transaction check
    Running transaction test
    Transaction test succeeded
    Running transaction
      Installing : dkms-2.2.0.3-30.git.7c3e7c5.el7.noarch                                                                      1/2
      Installing : sysdig-0.1.101-1.x86_64                                                                                     2/2
    
    Creating symlink /var/lib/dkms/sysdig/0.1.101/source ->
                     /usr/src/sysdig-0.1.101
    
    DKMS: add completed.
    
    Kernel preparation unnecessary for this kernel.  Skipping...
    
    Building module:
    cleaning build area...
    make KERNELRELEASE=3.10.0-229.7.2.el7.x86_64 -C /lib/modules/3.10.0-229.7.2.el7.x86_64/build M=/var/lib/dkms/sysdig/0.1.101/build....
    cleaning build area...
    
    DKMS: build completed.
    
    sysdig-probe:
    Running module version sanity check.
    - Original module
       - No original module exists within this kernel
    - Installation
       - Installing to /lib/modules/3.10.0-229.7.2.el7.x86_64/extra/
    Adding any weak-modules
    
    depmod...
    
    DKMS: install completed.
      Verifying  : dkms-2.2.0.3-30.git.7c3e7c5.el7.noarch                                                                      1/2
      Verifying  : sysdig-0.1.101-1.x86_64                                                                                     2/2
    
    Installed:
      sysdig.x86_64 0:0.1.101-1                                                                                        
    
    Dependency Installed:
      dkms.noarch 0:2.2.0.3-30.git.7c3e7c5.el7                                                                         
    
    Complete!
    Sysdig video explaining some of the features



    Videos

    Sysdig Youtube channel
    • Csysdig intro video:
    • Sysdig intro video:
     
    Last edited: Jul 19, 2015
  2. eva2000

    eva2000 Administrator Staff Member

    29,016
    6,585
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +9,775
    Local Time:
    12:38 PM
    Nginx 1.13.x
    MariaDB 5.5
    Code:
    sysdig -cl
    
    Category: CPU Usage
    -------------------
    spectrogram     Visualize OS latency in real time.
    subsecoffset    Visualize subsecond offset execution time.
    topcontainers_cpu
                    Top containers by CPU usage
    topprocs_cpu    Top processes by CPU usage
    
    Category: Errors
    ----------------
    topcontainers_error
                    Top containers by number of errors
    topfiles_errors Top files by number of errors
    topprocs_errors top processes by number of errors
    
    Category: I/O
    -------------
    echo_fds        Print the data read and written by processes.
    fdbytes_by      I/O bytes, aggregated by an arbitrary filter field
    fdcount_by      FD count, aggregated by an arbitrary filter field
    fdtime_by       FD time group by
    iobytes         Sum of I/O bytes on any type of FD
    iobytes_file    Sum of file I/O bytes
    spy_file        Echo any read/write made by any process to all files. Optionall
                    y, you can provide the name of one file to only intercept reads
                    /writes to that file.
    stderr          Print stderr of processes
    stdin           Print stdin of processes
    stdout          Print stdout of processes
    topcontainers_file
                    Top containers by R+W disk bytes
    topfiles_bytes  Top files by R+W bytes
    topfiles_time   Top files by time
    topprocs_file   Top processes by R+W disk bytes
    
    Category: Logs
    --------------
    spy_logs        Echo any write made by any process to a log file. Optionally, e
                    xport the events around each log message to file.
    spy_syslog      Print every message written to syslog. Optionally, export the e
                    vents around each syslog message to file.
    
    Category: Misc
    --------------
    around          Export to file the events around the where the given filter mat
                    ches.
    
    Category: Net
    -------------
    iobytes_net     Show total network I/O bytes
    spy_ip          Show the data exchanged with the given IP address
    spy_port        Show the data exchanged using the given IP port number
    topconns        Top network connections by total bytes
    topcontainers_net
                    Top containers by network I/O
    topports_server Top TCP/UDP server ports by R+W bytes
    topprocs_net    Top processes by network I/O
    
    Category: Performance
    ---------------------
    bottlenecks     Slowest system calls
    fileslower      Trace slow file I/O
    netlower        Trace slow network I/0
    proc_exec_time  Show process execution time
    scallslower     Trace slow syscalls
    topscalls       Top system calls by number of calls
    topscalls_time  Top system calls by time
    
    Category: Security
    ------------------
    list_login_shells
                    List the login shell IDs
    shellshock_detect
                    print shellshock attacks
    spy_users       Display interactive user activity
    
    Category: System State
    ----------------------
    lscontainers    List the running containers
    lsof            List (and optionally filter) the open file descriptors.
    netstat         List (and optionally filter) network connections.
    ps              List (and optionally filter) the machine processes.
    
    Use the -i flag to get detailed information about a specific chisel
     
  3. eva2000

    eva2000 Administrator Staff Member

    29,016
    6,585
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +9,775
    Local Time:
    12:38 PM
    Nginx 1.13.x
    MariaDB 5.5
    Example taking a trace file capture of tracefile.scap and querying the scap trace file for top processes. @Matt @RoldanLT @jeffwidman @palPalani @Tracy Perry @Steve Tozer @pamamolf would find Sysdig useful for sure :D

    Run command for 10 seconds and then CTRL+C exit
    Code:
    sysdig -w tracefile.scap
    Query file for top processes
    Code:
    sysdig -r tracefile.scap -ctopprocs_cpu
    CPU%            Process         PID          
    --------------------------------------------------------------------------------
    0.31%           sysdig          31858
    0.31%           nginx           3848
    0.31%           mysqld          1751
    0.00%           rs:main         609
    0.00%           bash            31804
    0.00%           auditd          581
    0.00%           tuned           611
    0.00%           rsyslogd        609
    0.00%           in:imjourn      609
    0.00%           memcached       2475
    Query file for top disk usage
    Code:
    sysdig -r tracefile.scap -ctopfiles_bytes
    Bytes           Filename     
    --------------------------------------------------------------------------------
    1.77KB          /proc/stat
    1024B           /proc/interrupts
    2B              /proc/irq/15/smp_affinity
    2B              /proc/irq/9/smp_affinity
    2B              /proc/irq/22/smp_affinity
    2B              /proc/irq/6/smp_affinity
    2B              /proc/irq/1/smp_affinity
    2B              /proc/irq/0/smp_affinity
    2B              /proc/irq/12/smp_affinity
    2B              /proc/irq/21/smp_affinity
     
    Last edited: Jul 18, 2015
  4. eva2000

    eva2000 Administrator Staff Member

    29,016
    6,585
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +9,775
    Local Time:
    12:38 PM
    Nginx 1.13.x
    MariaDB 5.5
    Interesting Sysdig blog post Sysdig Cloud - Sysdig vs DTrace vs Strace: a Technical Discussion

    Another Sysdig Blog article Sysdig Cloud - Sysdig + Logs: Advanced Log Analysis Made Easy

     
    Last edited: Jul 18, 2015
  5. eva2000

    eva2000 Administrator Staff Member

    29,016
    6,585
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +9,775
    Local Time:
    12:38 PM
    Nginx 1.13.x
    MariaDB 5.5
  6. eva2000

    eva2000 Administrator Staff Member

    29,016
    6,585
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +9,775
    Local Time:
    12:38 PM
    Nginx 1.13.x
    MariaDB 5.5
    Hmmm doesn't work on OpenVZ VPS though, kernel module related issue ?

    Code:
    Module build for the currently running kernel was skipped since the
    kernel source for this kernel does not seem to be installed.
      Verifying  : dkms-2.2.0.3-30.git.7c3e7c5.el6.noarch                                                                      1/2
      Verifying  : sysdig-0.1.101-1.i686      
    Code:
    csysdig
    error opening device /dev/sysdig0. Make sure you have root credentials and that the sysdig-probe module is loaded.
    Doesn't work with Linode Xen and their custom Linux kernels either :(

    Code:
    Module build for the currently running kernel was skipped since the
    kernel source for this kernel does not seem to be installed.
      Verifying  : sysdig-0.1.101-1.x86_64                                                                                     1/2
      Verifying  : dkms-2.2.0.3-30.git.7c3e7c5.el6.noarch                                                                      2/2
    
    Sysdig install doesn't work on non-Linode Xen VPS either
    Code:
    Module build for the currently running kernel was skipped since the
    kernel source for this kernel does not seem to be installed.
      Verifying  : sysdig-0.1.101-1.x86_64                                                                                     1/2
      Verifying  : dkms-2.2.0.3-30.git.7c3e7c5.el6.noarch   
    Sysdig does work on my DigitalOcean KVM and Vultr KVM VPSes though
    Code:
    DKMS: build completed.
    
    sysdig-probe:
    Running module version sanity check.
    - Original module
       - No original module exists within this kernel
    - Installation
       - Installing to /lib/modules/2.6.32-504.8.1.el6.x86_64/extra/
    Adding any weak-modules
    
    depmod...
    
    DKMS: install completed.
      Verifying  : sysdig-0.1.101-1.x86_64                                                                                     1/2
      Verifying  : dkms-2.2.0.3-30.git.7c3e7c5.el6.noarch                                                                      2/2
    
    Installed:
      sysdig.x86_64 0:0.1.101-1                                                                                                   
    
    Dependency Installed:
      dkms.noarch 0:2.2.0.3-30.git.7c3e7c5.el6        
     
    Last edited: Jul 19, 2015