Sysdig

eva2000 · Jul 18, 2015

Just came across Sysdig and thought I'd share with folks

Sysdig | Install

Sysdig | Wiki | Sysdig Quick Reference Guide

Sysdig | Wiki | Csysdig Overview

Sysdig | Wiki | Sysdig Examples

Sysdig | Wiki | Sysdig Documentation Wiki

Sysdig Mailing List Group

How to monitor and troubleshoot a Linux server using sysdig - Xmodulo

What does sysdig do and why should I use it?

Sysdig captures system calls and other system level events using a linux kernel facility called tracepoints, providing a rich set of real-time, system-level information.

Sysdig "packetizes" this information, so that you can do things like save it into trace files and easily filter it, a bit like you would do with tcpdump. This makes it very flexible to explore what processes are doing.

Sysdig is also packed with a set of scripts called Chisels that make it easier to extract useful information and do troubleshooting.

Sysdig is designed from the ground up for minimal overhead and is production ready.
Click to expand...

Sysdig Install

Installing Sysdig on Centmin Mod LEMP stack is easy as alot of required packages are already installed by Centmin Mod.
Code:
rpm --import https://s3.amazonaws.com/download.draios.com/DRAIOS-GPG-KEY.public
curl -s -o /etc/yum.repos.d/draios.repo http://download.draios.com/stable/rpm/draios.repo
yum -y install sysdig
Sysdig Usage
Running Sysdig
The easiest way to get started using sysdig is with csysdig:
Code:
~$ csysdig
Csysdig is a simple, intuitive, and fully customizable curses UI for sysdig. We highly recommend you just try running csysdig and playing around with it for yourself. Here's an overview of csysdig including a video and the man page.

You can also run sysdig in its raw form:
Code:
~$ sysdig
To take advantage of this information, you'll want to learn how to use filters and chisels. Here are some cool examplesto get you started. You might also want to check out this overview of sysdig and the user guide.

And finally, here are some other resources that tend to be helpful for new sysdig users:
Click to expand...
More examples at Sysdig | Wiki | Sysdig Examples.

Basic Command List

Capture all the events from the live system and print them to screen
Code:
sysdig
Capture all the events from the live system and save them to disk
Code:
sysdig -qw dumpfile.scap
Read events from a file and print them to screen
Code:
sysdig -r dumpfile.scap
Print all the open system calls invoked by cat
Code:
sysdig proc.name=cat and evt.type=open
Print the name of the files opened by cat
Code:
./sysdig -p"%evt.arg.name" proc.name=cat and evt.type=open
List the available chisels
Code:
./sysdig -cl
Run the spy_ip chisel for the 192.168.1.157 IP address:
Code:
sysdig –c spy_ip 192.168.1.157
See the top processes in terms of CPU usage
Code:
sysdig -c topprocs_cpu
See the top processes for CPU 0
Code:
sysdig -c topprocs_cpu evt.cpu=0
Observe the standard output of a process
Code:
sysdig -s4096 -A -c stdout proc.name=cat
See the files where most time has been spent
Code:
sysdig -c topfiles_time
See the files where apache spent most time
Code:
sysdig -c topfiles_time proc.name=httpd
See the top processes in terms of I/O errors
Code:
sysdig -c topprocs_errors
See the top files in terms of I/O errors
Code:
sysdig -c topfiles_errors
See all the failed disk I/O calls
Code:
sysdig fd.type=file and evt.failed=true
See all the failed file opens by httpd
Code:
sysdig "proc.name=httpd and evt.type=open and evt.failed=true"
See the system calls where most time has been spent
Code:
sysdig -c topscalls_time
See the top system calls returning errors
Code:
sysdig -c topscalls "evt.failed=true"
snoop failed file opens as they occur
Code:
sysdig -p "%12user.name %6proc.pid %12proc.name %3fd.num %fd.typechar %fd.name" evt.type=open and evt.failed=true
Print the file I/O calls that have a latency greater than 1ms:
Code:
sysdig -c fileslower 1
Dump system activity to file, so that sysdig can be used to process it later.
Code:
sysdig -w trace.scap
View the top network connections for a single container.
Code:
sysdig -pc -c topconns container.name=wordpress1
See the files where apache spends the most
time doing I/O.
Code:
sysdig -c topfiles_time proc.name=httpd
Show all the interactive commands executed inside a given container.
Code:
sysdig -pc -c spy_users container.name=wordpress1
Show every time a file is opened under /etc.
Code:
sysdig evt.type=open and fd.name contains /etc
Sample install output
Code:
yum -y install sysdig
Loaded plugins: fastestmirror, priorities
draios                                                                                                  | 2.9 kB  00:00:00
draios/x86_64/primary_db                                                                                | 8.6 kB  00:00:00
Loading mirror speeds from cached hostfile
* base: mirror.optus.net
* epel: mirror.overthewire.com.au
* extras: mirror.aarnet.edu.au
* rpmforge: mirror.ventraip.net.au
* updates: mirror.aarnet.edu.au
149 packages excluded due to repository priority protections
Resolving Dependencies
--> Running transaction check
---> Package sysdig.x86_64 0:0.1.101-1 will be installed
--> Processing Dependency: dkms for package: sysdig-0.1.101-1.x86_64
--> Running transaction check
---> Package dkms.noarch 0:2.2.0.3-30.git.7c3e7c5.el7 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

===============================================================================================================================
Package                  Arch                     Version                                      Repository                Size
===============================================================================================================================
Installing:
sysdig                   x86_64                   0.1.101-1                                    draios                   1.3 M
Installing for dependencies:
dkms                     noarch                   2.2.0.3-30.git.7c3e7c5.el7                   epel                      77 k

Transaction Summary
===============================================================================================================================
Install  1 Package (+1 Dependent package)

Total download size: 1.3 M
Installed size: 4.5 M
Downloading packages:
(1/2): dkms-2.2.0.3-30.git.7c3e7c5.el7.noarch.rpm                                                       |  77 kB  00:00:00
(2/2): sysdig-0.1.101-x86_64.rpm                                                                        | 1.3 MB  00:00:06
-------------------------------------------------------------------------------------------------------------------------------
Total                                                                                          212 kB/s | 1.3 MB  00:00:06
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : dkms-2.2.0.3-30.git.7c3e7c5.el7.noarch                                                                      1/2
  Installing : sysdig-0.1.101-1.x86_64                                                                                     2/2

Creating symlink /var/lib/dkms/sysdig/0.1.101/source ->
                 /usr/src/sysdig-0.1.101

DKMS: add completed.

Kernel preparation unnecessary for this kernel.  Skipping...

Building module:
cleaning build area...
make KERNELRELEASE=3.10.0-229.7.2.el7.x86_64 -C /lib/modules/3.10.0-229.7.2.el7.x86_64/build M=/var/lib/dkms/sysdig/0.1.101/build....
cleaning build area...

DKMS: build completed.

sysdig-probe:
Running module version sanity check.
- Original module
   - No original module exists within this kernel
- Installation
   - Installing to /lib/modules/3.10.0-229.7.2.el7.x86_64/extra/
Adding any weak-modules

depmod...

DKMS: install completed.
  Verifying  : dkms-2.2.0.3-30.git.7c3e7c5.el7.noarch                                                                      1/2
  Verifying  : sysdig-0.1.101-1.x86_64                                                                                     2/2

Installed:
  sysdig.x86_64 0:0.1.101-1                                                                                        

Dependency Installed:
  dkms.noarch 0:2.2.0.3-30.git.7c3e7c5.el7                                                                         

Complete!
Sysdig video explaining some of the features

Videos

Sysdig Youtube channel

Csysdig intro video:

Sysdig intro video:

eva2000 · Jul 18, 2015

Code:

sysdig -cl

Category: CPU Usage
-------------------
spectrogram     Visualize OS latency in real time.
subsecoffset    Visualize subsecond offset execution time.
topcontainers_cpu
                Top containers by CPU usage
topprocs_cpu    Top processes by CPU usage

Category: Errors
----------------
topcontainers_error
                Top containers by number of errors
topfiles_errors Top files by number of errors
topprocs_errors top processes by number of errors

Category: I/O
-------------
echo_fds        Print the data read and written by processes.
fdbytes_by      I/O bytes, aggregated by an arbitrary filter field
fdcount_by      FD count, aggregated by an arbitrary filter field
fdtime_by       FD time group by
iobytes         Sum of I/O bytes on any type of FD
iobytes_file    Sum of file I/O bytes
spy_file        Echo any read/write made by any process to all files. Optionall
                y, you can provide the name of one file to only intercept reads
                /writes to that file.
stderr          Print stderr of processes
stdin           Print stdin of processes
stdout          Print stdout of processes
topcontainers_file
                Top containers by R+W disk bytes
topfiles_bytes  Top files by R+W bytes
topfiles_time   Top files by time
topprocs_file   Top processes by R+W disk bytes

Category: Logs
--------------
spy_logs        Echo any write made by any process to a log file. Optionally, e
                xport the events around each log message to file.
spy_syslog      Print every message written to syslog. Optionally, export the e
                vents around each syslog message to file.

Category: Misc
--------------
around          Export to file the events around the where the given filter mat
                ches.

Category: Net
-------------
iobytes_net     Show total network I/O bytes
spy_ip          Show the data exchanged with the given IP address
spy_port        Show the data exchanged using the given IP port number
topconns        Top network connections by total bytes
topcontainers_net
                Top containers by network I/O
topports_server Top TCP/UDP server ports by R+W bytes
topprocs_net    Top processes by network I/O

Category: Performance
---------------------
bottlenecks     Slowest system calls
fileslower      Trace slow file I/O
netlower        Trace slow network I/0
proc_exec_time  Show process execution time
scallslower     Trace slow syscalls
topscalls       Top system calls by number of calls
topscalls_time  Top system calls by time

Category: Security
------------------
list_login_shells
                List the login shell IDs
shellshock_detect
                print shellshock attacks
spy_users       Display interactive user activity

Category: System State
----------------------
lscontainers    List the running containers
lsof            List (and optionally filter) the open file descriptors.
netstat         List (and optionally filter) network connections.
ps              List (and optionally filter) the machine processes.

Use the -i flag to get detailed information about a specific chisel

eva2000 · Jul 18, 2015

Example taking a trace file capture of tracefile.scap and querying the scap trace file for top processes. @Matt @RoldanLT @jeffwidman @palPalani @Tracy Perry @Steve Tozer @pamamolf would find Sysdig useful for sure

Run command for 10 seconds and then CTRL+C exit

Code:

sysdig -w tracefile.scap

Query file for top processes

Code:

sysdig -r tracefile.scap -ctopprocs_cpu
CPU%            Process         PID          
--------------------------------------------------------------------------------
0.31%           sysdig          31858
0.31%           nginx           3848
0.31%           mysqld          1751
0.00%           rs:main         609
0.00%           bash            31804
0.00%           auditd          581
0.00%           tuned           611
0.00%           rsyslogd        609
0.00%           in:imjourn      609
0.00%           memcached       2475

Query file for top disk usage

Code:

sysdig -r tracefile.scap -ctopfiles_bytes
Bytes           Filename     
--------------------------------------------------------------------------------
1.77KB          /proc/stat
1024B           /proc/interrupts
2B              /proc/irq/15/smp_affinity
2B              /proc/irq/9/smp_affinity
2B              /proc/irq/22/smp_affinity
2B              /proc/irq/6/smp_affinity
2B              /proc/irq/1/smp_affinity
2B              /proc/irq/0/smp_affinity
2B              /proc/irq/12/smp_affinity
2B              /proc/irq/21/smp_affinity

eva2000 · Jul 18, 2015

Interesting Sysdig blog post Sysdig Cloud - Sysdig vs DTrace vs Strace: a Technical Discussion

Sysdig has an architecture that is very similar to that of libpcap/tcpdump/wireshark. First, events are captured in the kernel by a small driver, called sysdig-probe, which leverages a kernel facility called tracepoints. Tracepoints make it possible to install a “handler” that is called from specific functions in the kernel. Currently, sysdig registers tracepoints for system calls on enter and exit, and for process scheduling events. Sysdig-probe’s handler for these events is super simple – it’s limited to copying the event details into a shared buffer, encoded for later consumption. The reason to keep the handler simple, as you can imagine, is performance, since the original kernel execution is “frozen” until the handler returns.

That’s all the driver does. The rest of the magic happens at user level.

The event buffer is memory-mapped into user space so that it can be accessed without any copy, minimizing CPU usage and cache misses. Two libraries, libscap and libsinsp, then offer support for reading, decoding, and parsing events. Specifically, libscap offers trace file management functionality, while libsinsp includes sophisticated state tracking functionality (e.g. you can use a file name instead of an FD number) and also filtering, event decoding, a Lua JIT compiler to run chisels, and much more. Finally, sysdig tops it off as a simple wrapper around these libraries.

But what if sysdig, libsinsp or libscap are not fast enough to keep up with the stream of events coming from the kernel? Will sysdig slow down my system just like strace (the wise reader asks)? Well of course not. In this scenario, the event buffer fills up, and sysdig-probe starts dropping the incoming events. So you will lose a bit of trace information, but the machine, and the other processes running on it, will not be slowed down.

This is a key advantage of sysdig’s architecture. It means the tracing overhead is very predictable, and it means sysdig is ideal for running in production environments. It also means that our chisels don’t need to be as carefully optimized as DTraces’s D scripts. And on top of that, chisels get to leverage the rich set of libraries written for Lua (want to pipe your chisel’s data to Redis? There’s a library for that!).

TLDR: Sysdig is versatile, easy to use and won’t break your system!
Click to expand...

Another Sysdig Blog article Sysdig Cloud - Sysdig + Logs: Advanced Log Analysis Made Easy

Log collection and analysis is one the most powerful tools in the hands of developers and operations teams. Inspecting logs is useful in a number of areas, including security, monitoring, debugging and troubleshooting. Often, however, dealing with logs is frustrating, because they give just a hint about a problem and don’t provide enough context to understand what the problem actually is or how to fix it.

Addressing the inherent shortcomings of logs has always been one of our many goals with sysdig. And the latest release of sysdig, in particular, adds some pretty cool functionality for log collection, inspection and analysis.
This blog post will demonstrate a couple interesting methods for using sysdig with logs, including:

Tailing all of the machine log files with a single command line, without the need to configure anything

Easily correlating log entries with the relevant system activity

Troubleshooting a failed PHP request with sysdig + logs

Note: this post will likely make more sense if you have a basic familiarity with sysdig. If you’ve never heard of sysdig, I suggest you start with the website or the wiki.
Click to expand...
Analyzing application logs with sysdig

Another chisel that is very useful for log spelunking is spy_logs. It works in a similar way to spy_syslog, but instead of showing messages written to /var/log/messages, it shows messages that are written to files that contain “.log” or “_log” in their name. (Note, if you want to tune this, just edit the FILE_FILTER constant at the beginning of the chisel Lua code). spy_logs automatically unpacks multi-line writes and renders each of them nicely on the screen, with some basic pattern-matching-based coloring.

Think about it as a filterable super-tail that shows all the log files in the system, with no configuration required. Just run it, and you get all of the logs in a single stream. It’s sort of magic. Don’t believe me? Here, for example, I ran the chisel for a minute or two on one of my VMs:
Code:
sysdig -c spy_logs
httpd /opt/lampp/logs/error_log [Wed Jul 30 20:07:49.180019 2014] [suexec:notice] [pid 28262] AH01232: suEXEC mechanism enabled (wrapper: /opt/lampp/bin/suexec)
httpd /opt/lampp/logs/error_log [Wed Jul 30 20:07:49.389739 2014] [auth_digest:notice] [pid 28263] AH01757: generating secret for digest authentication ...
httpd /opt/lampp/logs/error_log [Wed Jul 30 20:07:50.045576 2014] [ssl:warn] [pid 28263] AH01906: RSA server certificate is a CA certificate (BasicConstraints: CA == TRUE !?)
httpd /opt/lampp/logs/error_log [Wed Jul 30 20:07:50.045645 2014] [ssl:warn] [pid 28263] AH01909: RSA certificate configured for www.example.com:443 does NOT include an ID which matches the server name
httpd /opt/lampp/logs/error_log [Wed Jul 30 20:07:50.045720 2014] [lbmethod_heartbeat:notice] [pid 28263] AH02282: No slotmem from mod_heartmonitor
httpd /opt/lampp/logs/error_log [Wed Jul 30 20:07:50.943101 2014] [mpm_prefork:notice] [pid 28263] AH00163: Apache/2.4.4 (Unix) OpenSSL/1.0.1e PHP/5.4.16 mod_perl/2.0.8-dev Perl/v5.16.3 configured -- resuming normal operations
httpd /opt/lampp/logs/error_log [Wed Jul 30 20:07:50.943188 2014] [core:notice] [pid 28263] AH00094: Command line: '/opt/lampp/bin/httpd -E /opt/lampp/logs/error_log -D SSL -D PHP'
httpd /opt/lampp/logs/access_log 127.0.0.1 - - [30/Jul/2014:20:08:47 -0400] "GET /xampp/splash.php HTTP/1.1" 200 1321
httpd /opt/lampp/logs/access_log 127.0.0.1 - - [30/Jul/2014:20:08:48 -0400] "GET /xampp/xampp.css HTTP/1.1" 304 -
cupsd /var/log/cups/access_log localhost - - [30/Jul/2014:20:14:03 -0400] "POST / HTTP/1.1" 200 183 Renew-Subscription client-error-not-found
redis-server /var/log/redis/redis.log [
redis-server /var/log/redis/redis.log 27066
redis-server /var/log/redis/redis.log | signal handler] (
redis-server /var/log/redis/redis.log 1406766138
redis-server /var/log/redis/redis.log )
redis-server /var/log/redis/redis.log Received SIGTERM, scheduling shutdown...
redis-server /var/log/redis/redis.log [27066] 30 Jul 20:22:18.877 # User requested shutdown...
redis-server /var/log/redis/redis.log [27066] 30 Jul 20:22:18.877 * Saving the final RDB snapshot before exiting.
The first item in each line is the process name, the second is the name of the log file, and the rest of the line contains the actual message.
Click to expand...

eva2000 · Jul 18, 2015

Cool usage for Sysdig Sysdig Cloud - Fishing for Hackers: Analysis of a Linux Server Attack

eva2000 · Jul 19, 2015

Hmmm doesn't work on OpenVZ VPS though, kernel module related issue ?

Code:

Module build for the currently running kernel was skipped since the
kernel source for this kernel does not seem to be installed.
  Verifying  : dkms-2.2.0.3-30.git.7c3e7c5.el6.noarch                                                                      1/2
  Verifying  : sysdig-0.1.101-1.i686

Code:

csysdig
error opening device /dev/sysdig0. Make sure you have root credentials and that the sysdig-probe module is loaded.

Doesn't work with Linode Xen and their custom Linux kernels either

Code:

Module build for the currently running kernel was skipped since the
kernel source for this kernel does not seem to be installed.
  Verifying  : sysdig-0.1.101-1.x86_64                                                                                     1/2
  Verifying  : dkms-2.2.0.3-30.git.7c3e7c5.el6.noarch                                                                      2/2

Sysdig install doesn't work on non-Linode Xen VPS either

Code:

Module build for the currently running kernel was skipped since the
kernel source for this kernel does not seem to be installed.
  Verifying  : sysdig-0.1.101-1.x86_64                                                                                     1/2
  Verifying  : dkms-2.2.0.3-30.git.7c3e7c5.el6.noarch

Sysdig does work on my DigitalOcean KVM and Vultr KVM VPSes though

Code:

DKMS: build completed.

sysdig-probe:
Running module version sanity check.
- Original module
   - No original module exists within this kernel
- Installation
   - Installing to /lib/modules/2.6.32-504.8.1.el6.x86_64/extra/
Adding any weak-modules

depmod...

DKMS: install completed.
  Verifying  : sysdig-0.1.101-1.x86_64                                                                                     1/2
  Verifying  : dkms-2.2.0.3-30.git.7c3e7c5.el6.noarch                                                                      2/2

Installed:
  sysdig.x86_64 0:0.1.101-1                                                                                                   

Dependency Installed:
  dkms.noarch 0:2.2.0.3-30.git.7c3e7c5.el6

Log in / Register

Forums

Learn about Centmin Mod LEMP Stack today

Sysdig

eva2000 Administrator Staff Member

Sysdig Install

Sysdig Usage

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

.

Log in / Register

Useful Searches

Learn about Centmin Mod LEMP Stack today

Sysdig

eva2000 Administrator Staff Member

Sysdig Install

Sysdig Usage

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

.