Google's perfkit for testing cloud infrastructure.

s.molinari · Feb 15, 2015

Hey George.

What do you think?

Google Cloud Platform Blog: New open source tools for measuring cloud performance

Scott

eva2000 · Feb 15, 2015

Nice thanks for heads up Scott..

PerfKit Benchmarker sounds a bit like my bench.centminmod.com benchmark script (used to evaluate VPS/Server performance and connectivity) but on steroids

edit: more media coverage

Google whips up PerfKit tools to make cloud benchmarking easier | InfoWorld

Google Launches Open-Source, Cross-Cloud Benchmarking Tool | TechCrunch

seems right now PerfKit Benchmarker only supports Amazon AWS, Microsoft Azure and Google's GCE platforms though

nice set of benchmarks though - might borrow some for integration into bench.centminmod.com (already have a few like fio, ping, unixbench and also have basic support for sysbench)

In its current release these are the benchmarks that are executed:

bonnie++: GPL v2 (Bonnie++ Documentation)

cassandra_stress: Apache v2 (The Apache Cassandra Project)

cluster_boot: MIT license.

coremark: EEMBC (EEMBC -- The Embedded Microprocessor Benchmark Consortium)

fio: GPL v2 (https://github.com/axboe/fio/blob/master/LICENSE)

hadoop_terasort: Apache v2 (Welcome to Apache™ Hadoop®!)

hpcc: Original BSD license (HPCC)

iperf: BSD license(Iperf)

netperf: HP license (All Licenses - netperf)

mongodb: GNU AGPL v3.0 (MongoDB Licensing)

ycsb (used by mongodb): Apache V2 (YCSB/LICENSE.txt at master · brianfrankcooper/YCSB · GitHub)

memtier_benchmark: GPL v2 (RedisLabs/memtier_benchmark · GitHub)

sysbench_oltp: GPL v2 (sysbench in Launchpad)

unixbench: GPL v2 (byte-unixbench - A Unix benchmark suite - Google Project Hosting)

speccpu2006 - Spec CPU2006 (SPEC CPU™ 2006)

mesh_network: HP license (All Licenses - netperf)

copy_throughput: Apache v2.

object_storage_service: Apache v2.

ping: No license needed.

aerospike: Apache V2 for the client and GNU AGPL v3.0 for the server (Aerospike NoSQL Key Value Database | Products and Services)

Click to expand...

eva2000 · Feb 15, 2015

Going to borrow memtier_benchmark tool can benchmark both redis and memcached servers. I use to use Twemperf from twitter for memcached server benchmarks.

memcached server benchmarked

Code:

memtier_benchmark -P memcache_text -s 127.0.0.1 -p 11211 --random-data --data-size-range=4-204 --data-size-pattern=S --key-minimum=200 --key-maximum=400 --key-pattern=G:G --key-stddev=10 --key-median=300 2>&1 > memtier_benchmark.log

[RUN #1] Preparing benchmark client...
[RUN #1] Launching threads now...
[RUN #1 100%,  82 secs]  0 threads:     2000000 ops,   24720 (avg:   24334) ops/sec, 3.50MB/sec (avg: 3.47MB/sec),  8.08 (avg:  8.16) msec latency

Code:

cat memtier_benchmark.log
4         Threads
50        Connections per thread
10000     Requests per thread
Type        Ops/sec     Hits/sec   Misses/sec      Latency       KB/sec
------------------------------------------------------------------------
Sets        2214.42          ---          ---      8.17400       299.06
Gets       22119.88     21900.87       219.01      8.15900      3257.78
Totals     24334.30     21900.87       219.01      8.16000      3556.83


Request Latency Distribution
Type        <= msec      Percent
------------------------------------------------------------------------
SET               0         0.11
SET               1         1.52
SET               2         2.40
SET               3         3.15
SET               4         4.74
SET               5        10.27
SET               6        25.38
SET               7        50.51
SET               8        73.71
SET               9        86.63
SET              10        93.06
SET              11        96.13
SET              12        97.47
SET              13        98.09
SET              14        98.49
SET              15        98.71
SET              16        98.92
SET              17        99.10
SET              18        99.25
SET              19        99.39
SET              20        99.47
SET              21        99.56
SET              22        99.64
SET              23        99.68
SET              24        99.71
SET              25        99.74
SET              26        99.78
SET              27        99.80
SET              28        99.81
SET              29        99.84
SET              30        99.85
SET              31        99.87
SET              32        99.89
SET              33        99.90
SET              34        99.91
SET              35        99.92
SET              36        99.93
SET              37        99.94
SET              38        99.94
SET              39        99.95
SET              40        99.95
SET              41        99.96
SET              42        99.96
SET              43        99.97
SET              44        99.97
SET              45        99.97
SET              46        99.97
SET              47        99.98
SET              48        99.99
SET              49        99.99
SET              53        99.99
SET              54        99.99
SET              59       100.00
SET              60       100.00
---
GET               0         0.10
GET               1         1.38
GET               2         2.31
GET               3         3.09
GET               4         4.73
GET               5        10.04
GET               6        25.10
GET               7        50.69
GET               8        73.53
GET               9        86.73
GET              10        93.17
GET              11        96.26
GET              12        97.66
GET              13        98.32
GET              14        98.69
GET              15        98.93
GET              16        99.10
GET              17        99.25
GET              18        99.37
GET              19        99.47
GET              20        99.55
GET              21        99.63
GET              22        99.68
GET              23        99.72
GET              24        99.75
GET              25        99.78
GET              26        99.80
GET              27        99.82
GET              28        99.83
GET              29        99.85
GET              30        99.87
GET              31        99.88
GET              32        99.89
GET              33        99.90
GET              34        99.91
GET              35        99.91
GET              36        99.92
GET              37        99.93
GET              38        99.94
GET              39        99.94
GET              40        99.95
GET              41        99.95
GET              42        99.95
GET              43        99.96
GET              44        99.96
GET              45        99.96
GET              46        99.97
GET              47        99.98
GET              48        99.98
GET              49        99.99
GET              50        99.99
GET              51        99.99
GET              52        99.99
GET              53        99.99
GET              54        99.99
GET              56       100.00
GET              57       100.00
GET              58       100.00
GET              59       100.00
GET              60       100.00
GET              61       100.00
GET              62       100.00
GET              63       100.00

redis 2.8.19 benchmarked

Code:

memtier_benchmark -P redis -s 127.0.0.1 -p 6379 --random-data --data-size-range=4-204 --data-size-pattern=S --key-minimum=200 --key-maximum=400 --key-pattern=G:G --key-stddev=10 --key-median=300 2>&1 > memtier_benchmark.redis.log      
[RUN #1] Preparing benchmark client...
[RUN #1] Launching threads now...
[RUN #1 100%,  60 secs]  0 threads:     2000000 ops,   32549 (avg:   33281) ops/sec, 4.32MB/sec (avg: 4.30MB/sec),  6.13 (avg:  5.99) msec latency

Code:

cat memtier_benchmark.redis.log
4         Threads
50        Connections per thread
10000     Requests per thread
Type        Ops/sec     Hits/sec   Misses/sec      Latency       KB/sec
------------------------------------------------------------------------
Sets        3032.24          ---          ---      5.98800       436.15
Gets       30289.11     28122.66      2166.45      5.99200      3967.32
Totals     33321.35     28122.66      2166.45      5.99200      4403.47


Request Latency Distribution
Type        <= msec      Percent
------------------------------------------------------------------------
SET               0         0.07
SET               1         1.11
SET               2         5.00
SET               3        15.42
SET               4        34.90
SET               5        57.72
SET               6        74.66
SET               7        84.81
SET               8        90.85
SET               9        94.73
SET              10        96.87
SET              11        98.07
SET              12        98.84
SET              13        99.32
SET              14        99.62
SET              15        99.79
SET              16        99.85
SET              17        99.89
SET              18        99.91
SET              19        99.93
SET              20        99.96
SET              21        99.97
SET              22        99.99
SET              23        99.99
SET              24       100.00
SET              25       100.00
SET              28       100.00
---
GET               0         0.05
GET               1         1.03
GET               2         4.86
GET               3        15.35
GET               4        34.74
GET               5        57.71
GET               6        74.72
GET               7        84.78
GET               8        90.81
GET               9        94.71
GET              10        96.85
GET              11        98.09
GET              12        98.86
GET              13        99.33
GET              14        99.60
GET              15        99.78
GET              16        99.84
GET              17        99.87
GET              18        99.91
GET              19        99.93
GET              20        99.96
GET              21        99.97
GET              22        99.99
GET              23        99.99
GET              24       100.00
GET              25       100.00
GET              26       100.00
GET              28       100.00

redis AOF enabled with appendfsync everysec

Code:

memtier_benchmark -P redis -s 127.0.0.1 -p 6379 --random-data --data-size-range=4-204 --data-size-pattern=S --key-minimum=200 --key-maximum=400 --key-pattern=G:G --key-stddev=10 --key-median=300 2>&1 > memtier_benchmark.redis.aofenabled.log
[RUN #1] Preparing benchmark client...
[RUN #1] Launching threads now...
[RUN #1 100%,  60 secs]  0 threads:     2000000 ops,   34892 (avg:   32788) ops/sec, 4.64MB/sec (avg: 4.23MB/sec),  5.72 (avg:  6.08) msec latency

Code:

cat memtier_benchmark.redis.aofenabled.log
4         Threads
50        Connections per thread
10000     Requests per thread
Type        Ops/sec     Hits/sec   Misses/sec      Latency       KB/sec
------------------------------------------------------------------------
Sets        2983.83          ---          ---      6.08000       429.19
Gets       29805.53     27679.11      2126.42      6.08100      3904.52
Totals     32789.37     27679.11      2126.42      6.08100      4333.70


Request Latency Distribution
Type        <= msec      Percent
------------------------------------------------------------------------
SET               0         0.07
SET               1         0.75
SET               2         3.92
SET               3        13.54
SET               4        32.11
SET               5        55.51
SET               6        73.98
SET               7        84.69
SET               8        90.85
SET               9        94.78
SET              10        96.98
SET              11        98.10
SET              12        98.80
SET              13        99.25
SET              14        99.56
SET              15        99.74
SET              16        99.81
SET              17        99.86
SET              18        99.91
SET              19        99.93
SET              20        99.94
SET              21        99.96
SET              22        99.97
SET              23        99.97
SET              24        99.98
SET              25        99.98
SET              26        99.98
SET              27        99.99
SET              28        99.99
SET              29        99.99
SET              59       100.00
SET              60       100.00
SET              61       100.00
SET              62       100.00
---
GET               0         0.05
GET               1         0.75
GET               2         3.89
GET               3        13.46
GET               4        32.12
GET               5        55.56
GET               6        74.01
GET               7        84.60
GET               8        90.84
GET               9        94.74
GET              10        96.98
GET              11        98.12
GET              12        98.83
GET              13        99.27
GET              14        99.57
GET              15        99.73
GET              16        99.80
GET              17        99.85
GET              18        99.90
GET              19        99.92
GET              20        99.94
GET              21        99.97
GET              22        99.98
GET              23        99.98
GET              24        99.98
GET              25        99.98
GET              26        99.99
GET              27        99.99
GET              28        99.99
GET              29        99.99
GET              30       100.00
GET              56       100.00
GET              57       100.00
GET              59       100.00
GET              60       100.00
GET              61       100.00
GET              62       100.00

my custom redisinfo.sh script

Code:

./redisinfo.sh
--------------------------------------------------------------------------
redis-cli 2.8.19
--------------------------------------------------------------------------
redis-cli info
# Server
redis_version:2.8.19
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:8012391549b7b328
redis_mode:standalone
os:Linux 2.6.32-504.3.3.el6.x86_64 x86_64
arch_bits:64
multiplexing_api:epoll
gcc_version:4.4.7
process_id:26296
run_id:d370932bd82e7aeab4c750b2b2803131cc1470ab
tcp_port:6379
uptime_in_seconds:224
uptime_in_days:0
hz:10
lru_clock:14698116
config_file:/etc/redis.conf

# Clients
connected_clients:1
client_longest_output_list:0
client_biggest_input_buf:0
blocked_clients:0

# Memory
used_memory:810848
used_memory_human:791.84K
used_memory_rss:5468160
used_memory_peak:810848
used_memory_peak_human:791.84K
used_memory_lua:35840
mem_fragmentation_ratio:6.74
mem_allocator:jemalloc-3.6.0

# Persistence
loading:0
rdb_changes_since_last_save:0
rdb_bgsave_in_progress:0
rdb_last_save_time:1423984036
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:-1
rdb_current_bgsave_time_sec:-1
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok

# Stats
total_connections_received:3
total_commands_processed:2
instantaneous_ops_per_sec:0
total_net_input_bytes:42
total_net_output_bytes:3720
instantaneous_input_kbps:0.01
instantaneous_output_kbps:1.12
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:0
evicted_keys:0
keyspace_hits:0
keyspace_misses:0
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:0

# Replication
role:master
connected_slaves:0
master_repl_offset:0
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0

# CPU
used_cpu_sys:0.05
used_cpu_user:0.39
used_cpu_sys_children:0.00
used_cpu_user_children:0.00

# Keyspace

--------------------------------------------------------------------------
redis-cli --intrinsic-latency 10
Max latency so far: 8 microseconds.
Max latency so far: 22 microseconds.
Max latency so far: 63 microseconds.
Max latency so far: 74 microseconds.
Max latency so far: 743 microseconds.
Max latency so far: 1176 microseconds.
Max latency so far: 2541 microseconds.
Max latency so far: 7983 microseconds.
Max latency so far: 8382 microseconds.

1497243 total runs (avg latency: 6.6789 microseconds / 66789.43 nanoseconds per run).
Worst run took 1255x longer than the average latency.

--------------------------------------------------------------------------
vmstat 1 10
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
0  0      0 233104 118092 481232    0    0    67    48  147  582  3  3 95  0  0
0  0      0 233096 118092 481252    0    0     0    68   66   60  0  1 99  0  0
0  0      0 233104 118100 481236    0    0     0    28   66   81  0  0 100  0  0
0  0      0 233104 118100 481240    0    0     0     0   54   57  0  0 100  0  0
1  0      0 233104 118100 481240    0    0     0     0   61   64  0  1 99  0  0
0  0      0 233104 118100 481240    0    0     0     0   70   77  0  0 100  0  0
0  0      0 233104 118100 481240    0    0     0     0   56   67  0  0 100  0  0
2  0      0 233104 118100 481240    0    0     0     0   47   54  0  0 100  0  0
0  0      0 233104 118100 481240    0    0     0     0   46   55  1  0 99  0  0
1  0      0 233104 118100 481240    0    0     0     0   54   54  0  0 100  0  0
--------------------------------------------------------------------------

smem -kU redis
  PID User     Command                         Swap      USS      PSS      RSS
26296 redis    /usr/bin/redis-server 127.0        0     4.7M     4.7M     5.2M
--------------------------------------------------------------------------

--------------------------------------------------------------------------
smem -kmU redis
Map                                       PIDs   AVGPSS      PSS
/usr/lib/locale/locale-archive               1        0        0
[vdso]                                       1        0        0
[vsyscall]                                   1        0        0
/lib64/libdl-2.12.so                         1     8.0K     8.0K
/lib64/libm-2.12.so                          1     8.0K     8.0K
/lib64/ld-2.12.so                            1     9.0K     9.0K
/lib64/libpthread-2.12.so                    1    12.0K    12.0K
[stack]                                      1    20.0K    20.0K
/lib64/libc-2.12.so                          1    47.0K    47.0K
/usr/lib64/libjemalloc.so.1                  1    96.0K    96.0K
/usr/bin/redis-server                        1   360.0K   360.0K
<anonymous>                                  1     4.1M     4.1M
--------------------------------------------------------------------------

--------------------------------------------------------------------------
smem -tpk
  PID User     Command                         Swap      USS      PSS      RSS
2403 root     /sbin/mingetty /dev/tty1           0    80.0K    99.0K   584.0K
2407 root     /sbin/mingetty /dev/tty3           0    80.0K    99.0K   584.0K
2409 root     /sbin/mingetty /dev/tty4           0    80.0K    99.0K   584.0K
2411 root     /sbin/mingetty /dev/tty5           0    80.0K    99.0K   584.0K
2418 root     /sbin/mingetty /dev/tty6           0    80.0K    99.0K   584.0K
2405 root     /sbin/mingetty /dev/tty2           0    84.0K   103.0K   588.0K
  974 nsd      /usr/local/sbin/nsd -c /etc        0    84.0K   224.0K   908.0K
  977 nsd      /usr/local/sbin/nsd -c /etc        0   152.0K   263.0K   604.0K
2415 root     /sbin/udevd -d                     0    80.0K   303.0K   944.0K
2416 root     /sbin/udevd -d                     0    80.0K   303.0K   944.0K
  357 root     /sbin/udevd -d                     0   108.0K   312.0K   976.0K
  914 root     auditd                             0   308.0K   327.0K   776.0K
26321 root     /bin/bash ./redisinfo.sh           0   216.0K   401.0K     1.4M
1232 root     /bin/sh /usr/bin/mysqld_saf        0   316.0K   504.0K     1.6M
1026 dbus     dbus-daemon --system               0   528.0K   546.0K   908.0K
    1 root     /sbin/init                         0   440.0K   551.0K     1.4M
2390 root     crond                              0   644.0K   676.0K     1.2M
1209 root     /usr/sbin/sshd                     0   708.0K   794.0K     1.2M
1113 root     /usr/sbin/VBoxService              0   700.0K   798.0K     1.5M
  934 root     /sbin/rsyslogd -i /var/run/        0   876.0K   908.0K     1.5M
1218 ntp      ntpd -u ntp:ntp -p /var/run        0   956.0K     1.1M     2.0M
2362 root     /usr/libexec/postfix/master        0  1008.0K     1.4M     3.3M
2374 postfix  pickup -l -t fifo -u               0     1.0M     1.4M     3.3M
2375 postfix  qmgr -l -t fifo -u                 0     1.1M     1.5M     3.4M
2378 root     nginx: master process /usr/        0   788.0K     1.7M     2.8M
1124 root     /usr/sbin/console-kit-daemo        0     1.7M     1.8M     2.6M
2425 root     sshd: root@pts/0,pts/              0     2.2M     2.6M     4.3M
2427 root     -bash                              0     2.4M     2.7M     4.0M
11654 root     -bash                              0     2.4M     2.7M     4.0M
2268 root     /usr/sbin/haveged -w 3072 -        0     3.1M     3.1M     3.5M
  976 nsd      /usr/local/sbin/nsd -c /etc        0     4.2M     4.3M     4.7M
26296 redis    /usr/bin/redis-server 127.0        0     4.7M     4.7M     5.2M
2380 nginx    nginx: worker process              0     4.0M     4.9M     6.3M
26354 root     python /usr/bin/smem -tpk          0     5.8M     5.9M     6.6M
6215 nobody   /usr/local/bin/memcached -d        0     6.1M     6.1M     6.5M
6230 root     php-fpm: master process (/u        0     6.2M     6.3M     6.8M
1016 root     lfd - sleepin                      0    17.0M    17.1M    17.7M
2255 mysql    /usr/sbin/mysqld --basedir=        0    38.6M    38.8M    40.2M
-------------------------------------------------------------------------------
   38 9                                           0   108.8M   115.4M   146.2M
--------------------------------------------------------------------------

--------------------------------------------------------------------------
free -mlt
             total       used       free     shared    buffers     cached
Mem:           996        769        227         21        115        470
Low:           996        769        227
High:            0          0          0
-/+ buffers/cache:        183        812
Swap:         1839          0       1839
Total:        2836        769       2067
--------------------------------------------------------------------------

I'm liking memtier_benchmark

with test paramters

Code:

 memtier_benchmark -P memcache_text -s 127.0.0.1 -p 11211 --random-data --data-size-range=4-204 --data-size-pattern=S --key-minimum=200 --key-maximum=400 --key-pattern=G:G --key-stddev=10 --key-median=300 2>&1 > memtier_benchmark.log

default of box memcached server benchmark for Centmin Mod LEMP stack

Code:

cat memtier_benchmark.log
4         Threads
50        Connections per thread
10000     Requests per thread
Type        Ops/sec     Hits/sec   Misses/sec      Latency       KB/sec
------------------------------------------------------------------------
Sets        2214.42          ---          ---      8.17400       299.06
Gets       22119.88     21900.87       219.01      8.15900      3257.78
Totals     24334.30     21900.87       219.01      8.16000      3556.83

do some tuning and tweaking of memcached server on Centmin Mod LEMP stack

Code:

cat memtier_benchmark.log
4         Threads
50        Connections per thread
10000     Requests per thread
Type        Ops/sec     Hits/sec   Misses/sec      Latency       KB/sec
------------------------------------------------------------------------
Sets        3455.32          ---          ---      5.22500       466.64
Gets       34515.24     32113.53      2401.71      5.22100      4820.50
Totals     37970.56     32113.53      2401.71      5.22200      5287.13

more tweaking for memcached server

Code:

cat memtier_benchmark.log
4         Threads
50        Connections per thread
10000     Requests per thread
Type        Ops/sec     Hits/sec   Misses/sec      Latency       KB/sec
------------------------------------------------------------------------
Sets        3545.16          ---          ---      5.21700       478.77
Gets       35412.66     33082.59      2330.07      5.21300      4962.82
Totals     38957.82     33082.59      2330.07      5.21400      5441.59

Log in / Register

Forums

Want to subscribe to topics you're interested in?

Google's perfkit for testing cloud infrastructure.

s.molinari New Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

.

Log in / Register

Useful Searches

Want to subscribe to topics you're interested in?

Google's perfkit for testing cloud infrastructure.

s.molinari New Member

eva2000 Administrator Staff Member

eva2000 Administrator Staff Member

.