Welcome to Centmin Mod Community
Become a Member

Memcached Memcached - before and after graphs!

Discussion in 'Other Centmin Mod Installed software' started by deltahf, Feb 19, 2015.

  1. deltahf

    deltahf Premium Member Premium Member

    586
    264
    63
    Jun 8, 2014
    Ratings:
    +487
    Local Time:
    10:11 AM
    Just a reminder that memcached is awesome... I never bothered to configure it because I thought it wouldn't provide much benefit in a single server environment - I was very wrong, of course! Just activated it on a large WordPress + XenForo site (10+ million posts) which currently handles around 250k page views per day, hosted on a 16GB Linode (PM me for the link).

    Here's the NewRelic graph, with memcached activated at 7:45pm. Average total server response time dropped from 144ms to 101ms, a nearly 30% improvement. Thank you, Centminmod! :D

    Screen Shot 2015-02-18 at 10.15.40 PM.png

     
  2. eva2000

    eva2000 Administrator Staff Member

    54,339
    12,198
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,763
    Local Time:
    1:11 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
  3. eva2000

    eva2000 Administrator Staff Member

    54,339
    12,198
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,763
    Local Time:
    1:11 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    Was curious myself and for this very light traffic forum it's all PHP time :)

    newrelic_webtransactions_responsetime_01.png
    newrelic_webtransactions_responsetime_02.png
    newrelic_webtransactions_responsetime_03.png
     
    Last edited: Feb 19, 2015
  4. rdan

    rdan Well-Known Member

    5,444
    1,408
    113
    May 25, 2014
    Ratings:
    +2,201
    Local Time:
    11:11 PM
    Mainline
    10.2
  5. eva2000

    eva2000 Administrator Staff Member

    54,339
    12,198
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,763
    Local Time:
    1:11 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    Probably just the default TCP method for memcached I suspect :)

    FYI, there's various ways in SSH to get memcached server stats too

    Code:
    echo stats | nc 127.0.0.1 11211
    gives

    Code:
    echo stats | nc 127.0.0.1 11211
    STAT pid 22582
    STAT uptime 24
    STAT time 1424349314
    STAT version 1.4.22
    STAT libevent 2.0.21-stable
    STAT pointer_size 64
    STAT rusage_user 0.006666
    STAT rusage_system 0.033333
    STAT curr_connections 5
    STAT total_connections 18
    STAT connection_structures 6
    STAT reserved_fds 20
    STAT cmd_get 100
    STAT cmd_set 37
    STAT cmd_flush 0
    STAT cmd_touch 0
    STAT get_hits 64
    STAT get_misses 36
    STAT delete_misses 0
    STAT delete_hits 0
    STAT incr_misses 0
    STAT incr_hits 0
    STAT decr_misses 0
    STAT decr_hits 0
    STAT cas_misses 0
    STAT cas_hits 0
    STAT cas_badval 0
    STAT touch_hits 0
    STAT touch_misses 0
    STAT auth_cmds 0
    STAT auth_errors 0
    STAT bytes_read 204124
    STAT bytes_written 511405
    STAT limit_maxbytes 268435456
    STAT accepting_conns 1
    STAT listen_disabled_num 0
    STAT threads 4
    STAT conn_yields 0
    STAT hash_power_level 16
    STAT hash_bytes 524288
    STAT hash_is_expanding 0
    STAT slab_reassign_running 0
    STAT slabs_moved 0
    STAT malloc_fails 0
    STAT bytes 202454
    STAT curr_items 37
    STAT total_items 37
    STAT expired_unfetched 0
    STAT evicted_unfetched 0
    STAT evictions 0
    STAT reclaimed 0
    STAT crawler_reclaimed 0
    STAT lrutail_reflocked 0
    END
    via memcached-tool

    the options available

    Code:
    memcached-tool
    Usage: memcached-tool <host[:port] | /path/to/socket> [mode]
    
           memcached-tool 10.0.0.5:11211 display    # shows slabs
           memcached-tool 10.0.0.5:11211            # same.  (default is display)
           memcached-tool 10.0.0.5:11211 stats      # shows general stats
           memcached-tool 10.0.0.5:11211 settings   # shows settings stats
           memcached-tool 10.0.0.5:11211 sizes      # shows sizes stats
           memcached-tool 10.0.0.5:11211 dump       # dumps keys and values
    
    WARNING! sizes is a development command.
    As of 1.4 it is still the only command which will lock your memcached instance for some time.
    If you have many millions of stored items, it can become unresponsive for several minutes.
    Run this at your own risk. It is roadmapped to either make this feature optional
    or at least speed it up.
    for example stats

    Code:
    memcached-tool 127.0.0.1:11211 stats
    Code:
    memcached-tool 127.0.0.1:11211 stats
    #127.0.0.1:11211   Field       Value
             accepting_conns           1
                   auth_cmds           0
                 auth_errors           0
                       bytes      624900
                  bytes_read      701739
               bytes_written     5200282
                  cas_badval           0
                    cas_hits           0
                  cas_misses           0
                   cmd_flush           0
                     cmd_get         917
                     cmd_set         124
                   cmd_touch           0
                 conn_yields           0
       connection_structures           9
           crawler_reclaimed           0
            curr_connections           8
                  curr_items          98
                   decr_hits           0
                 decr_misses           0
                 delete_hits           0
               delete_misses           0
           evicted_unfetched           0
                   evictions           0
           expired_unfetched           0
                    get_hits         827
                  get_misses          90
                  hash_bytes      524288
           hash_is_expanding           0
            hash_power_level          16
                   incr_hits           0
                 incr_misses           0
                    libevent 2.0.21-stable
              limit_maxbytes   268435456
         listen_disabled_num           0
           lrutail_reflocked           0
                malloc_fails           0
                         pid       22582
                pointer_size          64
                   reclaimed           0
                reserved_fds          20
               rusage_system    0.079999
                 rusage_user    0.039999
       slab_reassign_running           0
                 slabs_moved           0
                     threads           4
                        time  1424349464
           total_connections          53
                 total_items         124
                  touch_hits           0
                touch_misses           0
                      uptime         174
                     version      1.4.22
    settings list

    Code:
    memcached-tool 127.0.0.1:11211 settings
    #127.0.0.1:11211   Field       Value
           auth_enabled_sasl          no
            binding_protocol auto-negotiate
                 cas_enabled         yes
                  chunk_size          72
              detail_enabled          no
               domain_socket        NULL
                   evictions          on
               flush_enabled         yes
               growth_factor        1.25
              hash_algorithm     jenkins
              hashpower_init           0
                       inter   127.0.0.1
               item_size_max     1048576
                 lru_crawler          no
           lru_crawler_sleep         100
         lru_crawler_tocrawl           0
                    maxbytes   268435456
                    maxconns        2048
               maxconns_fast          no
                 num_threads           4
         num_threads_per_udp           4
                      oldest           0
              reqs_per_event         200
               slab_automove           1
               slab_reassign         yes
             stat_key_prefix           :
            tail_repair_time           0
                 tcp_backlog        2048
                     tcpport       11211
                     udpport       11211
                       umask         700
                   verbosity           0
    Explanation for memcached-tool at MySQL :: MySQL 5.5 Reference Manual :: 16.6.4.6 Using memcached-tool and details on stats output MySQL :: MySQL 5.5 Reference Manual :: 16.6.4.1 memcached General Statistics

    Statistic Data type Description Version
    pid 32u Process ID of the memcached instance.
    uptime 32u Uptime (in seconds) for this memcached instance.
    time 32u Current time (as epoch).
    version string Version string of this instance.
    pointer_size string Size of pointers for this host specified in bits (32 or 64).
    rusage_user 32u:32u Total user time for this instance (seconds:microseconds).
    rusage_system 32u:32u Total system time for this instance (seconds:microseconds).
    curr_items 32u Current number of items stored by this instance.
    total_items 32u Total number of items stored during the life of this instance.
    bytes 64u Current number of bytes used by this server to store items.
    curr_connections 32u Current number of open connections.
    total_connections 32u Total number of connections opened since the server started running.
    connection_structures 32u Number of connection structures allocated by the server.
    cmd_get 64u Total number of retrieval requests (get operations).
    cmd_set 64u Total number of storage requests (set operations).
    get_hits 64u Number of keys that have been requested and found present.
    get_misses 64u Number of items that have been requested and not found.
    delete_hits 64u Number of keys that have been deleted and found present. 1.3.x
    delete_misses 64u Number of items that have been delete and not found. 1.3.x
    incr_hits 64u Number of keys that have been incremented and found present. 1.3.x
    incr_misses 64u Number of items that have been incremented and not found. 1.3.x
    decr_hits 64u Number of keys that have been decremented and found present. 1.3.x
    decr_misses 64u Number of items that have been decremented and not found. 1.3.x
    cas_hits 64u Number of keys that have been compared and swapped and found present. 1.3.x
    cas_misses 64u Number of items that have been compared and swapped and not found. 1.3.x
    cas_badvalue 64u Number of keys that have been compared and swapped, but the comparison (original) value did not match the supplied value. 1.3.x
    evictions 64u Number of valid items removed from cache to free memory for new items.
    bytes_read 64u Total number of bytes read by this server from network.
    bytes_written 64u Total number of bytes sent by this server to network.
    limit_maxbytes 32u Number of bytes this server is permitted to use for storage.
    threads 32u Number of worker threads requested.
    conn_yields 64u Number of yields for connections (related to the -R option). 1.4.0
     
    Last edited: Feb 19, 2015
  6. deltahf

    deltahf Premium Member Premium Member

    586
    264
    63
    Jun 8, 2014
    Ratings:
    +487
    Local Time:
    10:11 AM
    Awesome, I wish my forum had that much traffic. :) Why are you running multiple memcached servers in that environment? How does the software know which server will have the key it needs?

    Here's my memcached info after a day:

    Screen Shot 2015-02-19 at 8.49.58 PM.png

    TCP. From what I've heard, there is not much difference in speed between the two these days. What do you guys recommend?
     
  7. rdan

    rdan Well-Known Member

    5,444
    1,408
    113
    May 25, 2014
    Ratings:
    +2,201
    Local Time:
    11:11 PM
    Mainline
    10.2
    @deltahf with your traffic I think you better raise up your MEMCACHED MEMSIZE at least 1GB.
     
  8. deltahf

    deltahf Premium Member Premium Member

    586
    264
    63
    Jun 8, 2014
    Ratings:
    +487
    Local Time:
    10:11 AM
    OK, that seems a little large for what it has used so far. Should I expect it to grow by 8MB per day?
     
  9. eva2000

    eva2000 Administrator Staff Member

    54,339
    12,198
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,763
    Local Time:
    1:11 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    Better to monitor memory usage and adjust accordingly :)

    As to multiple memcached servers, you can use memcache PHP extensions consistent hash strategy / session redundancy option (already set in Centmin Mod default memcached server/memcache PHP extension setup) so adding more memcached servers if web app supports. However, there's not as great a performance boost for such as there are more advanced methods which allow multiple memcached servers pooled into one cluster with automatic sharding and failover which I've been playing with for my paid client work which allows you to scale memcached servers to 50,000+ concurrent connections easily to scale memcached server caching :)
     
  10. rdan

    rdan Well-Known Member

    5,444
    1,408
    113
    May 25, 2014
    Ratings:
    +2,201
    Local Time:
    11:11 PM
    Mainline
    10.2
    I expect from your forum size it will consume at least 512MB and up.
     
  11. Mask

    Mask Active Member

    108
    31
    28
    Nov 10, 2014
    Ratings:
    +37
    Local Time:
    8:11 PM
    Nginx 1.9.1
    MariaDB 10.0.19
    I actually tested it on a cloud server (Xen-OnApp based) and you are right... There was not much difference (at least in my test environment) after the sysctl tweak.
    However, I am still to test it out with my OpenVZ based VPS. (I host them on my own server)
    I have a feeling there me be difference in this case since you can not tweak sysctl in OpenVZ on container level.

    I will let you know once my server admin (and my friend) get's time for setting things up for me to test.
     
  12. deltahf

    deltahf Premium Member Premium Member

    586
    264
    63
    Jun 8, 2014
    Ratings:
    +487
    Local Time:
    10:11 AM
    Quick update after (almost) 1 week - XenForo is still not using that much memory. 201 million cache hits, though! :D

    Screen Shot 2015-02-25 at 1.23.28 AM.png
     
  13. eva2000

    eva2000 Administrator Staff Member

    54,339
    12,198
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,763
    Local Time:
    1:11 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    Nice to see it's working well.. 201 million hits saved from hitting the database :)
     
  14. rdan

    rdan Well-Known Member

    5,444
    1,408
    113
    May 25, 2014
    Ratings:
    +2,201
    Local Time:
    11:11 PM
    Mainline
    10.2
    Seriously 16MB is very small.
    Try at least 128MB or 256MB.

    This is what I have.
    upload_2015-2-25_20-22-24.png
     
  15. eva2000

    eva2000 Administrator Staff Member

    54,339
    12,198
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,763
    Local Time:
    1:11 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    You only need to allocate as much memory as your web app (XF) requires for memcached server caching. Allocating more memory doesn't do anything for performance :)
     
  16. deltahf

    deltahf Premium Member Premium Member

    586
    264
    63
    Jun 8, 2014
    Ratings:
    +487
    Local Time:
    10:11 AM
    Interesting - is that for your Xenforo install? I wonder how our memory usage could be so different after almost the exact same amount of time?
     
  17. eva2000

    eva2000 Administrator Staff Member

    54,339
    12,198
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,763
    Local Time:
    1:11 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    probably cause @RoldanLT and I have other addons which do caching like XenCache Xen Cache | XenForo Community and bd Cache [bd] Cache | XenForo Community and I have ngx_pagespeed enabled too with memcached offloading Nginx, PHP-FPM & MariaDB MySQL | Centmin Mod Community

    my memcached stats for this forum
    Code:
    Free: 76.3 MBytes (29.8%)    Hits: 5692032 (98.7%)
    Used: 179.7 MBytes (70.2%)    Misses: 77613 (1.3%)
    
    Cache Information
    Current Items(total)    41214 (355735)
    
     
  18. deltahf

    deltahf Premium Member Premium Member

    586
    264
    63
    Jun 8, 2014
    Ratings:
    +487
    Local Time:
    10:11 AM
    Ah, yes. I wonder why the XenForo developers chose not to cache the data which the "Xen Cache" plugin does...
     
  19. eva2000

    eva2000 Administrator Staff Member

    54,339
    12,198
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,763
    Local Time:
    1:11 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    yeah would think it would be useful for all :)
     
  20. rdan

    rdan Well-Known Member

    5,444
    1,408
    113
    May 25, 2014
    Ratings:
    +2,201
    Local Time:
    11:11 PM
    Mainline
    10.2
    Yes **** and a small Indian Forum of my Friend.
    And by the way, this is my config.php caching code.
    Code:
    // Cache
    $config['cache']['enabled'] = true;
    $config['cache']['frontend'] = 'Core';
    $config['cache']['frontendOptions']['cache_id_prefix'] = 'phc_';
    $config['cache']['cacheSessions'] = true;
    $config['cache']['backend']='Libmemcached';
    $config['cache']['backendOptions']=array(
            'compression'=>false,
            'servers' => array(
                    array(
                            'host'=>'127.0.0.1',
                            'port'=>'11211',
                            'persistent' => 'true'
                    )
          )
    );
    
    Yes :D
    I have BD Cache also.
    That's why it consumes more memory.
    But I remove Xen Cache for now, as it is not helping much for medium forum size only.