Join the community today
Become a Member

SSL HTTP/TLS Protocol and SSL Cipher Usage Statistics Logging

Discussion in 'Domains, DNS, Email & SSL Certificates' started by eva2000, Nov 21, 2015.

  1. eva2000

    eva2000 Administrator Staff Member

    26,598
    6,104
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +8,951
    Local Time:
    6:31 AM
    Nginx 1.11.x
    MariaDB 5.5
    Came across this article by Scott Helme Monitoring HTTP/2 usage in the wild and it's very useful stats :)

    So thought I'd do it on this forum's Centmin Mod Nginx server.

    Step 1. Add to /usr/local/nginx/conf/nginx.conf in http {} server context the following custom log format
    Code (Text):
    log_format ssl_custom '$ssl_protocol $ssl_cipher $request';


    Step 2. Add to your domain's Nginx vhost HTTPS/HTTP config file an additional access_log to accompany existing access_log entry
    Code (Text):
    access_log /home/nginx/domains/yourdomain.com/log/sslstats.log ssl_custom buffer=256k flush=5m;

    then restart Nginx and PHP-FPM
    Code (Text):
    nprestart


    Step 3. Wait for a few minutes (up to 5 minutes if low traffic server) so flush=5m can write access log entries to the access log

    Step 4. Tally stats in /home/nginx/domains/yourdomain.com/log/sslstats.log using these 3 commands. Some may have some empty 1st column fields which you can ignore:

    For HTTP Protocol Usage
    Code (Text):
    awk '{print $5}' /home/nginx/domains/yourdomain.com/log/sslstats.log | sort | uniq -c | sort -rn | awk '{print $2, $1}'


    For SSL Cipher Usage
    Code (Text):
    awk '{print $2}' /home/nginx/domains/yourdomain.com/log/sslstats.log | sort | uniq -c | sort -rn | awk '{print $2, $1}'


    For TLS protocol Usage
    Code (Text):
    awk '{print $1}' /home/nginx/domains/yourdomain.com/log/sslstats.log | sort | uniq -c | sort -rn | awk '{print $2, $1}';


    Example output
    Code (Text):
    HTTP/1.1 53777
    HTTP/2.0 20773
    HTTP/1.0 10148
    12
    
    ECDHE-RSA-AES128-GCM-SHA256 59489
    ECDHE-RSA-CHACHA20-POLY1305 14468
    ECDHE-RSA-AES128-SHA 6956
    ECDHE-RSA-AES128-SHA256 2220
    DHE-RSA-AES128-GCM-SHA256 1033
    DHE-RSA-AES128-SHA 540
    - 4
    
    TLSv1.2 77223
    TLSv1 7470
    TLSv1.1 13
    - 4


    upload_2015-11-21_14-26-36.png

    upload_2015-11-21_14-26-49.png

    upload_2015-11-21_14-26-56.png
     
    Last edited: Nov 23, 2015
    • Like Like x 1
  2. eva2000

    eva2000 Administrator Staff Member

    26,598
    6,104
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +8,951
    Local Time:
    6:31 AM
    Nginx 1.11.x
    MariaDB 5.5
    Lets extend the ssl stats logging to include $status and $http_user_agent for more details

    Code (Text):
    log_format ssl_customagent '$ssl_protocol $ssl_cipher $request $status $http_user_agent';

    and separate log at /home/nginx/domains/yourdomain.com/log/sslstats-agent.log
    Code (Text):
    access_log /home/nginx/domains/yourdomain.com/log/sslstats-agent.log ssl_customagent buffer=256k flush=5m;


    I added this in addition to above so as to not mess with already accumulated stats. So it became

    Code (Text):
    log_format ssl_custom '$ssl_protocol $ssl_cipher $request';
    log_format ssl_customagent '$ssl_protocol $ssl_cipher $request $status $http_user_agent';


    Code (Text):
    access_log /home/nginx/domains/yourdomain.com/log/sslstats.log ssl_custom buffer=256k flush=5m;
    access_log /home/nginx/domains/yourdomain.com/log/sslstats-agent.log ssl_customagent buffer=256k flush=5m;


    Then revised awk sort and count commands
    Code (Text):
    awk '{print $5}' /home/nginx/domains/yourdomain.com/log/sslstats-agent.log | sort | uniq -c | sort -rn | awk '{print $2, $1}'
    
    Code (Text):
    awk '{print $2}' /home/nginx/domains/yourdomain.com/log/sslstats-agent.log | sort | uniq -c | sort -rn | awk '{print $2, $1}'
    
    Code (Text):
    awk '{print $1}' /home/nginx/domains/yourdomain.com/log/sslstats-agent.log | sort | uniq -c | sort -rn | awk '{print $2, $1}'
    
    Code (Text):
    awk '{print $6}' /home/nginx/domains/yourdomain.com/log/sslstats-agent.log | sort | uniq -c | sort -rn | awk '{print $2, $1}'
    
    Code (Text):
    awk '{print $7}' /home/nginx/domains/yourdomain.com/log/sslstats-agent.log | sort | uniq -c | sort -rn | awk '{print $2, $1}'
    

    Code (Text):
    awk '{print $5, $7}' /home/nginx/domains/yourdomain.com/log/sslstats-agent.log | sort | uniq -c | sort -rn | awk '{print $2, $3, $1}';
    


    Code (Text):
    HTTP/1.1 195
    HTTP/2.0 145
    HTTP/1.0 51
    
    ECDHE-RSA-AES128-GCM-SHA256 207
    ECDHE-RSA-CHACHA20-POLY1305 119
    ECDHE-RSA-AES128-SHA256 31
    ECDHE-RSA-AES128-SHA 29
    DHE-RSA-AES128-GCM-SHA256 5
    
    TLSv1.2 362
    TLSv1 29
    
    200 335
    204 26
    301 16
    304 7
    303 3
    302 2
    404 1
    307 1
    

    Code (Text):
    Mozilla/5.0 221
    Mozilla/4.0 109
    Y!J-ASR/0.1 30
    NewRelicPinger/1.0 20
    Mozilla/5.0+(compatible; 5
    ltx71 2
    libwww-perl/5.833 2
    Mediapartners-Google/2.1 1
    Mediapartners-Google 1
    
    HTTP/2.0 Mozilla/5.0 145
    HTTP/1.1 Mozilla/4.0 109
    HTTP/1.1 Mozilla/5.0 75
    HTTP/1.0 Y!J-ASR/0.1 30
    HTTP/1.0 NewRelicPinger/1.0 20
    HTTP/1.1 Mozilla/5.0+(compatible; 5
    HTTP/1.1 ltx71 2
    HTTP/1.1 libwww-perl/5.833 2
    HTTP/1.1 Mediapartners-Google/2.1 1
    HTTP/1.1 Mediapartners-Google 1
    HTTP/1.0 Mozilla/5.0 1
    


    upload_2015-11-22_5-39-26.png

    Not entirely accurate user agent string though as not all column/fields are on the same column for all user agents
    Code (Text):
    awk '{print $5, $7, $8, $9, $10, $17, $18, $19}' /home/nginx/domains/yourdomain.com/log/sslstats-agent.log | sort | uniq -c | sort -rn
    
        487 HTTP/1.1 Mozilla/4.0 (compatible; MSIE 7.0; CLR 1.0.3705; .NET
        209 HTTP/2.0 Mozilla/5.0 (Macintosh; Intel Mac Gecko) Version/9.0.1 Safari/601.2.7
        140 HTTP/2.0 Mozilla/5.0 (Windows NT 6.2; Safari/537.36 OPR/33.0.1990.115
        132 HTTP/1.0 Y!J-ASR/0.1 crawler (http://www.yahoo-help.jp/app/answers/detail/p/595/a_id/42716/)
         92 HTTP/1.0 NewRelicPinger/1.0 (652248)
         69 HTTP/2.0 Mozilla/5.0 (Macintosh; Intel Mac Gecko) Chrome/46.0.2490.86 Safari/537.36
         56 HTTP/2.0 Mozilla/5.0 (Windows NT 10.0; Chrome/46.0.2490.86 Safari/537.36
         44 HTTP/1.1 Mozilla/5.0 (compatible; Yahoo! Slurp;
         41 HTTP/1.1 Mozilla/5.0 (Macintosh; Intel Mac Gecko) Chrome/46.0.2490.86 Safari/537.36
         39 HTTP/1.1 Mozilla/5.0 (Macintosh; Intel Mac Gecko) Version/9.0.1 Safari/601.2.7
         32 HTTP/1.1 Mozilla/5.0 (Windows NT 10.0; Safari/537.36
         30 HTTP/2.0 Mozilla/5.0 (Windows NT 6.1; Safari/537.36
         27 HTTP/1.1 Mozilla/5.0 (compatible; AhrefsBot/5.0; +http://ahrefs.com/robot/)
         22 HTTP/1.1 Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
         19 HTTP/1.1 Mozilla/5.0+(compatible; UptimeRobot/2.0; http://www.uptimerobot.com/)
         16 HTTP/2.0 Mozilla/5.0 (Linux; Android 5.1.1; Chrome/46.0.2490.76 Mobile Safari/537.36
         13 HTTP/2.0 Mozilla/5.0 (Windows NT 10.0; Safari/537.36
         13 HTTP/2.0 Mozilla/5.0 (Windows NT 10.0;
         13 HTTP/1.1 Mozilla/5.0 (Windows NT 6.1; Safari/537.1
         12 HTTP/1.1 Mozilla/5.0 (Windows NT 6.2; Safari/537.36 OPR/33.0.1990.115
         11 HTTP/1.1 ltx71 - (http://ltx71.com/)
         10 HTTP/2.0 Mozilla/5.0 (iPhone; CPU iPhone AppleWebKit/601.1.46 (KHTML, like
          9 HTTP/2.0 Mozilla/5.0 (Windows NT 6.3; Safari/537.36
          9 HTTP/2.0 Mozilla/5.0 (Windows NT 6.2; Safari/537.36
          9 HTTP/2.0 Mozilla/5.0 (Windows NT 10.0; Chrome/47.0.2526.69 Safari/537.36
          9 HTTP/2.0 Mozilla/5.0 (Windows NT 10.0; Chrome/47.0.2526.58 Safari/537.36
          9 HTTP/1.1 Mozilla/5.0 (Windows NT 6.1; Safari/537.36
          9 HTTP/1.1 Mediapartners-Google
          9 HTTP/1.1 libwww-perl/5.833
          8 HTTP/2.0 Mozilla/5.0 (X11; Linux x86_64;
          8 HTTP/2.0 Mozilla/5.0 (Windows NT 5.1;
          6 HTTP/2.0 Mozilla/5.0 (X11; Ubuntu; Linux
          6 HTTP/1.1 Mozilla/5.0 (iPhone; CPU iPhone AppleWebKit/600.1.4 (KHTML, like
          6 HTTP/1.1 Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)
          5 HTTP/1.1 Mozilla/5.0 (X11; Ubuntu; Linux
          5 HTTP/1.1 Mozilla/5.0 (compatible; proximic; +http://www.proximic.com/info/spider.php)
          5 HTTP/1.1 Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
          5 HTTP/1.1 Mediapartners-Google/2.1
          4 HTTP/1.1 Mozilla/5.0 (Windows NT 6.1;
          4 HTTP/1.1 Mozilla/5.0 (compatible; MSIE 10.0; Touch; Microsoft; Lumia
          3 HTTP/2.0 Mozilla/5.0 (Linux; Android 5.0.1; Gecko) Chrome/47.0.2526.69 Mobile
          2 HTTP/2.0 Mozilla/5.0 (Windows NT 6.1;
          2 HTTP/1.1 Zend_Http_Client
          2 HTTP/1.1 Mozilla/5.0 (Windows NT 10.0; Chrome/46.0.2490.86 Safari/537.36
          2 HTTP/1.0 Mozilla/5.0 (compatible; Qwantify/2.1w; +https://www.qwant.com/)/*
          2 HTTP/1.0 -
          1 HTTP/2.0 Mozilla/5.0 (X11; Linux x86_64) Chrome/45.0.2454.101 Safari/537.36
          1 HTTP/2.0 Mozilla/5.0 (X11; Linux x86_64)
          1 HTTP/2.0 Mozilla/5.0 (Windows NT 6.3;
          1 HTTP/1.1 Python-urllib/2.7
          1 HTTP/1.1 Mozilla/5.0 (X11; Linux x86_64) Chrome/34.0.1847.116 Safari/537.36
          1 HTTP/1.1 Mozilla/5.0 (X11; Linux x86_64;
          1 HTTP/1.1 Mozilla/5.0 (Windows NT 6.3;
          1 HTTP/1.1 Mozilla/5.0 (Windows NT 6.1; Chrome/46.0.2490.71 Safari/537.36
          1 HTTP/1.1 Mozilla/5.0 (Macintosh; Intel Mac Gecko) Version/8.0.8 Safari/600.8.9
          1 HTTP/1.1 Mozilla/5.0 (Macintosh; Intel Mac Gecko) Version/8.0.6 Safari/600.6.3
          1 HTTP/1.1 Mozilla/5.0 (Macintosh; Intel Mac Gecko) Version/6.2.3 Safari/537.85.12
          1 HTTP/1.1 Mozilla/5.0 (compatible; MSIE 10.0;
          1 HTTP/1.1 Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)
          1 HTTP/1.1 Google favicon
          1 HTTP/1.1 ADmantX Platform Semantic Analyzer - support@admantx.com
          1 HTTP/1.1 -
          1 HTTP/1.0 Mozilla/5.0 (Macintosh; Intel Mac
    


    Code (Text):
    awk '{print $5, $7, $8, $9, $10}' /home/nginx/domains/yourdomain.com/log/sslstats-agent.log | sort | uniq -c | sort -rn
        540 HTTP/1.1 Mozilla/4.0 (compatible; MSIE 7.0;
        333 HTTP/2.0 Mozilla/5.0 (Macintosh; Intel Mac
        173 HTTP/2.0 Mozilla/5.0 (Windows NT 6.2;
        147 HTTP/1.0 Y!J-ASR/0.1 crawler (http://www.yahoo-help.jp/app/answers/detail/p/595/a_id/42716/)
        104 HTTP/2.0 Mozilla/5.0 (Windows NT 10.0;
        103 HTTP/1.0 NewRelicPinger/1.0 (652248)
         86 HTTP/1.1 Mozilla/5.0 (Macintosh; Intel Mac
         51 HTTP/1.1 Mozilla/5.0 (compatible; Yahoo! Slurp;
         42 HTTP/2.0 Mozilla/5.0 (Windows NT 6.1;
         37 HTTP/1.1 Mozilla/5.0 (Windows NT 10.0;
         31 HTTP/1.1 Mozilla/5.0 (compatible; AhrefsBot/5.0; +http://ahrefs.com/robot/)
         28 HTTP/1.1 Mozilla/5.0 (Windows NT 6.1;
         24 HTTP/1.1 Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
         21 HTTP/1.1 Mozilla/5.0+(compatible; UptimeRobot/2.0; http://www.uptimerobot.com/)
         17 HTTP/2.0 Mozilla/5.0 (X11; Ubuntu; Linux
         16 HTTP/2.0 Mozilla/5.0 (Linux; Android 5.1.1;
         15 HTTP/1.1 Mozilla/5.0 (Windows NT 6.2;
         15 HTTP/1.1 ltx71 - (http://ltx71.com/)
         10 HTTP/2.0 Mozilla/5.0 (Windows NT 6.3;
         10 HTTP/2.0 Mozilla/5.0 (iPhone; CPU iPhone
         10 HTTP/1.1 libwww-perl/5.833
          9 HTTP/1.1 Mediapartners-Google
          8 HTTP/2.0 Mozilla/5.0 (X11; Linux x86_64;
          8 HTTP/2.0 Mozilla/5.0 (Windows NT 5.1;
          8 HTTP/1.1 Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)
          7 HTTP/1.1 Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
          6 HTTP/1.1 Mozilla/5.0 (X11; Ubuntu; Linux
          6 HTTP/1.1 Mozilla/5.0 (iPhone; CPU iPhone
          5 HTTP/1.1 Mozilla/5.0 (compatible; proximic; +http://www.proximic.com/info/spider.php)
          5 HTTP/1.1 Mozilla/5.0 (compatible; MSIE 10.0;
          5 HTTP/1.1 Mediapartners-Google/2.1
          3 HTTP/2.0 Mozilla/5.0 (Linux; Android 5.0.1;
          2 HTTP/2.0 Mozilla/5.0 (X11; Linux x86_64)
          2 HTTP/1.1 Zend_Http_Client
          2 HTTP/1.0 Mozilla/5.0 (compatible; Qwantify/2.1w; +https://www.qwant.com/)/*
          2 HTTP/1.0 -
          1 HTTP/1.1 Python-urllib/2.7
          1 HTTP/1.1 Mozilla/5.0 (X11; Linux x86_64)
          1 HTTP/1.1 Mozilla/5.0 (X11; Linux x86_64;
          1 HTTP/1.1 Mozilla/5.0 (Windows NT 6.3;
          1 HTTP/1.1 Mozilla/5.0 (compatible; YandexImages/3.0; +http://yandex.com/bots)
          1 HTTP/1.1 Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)
          1 HTTP/1.1 Google favicon
          1 HTTP/1.1 Apache-HttpClient/4.5 (Java/1.8.0_05)
          1 HTTP/1.1 ADmantX Platform Semantic Analyzer
          1 HTTP/1.1 -
          1 HTTP/1.0 Mozilla/5.0 (Macintosh; Intel Mac


    Or we can filter out some of the known non web browser user agent/bots from being logged via nginx mapping of $http_user_agent to variable $nobots
    Code (Text):
    map $http_user_agent $nobots {
        default 1;
        ~'Mediapartners-Google' 0;
        ~'Y!J-ASR' 0;
        ~AhrefsBot 0;
        ~Baiduspider 0;
        ~bingbot 0;
        ~DuckDuckGo-Favicons-Bot 0;
        ~Feedly 0;
        ~Googlebot-Imag 0;
        ~Googlebot 0;
        ~Google 0;
        ~GrapeshotCrawler 0;
        ~linkdexbot 0;
        ~NewRelicPinger 0;
        ~Pingdom 0;
        ~proximic 0;
        ~Qwantify 0;
        ~R6_CommentReader 0;
        ~TweetmemeBot 0;
        ~UptimeRobot 0;
        ~www.radian6.com 0;
        ~Yahoo 0;
        ~YandexBot 0;
        ~YandexImages 0;
    }
    

    so only log to /home/nginx/domains/yourdomain.com/log/sslstats-agent-nobots.log if not any of the above listed mapped bot identifiers
    Code (Text):
    access_log /home/nginx/domains/yourdomain.com/log/sslstats-agent-nobots.log ssl_customagent buffer=256k flush=5m if=$nobots;


    Code (Text):
    awk '{print $5}' /home/nginx/domains/yourdomain.com/log/sslstats-agent-nobots.log | sort | uniq -c | sort -rn | awk '{print $2, $1}'
    
    Code (Text):
    awk '{print $2}' /home/nginx/domains/yourdomain.com/log/sslstats-agent-nobots.log | sort | uniq -c | sort -rn | awk '{print $2, $1}'
    
    Code (Text):
    awk '{print $1}' /home/nginx/domains/yourdomain.com/log/sslstats-agent-nobots.log | sort | uniq -c | sort -rn | awk '{print $2, $1}'
    
    Code (Text):
    awk '{print $6}' /home/nginx/domains/yourdomain.com/log/sslstats-agent-nobots.log | sort | uniq -c | sort -rn | awk '{print $2, $1}'
    
    Code (Text):
    awk '{print $7}' /home/nginx/domains/yourdomain.com/log/sslstats-agent-nobots.log | sort | uniq -c | sort -rn | awk '{print $2, $1}'
    

    Code (Text):
    awk '{print $5, $7}' /home/nginx/domains/yourdomain.com/log/sslstats-agent-nobots.log | sort | uniq -c | sort -rn | awk '{print $2, $3, $1}';
    

    Code (Text):
    awk '{print $5, $7, $8, $9, $10, $17, $18, $19}' /home/nginx/domains/yourdomain.com/log/sslstats-agent-nobots.log | sort | uniq -c | sort -rn;
    
    awk '{print $5, $7, $8, $9, $10}' /home/nginx/domains/yourdomain.com/log/sslstats-agent-nobots.log | sort | uniq -c | sort -rn


    Code (Text):
    HTTP/2.0 92
    HTTP/1.1 75
    1
    
    ECDHE-RSA-CHACHA20-POLY1305 85
    ECDHE-RSA-AES128-GCM-SHA256 82
    1
    
    TLSv1.2 167
    1
    
    200 122
    204 31
    304 6
    404 4
    301 3
    303 1
    1
    
    Mozilla/5.0 118
    Mozilla/4.0 44
    ltx71 4
    libwww-perl/5.833 1
    1
    
    HTTP/2.0 Mozilla/5.0 92
    HTTP/1.1 Mozilla/4.0 44
    HTTP/1.1 Mozilla/5.0 26
    HTTP/1.1 ltx71 4
    HTTP/1.1 libwww-perl/5.833 1
      1
    

    Code (Text):
         44 HTTP/1.1 Mozilla/4.0 (compatible; MSIE 7.0; CLR 1.0.3705; .NET
         43 HTTP/2.0 Mozilla/5.0 (Windows NT 6.2; Safari/537.36 OPR/33.0.1990.115
         34 HTTP/2.0 Mozilla/5.0 (Macintosh; Intel Mac Gecko) Chrome/46.0.2490.86 Safari/537.36
         11 HTTP/1.1 Mozilla/5.0 (Macintosh; Intel Mac Gecko) Chrome/46.0.2490.86 Safari/537.36
          9 HTTP/1.1 Mozilla/5.0 (Windows NT 10.0; Safari/537.36
          6 HTTP/2.0 Mozilla/5.0 (Windows NT 10.0;
          5 HTTP/2.0 Mozilla/5.0 (Windows NT 10.0; Chrome/46.0.2490.86 Safari/537.36
          5 HTTP/1.1 Mozilla/5.0 (Windows NT 6.2; Safari/537.36 OPR/33.0.1990.115
          4 HTTP/1.1 ltx71 - (http://ltx71.com/)
          1 HTTP/2.0 Mozilla/5.0 (X11; Linux x86_64)
          1 HTTP/2.0 Mozilla/5.0 (Windows NT 10.0; Chrome/47.0.2526.69 Safari/537.36
          1 HTTP/2.0 Mozilla/5.0 (Windows NT 10.0; Chrome/47.0.2526.58 Safari/537.36
          1 HTTP/2.0 Mozilla/5.0 (Macintosh; Intel Mac
          1 HTTP/1.1 Mozilla/5.0 (Windows NT 10.0;
          1 HTTP/1.1 libwww-perl/5.833
          1
    
         44 HTTP/1.1 Mozilla/4.0 (compatible; MSIE 7.0;
         43 HTTP/2.0 Mozilla/5.0 (Windows NT 6.2;
         35 HTTP/2.0 Mozilla/5.0 (Macintosh; Intel Mac
         13 HTTP/2.0 Mozilla/5.0 (Windows NT 10.0;
         11 HTTP/1.1 Mozilla/5.0 (Macintosh; Intel Mac
         10 HTTP/1.1 Mozilla/5.0 (Windows NT 10.0;
          5 HTTP/1.1 Mozilla/5.0 (Windows NT 6.2;
          4 HTTP/1.1 ltx71 - (http://ltx71.com/)
          1 HTTP/2.0 Mozilla/5.0 (X11; Linux x86_64)
          1 HTTP/1.1 libwww-perl/5.833
          1
    


    Filtered bots changes the make up

    upload_2015-11-22_7-55-24.png

    upload_2015-11-22_7-55-33.png

    upload_2015-11-22_7-59-26.png

    upload_2015-11-22_7-55-51.png

    top 10 HTTP/TLS protocols with uri links and user agents

    Code (Text):
    awk '{print $5, $1, $2, $4, $6, $7, $8, $9, $10, $17, $18, $19}' /home/nginx/domains/yourdomain.com/log/sslstats-agent-nobots.log | sort | uniq -c | sort -rn | head -n10


    Code (Text):
       3073 HTTP/1.1 TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256 / 200 Mozilla/4.0 (compatible; MSIE 7.0; CLR 1.0.3705; .NET
        268 HTTP/1.1 TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256 /index.php?tabalerts 200 Mozilla/5.0 (Windows NT 10.0; Safari/537.36
        200 HTTP/2.0 TLSv1.2 ECDHE-RSA-CHACHA20-POLY1305 /index.php?tabalerts 200 Mozilla/5.0 (Macintosh; Intel Mac Gecko) Chrome/46.0.2490.80 Safari/537.36
         94 HTTP/2.0 TLSv1.2 ECDHE-RSA-CHACHA20-POLY1305 /index.php?tabalerts 200 Mozilla/5.0 (Windows NT 10.0; Chrome/46.0.2490.86 Safari/537.36
         49 HTTP/2.0 TLSv1.2 ECDHE-RSA-CHACHA20-POLY1305 /index.php?tabalerts 200 Mozilla/5.0 (Linux; Android 4.4.4; Chrome/46.0.2490.76 Mobile Safari/537.36
         49 HTTP/2.0 TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256 /index.php?tabalerts 200 Mozilla/5.0 (Macintosh; Intel Mac Gecko) Version/9.0.1 Safari/601.2.7
         39 HTTP/1.1 TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256 / 200 libwww-perl/5.833
         38 HTTP/2.0 TLSv1.2 ECDHE-RSA-CHACHA20-POLY1305 /?_xfResponseType=json 200 Mozilla/5.0 (Windows NT 10.0; Chrome/47.0.2526.69 Safari/537.36
         38 HTTP/2.0 TLSv1.2 ECDHE-RSA-CHACHA20-POLY1305 /?_xfResponseType=json 200 Mozilla/5.0 (Windows NT 10.0; Chrome/47.0.2526.58 Safari/537.36
         22 HTTP/2.0 TLSv1.2 ECDHE-RSA-CHACHA20-POLY1305 /index.php?tabalerts 200 Mozilla/5.0 (Windows NT 6.2; Safari/537.36 OPR/33.0.1990.115 


    So finding out what Mozilla/4.0 users are. It seems it's my Constellix Sonar server monitoring as the ips belong to Constellix !

    Code (Text):
    grep 'Mozilla\/4.0' access.log | awk '{print $1}' | sort | uniq -c | sort -rn | head -n10
      44501 178.62.112.179
      44088 176.58.90.149
      43434 212.71.238.144
      43409 149.154.152.196
      40960 178.62.215.141
      40781 128.199.194.53
      40540 199.38.182.28
      40137 106.186.122.163
      38101 23.239.17.158
      37719 208.111.40.251


    Constellix Sonar

    upload_2015-11-22_11-7-58.png

    So lets exclude Mozilla/4.0 user agent from top 10 listing and exclude the uri links

    Code (Text):
    awk '{print $5, $1, $2, $6, $7, $8, $9, $10, $17, $18, $19}' /home/nginx/domains/yourdomain.com/log/sslstats-agent-nobots.log | sort | uniq -c | sort -rn | grep -v 'Mozilla\/4.0' | head -n10
        341 HTTP/1.1 TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256 200 Mozilla/5.0 (Windows NT 10.0; Safari/537.36
        247 HTTP/2.0 TLSv1.2 ECDHE-RSA-CHACHA20-POLY1305 200 Mozilla/5.0 (Macintosh; Intel Mac Gecko) Chrome/46.0.2490.80 Safari/537.36
        149 HTTP/2.0 TLSv1.2 ECDHE-RSA-CHACHA20-POLY1305 200 Mozilla/5.0 (Windows NT 6.2; Safari/537.36 OPR/33.0.1990.115
        116 HTTP/2.0 TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256 200 Mozilla/5.0 (Macintosh; Intel Mac Gecko) Version/9.0.1 Safari/601.2.7
        110 HTTP/1.1 TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256 200 Mozilla/5.0 (Macintosh; Intel Mac Gecko) Chrome/46.0.2490.86 Safari/537.36
        107 HTTP/2.0 TLSv1.2 ECDHE-RSA-CHACHA20-POLY1305 200 Mozilla/5.0 (Windows NT 10.0; Chrome/46.0.2490.86 Safari/537.36
        101 HTTP/2.0 TLSv1.2 ECDHE-RSA-CHACHA20-POLY1305 204 Mozilla/5.0 (Windows NT 6.2; Safari/537.36 OPR/33.0.1990.115
         86 HTTP/2.0 TLSv1.2 ECDHE-RSA-CHACHA20-POLY1305 204 Mozilla/5.0 (Macintosh; Intel Mac Gecko) Chrome/46.0.2490.80 Safari/537.36
         81 HTTP/2.0 TLSv1.2 ECDHE-RSA-CHACHA20-POLY1305 200 Mozilla/5.0 (Macintosh; Intel Mac Gecko) Chrome/46.0.2490.86 Safari/537.36
         72 HTTP/1.1 TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256 404 ltx71 - (http://ltx71.com/)
    


    drill down top 20 and only HTTP status = 200 requests excluding 404s etc
    Code (Text):
    awk '$6 == '200' {print $5, $1, $2, $7, $8, $9, $10, $17, $18, $19}' /home/nginx/domains/yourdomain.com/log/sslstats-agent-nobots.log | sort | uniq -c | sort -rn | grep -v 'Mozilla\/4.0' | head -n20
        349 HTTP/1.1 TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256 Mozilla/5.0 (Windows NT 10.0; Safari/537.36
        247 HTTP/2.0 TLSv1.2 ECDHE-RSA-CHACHA20-POLY1305 Mozilla/5.0 (Macintosh; Intel Mac Gecko) Chrome/46.0.2490.80 Safari/537.36
        158 HTTP/2.0 TLSv1.2 ECDHE-RSA-CHACHA20-POLY1305 Mozilla/5.0 (Windows NT 6.2; Safari/537.36 OPR/33.0.1990.115
        121 HTTP/2.0 TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256 Mozilla/5.0 (Macintosh; Intel Mac Gecko) Version/9.0.1 Safari/601.2.7
        110 HTTP/1.1 TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256 Mozilla/5.0 (Macintosh; Intel Mac Gecko) Chrome/46.0.2490.86 Safari/537.36
        109 HTTP/2.0 TLSv1.2 ECDHE-RSA-CHACHA20-POLY1305 Mozilla/5.0 (Windows NT 10.0; Chrome/46.0.2490.86 Safari/537.36
         81 HTTP/2.0 TLSv1.2 ECDHE-RSA-CHACHA20-POLY1305 Mozilla/5.0 (Macintosh; Intel Mac Gecko) Chrome/46.0.2490.86 Safari/537.36
         69 HTTP/2.0 TLSv1.2 ECDHE-RSA-CHACHA20-POLY1305 Mozilla/5.0 (Windows NT 10.0; Safari/537.36
         49 HTTP/2.0 TLSv1.2 ECDHE-RSA-CHACHA20-POLY1305 Mozilla/5.0 (Linux; Android 4.4.4; Chrome/46.0.2490.76 Mobile Safari/537.36
         48 HTTP/1.1 TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256 Mozilla/5.0 (Windows NT 6.2; Safari/537.36 OPR/33.0.1990.115
         46 HTTP/1.1 TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256 Mozilla/5.0 (Macintosh; Intel Mac Gecko) Chrome/46.0.2490.80 Safari/537.36
         45 HTTP/2.0 TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256 Mozilla/5.0 (Windows NT 10.0;
         43 HTTP/2.0 TLSv1.2 ECDHE-RSA-CHACHA20-POLY1305 Mozilla/5.0 (Windows NT 10.0; Chrome/47.0.2526.69 Safari/537.36
         43 HTTP/2.0 TLSv1.2 ECDHE-RSA-CHACHA20-POLY1305 Mozilla/5.0 (Windows NT 10.0; Chrome/47.0.2526.58 Safari/537.36
         43 HTTP/1.1 TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256 libwww-perl/5.833
         30 HTTP/1.1 TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256 -
         20 HTTP/2.0 TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256 Mozilla/5.0 (X11; Linux x86_64;
         19 HTTP/1.1 TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256 Mozilla/5.0 (Macintosh; Intel Mac Gecko) Version/9.0.1 Safari/601.2.7
         18 HTTP/1.1 TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256 Mozilla/5.0 (iPad; CPU OS (KHTML, like Gecko)
         16 HTTP/2.0 TLSv1.2 ECDHE-RSA-CHACHA20-POLY1305 Mozilla/5.0 (Windows NT 6.3; Safari/537.36
    


    Then exclude the use-agent to get a more relevant end user usage profile
    Code (Text):
    awk '$6 == '200' {print $5, $1, $2}' /home/nginx/domains/yourdomain.com/log/sslstats-agent-nobots.log | sort | uniq -c | sort -rn | grep -v 'Mozilla\/4.0' | awk '{print $2, $4, $1}'                         
    
    HTTP/1.1 ECDHE-RSA-AES128-GCM-SHA256 4415
    HTTP/2.0 ECDHE-RSA-CHACHA20-POLY1305 1026
    HTTP/2.0 ECDHE-RSA-AES128-GCM-SHA256 246
    HTTP/1.0 ECDHE-RSA-AES128-GCM-SHA256 74
    HTTP/1.1 ECDHE-RSA-AES128-SHA 29
    HTTP/1.1 ECDHE-RSA-AES128-SHA256 19
    HTTP/1.1 DHE-RSA-AES128-GCM-SHA256 13
    HTTP/1.1 ECDHE-RSA-CHACHA20-POLY1305 8
    HTTP/1.0 ECDHE-RSA-AES128-SHA 3
    HTTP/1.1 DHE-RSA-AES128-SHA 1
    


    upload_2015-11-22_11-33-1.png

    Edit: FYI, server is setup to not log css, js or image requests to access_logs so only php/json requests logged so may skew HTTP/2 usage
     
    Last edited: Nov 23, 2015
  3. eva2000

    eva2000 Administrator Staff Member

    26,598
    6,104
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +8,951
    Local Time:
    6:31 AM
    Nginx 1.11.x
    MariaDB 5.5
    Updated nginx map for excluding bots from logging and added MSIE 7 agent which is for Constellix logging + re-enabled logging for css, js, images and webfonts requests

    Code (Text):
    map $http_user_agent $nobots {
        default 1;
        ~'Mozilla/4.0' 0;
        "~MSIE 7.0" 0;
        ~'Mediapartners-Google' 0;
        "~Y!J-ASR" 0;
        ~AhrefsBot 0;
        ~Baiduspider 0;
        ~bingbot 0;
        ~DuckDuckGo-Favicons-Bot 0;
        ~Feedly 0;
        ~FlipboardRSS 0;
        ~FlipboardProxy 0;
        ~Googlebot-Imag 0;
        ~Googlebot 0;
        ~Google 0;
        ~GrapeshotCrawler 0;
        ~linkdexbot 0;
        ~NewRelicPinger 0;
        ~Pingdom 0;
        ~proximic 0;
        ~Qwantify 0;
        ~R6_CommentReader 0;
        ~TweetmemeBot 0;
        ~UptimeRobot 0;
        ~www.radian6.com 0;
        ~Yahoo 0;
        ~YandexBot 0;
        ~YandexImages 0;
    }


    minor filter change from 200 status only to NOT 404 status
    Code (Text):
    awk '$6 != '404' {print $5, $1, $2}' /home/nginx/domains/yourdomain.com/log/sslstats-agent-nobots.log | sort | uniq -c | sort -rn | grep -v 'Mozilla\/4.0' | awk '{print $2, $4, $1}' 


    Code (Text):
    HTTP/2.0 ECDHE-RSA-CHACHA20-POLY1305 23600
    HTTP/1.1 ECDHE-RSA-AES128-GCM-SHA256 12824
    HTTP/2.0 ECDHE-RSA-AES128-GCM-SHA256 10910
    HTTP/1.0 ECDHE-RSA-AES128-GCM-SHA256 4005
    HTTP/1.1 ECDHE-RSA-AES128-SHA 690
    HTTP/1.1 DHE-RSA-AES128-SHA 610
    HTTP/1.1 ECDHE-RSA-AES128-SHA256 485
    HTTP/1.1 DHE-RSA-AES128-GCM-SHA256 304
    HTTP/1.1 ECDHE-RSA-CHACHA20-POLY1305 143
    HTTP/1.1 DHE-RSA-AES128-SHA256 137
    HTTP/1.1 ECDHE-RSA-AES128-SHA 95
    HTTP/1.0 DHE-RSA-AES128-SHA 87
    HTTP/1.0 ECDHE-RSA-AES128-SHA 50
    HTTP/1.1 ECDHE-RSA-AES128-SHA 9
    HTTP/1.1 DHE-RSA-AES128-SHA 9
    HTTP/1.0 ECDHE-RSA-AES128-SHA 4
    - - 3
    HTTP/1.1 - 2
      1
    
    HTTP/2.0 34527
    HTTP/1.1 15836
    HTTP/1.0 4956
    - 3
    1
    
    ECDHE-RSA-AES128-GCM-SHA256 28774
    ECDHE-RSA-CHACHA20-POLY1305 23747
    ECDHE-RSA-AES128-SHA 929
    DHE-RSA-AES128-SHA 882
    ECDHE-RSA-AES128-SHA256 487
    DHE-RSA-AES128-GCM-SHA256 306
    DHE-RSA-AES128-SHA256 192
    - 5
    1
    


    And the original 1st post logged stats with excluded css,js,image logging and included Constellix and non-browser client requests.
    Code (Text):
    HTTP/1.1 181237
    HTTP/2.0 66301
    HTTP/1.0 32534
    15
    
    ECDHE-RSA-AES128-GCM-SHA256 197478
    ECDHE-RSA-CHACHA20-POLY1305 46910
    ECDHE-RSA-AES128-SHA 23278
    ECDHE-RSA-AES128-SHA256 6489
    DHE-RSA-AES128-GCM-SHA256 3686
    DHE-RSA-AES128-SHA 2035
    DHE-RSA-AES128-SHA256 193
    - 11
    ECDHE-RSA-AES256-SHA 7
    
    TLSv1.2 254818
    TLSv1 25143
    TLSv1.1 115
    - 11
    
     
    Last edited: Nov 25, 2015
  4. eva2000

    eva2000 Administrator Staff Member

    26,598
    6,104
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +8,951
    Local Time:
    6:31 AM
    Nginx 1.11.x
    MariaDB 5.5
    Almost forgot about my stats logging for this so using latest configuration outlined here, my current stats for this very forum

    bot filtered version

    upload_2015-12-4_3-27-51.png

    upload_2015-12-4_3-28-50.png

    upload_2015-12-4_3-32-1.png

    Code (Text):
    HTTP/2.0 ECDHE-RSA-CHACHA20-POLY1305 101784
    HTTP/2.0 ECDHE-RSA-AES128-GCM-SHA256 52240
    HTTP/1.1 ECDHE-RSA-AES128-GCM-SHA256 49986
    HTTP/1.0 ECDHE-RSA-AES128-GCM-SHA256 16359
    HTTP/1.1 ECDHE-RSA-AES128-SHA 3562
    HTTP/1.1 DHE-RSA-AES128-SHA 2277
    HTTP/1.1 ECDHE-RSA-AES128-SHA256 1829
    HTTP/1.1 DHE-RSA-AES128-GCM-SHA256 1671
    HTTP/1.1 ECDHE-RSA-CHACHA20-POLY1305 887
    HTTP/1.1 DHE-RSA-AES128-SHA256 282
    HTTP/1.1 ECDHE-RSA-AES128-SHA 212
    HTTP/1.1 ECDHE-RSA-AES128-SHA 207
    HTTP/1.0 ECDHE-RSA-AES128-SHA 142
    HTTP/1.0 DHE-RSA-AES128-SHA 134
    HTTP/1.0 ECDHE-RSA-AES128-SHA 129
    HTTP/1.1 ECDHE-RSA-AES256-GCM-SHA384 14
    HTTP/1.1 DHE-RSA-AES128-SHA 9
    HTTP/1.1 DHE-RSA-AES128-SHA 3
    HTTP/1.0 ECDHE-RSA-AES128-SHA 2
    HTTP/1.0 DHE-RSA-AES128-SHA 2
    HTTP/1.1 ECDHE-RSA-AES256-SHA 1
    HTTP/1.0 ECDHE-RSA-AES256-SHA 1
    HTTP/1.0 DHE-RSA-AES256-SHA 1
    HTTP/1.0 ECDHE-RSA-AES256-SHA384 1
    HTTP/1.0 ECDHE-RSA-AES256-SHA 1
    HTTP/1.0 ECDHE-RSA-AES256-GCM-SHA384 1
    HTTP/1.0 ECDHE-RSA-AES128-SHA256 1
    HTTP/1.0 DHE-RSA-AES256-SHA256 1
    HTTP/1.0 DHE-RSA-AES256-SHA 1
    HTTP/1.0 DHE-RSA-AES256-GCM-SHA384 1
    HTTP/1.0 DHE-RSA-AES128-SHA256 1
    HTTP/1.0 DHE-RSA-AES128-SHA 1
    HTTP/1.0 DHE-RSA-AES128-GCM-SHA256 1
    HTTP/1.0 ECDHE-RSA-AES256-SHA 1
    HTTP/1.0 DHE-RSA-AES256-SHA 1
    400 - 1
      1
    
    HTTP/2.0 154092
    HTTP/1.1 68946
    HTTP/1.0 20831
    - 24
    400 1
    1
    
    ECDHE-RSA-AES128-GCM-SHA256 129127
    ECDHE-RSA-CHACHA20-POLY1305 102692
    ECDHE-RSA-AES128-SHA 4988
    DHE-RSA-AES128-SHA 3164
    ECDHE-RSA-AES128-SHA256 1835
    DHE-RSA-AES128-GCM-SHA256 1677
    DHE-RSA-AES128-SHA256 350
    - 36
    ECDHE-RSA-AES256-GCM-SHA384 15
    ECDHE-RSA-AES256-SHA 4
    DHE-RSA-AES256-SHA 3
    ECDHE-RSA-AES256-SHA384 1
    DHE-RSA-AES256-SHA256 1
    DHE-RSA-AES256-GCM-SHA384 1
    1


    original version
    Code (Text):
    HTTP/1.1 461623
    HTTP/2.0 185850
    HTTP/1.0 74495
    37
    
    ECDHE-RSA-AES128-GCM-SHA256 508097
    ECDHE-RSA-CHACHA20-POLY1305 125845
    ECDHE-RSA-AES128-SHA 57866
    DHE-RSA-AES128-GCM-SHA256 10618
    DHE-RSA-AES128-SHA 9184
    ECDHE-RSA-AES128-SHA256 8830
    DHE-RSA-AES128-SHA256 351
    ECDHE-RSA-AES256-GCM-SHA384 15
    ECDHE-RSA-AES256-SHA 11
    DHE-RSA-AES256-SHA 3
    ECDHE-RSA-AES256-SHA384 1
    DHE-RSA-AES256-SHA256 1
    DHE-RSA-AES256-GCM-SHA384 1
    - 1182
    
    TLSv1.2 654213
    TLSv1 66345
    TLSv1.1 265
    - 1182


    upload_2015-12-4_3-26-53.png

    upload_2015-12-4_3-27-1.png

    upload_2015-12-4_3-27-11.png
     
    Last edited: Dec 4, 2015