Want to subscribe to topics you're interested in?
Become a Member

Featured Cloudflare Cloudflare custom Nginx logging

Discussion in 'Domains, DNS, Email & SSL Certificates' started by eva2000, May 14, 2018.

Thread Status:
Not open for further replies.
  1. eva2000

    eva2000 Administrator Staff Member

    36,023
    7,901
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,183
    Local Time:
    6:03 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    If you're using Cloudflare in front of your Centmin Mod Nginx web server, then you may want to add custom Nginx access logging for Cloudflare related metrics such as CF-RAY header as well as SSL protocol and ssl ciphers served (previous example).

    I added additional logging formats for cf_custom, cf_custom2 and cf_custom3 into /usr/local/nginx/conf/nginx.conf
    Code (Text):
    log_format cf_custom '$remote_addr - $remote_user [$time_local]  '
                       '"$request" $status $body_bytes_sent '
                       '"$http_referer" "$http_user_agent" '
                       '$http_cf_ray';
                   
    log_format cf_custom2 '$remote_addr - $remote_user [$time_local] $request '
                   '"$status" $body_bytes_sent "$http_referer" '
                   '"$http_user_agent" "$http_x_forwarded_for" "$gzip_ratio"'
                   ' "$connection" "$connection_requests" "$request_time" $http_cf_ray';
               
    
    log_format cf_custom3 '$remote_addr - $remote_user [$time_local] $request '
                   '"$status" $body_bytes_sent "$http_referer" '
                   '"$http_user_agent" "$http_x_forwarded_for" "$gzip_ratio"'
                   ' "$connection" "$connection_requests" "$request_time" $http_cf_ray '
                   '$ssl_protocol $ssl_cipher';
    

    Then in my Nginx vhost config logs add 2 additional access log using defined custom log formats i.e. cf_custom2 and cf_custom3.

    So it changes from
    Code (Text):
      access_log /home/nginx/domains/domain.com/log/access.log combined buffer=256k flush=5m;
      error_log /home/nginx/domains/domain.com/log/error.log;
    

    to the below where cf-access.log only contains CF-RAY header and cfssl-access.log has CF-RAY with additional ssl protocol and ssl cipher logging.
    Code (Text):
      access_log /home/nginx/domains/domain.com/log/access.log combined buffer=256k flush=5m;
      access_log /home/nginx/domains/domain.com/log/cf-access.log cf_custom2 buffer=256k flush=5m;
      access_log /home/nginx/domains/domain.com/log/cfssl-access.log cf_custom3 buffer=256k flush=5m;
      error_log /home/nginx/domains/domain.com/log/error.log;
    

    Then you can use grep, fgrep, awk and other SSH tools to manipulate and inspect those new logs.

    Example of the field numbers for cf-access.log entry which logged Cloudflare request id and CF-RAY header = 41a8a20c1c953343-HKG meaning the request was served from Cloudflare Hong Kong datacenter :)
    Code (Text):
    tail -1 /home/nginx/domains/domain.com/log/cf-access.log | awk '{print NR": "$0; for(i=1;i<=NF;++i)print ""i":  "$i}'
    1: 180.76.15.158 - - [13/May/2018:22:41:32 +0000] GET /tags/php-54/ HTTP/1.1 "200" 45399 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)" "180.76.15.158" "-" "39" "1" "0.100" 41a8a20c1c953343-HKG
    1:  180.76.15.158
    2:  -
    3:  -
    4:  [13/May/2018:22:41:32
    5:  +0000]
    6:  GET
    7:  /tags/php-54/
    8:  HTTP/1.1
    9:  "200"
    10:  45399
    11:  "-"
    12:  "Mozilla/5.0
    13:  (compatible;
    14:  Baiduspider/2.0;
    15:  +http://www.baidu.com/search/spider.html)"
    16:  "180.76.15.158"
    17:  "-"
    18:  "39"
    19:  "1"
    20:  "0.100"
    21:  41a8a20c1c953343-HKG
    

    Example of field numbers for cfssl-access.log entry which shows CF-RAY header = 41a8a20c1c953343-HKG as well as SSL protocol used = TLS v1.2 and SSL cipher = ECDHE-RSA-AES256-GCM-SHA384. These 2 values are what Centmin Mod Nginx served to upstream Cloudflare (client) and not what Cloudflare served to the visitor. When Cloudflare is in front of Centmin Mod Nginx, then the SSL handshake and connection established is between Cloudflare and Centmin Mod Nginx. In my case it was between the middle Sucuri Cloudproxy which sandwiched between Cloudflare and Centmin Mod Nginx.
    Code (Text):
    tail -1 /home/nginx/domains/domain.com/log/cfssl-access.log | awk '{print NR": "$0; for(i=1;i<=NF;++i)print ""i":  "$i}'
    1: 180.76.15.158 - - [13/May/2018:22:41:32 +0000] GET /tags/php-54/ HTTP/1.1 "200" 45399 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)" "180.76.15.158" "-" "39" "1" "0.100" 41a8a20c1c953343-HKG TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384
    1:  180.76.15.158
    2:  -
    3:  -
    4:  [13/May/2018:22:41:32
    5:  +0000]
    6:  GET
    7:  /tags/php-54/
    8:  HTTP/1.1
    9:  "200"
    10:  45399
    11:  "-"
    12:  "Mozilla/5.0
    13:  (compatible;
    14:  Baiduspider/2.0;
    15:  +http://www.baidu.com/search/spider.html)"
    16:  "180.76.15.158"
    17:  "-"
    18:  "39"
    19:  "1"
    20:  "0.100"
    21:  41a8a20c1c953343-HKG
    22:  TLSv1.2
    23:  ECDHE-RSA-AES256-GCM-SHA384
    

    Inspecting cf-access.log to find the top Cloudflare datacenter regions in requests by inspecting CF-RAY headers logged

    For single /home/nginx/domains/domain.com/log/cf-access.log log
    Code (Text):
    awk '{print $NF}' /home/nginx/domains/domain.com/log/cf-access.log | awk -F '-' '{print $2}' | sort | uniq -c | sort -rn
       492
        71 SJC
        69 NRT
        66 DME
        64 ORD
        52 AMS
        50 SIN
        44 IAD
        40 LAX
        32 EWR
        31 LHR
        30 FRA
        23 HKG
        21 YVR
        17 BOM
        12 ATL
        11 DFW
         6 WAW
         6 MEL
         6 DUB
         5 HEL
         5 CDG
         4 SOF
         4 PDX
         4 ARN
         3 STL
         3 BCN
         3 ATH
         2 SEA
         2 EZE
         2 DEN
         1 VIE
         1 MIA
         1 LUX
         1 LAS
         1 KBP
    

    Nginx logs will get rotated and compressed so you may want to inspect all logs current uncompressed + log rotated and compressed together so you can use zcat/pzcat command instead.
    Code (Text):
    pzcat -f /home/nginx/domains/domain.com/log/cf-access.log{,-*} | awk '{print $NF}' | awk -F '-' '{print $2}' |  sed -e 's|\"||g' | sort | uniq -c | sort -rn
    
        281 DME
        230 NRT
        220 YVR
        135 SIN
        131 DFW
        131
        107 IAD
        101 AMS
         91 SJC
         76 LHR
         74 FRA
         67 ORD
         60 EWR
         49 PDX
         36 HKG
         30 BOM
         26 KIX
         24 LAX
         24 BCN
         19 MEL
         19 GRU
         12 CDG
          8 BOS
          7 AKL
          6 WAW
          6 OTP
          6 ATL
          5 PRG
          5 ATH
          4 SEA
          4 DUB
          4 ARN
          3 MIA
          2 KBP
          2 HEL
          2 EZE
          2 DEN
          2 BKK
          1 SOF
          1 LUX
          1 LAS
    

    You'll find the Cloudflare 3 letter datacenter region codes as outlined here.
    If you need to know the IATA airport codes use Google :)
    Code (Text):
    pzcat -f /home/nginx/domains/domain.com/log/cf-access.log{,-*} | awk '{print $NF}' | awk -F '-' '{print $2}' |  sed -e 's|\"||g' | sort | uniq -c | sort -rn | awk '{print $1, $2, "https://google.com/search?q=iata+code:+"$2}'    
    
    398 YVR https://google.com/search?q=iata+code:+YVR
    390 DME https://google.com/search?q=iata+code:+DME
    389 NRT https://google.com/search?q=iata+code:+NRT
    209 SIN https://google.com/search?q=iata+code:+SIN
    171 DFW https://google.com/search?q=iata+code:+DFW
    152  https://google.com/search?q=iata+code:+
    146 AMS https://google.com/search?q=iata+code:+AMS
    134 IAD https://google.com/search?q=iata+code:+IAD
    130 SJC https://google.com/search?q=iata+code:+SJC
    116 FRA https://google.com/search?q=iata+code:+FRA
    114 LHR https://google.com/search?q=iata+code:+LHR
    99 ORD https://google.com/search?q=iata+code:+ORD
    99 EWR https://google.com/search?q=iata+code:+EWR
    72 PDX https://google.com/search?q=iata+code:+PDX
    53 GRU https://google.com/search?q=iata+code:+GRU
    41 HKG https://google.com/search?q=iata+code:+HKG
    41 BOM https://google.com/search?q=iata+code:+BOM
    29 LAX https://google.com/search?q=iata+code:+LAX
    28 MEL https://google.com/search?q=iata+code:+MEL
    26 KIX https://google.com/search?q=iata+code:+KIX
    26 BCN https://google.com/search?q=iata+code:+BCN
    24 CDG https://google.com/search?q=iata+code:+CDG
    13 ARN https://google.com/search?q=iata+code:+ARN
    13 AKL https://google.com/search?q=iata+code:+AKL
    8 WAW https://google.com/search?q=iata+code:+WAW
    8 MIA https://google.com/search?q=iata+code:+MIA
    8 BOS https://google.com/search?q=iata+code:+BOS
    6 SEA https://google.com/search?q=iata+code:+SEA
    6 OTP https://google.com/search?q=iata+code:+OTP
    6 DUB https://google.com/search?q=iata+code:+DUB
    6 DEN https://google.com/search?q=iata+code:+DEN
    6 ATL https://google.com/search?q=iata+code:+ATL
    6 ATH https://google.com/search?q=iata+code:+ATH
    5 PRG https://google.com/search?q=iata+code:+PRG
    4 EZE https://google.com/search?q=iata+code:+EZE
    2 SOF https://google.com/search?q=iata+code:+SOF
    2 LUX https://google.com/search?q=iata+code:+LUX
    2 LAS https://google.com/search?q=iata+code:+LAS
    2 KBP https://google.com/search?q=iata+code:+KBP
    2 HEL https://google.com/search?q=iata+code:+HEL
    2 BKK https://google.com/search?q=iata+code:+BKK
    1 CPH https://google.com/search?q=iata+code:+CPH
    
     
    • Informative Informative x 2
  2. eva2000

    eva2000 Administrator Staff Member

    36,023
    7,901
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,183
    Local Time:
    6:03 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    Quick look at cfssl-access.log after 24+ hrs of logging on just one server behind Cloudflare.

    In log directory cfssl-access.log and cf-access.log logs both current, past day and gzip rotated prior day before that.
    Code (Text):
    ls -lhrt | egrep 'cfssl-|cf-'
    -rw-r--r-- 1 nginx nginx  16M May 16 04:16 cfssl-access.log-20180516
    -rw-r--r-- 1 nginx nginx  15M May 16 04:16 cf-access.log-20180516
    -rw-r--r-- 1 nginx nginx 2.6M May 16 04:16 cf-access.log-20180515.gz
    -rw-r--r-- 1 nginx nginx 2.7M May 16 04:16 cfssl-access.log-20180515.gz
    -rw-r--r-- 1 nginx nginx  16M May 17 00:15 cfssl-access.log
    -rw-r--r-- 1 nginx nginx  14M May 17 00:15 cf-access.log
    

    Look at Cloudflare datacenter regions hit according to IATA airport codes for top 10 datacenters via custom cf-access.log
    Code (Text):
    pzcat -f cf-access.log{,-*} | awk '{print $NF}' | awk -F '-' '{print $2}' |  sed -e 's|\"||g' | sort | uniq -c | sort -rn | awk '{print $1, $2, "https://google.com/search?q=iata+code:+"$2}' | head -n20
    12653 IAD https://google.com/search?q=iata+code:+IAD
    12290 FRA https://google.com/search?q=iata+code:+FRA
    11313 SEA https://google.com/search?q=iata+code:+SEA
    9698 AMS https://google.com/search?q=iata+code:+AMS
    9602 SIN https://google.com/search?q=iata+code:+SIN
    9296 DFW https://google.com/search?q=iata+code:+DFW
    8480 SJC https://google.com/search?q=iata+code:+SJC
    8263 NRT https://google.com/search?q=iata+code:+NRT
    7571 SYD https://google.com/search?q=iata+code:+SYD
    6326 LHR https://google.com/search?q=iata+code:+LHR
    6239 YVR https://google.com/search?q=iata+code:+YVR
    6148 EWR https://google.com/search?q=iata+code:+EWR
    5146 DME https://google.com/search?q=iata+code:+DME
    4955 HKG https://google.com/search?q=iata+code:+HKG
    3729 ARN https://google.com/search?q=iata+code:+ARN
    2877  https://google.com/search?q=iata+code:+
    2516 CDG https://google.com/search?q=iata+code:+CDG
    2240 LAX https://google.com/search?q=iata+code:+LAX
    2027 HEL https://google.com/search?q=iata+code:+HEL
    2012 BOM https://google.com/search?q=iata+code:+BOM
    

    What about stats for how Cloudflare is communicating SSL/TLS handshake wise to origin backend Centmin Mod Nginx server in terms of SSL Protocol and SSL ciphers via custom cfssl-access.log with filtered out SERVER_IP. Looks like majority of connections are via TLSv1.2 with either ECDHE-RSA-AES128-GCM-SHA256 or ECDHE-RSA-AES256-GCM-SHA384
    Code (Text):
    pzcat -f cfssl-access.log{,-*} | grep -v 'SERVER_IP' | awk '{n = 2; for (--n; n >= 0; n--){ printf "%s\t",$(NF-n)} print ""}' | sort | uniq -c | sort -rn | head -n20 | column -t
    108793  TLSv1.2  ECDHE-RSA-AES128-GCM-SHA256
    40651   TLSv1.2  ECDHE-RSA-AES256-GCM-SHA384
    22      TLSv1.2  ECDHE-RSA-CHACHA20-POLY1305
    4       -        -
    1       TLSv1    ECDHE-RSA-AES128-SHA
    1       TLSv1.1  ECDHE-RSA-AES128-SHA
    


    Optimising Cloudflare Origin Backend Communication Performance



    From the custom Cloudflare logging done on Centmin Mod Nginx backend origin server, I can see that my Centmin Mod Nginx origin server is using RSA 2048bit SSL certificate hence RSA based SSL/TLS handshake communication with Cloudflare on backend. Hence, I know I can improve the performance by switching my Centmin Mod Nginx to a new SSL certificate using ECC 256bit ECDSA if I wanted more speed.

    I can do this by either buying a ECC 256bit ECDSA SSL certificate, using Letsencrypt and change from RSA 2048bit default SSL cert to ECDSA 256bit SSL cert or use Cloudflare's freely provided Cloudflare Origin SSL certificates which have a validity of 15yrs and install the server on Centmin Mod Nginx backend server.

    For Cloudflare Origin SSL certificates install on Nginx How to install an Origin CA certificate in NGINX you're just changing the path of existing ssl_certificate and ssl_certificate_key paths.

    cloudflare-origin-certs.png
    For RSA 2048bit Cloudflare Origin SSL certificate

    cloudflare-origin-certs2.png

    For ECDSA 256bit Cloudflare Origin SSL certificate

    cloudflare-origin-certs3.png

    Note Cloudflare Origin SSL certificates only trusted by Cloudflare so untrusted when used on general web so if you have clients, tools etc that communicate directly with backend Nginx origin server, you may run into issues with Cloudflare Origin SSL certs so need proper trusted SSL certificates like paid or Letsencrypt SSL certificates instead.

    ECDSA Performance Boost



    If you want even more performance, selecting ECDSA 256bit SSL certificate usage for Centmin Mod Nginx backend origin to communicate with Cloudflare isn't enough as ECDSA performance depends on the Nginx crypto library it's built with - OpenSSL 1.0.2 or 1.1.0 or 1.1.1 version based has vary performance improvements for ECDSA.
    • OpenSSL 1.1.0 has ~2x times ECDSA performance as OpenSSL 1.0.2 LibreSSL 2.6/2.7 ECDSA.
    • OpenSSL 1.1.1 in beta has ~30-40% more ECDSA performance than OpenSSL 1.1.0. So up to 2.8x times faster ECDSA peformance than OpenSSL 1.0.2 LibreSSL 2.6/2.7 ECDSA.
    Centmin Mod 123.09beta01 and higher's Nginx server supports various crypto libraries and can support OpenSSL 1.0.2, OpenSSL 1.1.0 or OpenSSL 1.1.1 as well as LibreSSL 2.7+ as outlined here. Currently, Centmin Mod 123.09beta01 Nginx defaults to compiling Nginx against OpenSSL 1.1.0 branch latest, 1.1.0h. OpenSSL 1.1.1 is nearly final release once TLS 1.3 is finalized.
    For specific details on how to use Cloudflare Origin SSL certificates check out full guide at here.
     
  3. eva2000

    eva2000 Administrator Staff Member

    36,023
    7,901
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +12,183
    Local Time:
    6:03 PM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    Been a while so thought check logged CF access

    for top 20 IATA airport code based CF Datacenter connections to my origin
    Code (Text):
    pzcat -f cf-access.log{,-*} | awk '{print $NF}' | awk -F '-' '{print $2}' |  sed -e 's|\"||g' | sort | uniq -c | sort -rn | awk '{print $1, $2, "https://google.com/search?q=iata+code:+"$2}' | head -n20
    166880 SEA https://google.com/search?q=iata+code:+SEA
    108610 FRA https://google.com/search?q=iata+code:+FRA
    72431 DFW https://google.com/search?q=iata+code:+DFW
    47621 IAD https://google.com/search?q=iata+code:+IAD
    47618 AMS https://google.com/search?q=iata+code:+AMS
    38194 SIN https://google.com/search?q=iata+code:+SIN
    30617 LHR https://google.com/search?q=iata+code:+LHR
    27575 ORD https://google.com/search?q=iata+code:+ORD
    24423 SYD https://google.com/search?q=iata+code:+SYD
    23454 EWR https://google.com/search?q=iata+code:+EWR
    23165 SJC https://google.com/search?q=iata+code:+SJC
    18757 NRT https://google.com/search?q=iata+code:+NRT
    17968 DME https://google.com/search?q=iata+code:+DME
    15119 LAX https://google.com/search?q=iata+code:+LAX
    10865 CDG https://google.com/search?q=iata+code:+CDG
    9130 HKG https://google.com/search?q=iata+code:+HKG
    8139 KBP https://google.com/search?q=iata+code:+KBP
    7935 ARN https://google.com/search?q=iata+code:+ARN
    7225 MCI https://google.com/search?q=iata+code:+MCI
    4088 HEL https://google.com/search?q=iata+code:+HEL
    

    Top TLS protocol/ciphers communicated over for CF to origin connections
    Code (Text):
    pzcat -f cfssl-access.log{,-*} | grep -v 'SERVER_IP' | awk '{n = 2; for (--n; n >= 0; n--){ printf "%s\t",$(NF-n)} print ""}' | sort | uniq -c | sort -rn | head -n20 | column -t
    753574  TLSv1.2  ECDHE-ECDSA-AES128-GCM-SHA256
    107     TLSv1.2  ECDHE-ECDSA-AES256-GCM-SHA384
    81      -        -
    19      TLSv1.2  ECDHE-ECDSA-CHACHA20-POLY1305
    7       TLSv1    ECDHE-ECDSA-AES128-SHA
    
     
..
Thread Status:
Not open for further replies.