Welcome to Centmin Mod Community
Register Now

Featured Cloudflare Cloudflare custom Nginx logging

Discussion in 'Domains, DNS, Email & SSL Certificates' started by eva2000, May 14, 2018.

Thread Status:
Not open for further replies.
  1. eva2000

    eva2000 Administrator Staff Member

    54,107
    12,179
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,738
    Local Time:
    7:58 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    If you're using Cloudflare in front of your Centmin Mod Nginx web server, then you may want to add custom Nginx access logging for Cloudflare related metrics such as CF-RAY header as well as SSL protocol and ssl ciphers served (previous example).

    I added additional logging formats for cf_custom, cf_custom2 and cf_custom3 into /usr/local/nginx/conf/nginx.conf

    Updated with $http_cf_request_id logging
    Code (Text):
    log_format cf_custom '$remote_addr - $remote_user [$time_local]  '
                       '"$request" $status $body_bytes_sent '
                       '"$http_referer" "$http_user_agent" '
                       '$http_cf_ray $http_cf_request_id';
                 
    log_format cf_custom2 '$remote_addr - $remote_user [$time_local] $request '
                   '"$status" $body_bytes_sent "$http_referer" '
                   '"$http_user_agent" "$http_x_forwarded_for" "$gzip_ratio"'
                   ' "$connection" "$connection_requests" "$request_time" $http_cf_ray $http_cf_request_id';
             
    
    log_format cf_custom3 '$remote_addr - $remote_user [$time_local] $request '
                   '"$status" $body_bytes_sent "$http_referer" '
                   '"$http_user_agent" "$http_x_forwarded_for" "$gzip_ratio"'
                   ' "$connection" "$connection_requests" "$request_time" $http_cf_ray $http_cf_request_id '
                   '$ssl_protocol $ssl_cipher';
    

    Then in my Nginx vhost config logs add 2 additional access log using defined custom log formats i.e. cf_custom2 and cf_custom3.

    So it changes from
    Code (Text):
      access_log /home/nginx/domains/domain.com/log/access.log combined buffer=256k flush=5m;
      error_log /home/nginx/domains/domain.com/log/error.log;
    

    to the below where cf-access.log only contains CF-RAY header and cfssl-access.log has CF-RAY with additional ssl protocol and ssl cipher logging.
    Code (Text):
      access_log /home/nginx/domains/domain.com/log/access.log combined buffer=256k flush=5m;
      access_log /home/nginx/domains/domain.com/log/cf-access.log cf_custom2 buffer=256k flush=5m;
      access_log /home/nginx/domains/domain.com/log/cfssl-access.log cf_custom3 buffer=256k flush=5m;
      error_log /home/nginx/domains/domain.com/log/error.log;
    

    Then you can use grep, fgrep, awk and other SSH tools to manipulate and inspect those new logs.


    Example of the field numbers for cf-access.log entry which logged Cloudflare request id and CF-RAY header = 41a8a20c1c953343-HKG meaning the request was served from Cloudflare Hong Kong datacenter :)
    Code (Text):
    tail -1 /home/nginx/domains/domain.com/log/cf-access.log | awk '{print NR": "$0; for(i=1;i<=NF;++i)print ""i":  "$i}'
    1: 180.76.15.158 - - [13/May/2018:22:41:32 +0000] GET /tags/php-54/ HTTP/1.1 "200" 45399 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)" "180.76.15.158" "-" "39" "1" "0.100" 41a8a20c1c953343-HKG
    1:  180.76.15.158
    2:  -
    3:  -
    4:  [13/May/2018:22:41:32
    5:  +0000]
    6:  GET
    7:  /tags/php-54/
    8:  HTTP/1.1
    9:  "200"
    10:  45399
    11:  "-"
    12:  "Mozilla/5.0
    13:  (compatible;
    14:  Baiduspider/2.0;
    15:  +http://www.baidu.com/search/spider.html)"
    16:  "180.76.15.158"
    17:  "-"
    18:  "39"
    19:  "1"
    20:  "0.100"
    21:  41a8a20c1c953343-HKG
    

    Example of field numbers for cfssl-access.log entry which shows CF-RAY header = 41a8a20c1c953343-HKG as well as SSL protocol used = TLS v1.2 and SSL cipher = ECDHE-RSA-AES256-GCM-SHA384. These 2 values are what Centmin Mod Nginx served to upstream Cloudflare (client) and not what Cloudflare served to the visitor. When Cloudflare is in front of Centmin Mod Nginx, then the SSL handshake and connection established is between Cloudflare and Centmin Mod Nginx. In my case it was between the middle Sucuri Cloudproxy which sandwiched between Cloudflare and Centmin Mod Nginx.
    Code (Text):
    tail -1 /home/nginx/domains/domain.com/log/cfssl-access.log | awk '{print NR": "$0; for(i=1;i<=NF;++i)print ""i":  "$i}'
    1: 180.76.15.158 - - [13/May/2018:22:41:32 +0000] GET /tags/php-54/ HTTP/1.1 "200" 45399 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)" "180.76.15.158" "-" "39" "1" "0.100" 41a8a20c1c953343-HKG TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384
    1:  180.76.15.158
    2:  -
    3:  -
    4:  [13/May/2018:22:41:32
    5:  +0000]
    6:  GET
    7:  /tags/php-54/
    8:  HTTP/1.1
    9:  "200"
    10:  45399
    11:  "-"
    12:  "Mozilla/5.0
    13:  (compatible;
    14:  Baiduspider/2.0;
    15:  +http://www.baidu.com/search/spider.html)"
    16:  "180.76.15.158"
    17:  "-"
    18:  "39"
    19:  "1"
    20:  "0.100"
    21:  41a8a20c1c953343-HKG
    22:  TLSv1.2
    23:  ECDHE-RSA-AES256-GCM-SHA384
    

    Inspecting cf-access.log to find the top Cloudflare datacenter regions in requests by inspecting CF-RAY headers logged

    For single /home/nginx/domains/domain.com/log/cf-access.log log
    Code (Text):
    awk '{print $NF}' /home/nginx/domains/domain.com/log/cf-access.log | awk -F '-' '{print $2}' | sort | uniq -c | sort -rn
       492
        71 SJC
        69 NRT
        66 DME
        64 ORD
        52 AMS
        50 SIN
        44 IAD
        40 LAX
        32 EWR
        31 LHR
        30 FRA
        23 HKG
        21 YVR
        17 BOM
        12 ATL
        11 DFW
         6 WAW
         6 MEL
         6 DUB
         5 HEL
         5 CDG
         4 SOF
         4 PDX
         4 ARN
         3 STL
         3 BCN
         3 ATH
         2 SEA
         2 EZE
         2 DEN
         1 VIE
         1 MIA
         1 LUX
         1 LAS
         1 KBP
    

    Nginx logs will get rotated and compressed so you may want to inspect all logs current uncompressed + log rotated and compressed together so you can use zcat/pzcat command instead.
    Code (Text):
    pzcat -f /home/nginx/domains/domain.com/log/cf-access.log{,-*} | awk '{print $NF}' | awk -F '-' '{print $2}' |  sed -e 's|\"||g' | sort | uniq -c | sort -rn
    
        281 DME
        230 NRT
        220 YVR
        135 SIN
        131 DFW
        131
        107 IAD
        101 AMS
         91 SJC
         76 LHR
         74 FRA
         67 ORD
         60 EWR
         49 PDX
         36 HKG
         30 BOM
         26 KIX
         24 LAX
         24 BCN
         19 MEL
         19 GRU
         12 CDG
          8 BOS
          7 AKL
          6 WAW
          6 OTP
          6 ATL
          5 PRG
          5 ATH
          4 SEA
          4 DUB
          4 ARN
          3 MIA
          2 KBP
          2 HEL
          2 EZE
          2 DEN
          2 BKK
          1 SOF
          1 LUX
          1 LAS
    

    You'll find the Cloudflare 3 letter datacenter region codes as outlined here.
    If you need to know the IATA airport codes use Google :)
    Code (Text):
    pzcat -f /home/nginx/domains/domain.com/log/cf-access.log{,-*} | awk '{print $NF}' | awk -F '-' '{print $2}' |  sed -e 's|\"||g' | sort | uniq -c | sort -rn | awk '{print $1, $2, "https://google.com/search?q=iata+code:+"$2}'  
    
    398 YVR https://google.com/search?q=iata+code:+YVR
    390 DME https://google.com/search?q=iata+code:+DME
    389 NRT https://google.com/search?q=iata+code:+NRT
    209 SIN https://google.com/search?q=iata+code:+SIN
    171 DFW https://google.com/search?q=iata+code:+DFW
    152  https://google.com/search?q=iata+code:+
    146 AMS https://google.com/search?q=iata+code:+AMS
    134 IAD https://google.com/search?q=iata+code:+IAD
    130 SJC https://google.com/search?q=iata+code:+SJC
    116 FRA https://google.com/search?q=iata+code:+FRA
    114 LHR https://google.com/search?q=iata+code:+LHR
    99 ORD https://google.com/search?q=iata+code:+ORD
    99 EWR https://google.com/search?q=iata+code:+EWR
    72 PDX https://google.com/search?q=iata+code:+PDX
    53 GRU https://google.com/search?q=iata+code:+GRU
    41 HKG https://google.com/search?q=iata+code:+HKG
    41 BOM https://google.com/search?q=iata+code:+BOM
    29 LAX https://google.com/search?q=iata+code:+LAX
    28 MEL https://google.com/search?q=iata+code:+MEL
    26 KIX https://google.com/search?q=iata+code:+KIX
    26 BCN https://google.com/search?q=iata+code:+BCN
    24 CDG https://google.com/search?q=iata+code:+CDG
    13 ARN https://google.com/search?q=iata+code:+ARN
    13 AKL https://google.com/search?q=iata+code:+AKL
    8 WAW https://google.com/search?q=iata+code:+WAW
    8 MIA https://google.com/search?q=iata+code:+MIA
    8 BOS https://google.com/search?q=iata+code:+BOS
    6 SEA https://google.com/search?q=iata+code:+SEA
    6 OTP https://google.com/search?q=iata+code:+OTP
    6 DUB https://google.com/search?q=iata+code:+DUB
    6 DEN https://google.com/search?q=iata+code:+DEN
    6 ATL https://google.com/search?q=iata+code:+ATL
    6 ATH https://google.com/search?q=iata+code:+ATH
    5 PRG https://google.com/search?q=iata+code:+PRG
    4 EZE https://google.com/search?q=iata+code:+EZE
    2 SOF https://google.com/search?q=iata+code:+SOF
    2 LUX https://google.com/search?q=iata+code:+LUX
    2 LAS https://google.com/search?q=iata+code:+LAS
    2 KBP https://google.com/search?q=iata+code:+KBP
    2 HEL https://google.com/search?q=iata+code:+HEL
    2 BKK https://google.com/search?q=iata+code:+BKK
    1 CPH https://google.com/search?q=iata+code:+CPH
    
     
    Last edited: Aug 26, 2020
  2. eva2000

    eva2000 Administrator Staff Member

    54,107
    12,179
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,738
    Local Time:
    7:58 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    Quick look at cfssl-access.log after 24+ hrs of logging on just one server behind Cloudflare.

    In log directory cfssl-access.log and cf-access.log logs both current, past day and gzip rotated prior day before that.
    Code (Text):
    ls -lhrt | egrep 'cfssl-|cf-'
    -rw-r--r-- 1 nginx nginx  16M May 16 04:16 cfssl-access.log-20180516
    -rw-r--r-- 1 nginx nginx  15M May 16 04:16 cf-access.log-20180516
    -rw-r--r-- 1 nginx nginx 2.6M May 16 04:16 cf-access.log-20180515.gz
    -rw-r--r-- 1 nginx nginx 2.7M May 16 04:16 cfssl-access.log-20180515.gz
    -rw-r--r-- 1 nginx nginx  16M May 17 00:15 cfssl-access.log
    -rw-r--r-- 1 nginx nginx  14M May 17 00:15 cf-access.log
    

    Look at Cloudflare datacenter regions hit according to IATA airport codes for top 10 datacenters via custom cf-access.log
    Code (Text):
    pzcat -f cf-access.log{,-*} | awk '{print $NF}' | awk -F '-' '{print $2}' |  sed -e 's|\"||g' | sort | uniq -c | sort -rn | awk '{print $1, $2, "https://google.com/search?q=iata+code:+"$2}' | head -n20
    12653 IAD https://google.com/search?q=iata+code:+IAD
    12290 FRA https://google.com/search?q=iata+code:+FRA
    11313 SEA https://google.com/search?q=iata+code:+SEA
    9698 AMS https://google.com/search?q=iata+code:+AMS
    9602 SIN https://google.com/search?q=iata+code:+SIN
    9296 DFW https://google.com/search?q=iata+code:+DFW
    8480 SJC https://google.com/search?q=iata+code:+SJC
    8263 NRT https://google.com/search?q=iata+code:+NRT
    7571 SYD https://google.com/search?q=iata+code:+SYD
    6326 LHR https://google.com/search?q=iata+code:+LHR
    6239 YVR https://google.com/search?q=iata+code:+YVR
    6148 EWR https://google.com/search?q=iata+code:+EWR
    5146 DME https://google.com/search?q=iata+code:+DME
    4955 HKG https://google.com/search?q=iata+code:+HKG
    3729 ARN https://google.com/search?q=iata+code:+ARN
    2877  https://google.com/search?q=iata+code:+
    2516 CDG https://google.com/search?q=iata+code:+CDG
    2240 LAX https://google.com/search?q=iata+code:+LAX
    2027 HEL https://google.com/search?q=iata+code:+HEL
    2012 BOM https://google.com/search?q=iata+code:+BOM
    

    What about stats for how Cloudflare is communicating SSL/TLS handshake wise to origin backend Centmin Mod Nginx server in terms of SSL Protocol and SSL ciphers via custom cfssl-access.log with filtered out SERVER_IP. Looks like majority of connections are via TLSv1.2 with either ECDHE-RSA-AES128-GCM-SHA256 or ECDHE-RSA-AES256-GCM-SHA384
    Code (Text):
    pzcat -f cfssl-access.log{,-*} | grep -v 'SERVER_IP' | awk '{n = 2; for (--n; n >= 0; n--){ printf "%s\t",$(NF-n)} print ""}' | sort | uniq -c | sort -rn | head -n20 | column -t
    108793  TLSv1.2  ECDHE-RSA-AES128-GCM-SHA256
    40651   TLSv1.2  ECDHE-RSA-AES256-GCM-SHA384
    22      TLSv1.2  ECDHE-RSA-CHACHA20-POLY1305
    4       -        -
    1       TLSv1    ECDHE-RSA-AES128-SHA
    1       TLSv1.1  ECDHE-RSA-AES128-SHA
    


    Optimising Cloudflare Origin Backend Communication Performance



    From the custom Cloudflare logging done on Centmin Mod Nginx backend origin server, I can see that my Centmin Mod Nginx origin server is using RSA 2048bit SSL certificate hence RSA based SSL/TLS handshake communication with Cloudflare on backend. Hence, I know I can improve the performance by switching my Centmin Mod Nginx to a new SSL certificate using ECC 256bit ECDSA if I wanted more speed.

    I can do this by either buying a ECC 256bit ECDSA SSL certificate, using Letsencrypt and change from RSA 2048bit default SSL cert to ECDSA 256bit SSL cert or use Cloudflare's freely provided Cloudflare Origin SSL certificates which have a validity of 15yrs and install the server on Centmin Mod Nginx backend server.

    For Cloudflare Origin SSL certificates install on Nginx How to install an Origin CA certificate in NGINX you're just changing the path of existing ssl_certificate and ssl_certificate_key paths.

    cloudflare-origin-certs.png
    For RSA 2048bit Cloudflare Origin SSL certificate

    cloudflare-origin-certs2.png

    For ECDSA 256bit Cloudflare Origin SSL certificate

    cloudflare-origin-certs3.png

    Note Cloudflare Origin SSL certificates only trusted by Cloudflare so untrusted when used on general web so if you have clients, tools etc that communicate directly with backend Nginx origin server, you may run into issues with Cloudflare Origin SSL certs so need proper trusted SSL certificates like paid or Letsencrypt SSL certificates instead.

    ECDSA Performance Boost



    If you want even more performance, selecting ECDSA 256bit SSL certificate usage for Centmin Mod Nginx backend origin to communicate with Cloudflare isn't enough as ECDSA performance depends on the Nginx crypto library it's built with - OpenSSL 1.0.2 or 1.1.0 or 1.1.1 version based has vary performance improvements for ECDSA.
    • OpenSSL 1.1.0 has ~2x times ECDSA performance as OpenSSL 1.0.2 LibreSSL 2.6/2.7 ECDSA.
    • OpenSSL 1.1.1 in beta has ~30-40% more ECDSA performance than OpenSSL 1.1.0. So up to 2.8x times faster ECDSA peformance than OpenSSL 1.0.2 LibreSSL 2.6/2.7 ECDSA.
    Centmin Mod 123.09beta01 and higher's Nginx server supports various crypto libraries and can support OpenSSL 1.0.2, OpenSSL 1.1.0 or OpenSSL 1.1.1 as well as LibreSSL 2.7+ as outlined here. Currently, Centmin Mod 123.09beta01 Nginx defaults to compiling Nginx against OpenSSL 1.1.0 branch latest, 1.1.0h. OpenSSL 1.1.1 is nearly final release once TLS 1.3 is finalized.
    For specific details on how to use Cloudflare Origin SSL certificates check out full guide at here.
     
  3. eva2000

    eva2000 Administrator Staff Member

    54,107
    12,179
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,738
    Local Time:
    7:58 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    Been a while so thought check logged CF access

    for top 20 IATA airport code based CF Datacenter connections to my origin
    Code (Text):
    pzcat -f cf-access.log{,-*} | awk '{print $NF}' | awk -F '-' '{print $2}' |  sed -e 's|\"||g' | sort | uniq -c | sort -rn | awk '{print $1, $2, "https://google.com/search?q=iata+code:+"$2}' | head -n20
    166880 SEA https://google.com/search?q=iata+code:+SEA
    108610 FRA https://google.com/search?q=iata+code:+FRA
    72431 DFW https://google.com/search?q=iata+code:+DFW
    47621 IAD https://google.com/search?q=iata+code:+IAD
    47618 AMS https://google.com/search?q=iata+code:+AMS
    38194 SIN https://google.com/search?q=iata+code:+SIN
    30617 LHR https://google.com/search?q=iata+code:+LHR
    27575 ORD https://google.com/search?q=iata+code:+ORD
    24423 SYD https://google.com/search?q=iata+code:+SYD
    23454 EWR https://google.com/search?q=iata+code:+EWR
    23165 SJC https://google.com/search?q=iata+code:+SJC
    18757 NRT https://google.com/search?q=iata+code:+NRT
    17968 DME https://google.com/search?q=iata+code:+DME
    15119 LAX https://google.com/search?q=iata+code:+LAX
    10865 CDG https://google.com/search?q=iata+code:+CDG
    9130 HKG https://google.com/search?q=iata+code:+HKG
    8139 KBP https://google.com/search?q=iata+code:+KBP
    7935 ARN https://google.com/search?q=iata+code:+ARN
    7225 MCI https://google.com/search?q=iata+code:+MCI
    4088 HEL https://google.com/search?q=iata+code:+HEL
    

    Top TLS protocol/ciphers communicated over for CF to origin connections
    Code (Text):
    pzcat -f cfssl-access.log{,-*} | grep -v 'SERVER_IP' | awk '{n = 2; for (--n; n >= 0; n--){ printf "%s\t",$(NF-n)} print ""}' | sort | uniq -c | sort -rn | head -n20 | column -t
    753574  TLSv1.2  ECDHE-ECDSA-AES128-GCM-SHA256
    107     TLSv1.2  ECDHE-ECDSA-AES256-GCM-SHA384
    81      -        -
    19      TLSv1.2  ECDHE-ECDSA-CHACHA20-POLY1305
    7       TLSv1    ECDHE-ECDSA-AES128-SHA
    
     
  4. eva2000

    eva2000 Administrator Staff Member

    54,107
    12,179
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,738
    Local Time:
    7:58 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+

    Cloudflare Enables HTTPS TLS 1.3 Backend Origin Communication



    Cloudflare just announced they have officially enabled HTTPS TLS .13 backend origin communication with origin web servers which have HTTP/2 HTTPS TLS 1.3 enabled i.e. Cloudflare Strict SSL mode. Prior to this announcement, Cloudflare edge servers would communicate with your site's origin web server (Centmin Mod Nginx) using TLS 1.2 even if your origin web server supported TLS 1.3. Now with this change, you can speed up your page load speeds slightly as TLS 1.3 saves one round trip time (RTT) on the connection :)

    Centmin Mod 123.09beta01 and newer Nginx builds support HTTP/2 HTTPS TLS 1.3 out of the box now via either OpenSSL 1.1.1 branch (default) or optionally via BoringSSL. Details at Centmin Mod Nginx HTTP/2 HTTPS TLS 1.3 Support. So when you create a new Nginx HTTP/2 HTTPS site with Centmin Mod Nginx, it automatically now has TLS 1.3 enabled :)

    I can verify that Cloudflare is communicating with my Centmin Mod Nginx origin servers over TLS 1.3 protocol now using custom Cloudflare nginx logging I setup on my servers behind Cloudflare.

    For my cfssl-access.log log I can see TLS v1.3 being used for connection now with TLS_AES_256_GCM_SHA384 ssl cipher.
    Code (Text):
    tail -1 cfssl-access.log
    
    54.36.148.175 - - [05/Mar/2019:21:32:04 +0000] GET /threads/gcc-7-3-1-update.14858/ HTTP/1.1 "200" 97042 "-" "Mozilla/5.0 (compatible; AhrefsBot/6.1; +http://ahrefs.com/robot/)" "54.36.148.175" "-" "421824" "1" "0.400" 4b2f334b8bfb9cb3-AMS TLSv1.3 TLS_AES_256_GCM_SHA384
    

    TLS protocol and cipher stats - TLSv1.3 is starting to show up using ssl cipher, TLS_AES_256_GCM_SHA384 :D
    Code (Text):
    pzcat -f cfssl-access.log{,-*} | grep -v 'SERVER_IP' | awk '{n = 2; for (--n; n >= 0; n--){ printf "%s\t",$(NF-n)} print ""}' | sort | uniq -c | sort -rn | head -n20 | column -t
    299930  TLSv1.2  ECDHE-ECDSA-AES256-GCM-SHA384
    191033  TLSv1.2  ECDHE-ECDSA-CHACHA20-POLY1305
    173484  TLSv1.3  TLS_AES_256_GCM_SHA384
    43      -        -
    35      TLSv1.2  ECDHE-ECDSA-AES128-GCM-SHA256
    
     
  5. eva2000

    eva2000 Administrator Staff Member

    54,107
    12,179
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,738
    Local Time:
    7:58 AM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
Thread Status:
Not open for further replies.