Learn about Centmin Mod LEMP Stack today
Become a Member

Centmin Mod Maxmind GeoLite2 Free Database Download Changes

Discussion in 'Centmin Mod News' started by eva2000, Jan 3, 2020.

Thread Status:
Not open for further replies.
  1. eva2000

    eva2000 Administrator Staff Member

    44,425
    10,144
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +15,693
    Local Time:
    2:44 AM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    Maxmind announced significant changes to their GeoLite2 free database offerings and from December 30 2019 onwards, they will no longer be providing free public access to GeoLite2 free database downloads. Due to California Consumer Privacy Act (CCPA) laws, they will also now require folks to register for a free Maxmind account to be able to download and use their GeoLite2 free databases via a generated Maxmind token API key at https://www.maxmind.com/en/accounts/current/license-key. This will break alot of services/web apps which relied on downloading the GeoLite2 databases from a publicly known url link until those services/web apps update to use a Maxmind token API key.

    Note, Maxmind account registration signup doesn't allow VPN users to sign up at https://www.maxmind.com/en/geolite2/signup. You'd need to disable your VPN for just the registration phase. Once registered, you can re-enable your VPN to login to your Maxmind account.

    Centmin Mod Maxmind GeoLite2 Free Database Usage Updates



    Centmin Mod 123.09beta01 and newer branches make use of Maxmind's GeoLite2 free databases in two pieces of software:
    1. CSF Firewall's own GeoLite2 free database integration. CSF Firewall folks updated and released v13.09 to support Maxmind's new token API key method of downloading GeoLite2 databases and they use a new MM_LICENSE_KEY variable in /etc/csf/csf.conf for this. If you do not set a MM_LICENSE_KEY in CSF Firewall config file, your Login Failure Daemon (lfd) log at /var/log/lfd.log will show the following error message:
      Code (Text):
      Jan  2 03:15:05 host lfd[1488]: CC Error: Country Code Lookups setting MM_LICENSE_KEY must be set in /etc/csf/csf.conf to continue using the MaxMind databases
      .
    2. Centmin Mod Nginx's optional ngx_http_geoip2_module which is disabled by default. However, ngx_http2_geoip2_module can be optionally enabled by end users as outlined in the How to enable GeoIP 2 Lite Nginx Module Support.
    On January 2 2020, I updated Centmin Mod 123.09beta01 branch with changes to support Maxmind's new GeoLite2 account and token API method of downloading for both CSF Firewall and Nginx's ngx_http_geoip2_module.

    To update Centmin Mod 123.09beta01, you can just run in SSH, the cmupdate command and then run centmin.sh menu once and exit.
    Code (Text):
    cmupdate


    Once Centmin Mod 123.09beta01 is updated:
    • When you set MM_LICENSE_KEY in persistent config file at /etc/centminmod/custom_config.inc with your manually obtained and generated Maxmind account's token API key via https://www.maxmind.com/en/geolite2/signup and generate the token API in Services > My License Key section of your Maxmind account or via link at https://www.maxmind.com/en/accounts/current/license-key and then run centmin.sh menu once and then exit, then both CSF Firewall and Nginx geoip2 nginx module routines will download GeoLite2 database via Maxmind API for latest version.
      Code (Text):
      MM_LICENSE_KEY='your_maxmind_api_key'
    • If you do not set MM_LICENSE_KEY in persistent config file at /etc/centminmod/custom_config.inc, then Centmin Mod will use a new shared Maxmind token API key I created just for Centmin Mod usage.
    • When you have ngx_http_geoip2_module enabled, Centmin Mod also creates a GeoIP 2 database update cronjob
      Code (Text):
      20 2 * * 4 /usr/local/src/centminmod/tools/geoip2db-update.sh && nprestart >/dev/null 2>&1
      . You can wait for this cronjob to run to update your Centmin Mod system GeoLIte2 databases or manually run the cronjob once
      Code (Text):
      /usr/local/src/centminmod/tools/geoip2db-update.sh
    Relevant outlined commits for these changes.

    Verifying Maxmind API Downloaded GeoLite2 Database



    Maxmind GeoLite2 database last public release dated version is December 23-25, 2019 and the first GeoLite2 database to be released under the new registered Maxmind account token API downloaded version is dated December 31, 2019.

    CSF Firewall stores it's GeoLite2 database files in /var/lib/csf/Geo/ and based on it's CC_INTERVAL default, it will check and update database every 14 days.

    CSF Firewall last public GeoLite2 database release version dated December 23, 2019. On existing Centmin Mod 123.09beta01 installs, you will most likely see this dated version right now unless you did a fresh Centmin Mod 123.09beta01 or newer install - in which case you should see December 31, 2019 dated files.
    Code (Text):
    ls -lAh /var/lib/csf/Geo/
    total 16M
    -rw------- 1 root root   55 Dec 23 19:00 COPYRIGHT.txt
    -rw------- 1 root root  13M Dec 23 19:00 GeoLite2-Country-Blocks-IPv4.csv
    -rw------- 1 root root 3.8M Dec 23 19:00 GeoLite2-Country-Blocks-IPv6.csv
    -rw------- 1 root root 9.7K Dec 23 19:00 GeoLite2-Country-Locations-en.csv
    -rw------- 1 root root  433 Dec 23 19:00 LICENSE.txt
    -rw------- 1 root root  116 Dec 23 19:00 README.txt
    

    For Centmin Mod Nginx ngx_http_geoip2_module enabled servers, the GeoLite2 free database is stored at /usr/share/GeoIP (which also stores GeoIP legacy database as well).
    Code (Text):
    ls -lah /usr/share/GeoIP | grep GeoLite2            
    -rw-r--r--    1 root root 6.4M Dec 31 13:49 GeoLite2-ASN.mmdb
    -rw-r--r--    1 root root 3.6M Dec 31 13:49 GeoLite2-ASN.tar.gz
    -rw-r--r--    1 root root  59M Dec 31 16:33 GeoLite2-City.mmdb
    -rw-r--r--    1 root root  28M Dec 31 16:33 GeoLite2-City.tar.gz
    -rw-r--r--    1 root root 3.9M Dec 31 16:32 GeoLite2-Country.mmdb
    -rw-r--r--    1 root root 2.0M Dec 31 16:32 GeoLite2-Country.tar.gz
    


    GeoLite2 News Tracking Via Slack



    With GeoLite2 changes having a dramatic effect on Centmin Mod's products, I have setup a custom Slack channel to keep track of GeoIP/GeoLite2 news and announcements. A common way of tracking RSS feeds and news I outlined at How To Keep Informed Of Centmin Mod Related Updates. Slack documentation for adding RSS feeds here.

    RSS feeds for their blog and release notes pages
    geoip2news-slack-01.png
     
    Last edited: Jan 4, 2020
  2. eva2000

    eva2000 Administrator Staff Member

    44,425
    10,144
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +15,693
    Local Time:
    2:44 AM
    Nginx 1.17.x
    MariaDB 5.5/10.x
  3. eva2000

    eva2000 Administrator Staff Member

    44,425
    10,144
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +15,693
    Local Time:
    2:44 AM
    Nginx 1.17.x
    MariaDB 5.5/10.x

    CSF Firewall v14.00 Country Code Changes



    Looks like CSF Firewall v14.00 was released to switch away from using Maxmind's GeoLite2 database for Country code lookups for new CSF Firewall installs while older CSF Firewall installs continue to use Maxmind's GeoLite2 databases.
    from /etc/csf/csf.conf for existing CSF Firewall installs you can see CC_SRC = "1" set to use Maxmind GeoLite2 database instead of CC_SRC = "2" for new CSF Firewall installs for using different source databases via db-ip etc.
    Code (Text):
    # Set the following to your preferred source:
    #
    # "1" - MaxMind
    # "2" - db-ip, ipverse, iptoasn
    #
    # The default is "2" on new installations of csf, or set to "1" to use the
    # MaxMind databases after obtaining a license key
    CC_SRC = "1"
    

    So for Centmin Mod 123.09beta01 fresh installs after January 9th, 2020 you should be using CC_SRC = "2" for db-ip Country database and not require signing up for Maxmind account and not requiring to set up MM_LICENSE_KEY in /etc/csf/csf.conf.

    However, if you use and enable Centmin Mod Nginx's optional ngx_http_geoip2_module which is disabled by default as outlined in the How to enable GeoIP 2 Lite Nginx Module Support, then you still will need to set MM_LICENSE_KEY in persistent config file at /etc/centminmod/custom_config.inc with your manually obtained and generated Maxmind account's token API key via https://www.maxmind.com/en/geolite2/signup and generate the token API in Services > My License Key section of your Maxmind account or via link at https://www.maxmind.com/en/accounts/current/license-key and then run centmin.sh menu once and then exit.
     
  4. eva2000

    eva2000 Administrator Staff Member

    44,425
    10,144
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +15,693
    Local Time:
    2:44 AM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    From today (Jan 20, 2020) onwards, I will be setting CSF Firewall's CC_SRC back to using maxmind database instead of previous post mentioned db-ip source via CC_SRC="1" in /etc/csf/csf.conf. Folks have been reporting IP geolocation data for Country codes for db-ip sourced database have not been as accurate as with maxmind database.

    For existing Centmin Mod 123.09beta01 users, you can switch from CC_SRC="2" back to CC_SRC="1" via these commands which backup your /etc/csf/csf.conf, sed replacement and then CSF Firewall restart and then grep filter check that the config file has indeed changed CC_SRC="1"
    Code (Text):
    csf --profile backup before_cc_src_1_switch
    sed -i 's|CC_SRC = "2"|CC_SRC = "1"|' /etc/csf/csf.conf
    csf -ra
    grep 'CC_SRC' /etc/csf/csf.conf
    

    You can compare and test an IP's geolocation data using my own GeoIP lookup site at https://geoip.centminmod.com/v2/ which uses Maxmind's GeoLite2 database versus IP lookup at https://www.iplocation.net/ which shows results for 3 other databases, ip2location, ipinfo and db-ip database (the later is the CSF Firewall default).

    For instance, my Brisbane VPN IP address looked up in Maxmind GeoLite2 database, ip2location and ipinfo all show correct Brisbane, Queensland Australia for city, state and country. While db-ip reports incorrect Victorian city/state but correct country code.
     
Thread Status:
Not open for further replies.