Welcome to Centmin Mod Community
Become a Member

Centmin Mod Maxmind GeoLite2 Free Database Download Changes

Discussion in 'Centmin Mod News' started by eva2000, Jan 3, 2020.

Thread Status:
Not open for further replies.
  1. eva2000

    eva2000 Administrator Staff Member

    49,563
    11,375
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +17,678
    Local Time:
    8:46 PM
    Nginx 1.21.x
    MariaDB 10.x
    Maxmind announced significant changes to their GeoLite2 free database offerings and from December 30 2019 onwards, they will no longer be providing free public access to GeoLite2 free database downloads. Due to California Consumer Privacy Act (CCPA) laws, they will also now require folks to register for a free Maxmind account to be able to download and use their GeoLite2 free databases via a generated Maxmind token API key at https://www.maxmind.com/en/accounts/current/license-key. This will break alot of services/web apps which relied on downloading the GeoLite2 databases from a publicly known url link until those services/web apps update to use a Maxmind token API key.

    Note, Maxmind account registration signup doesn't allow VPN users to sign up at https://www.maxmind.com/en/geolite2/signup. You'd need to disable your VPN for just the registration phase. Once registered, you can re-enable your VPN to login to your Maxmind account.

    Centmin Mod Maxmind GeoLite2 Free Database Usage Updates



    Centmin Mod 123.09beta01 and newer branches make use of Maxmind's GeoLite2 free databases in two pieces of software:
    1. CSF Firewall's own GeoLite2 free database integration. CSF Firewall folks updated and released v13.09 to support Maxmind's new token API key method of downloading GeoLite2 databases and they use a new MM_LICENSE_KEY variable in /etc/csf/csf.conf for this. If you do not set a MM_LICENSE_KEY in CSF Firewall config file, your Login Failure Daemon (lfd) log at /var/log/lfd.log will show the following error message:
      Code (Text):
      Jan  2 03:15:05 host lfd[1488]: CC Error: Country Code Lookups setting MM_LICENSE_KEY must be set in /etc/csf/csf.conf to continue using the MaxMind databases
      .
    2. Centmin Mod Nginx's optional ngx_http_geoip2_module which is disabled by default. However, ngx_http2_geoip2_module can be optionally enabled by end users as outlined in the How to enable GeoIP 2 Lite Nginx Module Support.
    On January 2 2020, I updated Centmin Mod 123.09beta01 branch with changes to support Maxmind's new GeoLite2 account and token API method of downloading for both CSF Firewall and Nginx's ngx_http_geoip2_module.

    To update Centmin Mod 123.09beta01, you can just run in SSH, the cmupdate command and then run centmin.sh menu once and exit.
    Code (Text):
    cmupdate


    Once Centmin Mod 123.09beta01 is updated:
    • When you set MM_LICENSE_KEY in persistent config file at /etc/centminmod/custom_config.inc with your manually obtained and generated Maxmind account's token API key via https://www.maxmind.com/en/geolite2/signup and generate the token API in Services > My License Key section of your Maxmind account or via link at https://www.maxmind.com/en/accounts/current/license-key and then run centmin.sh menu once and then exit, then both CSF Firewall and Nginx geoip2 nginx module routines will download GeoLite2 database via Maxmind API for latest version.
      Code (Text):
      MM_LICENSE_KEY='your_maxmind_api_key'
    • If you do not set MM_LICENSE_KEY in persistent config file at /etc/centminmod/custom_config.inc, then Centmin Mod will use a new shared Maxmind token API key I created just for Centmin Mod usage.
    • When you have ngx_http_geoip2_module enabled, Centmin Mod also creates a GeoIP 2 database update cronjob
      Code (Text):
      20 2 * * 4 /usr/local/src/centminmod/tools/geoip2db-update.sh && nprestart >/dev/null 2>&1
      . You can wait for this cronjob to run to update your Centmin Mod system GeoLIte2 databases or manually run the cronjob once
      Code (Text):
      /usr/local/src/centminmod/tools/geoip2db-update.sh
    Relevant outlined commits for these changes.

    Verifying Maxmind API Downloaded GeoLite2 Database



    Maxmind GeoLite2 database last public release dated version is December 23-25, 2019 and the first GeoLite2 database to be released under the new registered Maxmind account token API downloaded version is dated December 31, 2019.


    CSF Firewall stores it's GeoLite2 database files in /var/lib/csf/Geo/ and based on it's CC_INTERVAL default, it will check and update database every 14 days.

    CSF Firewall last public GeoLite2 database release version dated December 23, 2019. On existing Centmin Mod 123.09beta01 installs, you will most likely see this dated version right now unless you did a fresh Centmin Mod 123.09beta01 or newer install - in which case you should see December 31, 2019 dated files.
    Code (Text):
    ls -lAh /var/lib/csf/Geo/
    total 16M
    -rw------- 1 root root   55 Dec 23 19:00 COPYRIGHT.txt
    -rw------- 1 root root  13M Dec 23 19:00 GeoLite2-Country-Blocks-IPv4.csv
    -rw------- 1 root root 3.8M Dec 23 19:00 GeoLite2-Country-Blocks-IPv6.csv
    -rw------- 1 root root 9.7K Dec 23 19:00 GeoLite2-Country-Locations-en.csv
    -rw------- 1 root root  433 Dec 23 19:00 LICENSE.txt
    -rw------- 1 root root  116 Dec 23 19:00 README.txt
    

    For Centmin Mod Nginx ngx_http_geoip2_module enabled servers, the GeoLite2 free database is stored at /usr/share/GeoIP (which also stores GeoIP legacy database as well).
    Code (Text):
    ls -lah /usr/share/GeoIP | grep GeoLite2            
    -rw-r--r--    1 root root 6.4M Dec 31 13:49 GeoLite2-ASN.mmdb
    -rw-r--r--    1 root root 3.6M Dec 31 13:49 GeoLite2-ASN.tar.gz
    -rw-r--r--    1 root root  59M Dec 31 16:33 GeoLite2-City.mmdb
    -rw-r--r--    1 root root  28M Dec 31 16:33 GeoLite2-City.tar.gz
    -rw-r--r--    1 root root 3.9M Dec 31 16:32 GeoLite2-Country.mmdb
    -rw-r--r--    1 root root 2.0M Dec 31 16:32 GeoLite2-Country.tar.gz
    


    GeoLite2 News Tracking Via Slack



    With GeoLite2 changes having a dramatic effect on Centmin Mod's products, I have setup a custom Slack channel to keep track of GeoIP/GeoLite2 news and announcements. A common way of tracking RSS feeds and news I outlined at How To Keep Informed Of Centmin Mod Related Updates. Slack documentation for adding RSS feeds here.

    RSS feeds for their blog and release notes pages
    geoip2news-slack-01.png
     
    Last edited: Jan 4, 2020
  2. eva2000

    eva2000 Administrator Staff Member

    49,563
    11,375
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +17,678
    Local Time:
    8:46 PM
    Nginx 1.21.x
    MariaDB 10.x
  3. eva2000

    eva2000 Administrator Staff Member

    49,563
    11,375
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +17,678
    Local Time:
    8:46 PM
    Nginx 1.21.x
    MariaDB 10.x

    CSF Firewall v14.00 Country Code Changes



    Looks like CSF Firewall v14.00 was released to switch away from using Maxmind's GeoLite2 database for Country code lookups for new CSF Firewall installs while older CSF Firewall installs continue to use Maxmind's GeoLite2 databases.
    from /etc/csf/csf.conf for existing CSF Firewall installs you can see CC_SRC = "1" set to use Maxmind GeoLite2 database instead of CC_SRC = "2" for new CSF Firewall installs for using different source databases via db-ip etc.
    Code (Text):
    # Set the following to your preferred source:
    #
    # "1" - MaxMind
    # "2" - db-ip, ipverse, iptoasn
    #
    # The default is "2" on new installations of csf, or set to "1" to use the
    # MaxMind databases after obtaining a license key
    CC_SRC = "1"
    

    So for Centmin Mod 123.09beta01 fresh installs after January 9th, 2020 you should be using CC_SRC = "2" for db-ip Country database and not require signing up for Maxmind account and not requiring to set up MM_LICENSE_KEY in /etc/csf/csf.conf.

    However, if you use and enable Centmin Mod Nginx's optional ngx_http_geoip2_module which is disabled by default as outlined in the How to enable GeoIP 2 Lite Nginx Module Support, then you still will need to set MM_LICENSE_KEY in persistent config file at /etc/centminmod/custom_config.inc with your manually obtained and generated Maxmind account's token API key via https://www.maxmind.com/en/geolite2/signup and generate the token API in Services > My License Key section of your Maxmind account or via link at https://www.maxmind.com/en/accounts/current/license-key and then run centmin.sh menu once and then exit.
     
  4. eva2000

    eva2000 Administrator Staff Member

    49,563
    11,375
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +17,678
    Local Time:
    8:46 PM
    Nginx 1.21.x
    MariaDB 10.x
    From today (Jan 20, 2020) onwards, I will be setting CSF Firewall's CC_SRC back to using maxmind database instead of previous post mentioned db-ip source via CC_SRC="1" in /etc/csf/csf.conf. Folks have been reporting IP geolocation data for Country codes for db-ip sourced database have not been as accurate as with maxmind database.

    For existing Centmin Mod 123.09beta01 users, you can switch from CC_SRC="2" back to CC_SRC="1" via these commands which backup your /etc/csf/csf.conf, sed replacement and then CSF Firewall restart and then grep filter check that the config file has indeed changed CC_SRC="1"
    Code (Text):
    csf --profile backup before_cc_src_1_switch
    sed -i 's|CC_SRC = "2"|CC_SRC = "1"|' /etc/csf/csf.conf
    csf -ra
    grep 'CC_SRC' /etc/csf/csf.conf
    

    You can compare and test an IP's geolocation data using my own GeoIP lookup site at https://geoip.centminmod.com/v2/ which uses Maxmind's GeoLite2 database versus IP lookup at https://www.iplocation.net/ which shows results for 3 other databases, ip2location, ipinfo and db-ip database (the later is the CSF Firewall default).

    For instance, my Brisbane VPN IP address looked up in Maxmind GeoLite2 database, ip2location and ipinfo all show correct Brisbane, Queensland Australia for city, state and country. While db-ip reports incorrect Victorian city/state but correct country code.
     
  5. eva2000

    eva2000 Administrator Staff Member

    49,563
    11,375
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +17,678
    Local Time:
    8:46 PM
    Nginx 1.21.x
    MariaDB 10.x
    Got some bad news today from folks at Maxmind. Because Centmin Mod LEMP stack is a free open source project, anyone can download and install it. So that means anyone can install CSF Firewall and thus the Geolite2 database dependency that CSF Firewall requires. Back in 2020, Maxmind required Geolite2 database users to register a Maxmind account to be able to download it. To make it easier for Centmin Mod users, I updated it so that I registered a Maxmind account and provided a shared API key so that CSF Firewall installs and thus Geolite2 databases continued to download and install uninterrupted. And provided Centmin Mod users with option to register their own Maxmind accounts to swap the shared API key for their own.

    However, because Centmin Mod can be downloaded and installed by anyone, it seems some users are from countries which are embargoed by the USA. Which means Maxmind saw this as a breach and has blocked my Maxmind account's access to Geolite2 database as such CSF Firewall will have issues using the database for IP/geolocation related functionality :(

    Registering Your Own Maxmind Account



    For existing users, you can register your own Maxmind account and get your own API key as outlined at https://community.centminmod.com/th...eolite2-free-database-download-changes.18959/. To that, you can set MM_LICENSE_KEY variable with your own API key in persistent config file at /etc/centminmod/custom_config.inc with your manually obtained and generated Maxmind account's token API key via GeoLite2 Sign Up | MaxMind and generate the token API in Services > My License Key section of your Maxmind account or via link at https://www.maxmind.com/en/accounts/current/license-key and then run centmin.sh menu once and then exit, then both CSF Firewall and Nginx geoip2 nginx module routines will download GeoLite2 database via Maxmind API for latest version.

    So if you want to continue using Maxmind Geolite2 database, you'd have to register your own Maxmind account and get own API key and these 2 variables in persistent config file /etc/centminmod/custom_config.inc
    Code (Text):
    MM_LICENSE_KEY='YOUR_OWN_MAXMIND_API_KEY'
    MM_CSF_SRC='y'
    


    To discuss this, please use discussion thread at https://community.centminmod.com/th...eolite2-free-database-download-changes.18960/

    Alternative For CSF Firewall Use DB-IP Geolocation Database



    I've just updated 124.00stable and 130.00beta01 switching CSF Firewall's geolocation database dependency away from Maxmind's Geolite2 database to DB-IP database. This only fixes CSF Firewall's usage. But Nginx's optional Geolite2 Nginx module and Modsecurity still need to find a solution for.

    You can switch from Maxmind Geolite2 database to alternative DB-IP geolocation database via setting MM_CSF_SRC='n' by default. For existing Centmin Mod users, you'll need to run cmupdate command to update your local code and then run centmin.sh menu once and exit to complete the update change

    You can verify the change by inspecting CSF Firewall's config file at /etc/csf/csf.conf

    Code (Text):
    grep -i -B3 ^CC_SRC /etc/csf/csf.conf
    #
    # The default is "2" on new installations of csf, or set to "1" to use the
    # MaxMind databases after obtaining a license key
    CC_SRC = "2"
    
     
  6. eva2000

    eva2000 Administrator Staff Member

    49,563
    11,375
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +17,678
    Local Time:
    8:46 PM
    Nginx 1.21.x
    MariaDB 10.x
    Good news. After I replied to Maxmind's email explaining my situation with Centmin Mod/CSF Firewall and GeoLite2 database usage, they agreed to not disable my Maxmind's account GeoLite2 database download/access :)

    But in future they may ban downloads at the IP address level for embargoed countries. So steps and measures outlined at https://community.centminmod.com/th...e-database-download-changes.18959/#post-93003 would still be applicable if you want to continue using Maxmind's GeoLite2 database with CSF Firewall via persistent config variables with your own registered Maxmind API key and setting MM_LICENSE_KEY='YOUR_OWN_MAXMIND_API_KEY' and MM_CSF_SRC='y' instead of using DP-IP geolocation database via the new default MM_CSF_SRC='n' setting.
     
  7. eva2000

    eva2000 Administrator Staff Member

    49,563
    11,375
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +17,678
    Local Time:
    8:46 PM
    Nginx 1.21.x
    MariaDB 10.x
  8. eva2000

    eva2000 Administrator Staff Member

    49,563
    11,375
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +17,678
    Local Time:
    8:46 PM
    Nginx 1.21.x
    MariaDB 10.x
    This serves as another reminder for folks running Centmin Mod in internationally sanctioned countries, that Maxmind is disabling their Geolite2 database download access in this countries come September 1, 2022 - next month. Better than their previous policy of blocking my entire Maxmind account!

    Got an email from Maxmind again. I assume these are older Centmin Mod installs prior to May 2022 or Centmin Mod installs that haven't updated Centmin Mod for the changes to using DP-IP etc.
    The below quoted statement by me is the current situation for Centmin Mod new installs since late May 2022 in that Centmin Mod doesn't use Maxmind Geolite2 database by default anymore for CSF Firewall but instead uses DP-IP geolocation database. However, Maxmind Geolite2 database is still used for optional Nginx Geolite2 Nginx module which isn't installed by default. If you need Maxmind Geolite2 database usage instead of DP-IP, follow below instructions and register your own Maxmind account and API key GeoLite2 Sign Up | MaxMind

     
Thread Status:
Not open for further replies.