Centmin Mod Maxmind GeoLite2 Free Database Download Changes

Maxmind announced significant changes to their GeoLite2 free database offerings and from December 30 2019 onwards, they will no longer be providing free public access to GeoLite2 free database downloads. Due to California Consumer Privacy Act (CCPA) laws, they will also now require folks to register for a free Maxmind account to be able to download and use their GeoLite2 free databases via a generated Maxmind token API key at https://www.maxmind.com/en/accounts/current/license-key. This will break alot of services/web apps which relied on downloading the GeoLite2 databases from a publicly known url link until those services/web apps update to use a Maxmind token API key.

Note, Maxmind account registration signup doesn't allow VPN users to sign up at https://www.maxmind.com/en/geolite2/signup. You'd need to disable your VPN for just the registration phase. Once registered, you can re-enable your VPN to login to your Maxmind account.

---

Centmin Mod Maxmind GeoLite2 Free Database Usage Updates

Centmin Mod 123.09beta01 and newer branches make use of Maxmind's GeoLite2 free databases in two pieces of software:

1. CSF Firewall's own GeoLite2 free database integration. CSF Firewall folks updated and released v13.09 to support Maxmind's new token API key method of downloading GeoLite2 databases and they use a new MM_LICENSE_KEY variable in /etc/csf/csf.conf for this. If you do not set a MM_LICENSE_KEY in CSF Firewall config file, your Login Failure Daemon (lfd) log at /var/log/lfd.log will show the following error message:
   Code (Text):

   Jan  2 03:15:05 host lfd[1488]: CC Error: Country Code Lookups setting MM_LICENSE_KEY must be set in /etc/csf/csf.conf to continue using the MaxMind databases

   .
2. Centmin Mod Nginx's optional ngx_http_geoip2_module which is disabled by default. However, ngx_http2_geoip2_module can be optionally enabled by end users as outlined in the How to enable GeoIP 2 Lite Nginx Module Support.

On January 2 2020, I updated Centmin Mod 123.09beta01 branch with changes to support Maxmind's new GeoLite2 account and token API method of downloading for both CSF Firewall and Nginx's ngx_http_geoip2_module.

To update Centmin Mod 123.09beta01, you can just run in SSH, the cmupdate command and then run centmin.sh menu once and exit.

Code (Text):

cmupdate

Once Centmin Mod 123.09beta01 is updated:

• When you set MM_LICENSE_KEY in persistent config file at /etc/centminmod/custom_config.inc with your manually obtained and generated Maxmind account's token API key via https://www.maxmind.com/en/geolite2/signup and generate the token API in Services > My License Key section of your Maxmind account or via link at https://www.maxmind.com/en/accounts/current/license-key and then run centmin.sh menu once and then exit, then both CSF Firewall and Nginx geoip2 nginx module routines will download GeoLite2 database via Maxmind API for latest version.
  Code (Text):

  MM_LICENSE_KEY='your_maxmind_api_key'

• If you do not set MM_LICENSE_KEY in persistent config file at /etc/centminmod/custom_config.inc, then Centmin Mod will use a new shared Maxmind token API key I created just for Centmin Mod usage.
• When you have ngx_http_geoip2_module enabled, Centmin Mod also creates a GeoIP 2 database update cronjob
  Code (Text):

  20 2 * * 4 /usr/local/src/centminmod/tools/geoip2db-update.sh && nprestart >/dev/null 2>&1

  . You can wait for this cronjob to run to update your Centmin Mod system GeoLIte2 databases or manually run the cronjob once
  Code (Text):

  /usr/local/src/centminmod/tools/geoip2db-update.sh

Relevant outlined commits for these changes.

• https://community.centminmod.com/threads/update-geoip2-lite-database-download-in-123-09beta01.18951/
• https://community.centminmod.com/th...ite2-database-routines-in-123-09beta01.18953/
• https://community.centminmod.com/th...ite2-database-routines-in-123-09beta01.18954/
• https://community.centminmod.com/th...ite2-database-routines-in-123-09beta01.18955/
• https://community.centminmod.com/th...ite2-database-routines-in-123-09beta01.18956/
• https://community.centminmod.com/th...mind-geolite2-routines-in-123-09beta01.18957/

Verifying Maxmind API Downloaded GeoLite2 Database

Maxmind GeoLite2 database last public release dated version is December 23-25, 2019 and the first GeoLite2 database to be released under the new registered Maxmind account token API downloaded version is dated December 31, 2019.

CSF Firewall stores it's GeoLite2 database files in /var/lib/csf/Geo/ and based on it's CC_INTERVAL default, it will check and update database every 14 days.

CSF Firewall last public GeoLite2 database release version dated December 23, 2019. On existing Centmin Mod 123.09beta01 installs, you will most likely see this dated version right now unless you did a fresh Centmin Mod 123.09beta01 or newer install - in which case you should see December 31, 2019 dated files.

Code (Text):

ls -lAh /var/lib/csf/Geo/
total 16M
-rw------- 1 root root   55 Dec 23 19:00 COPYRIGHT.txt
-rw------- 1 root root  13M Dec 23 19:00 GeoLite2-Country-Blocks-IPv4.csv
-rw------- 1 root root 3.8M Dec 23 19:00 GeoLite2-Country-Blocks-IPv6.csv
-rw------- 1 root root 9.7K Dec 23 19:00 GeoLite2-Country-Locations-en.csv
-rw------- 1 root root  433 Dec 23 19:00 LICENSE.txt
-rw------- 1 root root  116 Dec 23 19:00 README.txt

For Centmin Mod Nginx ngx_http_geoip2_module enabled servers, the GeoLite2 free database is stored at /usr/share/GeoIP (which also stores GeoIP legacy database as well).

Code (Text):

ls -lah /usr/share/GeoIP | grep GeoLite2            
-rw-r--r--    1 root root 6.4M Dec 31 13:49 GeoLite2-ASN.mmdb
-rw-r--r--    1 root root 3.6M Dec 31 13:49 GeoLite2-ASN.tar.gz
-rw-r--r--    1 root root  59M Dec 31 16:33 GeoLite2-City.mmdb
-rw-r--r--    1 root root  28M Dec 31 16:33 GeoLite2-City.tar.gz
-rw-r--r--    1 root root 3.9M Dec 31 16:32 GeoLite2-Country.mmdb
-rw-r--r--    1 root root 2.0M Dec 31 16:32 GeoLite2-Country.tar.gz

GeoLite2 News Tracking Via Slack

With GeoLite2 changes having a dramatic effect on Centmin Mod's products, I have setup a custom Slack channel to keep track of GeoIP/GeoLite2 news and announcements. A common way of tracking RSS feeds and news I outlined at How To Keep Informed Of Centmin Mod Related Updates. Slack documentation for adding RSS feeds here.

RSS feeds for their blog and release notes pages

• https://blog.maxmind.com/category/category-ip-geolocation/feed/
• https://dev.maxmind.com/feed/?post_type=release_note&product=geoip2

 

To discuss these changes or report bugs/issues check out the dedicated discussion thread at https://community.centminmod.com/th...eolite2-free-database-download-changes.18960/

 

CSF Firewall v14.00 Country Code Changes

Looks like CSF Firewall v14.00 was released to switch away from using Maxmind's GeoLite2 database for Country code lookups for new CSF Firewall installs while older CSF Firewall installs continue to use Maxmind's GeoLite2 databases.

from /etc/csf/csf.conf for existing CSF Firewall installs you can see CC_SRC = "1" set to use Maxmind GeoLite2 database instead of CC_SRC = "2" for new CSF Firewall installs for using different source databases via db-ip etc.

Code (Text):

# Set the following to your preferred source:
#
# "1" - MaxMind
# "2" - db-ip, ipverse, iptoasn
#
# The default is "2" on new installations of csf, or set to "1" to use the
# MaxMind databases after obtaining a license key
CC_SRC = "1"

So for Centmin Mod 123.09beta01 fresh installs after January 9th, 2020 you should be using CC_SRC = "2" for db-ip Country database and not require signing up for Maxmind account and not requiring to set up MM_LICENSE_KEY in /etc/csf/csf.conf.

However, if you use and enable Centmin Mod Nginx's optional ngx_http_geoip2_module which is disabled by default as outlined in the How to enable GeoIP 2 Lite Nginx Module Support, then you still will need to set MM_LICENSE_KEY in persistent config file at /etc/centminmod/custom_config.inc with your manually obtained and generated Maxmind account's token API key via https://www.maxmind.com/en/geolite2/signup and generate the token API in Services > My License Key section of your Maxmind account or via link at https://www.maxmind.com/en/accounts/current/license-key and then run centmin.sh menu once and then exit.

 

From today (Jan 20, 2020) onwards, I will be setting CSF Firewall's CC_SRC back to using maxmind database instead of previous post mentioned db-ip source via CC_SRC="1" in /etc/csf/csf.conf. Folks have been reporting IP geolocation data for Country codes for db-ip sourced database have not been as accurate as with maxmind database.

For existing Centmin Mod 123.09beta01 users, you can switch from CC_SRC="2" back to CC_SRC="1" via these commands which backup your /etc/csf/csf.conf, sed replacement and then CSF Firewall restart and then grep filter check that the config file has indeed changed CC_SRC="1"

Code (Text):

csf --profile backup before_cc_src_1_switch
sed -i 's|CC_SRC = "2"|CC_SRC = "1"|' /etc/csf/csf.conf
csf -ra
grep 'CC_SRC' /etc/csf/csf.conf

You can compare and test an IP's geolocation data using my own GeoIP lookup site at https://geoip.centminmod.com/v2/ which uses Maxmind's GeoLite2 database versus IP lookup at https://www.iplocation.net/ which shows results for 3 other databases, ip2location, ipinfo and db-ip database (the later is the CSF Firewall default).

For instance, my Brisbane VPN IP address looked up in Maxmind GeoLite2 database, ip2location and ipinfo all show correct Brisbane, Queensland Australia for city, state and country. While db-ip reports incorrect Victorian city/state but correct country code.

 

Got some bad news today from folks at Maxmind. Because Centmin Mod LEMP stack is a free open source project, anyone can download and install it. So that means anyone can install CSF Firewall and thus the Geolite2 database dependency that CSF Firewall requires. Back in 2020, Maxmind required Geolite2 database users to register a Maxmind account to be able to download it. To make it easier for Centmin Mod users, I updated it so that I registered a Maxmind account and provided a shared API key so that CSF Firewall installs and thus Geolite2 databases continued to download and install uninterrupted. And provided Centmin Mod users with option to register their own Maxmind accounts to swap the shared API key for their own.

However, because Centmin Mod can be downloaded and installed by anyone, it seems some users are from countries which are embargoed by the USA. Which means Maxmind saw this as a breach and has blocked my Maxmind account's access to Geolite2 database as such CSF Firewall will have issues using the database for IP/geolocation related functionality

Registering Your Own Maxmind Account

For existing users, you can register your own Maxmind account and get your own API key as outlined at https://community.centminmod.com/th...eolite2-free-database-download-changes.18959/. To that, you can set MM_LICENSE_KEY variable with your own API key in persistent config file at /etc/centminmod/custom_config.inc with your manually obtained and generated Maxmind account's token API key via GeoLite2 Sign Up | MaxMind and generate the token API in Services > My License Key section of your Maxmind account or via link at https://www.maxmind.com/en/accounts/current/license-key and then run centmin.sh menu once and then exit, then both CSF Firewall and Nginx geoip2 nginx module routines will download GeoLite2 database via Maxmind API for latest version.

So if you want to continue using Maxmind Geolite2 database, you'd have to register your own Maxmind account and get own API key and these 2 variables in persistent config file /etc/centminmod/custom_config.inc

Code (Text):

MM_LICENSE_KEY='YOUR_OWN_MAXMIND_API_KEY'
MM_CSF_SRC='y'

To discuss this, please use discussion thread at https://community.centminmod.com/th...eolite2-free-database-download-changes.18960/

Alternative For CSF Firewall Use DB-IP Geolocation Database

I've just updated 124.00stable and 130.00beta01 switching CSF Firewall's geolocation database dependency away from Maxmind's Geolite2 database to DB-IP database. This only fixes CSF Firewall's usage. But Nginx's optional Geolite2 Nginx module and Modsecurity still need to find a solution for.

You can switch from Maxmind Geolite2 database to alternative DB-IP geolocation database via setting MM_CSF_SRC='n' by default. For existing Centmin Mod users, you'll need to run cmupdate command to update your local code and then run centmin.sh menu once and exit to complete the update change

You can verify the change by inspecting CSF Firewall's config file at /etc/csf/csf.conf

Code (Text):

grep -i -B3 ^CC_SRC /etc/csf/csf.conf
#
# The default is "2" on new installations of csf, or set to "1" to use the
# MaxMind databases after obtaining a license key
CC_SRC = "2"

 

Good news. After I replied to Maxmind's email explaining my situation with Centmin Mod/CSF Firewall and GeoLite2 database usage, they agreed to not disable my Maxmind's account GeoLite2 database download/access

But in future they may ban downloads at the IP address level for embargoed countries. So steps and measures outlined at https://community.centminmod.com/th...e-database-download-changes.18959/#post-93003 would still be applicable if you want to continue using Maxmind's GeoLite2 database with CSF Firewall via persistent config variables with your own registered Maxmind API key and setting MM_LICENSE_KEY='YOUR_OWN_MAXMIND_API_KEY' and MM_CSF_SRC='y' instead of using DP-IP geolocation database via the new default MM_CSF_SRC='n' setting.

 

Updated Centmin Mod blog article for advanced Centmin Mod installation guide to mention this Maxmind GeoLite2 database change Centmin Mod Advanced Customised Installation Guide - Centmin Mod Blog

 

This serves as another reminder for folks running Centmin Mod in internationally sanctioned countries, that Maxmind is disabling their Geolite2 database download access in this countries come September 1, 2022 - next month. Better than their previous policy of blocking my entire Maxmind account!

Got an email from Maxmind again. I assume these are older Centmin Mod installs prior to May 2022 or Centmin Mod installs that haven't updated Centmin Mod for the changes to using DP-IP etc.

The below quoted statement by me is the current situation for Centmin Mod new installs since late May 2022 in that Centmin Mod doesn't use Maxmind Geolite2 database by default anymore for CSF Firewall but instead uses DP-IP geolocation database. However, Maxmind Geolite2 database is still used for optional Nginx Geolite2 Nginx module which isn't installed by default. If you need Maxmind Geolite2 database usage instead of DP-IP, follow below instructions and register your own Maxmind account and API key GeoLite2 Sign Up | MaxMind

 

Unfortunate news update. Looks like Maxmind folks now scan Github repositories for Maxmind GeoLite2 database API keys and came across Centmin Mod's shared API key I mentioned in quote above. They are deactivating that key and disabling its usage.

Solution:

For existing users, you can register your own Maxmind account and get your own API key as outlined at https://community.centminmod.com/th...eolite2-free-database-download-changes.18959/. To that, you can set MM_LICENSE_KEY variable with your own API key in persistent config file at /etc/centminmod/custom_config.inc with your manually obtained and generated Maxmind account's token API key via GeoLite2 Sign Up | MaxMind and generate the token API in Services > My License Key section of your Maxmind account or via link at https://www.maxmind.com/en/accounts/current/license-key and then run centmin.sh menu once and then exit, then both CSF Firewall and Nginx geoip2 nginx module routines will download GeoLite2 database via Maxmind API for latest version.

Both 124.00stable and 130.00beta01 by default out of the box already have switched CSF Firewall's geolocation database dependency away from Maxmind's Geolite2 database to DB-IP database. This only fixes CSF Firewall's usage. But Nginx's optional Geolite2 Nginx module and Modsecurity still need to find a solution for. You can switch from Maxmind Geolite2 database to alternative DB-IP geolocation database via setting MM_CSF_SRC='n' by default.

So if you want to continue using Maxmind Geolite2 database, you'd have to register your own Maxmind account and get own API key and these 2 variables in persistent config file /etc/centminmod/custom_config.inc

Code (Text):

MM_LICENSE_KEY='YOUR_OWN_MAXMIND_API_KEY'
MM_CSF_SRC='y'

To discuss this, please use discussion thread at https://community.centminmod.com/th...eolite2-free-database-download-changes.18960/

 

Maxmind GeoLite2 Database Download & Rate Limit Enforcement March 2024

Centmin Mod 130.00beta01 has been updated to tackle Maxmind's latest changes in GeoLite2 database download policy and rate limits https://community.centminmod.com/th...ase-routines-for-nginx-in-130-00beta01.24815/. You can discuss these changes in existing thread at https://community.centminmod.com/th...eolite2-free-database-download-changes.18960/.

• Maxmind is now enforcing GeoLite2 database download rate limits as of March 2024 to 30 daily direct API downloads and recommends users to download a copy locally to distribute to many servers if you hit the limits https://support.maxmind.com/hc/en-us/articles/4408216129947-Download-and-Update-Databases
• Maxmind has been slowly clamping down on the free GeoLite2 database downloads see https://community.centminmod.com/th...eolite2-free-database-download-changes.18960/ and https://community.centminmod.com/th...eolite2-free-database-download-changes.18959/.
• Centmin Mod uses GeoLIte2 databases in 2 places, 1 is via CSF Firewall it optionally can use GeoLite2 database for IP address geo meta data lookups but defaults to now using alternative geo meta databases like DB-IP instead of GeoLite2 in CSF Firewall config file /etc/csf/csf.conf CC_SRC = "2" variable set as outlined at https://community.centminmod.com/th...eolite2-free-database-download-changes.18960/. The Maxmind changes here will need CSF Firewall developers to update if impacted
• Centmin Mod's 2nd usage of GeoLite2 database is in optional Nginx GeoLite2 nginx module which isn't enabled by default unless you set in persistent config file /etc/centminmod/custom_config.inc the variable NGINX_GEOIPTWOLITE='y' prior to centmin.sh menu option 4 Nginx upgrade/recompiles or prior to initial Centmin Mod installation. Centmin Mod 130.00beta01 latest update is for this second usage case for Nginx GeoLite2 nginx module routine to use cached copy if the Centmin Mod user doesn't setup their own Maxmind API key as outlined under solutions header in post at .https://community.centminmod.com/th...eolite2-free-database-download-changes.18960/
• Existing Centmin Mod 130.00beta01 users can get this update via cmupdate command. Once that is tested to work well, will also backport changes to 124.00stable.