Welcome to Centmin Mod Community
Become a Member

CSF Discuss Maxmind GeoLite2 Free Database Download Changes

Discussion in 'Other Centmin Mod Installed software' started by eva2000, Jan 3, 2020.

  1. eva2000

    eva2000 Administrator Staff Member

    53,811
    12,159
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,711
    Local Time:
    8:52 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    This thread is dedicated to Maxmind's GeoLite2 free database changes outlined at https://community.centminmod.com/th...eolite2-free-database-download-changes.18959/

    Update:

    CSF Firewall v14.00 Country Code Changes


    Looks like CSF Firewall v14.00 was released to switch away from using Maxmind's GeoLite2 database for Country code lookups for new CSF Firewall installs while older CSF Firewall installs continue to use Maxmind's GeoLite2 databases.
    from /etc/csf/csf.conf for existing CSF Firewall installs you can see CC_SRC = "1" set to use Maxmind GeoLite2 database instead of CC_SRC = "2" for new CSF Firewall installs for using different source databases via db-ip etc.
    Code (Text):
    # Set the following to your preferred source:
    #
    # "1" - MaxMind
    # "2" - db-ip, ipverse, iptoasn
    #
    # The default is "2" on new installations of csf, or set to "1" to use the
    # MaxMind databases after obtaining a license key
    CC_SRC = "1"
    

    So for Centmin Mod 123.09beta01 fresh installs after January 9th, 2020 you should be using CC_SRC = "2" for db-ip Country database and not require signing up for Maxmind account and not requiring to set up MM_LICENSE_KEY in /etc/csf/csf.conf.
    However, if you use and enable Centmin Mod Nginx's optional ngx_http_geoip2_module which is disabled by default as outlined in the How to enable GeoIP 2 Lite Nginx Module Support, then you still will need to set MM_LICENSE_KEY in persistent config file at /etc/centminmod/custom_config.inc with your manually obtained and generated Maxmind account's token API key via https://www.maxmind.com/en/geolite2/signup and generate the token API in Services > My License Key section of your Maxmind account or via link at https://www.maxmind.com/en/accounts/current/license-key and then run centmin.sh menu once and then exit.

    October 28, 2023 Update


    Unfortunate news update. Looks like Maxmind folks now scan Github repositories for Maxmind GeoLite2 database API keys and came across Centmin Mod's shared API key I mentioned in quote above. They are deactivating that key and disabling its usage.

    Solution:
    For existing users, you can register your own Maxmind account and get your own API key as outlined at https://community.centminmod.com/th...eolite2-free-database-download-changes.18959/. For that, you can set MM_LICENSE_KEY variable with your own API key in persistent config file at /etc/centminmod/custom_config.inc with your manually obtained and generated Maxmind account's token API key via GeoLite2 Sign Up | MaxMind and generate the token API in Services > My License Key section of your Maxmind account or via link at https://www.maxmind.com/en/accounts/current/license-key and then run centmin.sh menu once and then exit, then both CSF Firewall and Nginx geoip2 nginx module routines will download GeoLite2 database via Maxmind API for latest version.

    Both 124.00stable and 130.00beta01 by default out of the box already have switched CSF Firewall's geolocation database dependency away from Maxmind's Geolite2 database to DB-IP database. This only fixes CSF Firewall's usage. But Nginx's optional Geolite2 Nginx module and Modsecurity still need to find a solution for. You can switch from Maxmind Geolite2 database to alternative DB-IP geolocation database via setting MM_CSF_SRC='n' by default.

    So if you want to continue using Maxmind Geolite2 database, you'd have to register your own Maxmind account and get own API key and these 2 variables in persistent config file /etc/centminmod/custom_config.inc
    Code (Text):
    MM_LICENSE_KEY='YOUR_OWN_MAXMIND_API_KEY'
    MM_CSF_SRC='y'
    

    To discuss this, please use discussion thread at https://community.centminmod.com/th...eolite2-free-database-download-changes.18960/
     
    Last edited: Oct 28, 2023
  2. Andy

    Andy Active Member

    542
    89
    28
    Aug 6, 2014
    Ratings:
    +132
    Local Time:
    4:52 AM
    I see a Dec 27th date
    ls -lAh /var/lib/csf/Geo/
    -rw------- 1 root root 55 Dec 27 10:00 COPYRIGHT.txt
    -rw------- 1 root root 13M Dec 27 10:00 GeoLite2-Country-Blocks-IPv4.csv
    -rw------- 1 root root 3.8M Dec 27 10:00 GeoLite2-Country-Blocks-IPv6.csv
    -rw------- 1 root root 9.7K Dec 27 10:00 GeoLite2-Country-Locations-en.csv
    -rw------- 1 root root 433 Dec 27 10:00 LICENSE.txt
    -rw------- 1 root root 116 Dec 27 10:00 README.txt
     
  3. eva2000

    eva2000 Administrator Staff Member

    53,811
    12,159
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,711
    Local Time:
    8:52 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    Yeah it's relative to when you installed Centmin Mod for CSF Firewall's database as by default CC_INTERVAL is set to update GeoLite2 database in CSF Firewall every 14 days. So roughly around Dec 23 time. So you'd expect an update 14 days from Dec 27th, 2019 if you updated Centmin Mod 123.09beta01 and have setup your own Maxmind API key in persistent config file at /etc/centminmod/custom_config.inc or if you use updated 123.09beta01's shared Maxmind API key which is the default once you update Centmin Mod 123.09beta01
     
  4. Jon Snow

    Jon Snow Active Member

    795
    159
    43
    Jun 30, 2017
    Ratings:
    +236
    Local Time:
    6:52 AM
    Nginx 1.13.9
    MariaDB 10.1.31
    So once we update, we'll be required to create an account and get an api key?

    Should we create our own account or just use yours? Keeping privacy and our privacy policies in mind. What's the purpose of the account and what information does it store?
     
  5. eva2000

    eva2000 Administrator Staff Member

    53,811
    12,159
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,711
    Local Time:
    8:52 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    Not required but recommended to use your own maxmind account and api key instead of the shared one as maxmind account allows you query and track geolite2 database downloads including ip address of server, and the server's web host ASN details

    see Significant Changes to Accessing and Using GeoLite2 Databases | MaxMind Blog

    basically Maxmind requires a way to contact folks forward on 'Do Not Sell' requests for those who download their GeoLite2 databases due to CCPA law requirements

    Do Not Sell https://www.privacypolicies.com/blog/create-display-do-not-sell-my-personal-information-page/

    Within your Maxmind account there's a Do No Sell page which lists IPs that requested their data not be shared/sold
     
    Last edited: Jan 9, 2020
  6. Jon Snow

    Jon Snow Active Member

    795
    159
    43
    Jun 30, 2017
    Ratings:
    +236
    Local Time:
    6:52 AM
    Nginx 1.13.9
    MariaDB 10.1.31
    Alright, I guess I'll make my own account then.
     
  7. eva2000

    eva2000 Administrator Staff Member

    53,811
    12,159
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,711
    Local Time:
    8:52 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    Yeah ideally you should.

    FYI, just checked the Centmin Mod maxmind shared API key download stats and it's currently peaking at 128 downloads per day ! :D
     
  8. eva2000

    eva2000 Administrator Staff Member

    53,811
    12,159
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,711
    Local Time:
    8:52 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    Update check shows Centmin Mod maxmind shared API key download stats have now broken 200-275 downloads per day ! :cool::D
     
  9. rdan

    rdan Well-Known Member

    5,439
    1,399
    113
    May 25, 2014
    Ratings:
    +2,188
    Local Time:
    6:52 PM
    Mainline
    10.2
    minus 20+ as I used my own license :D.
     
  10. eva2000

    eva2000 Administrator Staff Member

    53,811
    12,159
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,711
    Local Time:
    8:52 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    yeah I use my own license too so ~160+ not counted :)
     
  11. eva2000

    eva2000 Administrator Staff Member

    53,811
    12,159
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,711
    Local Time:
    8:52 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    Was curious and rechecked the stats, maxmind license downloads for past 12 months totalled 107,000+ with peaks of just under 800 license downloads/day :D That's quite a few Centmin Mod installs! :cool:
     
  12. eva2000

    eva2000 Administrator Staff Member

    53,811
    12,159
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,711
    Local Time:
    8:52 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    Got some bad news today from folks at Maxmind. Because Centmin Mod LEMP stack is a free open source project, anyone can download and install it. So that means anyone can install CSF Firewall and thus the Geolite2 database dependency that CSF Firewall requires. Back in 2020, Maxmind required Geolite2 database users to register a Maxmind account to be able to download it. To make it easier for Centmin Mod users, I updated it so that I registered a Maxmind account and provided a shared API key so that CSF Firewall installs and thus Geolite2 databases continued to download and install uninterrupted. And provided Centmin Mod users with option to register their own Maxmind accounts to swap the shared API key for their own.

    However, because Centmin Mod can be downloaded and installed by anyone, it seems some users are from countries which are embargoed by the USA. Which means Maxmind saw this as a breach and has blocked my Maxmind account's access to Geolite2 database as such CSF Firewall will have issues using the database for IP/geolocation related functionality :(

    Registering Your Own Maxmind Account



    For existing users, you can register your own Maxmind account and get your own API key as outlined at https://community.centminmod.com/th...eolite2-free-database-download-changes.18959/. So you can set MM_LICENSE_KEY variable with your own API key in persistent config file at /etc/centminmod/custom_config.inc with your manually obtained and generated Maxmind account's token API key via GeoLite2 Sign Up | MaxMind and generate the token API in Services > My License Key section of your Maxmind account or via link at https://www.maxmind.com/en/accounts/current/license-key and then run centmin.sh menu once and then exit, then both CSF Firewall and Nginx geoip2 nginx module routines will download GeoLite2 database via Maxmind API for latest version.

    So if you want to continue using Maxmind Geolite2 database, you'd have to register your own Maxmind account and get own API key and these 2 variables in persistent config file /etc/centminmod/custom_config.inc
    Code (Text):
    MM_LICENSE_KEY='YOUR_OWN_MAXMIND_API_KEY'
    MM_CSF_SRC='y'
    
     
  13. rdan

    rdan Well-Known Member

    5,439
    1,399
    113
    May 25, 2014
    Ratings:
    +2,188
    Local Time:
    6:52 PM
    Mainline
    10.2
    So for new CMM user and new Server install, maybe it's better to change the default to:
    CC_SRC = "2"

    And disable Nginx GEOIP.
     
  14. eva2000

    eva2000 Administrator Staff Member

    53,811
    12,159
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,711
    Local Time:
    8:52 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    Yeah looks like have no choice now for new Centmin Mod installations but to move away from Maxmind's Geolite2 database usage for CSF Firewall and switch to DB-IP database via CC_SRC=2. Centmin Mod has an option for this outlined at https://community.centminmod.com/th...src-routine-in-123-09beta01.19054/#post-81091. I stayed with Maxmind because they had a more accurate Geolocation database. But switching to DB-IP is better than having no access and breaking CSF Firewall.

    If you want to set CC_SRC = "2" to use DB-IP database in CSF Firewall /etc/csf/csf.conf, just set MM_CSF_SRC='n' no persistent config file /etc/centminmod/custom_config.inc and run and exit centmin.sh once.

    GeoIP Nginx stuff will be problematic too as would for Nginx Modsecurity optional module IIRC. Will have to rework them and find alternatives.
     
  15. eva2000

    eva2000 Administrator Staff Member

    53,811
    12,159
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,711
    Local Time:
    8:52 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+

    Alternative For CSF Firewall Use DB-IP Geolocation Database



    I've just updated 124.00stable and 130.00beta01 switching CSF Firewall's geolocation database dependency away from Maxmind's Geolite2 database to DB-IP database. This only fixes CSF Firewall's usage. But Nginx's optional Geolite2 Nginx module and Modsecurity still need to find a solution for.

    You can switch from Maxmind Geolite2 database to alternative DB-IP geolocation database via setting MM_CSF_SRC='n' by default. For existing Centmin Mod users, you'll need to run cmupdate command to update your local code and then run centmin.sh menu once and exit to complete the update change.

    You can verify the change by inspecting CSF Firewall's config file at /etc/csf/csf.conf

    Code (Text):
    grep -i -B3 ^CC_SRC /etc/csf/csf.conf
    #
    # The default is "2" on new installations of csf, or set to "1" to use the
    # MaxMind databases after obtaining a license key
    CC_SRC = "2"
    
     
  16. rdan

    rdan Well-Known Member

    5,439
    1,399
    113
    May 25, 2014
    Ratings:
    +2,188
    Local Time:
    6:52 PM
    Mainline
    10.2
    But for us with working MM License, we can safely use Maxmind right?
    And we also need to add on custom_config this:
     
  17. eva2000

    eva2000 Administrator Staff Member

    53,811
    12,159
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,711
    Local Time:
    8:52 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    Yes if you want to continue using Maxmind Geolite2 database, you'd have to register your own Maxmind account and get own API key and these 2 variables in persistent config file /etc/centminmod/custom_config.inc
    Code (Text):
    MM_LICENSE_KEY='YOUR_OWN_MAXMIND_API_KEY'
    MM_CSF_SRC='y'
    
     
  18. eva2000

    eva2000 Administrator Staff Member

    53,811
    12,159
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,711
    Local Time:
    8:52 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    Good news. After I replied to Maxmind's email explaining my situation with Centmin Mod/CSF Firewall and GeoLite2 database usage, they agreed to not disable my Maxmind's account GeoLite2 database download/access :)

    But in future they may ban downloads at the IP address level for embargoed countries. So steps and measures outlined at https://community.centminmod.com/th...e-database-download-changes.18959/#post-93003 would still be applicable if you want to continue using Maxmind's GeoLite2 database with CSF Firewall via persistent config variables with your own registered Maxmind API key and setting MM_LICENSE_KEY='YOUR_OWN_MAXMIND_API_KEY' and MM_CSF_SRC='y' instead of using DP-IP geolocation database via the new default MM_CSF_SRC='n' setting.
     
  19. eva2000

    eva2000 Administrator Staff Member

    53,811
    12,159
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,711
    Local Time:
    8:52 PM
    Nginx 1.27.x
    MariaDB 10.x/11.4+
    FYI, just checking my Maxmind account's GeoLite2 database download numbers they range from 300-1300 downloads per day. In past 12 months, there's been 212,245 downloads of GeoLite2 database from Centmin Mod installations associated with the default Centmin Mod Maxmind shared API key. That of course excludes folks who switched to DP-IP database via MM_CSF_SRC='n' or are using their own Maxmind account's API key :)
     
  20. Jon Snow

    Jon Snow Active Member

    795
    159
    43
    Jun 30, 2017
    Ratings:
    +236
    Local Time:
    6:52 AM
    Nginx 1.13.9
    MariaDB 10.1.31
    So we don't have to do anything if we live in a country that the USA doesn't like? (eg. Russia)