/ Register

Join the community today
Register Now

When to expect GeoIp2 in centminmod?

Discussion in 'Feature Requests & Suggestions' started by wmtech, May 9, 2018.

Tags:
Previous Thread Next Thread
Loading...
Page 1 of 2
  1. May 9, 2018 #1
    wmtech

    wmtech Active Member

    103
    28
    28
    Jul 22, 2017
    Ratings:
    +75
    Local Time:
    11:27 PM
    Maxmind does no longer publish updated legacy GeoIp databases (the last update was March) and will stop to provide any downloads of them at the end of this year.

    So we need to switch to GeoIp2 in the near future. Before I manually install the module, I would like to ask if there are plans to add GeoIp2 into centminmod soon?

    Thanks.
     
  2. May 9, 2018 #2
    eva2000

    eva2000 Administrator Staff Member

    39,828
    8,788
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +13,526
    Local Time:
    7:27 AM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    Now that you brought to my attention the eventual deprecation of GeoIP database updates, I am moving up the planned GeoIP2 support for Centmin Mod. So I can say soon ;)

    GeoLite2 Free Downloadable Databases « MaxMind Developer Site

    But seems Nginx GeoIP2 nginx module hasn't been updated in 2yrs leev/ngx_http_geoip2_module ?

    And no activity much on issues leev/ngx_http_geoip2_module ?
     
    • Like Like x 1
  3. May 9, 2018 #3
    wmtech

    wmtech Active Member

    103
    28
    28
    Jul 22, 2017
    Ratings:
    +75
    Local Time:
    11:27 PM
    Yes, but the module seems to be the recommended way to upgrade by Nginx:
    NGINX Docs | GeoIP2

    You can even deactivate automatic update of the (old) databases now, as there won't be any newer databases provided by Maxmind any more.

    Thanks!
     
  4. May 9, 2018 #4
    eva2000

    eva2000 Administrator Staff Member

    39,828
    8,788
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +13,526
    Local Time:
    7:27 AM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    from that notice suggests while March 2018 is latest update, updates will continue to happen until Jan 2019 ? Or read that wrongly ?
     
  5. May 9, 2018 #5
    eva2000

    eva2000 Administrator Staff Member

    39,828
    8,788
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +13,526
    Local Time:
    7:27 AM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    Hmm GeoIP2 Lite city database download doesn't seem to provide city related data fields ? But the GeoIP2 Lite nginx module's 4yr old instructions outline setting up city names leev/ngx_http_geoip2_module ? Seems City data is only available in paid GeoIP2 database and not GeoIP2 Lite database data GeoIP2 Release Notes « MaxMind Developer Site ?

    City database
    Code (Text):
    mmdblookup --file /usr/share/GeoIP/GeoLite2-City.mmdb --ip 8.8.8.8                                     

  {
    "continent":
      {
        "code":
          "NA" <utf8_string>
        "geoname_id":
          6255149 <uint32>
        "names":
          {
            "de":
              "Nordamerika" <utf8_string>
            "en":
              "North America" <utf8_string>
            "es":
              "Norteamérica" <utf8_string>
            "fr":
              "Amérique du Nord" <utf8_string>
            "ja":
              "北アメリカ" <utf8_string>
            "pt-BR":
              "América do Norte" <utf8_string>
            "ru":
              "Северная Америка" <utf8_string>
            "zh-CN":
              "北美洲" <utf8_string>
          }
      }
    "country":
      {
        "geoname_id":
          6252001 <uint32>
        "iso_code":
          "US" <utf8_string>
        "names":
          {
            "de":
              "USA" <utf8_string>
            "en":
              "United States" <utf8_string>
            "es":
              "Estados Unidos" <utf8_string>
            "fr":
              "États-Unis" <utf8_string>
            "ja":
              "アメリカ合衆国" <utf8_string>
            "pt-BR":
              "Estados Unidos" <utf8_string>
            "ru":
              "США" <utf8_string>
            "zh-CN":
              "美国" <utf8_string>
          }
      }
    "location":
      {
        "accuracy_radius":
          1000 <uint16>
        "latitude":
          37.751000 <double>
        "longitude":
          -97.822000 <double>
      }
    "registered_country":
      {
        "geoname_id":
          6252001 <uint32>
        "iso_code":
          "US" <utf8_string>
        "names":
          {
            "de":
              "USA" <utf8_string>
            "en":
              "United States" <utf8_string>
            "es":
              "Estados Unidos" <utf8_string>
            "fr":
              "États-Unis" <utf8_string>
            "ja":
              "アメリカ合衆国" <utf8_string>
            "pt-BR":
              "Estados Unidos" <utf8_string>
            "ru":
              "США" <utf8_string>
            "zh-CN":
              "美国" <utf8_string>
          }
      }
  }


    Country database
    Code (Text):
    mmdblookup --file /usr/share/GeoIP/GeoLite2-Country.mmdb --ip 8.8.8.8                 

  {
    "continent": 
      {
        "code": 
          "NA" <utf8_string>
        "geoname_id": 
          6255149 <uint32>
        "names": 
          {
            "de": 
              "Nordamerika" <utf8_string>
            "en": 
              "North America" <utf8_string>
            "es": 
              "Norteamérica" <utf8_string>
            "fr": 
              "Amérique du Nord" <utf8_string>
            "ja": 
              "北アメリカ" <utf8_string>
            "pt-BR": 
              "América do Norte" <utf8_string>
            "ru": 
              "Северная Америка" <utf8_string>
            "zh-CN": 
              "北美洲" <utf8_string>
          }
      }
    "country": 
      {
        "geoname_id": 
          6252001 <uint32>
        "iso_code": 
          "US" <utf8_string>
        "names": 
          {
            "de": 
              "USA" <utf8_string>
            "en": 
              "United States" <utf8_string>
            "es": 
              "Estados Unidos" <utf8_string>
            "fr": 
              "États-Unis" <utf8_string>
            "ja": 
              "アメリカ合衆国" <utf8_string>
            "pt-BR": 
              "Estados Unidos" <utf8_string>
            "ru": 
              "США" <utf8_string>
            "zh-CN": 
              "美国" <utf8_string>
          }
      }
    "registered_country": 
      {
        "geoname_id": 
          6252001 <uint32>
        "iso_code": 
          "US" <utf8_string>
        "names": 
          {
            "de": 
              "USA" <utf8_string>
            "en": 
              "United States" <utf8_string>
            "es": 
              "Estados Unidos" <utf8_string>
            "fr": 
              "États-Unis" <utf8_string>
            "ja": 
              "アメリカ合衆国" <utf8_string>
            "pt-BR": 
              "Estados Unidos" <utf8_string>
            "ru": 
              "США" <utf8_string>
            "zh-CN": 
              "美国" <utf8_string>
          }
      }
  }
     
  6. May 9, 2018 #6
    eva2000

    eva2000 Administrator Staff Member

    39,828
    8,788
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +13,526
    Local Time:
    7:27 AM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    • Like Like x 1
  7. May 10, 2018 #7
    wmtech

    wmtech Active Member

    103
    28
    28
    Jul 22, 2017
    Ratings:
    +75
    Local Time:
    11:27 PM
    The last update for the free download happened back in March 2018. No new updates until Jan 2019 when even the download will disappear. So updating it is unnecessary from now on.

    Updated legacy databases are provided for paying customers only between March 2018 and Jan 2019. It is a free add-on if you purchase the GeoIp2 databases.

    No city data in the CityLite GeoIP2 databases seems wrong. Will ask Maxmind about that.

    Thanks for your work!
     
  8. May 10, 2018 #8
    wmtech

    wmtech Active Member

    103
    28
    28
    Jul 22, 2017
    Ratings:
    +75
    Local Time:
    11:27 PM
    I downloaded the .csv version of the GeoIp2 CityLite database to take a look at the data and the city and region names are there.

    Code (Text):
    geoname_id,locale_code,continent_code,continent_name,country_iso_code,country_name,subdivision_1_iso_code,subdivision_1_name,subdivision_2_iso_code,subdivision_2_name,city_name,metro_code,time_zone,is_in_european_union
....
4832356,en,NA,"North America",US,"United States",ME,Maine,,,Berwick,500,America/New_York,0
4832366,en,NA,"North America",US,"United States",ME,Maine,,,Guilford,537,America/New_York,0
4832408,en,NA,"North America",US,"United States",ME,Maine,,,Hope,500,America/New_York,0
4832411,en,NA,"North America",US,"United States",CT,Connecticut,,,Danielson,533,America/New_York,0
4832425,en,NA,"North America",US,"United States",CT,Connecticut,,,Darien,501,America/New_York,0
4832458,en,NA,"North America",US,"United States",ME,Maine,,,Rockland,500,America/New_York,0
4832484,en,NA,"North America",US,"United States",ME,Maine,,,Union,500,America/New_York,0
4832498,en,NA,"North America",US,"United States",MI,Michigan,,,Champion,553,America/Detroit,0
....


    So there seems to be a problem in your test server either with the mmdb database or the mmdblookup tool. Did you check if the PHP vars are set correctly?

    The following data is available (more compared to legacy database):
    • geoname_id
    • locale_code
    • continent_code
    • continent_name
    • country_iso_code
    • country_name
    • subdivision_1_iso_code
    • subdivision_1_name
    • city_name
    • metro_code
    • time_zone
    • is_in_european_union
    Would be perfect to have all those data available in PHP vars. ;)
     
    Last edited: May 10, 2018
  9. May 10, 2018 #9
    eva2000

    eva2000 Administrator Staff Member

    39,828
    8,788
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +13,526
    Local Time:
    7:27 AM
    Nginx 1.15.x
    MariaDB 5.5/10.x
  10. May 10, 2018 #10
    wmtech

    wmtech Active Member

    103
    28
    28
    Jul 22, 2017
    Ratings:
    +75
    Local Time:
    11:27 PM
  11. May 10, 2018 #11
    eva2000

    eva2000 Administrator Staff Member

    39,828
    8,788
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +13,526
    Local Time:
    7:27 AM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    Ah i see the problem is with the ip 8.8.8.8 and not GeoIP2 Lite database - well sort of. The 8.8.8.8 ip doesn't have city data associated with it in GeoIP2 Lite database ! If i try a different IP, the city data is there though still not as accurate as old legacy GeoIP database it seems

    Also geo location is way off 1.1.1.1 test was ran against Los Angeles VPS and it returning Victoria, Australia location !

    test against 1.1.1.1
    Code (Text):
    mmdblookup --file /usr/share/GeoIP/GeoLite2-City.mmdb --ip 1.1.1.1 city names en

  "Research" <utf8_string>

    Code (Text):
    mmdblookup --file /usr/share/GeoIP/GeoLite2-City.mmdb --ip 1.1.1.1      

  {
    "city":
      {
        "geoname_id":
          2151718 <uint32>
        "names":
          {
            "en":
              "Research" <utf8_string>
          }
      }
    "continent":
      {
        "code":
          "OC" <utf8_string>
        "geoname_id":
          6255151 <uint32>
        "names":
          {
            "de":
              "Ozeanien" <utf8_string>
            "en":
              "Oceania" <utf8_string>
            "es":
              "Oceanía" <utf8_string>
            "fr":
              "Océanie" <utf8_string>
            "ja":
              "オセアニア" <utf8_string>
            "pt-BR":
              "Oceania" <utf8_string>
            "ru":
              "Океания" <utf8_string>
            "zh-CN":
              "大洋洲" <utf8_string>
          }
      }
    "country":
      {
        "geoname_id":
          2077456 <uint32>
        "iso_code":
          "AU" <utf8_string>
        "names":
          {
            "de":
              "Australien" <utf8_string>
            "en":
              "Australia" <utf8_string>
            "es":
              "Australia" <utf8_string>
            "fr":
              "Australie" <utf8_string>
            "ja":
              "オーストラリア" <utf8_string>
            "pt-BR":
              "Austrália" <utf8_string>
            "ru":
              "Австралия" <utf8_string>
            "zh-CN":
              "澳大利亚" <utf8_string>
          }
      }
    "location":
      {
        "accuracy_radius":
          1000 <uint16>
        "latitude":
          -37.700000 <double>
        "longitude":
          145.183300 <double>
        "time_zone":
          "Australia/Melbourne" <utf8_string>
      }
    "postal":
      {
        "code":
          "3095" <utf8_string>
      }
    "registered_country":
      {
        "geoname_id":
          2077456 <uint32>
        "iso_code":
          "AU" <utf8_string>
        "names":
          {
            "de":
              "Australien" <utf8_string>
            "en":
              "Australia" <utf8_string>
            "es":
              "Australia" <utf8_string>
            "fr":
              "Australie" <utf8_string>
            "ja":
              "オーストラリア" <utf8_string>
            "pt-BR":
              "Austrália" <utf8_string>
            "ru":
              "Австралия" <utf8_string>
            "zh-CN":
              "澳大利亚" <utf8_string>
          }
      }
    "subdivisions":
      [
        {
          "geoname_id":
            2145234 <uint32>
          "iso_code":
            "VIC" <utf8_string>
          "names":
            {
              "en":
                "Victoria" <utf8_string>
              "pt-BR":
                "Vitória" <utf8_string>
              "ru":
                "Виктория" <utf8_string>
            }
        }
      ]
  }
     
  12. May 10, 2018 #12
    eva2000

    eva2000 Administrator Staff Member

    39,828
    8,788
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +13,526
    Local Time:
    7:27 AM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    hmm seems 1.1.1.1 is Victoria, Australia !
    Code (Text):
    curl -s https://ipinfo.io/1.1.1.1                                 
{
  "ip": "1.1.1.1",
  "hostname": "1dot1dot1dot1.cloudflare-dns.com",
  "city": "Research",
  "region": "Victoria",
  "country": "AU",
  "loc": "-37.7000,145.1830",
  "postal": "3095",
  "org": "AS13335 Cloudflare, Inc."
     
  13. May 10, 2018 #13
    wmtech

    wmtech Active Member

    103
    28
    28
    Jul 22, 2017
    Ratings:
    +75
    Local Time:
    11:27 PM
    1.1.1.1 is also a global DNS address. Just try one of your server addresses or your broadband address. There should be the city name, the time zone and "is_in_european_union" also...
     
  14. May 10, 2018 #14
    eva2000

    eva2000 Administrator Staff Member

    39,828
    8,788
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +13,526
    Local Time:
    7:27 AM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    Yes it is but i ran the command on my Los Angeles VPS so no where near Australia hehe

    If i ran against my Los Angeles VPS IP address it returns correctly as Los Angeles.

    This seems a bit trickier if I want to keep both GeoIP legacy and GeoIP2 Lite variables in PHP setup. Will have to think about how to do that
     
  15. May 10, 2018 #15
    eva2000

    eva2000 Administrator Staff Member

    39,828
    8,788
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +13,526
    Local Time:
    7:27 AM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    @wmtech do you know where i can get a list of all the $geoip2_data variables for fastcgi_param ?

    only managed to find these
    Code (Text):
    fastcgi_param GEOIP2_COUNTRY_CODE $geoip2_data_country_code;
fastcgi_param GEOIP2_COUNTRY $geoip2_data_country_name;
fastcgi_param GEOIP2_CITY $geoip2_data_city_name;
fastcgi_param GEOIP2_POSTAL_CODE $geoip2_data_postal_code;


    when set in /usr/local/nginx/conf/php.conf becomes
    Code (Text):
    # Set php-fpm geoip variables
fastcgi_param GEOIP_COUNTRY_CODE $geoip_country_code;
fastcgi_param GEOIP_COUNTRY_CODE3 $geoip_country_code3;
fastcgi_param GEOIP_COUNTRY_NAME $geoip_country_name;
fastcgi_param GEOIP_CITY_COUNTRY_CODE $geoip_city_country_code;
fastcgi_param GEOIP_CITY_COUNTRY_CODE3 $geoip_city_country_code3;
fastcgi_param GEOIP_CITY_COUNTRY_NAME $geoip_city_country_name;
fastcgi_param GEOIP_REGION $geoip_region;
fastcgi_param GEOIP_CITY $geoip_city;
fastcgi_param GEOIP_POSTAL_CODE $geoip_postal_code;
fastcgi_param GEOIP_CITY_CONTINENT_CODE $geoip_city_continent_code;
fastcgi_param GEOIP_LATITUDE $geoip_latitude;
fastcgi_param GEOIP_LONGITUDE $geoip_longitude;

fastcgi_param GEOIP2_COUNTRY_CODE $geoip2_data_country_code;
fastcgi_param GEOIP2_COUNTRY $geoip2_data_country_name;
fastcgi_param GEOIP2_CITY $geoip2_data_city_name;
fastcgi_param GEOIP2_POSTAL_CODE $geoip2_data_postal_code;


    upload_2018-5-10_1-29-31.png
     
  16. May 10, 2018 #16
    wmtech

    wmtech Active Member

    103
    28
    28
    Jul 22, 2017
    Ratings:
    +75
    Local Time:
    11:27 PM
    Best would be to start GeoIp2 vars with GEOIP2_ and separate them from the regular GEOIP_ vars.

    So one can use the vars he likes. Legacy GeoIp will also not stop to be working as long as you don't delete the database file. It just will show old data.

    So everyone can start using GeoIp2 in scripts as liked.
     
  17. May 10, 2018 #17
    eva2000

    eva2000 Administrator Staff Member

    39,828
    8,788
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +13,526
    Local Time:
    7:27 AM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    yeah that's what i end up doing in previous post just need all the available variables :)
     
  18. May 10, 2018 #18
    wmtech

    wmtech Active Member

    103
    28
    28
    Jul 22, 2017
    Ratings:
    +75
    Local Time:
    11:27 PM
    You find them in the JSON returned from the mmdblookup tool. You just need to find an IP address where all data is returned (not all addresses show all data fields). I was successful with my local broadband address.

    For example from the city database:

    Code (Text):
    [15:37][[email protected] GeoIP]# mmdblookup --file /usr/share/GeoIP/GeoLite2-City.mmdb --ip 174.201.19.xx

  {
    "city":
      {
        "geoname_id":
          5178225 <uint32>
        "names":
          {
            "en":
              "Ambler" <utf8_string>
          }
      }
    "continent":
      {
        "code":
          "NA" <utf8_string>
        "geoname_id":
          6255149 <uint32>
        "names":
          {
            "de":
              "Nordamerika" <utf8_string>
            "en":
              "North America" <utf8_string>
            "es":
              "Norteamérica" <utf8_string>
            "fr":
              "Amérique du Nord" <utf8_string>
            "ja":
              "北アメリカ" <utf8_string>
            "pt-BR":
              "América do Norte" <utf8_string>
            "ru":
              "Северная Америка" <utf8_string>
            "zh-CN":
              "北美洲" <utf8_string>
          }
      }
    "country":
      {
        "geoname_id":
          6252001 <uint32>
        "iso_code":
          "US" <utf8_string>
        "names":
          {
            "de":
              "USA" <utf8_string>
            "en":
              "United States" <utf8_string>
            "es":
              "Estados Unidos" <utf8_string>
            "fr":
              "États-Unis" <utf8_string>
            "ja":
              "アメリカ合衆国" <utf8_string>
            "pt-BR":
              "Estados Unidos" <utf8_string>
            "ru":
              "США" <utf8_string>
            "zh-CN":
              "美国" <utf8_string>
          }
      }
    "location":
      {
        "accuracy_radius":
          100 <uint16>
        "latitude":
          40.184300 <double>
        "longitude":
          -75.216800 <double>
        "metro_code":
          504 <uint16>
        "time_zone":
          "America/New_York" <utf8_string>
      }
    "postal":
      {
        "code":
          "19002" <utf8_string>
      }
    "registered_country":
      {
        "geoname_id":
          6252001 <uint32>
        "iso_code":
          "US" <utf8_string>
        "names":
          {
            "de":
              "USA" <utf8_string>
            "en":
              "United States" <utf8_string>
            "es":
              "Estados Unidos" <utf8_string>
            "fr":
              "États-Unis" <utf8_string>
            "ja":
              "アメリカ合衆国" <utf8_string>
            "pt-BR":
              "Estados Unidos" <utf8_string>
            "ru":
              "США" <utf8_string>
            "zh-CN":
              "美国" <utf8_string>
          }
      }
    "subdivisions":
      [
        {
          "geoname_id":
            6254927 <uint32>
          "iso_code":
            "PA" <utf8_string>
          "names":
            {
              "en":
                "Pennsylvania" <utf8_string>
              "es":
                "Pensilvania" <utf8_string>
              "fr":
                "Pennsylvanie" <utf8_string>
              "ja":
                "ペンシルベニア州" <utf8_string>
              "pt-BR":
                "Pensilvânia" <utf8_string>
              "ru":
                "Пенсильвания" <utf8_string>
              "zh-CN":
                "宾夕法尼亚州" <utf8_string>
            }
        }
      ]
  }


    So you can get all those fields with the following code in NGINX:

    Code:
    geoip2 /usr/share/GeoIP/GeoLite2-City.mmdb {
   $geoip2_data_city_name default=London city names en;
   $geoip2_data_city_geonameid city geoname_id
   $geoip2_data_continent_code continent code
   $geoip2_data_continent_geonameid continent geoname_id
   $geoip2_data_continent_name continent names en
   $geoip2_data_country_geonameid country geoname_id
   $geoip2_data_country_iso country iso_code
   $geoip2_data_country_name country names en
   $geoip2_data_location_accuracyradius location accuracy_radius
   $geoip2_data_location_latitude location latitude
   $geoip2_data_location_longitude location longitude
   $geoip2_data_location_metrocode location metro_code
   $geoip2_data_location_timezone location time_zone
   $geoip2_data_postal_code postal code
   $geoip2_data_rcountry_geonameid registered_country geoname_id
   $geoip2_data_rcountry_iso registered_country iso_code
   $geoip2_data_rcountry_name registered_country names en
   $geoip2_data_region_geonameid subdivisions 0 geoname_id
   $geoip2_data_region_iso subdivisions 0 iso_code
   $geoip2_data_region_name subdivisions 0 names en
}
    The same can be done for the country database too. :)
     
    Last edited: May 10, 2018
  19. May 10, 2018 #19
    eva2000

    eva2000 Administrator Staff Member

    39,828
    8,788
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +13,526
    Local Time:
    7:27 AM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    ah I see i need to map nginx $geoip2_data_* variables to database fields before I can use them in fastcgi_params :)
     
    • Agree Agree x 1
  20. May 10, 2018 #20
    eva2000

    eva2000 Administrator Staff Member

    39,828
    8,788
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +13,526
    Local Time:
    7:27 AM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    Ok shaping up nicely :)

    upload_2018-5-10_2-22-31.png
     
    • Like Like x 1
Previous Thread Next Thread
Loading...
Page 1 of 2
..

.