Join the community today
Register Now

Beta Branch prep work for GeoIP2 Lite nginx module support 123.09beta01

Discussion in 'Centmin Mod Github Commits' started by eva2000, May 9, 2018.

  1. eva2000

    eva2000 Administrator Staff Member

    40,654
    9,032
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +13,905
    Local Time:
    4:55 AM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    prep work for GeoIP2 Lite nginx module support 123.09beta01

    - prep work for MaxMind GeoIP2 Lite database support via nginx GeoIP2 nginx module leev/ngx_http_geoip2_module due to eventual deprecation and shutdown of legacy GeoIP database updates and downloads after January 19th, 2019 GeoLite Legacy Downloadable Databases « MaxMind Developer Site
    - update adds support for downloading MaxMind GeoIP2 Lite databases for country and city downloads GeoLite2 Free Downloadable Databases « MaxMind Developer Site while still also downloading the legacy GeoIP databases from GeoLite Legacy Downloadable Databases « MaxMind Developer Site. Updated routine will only overwrite existing legacy GeoIP databases if a precheck of the download urls are considered working = HTTP 200 status code. When MaxMind eventually discontinues the download ability of legacy GeoIP database, the existing downloaded database will remain intact just there won't be any more updates from MaxMind and need to use the GeoIP2 Lite database instead
    - fresh Centmin Mod 123.09beta01 installs will have updated entries in /usr/local/nginx/conf/geoip.conf which are commented out by default for GeoIP2 Lite configuration. One will define the geoip2 /usr/share/GeoIP/GeoLite2-City.mmdb context and other will define the geoip2 /usr/share/GeoIP/GeoLite2-Country.mmdb context wrapped in {}. For existing Centmin Mod installs, you would need to add the missing configuration settings into /usr/local/nginx/conf/geoip.conf as per contents shown at centminmod/centminmod as end users may have customised their /usr/local/nginx/conf/geoip.conf file and overwriting the contents of end users customised file would not be a good idea.
    - to enable nginx GeoIP2 module support in Centmin Mod 123.09beta01 Nginx builds, need to add 2 variables into persistent config file at /etc/centminmod/custom_config.inc before recompiling nginx via centmin.sh menu option 4
    Code (Text):
    NGINX_GEOIPTWOLITE='y'
    NGXDYNAMIC_GEOIPTWOLITE='y'
    


    Continue reading...

    123.09beta01 branch
     
  2. eva2000

    eva2000 Administrator Staff Member

    40,654
    9,032
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +13,905
    Local Time:
    4:55 AM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    Example

    dynamic modules include file /usr/local/nginx/conf/dynamic-modules.conf
    Code (Text):
    # place custom load_module lines in this dynamic-modules-includes.conf
    # file so that they persistent i.e. for manually dropped in dynamic modules
    include /usr/local/nginx/conf/dynamic-modules-includes.conf;
    load_module "modules/ngx_http_image_filter_module.so";
    load_module "modules/ngx_http_headers_more_filter_module.so";
    load_module "modules/ngx_http_set_misc_module.so";
    load_module "modules/ngx_http_echo_module.so";
    load_module "modules/ngx_http_fancyindex_module.so";
    load_module "modules/ngx_http_vhost_traffic_status_module.so";
    load_module "modules/ngx_pagespeed.so";
    load_module "modules/ngx_http_brotli_filter_module.so";
    load_module "modules/ngx_http_brotli_static_module.so";
    load_module "modules/ngx_stream_module.so";
    load_module "modules/ngx_http_geoip2_module.so";
    

    dynamic module directory at /usr/local/nginx/modules
    Code (Text):
    -rwxr-xr-x  1 root root  20K May  9 12:56 ngx_http_brotli_filter_module.so
    -rwxr-xr-x  1 root root  14K May  9 12:56 ngx_http_brotli_static_module.so
    -rwxr-xr-x  1 root root  63K May  9 12:56 ngx_http_echo_module.so
    -rwxr-xr-x  1 root root 150K May  9 12:49 ngx_http_echo_module.so.old
    -rwxr-xr-x  1 root root  29K May  9 12:56 ngx_http_fancyindex_module.so
    -rwxr-xr-x  1 root root  47K May  9 12:49 ngx_http_fancyindex_module.so.old
    -rwxr-xr-x  1 root root  19K May  9 12:56 ngx_http_geoip2_module.so
    -rwxr-xr-x  1 root root  32K May  9 12:49 ngx_http_geoip2_module.so.old
    -rwxr-xr-x  1 root root  42K May  9 12:56 ngx_http_headers_more_filter_module.so
    -rwxr-xr-x  1 root root  76K May  9 12:49 ngx_http_headers_more_filter_module.so.old
    -rwxr-xr-x  1 root root  28K May  9 12:56 ngx_http_image_filter_module.so
    -rwxr-xr-x  1 root root  49K May  9 12:49 ngx_http_image_filter_module.so.old
    -rwxr-xr-x  1 root root  40K May  9 12:56 ngx_http_set_misc_module.so
    -rwxr-xr-x  1 root root 155K May  9 12:49 ngx_http_set_misc_module.so.old
    -rwxr-xr-x  1 root root 234K May  9 12:56 ngx_http_vhost_traffic_status_module.so
    -rwxr-xr-x  1 root root  15M May  9 12:56 ngx_pagespeed.so
    -rwxr-xr-x  1 root root  18K May  9 12:56 ngx_stream_geoip2_module.so
    -rwxr-xr-x  1 root root  24K May  9 12:49 ngx_stream_geoip2_module.so.old
    -rwxr-xr-x  1 root root 198K May  9 12:56 ngx_stream_module.so
    -rwxr-xr-x  1 root root 382K May  9 12:49 ngx_stream_module.so.old
    

    contents of /usr/local/nginx/conf/geoip.conf with geoip2 configs uncommented without hash in front
    Code (Text):
    # SET the path to the .dat file used for determining the visitors country from the IP-address ###
    geoip_country /usr/share/GeoIP/GeoIP.dat;
    
    # SET the path to the .dat file used for determining the visitors country from the IP-address ###
    geoip_city /usr/share/GeoIP/GeoIPCity.dat;
    
    # GeoIP2 Lite databases
      geoip2 /usr/share/GeoIP/GeoLite2-Country.mmdb {
        $geoip2_data_country_code country iso_code;
        $geoip2_data_country_name country names en;
      }
    
      geoip2 /usr/share/GeoIP/GeoLite2-City.mmdb {
        $geoip2_data_city_name city names en;
        $geoip2_data_postal_code postal code;
        $geoip2_data_latitude location latitude;
        $geoip2_data_longitude location longitude;
        $geoip2_data_state_name subdivisions 0 names en;
        $geoip2_data_state_code subdivisions 0 iso_code;
      }
    


    GeoIP2 nginx module compiled and loaded as a nginx dynamic module
    Code (Text):
    lsof | grep geoip2
    nginx       381            root  mem       REG         182,160625      18480   2243997 /usr/local/nginx/modules/ngx_http_geoip2_module.so
    nginx       382           nginx  mem       REG         182,160625      18480   2243997 /usr/local/nginx/modules/ngx_http_geoip2_module.so
    scheduler   382   396     nginx  mem       REG         182,160625      18480   2243997 /usr/local/nginx/modules/ngx_http_geoip2_module.so
    nginx       383           nginx  mem       REG         182,160625      18480   2243997 /usr/local/nginx/modules/ngx_http_geoip2_module.so
    scheduler   383   389     nginx  mem       REG         182,160625      18480   2243997 /usr/local/nginx/modules/ngx_http_geoip2_module.so
    
     
  3. eva2000

    eva2000 Administrator Staff Member

    40,654
    9,032
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +13,905
    Local Time:
    4:55 AM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    Using libmaxmindb tool test against IP 8.8.8.8
    Code (Text):
    /usr/local/bin/mmdblookup --file /usr/share/GeoIP/GeoLite2-Country.mmdb --ip 8.8.8.8 country names en
    
      "United States" <utf8_string>
    

    Code (Text):
    /usr/local/bin/mmdblookup --file /usr/share/GeoIP/GeoLite2-Country.mmdb --ip 8.8.8.8 country iso_code   
    
      "US" <utf8_string>
    

    though GeoIP2 Lite City database doesn't have field for city name ?
    Code (Text):
    mmdblookup --file /usr/share/GeoIP/GeoLite2-City.mmdb --ip 8.8.8.8 
    
      {
        "continent": 
          {
            "code": 
              "NA" <utf8_string>
            "geoname_id": 
              6255149 <uint32>
            "names": 
              {
                "de": 
                  "Nordamerika" <utf8_string>
                "en": 
                  "North America" <utf8_string>
                "es": 
                  "Norteamérica" <utf8_string>
                "fr": 
                  "Amérique du Nord" <utf8_string>
                "ja": 
                  "北アメリカ" <utf8_string>
                "pt-BR": 
                  "América do Norte" <utf8_string>
                "ru": 
                  "Северная Америка" <utf8_string>
                "zh-CN": 
                  "北美洲" <utf8_string>
              }
          }
        "country": 
          {
            "geoname_id": 
              6252001 <uint32>
            "iso_code": 
              "US" <utf8_string>
            "names": 
              {
                "de": 
                  "USA" <utf8_string>
                "en": 
                  "United States" <utf8_string>
                "es": 
                  "Estados Unidos" <utf8_string>
                "fr": 
                  "États-Unis" <utf8_string>
                "ja": 
                  "アメリカ合衆国" <utf8_string>
                "pt-BR": 
                  "Estados Unidos" <utf8_string>
                "ru": 
                  "США" <utf8_string>
                "zh-CN": 
                  "美国" <utf8_string>
              }
          }
        "location": 
          {
            "accuracy_radius": 
              1000 <uint16>
            "latitude": 
              37.751000 <double>
            "longitude": 
              -97.822000 <double>
          }
        "registered_country": 
          {
            "geoname_id": 
              6252001 <uint32>
            "iso_code": 
              "US" <utf8_string>
            "names": 
              {
                "de": 
                  "USA" <utf8_string>
                "en": 
                  "United States" <utf8_string>
                "es": 
                  "Estados Unidos" <utf8_string>
                "fr": 
                  "États-Unis" <utf8_string>
                "ja": 
                  "アメリカ合衆国" <utf8_string>
                "pt-BR": 
                  "Estados Unidos" <utf8_string>
                "ru": 
                  "США" <utf8_string>
                "zh-CN": 
                  "美国" <utf8_string>
              }
          }
      }
    
     
    style="display:inline-block;min-width:400px;max-width:970px;width:95%;height:90px" data-ad-client="ca-pub-6669518204467592" data-ad-slot="4024536743" data-ad-format="auto">
  4. eva2000

    eva2000 Administrator Staff Member

    40,654
    9,032
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +13,905
    Local Time:
    4:55 AM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    Specifics GeoLite Legacy Discontinuation Notice | MaxMind Support Center

     
  5. eva2000

    eva2000 Administrator Staff Member

    40,654
    9,032
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +13,905
    Local Time:
    4:55 AM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    Latest result working with GeoIP2 Lite and PHP-FPM fastcgi_param passed variables for PHP. Added variable to detect if IP address is in European Union (EU)

    centminmod-nginx-php-fpm-geoip2-02c.png
     
  6. eva2000

    eva2000 Administrator Staff Member

    40,654
    9,032
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +13,905
    Local Time:
    4:55 AM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    True to their word, maxmind removed GeoIP legacy database downloads after January 2, 2019 so GeoIP2 lite database support needs renewed attention :)
    For Centmin Mod use at least I have uploaded a copy of the old maxmind GeoIP legacy databases here. Though they won't have up to date data entries since maxmind no longer updates them.
     
  7. wmtech

    wmtech Active Member

    111
    30
    28
    Jul 22, 2017
    Ratings:
    +83
    Local Time:
    8:55 PM
    Those databases have not been updated for a very long time (1 year). Everybody should have switched to the new databases by now.

    But thanks for providing a download option for very late adopters. ;)
     
    Last edited: Jan 10, 2019
    • Agree Agree x 1
  8. rdan

    rdan Well-Known Member

    4,549
    1,090
    113
    May 25, 2014
    Ratings:
    +1,591
    Local Time:
    2:55 AM
    Mainline
    10.2
    Which database?
     
  9. wmtech

    wmtech Active Member

    111
    30
    28
    Jul 22, 2017
    Ratings:
    +83
    Local Time:
    8:55 PM
    GeoIP2 Lite
     
    • Like Like x 1
  10. eva2000

    eva2000 Administrator Staff Member

    40,654
    9,032
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +13,905
    Local Time:
    4:55 AM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    Yeah tricky part is for nginx / php-fpm and geoip/geoip2 usage is the PHP and nginx variables aren't 100% match between them - so switching between the 2 can break web app/php and nginx's usage of GeoIP. I'll have to do more testing to make sure such a switch closely matches the variables that web app scripts/php expect. While some web app/php scripts don't even support GeoIP2 databases - only GeoIP legacy.
     
    • Agree Agree x 1
  11. eva2000

    eva2000 Administrator Staff Member

    40,654
    9,032
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +13,905
    Local Time:
    4:55 AM
    Nginx 1.15.x
    MariaDB 5.5/10.x
    Checking to see if geoip2 nginx module works with latest Nginx 1.15.10 - built as a dynamic nginx module
    include file /usr/local/nginx/conf/geoip2.conf
    Code (Text):
    # GeoIP2 Lite databases
      #  geoip2 /usr/share/GeoIP/GeoLite2-Country.mmdb {
      #    $geoip2_data_country_code country iso_code;
      #    $geoip2_data_country_name country names en;
      #  }
    
      geoip2 /usr/share/GeoIP/GeoLite2-City.mmdb {
        $geoip2_data_city_name city names en;
        $geoip2_data_city_geonameid city geoname_id;
        $geoip2_data_continent_code continent code;
        $geoip2_data_continent_geonameid continent geoname_id;
        $geoip2_data_continent_name continent names en;
        $geoip2_data_country_geonameid country geoname_id;
        $geoip2_data_country_iso country iso_code;
        $geoip2_data_country_name country names en;
        $geoip2_data_country_is_eu country is_in_european_union;
        $geoip2_data_location_accuracyradius location accuracy_radius;
        $geoip2_data_location_latitude location latitude;
        $geoip2_data_location_longitude location longitude;
        $geoip2_data_location_metrocode location metro_code;
        $geoip2_data_location_timezone location time_zone;
        $geoip2_data_postal_code postal code;
        $geoip2_data_rcountry_geonameid registered_country geoname_id;
        $geoip2_data_rcountry_iso registered_country iso_code;
        $geoip2_data_rcountry_name registered_country names en;
        $geoip2_data_rcountry_is_eu registered_country is_in_european_union;
        $geoip2_data_region_geonameid subdivisions 0 geoname_id;
        $geoip2_data_region_iso subdivisions 0 iso_code;
        $geoip2_data_region_name subdivisions 0 names en;
      }
    

    manual lookup
    Code (Text):
    mmdblookup --file /usr/share/GeoIP/GeoLite2-City.mmdb --ip 8.8.8.8 
    
      {
        "continent": 
          {
            "code": 
              "NA" <utf8_string>
            "geoname_id": 
              6255149 <uint32>
            "names": 
              {
                "de": 
                  "Nordamerika" <utf8_string>
                "en": 
                  "North America" <utf8_string>
                "es": 
                  "Norteamérica" <utf8_string>
                "fr": 
                  "Amérique du Nord" <utf8_string>
                "ja": 
                  "北アメリカ" <utf8_string>
                "pt-BR": 
                  "América do Norte" <utf8_string>
                "ru": 
                  "Северная Америка" <utf8_string>
                "zh-CN": 
                  "北美洲" <utf8_string>
              }
          }
        "country": 
          {
            "geoname_id": 
              6252001 <uint32>
            "iso_code": 
              "US" <utf8_string>
            "names": 
              {
                "de": 
                  "USA" <utf8_string>
                "en": 
                  "United States" <utf8_string>
                "es": 
                  "Estados Unidos" <utf8_string>
                "fr": 
                  "États-Unis" <utf8_string>
                "ja": 
                  "アメリカ合衆国" <utf8_string>
                "pt-BR": 
                  "Estados Unidos" <utf8_string>
                "ru": 
                  "США" <utf8_string>
                "zh-CN": 
                  "美国" <utf8_string>
              }
          }
        "location": 
          {
            "accuracy_radius": 
              1000 <uint16>
            "latitude": 
              37.751000 <double>
            "longitude": 
              -97.822000 <double>
          }
        "registered_country": 
          {
            "geoname_id": 
              6252001 <uint32>
            "iso_code": 
              "US" <utf8_string>
            "names": 
              {
                "de": 
                  "USA" <utf8_string>
                "en": 
                  "United States" <utf8_string>
                "es": 
                  "Estados Unidos" <utf8_string>
                "fr": 
                  "États-Unis" <utf8_string>
                "ja": 
                  "アメリカ合衆国" <utf8_string>
                "pt-BR": 
                  "Estados Unidos" <utf8_string>
                "ru": 
                  "США" <utf8_string>
                "zh-CN": 
                  "美国" <utf8_string>
              }
          }
      }
    
     
    • Winner Winner x 1
  12. eva2000

    eva2000 Administrator Staff Member

    40,654
    9,032
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +13,905
    Local Time:
    4:55 AM
    Nginx 1.15.x
    MariaDB 5.5/10.x