Want more timely Centmin Mod News Updates?
Become a Member

Sysadmin AlmaLinux Elasticsearch log issue GeoIpDownloader Error

Discussion in 'System Administration' started by pamamolf, Mar 7, 2024.

  1. pamamolf

    pamamolf Well-Known Member

    4,049
    427
    83
    May 31, 2014
    Ratings:
    +826
    Local Time:
    2:28 PM
    Nginx-1.25.x
    MariaDB 10.3.x
    Hi,

    I am using the latest beta Centminmod beta on an Alma Linux server and sometimes i am getting some issues with Elastic search.

    Checking on the logs i can see:


    Code:
    [2024-03-06T13:10:15,531][ERROR][o.e.i.g.GeoIpDownloader  ] [server.domain.com] exception during geoip databases update
    org.elasticsearch.ElasticsearchException: not all primary shards of [.geoip_databases] index are active
        at org.elasticsearch.ingest.geoip.GeoIpDownloader.updateDatabases(GeoIpDownloader.java:137) ~[ingest-geoip-7.17.18.jar:7.17.18]
        at org.elasticsearch.ingest.geoip.GeoIpDownloader.runDownloader(GeoIpDownloader.java:284) [ingest-geoip-7.17.18.jar:7.17.18]
        at org.elasticsearch.ingest.geoip.GeoIpDownloaderTaskExecutor.nodeOperation(GeoIpDownloaderTaskExecutor.java:100) [ingest-geoip-7.17.18.jar:7.17.18]
        at org.elasticsearch.ingest.geoip.GeoIpDownloaderTaskExecutor.nodeOperation(GeoIpDownloaderTaskExecutor.java:46) [ingest-geoip-7.17.18.jar:7.17.18]
        at org.elasticsearch.persistent.NodePersistentTasksExecutor$1.doRun(NodePersistentTasksExecutor.java:42) [elasticsearch-7.17.18.jar:7.17.18]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:777) [elasticsearch-7.17.18.jar:7.17.18]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) [elasticsearch-7.17.18.jar:7.17.18]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [?:?]
        at java.lang.Thread.run(Thread.java:1583) [?:?]
    [2024-03-06T13:10:15,642][INFO ][o.e.i.g.DatabaseNodeService] [server.domain.com] retrieve geoip database [GeoLite2-Country.mmdb] from [.geoip_databases] to [/home/elasticsearch/geoip-databases/-yTaVssTTOq1QMEes0islw/GeoLite2-Country.mmdb.tmp.gz]
    [2024-03-06T13:10:15,643][INFO ][o.e.i.g.DatabaseNodeService] [server.domain.com] retrieve geoip database [GeoLite2-City.mmdb] from [.geoip_databases] to [/home/elasticsearch/geoip-databases/-yTaVssTTOq1QMEes0islw/GeoLite2-City.mmdb.tmp.gz]
    [2024-03-06T13:10:15,643][INFO ][o.e.i.g.DatabaseNodeService] [server.domain.com] retrieve geoip database [GeoLite2-ASN.mmdb] from [.geoip_databases] to [/home/elasticsearch/geoip-databases/-yTaVssTTOq1QMEes0islw/GeoLite2-ASN.mmdb.tmp.gz]
    [2024-03-06T13:10:15,839][INFO ][o.e.i.g.DatabaseNodeService] [server.domain.com] successfully reloaded changed geoip database file [/home/elasticsearch/geoip-databases/-yTaVssTTOq1QMEes0islw/GeoLite2-Country.mmdb]
    [2024-03-06T13:10:15,859][INFO ][o.e.i.g.DatabaseNodeService] [server.domain.com] successfully reloaded changed geoip database file [/home/elasticsearch/geoip-databases/-yTaVssTTOq1QMEes0islw/GeoLite2-ASN.mmdb]
    [2024-03-06T13:10:16,272][INFO ][o.e.i.g.DatabaseNodeService] [server.domain.com] successfully reloaded changed geoip database file [/home/elasticsearch/geoip-databases/-yTaVssTTOq1QMEes0islw/GeoLite2-City.mmdb]
    [2024-03-06T13:10:16,482][INFO ][o.e.c.r.a.AllocationService] [server.domain.com] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[content][0]]]).
    Elasticsearch version: 7.17.18

    Any ideas?

    Thanks in advance !
     
    Last edited: Mar 7, 2024
  2. eva2000

    eva2000 Administrator Staff Member

    51,726
    11,944
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,436
    Local Time:
    9:28 PM
    Nginx 1.25.x
    MariaDB 10.x
    AlmaLinux 8 or 9 ? How was Elasticsearch installed and configured? Have you checked if Elasticsearch has enough disk space? Did you try to manually configure GeoIP for Elasticsearch instead of letting it automatically handle it?

    Check if Elasticsearch connectivity to Maxmind servers to download GeoIP database is working and that no firewalls are in the way or if your server IP has been blocked. See GeoIP processor | Elasticsearch Guide [7.17] | Elastic

    1. Check ES server's health status is green

    Code (Text):
    curl -sX GET "http://localhost:9200/_cluster/health?pretty"
    


    2. If not green, check indices and shards for issues - specifically the .geoip_databases index health status

    Code (Text):
    curl -sX GET "http://localhost:9200/_cat/indices?v&pretty"
    


    3. Inspect shards for .geoip_databases index. If the shard(s) for this index are UNASSIGNED or in a state other than STARTED, it indicates an issue with shard allocation for this index.

    Code (Text):
    curl -sX GET "http://localhost:9200/_cat/shards/.geoip_databases?pretty"
    

    example out on working Elasticsearch server
    Code (Text):
    curl -sX GET "http://localhost:9200/_cat/shards/.geoip_databases?pretty"
    .geoip_databases 0 p STARTED 39 37mb 127.0.0.1 node22
    

    where
    • .geoip_databases: The name of the index, indicating it's the system index used by Elasticsearch to store GeoIP databases.
    • 0: The shard number. Since system indices like .geoip_databases typically don't require a lot of data, a single primary shard without replicas is common.
    • p: Indicates this shard is a primary shard.
    • STARTED: The status of the shard, indicating that it is active and operational. This is the state you want to see for a healthy shard.
    • 39: The unique document count in the shard.
    • 37mb: The size of the shard on disk, which is a reasonable size for the GeoIP databases.
    • 127.0.0.1: The IP address of the node on which this shard is located. Since it’s 127.0.0.1, this indicates the shard is hosted on the local machine.
    • node22: The name of the node hosting the shard, as specified in your Elasticsearch configuration.
    4. Use the Cluster Allocation Explain API to get more details on why the .geoip_databases shard is not allocated and can help identify the root cause, such as disk watermarks being exceeded, node connectivity issues, or other allocation problems.

    Code:
    curl -sX GET "http://localhost:9200/_cluster/allocation/explain?pretty" -H 'Content-Type: application/json' -d'
    {
      "index": ".geoip_databases",
      "shard": 0,
      "primary": true
    }'
    
     
  3. pamamolf

    pamamolf Well-Known Member

    4,049
    427
    83
    May 31, 2014
    Ratings:
    +826
    Local Time:
    2:28 PM
    Nginx-1.25.x
    MariaDB 10.3.x
    I disabled it and for the moment seems ok but i run the commands that you recommend anyway:

    Almalinux 8

    Elastic search installed using the official repo.

    So i got:

    curl -sX GET "http://localhost:9200/_cat/indices?v&pretty"

    Code:
    health status index   uuid                   pri rep docs.count docs.deleted store.size pri.store.size
    yellow open   content s1j0uwupRQ-3XmdLPHuqnA   1   1    3754715       919525      2.3gb          2.3gb
    curl -sX GET "http://localhost:9200/_cat/shards/.geoip_databases?pretty"

    Code:
    {
      "error" : {
        "root_cause" : [
          {
            "type" : "index_not_found_exception",
            "reason" : "no such index [.geoip_databases]",
            "resource.type" : "index_or_alias",
            "resource.id" : ".geoip_databases",
            "index_uuid" : "_na_",
            "index" : ".geoip_databases"
          }
        ],
        "type" : "index_not_found_exception",
        "reason" : "no such index [.geoip_databases]",
        "resource.type" : "index_or_alias",
        "resource.id" : ".geoip_databases",
        "index_uuid" : "_na_",
        "index" : ".geoip_databases"
      },
      "status" : 404
    }

    How can i check from where is coming the yellow status?

    Maybe is not related to Geoip....

    Thanks in advance !
     
  4. eva2000

    eva2000 Administrator Staff Member

    51,726
    11,944
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,436
    Local Time:
    9:28 PM
    Nginx 1.25.x
    MariaDB 10.x
    Just noticed that your error shows Elasticsearch is trying to download GeoIP2 databases to /home/elasticsearch. Does that directory exist? On my working Elasticsearch servers I see GeoIP2 databases downloaded to my Elasticsearch tmp directory which I configure with correct user/group permissions as part of Elasticsearch install/configs for myself and my clients.
     
  5. pamamolf

    pamamolf Well-Known Member

    4,049
    427
    83
    May 31, 2014
    Ratings:
    +826
    Local Time:
    2:28 PM
    Nginx-1.25.x
    MariaDB 10.3.x
    /home/elasticsearch exists

    Owner: elasticsearch [993]
    Group: elasticsearch [991]

    Octal: 0755
     
  6. eva2000

    eva2000 Administrator Staff Member

    51,726
    11,944
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +18,436
    Local Time:
    9:28 PM
    Nginx 1.25.x
    MariaDB 10.x
    hmm lots of ways to troubleshoot this. You'd need to find out what shard isn't assigned/allocated and then you can try allocating it. Or easy way if Elasticsearch data is reindexable, is to just purge the Elasticsearch data and reindex.

    You can also check to see if ES GeoIP is working with these commands
    Code (Text):
    curl -sX POST "http://localhost:9200/_ingest/pipeline/_simulate" -H 'Content-Type: application/json' -d'
    {
      "pipeline": {
        "processors": [
          {
            "geoip": {
              "field": "location"
            }
          }
        ]
      },
      "docs": [
        {
          "_source": {
            "location": "8.8.8.8"
          }
        }
      ]
    }' | jq -r
    

    example output
    Code (Text):
    {
      "docs": [
        {
          "doc": {
            "_index": "_index",
            "_type": "_doc",
            "_id": "_id",
            "_source": {
              "location": "8.8.8.8",
              "geoip": {
                "continent_name": "North America",
                "country_name": "United States",
                "location": {
                  "lon": -97.822,
                  "lat": 37.751
                },
                "country_iso_code": "US"
              }
            },
            "_ingest": {
              "timestamp": "2024-03-07T01:01:07.636282451Z"
            }
          }
        }
      ]
    }

    and
    Code (Text):
    curl -sX GET "http://localhost:9200/_ingest/geoip/stats" | jq -r
    

    Code (Text):
    {
      "stats": {
        "successful_downloads": 0,
        "failed_downloads": 0,
        "total_download_time": 0,
        "databases_count": 0,
        "skipped_updates": 3,
        "expired_databases": 0
      },
      "nodes": {
        "xabB00-kSu6bz84uA69-DA": {
          "databases": [
            {
              "name": "GeoLite2-ASN.mmdb"
            },
            {
              "name": "GeoLite2-Country.mmdb"
            },
            {
              "name": "GeoLite2-City.mmdb"
            }
          ],
          "files_in_temp": [
            "GeoLite2-ASN.mmdb_elastic-geoip-database-service-agreement-LICENSE.txt",
            "GeoLite2-ASN.mmdb_LICENSE.txt",
            "GeoLite2-City.mmdb_LICENSE.txt",
            "GeoLite2-Country.mmdb_elastic-geoip-database-service-agreement-LICENSE.txt",
            "GeoLite2-ASN.mmdb",
            "GeoLite2-City.mmdb_COPYRIGHT.txt",
            "GeoLite2-City.mmdb",
            "GeoLite2-City.mmdb_elastic-geoip-database-service-agreement-LICENSE.txt",
            "GeoLite2-Country.mmdb_LICENSE.txt",
            "GeoLite2-ASN.mmdb_COPYRIGHT.txt",
            "GeoLite2-Country.mmdb",
            "GeoLite2-Country.mmdb_COPYRIGHT.txt",
            "GeoLite2-City.mmdb_README.txt"
          ]
        }
      }
    }