Sysadmin AlmaLinux Elasticsearch log issue GeoIpDownloader Error

AlmaLinux 8 or 9 ? How was Elasticsearch installed and configured? Have you checked if Elasticsearch has enough disk space? Did you try to manually configure GeoIP for Elasticsearch instead of letting it automatically handle it?

Check if Elasticsearch connectivity to Maxmind servers to download GeoIP database is working and that no firewalls are in the way or if your server IP has been blocked. See GeoIP processor | Elasticsearch Guide [7.17] | Elastic

1. Check ES server's health status is green

Code (Text):

curl -sX GET "http://localhost:9200/_cluster/health?pretty"

2. If not green, check indices and shards for issues - specifically the .geoip_databases index health status

Code (Text):

curl -sX GET "http://localhost:9200/_cat/indices?v&pretty"

3. Inspect shards for .geoip_databases index. If the shard(s) for this index are UNASSIGNED or in a state other than STARTED, it indicates an issue with shard allocation for this index.

Code (Text):

curl -sX GET "http://localhost:9200/_cat/shards/.geoip_databases?pretty"

example out on working Elasticsearch server

Code (Text):

curl -sX GET "http://localhost:9200/_cat/shards/.geoip_databases?pretty"
.geoip_databases 0 p STARTED 39 37mb 127.0.0.1 node22

where

• .geoip_databases: The name of the index, indicating it's the system index used by Elasticsearch to store GeoIP databases.
• 0: The shard number. Since system indices like .geoip_databases typically don't require a lot of data, a single primary shard without replicas is common.
• p: Indicates this shard is a primary shard.
• STARTED: The status of the shard, indicating that it is active and operational. This is the state you want to see for a healthy shard.
• 39: The unique document count in the shard.
• 37mb: The size of the shard on disk, which is a reasonable size for the GeoIP databases.
• 127.0.0.1: The IP address of the node on which this shard is located. Since it’s 127.0.0.1, this indicates the shard is hosted on the local machine.
• node22: The name of the node hosting the shard, as specified in your Elasticsearch configuration.

4. Use the Cluster Allocation Explain API to get more details on why the .geoip_databases shard is not allocated and can help identify the root cause, such as disk watermarks being exceeded, node connectivity issues, or other allocation problems.

Code:

curl -sX GET "http://localhost:9200/_cluster/allocation/explain?pretty" -H 'Content-Type: application/json' -d'
{
  "index": ".geoip_databases",
  "shard": 0,
  "primary": true
}'

 

Just noticed that your error shows Elasticsearch is trying to download GeoIP2 databases to /home/elasticsearch. Does that directory exist? On my working Elasticsearch servers I see GeoIP2 databases downloaded to my Elasticsearch tmp directory which I configure with correct user/group permissions as part of Elasticsearch install/configs for myself and my clients.

 

hmm lots of ways to troubleshoot this. You'd need to find out what shard isn't assigned/allocated and then you can try allocating it. Or easy way if Elasticsearch data is reindexable, is to just purge the Elasticsearch data and reindex.

You can also check to see if ES GeoIP is working with these commands

Code (Text):

curl -sX POST "http://localhost:9200/_ingest/pipeline/_simulate" -H 'Content-Type: application/json' -d'
{
  "pipeline": {
    "processors": [
      {
        "geoip": {
          "field": "location"
        }
      }
    ]
  },
  "docs": [
    {
      "_source": {
        "location": "8.8.8.8"
      }
    }
  ]
}' | jq -r

example output

Code (Text):

{
  "docs": [
    {
      "doc": {
        "_index": "_index",
        "_type": "_doc",
        "_id": "_id",
        "_source": {
          "location": "8.8.8.8",
          "geoip": {
            "continent_name": "North America",
            "country_name": "United States",
            "location": {
              "lon": -97.822,
              "lat": 37.751
            },
            "country_iso_code": "US"
          }
        },
        "_ingest": {
          "timestamp": "2024-03-07T01:01:07.636282451Z"
        }
      }
    }
  ]
}

and

Code (Text):

curl -sX GET "http://localhost:9200/_ingest/geoip/stats" | jq -r

Code (Text):

{
  "stats": {
    "successful_downloads": 0,
    "failed_downloads": 0,
    "total_download_time": 0,
    "databases_count": 0,
    "skipped_updates": 3,
    "expired_databases": 0
  },
  "nodes": {
    "xabB00-kSu6bz84uA69-DA": {
      "databases": [
        {
          "name": "GeoLite2-ASN.mmdb"
        },
        {
          "name": "GeoLite2-Country.mmdb"
        },
        {
          "name": "GeoLite2-City.mmdb"
        }
      ],
      "files_in_temp": [
        "GeoLite2-ASN.mmdb_elastic-geoip-database-service-agreement-LICENSE.txt",
        "GeoLite2-ASN.mmdb_LICENSE.txt",
        "GeoLite2-City.mmdb_LICENSE.txt",
        "GeoLite2-Country.mmdb_elastic-geoip-database-service-agreement-LICENSE.txt",
        "GeoLite2-ASN.mmdb",
        "GeoLite2-City.mmdb_COPYRIGHT.txt",
        "GeoLite2-City.mmdb",
        "GeoLite2-City.mmdb_elastic-geoip-database-service-agreement-LICENSE.txt",
        "GeoLite2-Country.mmdb_LICENSE.txt",
        "GeoLite2-ASN.mmdb_COPYRIGHT.txt",
        "GeoLite2-Country.mmdb",
        "GeoLite2-Country.mmdb_COPYRIGHT.txt",
        "GeoLite2-City.mmdb_README.txt"
      ]
    }
  }
}