Join the community today
Register Now

Sysadmin Elasticsearch

Discussion in 'System Administration' started by ernestdefoe, Jul 17, 2019.

  1. ernestdefoe

    ernestdefoe New Member

    17
    6
    3
    Sep 8, 2014
    Ratings:
    +7
    Local Time:
    10:59 PM
    I am having issues getting ES to work. I originally installed the newest ES 7 version but found it was to new and removed it and installed ES6 and now I am having issues. ES will start but then quits and the curl -XGET command doesn't work I get the curl: (7) Failed to connect to localhost port 9200: Connection refused error.

    When I do the ps aux command it shows ES is running and then when I do the code systemctl status elasticsearch.service it shows it running then checking it again it shows
    Code:
    [00:25][root@srv445 ~]# systemctl status elasticsearch.service
    * elasticsearch.service - Elasticsearch
       Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
       Active: failed (Result: exit-code) since Wed 2019-07-17 00:25:30 UTC; 39s ago
         Docs: http://www.elastic.co
      Process: 16808 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=1/FAILURE)
     Main PID: 16808 (code=exited, status=1/FAILURE)
    
    Jul 17 00:25:20 srv445 systemd[1]: Started Elasticsearch.
    Jul 17 00:25:30 srv445 systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
    Jul 17 00:25:30 srv445 systemd[1]: Unit elasticsearch.service entered failed state.
    Jul 17 00:25:30 srv445 systemd[1]: elasticsearch.service failed.
    What can be going on that's keeping ES from working?
     
  2. ernestdefoe

    ernestdefoe New Member

    17
    6
    3
    Sep 8, 2014
    Ratings:
    +7
    Local Time:
    10:59 PM
    Sorry didn't know the correct forum for this. Thanks for moving it to the correct forum.
     
  3. ernestdefoe

    ernestdefoe New Member

    17
    6
    3
    Sep 8, 2014
    Ratings:
    +7
    Local Time:
    10:59 PM
    So no one has any ideas what could be going on?
     
  4. eva2000

    eva2000 Administrator Staff Member

    44,804
    10,216
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +15,834
    Local Time:
    1:59 PM
    Nginx 1.17.x
    MariaDB 5.5/10.x
    check the elasticsearch logs for clues
     
  5. ernestdefoe

    ernestdefoe New Member

    17
    6
    3
    Sep 8, 2014
    Ratings:
    +7
    Local Time:
    10:59 PM
    Well I don't know how to read any of that. Guess I'm just gonna give up. Thank you for the help.
     
  6. pamamolf

    pamamolf Premium Member Premium Member

    3,831
    374
    83
    May 31, 2014
    Ratings:
    +718
    Local Time:
    6:59 AM
    Nginx-1.17.x
    MariaDB 10.3.x
    Check at:
    Code:
    /var/log/elasticsearch/
    Something like:
    Code:
    elasticsearch.log
     
  7. ernestdefoe

    ernestdefoe New Member

    17
    6
    3
    Sep 8, 2014
    Ratings:
    +7
    Local Time:
    10:59 PM
    If I post the content of that log file could someone take a look cause I have no clue what I’m looking at?
     
  8. ernestdefoe

    ernestdefoe New Member

    17
    6
    3
    Sep 8, 2014
    Ratings:
    +7
    Local Time:
    10:59 PM
    Code:
    [2019-07-18T00:03:32,327][WARN ][o.e.b.JNANatives         ] [unknown] unable to install syscall filter:
    java.lang.UnsupportedOperationException: seccomp unavailable: CONFIG_SECCOMP not compiled into kernel, CONFIG_SECCOMP and CONFIG_SECCOMP_$
            at org.elasticsearch.bootstrap.SystemCallFilter.linuxImpl(SystemCallFilter.java:342) ~[elasticsearch-6.8.1.jar:6.8.1]
            at org.elasticsearch.bootstrap.SystemCallFilter.init(SystemCallFilter.java:617) ~[elasticsearch-6.8.1.jar:6.8.1]
            at org.elasticsearch.bootstrap.JNANatives.tryInstallSystemCallFilter(JNANatives.java:260) [elasticsearch-6.8.1.jar:6.8.1]
            at org.elasticsearch.bootstrap.Natives.tryInstallSystemCallFilter(Natives.java:113) [elasticsearch-6.8.1.jar:6.8.1]
            at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:108) [elasticsearch-6.8.1.jar:6.8.1]
            at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:170) [elasticsearch-6.8.1.jar:6.8.1]
            at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) [elasticsearch-6.8.1.jar:6.8.1]
            at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-6.8.1.jar:6.8.1]
            at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-6.8.1.jar:6.8.1]
            at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-6.8.1.jar:6.8.1]
            at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) [elasticsearch-cli-6.8.1.jar:6.8.1]
            at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-6.8.1.jar:6.8.1]
            at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:116) [elasticsearch-6.8.1.jar:6.8.1]
            at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) [elasticsearch-6.8.1.jar:6.8.1]
    [2019-07-18T00:03:33,734][ERROR][o.e.b.Bootstrap          ] [unknown] Exception
    org.elasticsearch.ElasticsearchException: java.io.IOException: failed to read [id:0, file:/var/lib/elasticsearch/nodes/0/_state/node-0.st]
            at org.elasticsearch.ExceptionsHelper.maybeThrowRuntimeAndSuppress(ExceptionsHelper.java:165) ~[elasticsearch-6.8.1.jar:6.8.1]
            at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestStateWithGeneration(MetaDataStateFormat.java:306) ~[elasticsearch-6.8.$
            at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:324) ~[elasticsearch-6.8.1.jar:6.8.1]
            at org.elasticsearch.env.NodeEnvironment.loadOrCreateNodeMetaData(NodeEnvironment.java:417) ~[elasticsearch-6.8.1.jar:6.8.1]
            at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:305) ~[elasticsearch-6.8.1.jar:6.8.1]
            at org.elasticsearch.node.Node.<init>(Node.java:296) ~[elasticsearch-6.8.1.jar:6.8.1]
            at org.elasticsearch.node.Node.<init>(Node.java:266) ~[elasticsearch-6.8.1.jar:6.8.1]
            at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.8.1.jar:6.8.1]
            at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.8.1.jar:6.8.1]
            at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) [elasticsearch-6.8.1.jar:6.8.1]
            at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-6.8.1.jar:6.8.1]
            at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-6.8.1.jar:6.8.1]
            at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-6.8.1.jar:6.8.1]
            at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) [elasticsearch-cli-6.8.1.jar:6.8.1]
           at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-6.8.1.jar:6.8.1]
            at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) [elasticsearch-cli-6.8.1.jar:6.8.1]
            at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-6.8.1.jar:6.8.1]
            at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:116) [elasticsearch-6.8.1.jar:6.8.1]
            at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) [elasticsearch-6.8.1.jar:6.8.1]
    Caused by: java.io.IOException: failed to read [id:0, file:/var/lib/elasticsearch/nodes/0/_state/node-0.st]
            at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestStateWithGeneration(MetaDataStateFormat.java:300) ~[elasticsearch-6.8.$
            ... 15 more
    Caused by: org.elasticsearch.common.xcontent.XContentParseException: [-1:36] [node_meta_data] unknown field [node_version], parser not fo$
            at org.elasticsearch.common.xcontent.ObjectParser.getParser(ObjectParser.java:369) ~[elasticsearch-x-content-6.8.1.jar:6.8.1]
            at org.elasticsearch.common.xcontent.ObjectParser.parse(ObjectParser.java:158) ~[elasticsearch-x-content-6.8.1.jar:6.8.1]
            at org.elasticsearch.common.xcontent.ObjectParser.apply(ObjectParser.java:182) ~[elasticsearch-x-content-6.8.1.jar:6.8.1]
            at org.elasticsearch.env.NodeMetaData$1.fromXContent(NodeMetaData.java:110) ~[elasticsearch-6.8.1.jar:6.8.1]
            at org.elasticsearch.env.NodeMetaData$1.fromXContent(NodeMetaData.java:94) ~[elasticsearch-6.8.1.jar:6.8.1]
            at org.elasticsearch.gateway.MetaDataStateFormat.read(MetaDataStateFormat.java:197) ~[elasticsearch-6.8.1.jar:6.8.1]
            at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestStateWithGeneration(MetaDataStateFormat.java:296) ~[elasticsearch-6.8.$
            ... 15 more
    [2019-07-18T00:03:33,742][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [unknown] uncaught exception in thread [main]
    org.elasticsearch.bootstrap.StartupException: ElasticsearchException[java.io.IOException: failed to read [id:0, file:/var/lib/elasticsear$
            at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-6.8.1.jar:6.8.1]
            at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-6.8.1.jar:6.8.1]
            at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.8.1.jar:6.8.1]
            at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.8.1.jar:6.8.1]
            at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.8.1.jar:6.8.1]
            at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:116) ~[elasticsearch-6.8.1.jar:6.8.1]
            at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) ~[elasticsearch-6.8.1.jar:6.8.1]
    Caused by: org.elasticsearch.ElasticsearchException: java.io.IOException: failed to read [id:0, file:/var/lib/elasticsearch/nodes/0/_stat$
            at org.elasticsearch.ExceptionsHelper.maybeThrowRuntimeAndSuppress(ExceptionsHelper.java:165) ~[elasticsearch-6.8.1.jar:6.8.1]
            at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestStateWithGeneration(MetaDataStateFormat.java:306) ~[elasticsearch-6.8.$
            at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:324) ~[elasticsearch-6.8.1.jar:6.8.1]
            at org.elasticsearch.env.NodeEnvironment.loadOrCreateNodeMetaData(NodeEnvironment.java:417) ~[elasticsearch-6.8.1.jar:6.8.1]
            at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:305) ~[elasticsearch-6.8.1.jar:6.8.1]
            at org.elasticsearch.env.NodeEnvironment.loadOrCreateNodeMetaData(NodeEnvironment.java:417) ~[elasticsearch-6.8.1.jar:6.8.1]
            at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:305) ~[elasticsearch-6.8.1.jar:6.8.1]
            at org.elasticsearch.node.Node.<init>(Node.java:296) ~[elasticsearch-6.8.1.jar:6.8.1]
            at org.elasticsearch.node.Node.<init>(Node.java:266) ~[elasticsearch-6.8.1.jar:6.8.1]
            at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.8.1.jar:6.8.1]
            at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.8.1.jar:6.8.1]
            at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.8.1.jar:6.8.1]
            at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-6.8.1.jar:6.8.1]
            ... 6 more
    Caused by: java.io.IOException: failed to read [id:0, file:/var/lib/elasticsearch/nodes/0/_state/node-0.st]
            at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestStateWithGeneration(MetaDataStateFormat.java:300) ~[elasticsearch-6.8.$
            at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:324) ~[elasticsearch-6.8.1.jar:6.8.1]
            at org.elasticsearch.env.NodeEnvironment.loadOrCreateNodeMetaData(NodeEnvironment.java:417) ~[elasticsearch-6.8.1.jar:6.8.1]
            at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:305) ~[elasticsearch-6.8.1.jar:6.8.1]
            at org.elasticsearch.node.Node.<init>(Node.java:296) ~[elasticsearch-6.8.1.jar:6.8.1]
            at org.elasticsearch.node.Node.<init>(Node.java:266) ~[elasticsearch-6.8.1.jar:6.8.1]
            at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.8.1.jar:6.8.1]
            at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.8.1.jar:6.8.1]
            at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.8.1.jar:6.8.1]
            at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-6.8.1.jar:6.8.1]
            ... 6 more
    Caused by: org.elasticsearch.common.xcontent.XContentParseException: [-1:36] [node_meta_data] unknown field [node_version], parser not fo$
            at org.elasticsearch.common.xcontent.ObjectParser.getParser(ObjectParser.java:369) ~[elasticsearch-x-content-6.8.1.jar:6.8.1]
            at org.elasticsearch.common.xcontent.ObjectParser.parse(ObjectParser.java:158) ~[elasticsearch-x-content-6.8.1.jar:6.8.1]
            at org.elasticsearch.common.xcontent.ObjectParser.apply(ObjectParser.java:182) ~[elasticsearch-x-content-6.8.1.jar:6.8.1]
            at org.elasticsearch.env.NodeMetaData$1.fromXContent(NodeMetaData.java:110) ~[elasticsearch-6.8.1.jar:6.8.1]
            at org.elasticsearch.env.NodeMetaData$1.fromXContent(NodeMetaData.java:94) ~[elasticsearch-6.8.1.jar:6.8.1]
            at org.elasticsearch.gateway.MetaDataStateFormat.read(MetaDataStateFormat.java:197) ~[elasticsearch-6.8.1.jar:6.8.1]
            at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestStateWithGeneration(MetaDataStateFormat.java:296) ~[elasticsearch-6.8.$
            at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:324) ~[elasticsearch-6.8.1.jar:6.8.1]
            at org.elasticsearch.env.NodeEnvironment.loadOrCreateNodeMetaData(NodeEnvironment.java:417) ~[elasticsearch-6.8.1.jar:6.8.1]
            at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:305) ~[elasticsearch-6.8.1.jar:6.8.1]
            at org.elasticsearch.env.NodeEnvironment.loadOrCreateNodeMetaData(NodeEnvironment.java:417) ~[elasticsearch-6.8.1.jar:6.8.1]
            at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:305) ~[elasticsearch-6.8.1.jar:6.8.1]
            at org.elasticsearch.node.Node.<init>(Node.java:296) ~[elasticsearch-6.8.1.jar:6.8.1]
            at org.elasticsearch.node.Node.<init>(Node.java:266) ~[elasticsearch-6.8.1.jar:6.8.1]
            at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.8.1.jar:6.8.1]
            at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.8.1.jar:6.8.1]
            at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.8.1.jar:6.8.1]
            at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-6.8.1.jar:6.8.1]
            ... 6 more
     
  9. eva2000

    eva2000 Administrator Staff Member

    44,804
    10,216
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +15,834
    Local Time:
    1:59 PM
    Nginx 1.17.x
    MariaDB 5.5/10.x
  10. diy

    diy Premium Member Premium Member

    44
    28
    18
    Jan 14, 2019
    USA
    Ratings:
    +43
    Local Time:
    11:59 PM
    nginx 1.17.x
    MariaDB 10.4.x
    I just did the same thing! I installed eleasticsearch 7.x then found out that IPS 4.4.7 does not support ES 7.x yet, so I uninstalled via yum and installed javajdk-devel 1.8 and ES v6.8 via yum. ES failed to start after editing the config to define the cluster, host & port, so I looked at the log and noticed that ES had failed to read/create the node lock.

    "Yum remove elasticsearch" had orphaned node dirs/files (& others) that ES 7.x had created in /var/lib/elasticsearch/ when the elasticsearch user & group were deleted from the system, so those files & directories were now owned by another user.

    The solution:
    Code:
    rm -rf /var/lib/elasticsearch/*
    chown elasticsearch:elasticsearch /var/lib/elasticsearch
    systemctl start elasticsearch.service