/ Register

Welcome to Centmin Mod Community
Register Now

Sysadmin Elasticsearch

Discussion in 'System Administration' started by ernestdefoe, Jul 17, 2019.

Previous Thread Next Thread
Loading...
  1. Jul 17, 2019 #1
    ernestdefoe

    ernestdefoe New Member

    17
    6
    3
    Sep 8, 2014
    Ratings:
    +7
    Local Time:
    3:57 PM
    I am having issues getting ES to work. I originally installed the newest ES 7 version but found it was to new and removed it and installed ES6 and now I am having issues. ES will start but then quits and the curl -XGET command doesn't work I get the curl: (7) Failed to connect to localhost port 9200: Connection refused error.

    When I do the ps aux command it shows ES is running and then when I do the code systemctl status elasticsearch.service it shows it running then checking it again it shows
    Code:
    [00:25][root@srv445 ~]# systemctl status elasticsearch.service
* elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Wed 2019-07-17 00:25:30 UTC; 39s ago
     Docs: http://www.elastic.co
  Process: 16808 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=1/FAILURE)
 Main PID: 16808 (code=exited, status=1/FAILURE)

Jul 17 00:25:20 srv445 systemd[1]: Started Elasticsearch.
Jul 17 00:25:30 srv445 systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Jul 17 00:25:30 srv445 systemd[1]: Unit elasticsearch.service entered failed state.
Jul 17 00:25:30 srv445 systemd[1]: elasticsearch.service failed.
    What can be going on that's keeping ES from working?
     
  2. Jul 17, 2019 #2
    ernestdefoe

    ernestdefoe New Member

    17
    6
    3
    Sep 8, 2014
    Ratings:
    +7
    Local Time:
    3:57 PM
    Sorry didn't know the correct forum for this. Thanks for moving it to the correct forum.
     
  3. Jul 18, 2019 #3
    ernestdefoe

    ernestdefoe New Member

    17
    6
    3
    Sep 8, 2014
    Ratings:
    +7
    Local Time:
    3:57 PM
    So no one has any ideas what could be going on?
     
  4. Jul 18, 2019 #4
    eva2000

    eva2000 Administrator Staff Member

    58,905
    12,490
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +19,122
    Local Time:
    6:57 AM
    Nginx 1.31.x
    MariaDB 10.x/11.4+/12.3+
    check the elasticsearch logs for clues
     
  5. Jul 18, 2019 #5
    ernestdefoe

    ernestdefoe New Member

    17
    6
    3
    Sep 8, 2014
    Ratings:
    +7
    Local Time:
    3:57 PM
    Well I don't know how to read any of that. Guess I'm just gonna give up. Thank you for the help.
     
  6. Jul 18, 2019 #6
    pamamolf

    pamamolf Well-Known Member

    4,125
    429
    83
    May 31, 2014
    Ratings:
    +841
    Local Time:
    11:57 PM
    Nginx-1.29.x
    MariaDB 10.6.x
    Check at:
    Code:
    /var/log/elasticsearch/
    Something like:
    Code:
    elasticsearch.log
     
  7. Jul 18, 2019 #7
    ernestdefoe

    ernestdefoe New Member

    17
    6
    3
    Sep 8, 2014
    Ratings:
    +7
    Local Time:
    3:57 PM
    If I post the content of that log file could someone take a look cause I have no clue what I’m looking at?
     
  8. Jul 18, 2019 #8
    ernestdefoe

    ernestdefoe New Member

    17
    6
    3
    Sep 8, 2014
    Ratings:
    +7
    Local Time:
    3:57 PM
    Code:
    [2019-07-18T00:03:32,327][WARN ][o.e.b.JNANatives         ] [unknown] unable to install syscall filter:
java.lang.UnsupportedOperationException: seccomp unavailable: CONFIG_SECCOMP not compiled into kernel, CONFIG_SECCOMP and CONFIG_SECCOMP_$
        at org.elasticsearch.bootstrap.SystemCallFilter.linuxImpl(SystemCallFilter.java:342) ~[elasticsearch-6.8.1.jar:6.8.1]
        at org.elasticsearch.bootstrap.SystemCallFilter.init(SystemCallFilter.java:617) ~[elasticsearch-6.8.1.jar:6.8.1]
        at org.elasticsearch.bootstrap.JNANatives.tryInstallSystemCallFilter(JNANatives.java:260) [elasticsearch-6.8.1.jar:6.8.1]
        at org.elasticsearch.bootstrap.Natives.tryInstallSystemCallFilter(Natives.java:113) [elasticsearch-6.8.1.jar:6.8.1]
        at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:108) [elasticsearch-6.8.1.jar:6.8.1]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:170) [elasticsearch-6.8.1.jar:6.8.1]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) [elasticsearch-6.8.1.jar:6.8.1]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-6.8.1.jar:6.8.1]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-6.8.1.jar:6.8.1]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-6.8.1.jar:6.8.1]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) [elasticsearch-cli-6.8.1.jar:6.8.1]
        at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-6.8.1.jar:6.8.1]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:116) [elasticsearch-6.8.1.jar:6.8.1]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) [elasticsearch-6.8.1.jar:6.8.1]
[2019-07-18T00:03:33,734][ERROR][o.e.b.Bootstrap          ] [unknown] Exception
org.elasticsearch.ElasticsearchException: java.io.IOException: failed to read [id:0, file:/var/lib/elasticsearch/nodes/0/_state/node-0.st]
        at org.elasticsearch.ExceptionsHelper.maybeThrowRuntimeAndSuppress(ExceptionsHelper.java:165) ~[elasticsearch-6.8.1.jar:6.8.1]
        at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestStateWithGeneration(MetaDataStateFormat.java:306) ~[elasticsearch-6.8.$
        at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:324) ~[elasticsearch-6.8.1.jar:6.8.1]
        at org.elasticsearch.env.NodeEnvironment.loadOrCreateNodeMetaData(NodeEnvironment.java:417) ~[elasticsearch-6.8.1.jar:6.8.1]
        at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:305) ~[elasticsearch-6.8.1.jar:6.8.1]
        at org.elasticsearch.node.Node.<init>(Node.java:296) ~[elasticsearch-6.8.1.jar:6.8.1]
        at org.elasticsearch.node.Node.<init>(Node.java:266) ~[elasticsearch-6.8.1.jar:6.8.1]
        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.8.1.jar:6.8.1]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.8.1.jar:6.8.1]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) [elasticsearch-6.8.1.jar:6.8.1]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-6.8.1.jar:6.8.1]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-6.8.1.jar:6.8.1]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-6.8.1.jar:6.8.1]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) [elasticsearch-cli-6.8.1.jar:6.8.1]
       at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-6.8.1.jar:6.8.1]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) [elasticsearch-cli-6.8.1.jar:6.8.1]
        at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-6.8.1.jar:6.8.1]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:116) [elasticsearch-6.8.1.jar:6.8.1]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) [elasticsearch-6.8.1.jar:6.8.1]
Caused by: java.io.IOException: failed to read [id:0, file:/var/lib/elasticsearch/nodes/0/_state/node-0.st]
        at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestStateWithGeneration(MetaDataStateFormat.java:300) ~[elasticsearch-6.8.$
        ... 15 more
Caused by: org.elasticsearch.common.xcontent.XContentParseException: [-1:36] [node_meta_data] unknown field [node_version], parser not fo$
        at org.elasticsearch.common.xcontent.ObjectParser.getParser(ObjectParser.java:369) ~[elasticsearch-x-content-6.8.1.jar:6.8.1]
        at org.elasticsearch.common.xcontent.ObjectParser.parse(ObjectParser.java:158) ~[elasticsearch-x-content-6.8.1.jar:6.8.1]
        at org.elasticsearch.common.xcontent.ObjectParser.apply(ObjectParser.java:182) ~[elasticsearch-x-content-6.8.1.jar:6.8.1]
        at org.elasticsearch.env.NodeMetaData$1.fromXContent(NodeMetaData.java:110) ~[elasticsearch-6.8.1.jar:6.8.1]
        at org.elasticsearch.env.NodeMetaData$1.fromXContent(NodeMetaData.java:94) ~[elasticsearch-6.8.1.jar:6.8.1]
        at org.elasticsearch.gateway.MetaDataStateFormat.read(MetaDataStateFormat.java:197) ~[elasticsearch-6.8.1.jar:6.8.1]
        at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestStateWithGeneration(MetaDataStateFormat.java:296) ~[elasticsearch-6.8.$
        ... 15 more
[2019-07-18T00:03:33,742][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [unknown] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: ElasticsearchException[java.io.IOException: failed to read [id:0, file:/var/lib/elasticsear$
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-6.8.1.jar:6.8.1]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-6.8.1.jar:6.8.1]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.8.1.jar:6.8.1]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.8.1.jar:6.8.1]
        at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.8.1.jar:6.8.1]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:116) ~[elasticsearch-6.8.1.jar:6.8.1]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) ~[elasticsearch-6.8.1.jar:6.8.1]
Caused by: org.elasticsearch.ElasticsearchException: java.io.IOException: failed to read [id:0, file:/var/lib/elasticsearch/nodes/0/_stat$
        at org.elasticsearch.ExceptionsHelper.maybeThrowRuntimeAndSuppress(ExceptionsHelper.java:165) ~[elasticsearch-6.8.1.jar:6.8.1]
        at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestStateWithGeneration(MetaDataStateFormat.java:306) ~[elasticsearch-6.8.$
        at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:324) ~[elasticsearch-6.8.1.jar:6.8.1]
        at org.elasticsearch.env.NodeEnvironment.loadOrCreateNodeMetaData(NodeEnvironment.java:417) ~[elasticsearch-6.8.1.jar:6.8.1]
        at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:305) ~[elasticsearch-6.8.1.jar:6.8.1]
        at org.elasticsearch.env.NodeEnvironment.loadOrCreateNodeMetaData(NodeEnvironment.java:417) ~[elasticsearch-6.8.1.jar:6.8.1]
        at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:305) ~[elasticsearch-6.8.1.jar:6.8.1]
        at org.elasticsearch.node.Node.<init>(Node.java:296) ~[elasticsearch-6.8.1.jar:6.8.1]
        at org.elasticsearch.node.Node.<init>(Node.java:266) ~[elasticsearch-6.8.1.jar:6.8.1]
        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.8.1.jar:6.8.1]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.8.1.jar:6.8.1]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.8.1.jar:6.8.1]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-6.8.1.jar:6.8.1]
        ... 6 more
Caused by: java.io.IOException: failed to read [id:0, file:/var/lib/elasticsearch/nodes/0/_state/node-0.st]
        at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestStateWithGeneration(MetaDataStateFormat.java:300) ~[elasticsearch-6.8.$
        at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:324) ~[elasticsearch-6.8.1.jar:6.8.1]
        at org.elasticsearch.env.NodeEnvironment.loadOrCreateNodeMetaData(NodeEnvironment.java:417) ~[elasticsearch-6.8.1.jar:6.8.1]
        at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:305) ~[elasticsearch-6.8.1.jar:6.8.1]
        at org.elasticsearch.node.Node.<init>(Node.java:296) ~[elasticsearch-6.8.1.jar:6.8.1]
        at org.elasticsearch.node.Node.<init>(Node.java:266) ~[elasticsearch-6.8.1.jar:6.8.1]
        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.8.1.jar:6.8.1]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.8.1.jar:6.8.1]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.8.1.jar:6.8.1]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-6.8.1.jar:6.8.1]
        ... 6 more
Caused by: org.elasticsearch.common.xcontent.XContentParseException: [-1:36] [node_meta_data] unknown field [node_version], parser not fo$
        at org.elasticsearch.common.xcontent.ObjectParser.getParser(ObjectParser.java:369) ~[elasticsearch-x-content-6.8.1.jar:6.8.1]
        at org.elasticsearch.common.xcontent.ObjectParser.parse(ObjectParser.java:158) ~[elasticsearch-x-content-6.8.1.jar:6.8.1]
        at org.elasticsearch.common.xcontent.ObjectParser.apply(ObjectParser.java:182) ~[elasticsearch-x-content-6.8.1.jar:6.8.1]
        at org.elasticsearch.env.NodeMetaData$1.fromXContent(NodeMetaData.java:110) ~[elasticsearch-6.8.1.jar:6.8.1]
        at org.elasticsearch.env.NodeMetaData$1.fromXContent(NodeMetaData.java:94) ~[elasticsearch-6.8.1.jar:6.8.1]
        at org.elasticsearch.gateway.MetaDataStateFormat.read(MetaDataStateFormat.java:197) ~[elasticsearch-6.8.1.jar:6.8.1]
        at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestStateWithGeneration(MetaDataStateFormat.java:296) ~[elasticsearch-6.8.$
        at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:324) ~[elasticsearch-6.8.1.jar:6.8.1]
        at org.elasticsearch.env.NodeEnvironment.loadOrCreateNodeMetaData(NodeEnvironment.java:417) ~[elasticsearch-6.8.1.jar:6.8.1]
        at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:305) ~[elasticsearch-6.8.1.jar:6.8.1]
        at org.elasticsearch.env.NodeEnvironment.loadOrCreateNodeMetaData(NodeEnvironment.java:417) ~[elasticsearch-6.8.1.jar:6.8.1]
        at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:305) ~[elasticsearch-6.8.1.jar:6.8.1]
        at org.elasticsearch.node.Node.<init>(Node.java:296) ~[elasticsearch-6.8.1.jar:6.8.1]
        at org.elasticsearch.node.Node.<init>(Node.java:266) ~[elasticsearch-6.8.1.jar:6.8.1]
        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.8.1.jar:6.8.1]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.8.1.jar:6.8.1]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.8.1.jar:6.8.1]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-6.8.1.jar:6.8.1]
        ... 6 more
     
  9. Jul 18, 2019 #9
    eva2000

    eva2000 Administrator Staff Member

    58,905
    12,490
    113
    May 24, 2014
    Brisbane, Australia
    Ratings:
    +19,122
    Local Time:
    6:57 AM
    Nginx 1.31.x
    MariaDB 10.x/11.4+/12.3+
  10. Nov 2, 2019 #10
    diy

    diy Member

    50
    32
    18
    Jan 14, 2019
    USA
    Ratings:
    +48
    Local Time:
    4:57 PM
    nginx 1.19.x
    MariaDB 10.4.x
    I just did the same thing! I installed eleasticsearch 7.x then found out that IPS 4.4.7 does not support ES 7.x yet, so I uninstalled via yum and installed javajdk-devel 1.8 and ES v6.8 via yum. ES failed to start after editing the config to define the cluster, host & port, so I looked at the log and noticed that ES had failed to read/create the node lock.

    "Yum remove elasticsearch" had orphaned node dirs/files (& others) that ES 7.x had created in /var/lib/elasticsearch/ when the elasticsearch user & group were deleted from the system, so those files & directories were now owned by another user.

    The solution:
    Code:
    rm -rf /var/lib/elasticsearch/*
chown elasticsearch:elasticsearch /var/lib/elasticsearch
systemctl start elasticsearch.service
     
Previous Thread Next Thread
Loading...

.