走过岁月......

wazuh server运行中很意外地elasticsearch服务起不来了，重装数次无效

# systemctl status elasticsearch

● elasticsearch.service - Elasticsearch

   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)

   Active: failed (Result: exit-code) since Wed 2019-09-11 11:08:50 CST; 621ms ago

     Docs: https://www.elastic.co

  Process: 57259 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=1/FAILURE)

Main PID: 57259 (code=exited, status=1/FAILURE)

Sep 11 11:08:46 wazuh-server systemd[1]: Started Elasticsearch.

Sep 11 11:08:47 wazuh-server elasticsearch[57259]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.

Sep 11 11:08:50 wazuh-server systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE

Sep 11 11:08:50 wazuh-server systemd[1]: Unit elasticsearch.service entered failed state.

Sep 11 11:08:50 wazuh-server systemd[1]: elasticsearch.service failed.

根据提示，重新安装了新版本jdk也不行

查看elasticsearch运行日志

# cat /var/log/elasticsearch/elasticsearch.log

[2019-09-11T11:08:50,096][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [node-1] uncaught exception in thread [main]

org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to obtain node locks, tried [[/var/lib/elasticsearch]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?

        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.3.1.jar:7.3.1]

        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.3.1.jar:7.3.1]

        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.3.1.jar:7.3.1]

        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-7.3.1.jar:7.3.1]

        at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.3.1.jar:7.3.1]

        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.3.1.jar:7.3.1]

        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.3.1.jar:7.3.1]

Caused by: java.lang.IllegalStateException: failed to obtain node locks, tried [[/var/lib/elasticsearch]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?

        at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:299) ~[elasticsearch-7.3.1.jar:7.3.1]

        at org.elasticsearch.node.Node.<init>(Node.java:278) ~[elasticsearch-7.3.1.jar:7.3.1]

        at org.elasticsearch.node.Node.<init>(Node.java:258) ~[elasticsearch-7.3.1.jar:7.3.1]

        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:221) ~[elasticsearch-7.3.1.jar:7.3.1]

        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-7.3.1.jar:7.3.1]

        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) ~[elasticsearch-7.3.1.jar:7.3.1]

        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.3.1.jar:7.3.1]

        ... 6 more

Caused by: java.io.IOException: failed to obtain lock on /var/lib/elasticsearch/nodes/0

        at org.elasticsearch.env.NodeEnvironment$NodeLock.<init>(NodeEnvironment.java:221) ~[elasticsearch-7.3.1.jar:7.3.1]

        at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:269) ~[elasticsearch-7.3.1.jar:7.3.1]

        at org.elasticsearch.node.Node.<init>(Node.java:278) ~[elasticsearch-7.3.1.jar:7.3.1]

        at org.elasticsearch.node.Node.<init>(Node.java:258) ~[elasticsearch-7.3.1.jar:7.3.1]

        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:221) ~[elasticsearch-7.3.1.jar:7.3.1]

        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-7.3.1.jar:7.3.1]

        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) ~[elasticsearch-7.3.1.jar:7.3.1]

        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.3.1.jar:7.3.1]

        ... 6 more

Caused by: java.nio.file.AccessDeniedException: /var/lib/elasticsearch/nodes/0/node.lock

        at sun.nio.fs.UnixException.translateToIOException(UnixException.java:90) ~[?:?]

        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]

        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116) ~[?:?]

        at sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:182) ~[?:?]

        at java.nio.channels.FileChannel.open(FileChannel.java:292) ~[?:?]

        at java.nio.channels.FileChannel.open(FileChannel.java:345) ~[?:?]

关键语句【maybe these locations are not writable or multiple nodes were started without increasing】

有可能是权限问题，给相关文件夹授权

# chmod -R 775 /var/lib/elasticsearch/

重启服务

# systemctl daemon-reload

# systemctl restart elasticsearch

搞定了！