说明：这只是本人在家里组网搭建的集群，所以内容排版可能比较随意了，没有耐心的同学直接跳过去看别的文章吧，我只是放在这里留作记录方便以后翻阅
ELK 的坑实在太多了，自己在物理机(多台)逐渐摸索的，安装最新版本的记录
为了好看可能我下面会出现多个这种的命令行
命令行 mkdir -p /iscsi/elk/kibana/config 命令行 mkdir -p /iscsi/elk/kibana/data 命令行 mkdir -p /iscsi/elk/kibana/logs # 可以使用 mkdir -p /iscsi/elk/kibana/{config,data,logs}代替在一个文件夹下创建多个目录```
ELK安装:
版本: 8.9.0
准备:
更新apt:
命令行 apt-get update
命令行 apt-get upgrade
命令行 apt-get install unzip
JDK安装:
- 1查看JDK版本: 命令行 apt search openjdk
- 2安装JDK: 命令行 apt install openjdk-21-jdk -y （找一个最高的来安装）
创建ELK目录:
命令行 mkdir -p /iscsi/elk
命令行 mkdir -p /iscsi/elk/elasticsearch
命令行 mkdir -p /iscsi/elk/elasticsearch/data
命令行 mkdir -p /iscsi/elk/elasticsearch/logs
命令行 mkdir -p /iscsi/elk/elasticsearch/plugins
命令行 mkdir -p /iscsi/elk/kibana
命令行 mkdir -p /iscsi/elk/kibana/config
命令行 mkdir -p /iscsi/elk/kibana/data
命令行 mkdir -p /iscsi/elk/kibana/logs
命令行 mkdir -p /iscsi/elk/logstash
命令行 mkdir -p /iscsi/elk/logstash/config
命令行 mkdir -p /iscsi/elk/logstash/logs
命令行 mkdir -p /iscsi/elk/logstash/pipeline
文件夹提权:
命令行 chmod 777 -R /iscsi/elk/*
创建用户和组:
1查看用户: 命令行 cat /etc/passwd
2查看组: 命令行 cat /etc/group
3修改原来uid为1000的用户: 命令行 usermod -u 1001 tonywoo
4新建elasticsearch 用户: 命令行 useradd -u 1000 -g root elasticsearch ??useradd -u 1000 -g tonywoo elasticsearch
6新建组: 命令行 groupadd elasticsearch
5修改elasticsearch 用户密码: 命令行 passwd elasticsearch
设置目录权限:
1设置目录拥有者: 命令行 chown -R elasticsearch:root /iscsi/elk/elasticsearch/
2设置目录拥有组: 命令行 chgrp -R 0 /iscsi/elk/elasticsearch
3备注: /iscsi/elk/elasticsearch 下的子目录的拥有者都要设置为 elasticsearch 这个用户
内存设置:
- 1查看用户内存权限: 命令行 sysctl -a|grep vm.max_map_count
- 2设置用户内存权限: 命令行 vim /etc/sysctl.conf
- 3禁止内存与硬盘交换: vm.swappiness=1
- 4配置最大映射数量: vm.max_map_count=262144
- 5使配置生效: 退出vim，命令行 sysctl -p
修改打开文件数:
- 1进入文件: 命令行 vim /etc/security/limits.conf
- 2追加内容:
`
# elasticsearch是用户，也可以使用*代替所有用户）
elasticsearch soft nofile 65536
elasticsearch hard nofile 65536
#内存锁定交换
soft memlock unlimited
hard memlock unlimited
`
查看docker网络: 命令行 docker network list
开始:
新建docker-compose文件: 命令行 touch /iscsi/elk/docker-compose-elk.yml
修改docker-compose文件如下:
`
version: '3.7'
services:
elasticsearch:
image: elasticsearch:8.9.0
container_name: elasticsearch
hostname: elasticsearch
restart: "no"
volumes:
- /etc/localtime:/etc/localtime
#- /iscsi/elk/elasticsearch/data:/usr/share/elasticsearch/data:rw
#- /iscsi/elk/elasticsearch/config:/usr/share/elasticsearch/config:rw
#- /iscsi/elk/elasticsearch/logs:/usr/share/elasticsearch/logs:rw
#- /iscsi/elk/elasticsearch/plugins:/usr/share/elasticsearch/plugins:rw
environment:
- TZ="Asia/Shanghai"
- ES_JAVA_OPTS=-Xms512m -Xmx512m
- discovery.type=single-node
ports:
- "9200:9200"
- "9300:9300"
networks:
elastic:
ipv4_address: 172.99.0.2
aliases:
- elasticsearch
kibana:
image: kibana:8.9.0
container_name: kibana
hostname: kibana
restart: "no"
volumes:
- /etc/localtime:/etc/localtime
#- /iscsi/elk/kibana/data:/usr/share/kibana/data:rw
#- /iscsi/elk/kibana/config:/usr/share/kibana/config:rw
#- /iscsi/elk/kibana/logs:/usr/share/kibana/logs:rw
ports:
- 5601:5601
depends_on:
- elasticsearch
networks:
elastic:
ipv4_address: 172.99.0.3
aliases:
- kibana
logstash:
image: logstash:8.9.0
container_name: logstash
hostname: logstash
restart: "no"
volumes:
- /etc/localtime:/etc/localtime
#- /iscsi/elk/logstash/config:/usr/share/logstash/config:rw
#- /iscsi/elk/logstash/logs:/usr/share/logstash/logs:rw
#- /iscsi/elk/logstash/pipeline:/usr/share/logstash/pipeline:rw
ports:
- 5044:5044
- 9066:9066
- 21068:21068
- "5000:5000/tcp"
- "5000:5000/udp"
depends_on:
- elasticsearch
networks:
elastic:
ipv4_address: 172.99.0.4
aliases:
- logstash
##自定义网络
networks:
elastic:
ipam:
driver: default
config:
- subnet: 172.99.0.0/16
`
上面带有#号 的后面要解除
首次启动: /iscsi/elk下执行 命令行 docker-compose -f docker-compose-elk.yml up -d
复制容器内目录到宿主机:
命令行 docker cp elasticsearch:/usr/share/elasticsearch/config /iscsi/elk/elasticsearch/
命令行 docker cp kibana:/usr/share/kibana/config /iscsi/elk/kibana/
命令行 docker cp logstash:/usr/share/logstash/config /iscsi/elk/logstash/
命令行 docker cp logstash:/usr/share/logstash/pipeline /iscsi/elk/log服务器托管stash/
命令行 chmod 777 -R /iscsi/elk/kibana/*
命令行 chmod 777 -R /iscsi/elk/logstash/*
命令行 chown -R elasticsearch:root /iscsi/elk/elasticsearch/config/
命令行 chown -R elasticsearch:root /iscsi/elk/elasticsearch/config/certs/
修改elasticsearch的jvm文件:
打开 /iscsi/elk/elasticsearch/config/jvm.options添加下面两项
命令行 vim /iscsi/elk/elasticsearch/config/jvm.options
`
-Xms512m
-Xmx512m
`
修改logstash的jvm文件:
打开 /iscsi/elk/logstash/config/jvm.options添加下面两项
命令行 vim /iscsi/elk/logstash/config/jvm.options 修改如下
#-Xms1g 改为 -Xms512m
#-Xmx1g 改为 -Xmx512m
放开注释: 放开docker-compose-elk.yml文件内挂载数据卷的注释
命令行 cd /iscsi/elk
命令行 docker-compose -f docker-compose-elk.yml stop
命令行 docker-compose -f docker-compose-elk.yml rm
命令行 docker-compose -f docker-compose-elk.yml up -d
配置SSL:
设置目录拥有者:
命令行 chown -R elasticsearch:root /iscsi/elk/elasticsearch/config/*
命令行 chown -R elasticsearch:root /iscsi/elk/elasticsearch/config/certs/*
进入elasticsearch容器:
命令行 docker exec -it elasticsearch /bin/bash
生成elastic-stack-ca.p12文件:
命令行 ./bin/elasticsearch-certutil ca
需要在 `Enter password for elastic-stack-ca.p12:` 哪里设置密码
生成elastic-certificates.p12:
命令行 ./bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12
`Enter password for CA(elastic-stack-ca.p12):`后输入 elastic-stack-ca.p12设置的密码
复制文件到config文件夹:
命令行 mv elastic-certificates.p12 config/certs/
命令行 mv elastic-stack-ca.p12 ./config/
备注 开放了docker-compose-elk.yml的注释复制文件到config宿主机文件夹会同时改变；elastic-stack-ca.p12文件后续也需要用到
设置文件拥有权:
退出容器
命令行 chmod 777 /iscsi/elk/elasticsearch/*
命令行 chmod 777 /iscsi/elk/elasticsearch/config/*
命令行 chmod 777 /iscsi/elk/elasticsearch/config/certs/*
命令行 chown -R elasticsearch:root /iscsi/elk/elasticsearch/*
设置elasticsearch.yml配置文件:
命令行 vim /iscsi/elk/elasticsearch/config/elasticsearch.yml
修改为如下
`
# Enable encryption and mutual authentication between cluster nodes
xpack.security.transport.ssl:
enabled: true
verification_mode: certificate
keystore.path: certs/elastic-certificates.p12
truststore.path: certs/elastic-certificates.p12
`
修改密码:
如果certificate设置了密码，需要执行一下两个命令
退回到容器根目录
命令行 ./bin/elasticsearch-keystore add xpack.security.transport.ssl.keystore.secure_password
命令行 ./bin/elasticsearch-keystore add xpack.security.transport.ssl.truststore.secure_password
备注 这个密码就是 elastic-certificates.p12 文件设置的密码
重启elasticsearch容器:
docker restart elasticsearch
配置elasticsearch和kibana开启https访问:
进入elasticsearch容器:
命令行 docker exec -it elasticsearch /bin/bash
生成elasticsearch-ssl-http.zip:
命令行 ./bin/elasticsearch-certutil http
操作如下
## Elasticsearch HTTP Certificate Utility
The 'http' command guides you through the process of generating certificates
for use on the HTTP (Rest) interface for Elasticsearch.
This tool will ask you a number of questions in order to generate the right
set of files for your needs.
## Do you wish to generate a Certificate Signing Request (CSR)?
A CSR is used when you want your certificate to be created by an existing
Certificate Authority (CA) that you do not control (that is, you don't have
access to the keys for that CA).
If you are in a corporate environment with a central security team, then you
may have an existing Corporate CA that can generate your certificate for you.
Infrastructure within your organisation may already be configured to trust this
CA, so it may be easier for clients to connect to Elasticsearch if you use a
CSR and send that request to the team that controls your CA.
If you choose not to generate a CSR, this tool will generate a new certificate
for you. That certificate will be signed by a CA under your control. This is a
quick and easy way to secure your cluster with TLS, but you will need to
configure all your clients to trust that custom CA.
## 生成CSR 输入n
Generate a CSR? [y/N]n
## Do you have an existing Certificate Authority (CA) key-pair that you wish to use to sign your certificate?
If you have an existing CA certificate and key, then you can use that CA to
sign your new http certificate. This allows you to use the same CA across
multiple Elasticsearch clusters which can make it easier to configure clients,
and may be easier for you to manage.
If you do not have an existing CA, one will be generated for you.
## 是否使用存在的ca 输入y（在基础配置时生成了）
Use an existing CA? [y/N]y
## What is the path to your CA?
Please enter the full pathname to the Certificate Authority that you wish to
use for signing your new http certificate. This can be in PKCS#12 (.p12), JKS
(.jks) or PEM (.crt, .key, .pem) format.
## 输入ca文件的地址
CA Path: /usr/share/elasticsearch/config/elastic-stack-ca.p12
Reading a PKCS12 keystore requires a password.
It is possible for the keystore's password to be blank,
in which case you can simply press > at the prompt
## 输入文件设置的密码
Password for elastic-stack-ca.p12:
## How long should your certificates be valid?
Every certificate has an expiry date. When the expiry date is reached clients
will stop trusting your certificate and TLS connections will fail.
Best practice suggests that you should either:
(a) set this to a short duration (90 - 120 days) and have automatic processes
to generate a new certificate before the old one expires, or
(b) set it to a longer duration (3 - 5 years) and then perform a manual update
a few months before it expires.
You may enter the validity period in years (e.g. 3Y), months (e.g. 18M), or days (e.g. 90D)
## 设置过期时间
For how long should your certificate be valid? [5y] 10y
## Do you wish to generate one certificate per node?
If you have multiple nodes in your cluster, then you may choose to generate a
separate certificate for each of these nodes. Each certificate will have its
own private key, and will be issued for a specific hostname or IP address.
Alternatively, you may wish to generate a single certificate that is valid
across all the hostnames or addresses in your cluster.
If all of your nodes will be accessed through a single domain
(e.g. node01.es.example.com, node02.es.example.com, etc) then you may find it
simpler to generate one certificate with a wildcard hostname (*.es.example.com)
and use that across all of your nodes.
However, if you do not have a common domain name, and you expect to add
additional nodes to your cluster in the future, then you should generate a
certificate per node so that you can more easily generate new certificates when
you provision new nodes.
## 是否为每一个节点生成证书 输入n
Generate a certificate per node? [y/N]n
## Which hostnames will be used to connect to your nodes?
These hostnames will be added as "DNS" names in the "Subject Alternative Name"
(SAN) field in your certificate.
You should list every hostname and variant that people will use to connect to
your cluster over http.
Do not list IP addresses here, you will be asked to enter them later.
If you wish to use a wildcard certificate (for example *.es.example.com) you
can enter that here.
## 节点的hostname,设置为elasticsearch，敲两次回车
Enter all the hostnames that you need, one per line.
When you are done, press > once more to move on to the next step.
elasticsearch
You entered the following hostnames.
- elasticsearch
## 配置是否正确
Is this correct [Y/n]y
## Which IP addresses will be used to connect to your nodes?
If your clients will ever connect to your nodes by numeric IP address, then you
can list these as valid IP "Subject Alternative Name" (SAN) fields in your
certificate.
If you do not have fixed IP addresses, or not wish to support direct IP access
to your cluster then you can just press > to skip this step.
服务器托管 ## 节点的ip(可以在宿主机通过命令docker inspect elasticsearch查看),设置为172.99.0.2，敲两次回车
Enter all the IP addresses that you need, one per line.
When you are done, press > once more to move on to the next step.
172.99.0.2
You entered the following IP addresses.
- 172.99.0.2
## 配置是否正确
Is this correct [Y/n]y
## Other certificate options
The generated certificate will have the following additional configuration
values. These values have been selected based on a combination of the
information you have provided above and secure defaults. You should not need to
change these values unless you have specific requirements.
Key Name: elasticsearch
Subject DN: CN=elasticsearch
Key Size: 2048
## 是否更改任意项
Do you wish to change any of these options? [y/N]n
## What password do you want for your private key(s)?
Your private key(s) will be stored in a PKCS#12 keystore file named "http.p12".
This type of keystore is always password protected, but it is possible to use a
blank password.
If you wish to use a blank password, simply press > at the prompt below.
## 输入生成文件的密码（可不设置，设置需要在后面进行配置）
Provide a password for the "http.p12" file: [> for none]
## 再次输入生成文件的密码
Repeat password to confirm:
## Where should we save the generated files?
A number of files will be generated including your private key(s),
public certificate(s), and sample configuration options for Elastic Stack products.
These files will be included in a single zip archive.
## 生成压缩文件的地址和名称，直接敲回车即可
What filename should be used for the output zip file? [/usr/share/elasticsearch/elasticsearch-ssl-http.zip]
移动elasticsearch-ssl-http.zip压缩包:
命令行 mv elasticsearch-ssl-http.zip ./config/
解压文件:
退出容器
命令行 unzip /iscsi/elk/elasticsearch/config/elasticsearch-ssl-http.zip
解压后会在原目录下新增两个目录分别是 elasticsearch 和 kibana
命令行 mv /iscsi/elk/elasticsearch/config/elasticsearch/http.p12 /iscsi/elk/elasticsearch/config/certs/
复制elasticsearch-ca.pem到kibana的config文件夹内:
命令行 cp /iscsi/elk/elasticsearch/config/kibana/elasticsearch-ca.pem /iscsi/elk/kibana/config/
删除文件夹:
命令行 rm -rf /iscsi/elk/elasticsearch/certs/kibana
文件提权:
命令行 chmod 777 /iscsi/elk/elasticsearch/config/certs/http.p12
设置http密码:
命令行 docker exec -it elasticsearch /bin/bash
命令行 ./bin/elasticsearch-keystore add xpack.security.http.ssl.keystore.secure_password
退出容器
重启elasticsearch容器:
命令行 docker restart elasticsearch
设置elastic用户的密码:
进入容器: 命令行 docker exec -it elasticsearch /bin/bash
设置密码: ./bin/elasticsearch-reset-password -u elastic -i
设置kibana_system密码:
命令行 ./bin/elasticsearch-reset-password -u kibana_system -i
生成kibana用https访问的公钥和私钥:
命令行 ./bin/elasticsearch-certutil csr -name kibana-server
备注 生成csr-bundle.zip文件夹
复制csr-bundle.zip到kibana:
退出容器
命令行 docker cp elasticsearch:/usr/share/elasticsearch/csr-bundle.zip /iscsi/elk/kibana/
解压csr-bundle.zip:
命令行 cd /iscsi/elk/kibana/
命令行 unzip /iscsi/elk/kibana/csr-bundle.zip
备注 解压后会生成 kibana-server 文件夹
移动文件到kibana的配置目录:
命令行 mv /iscsi/elk/kibana/kibana-server/* /iscsi/elk/kibana/config/
命令行 rm -rf /iscsi/elk/kibana/kibana-server
生成kibana-server.crt文件:
命令行 cd /iscsi/elk/kibana/config
命令行 openssl x509 -req -days 3650 -in kibana-server.csr -signkey kibana-server.key -out kibana-server.crt
文件提权:
命令行 chmod 777 elasticsearch-ca.pem kibana-server.csr kibana-server.key kibana-server.crt
修改kibana.yml文件:
命令行 vim /iscsi/elk/kibana/config/kibana.yml
最终文件为
`
#
# ** THIS IS AN AUTO-GENERATED FILE **
#
# Default Kibana configuration for docker target
server.host: "0.0.0.0"
server.shutdownTimeout: "5s"
elasticsearch.hosts: [ "https:/172.99.0.2:9200" ]
monitoring.ui.container.elasticsearch.enabled: true
elasticsearch.ssl.certificateAuthorities: ["/usr/share/kibana/config/elasticsearch-ca.pem"]
elasticsearch.username: "kibana_system"
elasticsearch.password: "tonglian@126.com"
server.ssl.certificate: "/usr/share/kibana/config/kibana-server.crt"
server.ssl.key: "/usr/share/kibana/config/kibana-server.key"
server.ssl.enabled: true
# 设置中文访问
i18n.locale: "zh-CN"
`
重启kiban:
命令行 docker restart kibana
配置logstash:
设置logstash_system密码:
命令行 docker exec -it elasticsearch /bin/bash
命令行 ./bin/elasticsearch-reset-password -u logstash_system -i
生成logstash.pem文件:
退出容器
命令行 openssl pkcs12 -in elasticsearch/config/certs/elastic-certificates.p12 -cacerts -nokeys -chain -out logstash.pem
移动logstash.pem文件到logstash配置文件目录下:
命令行 mv /iscsi/elk/elasticsearch/config/certs/logstash.pem /iscsi/elk/logstash/config/
提权logstash.pem:
命令行 chmod 777 /iscsi/elk/logstash/config/logstash.pem
配置logstash.yml文件:
命令行 vim /iscsi/elk/logstash/config/logstash.yml
最终文件
`
http.host: "0.0.0.0"
xpack.monitoring.elasticsearch.hosts: [ "https://172.99.0.2:9200" ]
#你的ca.pem 的所在路径
xpack.monitoring.elasticsearch.ssl.certificate_authority: "/usr/share/logstash/config/logstash.pem"
xpack.monitoring.elasticsearch.ssl.verification_mode: certificate
# 探嗅 es节点，设置为 false
xpack.monitoring.elasticsearch.sniffing: false
xpack.monitoring.elasticsearch.username: "logstash_system"
xpack.monitoring.elasticsearch.password: "tonglian@126.com"
`
配置logstash.conf文件:
命令行 vim /iscsi/elk/logstash/pipeline/logstash.conf
最终文件
`
input {
tcp {
port => 21068
codec => json_lines
}
}
output {
elasticsearch {
hosts => ["https://172.99.0.2:9200"]
index => "tonywoo-%{+YYYY.MM.dd}"
user => "elastic"
password => "tonglian@126.com"
ssl_enabled => true
ssl_certificate_authorities => ["/usr/share/logstash/config/logstash.pem"]
}
}
`
重启logstash:
docker restart logstash
设置自动启动:
`
cat > /etc/systemd/system/docker-compose-elk.service [Unit]
Description=Docker Compose Application Service
Requires=docker.service
After=docker.service
[Service]
Type=oneshot
RemainAfterExit=yes
WorkingDirectory=/iscsi/elk/
ExecStart=/iscsi/elk/docker-compose -f docker-compose-elk.yml up -d
ExecStop=/iscsi/elk/docker-compose -f docker-compose-elk.yml up down
TimeoutStartSec=0
[Install]
WantedBy=multi-user.target
`
回车
命令行 ctrl+d
服务器托管，北京服务器托管，服务器租用 http://www.fwqtg.net
