etcd监控
|
prometheus
|
25

1701 字
|
52 分钟

1 概述

<font color="#d83931">etcd是分布式系统中最关键数据的分布式可靠键值存储数据库</font>，重点是：
简单：定义明确、面向用户的API（gRPC）
安全：自动TLS，可选客户端证书身份验证
快速：以每秒10000次写入为基准
可靠：使用Raft正确分布
etcd是用Go编写的，并使用Raft共识算法来管理高可用的复制日志。

etcd被许多公司用于生产环境，开发团队在关键部署场景中支持它，其中etcd经常与Kubernetes、locksmith、vulcan、Doorman等应用程序合作。严格的稳健性测试进一步确保了可靠性。

etcd使用http协议工作，支持tls，生产环境中一般使用的是https访问。

etcdctl是一个用于与etcd服务器交互的命令行工具，用户大多通过输入或获取键的值与etcd进行交互。

etcdctl用来与etcd对话的API版本可以通过etcdctl_API环境变量设置为版本2或3。

默认情况下，"etcdctl 3.4+"的etcdctl使用v3 API，早期版本"etcdctl 3.3-"默认为v2 API。

值得注意的是，etcd v2版本写入的数据无法通过v3的版本进行查询，说白了，就是高版本不兼容低版本，相当于v3重写了etcd。

生产环境中，官方推荐使用v3版本。

1.1  Etcd 集群架构

1.1.1 集群组成

  • Leader：负责处理所有写请求，并同步数据到 Follower。

  • Follower：接收 Leader 的数据同步，可处理读请求。

  • Candidate：选举过程中的临时状态（Leader 宕机时触发）。

1.1.2 Raft 选举机制

  • Term（任期）：每个 Leader 任期唯一，递增。

  • 选举流程：

    1. Follower 检测 Leader 心跳超时 → 变为 Candidate。

    2. Candidate 发起投票，获得多数节点（N/2+1）同意后成为 Leader。

    3. 新 Leader 同步日志，维持心跳。

1.1.3 数据同步

  • Log Replication：

    • 客户端写请求 → Leader 写入日志 → 同步到 Follower → 提交（Commit）。

  • ReadIndex / LeaseRead：

    • 读请求可由 Follower 处理（需确保数据最新）。
      注意

  • Kubernetes 所有资源（Pods、Services）存储在 Etcd。

  • 如果 Etcd 故障，K8s 控制面将不可用。

2 集群部署

- etcd分布式集群搭建及高可用验证
推荐阅读：
	https://github.com/etcd-io/etcd
	https://etcd.io/docs/

etcd的双端口作用:
	- 2379: HTTP|HTTPS 
		对客户端使用。
		
	- 2380: TCP 
		etcd集群内部使用。
	2.安装etcd的方式
		包管理方式安装： 安装简单，但是版本低
apt -y install etcd-client
	
		二进制方式安装：可以安装任意版本（参考etcd相关的安装笔记即可）
https://www.cnblogs.com/cmy/p/18265994

 
	3.准备etcd程序包
		3.1 下载etcd的软件包
wget https://github.com/etcd-io/etcd/releases/download/v3.5.21/etcd-v3.5.21-linux-amd64.tar.gz

svip:
[root@node-exporter41 ~]# wget http://192.168.14.253/Resources/Prometheus/softwares/Etcd/etcd-v3.5.21-linux-amd64.tar.gz

		3.2 解压etcd的二进制程序包到PATH环境变量路径
[root@node-exporter41 ~]# tar -xf etcd-v3.5.21-linux-amd64.tar.gz -C /usr/local/bin etcd-v3.5.21-linux-amd64/etcd{,ctl}  --strip-components=1 
[root@node-exporter41 ~]# 
[root@node-exporter41 ~]# ll /usr/local/bin/etcd*
-rwxr-xr-x 1 cmy cmy 24072344 Mar 28 06:58 /usr/local/bin/etcd*
-rwxr-xr-x 1 cmy cmy 18419864 Mar 28 06:58 /usr/local/bin/etcdctl*
[root@node-exporter41 ~]# 
[root@node-exporter41 ~]# etcdctl version
etcdctl version: 3.5.21
API version: 3.5
		3.3 将软件包下发到所有节点
[root@node-exporter41 ~]# scp /usr/local/bin/etcd* 10.168.10.42:/usr/local/bin

[root@node-exporter41 ~]# scp /usr/local/bin/etcd* 10.168.10.43:/usr/local/bin


	4.准备etcd的证书文件
		4.1 安装cfssl证书管理工具
[root@node-exporter41 ~]# wget http://192.168.14.253/Resources/Prometheus/softwares/Etcd/cmy-cfssl-v1.6.5.zip
[root@node-exporter41 ~]#
[root@node-exporter41 ~]# unzip cmy-cfssl-v1.6.5.zip 
[root@node-exporter41 ~]#
[root@node-exporter41 ~]# apt install rename
[root@node-exporter41 ~]# 
[root@node-exporter41 ~]# rename -v "s/_1.6.5_linux_amd64//g" cfssl*
[root@node-exporter41 ~]# 
[root@node-exporter41 ~]# mv cfssl* /usr/local/bin/
[root@node-exporter41 ~]# 
[root@node-exporter41 ~]# chmod +x /usr/local/bin/cfssl*
[root@node-exporter41 ~]# 
[root@node-exporter41 ~]# ll /usr/local/bin/cfssl*
-rwxr-xr-x 1 root root 11890840 Jun 15 11:56 /usr/local/bin/cfssl*
-rwxr-xr-x 1 root root  8413336 Jun 15 11:56 /usr/local/bin/cfssl-certinfo*
-rwxr-xr-x 1 root root  6205592 Jun 15 11:56 /usr/local/bin/cfssljson*
[root@node-exporter41 ~]# 

		4.2 创建证书存储目录
[root@node-exporter41 ~]# mkdir -pv /cmy/certs/etcd && cd /cmy/certs/etcd/


		4.3 生成证书的CSR文件： 证书签发请求文件，配置了一些域名，公司，单位
[root@node-exporter41 etcd]# cat > etcd-ca-csr.json <<EOF
{
  "CN": "etcd",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Beijing",
      "L": "Beijing",
      "O": "etcd",
      "OU": "Etcd Security"
    }
  ],
  "ca": {
    "expiry": "876000h"
  }
}
EOF


		4.4 生成etcd CA证书和CA证书的key
[root@node-exporter41 etcd]# cfssl gencert -initca etcd-ca-csr.json | cfssljson -bare /cmy/certs/etcd/etcd-ca
[root@node-exporter41 etcd]# 
[root@node-exporter41 etcd]# ll /cmy/certs/etcd/etcd-ca*
-rw-r--r-- 1 root root 1050 Nov 15 10:42 /cmy/certs/etcd/etcd-ca.csr
-rw-r--r-- 1 root root  249 Nov 15 10:42 /cmy/certs/etcd/etcd-ca-csr.json
-rw------- 1 root root 1675 Nov 15 10:42 /cmy/certs/etcd/etcd-ca-key.pem
-rw-r--r-- 1 root root 1318 Nov 15 10:42 /cmy/certs/etcd/etcd-ca.pem
[root@node-exporter41 etcd]# 


		4.5 生成etcd证书的有效期为100年
[root@node-exporter41 etcd]# cat > ca-config.json <<EOF
{
  "signing": {
    "default": {
      "expiry": "876000h"
    },
    "profiles": {
      "kubernetes": {
        "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ],
        "expiry": "876000h"
      }
    }
  }
}
EOF


		4.6 生成证书的CSR文件： 证书签发请求文件，配置了一些域名，公司，单位
[root@node-exporter41 etcd]# cat > etcd-csr.json <<EOF
{
  "CN": "etcd",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "Beijing",
      "L": "Beijing",
      "O": "etcd",
      "OU": "Etcd Security"
    }
  ]
}
EOF


		4.7 基于自建的ectd ca证书生成etcd的证书
[root@node-exporter41 etcd]# cfssl gencert \
  -ca=/cmy/certs/etcd/etcd-ca.pem \
  -ca-key=/cmy/certs/etcd/etcd-ca-key.pem \
  -config=ca-config.json \
  --hostname=127.0.0.1,node-exporter41,node-exporter42,node-exporter43,10.168.10.41,10.168.10.42,10.168.10.43 \
  --profile=kubernetes \
  etcd-csr.json  | cfssljson -bare /cmy/certs/etcd/etcd-server

[root@k8s-master01 pki]# ll /cmy/certs/etcd/etcd-server*
-rw-r--r-- 1 root root 1131 Jun 24 15:18 /cmy/certs/etcd/etcd-server.csr
-rw------- 1 root root 1679 Jun 24 15:18 /cmy/certs/etcd/etcd-server-key.pem
-rw-r--r-- 1 root root 1464 Jun 24 15:18 /cmy/certs/etcd/etcd-server.pem
[root@k8s-master01 pki]# 


		4.8 将etcd证书拷贝到其他两个master节点
[root@node-exporter41 etcd]# scp -r /cmy/certs/ 10.168.10.42:/cmy

[root@node-exporter41 etcd]# scp -r /cmy/certs/ 10.168.10.43:/cmy

[root@node-exporter42 ~]# ll /cmy/certs/etcd/
total 44
drwxr-xr-x 2 root root 4096 Nov 15 10:49 ./
drwxr-xr-x 3 root root 4096 Nov 15 10:49 ../
-rw-r--r-- 1 root root  294 Nov 15 10:49 ca-config.json
-rw-r--r-- 1 root root 1050 Nov 15 10:49 etcd-ca.csr
-rw-r--r-- 1 root root  249 Nov 15 10:49 etcd-ca-csr.json
-rw------- 1 root root 1675 Nov 15 10:49 etcd-ca-key.pem
-rw-r--r-- 1 root root 1318 Nov 15 10:49 etcd-ca.pem
-rw-r--r-- 1 root root  210 Nov 15 10:49 etcd-csr.json
-rw-r--r-- 1 root root 1143 Nov 15 10:49 etcd-server.csr
-rw------- 1 root root 1679 Nov 15 10:49 etcd-server-key.pem
-rw-r--r-- 1 root root 1476 Nov 15 10:49 etcd-server.pem
[root@node-exporter42 ~]# 

[root@node-exporter43 ~]# ll /cmy/certs/etcd/
total 44
drwxr-xr-x 2 root root 4096 Nov 15 10:49 ./
drwxr-xr-x 3 root root 4096 Nov 15 10:49 ../
-rw-r--r-- 1 root root  294 Nov 15 10:49 ca-config.json
-rw-r--r-- 1 root root 1050 Nov 15 10:49 etcd-ca.csr
-rw-r--r-- 1 root root  249 Nov 15 10:49 etcd-ca-csr.json
-rw------- 1 root root 1675 Nov 15 10:49 etcd-ca-key.pem
-rw-r--r-- 1 root root 1318 Nov 15 10:49 etcd-ca.pem
-rw-r--r-- 1 root root  210 Nov 15 10:49 etcd-csr.json
-rw-r--r-- 1 root root 1143 Nov 15 10:49 etcd-server.csr
-rw------- 1 root root 1679 Nov 15 10:49 etcd-server-key.pem
-rw-r--r-- 1 root root 1476 Nov 15 10:49 etcd-server.pem
[root@node-exporter43 ~]# 
[root@node-exporter43 ~]# 


	5.创建etcd集群各节点配置文件
		5.1 node-exporter41节点的配置文件
[root@node-exporter41 ~]# mkdir -pv /cmy/softwares/etcd
[root@node-exporter41 ~]# 
[root@node-exporter41 ~]# cat > /cmy/softwares/etcd/etcd.config.yml <<'EOF'
name: 'node-exporter41'
data-dir: /var/lib/etcd
wal-dir: /var/lib/etcd/wal
snapshot-count: 5000
heartbeat-interval: 100
election-timeout: 1000
quota-backend-bytes: 0
listen-peer-urls: 'https://10.168.10.41:2380'
listen-client-urls: 'https://10.168.10.41:2379,http://127.0.0.1:2379'
max-snapshots: 3
max-wals: 5
cors:
initial-advertise-peer-urls: 'https://10.168.10.41:2380'
advertise-client-urls: 'https://10.168.10.41:2379'
discovery:
discovery-fallback: 'proxy'
discovery-proxy:
discovery-srv:
initial-cluster: 'node-exporter41=https://10.168.10.41:2380,node-exporter42=https://10.168.10.42:2380,node-exporter43=https://10.168.10.43:2380'
initial-cluster-token: 'etcd-k8s-cluster'
initial-cluster-state: 'new'
strict-reconfig-check: false
enable-v2: true
enable-pprof: true
proxy: 'off'
proxy-failure-wait: 5000
proxy-refresh-interval: 30000
proxy-dial-timeout: 1000
proxy-write-timeout: 5000
proxy-read-timeout: 0
client-transport-security:
  cert-file: '/cmy/certs/etcd/etcd-server.pem'
  key-file: '/cmy/certs/etcd/etcd-server-key.pem'
  client-cert-auth: true
  trusted-ca-file: '/cmy/certs/etcd/etcd-ca.pem'
  auto-tls: true
peer-transport-security:
  cert-file: '/cmy/certs/etcd/etcd-server.pem'
  key-file: '/cmy/certs/etcd/etcd-server-key.pem'
  peer-client-cert-auth: true
  trusted-ca-file: '/cmy/certs/etcd/etcd-ca.pem'
  auto-tls: true
debug: false
log-package-levels:
log-outputs: [default]
force-new-cluster: false
EOF



		5.2 node-exporter42节点的配置文件
[root@node-exporter42 ~]# mkdir -pv /cmy/softwares/etcd
[root@node-exporter42 ~]# 
[root@node-exporter42 ~]# cat > /cmy/softwares/etcd/etcd.config.yml <<'EOF'
name: 'node-exporter42'
data-dir: /var/lib/etcd
wal-dir: /var/lib/etcd/wal
snapshot-count: 5000
heartbeat-interval: 100
election-timeout: 1000
quota-backend-bytes: 0
listen-peer-urls: 'https://10.168.10.42:2380'
listen-client-urls: 'https://10.168.10.42:2379,http://127.0.0.1:2379'
max-snapshots: 3
max-wals: 5
cors:
initial-advertise-peer-urls: 'https://10.168.10.42:2380'
advertise-client-urls: 'https://10.168.10.42:2379'
discovery:
discovery-fallback: 'proxy'
discovery-proxy:
discovery-srv:
initial-cluster: 'node-exporter41=https://10.168.10.41:2380,node-exporter42=https://10.168.10.42:2380,node-exporter43=https://10.168.10.43:2380'
initial-cluster-token: 'etcd-k8s-cluster'
initial-cluster-state: 'new'
strict-reconfig-check: false
enable-v2: true
enable-pprof: true
proxy: 'off'
proxy-failure-wait: 5000
proxy-refresh-interval: 30000
proxy-dial-timeout: 1000
proxy-write-timeout: 5000
proxy-read-timeout: 0
client-transport-security:
  cert-file: '/cmy/certs/etcd/etcd-server.pem'
  key-file: '/cmy/certs/etcd/etcd-server-key.pem'
  client-cert-auth: true
  trusted-ca-file: '/cmy/certs/etcd/etcd-ca.pem'
  auto-tls: true
peer-transport-security:
  cert-file: '/cmy/certs/etcd/etcd-server.pem'
  key-file: '/cmy/certs/etcd/etcd-server-key.pem'
  peer-client-cert-auth: true
  trusted-ca-file: '/cmy/certs/etcd/etcd-ca.pem'
  auto-tls: true
debug: false
log-package-levels:
log-outputs: [default]
force-new-cluster: false
EOF


		5.3 node-exporter43节点的配置文件
[root@node-exporter43 ~]# mkdir -pv /cmy/softwares/etcd
[root@node-exporter43 ~]# 
[root@node-exporter43 ~]# cat > /cmy/softwares/etcd/etcd.config.yml <<'EOF'
name: 'node-exporter43'
data-dir: /var/lib/etcd
wal-dir: /var/lib/etcd/wal
snapshot-count: 5000
heartbeat-interval: 100
election-timeout: 1000
quota-backend-bytes: 0
listen-peer-urls: 'https://10.168.10.43:2380'
listen-client-urls: 'https://10.168.10.43:2379,http://127.0.0.1:2379'
max-snapshots: 3
max-wals: 5
cors:
initial-advertise-peer-urls: 'https://10.168.10.43:2380'
advertise-client-urls: 'https://10.168.10.43:2379'
discovery:
discovery-fallback: 'proxy'
discovery-proxy:
discovery-srv:
initial-cluster: 'node-exporter41=https://10.168.10.41:2380,node-exporter42=https://10.168.10.42:2380,node-exporter43=https://10.168.10.43:2380'
initial-cluster-token: 'etcd-k8s-cluster'
initial-cluster-state: 'new'
strict-reconfig-check: false
enable-v2: true
enable-pprof: true
proxy: 'off'
proxy-failure-wait: 5000
proxy-refresh-interval: 30000
proxy-dial-timeout: 1000
proxy-write-timeout: 5000
proxy-read-timeout: 0
client-transport-security:
  cert-file: '/cmy/certs/etcd/etcd-server.pem'
  key-file: '/cmy/certs/etcd/etcd-server-key.pem'
  client-cert-auth: true
  trusted-ca-file: '/cmy/certs/etcd/etcd-ca.pem'
  auto-tls: true
peer-transport-security:
  cert-file: '/cmy/certs/etcd/etcd-server.pem'
  key-file: '/cmy/certs/etcd/etcd-server-key.pem'
  peer-client-cert-auth: true
  trusted-ca-file: '/cmy/certs/etcd/etcd-ca.pem'
  auto-tls: true
debug: false
log-package-levels:
log-outputs: [default]
force-new-cluster: false
EOF

	6.所有节点编写etcd启动脚本
cat > /usr/lib/systemd/system/etcd.service <<'EOF'
[Unit]
Description=Jason Yin's Etcd Service
Documentation=https://coreos.com/etcd/docs/latest/
After=network.target

[Service]
Type=notify
ExecStart=/usr/local/bin/etcd --config-file=/cmy/softwares/etcd/etcd.config.yml
Restart=on-failure
RestartSec=10
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
Alias=etcd3.service
EOF


	7.所有节点启动etcd集群
systemctl daemon-reload && systemctl enable --now etcd
systemctl status etcd


	8.查看etcd集群状态
[root@node-exporter43 ~]# etcdctl --endpoints="https://10.168.10.41:2379,https://10.168.10.42:2379,https://10.168.10.43:2379" --cacert=/cmy/certs/etcd/etcd-ca.pem --cert=/cmy/certs/etcd/etcd-server.pem --key=/cmy/certs/etcd/etcd-server-key.pem  endpoint status --write-out=table
+------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|        ENDPOINT        |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| https://10.168.10.41:2379 | 9378902f41df91e9 |  3.5.21 |   20 kB |     false |      false |         2 |          9 |                  9 |        |
| https://10.168.10.42:2379 | 18f972748ec1bd96 |  3.5.21 |   20 kB |      true |      false |         2 |          9 |                  9 |        |
| https://10.168.10.43:2379 | a3dfd2d37c461ee9 |  3.5.21 |   20 kB |     false |      false |         2 |          9 |                  9 |        |
+------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
[root@node-exporter43 ~]# 




	9.验证etcd高可用集群
		9.1 停止leader节点
[root@node-exporter42 ~]# ss -ntl | egrep "2379|2380"
LISTEN 0      4096       127.0.0.1:2379      0.0.0.0:*          
LISTEN 0      4096       10.168.10.42:2379      0.0.0.0:*          
LISTEN 0      4096       10.168.10.42:2380      0.0.0.0:*          
[root@node-exporter42 ~]# 
[root@node-exporter42 ~]# systemctl stop etcd
[root@node-exporter42 ~]# 
[root@node-exporter42 ~]# ss -ntl | egrep "2379|2380"
[root@node-exporter42 ~]# 


		9.2 查看现有集群环境，发现新leader诞生
[root@node-exporter43 ~]# etcdctl --endpoints="https://10.168.10.41:2379,https://10.168.10.42:2379,https://10.168.10.43:2379" --cacert=/cmy/certs/etcd/etcd-ca.pem --cert=/cmy/certs/etcd/etcd-server.pem --key=/cmy/certs/etcd/etcd-server-key.pem  endpoint status --write-out=table
{"level":"warn","ts":"2025-05-15T11:54:52.261675+0800","logger":"etcd-client","caller":"v3@v3.5.21/retry_interceptor.go:63","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00035a000/10.168.10.41:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 10.168.10.42:2379: connect: connection refused\""}
Failed to get the status of endpoint https://10.168.10.42:2379 (context deadline exceeded)
+------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|        ENDPOINT        |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| https://10.168.10.41:2379 | 9378902f41df91e9 |  3.5.21 |   20 kB |      true |      false |         3 |         10 |                 10 |        |
| https://10.168.10.43:2379 | a3dfd2d37c461ee9 |  3.5.21 |   20 kB |     false |      false |         3 |         10 |                 10 |        |
+------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
[root@node-exporter43 ~]# 
[root@node-exporter43 ~]# 
[root@node-exporter43 ~]# 
[root@node-exporter43 ~]# etcdctl --endpoints="https://10.168.10.41:2379,https://10.168.10.43:2379" --cacert=/cmy/certs/etcd/etcd-ca.pem --cert=/cmy/certs/etcd/etcd-server.pem --key=/cmy/certs/etcd/etcd-server-key.pem  endpoint status --write-out=table
+------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|        ENDPOINT        |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| https://10.168.10.41:2379 | 9378902f41df91e9 |  3.5.21 |   20 kB |      true |      false |         3 |         10 |                 10 |        |
| https://10.168.10.43:2379 | a3dfd2d37c461ee9 |  3.5.21 |   20 kB |     false |      false |         3 |         10 |                 10 |        |
+------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
[root@node-exporter43 ~]# 


		9.3 再将之前的leader起来
[root@node-exporter42 ~]# ss -ntl | egrep "2379|2380"
[root@node-exporter42 ~]# 
[root@node-exporter42 ~]# 
[root@node-exporter42 ~]# systemctl start etcd
[root@node-exporter42 ~]# 
[root@node-exporter42 ~]# ss -ntl | egrep "2379|2380"
LISTEN 0      4096       127.0.0.1:2379      0.0.0.0:*          
LISTEN 0      4096       10.168.10.42:2379      0.0.0.0:*          
LISTEN 0      4096       10.168.10.42:2380      0.0.0.0:*          
[root@node-exporter42 ~]# 
[root@node-exporter42 ~]# 
[root@node-exporter42 ~]# etcdctl --endpoints="https://10.168.10.41:2379,https://10.168.10.42:2379,https://10.168.10.43:2379" --cacert=/cmy/certs/etcd/etcd-ca.pem --cert=/cmy/certs/etcd/etcd-server.pem --key=/cmy/certs/etcd/etcd-server-key.pem  endpoint status --write-out=table
+------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|        ENDPOINT        |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| https://10.168.10.41:2379 | 9378902f41df91e9 |  3.5.21 |   20 kB |      true |      false |         3 |         11 |                 11 |        |
| https://10.168.10.42:2379 | 18f972748ec1bd96 |  3.5.21 |   20 kB |     false |      false |         3 |         11 |                 11 |        |
| https://10.168.10.43:2379 | a3dfd2d37c461ee9 |  3.5.21 |   20 kB |     false |      false |         3 |         11 |                 11 |        |
+------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
[root@node-exporter42 ~]# 


	10.添加别名
		10.1 添加别名
[root@node-exporter41 ~]# vim .bashrc 
...
alias etcdctl='etcdctl --endpoints="10.168.10.41:2379,10.168.10.42:2379,10.168.10.43:2379" --cacert=/cmy/certs/etcd/etcd-ca.pem --cert=/cmy/certs/etcd/etcd-server.pem --key=/cmy/certs/etcd/etcd-server-key.pem ' 

...
[root@node-exporter41 ~]# source .bashrc
[root@node-exporter41 ~]# 
[root@node-exporter41 ~]# etcdctl endpoint status --write-out=table
+----------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|    ENDPOINT    |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+----------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| 10.168.10.41:2379 | 9378902f41df91e9 |  3.5.21 |   20 kB |      true |      false |         3 |         11 |                 11 |        |
| 10.168.10.42:2379 | 18f972748ec1bd96 |  3.5.21 |   20 kB |     false |      false |         3 |         11 |                 11 |        |
| 10.168.10.43:2379 | a3dfd2d37c461ee9 |  3.5.21 |   20 kB |     false |      false |         3 |         11 |                 11 |        |
+----------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
[root@node-exporter41 ~]# 
[root@node-exporter41 ~]# scp .bashrc 10.168.10.42:~
[root@node-exporter41 ~]#
[root@node-exporter41 ~]# scp .bashrc 10.168.10.43:~


		10.2 测试验证 【需要断开重连】
[root@node-exporter42 ~]# etcdctl endpoint status --write-out=table
+----------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|    ENDPOINT    |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+----------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| 10.168.10.41:2379 | 9378902f41df91e9 |  3.5.21 |   20 kB |     false |      false |         3 |         11 |                 11 |        |
| 10.168.10.42:2379 | 18f972748ec1bd96 |  3.5.21 |   20 kB |      true |      false |         3 |         11 |                 11 |        |
| 10.168.10.43:2379 | a3dfd2d37c461ee9 |  3.5.21 |   20 kB |     false |      false |         3 |         11 |                 11 |        |
+----------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
[root@node-exporter42 ~]# 


[root@node-exporter43 ~]# etcdctl endpoint status --write-out=table
+----------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|    ENDPOINT    |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+----------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| 10.168.10.41:2379 | 9378902f41df91e9 |  3.5.21 |   20 kB |     false |      false |         3 |         11 |                 11 |        |
| 10.168.10.42:2379 | 18f972748ec1bd96 |  3.5.21 |   20 kB |      true |      false |         3 |         11 |                 11 |        |
| 10.168.10.43:2379 | a3dfd2d37c461ee9 |  3.5.21 |   20 kB |     false |      false |         3 |         11 |                 11 |        |
+----------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
[root@node-exporter43 ~]#

3 etcd基本操作

	1.etcd基础操作概述
etcd的操作和zookeeper，Redis的操作类似，存储数据都是键值对。


	2.etcd增删改查基础操作
		2.1 写入数据KEY的school，value等于cmy
[root@node-exporter41 ~]# etcdctl put school cmy
OK
[root@node-exporter41 ~]# 
[root@node-exporter41 ~]# etcdctl put /class cmy
OK
[root@node-exporter41 ~]# 
[root@node-exporter41 ~]# etcdctl put classroom 教室5
OK
[root@node-exporter41 ~]# 
[root@node-exporter41 ~]# etcdctl put /etc/hosts 10.168.10.141 ceph141
OK
[root@node-exporter41 ~]# 
[root@node-exporter41 ~]# etcdctl put /cmy/docker/registry 轻量级镜像仓库
OK
[root@node-exporter41 ~]# 



		2.2 查看数据
[root@node-exporter41 ~]# etcdctl get school
school
cmy
[root@node-exporter41 ~]# 
[root@node-exporter41 ~]# etcdctl get school --keys-only
school

[root@node-exporter41 ~]# 
[root@node-exporter41 ~]# etcdctl get school --print-value-only
cmy
[root@node-exporter41 ~]# 
[root@node-exporter41 ~]# etcdctl get / --prefix --keys-only
/class

/etc/hosts

/cmy/docker/registry

[root@node-exporter41 ~]# 
[root@node-exporter41 ~]# etcdctl get / --prefix --print-value-only
cmy
10.168.10.141
轻量级镜像仓库
[root@node-exporter41 ~]# 
[root@node-exporter41 ~]# etcdctl get "" --prefix --keys-only  # 查看etcd所有数据
/class

/etc/hosts

/cmy/docker/registry

classroom

school

[root@node-exporter41 ~]# 
[root@node-exporter41 ~]# etcdctl get "" --prefix --print-value-only
cmy
10.168.10.141
轻量级镜像仓库
教室5
cmy
[root@node-exporter41 ~]# 
[root@node-exporter41 ~]# etcdctl get "" --prefix 
/class
cmy
/etc/hosts
10.168.10.141
/cmy/docker/registry
轻量级镜像仓库
classroom
教室5
school
cmy
[root@node-exporter41 ~]# 



		2.3 修改数据
[root@node-exporter42 ~]# etcdctl get school --print-value-only
cmy
[root@node-exporter42 ~]# 
[root@node-exporter41 ~]# etcdctl put school cmy
OK
[root@node-exporter41 ~]# 
[root@node-exporter41 ~]# etcdctl get school --print-value-only
cmy
[root@node-exporter41 ~]# 


	
		2.4 删除数据 
[root@node-exporter41 ~]# etcdctl get "" --prefix --keys-only
/class

/etc/hosts

/cmy/docker/registry

classroom

school

[root@node-exporter41 ~]# 
[root@node-exporter41 ~]# etcdctl del school
1
[root@node-exporter41 ~]# 
[root@node-exporter41 ~]# etcdctl del / --prefix
3
[root@node-exporter41 ~]# 
[root@node-exporter41 ~]# etcdctl get "" --prefix --keys-only
classroom

[root@node-exporter41 ~]#

4 集群备份与恢复

4.1 备份数据

1 原生数据【数据随机创建即可，用于模拟备份环节】
[root@node-exporter41 ~]# etcdctl get "" --prefix 
/cmy/cmy/first
陈梦元
/cmy/cmy/fourth
徐灵杰
/cmy/cmy/second
任鹏宇
/cmy/cmy/third
贾震
class
cmy
classroom
教室5
school
cmy
[root@node-exporter41 ~]# 



	2 创建快照用于备份数据
[root@node-exporter41 ~]# \etcdctl snapshot save /tmp/cmy-etcd-`date +%F`.backup
{"level":"info","ts":"2025-05-15T14:54:56.340375+0800","caller":"snapshot/v3_snapshot.go:65","msg":"created temporary db file","path":"/tmp/cmy-etcd-2025-05-15.backup.part"}
{"level":"info","ts":"2025-05-15T14:54:56.341243+0800","logger":"client","caller":"v3@v3.5.21/maintenance.go:212","msg":"opened snapshot stream; downloading"}
{"level":"info","ts":"2025-05-15T14:54:56.341274+0800","caller":"snapshot/v3_snapshot.go:73","msg":"fetching snapshot","endpoint":"127.0.0.1:2379"}
{"level":"info","ts":"2025-05-15T14:54:56.343658+0800","logger":"client","caller":"v3@v3.5.21/maintenance.go:220","msg":"completed snapshot read; closing"}
{"level":"info","ts":"2025-05-15T14:54:56.344576+0800","caller":"snapshot/v3_snapshot.go:88","msg":"fetched snapshot","endpoint":"127.0.0.1:2379","size":"20 kB","took":"now"}
{"level":"info","ts":"2025-05-15T14:54:56.344728+0800","caller":"snapshot/v3_snapshot.go:97","msg":"saved","path":"/tmp/cmy-etcd-2025-05-15.backup"}
Snapshot saved at /tmp/cmy-etcd-2025-05-15.backup
[root@node-exporter41 ~]# 
[root@node-exporter41 ~]# ll /tmp/cmy-etcd-`date +%F`.backup
-rw------- 1 root root 20512 May 15 14:54 /tmp/cmy-etcd-2025-05-15.backup
[root@node-exporter41 ~]# 

	3.查看快照的基本信息
[root@node-exporter41 ~]# etcdctl snapshot status /tmp/cmy-etcd-`date +%F`.backup -w table   # 查看官网文档的状态
Deprecated: Use `etcdutl snapshot status` instead.

+----------+----------+------------+------------+
|   HASH   | REVISION | TOTAL KEYS | TOTAL SIZE |
+----------+----------+------------+------------+
| 1583e0e8 |       15 |         24 |      20 kB |
+----------+----------+------------+------------+
[root@node-exporter41 ~]# 

	4.将快照拷贝到其他两个集群节点
[root@node-exporter41 ~]# scp /tmp/cmy-etcd-`date +%F`.backup 10.168.10.42:/tmp
[root@node-exporter41 ~]# scp /tmp/cmy-etcd-`date +%F`.backup 10.168.10.43:/tmp

4.2 删除数据


	5 删除所有数据【搞破坏】
[root@node-exporter41 ~]# etcdctl get "" --prefix 
/cmy/cmy/first
陈梦元
/cmy/cmy/fourth
徐灵杰
/cmy/cmy/second
任鹏宇
/cmy/cmy/third
贾震
class
cmy
classroom
教室5
school
cmy
[root@node-exporter41 ~]# 
[root@node-exporter41 ~]# etcdctl del "" --prefix 
7
[root@node-exporter41 ~]# 
[root@node-exporter41 ~]# etcdctl get "" --prefix 
[root@node-exporter41 ~]#

4.3 恢复数据

6 停止etcd集群
[root@node-exporter41 ~]# systemctl stop  etcd
[root@node-exporter42 ~]# systemctl stop  etcd
[root@node-exporter43 ~]# systemctl stop  etcd


	7.各节点恢复数据 【恢复的数据目录必须为空】
[root@node-exporter41 ~]# etcdctl snapshot restore /tmp/cmy-etcd-`date +%F`.backup --data-dir=/var/lib/etcd-2025
Deprecated: Use `etcdutl snapshot restore` instead.

2025-05-15T14:59:18+08:00	info	snapshot/v3_snapshot.go:265	restoring snapshot	{"path": "/tmp/cmy-etcd-2025-05-15.backup", "wal-dir": "/var/lib/etcd-2025/member/wal", "data-dir": "/var/lib/etcd-2025", "snap-dir": "/var/lib/etcd-2025/member/snap", "initial-memory-map-size": 0}
2025-05-15T14:59:18+08:00	info	membership/store.go:138	Trimming membership information from the backend...
2025-05-15T14:59:18+08:00	info	membership/cluster.go:421	added member	{"cluster-id": "cdf818194e3a8c32", "local-member-id": "0", "added-peer-id": "8e9e05c52164694d", "added-peer-peer-urls": ["http://localhost:2380"], "added-peer-is-learner": false}
2025-05-15T14:59:18+08:00	info	snapshot/v3_snapshot.go:293	restored snapshot	{"path": "/tmp/cmy-etcd-2025-05-15.backup", "wal-dir": "/var/lib/etcd-2025/member/wal", "data-dir": "/var/lib/etcd-2025", "snap-dir": "/var/lib/etcd-2025/member/snap", "initial-memory-map-size": 0}
[root@node-exporter41 ~]# 



[root@node-exporter42 ~]# etcdctl snapshot restore /tmp/cmy-etcd-`date +%F`.backup --data-dir=/var/lib/etcd-2025
Deprecated: Use `etcdutl snapshot restore` instead.

2025-05-15T14:59:42+08:00	info	snapshot/v3_snapshot.go:265	restoring snapshot	{"path": "/tmp/cmy-etcd-2025-05-15.backup", "wal-dir": "/var/lib/etcd-2025/member/wal", "data-dir": "/var/lib/etcd-2025", "snap-dir": "/var/lib/etcd-2025/member/snap", "initial-memory-map-size": 0}
2025-05-15T14:59:42+08:00	info	membership/store.go:138	Trimming membership information from the backend...
2025-05-15T14:59:42+08:00	info	membership/cluster.go:421	added member	{"cluster-id": "cdf818194e3a8c32", "local-member-id": "0", "added-peer-id": "8e9e05c52164694d", "added-peer-peer-urls": ["http://localhost:2380"], "added-peer-is-learner": false}
2025-05-15T14:59:42+08:00	info	snapshot/v3_snapshot.go:293	restored snapshot	{"path": "/tmp/cmy-etcd-2025-05-15.backup", "wal-dir": "/var/lib/etcd-2025/member/wal", "data-dir": "/var/lib/etcd-2025", "snap-dir": "/var/lib/etcd-2025/member/snap", "initial-memory-map-size": 0}
[root@node-exporter42 ~]# 


[root@node-exporter43 ~]# etcdctl snapshot restore /tmp/cmy-etcd-`date +%F`.backup --data-dir=/var/lib/etcd-2025
Deprecated: Use `etcdutl snapshot restore` instead.

2025-05-15T15:00:41+08:00	info	snapshot/v3_snapshot.go:265	restoring snapshot	{"path": "/tmp/cmy-etcd-2025-05-15.backup", "wal-dir": "/var/lib/etcd-2025/member/wal", "data-dir": "/var/lib/etcd-2025", "snap-dir": "/var/lib/etcd-2025/member/snap", "initial-memory-map-size": 0}
2025-05-15T15:00:41+08:00	info	membership/store.go:138	Trimming membership information from the backend...
2025-05-15T15:00:41+08:00	info	membership/cluster.go:421	added member	{"cluster-id": "cdf818194e3a8c32", "local-member-id": "0", "added-peer-id": "8e9e05c52164694d", "added-peer-peer-urls": ["http://localhost:2380"], "added-peer-is-learner": false}
2025-05-15T15:00:41+08:00	info	snapshot/v3_snapshot.go:293	restored snapshot	{"path": "/tmp/cmy-etcd-2025-05-15.backup", "wal-dir": "/var/lib/etcd-2025/member/wal", "data-dir": "/var/lib/etcd-2025", "snap-dir": "/var/lib/etcd-2025/member/snap", "initial-memory-map-size": 0}
[root@node-exporter43 ~]# 



	6 将恢复后的数据目录作为新的数据目录
[root@node-exporter41 ~]# grep "/var/lib/etcd" /cmy/softwares/etcd/etcd.config.yml 
data-dir: /var/lib/etcd
wal-dir: /var/lib/etcd/wal
[root@node-exporter41 ~]# 
[root@node-exporter41 ~]# 
[root@node-exporter41 ~]# sed -ri "s#(/var/lib/etcd)#\1-2025#g" /cmy/softwares/etcd/etcd.config.yml 
[root@node-exporter41 ~]# 
[root@node-exporter41 ~]# grep "/var/lib/etcd" /cmy/softwares/etcd/etcd.config.yml 
data-dir: /var/lib/etcd-2025
wal-dir: /var/lib/etcd-2025/wal
[root@node-exporter41 ~]# 


[root@node-exporter42 ~]# sed -ri "s#(/var/lib/etcd)#\1-2025#g" /cmy/softwares/etcd/etcd.config.yml 
[root@node-exporter42 ~]# 
[root@node-exporter42 ~]# grep "/var/lib/etcd" /cmy/softwares/etcd/etcd.config.yml
data-dir: /var/lib/etcd-2025
wal-dir: /var/lib/etcd-2025/wal
[root@node-exporter42 ~]# 


[root@node-exporter43 ~]# sed -ri "s#(/var/lib/etcd)#\1-2025#g" /cmy/softwares/etcd/etcd.config.yml 
[root@node-exporter43 ~]# 
[root@node-exporter43 ~]# grep "/var/lib/etcd" /cmy/softwares/etcd/etcd.config.yml
data-dir: /var/lib/etcd-2025
wal-dir: /var/lib/etcd-2025/wal
[root@node-exporter43 ~]# 

	7 启动etcd集群 
[root@node-exporter41 ~]# systemctl start etcd

[root@node-exporter42 ~]# systemctl start etcd
 
[root@node-exporter43 ~]# systemctl start etcd


	8 验证数据是否恢复 
[root@node-exporter41 ~]# etcdctl get "" --prefix --keys-only
/cmy/docker/dockerhub

/cmy/docker/harbor

/cmy/docker/registry

offic

school

service

[root@node-exporter41 ~]# 


	9.测试数据是否可以正常读写
[root@node-exporter41 ~]# etcdctl put xixi 哈哈
OK
[root@node-exporter41 ~]# 
[root@node-exporter41 ~]# etcdctl get "" --prefix --keys-only
/cmy/cmy/first

/cmy/cmy/fourth

/cmy/cmy/second

/cmy/cmy/third

class

classroom

school

xixi

[root@node-exporter41 ~]#

5 etcd-workbench使用

- etcd-workbench图形化管理etcd集群
参考链接:
	https://tzfun.github.io/etcd-workbench/
	https://github.com/tzfun/etcd-workbench/blob/master/README_ZH.md
	https://github.com/tzfun/etcd-workbench-web/blob/master/server/src/main/resources/etcd-workbench.conf
	
	1.拉取镜像 
docker pull tzfun/etcd-workbench:1.1.4

SVIP:
[root@node-exporter43 ~]# wget http://192.168.14.253/Resources/Prometheus/images/etcd-workbench/cmy-etcd-workbench-v1.1.4.tar.gz

[root@node-exporter43 ~]# docker load -i cmy-etcd-workbench-v1.1.4.tar.gz 


	2.运行etcd-workbench
[root@node-exporter43 ~]# cat etcd-workbench.conf 
[server]
# 服务监听的端口
port = 8002
# 链接超时时间
etcdExecuteTimeoutMillis = 3000
# 数据存储目录
dataDir = ./data

[auth]
# 启用认证功能
enable = true
# 指定用户名和密码
user = admin:cmy

[log]
# 指定日志的级别
level = INFO
# 日志存储目录
file = ./logs
# 日志文件的名称
fileName = etcd-workbench
# 指定日志的滚动大小
fileLimitSize = 100
# 日志打印的位置
printers = std,file
[root@node-exporter43 ~]# 
[root@node-exporter43 ~]# docker run -d -v /root/etcd-workbench.conf:/usr/tzfun/etcd-workbench/etcd-workbench.conf --name etcd-workbench --network host tzfun/etcd-workbench:1.1.4 
88e4dc60963e92f988a617727e7cf76db3e0d565096859ca63549bed7883fc46
[root@node-exporter43 ~]# 
[root@node-exporter43 ~]# ss -ntl | grep 8002
LISTEN 0      4096               *:8002             *:*          
[root@node-exporter43 ~]# 
[root@node-exporter43 ~]# docker ps -l
CONTAINER ID   IMAGE                        COMMAND                  CREATED         STATUS         PORTS     NAMES
88e4dc60963e   tzfun/etcd-workbench:1.1.4   "/bin/sh -c 'java   …"   9 seconds ago   Up 8 seconds             etcd-workbench
[root@node-exporter43 ~]# 



	3.访问etcd-workbench的webUI
http://10.168.10.43:8002/

用户名和密码使用你自定义的即可。


	4.拷贝证书到windows系统
cert-file: '/cmy/certs/etcd/etcd-server.pem'
key-file: '/cmy/certs/etcd/etcd-server-key.pem'
trusted-ca-file: '/cmy/certs/etcd/etcd-ca.pem'

	5.上传证书并测试
略，见视频。

6 prometheus监控etcd集群

6.1 prometheus端拷贝etcd证书

1.etcd https架构图解
略，见视频。
	
	
	2.使用curl工具连接etcd测试
[root@node-exporter41 ~]# curl -s -k --cacert /cmy/certs/etcd/etcd-ca.pem --cert /cmy/certs/etcd/etcd-server.pem  --key /cmy/certs/etcd/etcd-server-key.pem  https://10.168.10.41:2379/metrics  | wc -l
1692
[root@node-exporter41 ~]# 

 

- Prometheus监控etcd服务实战
参考链接:
	https://prometheus.io/docs/prometheus/latest/configuration/configuration/#tls_config
	
	
	1.prometheus端创建etcd证书目录
[root@prometheus-server31 prometheus-2.53.4.linux-amd64]# pwd
/cmy/softwares/prometheus-2.53.4.linux-amd64
[root@prometheus-server31 prometheus-2.53.4.linux-amd64]# 
[root@prometheus-server31 prometheus-2.53.4.linux-amd64]# mkdir -p certs/etcd
[root@prometheus-server31 prometheus-2.53.4.linux-amd64]# 



	2.将etcd的自建证书拷贝prometheus服务器
[root@prometheus-server31 prometheus-2.53.4.linux-amd64]# scp 10.168.10.41:/cmy/certs/etcd/etcd-{ca.pem,server-key.pem,server.pem} certs/etcd
 


	3.Prometheus查看证书文件
[root@prometheus-server31 prometheus-2.53.4.linux-amd64]# apt  install tree
[root@prometheus-server31 prometheus-2.53.4.linux-amd64]#
[root@prometheus-server31 prometheus-2.53.4.linux-amd64]# tree certs/etcd/
certs/etcd/
├── etcd-ca.pem
├── etcd-server-key.pem
└── etcd-server.pem

0 directories, 3 files
[root@prometheus-server31 prometheus-2.53.4.linux-amd64]#

6.2 修改Prometheus的配置文件

etcdctl move-leader

	



	4.修改Prometheus的配置文件【修改配置时，可以将中文注释删除，此处的中文注释是方便你理解的。】
[root@prometheus-server31 ~]# vim /cmy/softwares/prometheus-2.53.4.linux-amd64/prometheus.yml
...
  - job_name: "cmy-etcd-cluster"
  - job_name: "cmy-etcd-cluster"
    scheme: https
    tls_config:
        ca_file: certs/etcd/etcd-ca.pem
        cert_file: certs/etcd/etcd-server.pem
        key_file: certs/etcd/etcd-server-key.pem
    static_configs:
      - targets:
          - 10.168.10.41:2379
          - 10.168.10.42:2379
          - 10.168.10.43:2379
	
	5.检查配置文件是否正确
[root@prometheus-server31 prometheus-2.53.4.linux-amd64]# ./promtool check config prometheus.yml
Checking prometheus.yml
  SUCCESS: 1 rule files found
 SUCCESS: prometheus.yml is valid prometheus config file syntax

Checking cmy-linux96-rules.yml
  SUCCESS: 3 rules found

[root@prometheus-server31 prometheus-2.53.4.linux-amd64]# 


	
	6.热加载配置文件
[root@prometheus-server31 ~]# curl -X POST http://10.168.10.31:9090/-/reload 


	7.检查配置是否生效 
http://10.168.10.31:9090/targets


	8.grafana导入模板ID
21473
3070
10323
上一篇
下一篇