0.1 部署ceph集群的常用工具
推荐阅读:
https://docs.ceph.com/en/latest/install/#installing-ceph
ceph-deploy(ceph的原生部署工具):
(1)ceph-deploy是一种部署ceph的方法，它仅依赖于SSH访问服务器、而后借助sudo和一些Python模块就可以实现部署。
(2)它完全在工作站（管理主机）上运行，不需要任何服务、数据库或类似的东西。
(3)它不是一个通用的部署系统，它只是为Ceph设计的，并且是为那些希望在不需要安装Chef、Puppet或Juju的情况下使用合理的初始设置快速运行Ceph的用户而设计的。
(4)除了推送Ceph配置文件之外，它不会处理客户端配置，想要对安全设置、分区或目录位置进行精确控制的用户应该使用Chef或Puppet之类的工具。
ceph-ansible:
(1)我们可以使用ansible的playbook来部署Ceph;
(2)ceph的GitHub地址为: "https://github.com/ceph/ceph-ansible"
ceph-chef:
对应的GitHub地址: "https://github.com/ceph/ceph-chef"
puppet-ceph:
即使用puppet工具来部署ceph。
Cephadm:
和k8s的kubeadm一样好用，部署服务非常方便，推荐使用。
参考链接:
https://docs.ceph.com/en/latest/install/#other-methods
ceph:
https://github.com/ceph/ceph
0.2 ceph版本说明
推荐阅读:
https://docs.ceph.com/en/latest/releases/general/#understanding-the-release-cycle
x.0.z
开发版本，一般是用于内部测试。
x.1.z
测试版本，一般是开发人员测试使用。
x.2.z
稳定版本，生产环境中推荐使用。
1 集群部署
- cephadm快速部署ceph分布式集群
推荐阅读:
https://www.cnblogs.com/cmy/p/18370686
1.cephadm基础配置环境准备
1.1 基于cephadm部署前提条件，官方提的要求Ubuntu 22.04 LTS出了容器运行时其他都满足
- Python 3
- Systemd
- Podman or Docker for running containers
- Time synchronization (such as Chrony or the legacy ntpd)
- LVM2 for provisioning storage devices
参考链接:
https://docs.ceph.com/en/latest/cephadm/install/#requirements
1.2 设置时区
timedatectl set-timezone Asia/Shanghai
ll /etc/localtime
1.3.安装docker环境
略，国内的小伙伴建议二进制安装，yum或者apt安装都需要FQ，可以考虑使用国内的软件源。
当然，SVIP的小伙伴不要慌，用我给大家的一键部署脚步即可。
wget http://192.168.17.253/Resources/Docker/softwares/cmy-autoinstall-docker-docker-compose.tar.gz
tar xf cmy-autoinstall-docker-docker-compose.tar.gz
./install-docker.sh i
1.4.所有节点添加hosts文件解析
cat >> /etc/hosts <<EOF
10.168.10.141 ceph141
10.168.10.142 ceph142
10.168.10.143 ceph143
EOF
1.5 集群时间同步【Ubuntu系统可跳过】
参考链接:
https://www.cnblogs.com/cmy/p/14238720.html#3配置时间同步
1.6.集群环境准备
ceph141:
CPU: 1c
Memory: 2G
/dev/sdb：300GB
/dev/sdc: 500GB
/dev/sdd: 1024GB
ceph141:
CPU: 1c
Memory: 2G
/dev/sdb：300GB
/dev/sdc: 500GB
/dev/sdd: 1024GB
ceph143:
CPU: 1c
Memory: 2G
/dev/sdb：300GB
/dev/sdc: 500GB
/dev/sdd: 1024GB
[root@ceph141 ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
loop0 7:0 0 63.9M 1 loop /snap/core20/2318
loop1 7:1 0 87M 1 loop /snap/lxd/29351
loop2 7:2 0 64M 1 loop /snap/core20/2379
loop3 7:3 0 89.4M 1 loop /snap/lxd/31333
loop4 7:4 0 40.4M 1 loop /snap/snapd/20671
loop5 7:5 0 38.8M 1 loop /snap/snapd/21759
sda 8:0 0 100G 0 disk
├─sda1 8:1 0 1M 0 part
├─sda2 8:2 0 2G 0 part /boot
└─sda3 8:3 0 98G 0 part
└─ubuntu--vg-ubuntu--lv 253:0 0 49G 0 lvm /
sdb 8:16 0 300G 0 disk
sdc 8:32 0 500G 0 disk
sdd 8:48 0 1T 0 disk
sr0 11:0 1 1024M 0 rom
[root@ceph141 ~]#
[root@ceph142 ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
loop1 7:1 0 87M 1 loop /snap/lxd/29351
loop2 7:2 0 38.8M 1 loop /snap/snapd/21759
loop3 7:3 0 40.4M 1 loop /snap/snapd/20671
loop4 7:4 0 64M 1 loop /snap/core20/2379
loop5 7:5 0 89.4M 1 loop /snap/lxd/31333
loop6 7:6 0 63.7M 1 loop /snap/core20/2496
sda 8:0 0 100G 0 disk
├─sda1 8:1 0 1M 0 part
├─sda2 8:2 0 2G 0 part /boot
└─sda3 8:3 0 98G 0 part
└─ubuntu--vg-ubuntu--lv 253:0 0 49G 0 lvm /
sdb 8:16 0 300G 0 disk
sdc 8:32 0 500G 0 disk
sdd 8:48 0 1T 0 disk
sr0 11:0 1 1024M 0 rom
[root@ceph142 ~]#
[root@ceph143 ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
loop0 7:0 0 89.4M 1 loop /snap/lxd/31333
loop1 7:1 0 64M 1 loop /snap/core20/2379
loop2 7:2 0 87M 1 loop /snap/lxd/29351
loop3 7:3 0 40.4M 1 loop /snap/snapd/20671
loop5 7:5 0 38.8M 1 loop /snap/snapd/21759
loop6 7:6 0 63.7M 1 loop /snap/core20/2496
sda 8:0 0 100G 0 disk
├─sda1 8:1 0 1M 0 part
├─sda2 8:2 0 2G 0 part /boot
└─sda3 8:3 0 98G 0 part
└─ubuntu--vg-ubuntu--lv 253:0 0 49G 0 lvm /
sdb 8:16 0 300G 0 disk
sdc 8:32 0 500G 0 disk
sdd 8:48 0 1T 0 disk
sr0 11:0 1 1024M 0 rom
[root@ceph143 ~]#
2.安装cephadm工具
2.1.下载需要安装ceph版本的cephadm
CEPH_RELEASE=19.2.2
curl --silent --remote-name --location https://download.ceph.com/rpm-${CEPH_RELEASE}/el9/noarch/cephadm
SVIP:
wget http://192.168.17.253/Resources/Ceph/softwares/cephadm-v19.2.2
2.2 将cephadm添加到PATH环境变量
[root@ceph141 ~]# mv cephadm-v19.2.2 /usr/local/bin/cephadm
[root@ceph141 ~]#
[root@ceph141 ~]# chmod +x /usr/local/bin/cephadm
[root@ceph141 ~]#
[root@ceph141 ~]# ll /usr/local/bin/cephadm
-rwxr-xr-x 1 root root 787576 May 16 09:28 /usr/local/bin/cephadm*
[root@ceph141 ~]#
2.3 检查版本是否正确
[root@ceph141 ~]# cephadm version
cephadm version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)
[root@ceph141 ~]#
3.基于cephadm快速初始化ceph集群
3.1 提前导入镜像
[root@ceph141 ~]# cat > import-calico-v3.29.1.sh <<'EOF'
#!/bin/bash
URL=http://192.168.17.253/Resources/Ceph/images/cephadm/ceph-Squid-v19.2.2/
IMAGES=(cmy-alertmanager-v0.25.0.tar.gz cmy-ceph-v19.tar.gz cmy-grafana-v10.4.0.tar.gz cmy-node-exporter-v1.7.0.tar.gz cmy-prometheus-v2.51.0.tar.gz)
# echo ${IMAGES[*]}
for pkg in ${IMAGES[@]}
do
echo "Downdowning image ---> ${URL}/${pkg}"
wget ${URL}/${pkg}
echo "importing image ---> ${URL}/${pkg}"
docker load -i ${pkg}
done
EOF
[root@ceph141 ~]# bash import-calico-v3.29.1.sh
3.2 创建新集群
[root@ceph141 ~]# cephadm bootstrap --mon-ip 10.168.10.141 --cluster-network 10.168.10.0/24 --allow-fqdn-hostname
Verifying podman|docker is present...
Verifying lvm2 is present...
Verifying time synchronization is in place...
Unit systemd-timesyncd.service is enabled and running
Repeating the final host check...
docker (/usr/bin/docker) is present
systemctl is present
lvcreate is present
Unit systemd-timesyncd.service is enabled and running
Host looks OK
Cluster fsid: 48fcf2bc-31f6-11f0-8833-3507f15d877f
Verifying IP 10.168.10.141 port 3300 ...
Verifying IP 10.168.10.141 port 6789 ...
Mon IP `10.168.10.141` is in CIDR network `10.168.10.0/24`
Mon IP `10.168.10.141` is in CIDR network `10.168.10.0/24`
Pulling container image quay.io/ceph/ceph:v19...
Ceph version: ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)
Extracting ceph user uid/gid from container image...
Creating initial keys...
Creating initial monmap...
Creating mon...
Waiting for mon to start...
Waiting for mon...
mon is available
Assimilating anything we can from ceph.conf...
Generating new minimal ceph.conf...
Restarting the monitor...
Setting public_network to 10.168.10.0/24 in mon config section
Setting cluster_network to 10.168.10.0/24
Wrote config to /etc/ceph/ceph.conf
Wrote keyring to /etc/ceph/ceph.client.admin.keyring
Creating mgr...
Verifying port 0.0.0.0:9283 ...
Verifying port 0.0.0.0:8765 ...
Verifying port 0.0.0.0:8443 ...
Waiting for mgr to start...
Waiting for mgr...
mgr not available, waiting (1/15)...
mgr not available, waiting (2/15)...
mgr not available, waiting (3/15)...
mgr not available, waiting (4/15)...
mgr not available, waiting (5/15)...
mgr not available, waiting (6/15)...
mgr not available, waiting (7/15)...
mgr not available, waiting (8/15)...
mgr is available
Enabling cephadm module...
Waiting for the mgr to restart...
Waiting for mgr epoch 5...
mgr epoch 5 is available
Setting orchestrator backend to cephadm...
Generating ssh key...
Wrote public SSH key to /etc/ceph/ceph.pub
Adding key to root@localhost authorized_keys...
Adding host ceph141...
Deploying mon service with default placement...
Deploying mgr service with default placement...
Deploying crash service with default placement...
Deploying ceph-exporter service with default placement...
Deploying prometheus service with default placement...
Deploying grafana service with default placement...
Deploying node-exporter service with default placement...
Deploying alertmanager service with default placement...
Enabling the dashboard module...
Waiting for the mgr to restart...
Waiting for mgr epoch 9...
mgr epoch 9 is available
Generating a dashboard self-signed certificate...
Creating initial admin user...
Fetching dashboard port number...
Ceph Dashboard is now available at:
URL: https://ceph141:8443/
User: admin
Password: 35l2y1qayk
Enabling client.admin keyring and conf on hosts with "admin" label
Saving cluster configuration to /var/lib/ceph/48fcf2bc-31f6-11f0-8833-3507f15d877f/config directory
You can access the Ceph CLI as following in case of multi-cluster or non-default config:
sudo /usr/local/bin/cephadm shell --fsid 48fcf2bc-31f6-11f0-8833-3507f15d877f -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring
Or, if you are only running a single cluster on this host:
sudo /usr/local/bin/cephadm shell
Please consider enabling telemetry to help improve Ceph:
ceph telemetry on
For more information see:
https://docs.ceph.com/en/latest/mgr/telemetry/
Bootstrap complete.
[root@ceph141 ~]#
3.1 注意观察输出信息，记录dashboard账号信息，并重置密码
https://10.168.10.141:8443/#/hosts
温馨提示:
首次登录需要修改密码，建议密码为你的名字。
初始密码无法登陆【需要安装管理节点】
除了上面在WebUI的方式修改密码外，也可以基于命令行方式修改密码，只不过在应用时可能需要等待一段时间才能生效。【目前官方已经弃用，大概需要等30s-1min】
[root@ceph141 ~]# echo jasonyin2020 | ceph dashboard set-login-credentials admin -i -
******************************************************************
*** WARNING: this command is deprecated. ***
*** Please use the ac-user-* related commands to manage users. ***
******************************************************************
Username and password updated
[root@ceph141 ~]#
配置ceph集群的管理节点的两种方式
1.方式一：在宿主机安装ceph管理命令【推荐】
1.1 添加软件源
[root@ceph141 ~]# cephadm add-repo --release squid
Installing repo GPG key from https://download.ceph.com/keys/release.gpg...
Installing repo file at /etc/apt/sources.list.d/ceph.list...
Updating package list...
Completed adding repo.
[root@ceph141 ~]#
[root@ceph141 ~]# ll /etc/apt/sources.list.d/ceph.list
-rw-r--r-- 1 root root 55 Mar 31 16:38 /etc/apt/sources.list.d/ceph.list
[root@ceph141 ~]#
[root@ceph141 ~]# ll /etc/apt/trusted.gpg.d/ceph.release.gpg
-rw-r--r-- 1 root root 1143 Mar 31 16:38 /etc/apt/trusted.gpg.d/ceph.release.gpg
[root@ceph141 ~]#
[root@ceph141 ~]# ll /etc/ceph/
total 20
drwxr-xr-x 2 root root 4096 Mar 31 15:37 ./
drwxr-xr-x 130 root root 4096 Mar 31 15:30 ../
-rw------- 1 root root 151 Mar 31 15:37 ceph.client.admin.keyring
-rw-r--r-- 1 root root 171 Mar 31 15:37 ceph.conf
-rw-r--r-- 1 root root 595 Mar 31 15:33 ceph.pub
[root@ceph141 ~]#
[root@ceph141 ~]# sed -i 's#download.ceph.com#mirrors.tuna.tsinghua.edu.cn#' /etc/apt/sources.list.d/ceph.list
[root@ceph141 ~]#
[root@ceph141 ~]# cat /etc/apt/sources.list.d/ceph.list
deb https://mirrors.tuna.tsinghua.edu.cn/debian-squid/ jammy main
[root@ceph141 ~]#
推荐阅读:
https://mirrors.tuna.tsinghua.edu.cn/help/ceph/
1.2 安装ceph通用工具包
[root@ceph141 ~]# apt update
[root@ceph141 ~]# apt -y install ceph-common
1.3 测试使用
[root@ceph141 ~]# ceph version
ceph version 19.2.2 (58a7fab8be0a062d730ad7da874972fd3fba59fb) squid (stable)
[root@ceph141 ~]#
[root@ceph141 ~]# ceph -s
cluster:
id: 11e66474-0e02-11f0-82d6-4dcae3d59070
health: HEALTH_WARN
OSD count 0 < osd_pool_default_size 3
services:
mon: 1 daemons, quorum ceph141 (age 69m)
mgr: ceph141.mbakds(active, since 65m)
osd: 0 osds: 0 up, 0 in
data:
pools: 0 pools, 0 pgs
objects: 0 objects, 0 B
usage: 0 B used, 0 B / 0 B avail
pgs:
[root@ceph141 ~]#
[root@ceph141 ~]# ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 0 root default
[root@ceph141 ~]#
2.基于容器管理ceph集群
2.1 启动shell工具
[root@ceph141 ~]# cephadm shell
Inferring fsid 11e66474-0e02-11f0-82d6-4dcae3d59070
Inferring config /var/lib/ceph/11e66474-0e02-11f0-82d6-4dcae3d59070/mon.ceph141/config
Not using image 'sha256:f2efb0401a30ec7eda97b6da76b314bd081fcb910cc5dcd826bc7c72c9dfdd7d' as it's not in list of non-dangling images with ceph=True label
root@ceph141:/#
root@ceph141:/# ceph --version
ceph version 19.2.2 (0eceb0defba60152a8182f7bd87d164b639885b8) squid (stable)
root@ceph141:/#
root@ceph141:/#
root@ceph141:/# ceph -s
cluster:
id: 11e66474-0e02-11f0-82d6-4dcae3d59070
health: HEALTH_WARN
OSD count 0 < osd_pool_default_size 3
services:
mon: 1 daemons, quorum ceph141 (age 71m)
mgr: ceph141.mbakds(active, since 67m)
osd: 0 osds: 0 up, 0 in
data:
pools: 0 pools, 0 pgs
objects: 0 objects, 0 B
usage: 0 B used, 0 B / 0 B avail
pgs:
root@ceph141:/#
2.2 本质上是启动了一个容器【当命令执行结束时，容器会自动删除！】
[root@ceph141 ~]# docker ps -l
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
c72582492e3a quay.io/ceph/ceph:v19 "bash" 4 seconds ago Up 3 seconds blissful_proskuriakova
[root@ceph141 ~]#
1.1 ceph集群添加主机实战
1.扩容时准备操作
建议将docker和镜像重新从ceph141节点导入到ceph142和ceph143节点。
1.1 拷贝镜像
[root@ceph141 ~]# scp cmy-alertmanager-v0.25.0.tar.gz cmy-ceph-v19.tar.gz cmy-grafana-v10.4.0.tar.gz cmy-node-exporter-v1.7.0.tar.gz cmy-prometheus-v2.51.0.tar.gz 10.168.10.142:~
[root@ceph141 ~]# scp cmy-alertmanager-v0.25.0.tar.gz cmy-ceph-v19.tar.gz cmy-grafana-v10.4.0.tar.gz cmy-node-exporter-v1.7.0.tar.gz cmy-prometheus-v2.51.0.tar.gz 10.168.10.143:~
1.2 导入镜像
[root@ceph142 ~]# for i in cmy-alertmanager-v0.25.0.tar.gz cmy-ceph-v19.tar.gz cmy-grafana-v10.4.0.tar.gz cmy-node-exporter-v1.7.0.tar.gz cmy-prometheus-v2.51.0.tar.gz; do docker load -i $i ;done
[root@ceph143 ~]# for i in cmy-alertmanager-v0.25.0.tar.gz cmy-ceph-v19.tar.gz cmy-grafana-v10.4.0.tar.gz cmy-node-exporter-v1.7.0.tar.gz cmy-prometheus-v2.51.0.tar.gz; do docker load -i $i ;done
2.查看现有的集群主机列表
[root@ceph141 ~]# ceph orch host ls
HOST ADDR LABELS STATUS
ceph141 10.168.10.141 _admin
1 hosts in cluster
[root@ceph141 ~]#
3.把秘钥放到其他服务器上
[root@ceph141 ~]# ssh-copy-id -f -i /etc/ceph/ceph.pub ceph142
[root@ceph141 ~]# ssh-copy-id -f -i /etc/ceph/ceph.pub ceph143
4.将秘钥节点加入到集群
[root@ceph141 ~]# ceph orch host add ceph142 10.168.10.142
Added host 'ceph142' with addr '10.168.10.142'
[root@ceph141 ~]#
[root@ceph141 ~]# ceph orch host add ceph143 10.168.10.143
Added host 'ceph143' with addr '10.168.10.143'
[root@ceph141 ~]#
温馨提示：
将集群加入成功后，会自动创建"/var/lib/ceph/<Ceph_Cluster_ID>"相关数据目录。
5.再次查看集群的主机列表
[root@ceph141 ~]# ceph orch host ls
HOST ADDR LABELS STATUS
ceph141 10.168.10.141 _admin
ceph142 10.168.10.142
ceph143 10.168.10.143
3 hosts in cluster
[root@ceph141 ~]#
6.通过查看WebUI观察ceph集群有多少个主机。
https://10.168.10.141:8443/#/hosts
1.2 ceph集群移除主机实战
2.1 卸载集群前状态查看
[root@ceph141 ~]# ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-11 7.68741 root linux97
-13 2.56247 host ceph141-ssd
9 hdd 0.78119 osd.9 up 1.00000 1.00000
-15 2.56247 host ceph142-ssd
10 hdd 0.78119 osd.10 up 1.00000 1.00000
-17 2.56247 host ceph143-ssd
11 hdd 0.78119 osd.11 up 1.00000 1.00000
-1 6.90623 root default
-3 2.56247 host ceph141
0 hdd 0.29300 osd.0 up 1.00000 1.00000
1 hdd 0.48830 osd.1 up 1.00000 1.00000
2 hdd 1.00000 osd.2 up 1.00000 1.00000
-5 1.78130 host ceph142
3 hdd 0.29300 osd.3 up 1.00000 1.00000
4 hdd 0.48830 osd.4 up 1.00000 1.00000
5 hdd 1.00000 osd.5 up 1.00000 1.00000
-7 2.56247 host ceph143
6 hdd 0.29300 osd.6 up 1.00000 1.00000
7 hdd 0.48830 osd.7 up 1.00000 1.00000
8 hdd 1.00000 osd.8 up 1.00000 1.00000
[root@ceph141 ~]#
[root@ceph141 ~]# ceph -s
cluster:
id: 48fcf2bc-31f6-11f0-8833-3507f15d877f
health: HEALTH_OK
services:
mon: 3 daemons, quorum ceph141,ceph142,ceph143 (age 6h)
mgr: ceph142.rngppx(active, since 6h), standbys: ceph141.rzrqkk
mds: 1/1 daemons up, 1 standby
osd: 12 osds: 12 up (since 3m), 12 in (since 5m)
rgw: 2 daemons active (2 hosts, 1 zones)
data:
volumes: 1/1 healthy
pools: 12 pools, 625 pgs
objects: 3.94k objects, 472 MiB
usage: 4.6 GiB used, 7.7 TiB / 7.7 TiB avail
pgs: 625 active+clean
[root@ceph141 ~]#
[root@ceph141 ~]# ceph orch device ls
HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS
ceph141 /dev/sdb hdd 300G No 5m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected
ceph141 /dev/sdc hdd 500G No 5m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected
ceph141 /dev/sdd hdd 1024G No 5m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected
ceph141 /dev/sde hdd 800G No 5m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected
ceph141 /dev/sr0 hdd VMware_Virtual_SATA_CDRW_Drive_01000000000000000001 1023M No 5m ago Failed to determine if device is BlueStore, Insufficient space (<5GB)
ceph142 /dev/sdb hdd 300G No 4m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected
ceph142 /dev/sdc hdd 500G No 4m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected
ceph142 /dev/sdd hdd 1024G No 4m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected
ceph142 /dev/sde hdd 800G No 4m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected
ceph142 /dev/sr0 hdd VMware_Virtual_SATA_CDRW_Drive_01000000000000000001 1023M No 4m ago Failed to determine if device is BlueStore, Insufficient space (<5GB)
ceph143 /dev/sdb hdd 300G No 4m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected
ceph143 /dev/sdc hdd 500G No 4m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected
ceph143 /dev/sdd hdd 1024G No 4m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected
ceph143 /dev/sde hdd 800G No 4m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected
ceph143 /dev/sr0 hdd VMware_Virtual_SATA_CDRW_Drive_01000000000000000001 1023M No 4m ago Failed to determine if device is BlueStore, Insufficient space (<5GB)
[root@ceph141 ~]#
2.2 停止需要下线节点的所有osd守护进程
[root@ceph141 ~]# ceph orch daemon stop osd.6
Scheduled to stop osd.6 on host 'ceph143'
[root@ceph141 ~]#
[root@ceph141 ~]# ceph orch daemon stop osd.7
Scheduled to stop osd.7 on host 'ceph143'
[root@ceph141 ~]#
[root@ceph141 ~]# ceph orch daemon stop osd.8
Scheduled to stop osd.8 on host 'ceph143'
[root@ceph141 ~]#
[root@ceph141 ~]# ceph orch daemon stop osd.11
Scheduled to stop osd.11 on host 'ceph143'
[root@ceph141 ~]#
[root@ceph141 ~]# ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-11 7.68741 root linux97
-13 2.56247 host ceph141-ssd
9 hdd 0.78119 osd.9 up 1.00000 1.00000
-15 2.56247 host ceph142-ssd
10 hdd 0.78119 osd.10 up 1.00000 1.00000
-17 2.56247 host ceph143-ssd
11 hdd 0.78119 osd.11 down 1.00000 1.00000
-1 6.90623 root default
-3 2.56247 host ceph141
0 hdd 0.29300 osd.0 up 1.00000 1.00000
1 hdd 0.48830 osd.1 up 1.00000 1.00000
2 hdd 1.00000 osd.2 up 1.00000 1.00000
-5 1.78130 host ceph142
3 hdd 0.29300 osd.3 up 1.00000 1.00000
4 hdd 0.48830 osd.4 up 1.00000 1.00000
5 hdd 1.00000 osd.5 up 1.00000 1.00000
-7 2.56247 host ceph143
6 hdd 0.29300 osd.6 down 1.00000 1.00000
7 hdd 0.48830 osd.7 down 1.00000 1.00000
8 hdd 1.00000 osd.8 down 1.00000 1.00000
[root@ceph141 ~]#
2.3 清理OSD数据和配置
[root@ceph141 ~]# ceph osd purge 6 --force
purged osd.6
[root@ceph141 ~]#
[root@ceph141 ~]# ceph osd purge 7 --force
purged osd.7
[root@ceph141 ~]#
[root@ceph141 ~]# ceph osd purge 8 --force
purged osd.8
[root@ceph141 ~]#
[root@ceph141 ~]# ceph osd purge 11 --force
purged osd.11
[root@ceph141 ~]#
[root@ceph141 ~]# ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-11 5.12494 root linux97
-13 2.56247 host ceph141-ssd
9 hdd 0.78119 osd.9 up 1.00000 1.00000
-15 2.56247 host ceph142-ssd
10 hdd 0.78119 osd.10 up 1.00000 1.00000
-17 0 host ceph143-ssd
-1 4.34377 root default
-3 2.56247 host ceph141
0 hdd 0.29300 osd.0 up 1.00000 1.00000
1 hdd 0.48830 osd.1 up 1.00000 1.00000
2 hdd 1.00000 osd.2 up 1.00000 1.00000
-5 1.78130 host ceph142
3 hdd 0.29300 osd.3 up 1.00000 1.00000
4 hdd 0.48830 osd.4 up 1.00000 1.00000
5 hdd 1.00000 osd.5 up 1.00000 1.00000
-7 0 host ceph143
[root@ceph141 ~]#
2.4 移除所有OSD后，从CRUSH map中删除主机
[root@ceph141 ~]# ceph osd crush rm ceph143
removed item id -7 name 'ceph143' from crush map
[root@ceph141 ~]#
[root@ceph141 ~]# ceph osd crush rm ceph143-ssd
removed item id -17 name 'ceph143-ssd' from crush map
[root@ceph141 ~]#
[root@ceph141 ~]# ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-11 5.12494 root linux97
-13 2.56247 host ceph141-ssd
9 hdd 0.78119 osd.9 up 1.00000 1.00000
-15 2.56247 host ceph142-ssd
10 hdd 0.78119 osd.10 up 1.00000 1.00000
-1 4.34377 root default
-3 2.56247 host ceph141
0 hdd 0.29300 osd.0 up 1.00000 1.00000
1 hdd 0.48830 osd.1 up 1.00000 1.00000
2 hdd 1.00000 osd.2 up 1.00000 1.00000
-5 1.78130 host ceph142
3 hdd 0.29300 osd.3 up 1.00000 1.00000
4 hdd 0.48830 osd.4 up 1.00000 1.00000
5 hdd 1.00000 osd.5 up 1.00000 1.00000
[root@ceph141 ~]#
2.5 自动驱逐被下线节点的服务组件
[root@ceph141 ~]# ceph orch host ls
HOST ADDR LABELS STATUS
ceph141 10.0.0.141 _admin
ceph142 10.0.0.142
ceph143 10.0.0.143
3 hosts in cluster
[root@ceph141 ~]#
[root@ceph141 ~]# ceph orch host drain ceph143
Scheduled to remove the following daemons from host 'ceph143'
type id
-------------------- ---------------
osd 6
osd 8
osd 7
node-exporter ceph143
osd 11
mon ceph143
ceph-exporter ceph143
crash ceph143
[root@ceph141 ~]#
[root@ceph141 ~]# ceph orch host drain ceph143 # 发现4个OSD是无法自动驱逐的，因为这4个osd已经被我们停止了
Scheduled to remove the following daemons from host 'ceph143'
type id
-------------------- ---------------
osd 6
osd 8
osd 7
osd 11
[root@ceph141 ~]#
[root@ceph141 ~]# ceph orch host ls
HOST ADDR LABELS STATUS
ceph141 10.0.0.141 _admin
ceph142 10.0.0.142
ceph143 10.0.0.143 _no_schedule,_no_conf_keyring
3 hosts in cluster
[root@ceph141 ~]#
2.6 手动删除停止的OSD组件【停止的OSD无法被驱逐，如果想要跳过此步骤，可以尝试先进行驱逐，再执行停止的相关动作】
[root@ceph141 ~]# ceph orch daemon rm osd.6 --force
Removed osd.6 from host 'ceph143'
[root@ceph141 ~]#
[root@ceph141 ~]# ceph orch daemon rm osd.7 --force
Removed osd.7 from host 'ceph143'
[root@ceph141 ~]#
[root@ceph141 ~]# ceph orch daemon rm osd.8 --force
Removed osd.8 from host 'ceph143'
[root@ceph141 ~]#
[root@ceph141 ~]# ceph orch daemon rm osd.11 --force
Removed osd.11 from host 'ceph143'
[root@ceph141 ~]#
[root@ceph141 ~]# ceph orch host drain ceph143
Scheduled to remove the following daemons from host 'ceph143'
type id
-------------------- ---------------
[root@ceph141 ~]#
2.7 删除节点
[root@ceph141 ~]# ceph orch host ls
HOST ADDR LABELS STATUS
ceph141 10.0.0.141 _admin
ceph142 10.0.0.142
ceph143 10.0.0.143 _no_schedule,_no_conf_keyring
3 hosts in cluster
[root@ceph141 ~]#
[root@ceph141 ~]# ceph orch host rm ceph143
Removed host 'ceph143'
[root@ceph141 ~]#
[root@ceph141 ~]# ceph orch host ls
HOST ADDR LABELS STATUS
ceph141 10.0.0.141 _admin
ceph142 10.0.0.142
2 hosts in cluster
[root@ceph141 ~]#
[root@ceph141 ~]# ceph orch device ls
HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS
ceph141 /dev/sdb hdd 300G No 16m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected
ceph141 /dev/sdc hdd 500G No 16m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected
ceph141 /dev/sdd hdd 1024G No 16m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected
ceph141 /dev/sde hdd 800G No 16m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected
ceph141 /dev/sr0 hdd VMware_Virtual_SATA_CDRW_Drive_01000000000000000001 1023M No 16m ago Failed to determine if device is BlueStore, Insufficient space (<5GB)
ceph142 /dev/sdb hdd 300G No 15m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected
ceph142 /dev/sdc hdd 500G No 15m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected
ceph142 /dev/sdd hdd 1024G No 15m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected
ceph142 /dev/sde hdd 800G No 15m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected
ceph142 /dev/sr0 hdd VMware_Virtual_SATA_CDRW_Drive_01000000000000000001 1023M No 15m ago Failed to determine if device is BlueStore, Insufficient space (<5GB)
[root@ceph141 ~]#
2.8 查看客户端的设备信息
[root@ceph143 ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
...
└─ceph--ac55aa2b--2136--420c--9cf5--9853202d6f8e-osd--block--8de87548--557c--4d30--a767--0fa610c2142e
253:0 0 300G 0 lvm
sdc 8:32 0 500G 0 disk
└─ceph--74600ec4--4924--4926--8218--0128d6cdbdf7-osd--block--77c5a1f0--8299--4fee--96a3--b01535a69e64
253:2 0 500G 0 lvm
sdd 8:48 0 1T 0 disk
└─ceph--79b405e3--1c85--432d--b3c4--5973fba7694f-osd--block--d926eac9--e5bf--4199--8ac3--6592303d1df0
253:1 0 1024G 0 lvm
sde 8:64 0 800G 0 disk
└─ceph--3f3d46b5--b764--495f--9c36--cb874a7f5003-osd--block--192f71bb--da25--4d9f--89eb--08abf4cbfa04
253:4 0 800G 0 lvm
sr0 11:0 1 1024M 0 rom
[root@ceph143 ~]#
2.9 查看本地的OSD编号和对应的磁盘设备对应关系
[root@ceph143 ~]# cat /var/lib/ceph/48fcf2bc-31f6-11f0-8833-3507f15d877f/removed/osd.6_2025-05-21T08\:01\:26.044940Z/fsid
8de87548-557c-4d30-a767-0fa610c2142e
[root@ceph143 ~]#
[root@ceph143 ~]# cat /var/lib/ceph/48fcf2bc-31f6-11f0-8833-3507f15d877f/removed/osd.7_2025-05-21T08\:01\:32.132483Z/fsid
77c5a1f0-8299-4fee-96a3-b01535a69e64
[root@ceph143 ~]#
[root@ceph143 ~]# cat /var/lib/ceph/48fcf2bc-31f6-11f0-8833-3507f15d877f/removed/osd.8_2025-05-21T08\:01\:37.339681Z/fsid
d926eac9-e5bf-4199-8ac3-6592303d1df0
[root@ceph143 ~]#
[root@ceph143 ~]# cat /var/lib/ceph/48fcf2bc-31f6-11f0-8833-3507f15d877f/removed/osd.11_2025-05-21T08\:01\:42.160129Z/fsid
192f71bb-da25-4d9f-89eb-08abf4cbfa04
[root@ceph143 ~]#
[root@ceph143 ~]# dmsetup status
ceph--3f3d46b5--b764--495f--9c36--cb874a7f5003-osd--block--192f71bb--da25--4d9f--89eb--08abf4cbfa04: 0 1677713408 linear
ceph--74600ec4--4924--4926--8218--0128d6cdbdf7-osd--block--77c5a1f0--8299--4fee--96a3--b01535a69e64: 0 1048567808 linear
ceph--79b405e3--1c85--432d--b3c4--5973fba7694f-osd--block--d926eac9--e5bf--4199--8ac3--6592303d1df0: 0 2147475456 linear
ceph--ac55aa2b--2136--420c--9cf5--9853202d6f8e-osd--block--8de87548--557c--4d30--a767--0fa610c2142e: 0 629137408 linear
ubuntu--vg-ubuntu--lv: 0 102752256 linear
[root@ceph143 ~]#
2.10 客户端解除ceph对磁盘的占用
[root@ceph143 ~]# dmsetup remove ceph--3f3d46b5--b764--495f--9c36--cb874a7f5003-osd--block--192f71bb--da25--4d9f--89eb--08abf4cbfa04
[root@ceph143 ~]# dmsetup remove ceph--74600ec4--4924--4926--8218--0128d6cdbdf7-osd--block--77c5a1f0--8299--4fee--96a3--b01535a69e64
[root@ceph143 ~]# dmsetup remove ceph--79b405e3--1c85--432d--b3c4--5973fba7694f-osd--block--d926eac9--e5bf--4199--8ac3--6592303d1df0
[root@ceph143 ~]# dmsetup remove ceph--ac55aa2b--2136--420c--9cf5--9853202d6f8e-osd--block--8de87548--557c--4d30--a767--0fa610c2142e
[root@ceph143 ~]#
2.11 再次查看本地磁盘设备，观察是否解除占用
[root@ceph143 ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
...
sdb 8:16 0 300G 0 disk
sdc 8:32 0 500G 0 disk
sdd 8:48 0 1T 0 disk
sde 8:64 0 800G 0 disk
[root@ceph143 ~]#
1.3 添加OSD设备到ceph集群及底层lvm技术原理
1.添加OSD之前环境查看
1.1 查看集群可用的设备【每个设备想要加入到集群，则其大小不得小于5GB】
[root@ceph141 ~]# ceph orch device ls
HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS
ceph141 /dev/sdb hdd 300G Yes 28m ago
ceph141 /dev/sdc hdd 500G Yes 28m ago
ceph141 /dev/sdd hdd 1024G Yes 28m ago
ceph141 /dev/sr0 hdd VMware_Virtual_SATA_CDRW_Drive_01000000000000000001 1023M No 28m ago Failed to determine if device is BlueStore, Insufficient space (<5GB)
ceph142 /dev/sdb hdd 300G Yes 3m ago
ceph142 /dev/sdc hdd 500G Yes 3m ago
ceph142 /dev/sdd hdd 1024G Yes 3m ago
ceph142 /dev/sr0 hdd VMware_Virtual_SATA_CDRW_Drive_01000000000000000001 1023M No 3m ago Failed to determine if device is BlueStore, Insufficient space (<5GB)
ceph143 /dev/sdb hdd 300G Yes 20m ago
ceph143 /dev/sdc hdd 500G Yes 20m ago
ceph143 /dev/sdd hdd 1024G Yes 20m ago
ceph143 /dev/sr0 hdd VMware_Virtual_SATA_CDRW_Drive_01000000000000000001 1023M No 20m ago Failed to determine if device is BlueStore, Insufficient space (<5GB)
[root@ceph141 ~]#
温馨提示:
如果一个设备想要加入ceph集群，要求满足3个条件:
- 1.设备未被使用;
- 2.设备的存储大小必须大于5GB;
- 3.需要等待一段时间，快则30s，慢则3分钟，线下教学有人笔记本性能不高甚至有等了25min或者40min才等出现设备结果;
如果想要快速显示，则可以在相应节点手动导入"quay.io/ceph/ceph"镜像即可。
ceph底层使用lvm技术栈，感兴趣的效果可参考链接:
https://www.cnblogs.com/cmy/p/18594123
1.2 查看各节点的空闲设备信息
[root@ceph141 ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
...
sdb 8:16 0 300G 0 disk
sdc 8:32 0 500G 0 disk
sdd 8:48 0 1T 0 disk
...
[root@ceph141 ~]#
[root@ceph142 ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
...
sdb 8:16 0 300G 0 disk
sdc 8:32 0 500G 0 disk
sdd 8:48 0 1T 0 disk
...
[root@ceph142 ~]#
[root@ceph143 ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
...
sdb 8:16 0 300G 0 disk
sdc 8:32 0 500G 0 disk
sdd 8:48 0 1T 0 disk
...
[root@ceph143 ~]#
1.3 查看OSD列表
[root@ceph141 ~]# ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 0 root default
[root@ceph141 ~]#
[root@ceph141 ~]# ceph -s
cluster:
id: 48fcf2bc-31f6-11f0-8833-3507f15d877f
health: HEALTH_WARN
OSD count 1 < osd_pool_default_size 3
services:
mon: 3 daemons, quorum ceph141,ceph142,ceph143 (age 27m)
mgr: ceph141.rzrqkk(active, since 2h), standbys: ceph142.rngppx
osd: 1 osds: 0 up, 1 in (since 2s)
data:
pools: 0 pools, 0 pgs
objects: 0 objects, 0 B
usage: 0 B used, 0 B / 0 B avail
pgs:
[root@ceph141 ~]#
2.添加OSD设备到集群
2.1 添加ceph141节点的设备到ceph集群
[root@ceph141 ~]# ceph orch daemon add osd ceph141:/dev/sdb
Created osd(s) 0 on host 'ceph141'
[root@ceph141 ~]#
[root@ceph141 ~]# ceph orch daemon add osd ceph141:/dev/sdc
Created osd(s) 1 on host 'ceph141'
[root@ceph141 ~]#
[root@ceph141 ~]# ceph orch daemon add osd ceph141:/dev/sdd
Created osd(s) 2 on host 'ceph141'
[root@ceph141 ~]#
[root@ceph141 ~]#
[root@ceph141 ~]# ceph orch daemon add osd ceph142:/dev/sdb
Created osd(s) 3 on host 'ceph142'
[root@ceph141 ~]#
[root@ceph141 ~]# ceph orch daemon add osd ceph142:/dev/sdc
Created osd(s) 4 on host 'ceph142'
[root@ceph141 ~]#
[root@ceph141 ~]# ceph orch daemon add osd ceph142:/dev/sdd
Created osd(s) 5 on host 'ceph142'
[root@ceph141 ~]#
[root@ceph141 ~]# ceph orch daemon add osd ceph143:/dev/sdb
Created osd(s) 6 on host 'ceph143'
[root@ceph141 ~]#
[root@ceph141 ~]# ceph orch daemon add osd ceph143:/dev/sdc
Created osd(s) 7 on host 'ceph143'
[root@ceph141 ~]#
[root@ceph141 ~]# ceph orch daemon add osd ceph143:/dev/sdd
Created osd(s) 8 on host 'ceph143'
[root@ceph141 ~]#
温馨提示:
- 1.此步骤会在"/var/lib/ceph/<Ceph_Cluster_ID>/osd.<OSD_ID>/fsid"文件中记录对应ceph的OSD编号对应本地的磁盘设备标识。
- 2.比如查看ceph142节点的硬盘和OSD的对应关系如下:
[root@ceph142 ~]# ll -d /var/lib/ceph/3cb12fba-5f6e-11ef-b412-9d303a22b70f/osd.*
drwx------ 2 167 167 4096 Aug 21 15:18 /var/lib/ceph/3cb12fba-5f6e-11ef-b412-9d303a22b70f/osd.2/
drwx------ 2 167 167 4096 Aug 21 15:19 /var/lib/ceph/3cb12fba-5f6e-11ef-b412-9d303a22b70f/osd.3/
drwx------ 2 167 167 4096 Aug 21 15:22 /var/lib/ceph/3cb12fba-5f6e-11ef-b412-9d303a22b70f/osd.6/
[root@ceph142 ~]#
[root@ceph142 ~]# cat /var/lib/ceph/3cb12fba-5f6e-11ef-b412-9d303a22b70f/osd.*/fsid
68ff55fb-358a-4014-ba0e-075adb18c6d9
b9096186-53af-4ca0-b233-01fd913bdaba
d4ccefb2-5812-4ca2-97ca-9642ff4539f2
[root@ceph142 ~]#
[root@ceph142 ~]# lsblk # 不难发现，ceph底层是基于lvm技术磁盘的。
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
...
sdb 8:16 0 300G 0 disk
└─ceph--bb7e7dd0--d4e2--4da2--9cfb--da1dcd70222d-osd--block--68ff55fb--358a--4014--ba0e--075adb18c6d9
253:1 0 300G 0 lvm
sdc 8:32 0 500G 0 disk
└─ceph--5b511438--e561--456f--a33e--82bfc9c4abfd-osd--block--b9096186--53af--4ca0--b233--01fd913bdaba
253:2 0 500G 0 lvm
sdd 8:48 0 1000G 0 disk
└─ceph--0d9e77e1--051d--4ba6--8274--cfd85e213ab9-osd--block--d4ccefb2--5812--4ca2--97ca--9642ff4539f2
253:3 0 1000G 0 lvm
...
[root@ceph142 ~]#
LVM学习资料：
https://www.cnblogs.com/cmy/p/18594123
2.2 查看集群的osd总容量大小
[root@ceph141 ~]# ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 5.34389 root default
-3 1.78130 host ceph141
0 hdd 0.29300 osd.0 up 1.00000 1.00000
1 hdd 0.48830 osd.1 up 1.00000 1.00000
2 hdd 1.00000 osd.2 up 1.00000 1.00000
-5 1.78130 host ceph142
3 hdd 0.29300 osd.3 up 1.00000 1.00000
4 hdd 0.48830 osd.4 up 1.00000 1.00000
5 hdd 1.00000 osd.5 up 1.00000 1.00000
-7 1.78130 host ceph143
6 hdd 0.29300 osd.6 up 1.00000 1.00000
7 hdd 0.48830 osd.7 up 1.00000 1.00000
8 hdd 1.00000 osd.8 up 1.00000 1.00000
[root@ceph141 ~]#
2.3 查看集群的大小
[root@ceph141 ~]# ceph -s
cluster:
id: 11e66474-0e02-11f0-82d6-4dcae3d59070
health: HEALTH_OK
services:
mon: 3 daemons, quorum ceph141,ceph142,ceph143 (age 42m)
mgr: ceph141.mbakds(active, since 2h), standbys: ceph142.qgifwo
osd: 9 osds: 9 up (since 2m), 9 in (since 2m)
data:
pools: 1 pools, 1 pgs
objects: 2 objects, 449 KiB
usage: 246 MiB used, 5.3 TiB / 5.3 TiB avail
pgs: 1 active+clean
[root@ceph141 ~]#
3.当然，也可以在dashboard中查看OSD信息
https://ceph141:8443/#/osd
- 测试ceph集群可用性
1.创建存储池
[root@ceph141 ~]# ceph osd pool create cmy
pool 'cmy' created
[root@ceph141 ~]#
[root@ceph141 ~]# ceph osd pool ls
.mgr
cmy
[root@ceph141 ~]#
2.往存储池上传文件
[root@ceph141 ~]# rados put sys.txt /etc/os-release -p cmy
3.查看存储池上传的文件
[root@ceph141 ~]# rados ls -p cmy
sys.txt
[root@ceph141 ~]#
4.查看存储池文件的状态信息
[root@ceph141 ~]# rados -p cmy stat sys.txt
cmy/sys.txt mtime 2025-05-16T11:56:26.000000+0800, size 427
[root@ceph141 ~]#
5.查看PG的副本在哪些OSD上
[root@ceph141 ~]# ceph osd map cmy sys.txt
osdmap e67 pool 'cmy' (2) object 'sys.txt' -> pg 2.486f5322 (2.2) -> up ([5,1,8], p5) acting ([5,1,8], p5)
[root@ceph141 ~]#
6.删除文件
[root@ceph141 ~]# rados -p cmy rm sys.txt
[root@ceph141 ~]#
[root@ceph141 ~]# rados -p cmy ls
[root@ceph141 ~]#
[root@ceph141 ~]# ceph osd map cmy sys.txt # 删除文件后不难发现映射信息还在，那如何删除这些映射信息呢？我们后面会陆续讲解到。
osdmap e66 pool 'cmy' (2) object 'sys.txt' -> pg 2.486f5322 (2.2) -> up ([5,1,8], p5) acting ([5,1,8], p5)
[root@ceph141 ~]#
7.查看系统的使用量
[root@ceph141 ~]# rados df
POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR USED COMPR UNDER COMPR
.mgr 3.5 MiB 2 0 6 0 0 0 96 82 KiB 113 1.3 MiB 0 B 0 B
cmy 0 B 0 0 0 0 0 0 1 0 B 1 0 B 0 B 0 B
total_objects 2
total_used 247 MiB
total_avail 5.3 TiB
total_space 5.3 TiB
[root@ceph141 ~]#
8.集群关键拍快照
[root@ceph141 ~]# init 0
[root@ceph142 ~]# init 0
[root@ceph143 ~]# init 0
略，见视频。
