2025年
mds写入失败
2025/11/09 - ceph 集群 cephfs 故障处理
出现 1 clients failing to respond to cache pressure
1 clients failing to respond to cache pressure 表示:有 1 个 Ceph 客户端(如挂载 Ceph 文件系统的服务器、使用 RBD 的虚拟机、调用 RGW 的应用等)没有响应 Ceph 集群发送的 “缓存压力调节请求”。
Ceph 的 OSD(对象存储守护进程)会使用系统缓存(页缓存、BlueStore 缓存等)加速数据读写。当 OSD 的缓存使用率过高(比如接近阈值)时,集群会主动向客户端发送 “缓存压力通知(cache pressure hints)”,目的是:
- 让客户端减少缓存占用(比如主动 flush 脏数据到集群、降低读写请求的缓存依赖);
- 避免 OSD 缓存溢出导致性能骤降、IO 阻塞或集群不稳定。
正常情况下,客户端收到通知后会配合调整;而这个告警的核心是:有 1 个客户端没按预期配合,导致集群无法通过客户端侧缓解 OSD 的缓存压力。
主要问题出现在客户端,需要排查
- 检查ceph详细健康状态
ceph health detail可以看到哪个客户端有问题 - 检查客户端使用的ceph版本是否和服务器的版本匹配,或者兼容
- 客户端是否无响应
一般情况下不用管,只需要持续监控 mds 服务是否正常即可
如果长时间无法恢复
ceph mds client recall 5576386 # 强制触发客户端缓存回收
ceph mds client evict 5576386 # 若回收无效,直接断开客户端连接(需确认客户端可重启)
# 查看 recall_caps 字段
ceph tell mds.${id} session ls
参考文章
- https://www.cnblogs.com/bugutian/p/11844050.html
- https://cloud.tencent.com/developer/article/1664555
- https://blog.csdn.net/signmem/article/details/148918060
Pool 池满了
结论就是扩大pool的池的配额,或者直接设置为0
2025/11/12 - ceph集群Pool池满了
删除rbd的镜像后,仍然还有大量空间
| 命令 | 显示的总量 | 统计范围说明 |
|---|---|---|
rbd du |
214 GiB | 仅统计当前存在的 RBD 镜像的实际数据量(逻辑大小) |
ceph df |
8.7 TiB | 统计整个 pool 内所有对象的逻辑大小(含历史残留) |
手动检查池子里面的对象,和 ceph df中查看的OBJECTS数量 2.36M基本一致
# 手动检查了池子里面的对象, 和
root@:~/public/# rbd trash ls -p rbd-pool
root@:~/public/# rados ls -p rbd-pool | wc -l
2364302
- 想查看下大image的一个占比,先确认对象分布情况(选 used/provisioned 差距大的)
root@:~/public/# rbd du -p rbd-pool | grep csi-vol-4502641b-7430-4b9d-8011-356d6d9d4a71 csi-vol-4502641b-7430-4b9d-8011-356d6d9d4a71 500 GiB 107 GiB root@:~/public/# rbd info -p rbd-pool csi-vol-4502641b-7430-4b9d-8011-356d6d9d4a71 rbd image 'csi-vol-4502641b-7430-4b9d-8011-356d6d9d4a71': size 500 GiB in 128000 objects order 22 (4 MiB objects) snapshot_count: 0 id: 512a6bd57f425f block_name_prefix: rbd_data.512a6bd57f425f format: 2 features: layering, exclusive-lock, object-map, fast-diff, deep-flatten op_features: flags: create_timestamp: Thu Nov 6 13:43:06 2025 access_timestamp: Thu Nov 6 13:43:06 2025 modify_timestamp: Thu Nov 6 13:43:06 2025 # 手动统计该镜像的实际对象占用 root@:~/public/# rados ls -p rbd-pool | grep $(rbd info -p rbd-pool csi-vol-4502641b-7430-4b9d-8011-356d6d9d4a71 | grep block_name_prefix | awk '{print $2}') | wc -l 27355 ## 或者导出后可以反复使用 rados ls -p rbd-pool > rbd-pool-objects grep $(rbd info -p rbd-pool csi-vol-4502641b-7430-4b9d-8011-356d6d9d4a71 | grep block_name_prefix | awk '{print $2}') rbd-pool-objects | wc -l
每个对象4M,通过计算 27355个对象,使用量就是107G,说明这一块没问题
查看rbd的object-map状态
root@:~//personal-configs/scripts# rbd ls -p rbd-pool | head -5 | while read img; do
echo "=== $img ==="
rbd object-map check -p rbd-pool $img
done
=== csi-vol-0ae25b8d-c8cf-4e05-9c09-e3f11d7b32ef ===
Object Map Check: 100% complete...done.
=== csi-vol-123db2dd-836d-41a7-8789-266d858714ee ===
Object Map Check: 100% complete...done.
=== csi-vol-13aa50a7-f01e-4b0d-a6e2-476fdbebfa49 ===
Object Map Check: 100% complete...done.
=== csi-vol-14b57378-5285-4e2e-b474-086b782676e9 ===
Object Map Check: 100% complete...done.
=== csi-vol-1888c23e-37ae-44ee-9ac5-1b5dbe32dbdb ===
Object Map Check: 72% complete...2025-11-19T16:34:45.256+0800 737a7ea006c0 -1 librbd::ObjectMapIterateRequest: object map operation encountered an error: (85) Interrupted system call should be restarted
Object Map Check: 81% complete...2025-11-19T16:34:48.341+0800 737a7f4006c0 -1 librbd::ObjectMapIterateRequest: object map operation encountered an error: (85) Interrupted system call should be restarted
2025-11-19T16:34:50.374+0800 737a7f4006c0 -1 librbd::ObjectMapIterateRequest: object map operation encountered an error: (85) Interrupted system call should be restarted
2025-11-19T16:34:51.420+0800 737a7ea006c0 -1 librbd::ObjectMapIterateRequest: object map operation encountered an error: (85) Interrupted system call should be restarted
Object Map Check: 84% complete...2025-11-19T16:34:54.473+0800 737a7ea006c0 -1 librbd::ObjectMapIterateRequest: object map operation encountered an error: (85) Interrupted system call should be restarted
2025-11-19T16:34:54.512+0800 737a7f4006c0 -1 librbd::ObjectMapIterateRequest: object map operation encountered an error: (85) Interrupted system call should be restarted
Object Map Check: 100% complete...done.
root@:~//personal-configs/scripts#
统计所有对象后
root@:~/public/# awk -F'.' '{print $1"."$2}' rbd-pool-objects | sort | uniq -c | sort -nr
2308628 rbd_data.395060268cf494
27355 rbd_data.512a6bd57f425f
14258 rbd_data.455f4acb2c52a4
3593 rbd_data.3935a4fee583f3
2471 rbd_data.39506099c962df
2292 rbd_data.3950601b813079
1172 rbd_data.4254859861e5f4
456 rbd_data.455f4aefa05d03
430 rbd_data.455f4a29727af6
428 rbd_data.455f4ab70761b5
392 rbd_data.39f1a3ca1aad5b
212 rbd_data.512a6b7726a29
205 rbd_data.512a6b5aa51229
200 rbd_data.455f4a8570e014
188 rbd_data.512a6b54210242
168 rbd_data.455f4a809e6edb
162 rbd_data.395060666e2396
151 rbd_data.512a6bbf8a02be
144 rbd_data.3935a47f9b5ff0
141 rbd_data.455f4ac4665426
122 rbd_data.39f1a3d634125a
116 rbd_data.3950607813e29c
116 rbd_data.39506044e6ae1f
112 rbd_data.455f4a7d4d42f7
107 rbd_data.3935a4a7b1d677
72 rbd_data.455f4adaf4bcf5
71 rbd_data.39f1a3f58ea321
71 rbd_data.39f1a3505eb8f9
68 rbd_data.455f4a7283667f
45 rbd_data.455f4a80a1beac
45 csi.volume
40 rbd_data.3942bf746b2609
38 rbd_data.3935a46c425be3
23 rbd_data.455f4a5bbfab69
17 rbd_data.512a6b87f64d58
16 rbd_data.512a6b6f2a7c8a
16 rbd_data.512a6b1598424f
16 rbd_data.455f4a5d1b90aa
13 rbd_data.3935a482dada27
7 rbd_data.455f4add4612a
7 rbd_data.455f4a8ca0252f
3 rbd_object_map.48d15db3a2c9fa
1 rbd_trash.
1 rbd_task.
1 rbd_object_map.5c500b0136e7e
1 rbd_object_map.512a6bd57f425f
1 rbd_object_map.512a6bbf8a02be
1 rbd_object_map.512a6b87f64d58
1 rbd_object_map.512a6b7726a29
1 rbd_object_map.512a6b6f2a7c8a
1 rbd_object_map.512a6b5aa51229
1 rbd_object_map.512a6b54210242
1 rbd_object_map.512a6b1598424f
1 rbd_object_map.48d15d6bf3707d
1 rbd_object_map.455f4aefa05d03
1 rbd_object_map.455f4add4612a
1 rbd_object_map.455f4adaf4bcf5
1 rbd_object_map.455f4acb2c52a4
1 rbd_object_map.455f4ac4665426
1 rbd_object_map.455f4ab70761b5
1 rbd_object_map.455f4a8ca0252f
1 rbd_object_map.455f4a89fc9daa
1 rbd_object_map.455f4a8570e014
1 rbd_object_map.455f4a80a1beac
1 rbd_object_map.455f4a809e6edb
1 rbd_object_map.455f4a7d4d42f7
1 rbd_object_map.455f4a7283667f
1 rbd_object_map.455f4a5d1b90aa
1 rbd_object_map.455f4a5bbfab69
1 rbd_object_map.455f4a54ee019b
1 rbd_object_map.455f4a29727af6
1 rbd_object_map.4254859861e5f4
1 rbd_object_map.3c65277e3f6e0
1 rbd_object_map.39f1a3f58ea321
1 rbd_object_map.39f1a3d634125a
1 rbd_object_map.39f1a3ca1aad5b
1 rbd_object_map.39f1a3505eb8f9
1 rbd_object_map.39506099c962df
1 rbd_object_map.3950607813e29c
1 rbd_object_map.395060666e2396
1 rbd_object_map.39506044e6ae1f
1 rbd_object_map.3950601b813079
1 rbd_object_map.3942bf746b2609
1 rbd_object_map.3935a4fee583f3
1 rbd_object_map.3935a4a7b1d677
1 rbd_object_map.3935a482dada27
1 rbd_object_map.3935a47f9b5ff0
1 rbd_object_map.3935a46c425be3
1 rbd_info.
1 rbd_id.testing
1 rbd_id.test-image-2
1 rbd_id.test-image
1 rbd_id.csi-vol-f58fb300-6944-4b5d-8494-8842eb18a31e
1 rbd_id.csi-vol-f3c3f4b6-ed2f-4b3a-89f9-28e56325e56f
1 rbd_id.csi-vol-f0bff173-e28b-4db5-a26f-877626605a99
1 rbd_id.csi-vol-eeaa22b5-8a49-45a8-908b-8e63d82cabac
1 rbd_id.csi-vol-e370eb24-a3a4-493b-a9fb-233c39bea167
1 rbd_id.csi-vol-d2ee2516-d861-482e-be55-6b98480f5875
1 rbd_id.csi-vol-cc2822b6-e0d2-41a5-9d79-a543e4cf3f27
1 rbd_id.csi-vol-c69f9306-d9dc-48c2-925f-12e67f27689f
1 rbd_id.csi-vol-bcc71272-f7ba-4e5d-94a6-35231cb685a0
1 rbd_id.csi-vol-b478245b-5204-40a0-891e-c1677d08ccc1
1 rbd_id.csi-vol-adee51c0-88a8-4d1a-9e5b-1efb19aca053
1 rbd_id.csi-vol-ad3ee245-a69c-4e76-b7c6-1dcbb88a4d17
1 rbd_id.csi-vol-a8a76c04-f0c9-458a-8539-a2364cc52d15
1 rbd_id.csi-vol-a7d7795e-843e-415f-8b57-565eb5de9a18
1 rbd_id.csi-vol-a605bbc4-4d67-4d10-82b4-ba7aa9dac2a5
1 rbd_id.csi-vol-a52ee0ce-ab12-4362-9574-1132a2cc2e2d
1 rbd_id.csi-vol-a38a34c0-6973-4c95-809e-c213f0bd1ad8
1 rbd_id.csi-vol-9c267c2f-f5d7-42f9-abe0-9a649a49ea06
1 rbd_id.csi-vol-982bd666-d83b-4486-b5f4-e1955b1a6d03
1 rbd_id.csi-vol-9674e35e-085b-4b7f-948d-58eec8a0e8d3
1 rbd_id.csi-vol-93cd8c65-cde0-4b5b-b66d-27fc46c8af21
1 rbd_id.csi-vol-899bf539-30aa-4185-892e-5b90c9d4cf61
1 rbd_id.csi-vol-76445349-b611-49d5-8ef1-a05737e7fce5
1 rbd_id.csi-vol-6f81ffcf-ef12-4cf3-a4f8-336e92124d62
1 rbd_id.csi-vol-649417b2-589e-4fbc-8891-4fba611cda2d
1 rbd_id.csi-vol-6142b2f0-3da4-4abb-95d6-9015b8bb0409
1 rbd_id.csi-vol-5bf0df30-eeed-4414-8561-66615b96b0b0
1 rbd_id.csi-vol-4a9db4b1-64b6-464c-8bf2-b1f9ba524867
1 rbd_id.csi-vol-48cd8fb3-b424-4d60-a03f-a3c536f26657
1 rbd_id.csi-vol-4613a01c-dcb1-4c68-96b0-35313a28ab42
1 rbd_id.csi-vol-4502641b-7430-4b9d-8011-356d6d9d4a71
1 rbd_id.csi-vol-3f2f2d15-e0f5-4fcf-84cf-2ab7c3e48340
1 rbd_id.csi-vol-3eeec040-6965-4c6b-9e56-5aaae7a25830
1 rbd_id.csi-vol-3da7dfba-2cec-4b33-96a0-1a55dc9c247f
1 rbd_id.csi-vol-2a968f97-16ec-4aec-b92d-9e4bac2c9614
1 rbd_id.csi-vol-23e3ced0-9796-4c1f-870f-40b9f19c2f69
1 rbd_id.csi-vol-1ec3000f-df23-4d44-a5e5-abeb49db71af
1 rbd_id.csi-vol-1888c23e-37ae-44ee-9ac5-1b5dbe32dbdb
1 rbd_id.csi-vol-14b57378-5285-4e2e-b474-086b782676e9
1 rbd_id.csi-vol-13aa50a7-f01e-4b0d-a6e2-476fdbebfa49
1 rbd_id.csi-vol-123db2dd-836d-41a7-8789-266d858714ee
1 rbd_id.csi-vol-0ae25b8d-c8cf-4e05-9c09-e3f11d7b32ef
1 rbd_header.5c500b0136e7e
1 rbd_header.512a6bd57f425f
1 rbd_header.512a6bbf8a02be
1 rbd_header.512a6b87f64d58
1 rbd_header.512a6b7726a29
1 rbd_header.512a6b6f2a7c8a
1 rbd_header.512a6b5aa51229
1 rbd_header.512a6b54210242
1 rbd_header.512a6b1598424f
1 rbd_header.48d15db3a2c9fa
1 rbd_header.48d15d6bf3707d
1 rbd_header.455f4aefa05d03
1 rbd_header.455f4add4612a
1 rbd_header.455f4adaf4bcf5
1 rbd_header.455f4acb2c52a4
1 rbd_header.455f4ac4665426
1 rbd_header.455f4ab70761b5
1 rbd_header.455f4a8ca0252f
1 rbd_header.455f4a89fc9daa
1 rbd_header.455f4a8570e014
1 rbd_header.455f4a80a1beac
1 rbd_header.455f4a809e6edb
1 rbd_header.455f4a7d4d42f7
1 rbd_header.455f4a7283667f
1 rbd_header.455f4a5d1b90aa
1 rbd_header.455f4a5bbfab69
1 rbd_header.455f4a54ee019b
1 rbd_header.455f4a29727af6
1 rbd_header.4254859861e5f4
1 rbd_header.3c65277e3f6e0
1 rbd_header.39f1a3f58ea321
1 rbd_header.39f1a3d634125a
1 rbd_header.39f1a3ca1aad5b
1 rbd_header.39f1a3505eb8f9
1 rbd_header.39506099c962df
1 rbd_header.3950607813e29c
1 rbd_header.395060666e2396
1 rbd_header.39506044e6ae1f
1 rbd_header.3950601b813079
1 rbd_header.3942bf746b2609
1 rbd_header.3935a4fee583f3
1 rbd_header.3935a4a7b1d677
1 rbd_header.3935a482dada27
1 rbd_header.3935a47f9b5ff0
1 rbd_header.3935a46c425be3
1 rbd_directory.
1 rbd_children.
1 csi.volumes
我统计了所有对象前缀,发现rbd_data.395060268cf494就是这个最多,基本就是8.8Ti的数据量,可以确定是这个
现在问题根源已经找到,就是孤儿对象前缀
- 那么为什么rbd du 不显示?
rbd du只读取元数据正确的镜像(通过rbd_id.<name>→rbd_header.<id>→rbd_data.<id>追踪)rbd_data.395060268cf494的元数据(rbd_id.*和rbd_header.395060268cf494)已不存在
为什么会出现?(CSI典型场景)
当 Kubernetes 删除 PVC 时:CSI 调用
rbd rm删除镜像元数据成功删除(
rbd_id和rbd_header被删)数据对象删除失败(OSD 响应超时、网络中断、OSD 宕机)
结果是:镜像消失,数据残留
为什么
rbd trash ls为空?
因为 CSI 的rbd rm直接删除,不经过 trash。trash 只包含rbd trash move命令移入的镜像。
通过脚本批量删除的方式,来对孤儿对象进行删除
# 确认该前缀没有对应任何现有镜像
rbd ls -p rbd-pool | while read img; do
rbd info -p rbd-pool $img | grep "id: "
done | grep 395060268cf494
# 先将之前导出的 rados ls -p rbd-pool 的文件 rbd-pool-objects 提取孤儿对象前缀,手动确认,避免删除
grep rbd_data.395060268cf494 rbd-pool-objects > need_to_delete_objects
# 然后脚本批量删除,选个性能好的机器
cat > cleanup_orphan.sh <<'EOF'
#!/bin/bash
POOL="rbd-pool"
PREFIX="rbd_data.395060268cf494"
BATCH_SIZE=1000
WORKERS=32
OBJECTS_FILE="need_to_delete_objects"
echo "Starting cleanup of orphan prefix: $PREFIX"
echo "Total objects: $(wc -l < "$OBJECTS_FILE")"
# 从文件读取对象列表并分批删除
cat "$OBJECTS_FILE" | \
xargs -P $WORKERS -n $BATCH_SIZE -I {} bash -c "
if ! rados rm -p $POOL '{}'; then
echo 'FAILED: {}' >&2
echo '{}' >> 'failed_objects.txt'
fi"
echo "Cleanup completed."
EOF
chmod +x cleanup_orphan.sh
nohup ./cleanup_orphan.sh > cleanup_orphan.log 2>&1 &
添加OSD的时候,出现noout标识
root@:~# ceph health detail
HEALTH_WARN 1 OSDs or CRUSH {nodes, device-classes} have {NOUP,NODOWN,NOIN,NOOUT} flags set
[WRN] OSD_FLAGS: 1 OSDs or CRUSH {nodes, device-classes} have {NOUP,NODOWN,NOIN,NOOUT} flags set
host ceph-store-01 has flags noout
这个其实是ceph的管理方式,有可能是rook-ceph调整后没有恢复,可以手动恢复
| 作用 | 使用场景 | |
|---|---|---|
| noup | 防止OSD进程启动后自动标记为(up)状态 | 防止OSD反复重启,常用于调试 |
| nodown | 忽略OSD宕机报告,防止其标记为(down)状态 | 网络故障排除 |
| noin | 防止OSD在启动后自动加入集群(in)状态 | 需要手动确认OSD健康后再让其加入 |
| noout | 最重要,防止OSD宕机超时后自动标记为out(移出集群)而避免不必要的数据迁移 | 主机维护、计划内停机、磁盘更换 |
# 手动创建flag
ceph osd set <flag-name> # 作用所有osd
ceph osd set-group <flag-name> <who> # who 可以是osd或者node
# 手动移除flag
ceph osd unset <flag-name> # 作用所有osd
ceph osd unset-group <flag-name> <who> # who 可以是osd或者node
添加rgw时,出现cephobjectstore无法删除
问题出现的原因是rook-ceph期望与我手动创建的不符导致,需要手动删除。目标是s3-dev-pool
删除前先看看有没有桶,如果有,最后先手动删除掉
首先手动删除池子
ceph osd pool rm --yes-i-really-really-mean-it s3-dev-pool.rgw.control
ceph osd pool rm --yes-i-really-really-mean-it s3-dev-pool.rgw.control s3-dev-pool.rgw.control
ceph osd pool rm --yes-i-really-really-mean-it s3-dev-pool.rgw.meta s3-dev-pool.rgw.meta
ceph osd pool rm --yes-i-really-really-mean-it s3-dev-pool.rgw.log s3-dev-pool.rgw.log
ceph osd pool rm --yes-i-really-really-mean-it s3-dev-pool.rgw.buckets.index s3-dev-pool.rgw.buckets.index
ceph osd pool rm --yes-i-really-really-mean-it s3-dev-pool.rgw.buckets.non-ec s3-dev-pool.rgw.buckets.non-ec
ceph osd pool rm --yes-i-really-really-mean-it s3-dev-pool.rgw.otp s3-dev-pool.rgw.otp
ceph osd pool rm --yes-i-really-really-mean-it s3-dev-pool.rgw.buckets.data s3-dev-pool.rgw.buckets.data
# 或者直接批量删除(慎用!)
for pool in $(ceph osd pool ls | grep s3-dev-pool);do ceph osd pool rm $pool $pool --yes-i-really-really-mean-it; done
此时crd 删除一直处于卡住的状态,手动停止掉
kubectl patch cephobjectstore s3-dev-pool -n rook-ceph \
--type='json' \
-p='[{"op": "remove", "path": "/metadata/finalizers"}]'
使其能够强制删除,并且删除了 orch里面的 rgw.s3-dev-pool
但是仍然有元数据残留导致,此处多站点管理,最好通过dashboard来处理
# 检查元数据状态,看是否真的有残留
radosgw-admin realm list
radosgw-admin zonegroup list
radosgw-admin zone list
radosgw-admin bucket list
# 查看默认配置
radosgw-admin realm get
radosgw-admin zonegroup get
radosgw-admin zone get
# 查看 s3-dev-pool 的详情
radosgw-admin zone get --rgw-zone=s3-dev-pool
# 检查 RADOS 中相关对象是否有残留
rados -p .rgw.root ls | grep s3-dev-pool
for zone in s3-dev-pool s3-pool default; do
echo "Zone: $zone"
radosgw-admin zone get --rgw-zone=$zone 2>/dev/null | grep -E '(name|realm_id|pool)'
echo "---"
done
检查发现 s3-dev-pool 和 s3-pool 使用的是不同的realm,需要明确指定,先查看下realm
# 两个命令等效
radosgw-admin realm get --rgw-realm=s3-dev-pool
radosgw-admin realm get --rgw-realm-id=5709c714-31ee-4031-843b-5b8566134b15
最终检查结果为同一个zonegroup,放在了一个realm中,是多次配置错误导致,需要清理
# 清理
radosgw-admin zone delete --rgw-zone=s3-dev-pool
radosgw-admin zonegroup delete --rgw-zonegroup=s3-dev-pool
radosgw-admin realm rm --rgw-realm=s3-dev-pool
# 验证已经删除掉
radosgw-admin realm list
radosgw-admin zonegroup list
radosgw-admin zone list
rados -p .rgw.root ls | grep s3-dev-pool
删除对应的CRUSH规则,查看对应创建的
# 查看ec规则和crush规则
ceph osd erasure-code-profile ls
ceph osd crush rule ls # 也可用用dump
# 如果有残留,就删除
ceph osd erasure-code-profile rm s3-dev-pool.rgw.buckets.data_ecprofile
ceph osd crush rule rm s3-dev-pool.rgw.buckets.data
## 也可用用批量清理(慎用!)
for rule in $(ceph osd crush rule ls | grep s3-dev-pool); do ceph osd crush rule rm $rule; done
OMAP 过大告警
2025/11/26 - CephFS OMAP过大告警
orch ls 中信息没有更新
当添加osd或者添加rgw后,orch信息没有更新
原因是因为rook和orck的通讯存在问题,需要重启下 mgr 的 rook 模块
ceph mgr module disable rook
sleep 5
ceph mgr module enable rook
此步骤不会对业务造成影响
rgw创建后,dashboard中无法访问
Credentials not found for RGW Daemon: 's3-dev-pool'
重新生成一遍
ceph dashboard set-rgw-credentials
rook-ceph删除cephfs
在 Rook-Ceph 里把 CephFileSystem(CephFS)彻底删掉,需要按“业务 → Rook → Ceph”三个层次倒序清理
业务侧清理
# 删除掉用到Cephfs的Pod和PVC kubectl delete pvc -A --selector \ -l "volume.beta.kubernetes.io/storage-class in (rook-cephfs,csi-cephfs)" # 删除掉快照 kubectl delete volumesnapshot -A --all kubectl delete volumesnapshotcontent -A --all清理Rook层(SC → CephFilesystemSubVolumeGroup → CephFilesystem)
# sc kubectl delete sc rook-cephfs # 删除subvolumeGroup kubectl -n rook-ceph delete cephfilesystemsubvolumegroup csi # 删除CephFilesystem本体 kubectl -n rook-ceph delete cephfilesystem myfs
这一步最容易卡住:
如果 subvolumeGroup 状态一直 Progressing,先确认上面所有 PVC、Pod 已释放;
仍卡住的,把 finalizer 手动置空:
kubectl -n rook-ceph patch cephfilesystemsubvolumegroup csi \ --type merge -p '{"metadata":{"finalizers":[]}}' kubectl -n rook-ceph patch cephfilesystem myfs \ --type merge -p '{"metadata":{"finalizers":[]}}'清理Ceph层(pool)
# 删除文件系统 ceph fs set myfs down true ceph fs rm myfs --yes-i-really-mean-it # 删除池 ceph osd pool rm myfs-replicated myfs-replicated --yes-i-really-really-mean-it ceph osd pool rm myfs-metadata myfs-metadata --yes-i-really-really-mean-it
创建基于EC的CephFS
2025/12/03 - 创建基于纠删码的CephFS
cephfs 数据损坏故障处理
2025/12/26 - [故障]cephfs 文件损坏
MDSs report damaged metadata
[ERR] MDS_DAMAGE: 1 MDSs report damaged metadata
mds.public-fs-b(mds.0): Metadata damage detected
关键操作
# 查看详细的损坏列表
ceph tell mds.<name> damage ls
# 尝试恢复元数据
ceph tell mds.<name> scrub start <path> recursive repair
# 查看 scrub 状态(可能卡住了)
ceph tell mds.<name> scrub status
# 如果没有自动清理掉,或者长时间卡住
# 停止当前 scrub
ceph tell mds.<name> scrub abort
# 对每个 damage ID 执行删除(按 ID 号替换 <id>)
ceph tell mds.<name> damage rm <damage_id>
2026
RBD镜像锁竞争问题
常出现在k8s调度,前一个还没有释放,后一个已经要加进来了
-device-type krbd --options noudev], rbd error output: rbd: sysfs write failed
rbd: map failed: (13)Permission denied
- 根本原因是节点尝试挂载的时候,需要打破旧锁,并将旧客户端加入黑名单(blocklist)
- 但是没有足够的执行权限去操作blocklist
但是要操作blocklist 是需要mon的rwx权限的,太大了
所以可以手动清理旧锁,或者干脆等待,有时候需要20分钟才能释放
# 在 Ceph 管理节点查看当前锁持有者
rbd lock list rbd-pool/csi-vol-8c3ebf79-7971-4c02-b16f-5bac050cd9b6
# 查看当前客户端列表
rbd status rbd-pool/csi-vol-8c3ebf79-7971-4c02-b16f-5bac050cd9b6
# 手动 blocklist 旧客户端(需要 admin 权限)
ceph osd blocklist add 10.0.0.xxx # client42790400 对应的 IP
ceph osd blocklist add 10.10.128.223:0/3360680447
# 或者解除镜像锁(如果确定旧客户端已死)
rbd lock remove rbd-pool/csi-vol-8c3ebf79-7971-4c02-b16f-5bac050cd9b6 <lock-id> client42790400
rbd lock remove rbd-pool/csi-vol-8c3ebf79-7971-4c02-b16f-5bac050cd9b6 "auto 18446462598732840961" client.45160975
格式化硬盘导致无法识别OSD
因为格式化硬盘,导致无法识别OSD
Linux资源不足
Linux 资源不足 导致的问题
因为IB地址冲突导致的OSD掉线
2026/02/09 - B200的IB地址配置与存储冲突
4*15.7Ti 长时间感觉没有释放空间
清理后,6个小时过去,空间还未释放
确认问题为孤儿对象堆积,还是继续等待
PVC Pending
2026/04/02 -[故障总结] PVC Pending
转载请注明来源,欢迎对文章中的引用来源进行考证,欢迎指出任何有错误或不够清晰的表达。可以在下面评论区评论,也可以邮件至 365433079@qq.com