Ceph 故障处理集

2025年

mds写入失败

2025/11/09 - ceph 集群 cephfs 故障处理

出现 1 clients failing to respond to cache pressure

1 clients failing to respond to cache pressure 表示:有 1 个 Ceph 客户端(如挂载 Ceph 文件系统的服务器、使用 RBD 的虚拟机、调用 RGW 的应用等)没有响应 Ceph 集群发送的 “缓存压力调节请求”。

Ceph 的 OSD(对象存储守护进程)会使用系统缓存(页缓存、BlueStore 缓存等)加速数据读写。当 OSD 的缓存使用率过高(比如接近阈值)时,集群会主动向客户端发送 “缓存压力通知(cache pressure hints)”,目的是:

  • 让客户端减少缓存占用(比如主动 flush 脏数据到集群、降低读写请求的缓存依赖);
  • 避免 OSD 缓存溢出导致性能骤降、IO 阻塞或集群不稳定。
    正常情况下,客户端收到通知后会配合调整;而这个告警的核心是:有 1 个客户端没按预期配合,导致集群无法通过客户端侧缓解 OSD 的缓存压力。

主要问题出现在客户端,需要排查

  1. 检查ceph详细健康状态 ceph health detail可以看到哪个客户端有问题
  2. 检查客户端使用的ceph版本是否和服务器的版本匹配,或者兼容
  3. 客户端是否无响应

一般情况下不用管,只需要持续监控 mds 服务是否正常即可

如果长时间无法恢复

ceph mds client recall 5576386  # 强制触发客户端缓存回收
ceph mds client evict 5576386  # 若回收无效,直接断开客户端连接(需确认客户端可重启)

# 查看 recall_caps 字段 ceph tell mds.${id} session ls

参考文章

结论就是扩大pool的池的配额,或者直接设置为0

2025/11/12 - ceph集群Pool池满了

删除rbd的镜像后,仍然还有大量空间

命令 显示的总量 统计范围说明
rbd du 214 GiB 仅统计当前存在的 RBD 镜像的实际数据量(逻辑大小)
ceph df 8.7 TiB 统计整个 pool 内所有对象的逻辑大小(含历史残留)

手动检查池子里面的对象,和 ceph df中查看的OBJECTS数量 2.36M基本一致

# 手动检查了池子里面的对象, 和
root@:~/public/# rbd trash ls -p rbd-pool
root@:~/public/# rados ls -p rbd-pool | wc -l
2364302
  • 想查看下大image的一个占比,先确认对象分布情况(选 used/provisioned 差距大的)
    root@:~/public/# rbd du -p rbd-pool | grep csi-vol-4502641b-7430-4b9d-8011-356d6d9d4a71
    csi-vol-4502641b-7430-4b9d-8011-356d6d9d4a71                           500 GiB  107 GiB
    
    root@:~/public/# rbd info -p rbd-pool csi-vol-4502641b-7430-4b9d-8011-356d6d9d4a71 rbd image 'csi-vol-4502641b-7430-4b9d-8011-356d6d9d4a71': size 500 GiB in 128000 objects order 22 (4 MiB objects) snapshot_count: 0 id: 512a6bd57f425f block_name_prefix: rbd_data.512a6bd57f425f format: 2 features: layering, exclusive-lock, object-map, fast-diff, deep-flatten op_features: flags: create_timestamp: Thu Nov 6 13:43:06 2025 access_timestamp: Thu Nov 6 13:43:06 2025 modify_timestamp: Thu Nov 6 13:43:06 2025
    # 手动统计该镜像的实际对象占用 root@:~/public/# rados ls -p rbd-pool | grep $(rbd info -p rbd-pool csi-vol-4502641b-7430-4b9d-8011-356d6d9d4a71 | grep block_name_prefix | awk '{print $2}') | wc -l 27355
    ## 或者导出后可以反复使用 rados ls -p rbd-pool > rbd-pool-objects grep $(rbd info -p rbd-pool csi-vol-4502641b-7430-4b9d-8011-356d6d9d4a71 | grep block_name_prefix | awk '{print $2}') rbd-pool-objects | wc -l

每个对象4M,通过计算 27355个对象,使用量就是107G,说明这一块没问题

查看rbd的object-map状态


root@:~//personal-configs/scripts# rbd ls -p rbd-pool | head -5 | while read img; do echo "=== $img ===" rbd object-map check -p rbd-pool $img done === csi-vol-0ae25b8d-c8cf-4e05-9c09-e3f11d7b32ef === Object Map Check: 100% complete...done. === csi-vol-123db2dd-836d-41a7-8789-266d858714ee === Object Map Check: 100% complete...done. === csi-vol-13aa50a7-f01e-4b0d-a6e2-476fdbebfa49 === Object Map Check: 100% complete...done. === csi-vol-14b57378-5285-4e2e-b474-086b782676e9 === Object Map Check: 100% complete...done. === csi-vol-1888c23e-37ae-44ee-9ac5-1b5dbe32dbdb === Object Map Check: 72% complete...2025-11-19T16:34:45.256+0800 737a7ea006c0 -1 librbd::ObjectMapIterateRequest: object map operation encountered an error: (85) Interrupted system call should be restarted Object Map Check: 81% complete...2025-11-19T16:34:48.341+0800 737a7f4006c0 -1 librbd::ObjectMapIterateRequest: object map operation encountered an error: (85) Interrupted system call should be restarted 2025-11-19T16:34:50.374+0800 737a7f4006c0 -1 librbd::ObjectMapIterateRequest: object map operation encountered an error: (85) Interrupted system call should be restarted 2025-11-19T16:34:51.420+0800 737a7ea006c0 -1 librbd::ObjectMapIterateRequest: object map operation encountered an error: (85) Interrupted system call should be restarted Object Map Check: 84% complete...2025-11-19T16:34:54.473+0800 737a7ea006c0 -1 librbd::ObjectMapIterateRequest: object map operation encountered an error: (85) Interrupted system call should be restarted 2025-11-19T16:34:54.512+0800 737a7f4006c0 -1 librbd::ObjectMapIterateRequest: object map operation encountered an error: (85) Interrupted system call should be restarted Object Map Check: 100% complete...done. root@:~//personal-configs/scripts#

统计所有对象后

root@:~/public/# awk -F'.' '{print $1"."$2}' rbd-pool-objects | sort | uniq -c | sort -nr
2308628 rbd_data.395060268cf494
  27355 rbd_data.512a6bd57f425f
  14258 rbd_data.455f4acb2c52a4
   3593 rbd_data.3935a4fee583f3
   2471 rbd_data.39506099c962df
   2292 rbd_data.3950601b813079
   1172 rbd_data.4254859861e5f4
    456 rbd_data.455f4aefa05d03
    430 rbd_data.455f4a29727af6
    428 rbd_data.455f4ab70761b5
    392 rbd_data.39f1a3ca1aad5b
    212 rbd_data.512a6b7726a29
    205 rbd_data.512a6b5aa51229
    200 rbd_data.455f4a8570e014
    188 rbd_data.512a6b54210242
    168 rbd_data.455f4a809e6edb
    162 rbd_data.395060666e2396
    151 rbd_data.512a6bbf8a02be
    144 rbd_data.3935a47f9b5ff0
    141 rbd_data.455f4ac4665426
    122 rbd_data.39f1a3d634125a
    116 rbd_data.3950607813e29c
    116 rbd_data.39506044e6ae1f
    112 rbd_data.455f4a7d4d42f7
    107 rbd_data.3935a4a7b1d677
     72 rbd_data.455f4adaf4bcf5
     71 rbd_data.39f1a3f58ea321
     71 rbd_data.39f1a3505eb8f9
     68 rbd_data.455f4a7283667f
     45 rbd_data.455f4a80a1beac
     45 csi.volume
     40 rbd_data.3942bf746b2609
     38 rbd_data.3935a46c425be3
     23 rbd_data.455f4a5bbfab69
     17 rbd_data.512a6b87f64d58
     16 rbd_data.512a6b6f2a7c8a
     16 rbd_data.512a6b1598424f
     16 rbd_data.455f4a5d1b90aa
     13 rbd_data.3935a482dada27
      7 rbd_data.455f4add4612a
      7 rbd_data.455f4a8ca0252f
      3 rbd_object_map.48d15db3a2c9fa
      1 rbd_trash.
      1 rbd_task.
      1 rbd_object_map.5c500b0136e7e
      1 rbd_object_map.512a6bd57f425f
      1 rbd_object_map.512a6bbf8a02be
      1 rbd_object_map.512a6b87f64d58
      1 rbd_object_map.512a6b7726a29
      1 rbd_object_map.512a6b6f2a7c8a
      1 rbd_object_map.512a6b5aa51229
      1 rbd_object_map.512a6b54210242
      1 rbd_object_map.512a6b1598424f
      1 rbd_object_map.48d15d6bf3707d
      1 rbd_object_map.455f4aefa05d03
      1 rbd_object_map.455f4add4612a
      1 rbd_object_map.455f4adaf4bcf5
      1 rbd_object_map.455f4acb2c52a4
      1 rbd_object_map.455f4ac4665426
      1 rbd_object_map.455f4ab70761b5
      1 rbd_object_map.455f4a8ca0252f
      1 rbd_object_map.455f4a89fc9daa
      1 rbd_object_map.455f4a8570e014
      1 rbd_object_map.455f4a80a1beac
      1 rbd_object_map.455f4a809e6edb
      1 rbd_object_map.455f4a7d4d42f7
      1 rbd_object_map.455f4a7283667f
      1 rbd_object_map.455f4a5d1b90aa
      1 rbd_object_map.455f4a5bbfab69
      1 rbd_object_map.455f4a54ee019b
      1 rbd_object_map.455f4a29727af6
      1 rbd_object_map.4254859861e5f4
      1 rbd_object_map.3c65277e3f6e0
      1 rbd_object_map.39f1a3f58ea321
      1 rbd_object_map.39f1a3d634125a
      1 rbd_object_map.39f1a3ca1aad5b
      1 rbd_object_map.39f1a3505eb8f9
      1 rbd_object_map.39506099c962df
      1 rbd_object_map.3950607813e29c
      1 rbd_object_map.395060666e2396
      1 rbd_object_map.39506044e6ae1f
      1 rbd_object_map.3950601b813079
      1 rbd_object_map.3942bf746b2609
      1 rbd_object_map.3935a4fee583f3
      1 rbd_object_map.3935a4a7b1d677
      1 rbd_object_map.3935a482dada27
      1 rbd_object_map.3935a47f9b5ff0
      1 rbd_object_map.3935a46c425be3
      1 rbd_info.
      1 rbd_id.testing
      1 rbd_id.test-image-2
      1 rbd_id.test-image
      1 rbd_id.csi-vol-f58fb300-6944-4b5d-8494-8842eb18a31e
      1 rbd_id.csi-vol-f3c3f4b6-ed2f-4b3a-89f9-28e56325e56f
      1 rbd_id.csi-vol-f0bff173-e28b-4db5-a26f-877626605a99
      1 rbd_id.csi-vol-eeaa22b5-8a49-45a8-908b-8e63d82cabac
      1 rbd_id.csi-vol-e370eb24-a3a4-493b-a9fb-233c39bea167
      1 rbd_id.csi-vol-d2ee2516-d861-482e-be55-6b98480f5875
      1 rbd_id.csi-vol-cc2822b6-e0d2-41a5-9d79-a543e4cf3f27
      1 rbd_id.csi-vol-c69f9306-d9dc-48c2-925f-12e67f27689f
      1 rbd_id.csi-vol-bcc71272-f7ba-4e5d-94a6-35231cb685a0
      1 rbd_id.csi-vol-b478245b-5204-40a0-891e-c1677d08ccc1
      1 rbd_id.csi-vol-adee51c0-88a8-4d1a-9e5b-1efb19aca053
      1 rbd_id.csi-vol-ad3ee245-a69c-4e76-b7c6-1dcbb88a4d17
      1 rbd_id.csi-vol-a8a76c04-f0c9-458a-8539-a2364cc52d15
      1 rbd_id.csi-vol-a7d7795e-843e-415f-8b57-565eb5de9a18
      1 rbd_id.csi-vol-a605bbc4-4d67-4d10-82b4-ba7aa9dac2a5
      1 rbd_id.csi-vol-a52ee0ce-ab12-4362-9574-1132a2cc2e2d
      1 rbd_id.csi-vol-a38a34c0-6973-4c95-809e-c213f0bd1ad8
      1 rbd_id.csi-vol-9c267c2f-f5d7-42f9-abe0-9a649a49ea06
      1 rbd_id.csi-vol-982bd666-d83b-4486-b5f4-e1955b1a6d03
      1 rbd_id.csi-vol-9674e35e-085b-4b7f-948d-58eec8a0e8d3
      1 rbd_id.csi-vol-93cd8c65-cde0-4b5b-b66d-27fc46c8af21
      1 rbd_id.csi-vol-899bf539-30aa-4185-892e-5b90c9d4cf61
      1 rbd_id.csi-vol-76445349-b611-49d5-8ef1-a05737e7fce5
      1 rbd_id.csi-vol-6f81ffcf-ef12-4cf3-a4f8-336e92124d62
      1 rbd_id.csi-vol-649417b2-589e-4fbc-8891-4fba611cda2d
      1 rbd_id.csi-vol-6142b2f0-3da4-4abb-95d6-9015b8bb0409
      1 rbd_id.csi-vol-5bf0df30-eeed-4414-8561-66615b96b0b0
      1 rbd_id.csi-vol-4a9db4b1-64b6-464c-8bf2-b1f9ba524867
      1 rbd_id.csi-vol-48cd8fb3-b424-4d60-a03f-a3c536f26657
      1 rbd_id.csi-vol-4613a01c-dcb1-4c68-96b0-35313a28ab42
      1 rbd_id.csi-vol-4502641b-7430-4b9d-8011-356d6d9d4a71
      1 rbd_id.csi-vol-3f2f2d15-e0f5-4fcf-84cf-2ab7c3e48340
      1 rbd_id.csi-vol-3eeec040-6965-4c6b-9e56-5aaae7a25830
      1 rbd_id.csi-vol-3da7dfba-2cec-4b33-96a0-1a55dc9c247f
      1 rbd_id.csi-vol-2a968f97-16ec-4aec-b92d-9e4bac2c9614
      1 rbd_id.csi-vol-23e3ced0-9796-4c1f-870f-40b9f19c2f69
      1 rbd_id.csi-vol-1ec3000f-df23-4d44-a5e5-abeb49db71af
      1 rbd_id.csi-vol-1888c23e-37ae-44ee-9ac5-1b5dbe32dbdb
      1 rbd_id.csi-vol-14b57378-5285-4e2e-b474-086b782676e9
      1 rbd_id.csi-vol-13aa50a7-f01e-4b0d-a6e2-476fdbebfa49
      1 rbd_id.csi-vol-123db2dd-836d-41a7-8789-266d858714ee
      1 rbd_id.csi-vol-0ae25b8d-c8cf-4e05-9c09-e3f11d7b32ef
      1 rbd_header.5c500b0136e7e
      1 rbd_header.512a6bd57f425f
      1 rbd_header.512a6bbf8a02be
      1 rbd_header.512a6b87f64d58
      1 rbd_header.512a6b7726a29
      1 rbd_header.512a6b6f2a7c8a
      1 rbd_header.512a6b5aa51229
      1 rbd_header.512a6b54210242
      1 rbd_header.512a6b1598424f
      1 rbd_header.48d15db3a2c9fa
      1 rbd_header.48d15d6bf3707d
      1 rbd_header.455f4aefa05d03
      1 rbd_header.455f4add4612a
      1 rbd_header.455f4adaf4bcf5
      1 rbd_header.455f4acb2c52a4
      1 rbd_header.455f4ac4665426
      1 rbd_header.455f4ab70761b5
      1 rbd_header.455f4a8ca0252f
      1 rbd_header.455f4a89fc9daa
      1 rbd_header.455f4a8570e014
      1 rbd_header.455f4a80a1beac
      1 rbd_header.455f4a809e6edb
      1 rbd_header.455f4a7d4d42f7
      1 rbd_header.455f4a7283667f
      1 rbd_header.455f4a5d1b90aa
      1 rbd_header.455f4a5bbfab69
      1 rbd_header.455f4a54ee019b
      1 rbd_header.455f4a29727af6
      1 rbd_header.4254859861e5f4
      1 rbd_header.3c65277e3f6e0
      1 rbd_header.39f1a3f58ea321
      1 rbd_header.39f1a3d634125a
      1 rbd_header.39f1a3ca1aad5b
      1 rbd_header.39f1a3505eb8f9
      1 rbd_header.39506099c962df
      1 rbd_header.3950607813e29c
      1 rbd_header.395060666e2396
      1 rbd_header.39506044e6ae1f
      1 rbd_header.3950601b813079
      1 rbd_header.3942bf746b2609
      1 rbd_header.3935a4fee583f3
      1 rbd_header.3935a4a7b1d677
      1 rbd_header.3935a482dada27
      1 rbd_header.3935a47f9b5ff0
      1 rbd_header.3935a46c425be3
      1 rbd_directory.
      1 rbd_children.
      1 csi.volumes

我统计了所有对象前缀,发现rbd_data.395060268cf494就是这个最多,基本就是8.8Ti的数据量,可以确定是这个

现在问题根源已经找到,就是孤儿对象前缀

  1. 那么为什么rbd du 不显示?
  • rbd du只读取元数据正确的镜像(通过rbd_id.<name>→rbd_header.<id>→rbd_data.<id> 追踪)
  • rbd_data.395060268cf494的元数据(rbd_id.*和rbd_header.395060268cf494)已不存在
  1. 为什么会出现?(CSI典型场景)
    当 Kubernetes 删除 PVC 时:

  2. CSI 调用rbd rm 删除镜像

  3. 元数据成功删除(rbd_id和rbd_header 被删)

  4. 数据对象删除失败(OSD 响应超时、网络中断、OSD 宕机)

  5. 结果是:镜像消失,数据残留

  6. 为什么rbd trash ls 为空?
    因为 CSI 的 rbd rm 直接删除,不经过 trash。trash 只包含 rbd trash move 命令移入的镜像。

通过脚本批量删除的方式,来对孤儿对象进行删除

# 确认该前缀没有对应任何现有镜像
rbd ls -p rbd-pool | while read img; do
  rbd info -p rbd-pool $img | grep "id: "
done | grep 395060268cf494

# 先将之前导出的 rados ls -p rbd-pool 的文件 rbd-pool-objects 提取孤儿对象前缀,手动确认,避免删除 grep rbd_data.395060268cf494 rbd-pool-objects > need_to_delete_objects
# 然后脚本批量删除,选个性能好的机器 cat > cleanup_orphan.sh <<'EOF' #!/bin/bash POOL="rbd-pool" PREFIX="rbd_data.395060268cf494" BATCH_SIZE=1000 WORKERS=32 OBJECTS_FILE="need_to_delete_objects"
echo "Starting cleanup of orphan prefix: $PREFIX" echo "Total objects: $(wc -l < "$OBJECTS_FILE")"
# 从文件读取对象列表并分批删除 cat "$OBJECTS_FILE" | \ xargs -P $WORKERS -n $BATCH_SIZE -I {} bash -c " if ! rados rm -p $POOL '{}'; then echo 'FAILED: {}' >&2 echo '{}' >> 'failed_objects.txt' fi"
echo "Cleanup completed." EOF
chmod +x cleanup_orphan.sh nohup ./cleanup_orphan.sh > cleanup_orphan.log 2>&1 &

添加OSD的时候,出现noout标识

root@:~# ceph health detail
HEALTH_WARN 1 OSDs or CRUSH {nodes, device-classes} have {NOUP,NODOWN,NOIN,NOOUT} flags set
[WRN] OSD_FLAGS: 1 OSDs or CRUSH {nodes, device-classes} have {NOUP,NODOWN,NOIN,NOOUT} flags set
    host ceph-store-01 has flags noout

这个其实是ceph的管理方式,有可能是rook-ceph调整后没有恢复,可以手动恢复

作用 使用场景
noup 防止OSD进程启动后自动标记为(up)状态 防止OSD反复重启,常用于调试
nodown 忽略OSD宕机报告,防止其标记为(down)状态 网络故障排除
noin 防止OSD在启动后自动加入集群(in)状态 需要手动确认OSD健康后再让其加入
noout 最重要,防止OSD宕机超时后自动标记为out(移出集群)而避免不必要的数据迁移 主机维护、计划内停机、磁盘更换
# 手动创建flag
ceph osd set <flag-name> # 作用所有osd
ceph osd set-group <flag-name> <who> # who 可以是osd或者node

# 手动移除flag ceph osd unset <flag-name> # 作用所有osd ceph osd unset-group <flag-name> <who> # who 可以是osd或者node

添加rgw时,出现cephobjectstore无法删除

问题出现的原因是rook-ceph期望与我手动创建的不符导致,需要手动删除。目标是s3-dev-pool

删除前先看看有没有桶,如果有,最后先手动删除掉

首先手动删除池子

ceph osd pool rm --yes-i-really-really-mean-it s3-dev-pool.rgw.control
ceph osd pool rm --yes-i-really-really-mean-it s3-dev-pool.rgw.control s3-dev-pool.rgw.control
ceph osd pool rm --yes-i-really-really-mean-it s3-dev-pool.rgw.meta s3-dev-pool.rgw.meta
ceph osd pool rm --yes-i-really-really-mean-it s3-dev-pool.rgw.log s3-dev-pool.rgw.log
ceph osd pool rm --yes-i-really-really-mean-it s3-dev-pool.rgw.buckets.index s3-dev-pool.rgw.buckets.index
ceph osd pool rm --yes-i-really-really-mean-it s3-dev-pool.rgw.buckets.non-ec s3-dev-pool.rgw.buckets.non-ec
ceph osd pool rm --yes-i-really-really-mean-it s3-dev-pool.rgw.otp s3-dev-pool.rgw.otp
ceph osd pool rm --yes-i-really-really-mean-it s3-dev-pool.rgw.buckets.data s3-dev-pool.rgw.buckets.data

# 或者直接批量删除(慎用!) for pool in $(ceph osd pool ls | grep s3-dev-pool);do ceph osd pool rm $pool $pool --yes-i-really-really-mean-it; done

此时crd 删除一直处于卡住的状态,手动停止掉

kubectl patch cephobjectstore s3-dev-pool -n rook-ceph \
  --type='json' \
  -p='[{"op": "remove", "path": "/metadata/finalizers"}]'

使其能够强制删除,并且删除了 orch里面的 rgw.s3-dev-pool

但是仍然有元数据残留导致,此处多站点管理,最好通过dashboard来处理

# 检查元数据状态,看是否真的有残留
radosgw-admin realm list
radosgw-admin zonegroup list
radosgw-admin zone list
radosgw-admin bucket list

# 查看默认配置 radosgw-admin realm get radosgw-admin zonegroup get radosgw-admin zone get
# 查看 s3-dev-pool 的详情 radosgw-admin zone get --rgw-zone=s3-dev-pool
# 检查 RADOS 中相关对象是否有残留 rados -p .rgw.root ls | grep s3-dev-pool
for zone in s3-dev-pool s3-pool default; do
    echo "Zone: $zone"
    radosgw-admin zone get --rgw-zone=$zone 2>/dev/null | grep -E '(name|realm_id|pool)'
    echo "---"
done

检查发现 s3-dev-pool 和 s3-pool 使用的是不同的realm,需要明确指定,先查看下realm

# 两个命令等效
radosgw-admin realm get --rgw-realm=s3-dev-pool
radosgw-admin realm get --rgw-realm-id=5709c714-31ee-4031-843b-5b8566134b15

最终检查结果为同一个zonegroup,放在了一个realm中,是多次配置错误导致,需要清理

# 清理
radosgw-admin zone delete --rgw-zone=s3-dev-pool
radosgw-admin zonegroup delete --rgw-zonegroup=s3-dev-pool
radosgw-admin realm rm --rgw-realm=s3-dev-pool

# 验证已经删除掉 radosgw-admin realm list radosgw-admin zonegroup list radosgw-admin zone list rados -p .rgw.root ls | grep s3-dev-pool

删除对应的CRUSH规则,查看对应创建的

# 查看ec规则和crush规则
ceph osd erasure-code-profile ls
ceph osd crush rule ls  # 也可用用dump

# 如果有残留,就删除 ceph osd erasure-code-profile rm s3-dev-pool.rgw.buckets.data_ecprofile ceph osd crush rule rm s3-dev-pool.rgw.buckets.data
## 也可用用批量清理(慎用!) for rule in $(ceph osd crush rule ls | grep s3-dev-pool); do ceph osd crush rule rm $rule; done

OMAP 过大告警

2025/11/26 - CephFS OMAP过大告警

orch ls 中信息没有更新

当添加osd或者添加rgw后,orch信息没有更新

原因是因为rook和orck的通讯存在问题,需要重启下 mgr 的 rook 模块

ceph mgr module disable rook
sleep 5
ceph mgr module enable rook

此步骤不会对业务造成影响

rgw创建后,dashboard中无法访问

Credentials not found for RGW Daemon: 's3-dev-pool'

重新生成一遍

ceph dashboard set-rgw-credentials

rook-ceph删除cephfs

在 Rook-Ceph 里把 CephFileSystem(CephFS)彻底删掉,需要按“业务 → Rook → Ceph”三个层次倒序清理

  • 业务侧清理

    # 删除掉用到Cephfs的Pod和PVC
    kubectl delete pvc -A --selector \
    -l "volume.beta.kubernetes.io/storage-class in (rook-cephfs,csi-cephfs)"
    
    # 删除掉快照 kubectl delete volumesnapshot -A --all kubectl delete volumesnapshotcontent -A --all
  • 清理Rook层(SC → CephFilesystemSubVolumeGroup → CephFilesystem)

    # sc
    kubectl delete sc rook-cephfs
    
    # 删除subvolumeGroup kubectl -n rook-ceph delete cephfilesystemsubvolumegroup csi
    # 删除CephFilesystem本体 kubectl -n rook-ceph delete cephfilesystem myfs

这一步最容易卡住:

  • 如果 subvolumeGroup 状态一直 Progressing,先确认上面所有 PVC、Pod 已释放;

  • 仍卡住的,把 finalizer 手动置空:

    kubectl -n rook-ceph patch cephfilesystemsubvolumegroup csi \
    --type merge -p '{"metadata":{"finalizers":[]}}'
    kubectl -n rook-ceph patch cephfilesystem myfs \
    --type merge -p '{"metadata":{"finalizers":[]}}'
    
  • 清理Ceph层(pool)

    # 删除文件系统
    ceph fs set myfs down true
    ceph fs rm myfs --yes-i-really-mean-it
    
    # 删除池 ceph osd pool rm myfs-replicated myfs-replicated --yes-i-really-really-mean-it ceph osd pool rm myfs-metadata myfs-metadata --yes-i-really-really-mean-it

创建基于EC的CephFS

2025/12/03 - 创建基于纠删码的CephFS

cephfs 数据损坏故障处理

2025/12/26 - [故障]cephfs 文件损坏

MDSs report damaged metadata

[ERR] MDS_DAMAGE: 1 MDSs report damaged metadata mds.public-fs-b(mds.0): Metadata damage detected

关键操作

# 查看详细的损坏列表
ceph tell mds.<name> damage ls

# 尝试恢复元数据 ceph tell mds.<name> scrub start <path> recursive repair
# 查看 scrub 状态(可能卡住了) ceph tell mds.<name> scrub status
# 如果没有自动清理掉,或者长时间卡住 # 停止当前 scrub ceph tell mds.<name> scrub abort
# 对每个 damage ID 执行删除(按 ID 号替换 <id>) ceph tell mds.<name> damage rm <damage_id>

2026

RBD镜像锁竞争问题

常出现在k8s调度,前一个还没有释放,后一个已经要加进来了

-device-type krbd --options noudev], rbd error output: rbd: sysfs write failed
rbd: map failed: (13)Permission denied
  • 根本原因是节点尝试挂载的时候,需要打破旧锁,并将旧客户端加入黑名单(blocklist)
  • 但是没有足够的执行权限去操作blocklist

但是要操作blocklist 是需要mon的rwx权限的,太大了

所以可以手动清理旧锁,或者干脆等待,有时候需要20分钟才能释放

# 在 Ceph 管理节点查看当前锁持有者
rbd lock list rbd-pool/csi-vol-8c3ebf79-7971-4c02-b16f-5bac050cd9b6

# 查看当前客户端列表 rbd status rbd-pool/csi-vol-8c3ebf79-7971-4c02-b16f-5bac050cd9b6
# 手动 blocklist 旧客户端(需要 admin 权限) ceph osd blocklist add 10.0.0.xxx # client42790400 对应的 IP ceph osd blocklist add 10.10.128.223:0/3360680447
# 或者解除镜像锁(如果确定旧客户端已死) rbd lock remove rbd-pool/csi-vol-8c3ebf79-7971-4c02-b16f-5bac050cd9b6 <lock-id> client42790400
rbd lock remove rbd-pool/csi-vol-8c3ebf79-7971-4c02-b16f-5bac050cd9b6 "auto 18446462598732840961" client.45160975

格式化硬盘导致无法识别OSD

因为格式化硬盘,导致无法识别OSD

Linux资源不足

Linux 资源不足 导致的问题

因为IB地址冲突导致的OSD掉线

2026/02/09 - B200的IB地址配置与存储冲突

4*15.7Ti 长时间感觉没有释放空间

清理后,6个小时过去,空间还未释放

确认问题为孤儿对象堆积,还是继续等待

PVC Pending

2026/04/02 -[故障总结] PVC Pending


转载请注明来源,欢迎对文章中的引用来源进行考证,欢迎指出任何有错误或不够清晰的表达。可以在下面评论区评论,也可以邮件至 365433079@qq.com