运维宝典

## Cephfs重建后，无法启动Pod
现象：

```shell
[root@host4193 storeone]# kubectl logs -f csi-cephfsplugin-swqxb -n binone-components
error: a container name must be specified for pod csi-cephfsplugin-swqxb, choose one of: [driver-registrar csi-cephfsplugin]
[root@host4193 storeone]# kubectl logs -f csi-cephfsplugin-swqxb -n binone-components csi-cephfsplugin
E0710 05:41:20.328170       1 utils.go:200] ID: 4 Req-ID: 0001-0011-binone-components-0000000000000001-b888c8d3-1af4-11ee-98d3-000000d24c3b GRPC error: rpc error: code = Internal desc = rpc error: code = Internal desc = pool not found: fscID (1) not found in Ceph cluster
E0710 05:43:22.440693       1 utils.go:200] ID: 6 Req-ID: 0001-0011-binone-components-0000000000000001-b888c8d3-1af4-11ee-98d3-000000d24c3b GRPC error: rpc error: code = Internal desc = rpc error: code = Internal desc = pool not found: fscID (1) not found in Ceph cluster
^C
```
参考：

- https://bugzilla.redhat.com/show_bug.cgi?id=1973256
- https://github.com/rook/rook/issues/4660
- https://bugzilla.redhat.com/show_bug.cgi?id=1975608
解决方法：

`ceph fs new <name> --fcsid 1 --force`
## MDS daemon damaged导致BinOne都不可用
#### 现象

![](/media/202311/clipboard-202307071800-sfudo_1701252962.png)

#### 原因

不清楚

#### 解决方法

1. 创建恢复用的filesystem，并使用原来ceph-filesystem的data pool

```shell
ceph fs flag set enable_multiple true --yes-i-really-mean-it
ceph fs new recovery-fs recovery ceph-filesystem-metadata --allow-dangerous-metadata-overlay
cephfs-data-scan init --force-init --filesystem recovery-fs --alternate-pool recovery
cephfs-data-scan scan_extents --force-pool --alternate-pool recovery --filesystem ceph-filesystem ceph-filesystem-data0
cephfs-data-scan scan_inodes --alternate-pool recovery --filesystem ceph-filesystem --force-corrupt --force-init ceph-filesystem-data0
cephfs-data-scan scan_links --filesystem recovery-fs
```
2. 删除老的ceph-filesystem

```shell
ceph fs fail ceph-file-system
ceph fs rm ceph-filesystem --yes-i-really-mean-it
```
3. 重启mds pod

4. 重新创建filesystem，并使用原来的ceph-filesystem-data0 pool

```shell
ceph osd pool create binone 8
ceph fs new ceph-filesystem --fscid 1 --force binone ceph-filesystem-data0 --allow-dangerous-metadata-overlay
cephfs-data-scan init --force-init --filesystem ceph-filesystem --alternate-pool binone

cephfs-data-scan scan_extents --force-pool --alternate-pool binone --filesystem recovery-fs ceph-filesystem-data0
cephfs-data-scan scan_inodes --alternate-pool binone --filesystem recovery-fs --force-corrupt --force-init ceph-filesystem-data0
cephfs-data-scan scan_links --filesystem ceph-filesystem
```
5. 重启mds pod

最终还是未解决问题。最终harbor-registry pod还是无法启动，查看csi-cephfsplugin，日志如下：

```
E0710 07:45:12.184738       1 omap.go:78] ID: 16 Req-ID: 0001-0011-binone-components-0000000000000001-b888c8d3-1af4-11ee-98d3-000000d24c3b omap not found (pool="binone", namespace="csi", name="csi.volume.b888c8d3-1af4-11ee-98d3-000000d24c3b"): rados: ret=-2, No such file or directory
W0710 07:45:12.184787       1 voljournal.go:690] ID: 16 Req-ID: 0001-0011-binone-components-0000000000000001-b888c8d3-1af4-11ee-98d3-000000d24c3b unable to read omap keys: pool or key missing: key not found: rados: ret=-2, No such file or directory
E0710 07:45:12.193752       1 volume.go:146] ID: 16 Req-ID: 0001-0011-binone-components-0000000000000001-b888c8d3-1af4-11ee-98d3-000000d24c3b failed to get subvolume info for the vol csi-vol-b888c8d3-1af4-11ee-98d3-000000d24c3b: rados: ret=-61, No data available: "error in getxattr"
E0710 07:45:12.193907       1 utils.go:200] ID: 16 Req-ID: 0001-0011-binone-components-0000000000000001-b888c8d3-1af4-11ee-98d3-000000d24c3b GRPC error: rpc error: code = Internal desc = rpc error: code = Internal desc = rados: ret=-61, No data available: "error in getxattr"
```
查看`subvolume info`报错：
` ceph> fs subvolume info ceph-filesystem csi-vol-b888c8d3-1af4-11ee-98d3-000000d24c3b csi Error ENODATA: error in getxattr `

数据未丢失：

![](/media/202311/2023-07-10_154806_42119109854378721045914_1701251039.png)

## 虚拟机
#### 无法删除vmi
解决方法：
```shell
$ kubectl edit vmi test-vmi1, remove

Finalizers:
    foregroundDeleteVirtualMachine
```

### 虚拟机总是调度到特定节点上
##### 解决方法

给节点打标签：

```shell
kubectl label node 11.0.5.111 def-provider.provider-network.kubernetes.io/exclude=true
```