openstack对接ceph存储

## 1. 现状
- NFS Server
- NFS server导出以下目录：

nova虚拟机实例的状态存储：/nfs_data/instances/
虚拟机镜像存储：/var/lib/glance/images/
## Nova
nova.conf

```
[DEFAULT]
log_dir = /var/log/nova
lock_path = /var/lock/nova
state_path = /var/lib/nova
saleone.properties
```

config/scaleone.properties:share_mount_cmd="mount ${nfs_server}:${nfs_location} /var/lib/nova/instances/"
Cinder (块存储)
cinder.conf配置文件

```markdown
[DEFAULT]
volume_name_template = volume-%s
volume_group = cinder-volumes
volumes_dir = /var/lib/cinder/volumes
enabled_backends = lvm
glance_api_servers = http://controller:9292

[lvm]
volume_driver = cinder.volume.drivers.lvm.LVMVolumeDriver
volume_group = cinder-volumes
iscsi_protocol = iscsi
target_helper = tgtadm
```
## Glance (虚拟机镜像存储和管理)
### 2. 解决方案
### 软件准备
- ceph
ceph.tar.gz
### ceph户端安装
在安装openstack软件之前，必须先安装ceph，并复制好ceph.conf和授权文件。
存储模式配成ceph，挂载命令配成cephfs的挂载命令。例：

```markdown
share_disk=ceph
share_mount_cmd="ceph-fuse -o nonempty /var/lib/nova/instances/"
```
安装步骤：

1. 在每个机器上解压ceph.tar.gz
2. 执行ceph/packages/rpm目录下的install.sh脚本
`bash ceph/packages/rpm/install.sh`
## Nova使用ceph的文件存储
1. 在计算节点上停止nova服务
`systemctl stop openstack-nova-compute`
2. 在计算节点上取消nfs挂载
`umount /var/lib/nova/instances`
3. 启动ceph-client容器
```markdown
#!/bin/bash
nerdctl run --hostname=ceph-client \
        --name=ceph-client \
        --privileged \
        -d \
        -v /var/run/udev:/var/run/udev \
        -v /opt/:/opt \
        -v /dev:/dev \
        -v /etc/hosts:/etc/hosts \
        -v /etc/ceph:/etc/ceph \
        -v /var/lib/nova:/var/lib/nova:shared \
        --entrypoint=/sbin/init \
        registry.binone.tech/ceph/ceph-client:7.2.1
```
4. 在ceph-client容器中，执行ceph-fuse -o nonempty /var/lib/nova/instances命令将ceph的文件系统挂载到nova目录，

``` shell
[root@ceph-client rpm]# ceph-fuse -o nonempty /var/lib/nova/instances
ceph-fuse[641]: starting fuse
ceph-fuse            282G     0  282G   0% /var/lib/nova/instances

[root@ceph-client /]# df -h
Filesystem               Size  Used Avail Use% Mounted on
overlay                  913G   13G  901G   2% /
tmpfs                     24G     0   24G   0% /sys/fs/cgroup
tmpfs                     24G  8.8M   24G   1% /run
/dev/mapper/centos-root  913G   13G  901G   2% /opt
devtmpfs                  24G  4.0K   24G   1% /dev
tmpfs                     24G     0   24G   0% /dev/shm
ceph-fuse            282G     0  282G   0% /var/lib/nova/instances
```
5. `chown nova:nova /var/lib/nova/instances`

6. 在宿主机上启动nova服务

```markdown
systemctl start openstack-nova-compute
```
## Glance使用ceph文件存储
1. 在ceph上创建pool、glance卷、以及用户
```markdown
ceph osd pool create glance_metadata
ceph osd pool create glance
ceph fs flag set enable_multiple true
ceph fs new glance glance_metadata glance

ceph auth get-or-create client.glance mon 'allow r' \
  mds 'allow r, allow rw path=/glance' osd 'allow rw pool=glance'
```
2. 在控制节点上停止glance服务
```markdown
systemctl stop openstack-glance-api
systemctl stop openstack-glance-registry
```
3. 将ceph上的用户glance的keyring，导出到控制节点中的/etc/ceph/目录下

4. 启动ceph-client容器

```markdown
#!/bin/bash

nerdctl run --hostname=ceph-client \
        --name=ceph-client \
        --privileged \
        -d \
        -v /var/run/udev:/var/run/udev \
        -v /opt/:/opt \
        -v /dev:/dev \
        -v /etc/hosts:/etc/hosts \
        -v /etc/ceph:/etc/ceph \
        -v /var/lib/glance:/var/lib/glance:shared \
        --entrypoint=/sbin/init \
        registry.binone.tech/ceph/ceph-client:7.2.1
```
5. 在ceph-client容器中挂载glance存储卷
```markdown
ceph-fuse -n client.glance -o nonempty /var/lib/glance/images
```
## Cinder使用ceph
1. 在ceph中创建块设备

```markdown
 ceph osd pool create cinder
 rbd create --size 10000 cinder/vol
 rbd info cinder/vol
 rbd feature disable cinder/vol fast-diff
 rbd feature disable cinder/vol deep-flatten
 rbd feature disable cinder/vol exclusive-lock
```
2. 在计算节点上停止cinder service

```markdown
 systemctl stop openstack-cinder-volume
```
3. 启动ceph-client容器

```markdown
 #!/bin/bash

nerdctl run --hostname=ceph-client \
         --name=ceph-client \
         --privileged \
         -d \
         -v /var/run/udev:/var/run/udev \
         -v /opt/:/opt \
         -v /dev:/dev \
         -v /etc/hosts:/etc/hosts \
         -v /etc/ceph:/etc/ceph \
         -v /var/lib/glance:/var/lib/glance:shared \
         --entrypoint=/sbin/init \
         registry.binone.tech/ceph/ceph-client:7.2.1
```
4. 在容器ceph-client容器中映射块设备

```markdown
 rbd device map cinder/vol
```
5. 在计算节点上，修改lvm配置/etc/lvm/lvm.conf，以支持创建/dev/rbd类型的pvc

```markdown
 filter = [ "a/sda/","a/sdc/", "a/sdd/", "a/rbd/", "r/.*/" ]
 types = [ "sdd", 1024, "rbd", 1024 ]
```
6. 在计算节点上创建pv和vg

```markdown
 pvcreate /dev/rbd0
 vgcreate -q cinder-volumes /dev/rbd0
```
7. 在计算节点上启动cinder service

```markdown
 systemctl start openstack-cinder-volume
```