在 kubernetes 中使用 glusterfs 作为 pv。
environment Link to heading
- centos7
- 3.10.0-957.27.2.el7.x86_64
机器（virtualbox 虚拟机） Link to heading
- centos10 - 172.27.32.165 - kubernetes master节点/glusterfs节点
- centos12l - 172.27.32.182 - glusterfs节点/kubernetes node节点
- centos11 - 172.27.32.164 - glusterfs节点/kubernetes node节点
virtualbox 添加硬盘 Link to heading
- 关闭虚拟机
- 设置-存储-控制器SATA-新建磁盘-固定大小
- 启动虚拟机
开机后执行fdisk -l，其中 /dev/sdb 为新创建出的磁盘。
[root@centos12l ~]$ fdisk -l
Disk /dev/sda: 54.5 GB, 54495248384 bytes, 106436032 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x0001552e
Device Boot Start End Blocks Id System
/dev/sda1 * 2048 2099199 1048576 83 Linux
/dev/sda2 2099200 106434559 52167680 8e Linux LVM
Disk /dev/sdb: 10.7 GB, 10737418240 bytes, 20971520 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/mapper/centos-root: 51.3 GB, 51266977792 bytes, 100130816 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/mapper/centos-swap: 2147 MB, 2147483648 bytes, 4194304 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
如果使用gluster-kubernetes提供的gk-deploy脚本配置glusterfs，无需进行一下的磁盘挂载操作。
磁盘挂载 Link to heading
以下为手动部署 glusterfs 的步骤，使用gk-deploy在kubernetes上使用glusterfs跳过此步，否则会创建失败。
执行fdisk /dev/sdb并根据提示进行磁盘写入
[root@centos12l ~]$ fdisk /dev/sdb
Welcome to fdisk (util-linux 2.23.2).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.
Device does not contain a recognized partition table
Building a new DOS disklabel with disk identifier 0xf6e6b69c.
Command (m for help): n
Partition type:
p primary (0 primary, 0 extended, 4 free)
e extended
Select (default p): p
Partition number (1-4, default 1):
First sector (2048-20971519, default 2048):
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-20971519, default 20971519):
Using default value 20971519
Partition 1 of type Linux and of size 10 GiB is set
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.
需要 ext4 模块，lsmod | grep ext4 查看是否已加载，没有的话执行modprobe ext4加载。
[root@centos12l ~]$ lsmod | grep ext4
ext4 579979 0
mbcache 14958 1 ext4
jbd2 107478 1 ext4
执行mkfs.ext4 /dev/sdb1格式化磁盘。
[root@centos12l ~]$ mkfs.ext4 /dev/sdb1
mke2fs 1.42.9 (28-Dec-2013)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
655360 inodes, 2621184 blocks
131059 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=2151677952
80 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632
Allocating group tables: done
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done
将磁盘挂载到data目录
mkdir /data
mount -t ext4 /dev/sdb1 /data
[root@centos12l ~]$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/centos-root 48G 2.7G 46G 6% /
devtmpfs 908M 0 908M 0% /dev
tmpfs 920M 0 920M 0% /dev/shm
tmpfs 920M 9.2M 910M 1% /run
tmpfs 920M 0 920M 0% /sys/fs/cgroup
/dev/sda1 1014M 189M 826M 19% /boot
tmpfs 184M 0 184M 0% /run/user/0
/dev/sdb1 9.8G 37M 9.2G 1% /data
写入fstab开机自动挂载
vim /etc/fstab
/dev/sdb1 /data ext4 defaults 0 0
安装 glusterfs server Link to heading
yum install -y centos-release-gluster
yum install -y glusterfs-server
systemctl start glusterd
systemctl enable glusterd
以下命名从glusterfs节点中选取一台执行即可。
gluster peer probe 172.27.32.182，添加远程节点。
[root@centos11 ~]$ gluster peer probe 172.27.32.182
peer probe: success.
gluster peer status，查看远程节点状态。
root@centos11 ~]$ gluster peer status
Number of Peers: 1
Hostname: 172.27.32.182
Uuid: 3ad2f5fc-2cd6-4d0a-a42d-d3325eb0c687
State: Peer in Cluster (Connected)
gluster pool list，查看节点列表。
[root@centos11 ~]$ gluster pool list
UUID Hostname State
3ad2f5fc-2cd6-4d0a-a42d-d3325eb0c687 172.27.32.182 Connected
1717b41d-c7cd-457e-bfe3-1c825d837488 localhost Connected
安装 glusterfs Link to heading
glusterfs 需要以下内核模块：
- dm_snapshot
- dm_mirror
- dm_thin_pool
执行lsmod | grep <name>查看模块是否存在，如果不存在的话执行modprobe <name>加载模块。
安装 glusterfs：
# install
yum install glusterfs-fuse -y
# version
glusterfs --version
创建 glusterfs volume Link to heading
以下为手动部署 glusterfs 的步骤，使用gk-deploy在kubernetes上使用glusterfs跳过此步，否则会创建失败。
mkdir /data/gvol，在节点上创建 volume 的目录。
以下命名从glusterfs节点中选取一台执行即可。
gluster volume create gvol1 replica 2 172.27.32.182:/data/gvol 172.27.32.164:/data/gvol，创建volume。
# 提示两个节点容易发生脑裂，测试目的可以直接选择继续，生产建议3个节点。
[root@centos11 ~]$ gluster volume create gvol1 replica 2 172.27.32.182:/data/gvol 172.27.32.164:/data/gvol
Replica 2 volumes are prone to split-brain. Use Arbiter or Replica 3 to avoid this. See: http://docs.gluster.org/en/latest/Administrator%20Guide/Split%20brain%20and%20ways%20to%20deal%20with%20it/.
Do you still want to continue?
(y/n) y
volume create: gvol1: success: please start the volume to access data
此 volume 为 Repicate 类型，其他类型可查看官方文档volume-types。
gluster volume start gvol1，启动 volume。
[root@centos11 ~]$ gluster volume start gvol1
volume start: gvol1: success
gluster volume info gvol1，查看 volume 信息。
[root@centos11 ~]$ gluster volume info gvol1
Volume Name: gvol1
Type: Replicate
Volume ID: ed8662a9-a698-4730-8ac7-de579890b720
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: 172.27.32.182:/data/vol1
Brick2: 172.27.32.164:/data/vol1
Options Reconfigured:
transport.address-family: inet
nfs.disable: on
performance.client-io-threads: off
挂载 glusterfs volume，将 glusterfs 的 vloume gvol1 挂载到/data/gfs目录下。
mkdir -p /data/gfs
mount -t glusterfs 172.27.32.164:/gvol1 /data/gfs
# df -h 可以看到已经挂载上
[root@centos11 gfs]$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/centos-root 46G 3.3G 42G 8% /
devtmpfs 1.9G 0 1.9G 0% /dev
tmpfs 1.9G 0 1.9G 0% /dev/shm
tmpfs 1.9G 9.5M 1.9G 1% /run
tmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup
/dev/sda1 1014M 189M 826M 19% /boot
tmpfs 379M 0 379M 0% /run/user/0
/dev/sdb1 9.8G 37M 9.2G 1% /data
172.27.32.182:/gvol1 9.8G 136M 9.2G 2% /data/gfs
在 data/gfs 下写入或更改文件，会自动同步到所有glusterfs节点 gvol1 下的目录中。
将如下配置添加到/etc/fstab，当系统重启后自动 mount 目录。
172.27.32.182:/gvol1 /data/gfs glusterfs defaults,_netdev 0 0
gluster-kubernetes Link to heading
gluster-kubernetes 项目由官方提供的脚本，在kubernetes 上集成 glusterfs。
以下命令未特殊说明的都是在kubernetes master节点上执行。
# 下载项目代码到 /root 文件夹下
git clone https://github.com/gluster/gluster-kubernetes.git
# 进入到 deploy 目录，这也是脚本所在的工作目录
# deploy/kube-templates/ 文件夹下为需要在kubernetes上创建的资源的 yaml 文件。
cd /gluster-kubernetes/deploy
# 修改 topology.json, 描述你的 glusterfs 集群的信息
mv topology.json.sample topology.json
vim topology.json
{
"clusters": [
{
"nodes": [
{
"node": {
"hostnames": {
"manage": [
"172.27.32.164"
],
"storage": [
"172.27.32.164"
]
},
"zone": 1
},
"devices": [
"/dev/sdc"
]
},
{
"node": {
"hostnames": {
"manage": [
"172.27.32.182"
],
"storage": [
"172.27.32.182"
]
},
"zone": 1
},
"devices": [
"/dev/sdc"
]
}
]
}
]
}
gk-deploy需要为初始化过的磁盘，所以这里在两个节点上分别挂载了新的磁盘设备/dev/sdc，并执行以下命令。
# 每台 glusterfs 节点上执行如下命令
# /dev/sdc 需要是新的为初始化的设备
dd if=/dev/urandom of=/dev/sdc bs=512 count=64
使用 base64 生成 heketi 所需要的key，此节点需要指定能够登陆 glusterfs 节点的私钥。
如果不指定 –ssh-keyfile, gk-deploy会默认在kubernetes内创建新的 glusterfs pod，而不是使用本地已有的。
# generate key
echo -n hello | base64
# gk-deploy
./gk-deploy -h
./gk-deploy --admin-key aGVsbG8= --user-key aGVsbG8= --ssh-keyfile /root/.ssh/id_rsa
[root@centos10 deploy]$ ./gk-deploy --admin-key aGVsbG8= --user-key aGVsbG8= --ssh-keyfile /root/.ssh/id_rsa
Welcome to the deployment tool for GlusterFS on Kubernetes and OpenShift.
Before getting started, this script has some requirements of the execution
environment and of the container platform that you should verify.
The client machine that will run this script must have:
* Administrative access to an existing Kubernetes or OpenShift cluster
* Access to a python interpreter 'python'
Each of the nodes that will host GlusterFS must also have appropriate firewall
rules for the required GlusterFS ports:
* 2222 - sshd (if running GlusterFS in a pod)
* 24007 - GlusterFS Management
* 24008 - GlusterFS RDMA
* 49152 to 49251 - Each brick for every volume on the host requires its own
port. For every new brick, one new port will be used starting at 49152. We
recommend a default range of 49152-49251 on each host, though you can adjust
this to fit your needs.
The following kernel modules must be loaded:
* dm_snapshot
* dm_mirror
* dm_thin_pool
For systems with SELinux, the following settings need to be considered:
* virt_sandbox_use_fusefs should be enabled on each node to allow writing to
remote GlusterFS volumes
In addition, for an OpenShift deployment you must:
* Have 'cluster_admin' role on the administrative account doing the deployment
* Add the 'default' and 'router' Service Accounts to the 'privileged' SCC
* Have a router deployed that is configured to allow apps to access services
running in the cluster
Do you wish to proceed with deployment?
[Y]es, [N]o? [Default: Y]: y
Using Kubernetes CLI.
Using namespace "default".
Checking for pre-existing resources...
GlusterFS pods ... not found.
deploy-heketi pod ... not found.
heketi pod ... not found.
gluster-s3 pod ... not found.
Creating initial resources ... serviceaccount "heketi-service-account" created
clusterrolebinding.rbac.authorization.k8s.io "heketi-sa-view" created
clusterrolebinding.rbac.authorization.k8s.io "heketi-sa-view" labeled
OK
secret "heketi-config-secret" created
secret "heketi-config-secret" labeled
service "deploy-heketi" created
deployment.extensions "deploy-heketi" created
Waiting for deploy-heketi pod to start ... OK
Creating cluster ... ID: e5558e7dacc4f24c75f62a68168105fc
Allowing file volumes on cluster.
Allowing block volumes on cluster.
Creating node 172.27.32.164 ... ID: 23abb66f328935c437b6d0274388027f
Adding device /dev/sdc ... OK
Creating node 172.27.32.182 ... ID: 7f99fb669bad6434cdf16258e507dbb7
Adding device /dev/sdc ... OK
heketi topology loaded.
Error: Failed to allocate new volume: No space
command terminated with exit code 255
Failed on setup openshift heketi storage
This may indicate that the storage must be wiped and the GlusterFS nodes must be reset.
直接执行会发生如上报错 No space，这是由于我们的 glusterfs 集群只有两个节点，和 heketi 默认至少需要三个节点，可以在执行gk-deploy时加上--single-ndoe参数跳过此报错。
再次执行之前，需要对环境做下清理，在glusterfs节点上执行以下命令。（这里的pv跟k8s的pv概念没有关系）
# 查看 pv，第二行 /dev/sdc 即为 heketi 创建的 pv，需要删除
[root@centos11 ~]$ pvs
PV VG Fmt Attr PSize PFree
/dev/sda2 centos lvm2 a-- <49.00g 4.00m
/dev/sdc vg_bf7e75e181a24a59edc0d38e33d5ee9c lvm2 a-- 7.87g 7.87g
## 删除pv
[root@centos11 ~]$ pvremove /dev/sdc -ff
WARNING: PV /dev/sdc is used by VG vg_bf7e75e181a24a59edc0d38e33d5ee9c.
Really WIPE LABELS from physical volume "/dev/sdc" of volume group "vg_bf7e75e181a24a59edc0d38e33d5ee9c" [y/n]? y
WARNING: Wiping physical volume label from /dev/sdc of volume group "vg_bf7e75e181a24a59edc0d38e33d5ee9c".
Labels on physical volume "/dev/sdc" successfully wiped.
清理gk-deploy脚本创建的 kubernetes 资源。
kubectl delete sa heketi-service-account
kubectl delete clusterrolebinding heketi-sa-view
kubectl delete secret heketi-config-secret
kubectl delete svc deploy-heketi
kubectl delete deploy deploy-heketi
修改 topology.json
{
"clusters": [
{
"nodes": [
{
"node": {
"hostnames": {
"manage": [
"172.27.32.164"
],
"storage": [
"172.27.32.164"
]
},
"zone": 1
},
"devices": [
"/dev/sdc"
]
},
{
"node": {
"hostnames": {
"manage": [
"172.27.32.182"
],
"storage": [
"172.27.32.182"
]
},
"zone": 1
},
"devices": [
"/dev/sdc"
]
},
{
"node": {
"hostnames": {
"manage": [
"172.27.32.165"
],
"storage": [
"172.27.32.165"
]
},
"zone": 1
},
"devices": [
"/dev/sdc"
]
}
]
}
]
}
再次创建：
# heketi 默认要求至少3个glusterfs 节点，否则会报错 no space，添加一个节点 172.27.32.165
[root@centos10 deploy]$ ./gk-deploy --admin-key aGVsbG8= --user-key aGVsbG8= --ssh-keyfile /root/.ssh/id_rsa
Welcome to the deployment tool for GlusterFS on Kubernetes and OpenShift.
Before getting started, this script has some requirements of the execution
environment and of the container platform that you should verify.
The client machine that will run this script must have:
* Administrative access to an existing Kubernetes or OpenShift cluster
* Access to a python interpreter 'python'
Each of the nodes that will host GlusterFS must also have appropriate firewall
rules for the required GlusterFS ports:
* 2222 - sshd (if running GlusterFS in a pod)
* 24007 - GlusterFS Management
* 24008 - GlusterFS RDMA
* 49152 to 49251 - Each brick for every volume on the host requires its own
port. For every new brick, one new port will be used starting at 49152. We
recommend a default range of 49152-49251 on each host, though you can adjust
this to fit your needs.
The following kernel modules must be loaded:
* dm_snapshot
* dm_mirror
* dm_thin_pool
For systems with SELinux, the following settings need to be considered:
* virt_sandbox_use_fusefs should be enabled on each node to allow writing to
remote GlusterFS volumes
In addition, for an OpenShift deployment you must:
* Have 'cluster_admin' role on the administrative account doing the deployment
* Add the 'default' and 'router' Service Accounts to the 'privileged' SCC
* Have a router deployed that is configured to allow apps to access services
running in the cluster
Do you wish to proceed with deployment?
[Y]es, [N]o? [Default: Y]: y
Using Kubernetes CLI.
Using namespace "default".
Checking for pre-existing resources...
GlusterFS pods ... not found.
deploy-heketi pod ... not found.
heketi pod ... not found.
gluster-s3 pod ... not found.
Creating initial resources ... serviceaccount "heketi-service-account" created
clusterrolebinding.rbac.authorization.k8s.io "heketi-sa-view" created
clusterrolebinding.rbac.authorization.k8s.io "heketi-sa-view" labeled
OK
secret "heketi-config-secret" created
secret "heketi-config-secret" labeled
service "deploy-heketi" created
deployment.extensions "deploy-heketi" created
Waiting for deploy-heketi pod to start ... OK
Creating cluster ... ID: d46fce0516378e5aa913bd1baf97d08b
Allowing file volumes on cluster.
Allowing block volumes on cluster.
Creating node 172.27.32.164 ... ID: 8058c666087f1b411738b802b6cf1d5d
Adding device /dev/sdc ... OK
Creating node 172.27.32.182 ... ID: b08d71449838ed66e2d5aa10f2b8771b
Adding device /dev/sdc ... OK
Creating node 172.27.32.165 ... ID: 9b173570da2fee54f25ed03e74f11c72
Adding device /dev/sdc ... OK
heketi topology loaded.
Saving /tmp/heketi-storage.json
secret "heketi-storage-secret" created
endpoints "heketi-storage-endpoints" created
service "heketi-storage-endpoints" created
job.batch "heketi-storage-copy-job" created
service "heketi-storage-endpoints" labeled
pod "deploy-heketi-bf46f97fb-k42wr" deleted
service "deploy-heketi" deleted
deployment.apps "deploy-heketi" deleted
job.batch "heketi-storage-copy-job" deleted
secret "heketi-storage-secret" deleted
service "heketi" created
deployment.extensions "heketi" created
Waiting for heketi pod to start ...
OK
Flag --show-all has been deprecated, will be removed in an upcoming release
heketi is now running and accessible via http://10.244.106.5:8080 . To run
administrative commands you can install 'heketi-cli' and use it as follows:
# heketi-cli -s http://10.244.106.5:8080 --user admin --secret '<ADMIN_KEY>' cluster list
You can find it at https://github.com/heketi/heketi/releases . Alternatively,
use it from within the heketi pod:
# /usr/bin/kubectl -n default exec -i heketi-77f4797494-8sqng -- heketi-cli -s http://localhost:8080 --user admin --secret '<ADMIN_KEY>' cluster list
For dynamic provisioning, create a StorageClass similar to this:
---
apiVersion: storage.k8s.io/v1beta1
kind: StorageClass
metadata:
name: glusterfs-storage
provisioner: kubernetes.io/glusterfs
parameters:
resturl: "http://10.244.106.5:8080"
restuser: "user"
restuserkey: "aGVsbG8="
Deployment complete!
创建成功！
kubectl delete secret heketi-storage-secret
kubectl delete endpoint heketi-storage-endpoints
kubectl delete svc heketi-storage-endpoints
kubectl delete job heketi-storage-copy-job
kubectl delete svc heketi
kubectl delete deploy heketi
创建 storageclass 测试 Link to heading
kubectl create -f glusterfs-storage.yaml
---
apiVersion: storage.k8s.io/v1beta1
kind: StorageClass
metadata:
name: glusterfs-storage
provisioner: kubernetes.io/glusterfs
parameters:
resturl: "http://10.254.130.133:8080" # heketi service ip
restuser: "admin"
restuserkey: "aGVsbG8="
kubectl create -f pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: gluster-pvc-test
spec:
accessModes:
- ReadWriteOnce
storageClassName: glusterfs-storage
resources:
requests:
storage: 1Gi
创建pvc，等待一段时间后，从Pending变为Bound状态
[root@centos10 gfs]$ kubectl get pvc -w
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
gluster-pvc-test Pending glusterfs-storage 13s
gluster-pvc-test Pending pvc-cacc8019-d9e4-11e9-b223-0800272600e0 0 glusterfs-storage 25s
gluster-pvc-test Bound pvc-cacc8019-d9e4-11e9-b223-0800272600e0 1Gi RWO glusterfs-storage 25s
自动创建了对应pv
[root@centos10 gfs]$ kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-cacc8019-d9e4-11e9-b223-0800272600e0 1Gi RWO Delete Bound default/gluster-pvc-test glusterfs-storage 52s
kubectl create -f nginx.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx
spec:
selector:
matchLabels:
app: nginx
replicas: 1
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: harbor.guahao-inc.com/test4engine/nginx:1.15-alpine
volumeMounts:
- mountPath: "/root/"
name: root
ports:
- containerPort: 80
resources:
limits:
cpu: "1"
memory: 5Mi
requests:
2: 500m
memory: 5Mi
volumes:
- name: root
persistentVolumeClaim:
claimName: gluster-pvc-test
创建Pod绑定到PVC上，pod running。
[root@centos10 gfs]# kubectl get pod -owide
NAME READY STATUS RESTARTS AGE IP NODE
...............
nginx-6dc67b9dc5-84sfg 1/1 Running 0 2m 10.244.145.35 172.27.32.164
# 向 pod 中写入数据
kubectl exec -it nginx-6dc67b9dc5-84sfg sh
echo hello > /root/hello.txt
# 每台 glusterfs node 节点都会生成类似如下目录（每台节点的vg_xxx和brick_xxx名称不同），并可看到目录中有刚才写入pod的文件.
ls /var/lib/heketi/mounts/vg_2c32b0932a02c5b2098de24592b9a2f1/brick_1bf403d9677e9ae11e370f5fcaf8b9bb/brick/
# 删除 deployment，pv 数据仍然存在,重新创建deployment，仍然可以绑定到原有pv。
kubectl delete -f nginx.yaml
# 删除 pvc，对应pv会变成Released状态，并稍后被删除。
kubectl delete -f pvc.yaml
[root@centos10 gfs]$ kubectl get pv -w
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-cacc8019-d9e4-11e9-b223-0800272600e0 1Gi RWO Delete Relegiased default/gluster-pvc-test glusterfs-storage 9h
pvc-cacc8019-d9e4-11e9-b223-0800272600e0 1Gi RWO Delete Failed default/gluster-pvc-test glusterfs-storage 9h