如何构建一个一劳永逸的高可用Harbor容器镜像仓库(3)

/ cncf / 没有评论 / 76浏览

juiceFS简介

https://juicefs.com/docs/zh/intro.html

需要强调的是,juiceFS只是一个中间服务,实际存储还是在我们云账号的对象存储里面

juiceFS每个区域都有单独的服务,可以做多副本在不同的区域。 提供Prometheus的监控接口,以token的形式

juiceFS磁盘压力测试

顺序读写测试

10G数据 单位100k

# dd if=/dev/zero of=./file1 bs=100k count=100000
100000+0 records in
100000+0 records out
10240000000 bytes (10 GB, 9.5 GiB) copied, 13.7332 s, 746 MB/s

1G数据,单位10k

# dd if=/dev/zero of=./file1 bs=10k count=100000
100000+0 records in
100000+0 records out
1024000000 bytes (1.0 GB, 977 MiB) copied, 5.10316 s, 201 MB/s

随机读写测试

1G数据,单位10M

# dd if=/dev/urandom  of=./file2 bs=10M count=100
100+0 records in
100+0 records out
1048576000 bytes (1.0 GB, 1000 MiB) copied, 81.9476 s, 12.8 MB/s

1G数据,单位100k

# dd if=/dev/urandom  of=./file2 bs=100k count=10000
10000+0 records in
10000+0 records out
1024000000 bytes (1.0 GB, 977 MiB) copied, 79.9982 s, 12.8 MB/s

注意

这是在主机默认磁盘测试的随机读写,以此来和juiceFS对比读写

# dd if=/dev/urandom  of=./file2 bs=10M count=100
100+0 records in
100+0 records out
1048576000 bytes (1.0 GB, 1000 MiB) copied, 80.8556 s, 13.0 MB/s

k8s集群模拟高并发的去pull 镜像

从左到右的操作分别是:Harbor实例1 | 在k8s所有节点模拟高并发的pull操作 | Harbor实例2

该镜像大小为500M

image-20190925121842454

监控图

重点关注监控图毛刺部分

结论

juiceFS的读写性能和主机磁盘相差无几,但是性价比和灵活性却高的多。

不需要在担心扩容问题,以此来应对harbor镜像存储目录快速膨胀的问题。

综合下来,juiceFS的稳定性,易扩展性,性价比,都是最合适的。

附:

新集群的登陆问题在部分k8s集群节点登陆验证

$  ansible all -m shell -a "echo password | docker login --username username --password-stdin  harbor.**.com" -i harbor_login_20190917_test.host
172.23.11.69 | CHANGED | rc=0 >>
Login SucceededWARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

172.23.12.111 | CHANGED | rc=0 >>
Login SucceededWARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

172.23.12.42 | CHANGED | rc=0 >>
Login SucceededWARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

172.23.75.73 | CHANGED | rc=0 >>
Login SucceededWARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

172.23.98.34 | CHANGED | rc=0 >>
Login SucceededWARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

172.23.10.186 | CHANGED | rc=0 >>
Login SucceededWARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

172.23.12.109 | CHANGED | rc=0 >>
Login SucceededWARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

172.23.11.227 | CHANGED | rc=0 >>
Login SucceededWARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store