做导购网站 商品,怎么免费建立自己的网站,白云区专业网站建设,信阳网站设计harbor#xff08;docker仓库#xff09;仓库部署 - 高可用 1. harbor高可用1.1 方案说明1. 双主复制2. 多harbor实例共享后端存储 1.2 部署高可用#xff08;多harbor实例共享后端存储#xff09;1. 服务器划分2. 安装harbor#xff08;先部署一套Harbor#xff0c;用于… harbordocker仓库仓库部署 - 高可用 1. harbor高可用1.1 方案说明1. 双主复制2. 多harbor实例共享后端存储 1.2 部署高可用多harbor实例共享后端存储1. 服务器划分2. 安装harbor先部署一套Harbor用于将其所有表结构导出3. 安装Postgresql4. 安装nginx5. 安装nfs6. 安装redis7. 部署harbor8. 修改nginx配置9. docker登录harbor10. harbor修改11. 修改nginx12. docker推送 1. harbor高可用
目前有两种主流的方案来解决这个问题
双主复制多harbor实例共享后端存储
1.1 方案说明
1. 双主复制
所谓的双主复制其实就是复用主从同步实现两个harbor节点之间的双向同步来保证数据的一致性然后在两台harbor前端顶一个负载均衡器将进来的请求分流到不同的实例中去只要有一个实例中有了新的镜像就是自动的同步复制到另外的的实例中去这样实现了负载均衡也避免了单点故障在一定程度上实现了Harbor的高可用性 这个方案有一个问题就是有可能两个Harbor实例中的数据不一致。假设如果一个实例A挂掉了这个时候有新的镜像进来那么新的镜像就会在另外一个实例B中后面即使恢复了挂掉的A实例Harbor实例B也不会自动去同步镜像这样只能手动的先关掉Harbor实例B的复制策略然后再开启复制策略才能让实例B数据同步让两个实例的数据一致。 在实际生产使用中主从复制十分的不靠谱所以这里就不配置了。
2. 多harbor实例共享后端存储
利用共享存储和共享数据库来实现服务的高可用性和数据的冗余 这个方案在实际生产环境中部署需要考虑三个问题
共享存储的选取Harbor的后端存储目前支持AWS S3、Openstack Swift, Ceph等在我们的实验环境里就直接使用nfs。Session在不同的实例上共享这个现在其实已经不是问题了在最新的harbor中默认session会存放在redis中我们只需要将redis独立出来即可。可以通过redis sentinel或者redis cluster等方式来保证redis的可用性。在我们的实验环境里仍然使用单台redis。Harbor多实例数据库问题这个也只需要将harbor中的数据库拆出来独立部署即可。让多实例共用一个外部数据库数据库的高可用也可以通过数据库的高可用方案保证。可选择的数据库包括PostgreSqlmysql等等。 1.2 部署高可用多harbor实例共享后端存储
1. 服务器划分
服务器IP说明k8s-harbor-01.xx.net192.168.17.220harbor1服务器k8s-harbor-02.xx.net192.168.17.221harbor2服务器k8s-harbor-lb-01.xx.net192.168.17.225nginx,redis,mysql,nfs
我们将在k8s-harbor-lb-01.xx.net部署nginxredismysqlnfs等服务生产环境中应该分开并且配置成为高可用
2. 安装harbor先部署一套Harbor用于将其所有表结构导出
下载安装包并上传到服务器
tar xvf harbor-offline-installer-v2.7.2.tgz
cd harbormkdir certs #证书
cd certs/
openssl genrsa -out ./harbor-ca.key #key
openssl req -x509 -new -nodes -key ./harbor-ca.key -subj /CNharbor.xx.net -days 7120 -out ./harbor-ca.crt #认证配置harbor.yml
cp harbor.yml.tmpl harbor.yml
[rootk8s-harbor-01 harbor]# egrep -v ^$|^#|^ # harbor.yml
hostname: harbor.xx.net
http:port: 80
https:port: 443certificate: /opt/harbor/certs/harbor-ca.crtprivate_key: /opt/harbor/certs/harbor-ca.key
harbor_admin_password: 123456
...启动harbor
./install.sh --with-trivy --with-chartmuseum
--with-trivy #镜像漏洞检测
--with-chartmuseum #Chart仓库服务本节搭建的目的是导出postgresql数据库到其他服务器接着导出数据库
docker ps
docker exec -it harbor-db /bin/bash进入容器
## 执行 psql 进入数据库
postgres [ / ]$ psql
psql (9.6.14)
Type help for help.## 查看当前所有的数据库postgres、template0、template1为默认数据库
postgres# \lList of databasesName | Owner | Encoding | Collate | Ctype | Access privileges
-----------------------------------------------------------------------------------notaryserver | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | Tc/postgres | | | | | postgresCTc/postgres| | | | | serverCTc/postgresnotarysigner | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | Tc/postgres | | | | | postgresCTc/postgres| | | | | signerCTc/postgrespostgres | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | registry | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | template0 | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | c/postgres | | | | | postgresCTc/postgrestemplate1 | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | c/postgres | | | | | postgresCTc/postgres
(6 rows)postgres# \q
## 导出表结构及数据
postgres [ / ]$ pg_dump -U postgres registry /tmp/registry.sql
postgres [ / ]$ pg_dump -U postgres notaryserver /tmp/notaryserver.sql
postgres [ / ]$ pg_dump -U postgres notarysigner /tmp/notarysigner.sql-U 数据库用户-p 访问端口-f 指定文件和 功能一样-h 指定数据库地址-s 表示只导出表结构不导数据导出到宿主机
docker cp 8d69069a2cd7:/tmp/registry.sql ./
docker cp 8d69069a2cd7:/tmp/notaryserver.sql ./
docker cp 8d69069a2cd7:/tmp/notarysigner.sql ./3. 安装Postgresql
yum install -y https://download.postgresql.org/pub/repos/yum/reporpms/EL-7-x86_64/pgdg-redhat-repo-latest.noarch.rpm
yum install -y postgresql13-server/usr/pgsql-13/bin/postgresql-13-setup initdb #初始化数据库# 修改远程访问配置
vim /var/lib/pgsql/13/data/postgresql.conf
...
#将 listen_addresses localhost 修改为
listen_addresses *# 添加信任的远程连接,生产中不要添加0.0.0.0
vim /var/lib/pgsql/13/data/pg_hba.conf
...
host all all 0.0.0.0/0 trust
# host all all 0.0.0.0/0 md5
# 最后一列如果是trust则登录pg不需要密码若为md5则需要密码# start and enable server
systemctl enable postgresql-13
systemctl start postgresql-13# 检查服务是否启动成功
#ps看进程 或 ss看端口号#给postgresql设置密码增强安全性
su - postgres
-bash-4.2$ psql
## 直接写入新密码
postgres# \password
Enter new password for user postgres:
Enter it again:
postgres# \q#验证
psql -U postgres -h localhost
Password for user postgres:
psql (13.14)
Type help for help.postgres# exit#把全部信任改为指定IP
[rootk8s-harbor-lb-01 ~]# tail -3 /var/lib/pgsql/13/data/pg_hba.conf
host all all 192.168.17.220/24 trust
host all all 192.168.17.221/24 trust
#host all all 0.0.0.0/0 trust将备份的数据导入进单独部署的postgresql中
## 创建数据库
postgres# CREATE DATABASE registry;
postgres# CREATE DATABASE notaryserver;
postgres# CREATE DATABASE notarysigner;将harbor服务器的导出的SQL拷贝到本机
scp -r 192.168.17.220:/opt/harbor/postgresql_export /opt/postgresql_export
notaryserver.sql 100% 491 213.2KB/s 00:00
notarysigner.sql 100% 491 281.3KB/s 00:00
registry.sql 100% 101KB 19.7MB/s 00:00导入数据
##
psql -h localhost -U postgres -p 5432 -d registry -f registry.sql
psql -h localhost -U postgres -p 5432 -d notaryserver -f notaryserver.sql
psql -h localhost -U postgres -p 5432 -d notarysigner -f notarysigner.sql -U 数据库用户-p 访问端口-f 指定文件和 功能一样-h 指定数据库地址-d 指定数据库名4. 安装nginx
rpm -Uvh http://nginx.org/packages/centos/7/noarch/RPMS/nginx-release-centos-7-0.el7.ngx.noarch.rpm
yum install -y nginx配置nginx.conf
[rootk8s-harbor-lb-01 ~]# cat /etc/nginx/nginx.conf
...include /etc/nginx/conf.d/*.conf;upstream harborsvr {server 192.168.17.220:80 weight2;server 192.168.17.221:80 weight1;}server {listen 80;server_name 192.168.17.225;location / {proxy_pass http://harborsvr;}}
}启动
systemctl start nginx
systemctl status nginx
systemctl enable nginx5. 安装nfs
yum install -y nfs-utils# 编辑/etc/exports文件
/data *(rw,no_root_squash)chmod 777 -R /datasystemctl start nfs-server
systemctl enable nfs-servermdkir /data/
mount -t nfs hostname:/data/ /data/6. 安装redis
yum install epel-release -y
yum install redis -y##
vim /etc/redis.conf
...
bind 0.0.0.0 # 设置所有主机可以连接
requirepass 123456 # 设置客户端连接密码
daemonize yes # 打开守护进程模式
...## 启动redis
systemctl start redis
systemctl enable redis7. 部署harbor
我们将第2段中部署的harbor进行修改配置文件
[rootk8s-harbor-01 harbor]# docker-compose down编辑配置文件需要更改的主要有以下几点 1.hostname 改为主机ip或完全限定域名不要使用127.0.0.1或localhost 2.https选项如需要指定crt和key的路径若不需要直接注释掉 3.harbor_admin_password默认密码可以更改 4.data_volume数据默认存储位置设计为共享路径 5.注释掉database模块 及 Clair模块 6.开启external_database 和 external_redis模块及正确配置其中参数 7.集群内所有harbor配置均一样改一下hostname值即可 修改配置文件经过后面挂掉之后重新改的配置文件
[rootk8s-harbor-01 harbor]# egrep -v ^$|^#|^ # harbor.yml
hostname: harbor.xx.net
http:port: 80
https:port: 443certificate: /opt/harbor/certs/harbor-ca.crtprivate_key: /opt/harbor/certs/harbor-ca.key
harbor_admin_password: 123456
data_volume: /data
trivy:ignore_unfixed: falseskip_update: falseoffline_scan: falsesecurity_check: vulninsecure: false
jobservice:max_job_workers: 10
notification:webhook_job_max_retry: 10
chart:absolute_url: disabled
log:level: infolocal:# Log files are rotated log_rotate_count times before being removed. If count is 0, old versions are removed rather than rotated.rotate_count: 50# Log files are rotated only if they grow bigger than log_rotate_size bytes. If size is followed by k, the size is assumed to be in kilobytes.# If the M is used, the size is in megabytes, and if G is used, the size is in gigabytes. So size 100, size 100k, size 100M and size 100G# are all valid.rotate_size: 200M# The directory on your host that store loglocation: /var/log/harbor
_version: 2.7.0
external_database:harbor:host: 192.168.17.225port: 5432db_name: registryusername: postgrespassword: 123456ssl_mode: disablemax_idle_conns: 2max_open_conns: 0notary_signer:host: 192.168.17.225port: 5432db_name: notarysignerusername: postgrespassword: 123456ssl_mode: disablenotary_server:host: 192.168.17.225port: 5432db_name: notaryserverusername: postgrespassword: 123456ssl_mode: disable
external_redis:host: 192.168.17.225:6379password: 123456registry_db_index: 1jobservice_db_index: 2chartmuseum_db_index: 3chair_db_index: 4trivy_db_index: 5idle_timeout_seconds: 30
proxy:http_proxy:https_proxy:no_proxy:components:- core- jobservice- trivy
metric:enabled: falseport: 9090path: /metrics
upload_purging:enabled: trueage: 168hinterval: 24hdryrun: false
cache:enabled: falseexpire_hours: 24
启动harbor的过程中发现jobservice容器无法启动怀疑是数据库连接失败
[rootk8s-harbor-01 harbor]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
767ac8431315 goharbor/harbor-jobservice:v2.7.2 /harbor/entrypoint.… 18 hours ago Restarting (2) 17 seconds ago harbor-jobservice
...日志
[rootk8s-harbor-01 harbor]# docker logs 767
Appending internal tls trust CA to ca-bundle ...
find: /etc/harbor/ssl: No such file or directory
Internal tls trust CA appending is Done.
2024-02-21T17:00:15Z [ERROR] [/pkg/registry/client.go:82]: Failed to parse REGISTRY_HTTP_CLIENT_TIMEOUT: strconv.ParseInt: parsing : invalid syntax, use default value: 30m0s
2024-02-21T17:00:15Z [INFO] [/controller/artifact/annotation/parser.go:71]: the annotation parser to parser artifact annotation version v1alpha1 registered
2024-02-21T17:00:15Z [INFO] [/controller/artifact/processor/processor.go:59]: the processor to process media type application/vnd.wasm.config.v1json registered
2024-02-21T17:00:15Z [ERROR] [/lib/config/config.go:81]: failed to get config manager
2024-02-21T17:00:15Z [ERROR] [/lib/config/config.go:81]: failed to get config manager在harbor服务器安装postgresql客户端
yum install https://download.postgresql.org/pub/repos/yum/reporpms/EL-7-x86_64/pgdg-redhat-repo-latest.noarch.rpm -y
yum install postgresql13 -y[rootk8s-harbor-01 harbor]# psql -U postgres -h 192.168.17.225 -p 5432
psql (13.14)
Type help for help.postgres# \lList of databasesName | Owner | Encoding | Collate | Ctype | Access privileges
-----------------------------------------------------------------------------------notaryserver | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 |notarysigner | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 |postgres | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 |registry | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 |template0 | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | c/postgres | | | | | postgresCTc/postgrestemplate1 | postgres | UTF8 | en_US.UTF-8 | en_US.UTF-8 | c/postgres | | | | | postgresCTc/postgres
(6 rows)postgres#发现可以连接注释掉缓存数据库redis配置发现可以正常启动说明是redis配置有问题
[rootk8s-harbor-01 harbor]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
dccc714a7abe goharbor/harbor-jobservice:v2.7.2 /harbor/entrypoint.… About a minute ago Up About a minute (healthy) harbor-jobservice经过调查原来是redis的password修改过和默认的不匹配 harbor1服务器配置完成接下来配置harbor2服务器相同配置改下域名就可以了
[rootk8s-harbor-02 harbor]# grep hostname harbor.yml
# The IP address or hostname to access admin UI and registry service.
hostname: harbor2.xx.net8. 修改nginx配置
由于我们安装的harbor通过http跳转到https访问所以前面设置的nginx的负载均衡的配置需要进行修改否则无法访问
[rootk8s-harbor-lb-01 nginx]# cat /etc/nginx/nginx.conf
...
http {include /etc/nginx/mime.types;default_type application/octet-stream;log_format main $remote_addr - $remote_user [$time_local] $request $status $body_bytes_sent $http_referer $http_user_agent $http_x_forwarded_for;access_log /var/log/nginx/access.log main;sendfile on;#tcp_nopush on;keepalive_timeout 65;#gzip on;include /etc/nginx/conf.d/*.conf;upstream harborsvrs {server 192.168.17.220:443 weight2;server 192.168.17.221:443 weight1;}server {listen 443;server_name 192.168.17.225;location / {proxy_pass https://harborsvrs/;proxy_set_header Host $host;proxy_set_header X-Real-IP $remote_addr;proxy_set_header X-Forwarded-Proto $scheme;}}
}修改后重启nginx
9. docker登录harbor
[rootk8s-master-01 ansible]# docker login 192.168.17.225:443
Username: admin
Password:
Error response from daemon: Get http://192.168.17.225:443/v2/: Get https://harbor1.xx.net/service/token?accountadminclient_iddockeroffline_tokentrueserviceharbor-registry: dial tcp: lookup harbor1.xx.net on 8.8.8.8:53: no such host解析不了harbor的域名在没有DNS服务器的情况下修改/etc/hosts文件
[rootk8s-master-01 ansible]# cat /etc/hosts
...
192.168.17.220 k8s-harbor-01.xx.net harbor1 harbor1.xx.net
192.168.17.221 k8s-harbor-02.xx.net harbor2 harbor2.xx.net登录
[rootk8s-master-01 ansible]# docker login 192.168.17.225:443
Username: admin
Password:
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-storeLogin Succeeded由于推送的时候出现问题笔者一直未能解决所以将harbor回退到使用http协议
[rootk8s-master-01 ansible]# docker push harbor2.xx.net/alpine/alpine:latest
The push refers to repository [harbor2.xx.net/alpine/alpine]
d4fc045c9e3a: Layer already exists
unauthorized: unauthorized to access repository: alpine/alpine, action: push: unauthorized to access repository: alpine/alpine, action: push10. harbor修改
[rootk8s-harbor-01 harbor]# head -20 harbor.yml
# Configuration file of Harbor# The IP address or hostname to access admin UI and registry service.
# DO NOT use localhost or 127.0.0.1, because Harbor needs to be accessed by external clients.
#hostname: harbor1.xx.net
hostname: 192.168.17.220# http related config
http:# port for http, default is 80. If https enabled, this port will redirect to https portport: 80# https related config
#https:# https port for harbor, default is 443
# port: 443# The path of cert and key files for nginx
# certificate: /opt/harbor/certs/harbor-ca.crt
# private_key: /opt/harbor/certs/harbor-ca.key
修改harbor1和harbor2服务器的配置文件并重启harbor
11. 修改nginx
http {include /etc/nginx/mime.types;default_type application/octet-stream;log_format main $remote_addr - $remote_user [$time_local] $request $status $body_bytes_sent $http_referer $http_user_agent $http_x_forwarded_for;access_log /var/log/nginx/access.log main;sendfile on;#tcp_nopush on;keepalive_timeout 65;#gzip on;include /etc/nginx/conf.d/*.conf;upstream harborsvrs {server 192.168.17.220:80 weight2;server 192.168.17.221:80 weight1;}server {listen 80;server_name 192.168.17.225;autoindex on;location / {proxy_pass http://harborsvrs/;}}
}通过负载均衡可以访问harbor
12. docker推送
[rootk8s-master-01 ansible]# docker login 192.168.17.221:80 -u admin
Password:
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-storeLogin Succeeded
[rootk8s-master-01 ansible]# docker tag alpine:latest 192.168.17.221:80/alpine/alpine:v1
[rootk8s-master-01 ansible]# docker push 192.168.17.221:80/alpine/alpine:v1
The push refers to repository [192.168.17.221:80/alpine/alpine]
d4fc045c9e3a: Layer already exists
v1: digest: sha256:6457d53fb065d6f250e1504b9bc42d5b6c65941d57532c072d929dd0628977d0 size: 528
[rootk8s-master-01 ansible]# docker login 192.168.17.220:80 -u admin
Password:
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-storeLogin Succeeded
[rootk8s-master-01 ansible]# docker push 192.168.17.220:80/alpine/alpine:latest
The push refers to repository [192.168.17.220:80/alpine/alpine]
d4fc045c9e3a: Layer already exists
latest: digest: sha256:6457d53fb065d6f250e1504b9bc42d5b6c65941d57532c072d929dd0628977d0 size: 528在其他服务器登录harbor后也可以正常推送
