用模板建商城购物网站,抖音营销网站建设价格,服装厂家东莞网站建设,重庆网站设计最佳科技作者#xff1a;张华 发表于#xff1a;2023-03-10 版权声明#xff1a;可以任意转载#xff0c;转载时请务必以超链接形式标明文章原始出处和作者信息及本版权声明
问题
想创建一个local local test bed, 用来方便做各种云实验#xff0c;如openstack, k8s, ovn, lxd等…作者张华 发表于2023-03-10 版权声明可以任意转载转载时请务必以超链接形式标明文章原始出处和作者信息及本版权声明
问题
想创建一个local local test bed, 用来方便做各种云实验如openstack, k8s, ovn, lxd等实验限制条件为
只有一台物理机且为单网卡尽量使用各种cache来应付特色网络
apt cache
首先就是apt cache:
sudo apt install apt-cacher-ng -y
echo PassThroughPattern: .* |sudo tee -a /etc/apt-cacher-ng/acng.conf
sudo systemctl restart apt-cacher-ng.service sudo systemctl enable apt-cacher-ng.service
du -sh /var/cache/apt-cacher-ng/
#vim /var/lib/dpkg/info/apt-cacher-ng.postinst
#dpkg --configure apt-cacher-ng#change the dir from /var/cache/apt-cacher-ng/ to /mnt/udisk/apt-cacher-ng
cat EOF |sudo tee -a /etc/fstab
#use blkid to see uuid
UUIDd63d7251-ec3d-4ef5-aa92-f3d4c480f20c /mnt/udisk ext4 defaults 0 2
EOF
mkfs.ext4 -F -L udisk /dev/sdb1
mkdir /mnt/udisk/apt-cacher-ng
chown -R apt-cacher-ng:apt-cacher-ng /mnt/udisk/apt-cacher-ng
sudo sed -i s/CacheDir: \/var\/cache\/apt-cacher-ng/CacheDir: \/mnt\/udisk\/apt-cacher-ng/g /etc/apt-cacher-ng/acng.conf
du -sh /mnt/udisk/apt-cacher-ng#Use apt cache proxy
echo Acquire::http::Proxy http://proxy:3142; | sudo tee /etc/apt/apt.conf.d/01acngpip mirror
#use pip mirror, or use this instead: PYPI_ALTERNATIVE_URLhttp://mirrors.aliyun.com/pypi/simple
mkdir -p ~/.pip
cat EOF |tee ~/.pip/pip.conf
[global]
trusted-hostmirrors.aliyun.com
index-url http://mirrors.aliyun.com/pypi/simple
disable-pip-version-check true
timeout 120
EOFimage mirror
注下列的sstream-mirror不知为什么在特色网络下始终都是0%, 待查
archive.ubuntu.com, 可用 http://mirrors.clooud.tencent.com/ubuntu 代替ports.ubuntu.com, 可用http://ports.ubuntu.com/ubuntu-ports 代替
maas.io与cloud-images.ubuntu.com自己做mirror, 方法如下
#https://blog.csdn.net/quqi99/article/details/78456909
sudo apt -y install simplestreams -y
#for cloud-images.ubuntu.com
sudo sstream-mirror --keyring/usr/share/keyrings/ubuntu-cloudimage-keyring.gpg --progress --max1 --pathstreams/v1/index.json \https://cloud-images.ubuntu.com/releases/ /images/simplestreams archamd64 release~(jammy) \ftype~(lxd.tar.xz|squashfs|root.tar.xz|root.tar.gz|disk1.img|.json|.sjson)
#for images.maas.io
sudo sstream-mirror --keyring/usr/share/keyrings/ubuntu-cloudimage-keyring.gpg https://images.maas.io/ephemeral-v3/stable \/images/simplestreams archamd64 release~(jammy) --max1 --progress
sudo sstream-mirror --keyring/usr/share/keyrings/ubuntu-cloudimage-keyring.gpg https://images.maas.io/ephemeral-v3/stable \/images/simplestreams os~(grub*|pxelinux) --max1 --progress然后解决密钥:
#https://goharbor.io/docs/2.6.0/install-config/configure-https/
openssl genrsa -out ca.key 4096
openssl req -x509 -new -nodes -sha512 -days 3650 -subj /CCN/STBeijing/LBeijing/Oexample/OUPersonal/CNnode1.lan -key ca.key -out ca.crt
openssl genrsa -out node1.lan.key 4096
openssl req -sha512 -new -subj /CCN/STBeijing/LBeijing/Oexample/OUPersonal/CNnode1.lan -key node1.lan.key -out node1.lan.csr
#complies with the Subject Alternative Name (SAN) and x509 v3 extension requirements to avoid x509: certificate relies on legacy Common Name field, use SANs instead
cat v3.ext -EOF
authorityKeyIdentifierkeyid,issuer
basicConstraintsCA:FALSE
keyUsage digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
extendedKeyUsage serverAuth
subjectAltName alt_names[alt_names]
DNS.1node1.lan
DNS.2node1
DNS.3hostname
EOF
openssl x509 -req -sha512 -days 3650 -extfile v3.ext -CA ca.crt -CAkey ca.key -CAcreateserial -in node1.lan.csr -out node1.lan.crt
#for docker, the Docker daemon interprets .crt files as CA certificates and .cert files as client certificates.
openssl x509 -inform PEM -in node1.lan.crt -out node1.lan.cert设置nginx为https, 另外由于上面使用了一个新目录/images/simplestreams作为root那需要将/etc/nginx/nginx.conf中添加’user root;来避免权限问题
$ cat /etc/nginx/sites-available/default
server {listen 443 ssl http2;listen [::]:443 ssl http2;server_name node1.lan;ssl_certificate /home/hua/ca/node1.lan.crt;ssl_certificate_key /home/hua/ca/node1.lan.key;#ssl_protocols TLSv1.2;ssl_prefer_server_ciphers on; location / {root /images/simplestreams;index index.html;}
}测试:
curl --resolve node1.lan:443:192.168.99.235 --cacert ~/ca/ca.crt https://node1.lan:443/streams/v1/index.json
sudo cp ~/ca/ca.crt /usr/local/share/ca-certificates/ca.crt
sudo chmod 644 /usr/local/share/ca-certificates/ca.crt
sudo update-ca-certificates --fresh
curl --resolve node1.lan:443:192.168.99.235 https://node1.lan:443/streams/v1/index.json物理机网络设计
因为只有一台机器node1, 只有一个网卡eno1:
br-eth0, 虽然lxd也支持ovs bridge, 但为了更方便使用wol更稳定的management网络我们还是决定使用用这个仅有的网卡eno1创建一个linux bridge (br-eth0), 这样br-eth0足够做各种lxd实验了br-data, 若要做openstack实验还需要创建一个ovs bridge (br-data), 只用node1这一台物理机做实验它并不需要物理网卡若今后还想加物理机可以创建一对linux peers来连接br-eth0与br-data这linux peers的一端加入br-data即可。另外即使要创建多机环境用vagrant, lxd很多方案可以解决并没有加多物理机的需求
下列netplan配置创建了br-eth0也让br-eth0支持wol通过魔术包唤醒也创建了一个没有dhcp的br-maas用于maas实验
cat EOF |sudo tee /etc/netplan/90-local.yaml
network:version: 2renderer: networkdethernets:eno1:dhcp4: nomatch:macaddress: f8:32:e4:be:87:cdwakeonlan: truebridges:br-eth0:dhcp4: yesinterfaces:- eno1#Use etherwake F8:32:E4:BE:87:CD to wol in bridgemacaddress: f8:32:e4:be:87:cdbr-maas:#br-maas without dhcp enabled so its for maasdhcp4: falseaddresses:- 192.168.9.1/24routes:- to: defaultvia: 192.168.99.1nameservers:addresses:- 192.168.99.1
EOF
sudo netplan generate
sudo netplan apply使用netplan的配置是想运行一些post script hook时不方便, 未测试下面使用networkd-dispatcher hook的曲线救国方法.
sudo systemctl stop NetworkManager.service
sudo systemctl disable NetworkManager.service
sudo systemctl stop NetworkManager-wait-online.service
sudo systemctl disable NetworkManager-wait-online.service
sudo systemctl stop NetworkManager-dispatcher.service
sudo systemctl disable NetworkManager-dispatcher.service
sudo apt install netplan.io openvswitch-switch -y
sudo apt install -y networkd-dispatcher -y
cat EOF |sudo tee /etc/networkd-dispatcher/off.d/start.sh
#!/bin/bash -e
#IFACEeno1
if [ \$IFACE eno1 -o \$IFACE br-eth0 ]; thenif ip link show eno1 | grep state DOWN /dev/null !(arp -ni br-data | grep ether /dev/null); thendate /tmp/start.txt;/usr/bin/ovs-vsctl --may-exist add-port br-eth0 eno1ip l add name veth-br-eth0 type veth peer name veth-exip l set dev veth-br-eth0 upip l set dev veth-ex upip l set veth-br-eth0 master br-eth0fi
fi
EOF
cat EOF |sudo tee /etc/networkd-dispatcher/routable.d/stop.sh
#!/bin/bash -e
if [ \$IFACE eno1 -o \$IFACE br-eth0 ]; thenif ip link show eno1 | grep state UP /dev/null || arp -ni br-data | grep ether /dev/null; thendate /tmp/stop.txt;systemctl stop hostapd;fi
fi
EOF
sudo chmod x /etc/networkd-dispatcher/off.d/start.sh
sudo chmod x /etc/networkd-dispatcher/off.d/stop.sh直接创建ovs-bridge的方法如下但我们的设计并没有使用ovs-bridge的需求
auto br-eth0
allow-ovs br-eth0
iface br-eth0 inet static
pre-up /usr/bin/ovs-vsctl -- --may-exist add-br br-eth0
pre-up /usr/bin/ovs-vsctl -- --may-exist add-port br-eth0 eno1address 192.168.99.125gateway 192.168.99.1network 192.168.99.0netmask 255.255.255.0broadcast 192.168.99.255
ovs_type OVSBridge
ovs_ports eno1#sudo ip -6 addr add 2001:2:3:4500:fa32:e4ff:febe:87cd/64 dev br-eth0
iface br-phy inet6 static
pre-up modprobe ipv6
address 2001:2:3:4500:fa32:e4ff:febe:87cd
netmask 64
gateway 2001:2:3:4500::1auto eno1
allow-br-phy eno1
iface eno1 inet manual
ovs_bridge br-eth0
ovs_type OVSPort使用Networkmanager来代替netplan的方法容易支持post script hook:
rootnode1:~# cat /etc/network/interfaces
auto lo
iface lo inet loopback
auto eth0
iface eth0 inet manual
auto br-eth0
iface br-eth0 inet staticaddress 192.168.99.124/24gateway 192.168.99.1bridge_ports eth0dns-nameservers 192.168.99.1bridge_stp onbridge_fd 0bridge_maxwait 0up echo -n 0 /sys/devices/virtual/net/$IFACE/bridge/multicast_snooping
# for stateless its inet6 auto, for stateful its inet6 dhcp
iface br-eth0 inet6 auto#iface eth0 inet6 static#address 2001:192:168:99::135 #gateway 2001:192:168:99::1#netmask 64# use SLAAC to get global IPv6 address from the router# we may not enable ipv6 forwarding, otherwise SLAAC gets disabled# sleep 5 is due a bug and dhcp 1 indicates that info should be obtained from dhcpv6 server for statelessup echo 0 /proc/sys/net/ipv6/conf/$IFACE/disable_ipv6up sleep 5autoconf 1accept_ra 2dhcp 1devstack实验
单机实验br-data不用veth-ex也行br-ex用veth-ex也行(这时就可以当br-ex用了
OVS_PHYSICAL_BRIDGEbr-data将用于这里(ovs-vsctl set open . external-ids:ovn-bridge-mappingsphysnet1:br-data)如果它想要访问外网的话可以再设置PUBLIC_INTERFACEveth-ex 其中veth-ex为br-eth0(linux bridge)与br-data(ovs bridge)之间的linux peersPUBLIC_INTERFACE也就是br-ex, 也可以让它使用br-data
配置如下未测试, 仅供参考
cat EOF |tee local.conf
[[local|localrc]]
#make rabbitmq-server to run well
#echo 10.0.1.1 node1 |sudo tee -a /etc/hosts
#sudo pip install --upgrade setuptools
#when USE_VENVTrue and hitting pip issue, eg: install_ipip.sh related issues, can try:
#find /bak/openstack -name *.venv |xargs rm -rf {}
#https://docs.openstack.org/devstack/latest/configuration.html
#TARGET_BRANCHstable/zed
#PYPI_ALTERNATIVE_URLhttp://mirrors.aliyun.com/pypi/simple
sudo ovs-vsctl show
sudo ip l add name veth-br-eth0 type veth peer name veth-ex /dev/null 21
sudo ip l set dev veth-br-eth0 up
sudo ip l set dev veth-ex up
sudo ip l set veth-br-eth0 master br-eth0
sudo ovs-vsctl --may-exist add-br br-data
sudo ovs-vsctl --may-exist add-port br-data veth-ex
sudo ip addr add 10.0.1.1/24 dev br-data /dev/null /dev/null 21
USE_VENVFalse
OFFLINEFalse
DEST/bak/openstack
PUBLIC_INTERFACEveth-ex
OVS_PHYSICAL_BRIDGEbr-data
PUBLIC_BRIDGEbr-data
HOST_IP10.0.1.1
FIXED_RANGE10.0.1.0/24
NETWORK_GATEWAY10.0.1.1
PUBLIC_NETWORK_GATEWAY192.168.99.1
FLOATING_RANGE192.168.99.0/24
Q_FLOATING_ALLOCATION_POOLstart192.168.99.240,end192.168.99.249
disable_service tempest
disable_service horizon
disable_service memory_tracker
ADMIN_PASSWORDpassword
DATABASE_PASSWORD\$ADMIN_PASSWORD
RABBIT_PASSWORD\$ADMIN_PASSWORD
SERVICE_PASSWORD\$ADMIN_PASSWORD
IP_VERSION4
SYSLOGFalse
VERBOSETrue
LOGFILE\$DEST/logs/stack.log
ENABLE_DEBUG_LOG_LEVELFalse
SCREEN_LOGDIR\$DEST/logs
LOG_COLORFalse
LOGDAYS5
Q_USE_DEBUG_COMMANDFalse
WSGI_MODEmod_wsgi
KEYSTONE_USE_MOD_WSGIFalse
NOVA_USE_MOD_WSGIFalse
CINDER_USE_MOD_WSGIFalse
MYSQL_GATHER_PERFORMANCEFalse
DOWNLOAD_DEFAULT_IMAGESFalse
IMAGE_URLShttp://download.cirros-cloud.net/0.6.1/cirros-0.6.1-x86_64-disk.img
heartbeat_timeout_threshold7200
#GIT_BASEhttp://git.trystack.cn
EOF安装lxd
配置lxd默认创建两块网卡:
eth0: br-eth0eth1: lxdbr0 with dhcp
对于maas lxd 容器可能还需要一块没有dhcp的网卡(上面netplan中创建了br-maas), 可这样用它lxc config device add maas eth2 nic nameeth2 nictypebridged parentbr-maas
sudo snap install lxd --classic
sudo usermod -aG $USER lxd
sudo chown -R $USER ~/.config/
export EDITORvim
# MUST NOT use sudo, so must cd to home dir to run it
cd ~ lxd init --auto
#lxc network set lxdbr0 ipv4.address10.10.10.1/24
#lxc network set lxdbr0 ipv6.address none#Change the default storage
lxc profile device remove default root
lxc storage delete default
cat EOF | sudo tee -a /etc/fstab
#mount -o bind /images/lxd /var/snap/lxd/common/lxd/storage-pools
/var/snap/lxd/common/lxd/storage-pools /images/lxd none bind 0 0
EOF
mkdir /images/lxd sudo mount -a
sudo systemctl restart snap.lxd.daemon
lxc storage create default dir lxc storage show default
lxc profile device add default root disk path/ pooldefault
lxd sql global SELECT * FROM storage_pools_config#Use br-data for lxd containers
cat EOF |tee /tmp/default.yaml
config:boot.autostart: truelinux.kernel_modules: openvswitch,nbd,ip_tables,ip6_tablessecurity.nesting: truesecurity.privileged: true
description:
devices:eth0:name: eth0nictype: bridgedparent: br-datatype: niceth1:mtu: 9000name: eth1nictype: bridgedparent: lxdbr0type: nickvm:path: /dev/kvmtype: unix-charmem:path: /dev/memtype: unix-charroot:path: /pool: defaulttype: disktun:path: /dev/net/tuntype: unix-char
name: default
EOF
cat /tmp/default.yaml |lxc profile edit defaultwget https://cloud-images.ubuntu.com/releases/22.04/release/ubuntu-22.04-server-cloudimg-amd64-lxd.tar.xz
wget https://cloud-images.ubuntu.com/releases/22.04/release/ubuntu-22.04-server-cloudimg-amd64.squashfs
lxc image import ./ubuntu-22.04-server-cloudimg-amd64-lxd.tar.xz ./ubuntu-22.04-server-cloudimg-amd64.squashfs --alias jammy
lxc image listlxc launch jammy maas
lxc config show maas --expanded
lxc exec maas bash注上面的配置在安装maas snap版本时会报错security profiles (cannot setup udev for snap “maas”: cannot reload udev rules: exit status 1 继续使用’lxc profile edit default’来加入
#https://discourse.maas.io/t/install-with-lxd/757/2
config:raw.lxc: |-lxc.mount.autosys:rwlxc.cgroup.devices.allow c 10:237 rwmlxc.apparmor.profile unconfinedlxc.cgroup.devices.allow b 7:* rwm若容器里如果上不了网如无法访问api.snapcraft.io是因为lxd容易默认使用了eth1上的dns10.10.10.1下面的配置可让eth0, eth1, eth2都默认使用dns192.168.99.1来避免特色网络对api.snapcraft.io的污染
lxc exec maas bash
cat EOF |sudo tee /etc/netplan/50-cloud-init.yaml
#make 192.168.99.1 as default dns instead of 10.10.10.1
network:version: 2renderer: networkdethernets:eth0:dhcp4: falseaddresses:- 192.168.99.221/24routes:- to: defaultvia: 192.168.99.1nameservers:addresses:- 192.168.99.1eth1:dhcp4: truenameservers:addresses:- 192.168.99.1eth2:dhcp4: falseaddresses:- 192.168.9.3/24nameservers:addresses:- 192.168.9.3
EOF
#In systemd 239 systemd-resolve has been renamed to resolvectl
resolvectl status
cat /run/systemd/netif/leases/*
nslookup api.snapcraft.io安装maas
sudo snap install maas --channel3.3/stable
sudo apt install -y postgresql
sudo -iu postgres psql -d template1 -U postgres
CREATE USER maas WITH ENCRYPTED PASSWORD password;
CREATE DATABASE maasdb;
GRANT all privileges on database maasdb to maas;
\c maasdb
cat EOF | sudo tee -a /etc/postgresql/14/main/pg_hba.conf
host maas maasdb 0/0 md5
EOF
#This maas container has 3 IPs: eth0192.168.99.221 eth110.10.10.238 eth2192.168.9.3
sudo /snap/bin/maas init regionrack --maas-url http://192.168.99.221:5240/MAAS --database-uri postgres://maas:passwordlocalhost/maasdb
sudo /snap/bin/maas createadmin --username admin --password password --email adminexample.com --ssh-import lp:zhhuabj
sudo /snap/bin/maas apikey --username admin |tee ~/admin-api-key
sudo /snap/bin/maas status
#login into http://192.168.99.221:5240/MAAS/r/
#change mirror: http://mirrors.cloud.tencent.com/ubuntu/ for http://archive.ubuntu.com/ubuntu
#change mirror: https://mirrors.tuna.tsinghua.edu.cn/ubuntu-ports/ for http://ports.ubuntu.com/ubuntu-ports
apikey$(sudo maas apikey --username admin)
maas login admin http://127.0.0.1:5240/MAAS $apikey
maas root boot-source update 1 urlhttps://node1.lan:443 keyring_filename/usr/share/keyrings/ubuntu-cloudimage-keyring.gpg
maas admin boot-resources import创建openstack实验环境
在单机上创建openstack实验环境的方法有
juju/charm lxd: 直接部署在lxd容器中好像并发安装时速度较慢, 还是用虚机吧用虚机的话vagrant是一个选择multipath也行(但multipath在运行qemu provider时cpu较高)若创建openstack over openstack环境底层openstack采用devstack或者microstack来搭建然后使用juju openstack provider来通过juju来在租户之下搭建上层openstack环境。这种情况下不需要修改现有bundle中的machine配置这样是为每一个openstack组件都会新启动一个虚机来安装。问题是这一台物理机可能性能无法运行如此多的虚机吧、这台物理机是4核的那创建三台虚机差不多了一台做controller, 两台做compute. 在lxd容器中安装maas, 用maas通过pxe来自动安装这三台虚机。然后使用juju maas provider来通过juju管理上层openstack环境各种控制服务安装在controller虚机的lxd容器里
待续, 目前的问题主要是特色网络造成镜像无法下载使用sstream-mirror做mirror时也下载不下来。
