宁波网站建设论坛,wordpress用户数据备份,贵阳app开发制作,哪里app开发公司好引言
在软件部署的世界中#xff0c;Jenkins已经成为自动化流程的代名词。不断变化的技术环境要求我们持续改进部署流程以满足现代应用部署的需要。在本篇博客中#xff0c;作为一位资深运维工程师#xff0c;我将分享如何将Jenkins Pipeline进化至不仅能支持部署应用直至R…引言
在软件部署的世界中Jenkins已经成为自动化流程的代名词。不断变化的技术环境要求我们持续改进部署流程以满足现代应用部署的需要。在本篇博客中作为一位资深运维工程师我将分享如何将Jenkins Pipeline进化至不仅能支持部署应用直至Running状态检测同时也能兼顾Deployment和StatefulSet资源的轮询更新并详细介绍滚动更新策略的配置方法。
初始Jenkins Pipeline分析
参照前文Jenkins Pipeline 脚本优化实践从繁琐到简洁初始化pipeline如下
pipeline {agent none // Use none at the top level, each stage will define its own agent.environment {REGISTRY swr.cn-north-4.myhuaweicloud.com/master-metaspaceKUBE_CONFIG --namespacemaster-metaspace --contextmasterKUBE_YAML_PATH /home/jenkins/workspace/yaml/master-metaspace// Assume that data is defined elsewhere or injected as a parameter.BASE_WORKSPACE xxxxxxx // 定义一个基础工作空间路径}stages {stage(GetCode) {agent { label build01 }steps {script {checkout scm: [$class: GitSCM,branches: [[name: env.branchName]],extensions: [[$class: CloneOption, depth: 1, noTags: false, shallow: true]],userRemoteConfigs: [[credentialsId: xxxx, url: env.gitHttpURL]]]}}}stage(Docker Builds) {parallel {stage(Build dataloader-game-ucenter) {agent { label build01 }when { environment name: dataloader, value: true }steps {buildAndPushDockerImage(dataloader-game-ucenter, env.data, env.BASE_WORKSPACE)}}stage(Build datawriter-game-ucenter) {agent { label build01 }when { environment name: datawriter, value: true }steps {buildAndPushDockerImage(datawriter-game-ucenter, env.data, env.BASE_WORKSPACE)}}stage(Build game-ucenter) {agent { label build01 }when { environment name: game-ucenter, value: true }steps {buildAndPushDockerImage(game-ucenter, env.data, env.BASE_WORKSPACE)}}}}stage(Development Deployment) {parallel {stage(Deploy datawriter-game-ucenter) {when { environment name: datawriter-game-ucenter, value: true }agent { label huaweiyun-xx }steps {deployToKubernetes(datawriter-game-ucenter)}}stage(Deploy dataloader-game-ucenter) {when { environment name: dataloader, value: true }agent { label huaweiyun-xx }steps {deployToKubernetes(dataloader-game-ucenter)}}stage(Deploy game-ucenter) {when { environment name: game-ucenter, value: true }agent { label huaweiyun-xx }steps {deployToKubernetes(game-ucenter)}}}}}
}// Define methods outside pipeline to avoid repetitiondef buildAndPushDockerImage(String imageName, String tag, String workspacePath) {sh cd ${workspacePath} echo Current directory: \$(pwd) // 使用基础工作空间变量sh cd ${workspacePath}/${imageName} docker build --build-arg NODE_ENV$imageName -t $REGISTRY/$imageName:$tag .withCredentials([usernamePassword(credentialsId: hw-registry, passwordVariable: dockerPassword, usernameVariable: dockerUser)]) {sh docker login -u $dockerUser -p $dockerPassword $REGISTRYsh docker push $REGISTRY/$imageName:$tag}
}def deployToKubernetes(String kubernetesComponent) {String templateFile ${KUBE_YAML_PATH}/${kubernetesComponent}.tplString outputFile ${KUBE_YAML_PATH}/${kubernetesComponent}.yamlsh sed -e s/{data}/$data/g $templateFile $outputFilesh sudo kubectl apply -f $outputFile $KUBE_CONFIG
}初始的Jenkins Pipeline定义了一个基本的CI/CD流程涵盖了代码拉取、Docker镜像构建、推送及在Kubernetes环境中的部署。然而流程中缺少了对部署状态的检查这是在确保部署稳定性方面至关重要的一个环节。
进化 I探针引入Deployment部署
现代应用部署不仅仅需要一个“部署到Kubernetes”的指令更需要在部署后进行健康检查。对于Deployment类型的应用来说我们需要在所有Pods运行并处于READY状态后才认为部署成功。
状态检测方法介绍
为此我们引入了checkKubernetesResourceStatus方法来检查资源的状态。该方法通过kubectl的get命令和jsonpath查询输出来轮询检查ready副本数。如果指定时间内资源不达状态则流程失败。
Jenkinsfile变更详解
引入checkKubernetesResourceStatus方法来检测deployment各个阶段部署的状态。
def checkKubernetesResourceStatus(String deploymentName, String namespace) {int attempts 30 // Set the number of retry attemptsint sleepTime 10 // Set the sleep time between attempts in secondsString readyReplicasJsonPath .status.readyReplicasfor (int i 1; i attempts; i) {// Check the deployment statusString statusCheck sh (script: kubectl get deployment ${deploymentName} --namespace${namespace} -o jsonpath\{${readyReplicasJsonPath}}\,returnStdout: true).trim()// If the number of ready replicas is not empty and greater than 0if (statusCheck statusCheck.isInteger() statusCheck.toInteger() 0) {echo Deployment ${deploymentName} is ready.return} else {echo Waiting for Deployment ${deploymentName} to be ready. Attempt ${i}/${attempts}sleep sleepTime}}error Deployment ${deploymentName} did not become ready after ${attempts} attempts
}以Deploy game-ucenter stage为例 stage(Deploy game-ucenter) {when { environment name: game-ucenter, value: true }agent { label xxxx }steps {deployToKubernetes(game-ucenter)checkKubernetesResourceStatus(game-ucenter, master-metaspace)}}game-ucenter模板文件如下
apiVersion: apps/v1
kind: Deployment
metadata:name: game-ucenter
spec:replicas: 1selector:matchLabels:app: game-ucentertemplate:metadata:labels:app: game-ucenterspec:containers:- name: game-ucenterimage: xxxxxx/xxxx/game-ucenter:{data}envFrom:- configMapRef:name: deployports:- containerPort: 80resources:requests:memory: 4096Mcpu: 2000mlimits:memory: 4096Mcpu: 2000m livenessProbe:httpGet:scheme: HTTPpath: /test.htmlport: 80initialDelaySeconds: 20periodSeconds: 120successThreshold: 1failureThreshold: 3readinessProbe:httpGet:scheme: HTTPpath: /test.htmlport: 80initialDelaySeconds: 20periodSeconds: 120imagePullSecrets: - name: xxxx
---apiVersion: v1
kind: Service
metadata:name: game-ucenterlabels:app: game-ucenter
spec:ports:- port: 80protocol: TCPtargetPort: 80selector:app: game-ucenter 尝试修改以下**replicas: 3 **重新运行以下pipeline 也没有问题pipeline 脚本有效
进化 II兼容StatefulSet的健康检查
考虑到某些应用可能采用StatefulSet作为工作负载类型我们必须确保Jenkins Pipeline能够针对不同的工作负载类型执行健康检查。
状态检测兼容性改进
为了适配StatefulSet我们对checkKubernetesResourceStatus方法做了略微修改使其可以接受一个resourceType参数来区分资源类型进而查询对应的状态字段代码片段如下
def checkKubernetesResourceStatus(String resourceName, String namespace, String resourceType) {int attempts 30 // Set the number of retry attemptsint sleepTime 10 // Set the sleep time between attempts in secondsString readyReplicasJsonPath resourceType deployment ? .status.readyReplicas : .status.readyReplicasfor (int i 1; i attempts; i) {// Check the resource statusString statusCheck sh (script: kubectl get ${resourceType} ${resourceName} --namespace${namespace} -o jsonpath\{${readyReplicasJsonPath}}\,returnStdout: true).trim()// If the number of ready replicas is not empty and equal to the desired numberif (statusCheck statusCheck.isInteger() statusCheck.toInteger() 0) {echo ${resourceType} ${resourceName} is ready.return} else {echo Waiting for ${resourceType} ${resourceName} to be ready. Attempt ${i}/${attempts}sleep(sleepTime)}}error ${resourceType} ${resourceName} did not become ready after ${attempts} attempts
}修改game-ucenter stage: stage(Deploy game-ucenter) {when { environment name: game-ucenter, value: true }agent { label k8s-node-06 }steps {deployToKubernetes(game-ucenter)checkKubernetesResourceStatus(game-ucenter, master-metaspace, deployment)}}创建一个statefulset datawriter-game-ucenter stage: stage(Deploy datawriter-game-ucenter) {when { environment name: datawriter-game-ucenter, value: true }agent { label xxxxx }steps {deployToKubernetes(datawriter-game-ucenter)checkKubernetesResourceStatus(datawriter-game-ucenter, master-metaspace, statefulset)}}注意:我这里截图还是用了game-ucenter做的测试其实我想用我的datawriter-game-ucenterbut这个服务是一个node应用没有没有livenessProbe readinessProbe所以截图我还是使用了game-ucenter!
进化 III引入滚动更新策略配置和检测
当我们更新Deployment资源时通常会采用滚动更新策略以逐步替换旧Pods最小化部署时的中断。
更新策略检测逻辑
def checkDeploymentUpdateStatus(String deploymentName, String namespace) {int attempts 30 // Set the number of retry attemptsint sleepTime 10 // Set the sleep time between attempts in secondsecho Checking the update status of Deployment: ${deploymentName}for (int i 1; i attempts; i) {String updateStatus sh (script: kubectl rollout status deployment/${deploymentName} --namespace${namespace},returnStdout: true).trim()if (updateStatus.contains(successfully rolled out)) {echo Update status: ${updateStatus}return} else {echo Waiting for Deployment ${deploymentName} to successfully roll out. Attempt ${i}/${attempts}sleep(sleepTime)}}error Deployment ${deploymentName} did not successfully roll out after ${attempts} attempts
}我们增加了checkDeploymentUpdateStatus方法该方法通过kubectl命令rollout status监控Deployment的更新状态。当检测到successfully rolled out时表示滚动更新成功。如果在给定时间内更新没有成功则流程将失败。
继续考虑一下如果statefulset多实例呢不想写两个了整合成一个方法如下
def checkRolloutStatus(String resourceName, String namespace, String resourceType) {int attempts 30 // Set the number of retry attemptsint sleepTime 10 // Set the sleep time between attempts in secondsif (!(resourceType in [deployment, statefulset])) {error Unknown resource type: ${resourceType}. Only deployment and statefulset are supported.}echo Checking the update status of ${resourceType} ${resourceName} in namespace ${namespace}for (int i 1; i attempts; i) {String rolloutCommand kubectl rollout status ${resourceType}/${resourceName} --namespace${namespace}String updateStatus sh (script: rolloutCommand,returnStdout: true).trim()if (updateStatus.contains(successfully rolled out) || updateStatus.contains(partitioned roll out complete)) {echo Update status: ${updateStatus}return} else {echo Waiting for ${resourceType} ${resourceName} to successfully roll out. Attempt ${i}/${attempts}.sleep(sleepTime)}}error ${resourceType} ${resourceName} did not successfully roll out after ${attempts} attempts in namespace ${namespace}
}Jenkinsfile更新实现
经过上述进化Jenkinsfile中现在包含了完整的部署状态检查逻辑以应对不同类型资源的部署监控需求。
pipeline {agent none // Use none at the top level, each stage will define its own agent.environment {REGISTRY ccr.ccs.tencentyun.com/xxxxxKUBE_CONFIG --namespacemaster-metaspaceKUBE_YAML_PATH /home/jenkins/workspace/yaml/master-metaspace// Assume that data is defined elsewhere or injected as a parameter.BASE_WORKSPACE xxxxxx // 定义一个基础工作空间路径}stages {stage(GetCode) {agent { label build01 }steps {script {checkout scm: [$class: GitSCM,branches: [[name: env.branchName]],extensions: [[$class: CloneOption, depth: 1, noTags: false, shallow: true]],userRemoteConfigs: [[credentialsId: xxxxx, url: env.gitHttpURL]]]}}}stage(Docker Builds) {parallel {stage(Build dataloader-game-ucenter) {agent { label build01 }when { environment name: dataloader-game-ucenter, value: true }steps {buildAndPushDockerImage(dataloader-game-ucenter, env.data, env.BASE_WORKSPACE)}}stage(Build datawriter-game-ucenter) {agent { label build01 }when { environment name: datawriter-game-ucenter, value: true }steps {buildAndPushDockerImage(datawriter-game-ucenter, env.data, env.BASE_WORKSPACE)}}stage(Build game-ucenter) {agent { label build01 }when { environment name: game-ucenter, value: true }steps {buildAndPushDockerImage(game-ucenter, env.data, env.BASE_WORKSPACE)}}}}stage(Development Deployment) {parallel {stage(Deploy datawriter-game-ucenter) {when { environment name: datawriter-game-ucenter, value: true }agent { label xxxx }steps {deployToKubernetes(datawriter-game-ucenter)checkKubernetesResourceStatus(datawriter-game-ucenter, master-metaspace, statefulset)checkRolloutStatus(datawriter-game-ucenter, master-metaspace, statefulset)}}stage(Deploy dataloader-game-ucenter) {when { environment name: dataloader-game-ucenter, value: true }agent { label xxxx }steps {deployToKubernetes(dataloader-game-ucenter)checkKubernetesResourceStatus(dataloader-game-ucenter, master-metaspace, statefulset)}}stage(Deploy game-ucenter) {when { environment name: game-ucenter, value: true }agent { label xxxx }steps {deployToKubernetes(game-ucenter)checkRolloutStatus(game-ucenter, master-metaspace, deployment)checkKubernetesResourceStatus(game-ucenter, master-metaspace, deployment)}}}}}
}// Define methods outside pipeline to avoid repetitiondef buildAndPushDockerImage(String imageName, String tag, String workspacePath) {sh cd ${workspacePath} echo Current directory: \$(pwd) // 使用基础工作空间变量sh cd ${workspacePath}/${imageName} docker build --build-arg NODE_ENVgame-ucenter -t $REGISTRY/$imageName:$tag .withCredentials([usernamePassword(credentialsId: xxxxx, passwordVariable: dockerPassword, usernameVariable: dockerUser)]) {sh docker login -u $dockerUser -p $dockerPassword $REGISTRYsh docker push $REGISTRY/$imageName:$tag}
}def deployToKubernetes(String kubernetesComponent) {String templateFile ${KUBE_YAML_PATH}/${kubernetesComponent}.tplString outputFile ${KUBE_YAML_PATH}/${kubernetesComponent}.yamlsh sed -e s/{data}/$data/g $templateFile $outputFilesh sudo kubectl apply -f $outputFile $KUBE_CONFIG
}def checkRolloutStatus(String resourceName, String namespace, String resourceType) {int attempts 30 // 设置重试次数int sleepTime 10 // 设置重试间隔时间秒if (!(resourceType in [deployment, statefulset])) {error 未知资源类型${resourceType}。只支持 deployment 和 statefulset 。}echo 正在检查${resourceType} ${resourceName} 在命名空间 ${namespace} 的更新状态for (int i 1; i attempts; i) {String rolloutCommand kubectl rollout status ${resourceType}/${resourceName} --namespace${namespace}try {String updateStatus sh (script: rolloutCommand,returnStdout: true).trim()// 添加对 partitioned roll out complete 状态的检查if (updateStatus.contains(successfully rolled out) || updateStatus.contains(partitioned roll out complete)) {echo 更新状态${updateStatus}return} else {echo 等待 ${resourceType} ${resourceName} 成功发布。尝试次数${i}/${attempts}。sleep(sleepTime)}} catch (Exception e) {echo 获取更新状态时发生错误${e.getMessage()}。尝试次数${i}/${attempts}。sleep(sleepTime)}}error ${resourceType} ${resourceName} 在命名空间 ${namespace} 内未能在 ${attempts} 次尝试之后成功发布
}
def checkKubernetesResourceStatus(String resourceName, String namespace, String resourceType) {int attempts 30 // Set the number of retry attemptsint sleepTime 10 // Set the sleep time between attempts in secondsString readyReplicasJsonPath resourceType deployment ? .status.readyReplicas : .status.readyReplicasfor (int i 1; i attempts; i) {// Check the resource statusString statusCheck sh (script: kubectl get ${resourceType} ${resourceName} --namespace${namespace} -o jsonpath\{${readyReplicasJsonPath}}\,returnStdout: true).trim()// If the number of ready replicas is not empty and equal to the desired numberif (statusCheck statusCheck.isInteger() statusCheck.toInteger() 0) {echo ${resourceType} ${resourceName} is ready.return} else {echo Waiting for ${resourceType} ${resourceName} to be ready. Attempt ${i}/${attempts}sleep(sleepTime)}}error ${resourceType} ${resourceName} did not become ready after ${attempts} attempts
}// 更新后的Jenkins Pipeline代码详细定义参照本文开头给出的代码总结
本篇博客通过对Jenkins Pipeline的进化过程展开讲解展现了如何从简单的部署任务转变为一个健壮且兼顾各类工作负载状态监测的CI/CD流程。我们强化了状态检测的逻辑引入了更新策略的检测并保持了对不同Kubernetes资源类型的兼容性。这些改进确保了自动化流程能够与现代部署实践保持同步给运维团队带来极大便利并最大化地保障了部署的可靠性。
后记
由于篇幅限制本篇文章未作其他更详细演示。然而在实际应用中运维团队可以根据自己的具体需求和环境进一步丰富和细化每个步骤的实现确保Pipeline的健壮性和高可用性以适应不断变化的技术挑战。
