佛山建设网站,专业网站建设网站研发深圳公司,网站建设要些什么,怎么做网页别人可以看到图片DolphinScheduler概述 Apache DolphinScheduler是一个分布式、易扩展的可视化DAG工作流任务调度平台。致力于解决数据处理流程中错综复杂的依赖关系#xff0c;使调度系统在数据处理流程中开箱即用
DolphinScheduler核心架构
DolphinScheduler的主要角色如下#xff1a; Ma…DolphinScheduler概述 Apache DolphinScheduler是一个分布式、易扩展的可视化DAG工作流任务调度平台。致力于解决数据处理流程中错综复杂的依赖关系使调度系统在数据处理流程中开箱即用
DolphinScheduler核心架构
DolphinScheduler的主要角色如下 MasterServer采用分布式无中心设计理念MasterServer主要负责 DAG 任务切分、任务提交、任务监控并同时监听其它MasterServer和WorkerServer的健康状态。 WorkerServer也采用分布式无中心设计理念WorkerServer主要负责任务的执行和提供日志服务。 ZooKeeper服务系统中的MasterServer和WorkerServer节点都通过ZooKeeper来进行集群管理和容错。 Alert服务提供告警相关服务。 API接口层主要负责处理前端UI层的请求。 UI系统的前端页面提供系统的各种可视化操作界面。 DolphinScheduler部署说明 操作系统版本要求 操作系统 版本 Red Hat Enterprise Linux 7.0 及以上 CentOS 7.0 及以上 Oracle Enterprise Linux 7.0 及以上 Ubuntu LTS 16.04 及以上 服务器硬件要求 CPU 内存 硬盘类型 网络 实例数量 4核 8 GB SAS 千兆网卡 1 部署模式
DolphinScheduler支持多种部署模式包括单机模式Standalone、伪集群模式Pseudo-Cluster、集群模式Cluster等 单机模式 单机模式standalone模式下所有服务均集中于一个StandaloneServer进程中并且其中内置了注册中心Zookeeper和数据库H2。只需配置JDK环境就可一键启动DolphinScheduler快速体验其功能。 伪集群模式 伪集群模式Pseudo-Cluster是在单台机器部署 DolphinScheduler 各项服务该模式下master、worker、api server、logger server等服务都只在同一台机器上。Zookeeper和数据库需单独安装并进行相应配置。 集群模式
集群模式Cluster与伪集群模式的区别就是在多台机器部署 DolphinScheduler各项服务并且Master、Worker等服务可配置多个
DolphinScheduler集群模式部署 集群规划 集群模式下可配置多个Master及多个Worker。通常可配置2~3个Master若干个Worker。由于集群资源有限此处配置一个Master三个Worker集群规划如下。 hadoop102 master、worker hadoop103 worker hadoop104 worker
前置准备工作
三台节点均需部署JDK1.8并配置相关环境变量。
2需部署数据库支持MySQL5.7或者PostgreSQL8.2.15。
3需部署Zookeeper3.4.6。
4如果启用 HDFS 文件系统则需要 Hadoop2.6环境。
5三台节点均需安装进程树分析工具psmisc。
注意三台节点都需要安装 sudo yum install -y psmisc
解压DolphinScheduler安装包
1上传DolphinScheduler安装包到hadoop102节点的/opt/software目录
2解压安装包到当前目录
注解压目录并非最终的安装目录
tar -zxvf apache-dolphinscheduler-2.0.5-bin
创建元数据库及用户 DolphinScheduler 元数据存储在关系型数据库中故需创建相应的数据库和用户
创建数据库 CREATE DATABASE dolphinscheduler DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
创建用户
CREATE USER dolphinscheduler% IDENTIFIED BY dolphinscheduler;
注
若出现以下错误信息表明新建用户的密码过于简单。
ERROR 1819 (HY000): Your password does not satisfy the current policy requirements
可提高密码复杂度或者执行以下命令降低MySQL密码强度级别。 mysqlset global validate_password_policy0; mysql set global validate_password_length4;
赋予用户相应权限 GRANT ALL PRIVILEGES ON dolphinscheduler.* TO dolphinscheduler%;
刷新 flush privileges;
配置一键部署脚本
修改解压目录下的conf/config目录下的install_config.conf文件。
vim conf/config/install_config.conf ipshadoop102,hadoop103,hadoop104
# 将要部署任一 DolphinScheduler 服务的服务器主机名或 ip 列表
# Port of SSH protocol, default value is 22. For now we only support same port in all ips machine
sshPort22
# A comma separated list of machine hostname or IP would be installed Master server, it
# must be a subset of configuration ips.
# Example for hostnames: mastersds1,ds2, Example for IPs: masters192.168.8.1,192.168.8.2
mastershadoop102
# master 所在主机名列表必须是 ips 的子集
# A comma separated list of machine hostname:workerGroup or IP:workerGroup.All hostname or IP must be a
# subset of configuration ips, And workerGroup have default value as default, but we recommend you declare behind the hosts
# Example for hostnames: workersds1:default,ds2:default,ds3:default, Example for IPs: workers192.168.8.1:default,192.168.8.2:default,192.168.8.3:default
workershadoop102:default,hadoop103:default,hadoop104:default
# worker主机名及队列此处的 ip 必须在 ips 列表中
# A comma separated list of machine hostname or IP would be installed Alert server, it
# must be a subset of configuration ips.
# Example for hostname: alertServerds3, Example for IP: alertServer192.168.8.3
alertServerhadoop102
# 告警服务所在服务器主机名
# A comma separated list of machine hostname or IP would be installed API server, it
# must be a subset of configuration ips.
# Example for hostname: apiServersds1, Example for IP: apiServers192.168.8.1
apiServershadoop102
# api服务所在服务器主机名
# A comma separated list of machine hostname or IP would be installed Python gateway server, it
# must be a subset of configuration ips.
# Example for hostname: pythonGatewayServersds1, Example for IP: pythonGatewayServers192.168.8.1
# pythonGatewayServersds1
# 不需要的配置项可以保留默认值也可以用 # 注释
# The directory to install DolphinScheduler for all machine we config above. It will automatically be created by install.sh script if not exists.
# Do not set this configuration same as the current path (pwd)
installPath/opt/module/dolphinscheduler
# DS 安装路径如果不存在会创建
# The user to deploy DolphinScheduler for all machine we config above. For now user must create by yourself before running install.sh
# script. The user needs to have sudo privileges and permissions to operate hdfs. If hdfs is enabled than the root directory needs
# to be created by this user
deployUseratguigu
# 部署用户任务执行服务是以 sudo -u {linux-user} 切换不同 Linux 用户的方式来实现多租户运行作业因此该用户必须有免密的 sudo 权限。
# The directory to store local data for all machine we config above. Make sure user deployUser have permissions to read and write this directory.
dataBasedirPath/tmp/dolphinscheduler
# 前文配置的所有节点的本地数据存储路径需要确保部署用户拥有该目录的读写权限
# DolphinScheduler ENV
# JAVA_HOME, we recommend use same JAVA_HOME in all machine you going to install DolphinScheduler
# and this configuration only support one parameter so far.
javaHome/opt/module/jdk1.8.0_212
# JAVA_HOME 路径
# DolphinScheduler API service port, also this is your DolphinScheduler UI components URL port, default value is 12345
apiServerPort12345
# Database
# NOTICE: If database value has special characters, such as .*[]^${}\?|()#, Please add prefix \ for escaping.
# The type for the metadata database
# Supported values: postgresql, mysql, h2.
# 注意数据库相关配置的 value 必须加引号否则配置无法生效
DATABASE_TYPEmysql
# 数据库类型
# Spring datasource url, following HOST:PORT/database?parameter format, If you using mysql, you could use jdbc
# string jdbc:mysql://127.0.0.1:3306/dolphinscheduler?useUnicodetruecharacterEncodingUTF-8 as example
# SPRING_DATASOURCE_URL${SPRING_DATASOURCE_URL:-jdbc:h2:mem:dolphinscheduler;MODEMySQL;DB_CLOSE_DELAY-1;DATABASE_TO_LOWERtrue}
SPRING_DATASOURCE_URLjdbc:mysql://hadoop102:3306/dolphinscheduler?useUnicodetruecharacterEncodingUTF-8
# 数据库 URL
# Spring datasource username
# SPRING_DATASOURCE_USERNAME${SPRING_DATASOURCE_USERNAME:-sa}
SPRING_DATASOURCE_USERNAMEdolphinscheduler
# 数据库用户名
# Spring datasource password
# SPRING_DATASOURCE_PASSWORD${SPRING_DATASOURCE_PASSWORD:-}
SPRING_DATASOURCE_PASSWORDdolphinscheduler
# 数据库密码
# Registry Server
# Registry Server plugin name, should be a substring of registryPluginDir, DolphinScheduler use this for verifying configuration consistency
registryPluginNamezookeeper
# 注册中心插件名称DS 通过注册中心来确保集群配置的一致性
# Registry Server address.
registryServershadoop102:2181,hadoop103:2181,hadoop104:2181
# 注册中心地址即 Zookeeper 集群的地址
# Registry Namespace
registryNamespacedolphinscheduler
# DS 在 Zookeeper 的结点名称
# Worker Task Server
# Worker Task Server plugin dir. DolphinScheduler will find and load the worker task plugin jar package from this dir.
taskPluginDirlib/plugin/task
# resource storage type: HDFS, S3, NONE
resourceStorageTypeHDFS
# 资源存储类型
# resource store on HDFS/S3 path, resource file will store to this hdfs path, self configuration, please make sure the directory exists on hdfs and has read write permissions. /dolphinscheduler is recommended
resourceUploadPath/dolphinscheduler
# 资源上传路径
# if resourceStorageType is HDFSdefaultFS write namenode addressHA, you need to put core-site.xml and hdfs-site.xml in the conf directory.
# if S3write S3 addressHAfor example s3a://dolphinscheduler
# NoteS3 be sure to create the root directory /dolphinscheduler
defaultFShdfs://hadoop102:8020
# 默认文件系统
# if resourceStorageType is S3, the following three configuration is required, otherwise please ignore
s3Endpointhttp://192.168.xx.xx:9010
s3AccessKeyxxxxxxxxxx
s3SecretKeyxxxxxxxxxx
# resourcemanager port, the default value is 8088 if not specified
resourceManagerHttpAddressPort8088
# yarn RM http 访问端口
# if resourcemanager HA is enabled, please set the HA IPs; if resourcemanager is single node, keep this value empty
yarnHaIps
# Yarn RM 高可用 ip若未启用 RM 高可用则将该值置空
# if resourcemanager HA is enabled or not use resourcemanager, please keep the default value; If resourcemanager is single node, you only need to replace yarnIp1 to actual resourcemanager hostname
singleYarnIphadoop103
# Yarn RM 主机名若启用了 HA 或未启用 RM保留默认值
# who has permission to create directory under HDFS/S3 root path
# Note: if kerberos is enabled, please config hdfsRootUser
hdfsRootUseratguigu
# 拥有 HDFS 根目录操作权限的用户
# kerberos config
# whether kerberos starts, if kerberos starts, following four items need to config, otherwise please ignore
kerberosStartUpfalse
# kdc krb5 config file path
krb5ConfPath$installPath/conf/krb5.conf
# keytab username,watch out the sign should followd by \\
keytabUserNamehdfs-mycluster\\ESZ.COM
# username keytab path
keytabPath$installPath/conf/hdfs.headless.keytab
# kerberos expire time, the unit is hour
kerberosExpireTime2
# use sudo or not
sudoEnabletrue
# worker tenant auto create
workerTenantAutoCreatefalse
初始化数据库 拷贝MySQL驱动到DolphinScheduler的解压目录下的lib中要求使用 MySQL JDBC Driver 8.0.16。
cp /opt/software/mysql-connector-java-8.0.16.jar lib/
执行数据库初始化脚本 数据库初始化脚本位于DolphinScheduler解压目录下的script目录中即/opt/software/ds/apache-dolphinscheduler-2.0.5-bin/script/。
script/create-dolphinscheduler.sh
一键部署DolphinScheduler
启动hadoop集群
hdp.sh start
启动Zookeeper集群
zk.sh start
一键部署并启动DolphinScheduler ./install.sh
查看DolphinScheduler进程 访问DolphinScheduler UI DolphinScheduler UI地址为http://hadoop102:12345/dolphinscheduler
初始用户的用户名为admin密码为dolphinscheduler123 DolphinScheduler启停命令 DolphinScheduler的启停脚本均位于其安装目录的bin目录下
一键启停所有服务
./bin/start-all.sh
./bin/stop-all.sh
注意同Hadoop的启停脚本进行区分。 启停 Master
./bin/dolphinscheduler-daemon.sh start master-server
./bin/dolphinscheduler-daemon.sh stop master-server 启停 Worker
./bin/dolphinscheduler-daemon.sh start worker-server
./bin/dolphinscheduler-daemon.sh stop worker-server 启停 Api
./bin/dolphinscheduler-daemon.sh start api-server
./bin/dolphinscheduler-daemon.sh stop api-server 启停 Logger
./bin/dolphinscheduler-daemon.sh start logger-server
./bin/dolphinscheduler-daemon.sh stop logger-server 启停 Alert
./bin/dolphinscheduler-daemon.sh start alert-server
./bin/dolphinscheduler-daemon.sh stop alert-server
