定制网站建设简介,什么网站可以有人做详情页,昆明网络推广优化,做试客需要去哪些网站一、CDH介绍
1.CDH 是一个强大的商业版数据中心管理工具
提供了各种能够快速稳定运行的数据计算框架#xff0c;如Spark#xff1b; 使用Apache Impala做为对HDFS、HBase的高性能SQL查询引擎#xff1b; 使用Hive数据仓库工具帮助用户分析数据#xff1b; 提供CM安装HBas…一、CDH介绍
1.CDH 是一个强大的商业版数据中心管理工具
提供了各种能够快速稳定运行的数据计算框架如Spark 使用Apache Impala做为对HDFS、HBase的高性能SQL查询引擎 使用Hive数据仓库工具帮助用户分析数据 提供CM安装HBase分布式列式NoSQL数据库 包含原生的Hadoop搜索引擎以及Cloudera Navigator Optimizer去对Hadoop上的计算任务进行一个可视化的协调优化提高运行效率 提供的各种软件能让用户在一个可视化的UI界面中方便地管理、配置和监控Hadoop以及其它所有相关组件并有一定的容错容灾处理 提供了基于角色的访问控制安全管理。
2.CDH提供一下特性
灵活性能够存储各种类型的数据并使用各种不同的计算框架进行操作包括批处理、交互式sql、文本搜索、机器学习和统计计算集成性能够快速集成和运行一个完整的Hadoop平台适用于各种不同的硬件和软件安全性处理和控制敏感数据扩展性能够部署多种应用并扩展和扩充它们以满足你的需求高可用性可以放心地用于关键的商业任务兼容性可以利用你现有的IT基础设施和资源。
二、Cloudera Manager
1.CM介绍 CM是一种管理CDH集群的端到端的应用
CM通过对CDH集群的各部分提供精细的可视化和控制建立了企业级部署的标准增强了操作人员的能力以提升性能、提升服务质量、提高合规性、降低管理成本安装过程自动化使集群部署时间从几个星期减少到几分钟集群范围、实时的主机和服务运行情况的视图单一的中央控制台以对集群配置进行变更采用全方位的报告和诊断工具来帮助你优化性能和利用率CM的核心是Cloudera Manager Server。Server承载了管理员控制台Admin Console Web Server和应用逻辑并负责安装软件、配置、启动、停止服务以及管理运行有服务的集群。
2.与Cloudera Manager Server一同工作的组件
代理在每个主机上安装负责启动和停止进行、解包配置、触发安装、监控主机管理服务由一组角色组成的服务执行各种监视、告警、报告功能数据库存储配置和监控信息。通常多个逻辑数据库在一个或多个数据库服务器上运行。例如CMS和监控服务使用不同的逻辑数据库Cloudera仓库CM提供的用于软件分发的仓库客户端与服务器交互的借口包括管理控制台管理员使用该基于web的界面以管理集群和CM、API开发者可使用API创建自定义的CM应用。
3.CM功能
1状态管理
CMS维护了集群的各种状态。状态可分为两类模块和运行时两者都存储于CMS的数据库中模块中包含集群、主机、服务、角色、配置。运行时包含进程、命令。
2配置管理 CM在多个层面定义了配置如
服务层面可定义整个服务实例层面的配置如HDFS服务的默认副本因子角色组层面可定义某个角色组的配置如DataNode的处理线程数量可根据DataNodes的不同分组进行不同的配置角色层面可覆盖从角色组层面继承的配置。这种配置需要谨慎使用因为会造成角色组中的配置分歧。如因为排错需求临时启用某个角色实例的DEBUG日志主机层面根据监控、软件管理、资源管理的不同有不同的配置CM自身也有很多与管理操作相关的配置。
3进程管理
非CM管理的集群使用脚本进行角色进程的启动但在CM管理的集群中这类脚本不起作用CM管理的集群中只能使用CM进行角色进程的启停。CM使用开源的进程管理工具名为supervisord其会启动进程、重定向日志、通知进程失 败、为进程设置正确的用户ID等等CM支持自动重启一个崩溃的进程。 如果一个进程在启动后频繁崩溃还会被打上非健康标记停止CMS和CM代理不会使正在运行的进程被中止。
4软件管理 CM支持两种软件分发格式:packages和parcels
package是一种二进制分发格式包含编译的代码和元数据如包 述、版本、依赖项。包管理系统评估此元数据以允许包搜索、执行升级、确保包的所有依赖关系得到满足。CM使用本地操作系统支持的包管理程序parcel也是一种二进制分发格式包含CM需要使用的附加元数据。其与package的区别有:可安装同一个parcel的多个版本并激活其中一个; parcel可安装到任何路径;通过parcel安装CM会自动下载并激活和每 个节点操作系统版本匹配的parcel包解决某些操作系统版本不一致问 题。
5主机管理
CM 供了多种功能以管理Hadoop集群的主机。第一次运行CM管理员控制台时可搜索主机并添加到集群一旦选中了主机就可以为其分配CDH 角色。CM会在主机上自动部署作为集群托管节点的所有软件:JDKCM 代理CDHImpalaSolr等等服务部署并运行后管理员控制台中的“Hosts”区域显示集群中托管 主机的总体状态。 供的信息包括主机上的CDH版本、主机所属的集群、 运行在主机上的角色的数量。Cloudera管理服务中的主机监控角色执行 健康检查并收集主机的统计信息以允许你监控主机的健康和性能。
6资源管理 CM允许使用两种资源管理方式:
静态资源池:使用Linux cgroups在多个服务间静态地进行资源隔离如 HBase、Impala、YARN分别使用一定百分比的资源。静态资源池默认不启 用动态资源池:用于某些服务内部的资源管理如YARN的各种资源调度器; Impala也可对不同池中的查询动态分配资源。
7用户管理
访问CM通过用户账户进行控制。用户账户标识如何对用户进行身份验证并确定授予用户的权限CM 供了多种用户认证机制。可以配置CM使用CM数据库认证用户或使用某种外部认证服务。外部认证服务可以是LDAP服务器或者指定的其 他服务。CM还支持使用安全断言标记语言(SAML)来实现单点登录
8安全管理
认证:认证是指用户或服务证明其有访问某种系统资源的权限。Cloudera集群支持操作系统账号认证、LDAP、Kerberos等认证方式。LDAP和Kerberos并不是互斥的很多时候可以一起使用授权:授权关注谁可以存取或控制指定的资源或服务。CDH目前支持以 下几种权限控制:传统的POSIX形式的目录和文件权限控制;HDFS扩展 的ACL细粒度权限控制;HBase可对用户和组设置各种操作的ACL;使用Apache Sentry进行基于角色的权限控制加密:集群不同层面存储和传输的数据支持不同的加密方式。
9监控管理
Cloudera Management Service实现了多种管理特性包括活动监控、主机监控、服务监控、事件服务、告警发布、报表管理等。
三、Cloudera Manager 服务部署
1.环境准备
1服务器资源配置 A.配置不用过于纠结部署学习的4核、8核都可以内存最好16G以上不然后期安装服务的时候可能安装几个就满了web 监控部分会实时显示资源使用情况的。
iphostname配置作用192.168.101.105hadoop10520核32G、200Gserver、agent192.168.101.106hadoop10616核16G、100Gagent192.168.101.107hadoop10716核16G、100GagentB.firewalld、hostname 、selinux 配置
#第一次安装部署建议关闭防火墙可以避免后面过程出现很多问题后面在 web 中配置集群时安装软件会用到不同的端口,三台节点都执行
[roothadoop105 ~]# systemctl stop firewalld
[roothadoop105 ~]# systemctl disable firewalld#三台服务器都需要配置 hostname 解析
[rootlocalhost ~]# vi /etc/hosts
192.168.101.105 hadoop105
192.168.101.106 hadoop106
192.168.101.107 hadoop107
#测试解析是否正常
[rootlocalhost ~]# ping hadoop105
PING hadoop105 (192.168.101.105) 56(84) bytes of data.
64 bytes from hadoop105 (192.168.101.105): icmp_seq1 ttl64 time0.270 ms
64 bytes from hadoop105 (192.168.101.105): icmp_seq2 ttl64 time0.117 ms
64 bytes from hadoop105 (192.168.101.105): icmp_seq3 ttl64 time0.097 ms
64 bytes from hadoop105 (192.168.101.105): icmp_seq4 ttl64 time0.102 ms[rootlocalhost ~]# ping hadoop106
PING hadoop106 (192.168.101.106) 56(84) bytes of data.
64 bytes from hadoop106 (192.168.101.106): icmp_seq1 ttl64 time0.235 ms
64 bytes from hadoop106 (192.168.101.106): icmp_seq2 ttl64 time0.109 ms
64 bytes from hadoop106 (192.168.101.106): icmp_seq3 ttl64 time0.096 ms
64 bytes from hadoop106 (192.168.101.106): icmp_seq4 ttl64 time0.103 ms[rootlocalhost ~]# ping hadoop107
PING hadoop107 (192.168.101.107) 56(84) bytes of data.
64 bytes from hadoop107 (192.168.101.107): icmp_seq1 ttl64 time0.131 ms
64 bytes from hadoop107 (192.168.101.107): icmp_seq2 ttl64 time0.094 ms
64 bytes from hadoop107 (192.168.101.107): icmp_seq3 ttl64 time0.083 ms
64 bytes from hadoop107 (192.168.101.107): icmp_seq4 ttl64 time0.093 ms#关闭selinux
[roothadoop105 ~]# setenforce 0
[roothadoop106 ~]# setenforce 0
[roothadoop107 ~]# setenforce 0#永久关闭selinux
sed -ri s/SELINUXenforcing/SELINUXdisabled/ /etc/selinux/config
2服务器SSH免密配置 SSH免密配置用于部署过程方便个人操作文件传输等此步骤于CM服务部署没有直接关系可忽略不做但后期文件互传工作会相对麻烦。
#生成公钥和私钥
[roothadoop105 ~]# ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Created directory /root/.ssh.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:AmIOLBmYoOEVJHpFtC11MCkeDZOH/mH/4vbk/aR9bs4 roothadoop105
The keys randomart image is:
---[RSA 2048]----
|*o.*Bo. |
|Bo.o |
|*oo. |
|. .ooo |
| . o.oS |
| ... |
| .. . |
| oo.. .o|
| o.oo o.oE|
----[SHA256]-----
#公钥发送到目标主机上
[roothadoop105 ~]# ssh-copy-id hadoop105
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: /root/.ssh/id_rsa.pub
The authenticity of host hadoop105 (192.168.101.105) cant be established.
ECDSA key fingerprint is SHA256:BbZRGioTZYeFPNUrXxuZoIE3zXuooz5Lg6g4kascIE.
ECDSA key fingerprint is MD5:28:70:64:a2:b1:09:3f:1c:c3:27:2a:45:12:ac:5b:b9.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
roothadoop105s password: Number of key(s) added: 1Now try logging into the machine, with: ssh hadoop105
and check to make sure that only the key(s) you wanted were added.[roothadoop105 ~]# ssh-copy-id hadoop106
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: /root/.ssh/id_rsa.pub
The authenticity of host hadoop106 (192.168.101.106) cant be established.
ECDSA key fingerprint is SHA256:79uhkuskGIRi/wotMr5Lr3rXispB9u75B/SzOIvf3k.
ECDSA key fingerprint is MD5:63:41:6c:83:9a:f6:d2:96:ed:27:22:79:f9:f8:90:bf.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
roothadoop106s password: Number of key(s) added: 1Now try logging into the machine, with: ssh hadoop106
and check to make sure that only the key(s) you wanted were added.[roothadoop105 ~]# ssh-copy-id hadoop107
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: /root/.ssh/id_rsa.pub
The authenticity of host hadoop107 (192.168.101.107) cant be established.
ECDSA key fingerprint is SHA256:IUjVYd8gXsS9JHyWMxB56JK56o4sUPzZkWoJ3tW1x08.
ECDSA key fingerprint is MD5:98:30:97:44:7f:7c:9f:87:7b:63:7f:ac:81:24:71:bb.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
roothadoop107s password: Number of key(s) added: 1Now try logging into the machine, with: ssh hadoop107
and check to make sure that only the key(s) you wanted were added#测试免密登录是否成功不需要输入密码登录目标即成功
[roothadoop105 ~]# ssh hadoop106
Last login: Thu Feb 23 13:43:31 2023 from 192.168.101.144
[roothadoop106 ~]# exit
登出
Connection to hadoop106 closed.
[roothadoop105 ~]# ssh hadoop107
Last login: Thu Feb 23 13:43:33 2023 from 192.168.101.144#另外两台也按此操作即可主要用于server端为agent下发文件agent可以不配置即另外两台可以不做
3同步脚本配置
#安装rsync同步服务服务三台都需要安装
[roothadoop105 ~]# yum install -y rsync
#创建脚本信息如下
[roothadoop105 ~]# vi xsync.sh
#!/bin/bash
#1 获取输入参数个数如果没有参数直接退出
pcount$#
if((pcount0)); then
echo no args;
exit;
fi#2 获取文件名称
p1$1
fnamebasename $p1
echo fname$fname #3 获取上级目录到绝对路径
pdircd -P $(dirname $p1); pwd
echo pdir$pdir#4 获取当前用户名称
userwhoami#5 循环
for((host105; host107; host)); doecho ------------------- hadoop$host --------------rsync -av $pdir/$fname $userhadoop$host:$pdir
done
#赋予执行权限
[roothadoop105 ~]# chmod 755 xsync.sh#测试同步文件在另外两台/root目录下看到文件即为成功
[roothadoop105 ~]# touch test
[roothadoop105 ~]# ./xsync.sh test
fnametest
pdir/root
------------------- hadoop105 --------------
sending incremental file listsent 43 bytes received 12 bytes 110.00 bytes/sec
total size is 0 speedup is 0.00
------------------- hadoop106 --------------
sending incremental file list
testsent 86 bytes received 35 bytes 242.00 bytes/sec
total size is 0 speedup is 0.00
------------------- hadoop107 --------------
sending incremental file list
testsent 86 bytes received 35 bytes 80.67 bytes/sec
total size is 0 speedup is 0.004JDK环境配置
#上传jdk文件并同步给另外两台
[roothadoop105 ~]# /root/xsync.sh /opt/soft/oracle-j2sdk1.8-1.8.0update181-1.x86_64.rpm
#安装jdk服务三台同步安装也可安装其中一台把/usr/java/同步到另外两台
[roothadoop105 soft]# rpm -ivh oracle-j2sdk1.8-1.8.0update181-1.x86_64.rpm
警告oracle-j2sdk1.8-1.8.0update181-1.x86_64.rpm: 头V3 RSA/SHA256 Signature, 密钥 ID b0b19c9f: NOKEY
准备中... ################################# [100%]
正在升级/安装...1:oracle-j2sdk1.8-1.8.0update181-1################################# [100%]
#编辑profile新增配置
[roothadoop105 ~]# vi /etc/profile
export JAVA_HOME/usr/java/jdk1.8.0_181-cloudera
export CLASSPATH.:$CLASSPATH:$JAVA_HOME/lib
export PATH$PATH:$JAVA_HOME/bin
#生效文件
[roothadoop105 ~]# source /etc/profile
#检测安装结果
[roothadoop105 ~]# java -version
java version 1.8.0_181
Java(TM) SE Runtime Environment (build 1.8.0_181-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode)
#同步配置到另外两台
[roothadoop105 ~]# /root/xsync.sh /etc/profile
#另外两台生效配置
[roothadoop106 ~]# source /etc/profile
[roothadoop107 ~]# source /etc/profile
5集群整体操作脚本配置
#创建脚本
[roothadoop105 ~]# vi xcall.sh
#! /bin/bash
for i in hadoop105 hadoop106 hadoop107
doecho --------- $i ----------ssh $i $*
done
#修改脚本权限
[roothadoop105 ~]# chmod 755 xcall.sh
#环境变量配置追加到.bashrc文件
[roothadoop105 ~]# cat /etc/profile ~/.bashrc
[roothadoop106 ~]# cat /etc/profile ~/.bashrc
[roothadoop107 ~]# cat /etc/profile ~/.bashrc
#测试集群
[roothadoop105 ~]# ./xcall.sh jps
--------- hadoop105 ----------
3583 Jps
--------- hadoop106 ----------
3300 Jps
--------- hadoop107 ----------
3276 Jps
6mysql安装与配置
#mysql仅安装在server节点
#卸载掉本机自带的mariadb服务
[roothadoop105 ~]# rpm -qa |grep mari
mariadb-libs-5.5.68-1.el7.x86_64
[roothadoop105 ~]# rpm -e mariadb-libs-5.5.68-1.el7.x86_64 --nodeps
#安装相关依赖
[roothadoop105 soft]# yum install -y net-tools
[roothadoop105 soft]# yum install -y perl
#安装mysql服务
[roothadoop105 soft]# rpm -ivh mysql*
警告mysql57-community-release-el7-10.noarch.rpm: 头V3 DSA/SHA1 Signature, 密钥 ID 5072e1f5: NOKEY
准备中... ################################# [100%]
正在升级/安装...1:mysql-community-common-5.7.26-1.e################################# [ 17%]2:mysql-community-libs-5.7.26-1.el7################################# [ 33%]3:mysql-community-client-5.7.26-1.e################################# [ 50%]4:mysql-community-server-5.7.26-1.e################################# [ 67%]5:mysql-community-libs-compat-5.7.2################################# [ 83%]6:mysql57-community-release-el7-10 ################################# [100%]
#启动数据库
[roothadoop105 soft]# systemctl start mysqld
#查看mysql初始密码
[roothadoop105 soft]# cat /var/log/mysqld.log |grep pass
2023-02-24T08:32:49.190770Z 1 [Note] A temporary password is generated for rootlocalhost: nMNGt3nmy!V
#使用默认密码登录
[roothadoop105 soft]# mysql -uroot -p
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 5.7.26Copyright (c) 2000, 2019, Oracle and/or its affiliates. All rights reserved.Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.Type help; or \h for help. Type \c to clear the current input statement.mysql #修改mysql密码
mysql alter user rootlocalhost identified by 你的密码;
Query OK, 0 rows affected (0.00 sec)
#创建scm用户
mysql GRANT ALL ON scm.* TO scm% IDENTIFIED BY scm用户密码;
Query OK, 0 rows affected, 1 warning (0.00 sec)
#创建scm数据库
mysql CREATE DATABASE scm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
Query OK, 1 row affected (0.00 sec)
#创建hive数据库
mysql CREATE DATABASE hive DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
Query OK, 1 row affected (0.00 sec)
#创建oozie数据库
mysql CREATE DATABASE oozie DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
Query OK, 1 row affected (0.00 sec)
#创建hue数据库
mysql CREATE DATABASE hue DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
Query OK, 1 row affected (0.00 sec)#开启远程登录数据库
mysql use mysql;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -ADatabase changed
mysql update user set host% where userroot;
Query OK, 1 row affected (0.00 sec)
Rows matched: 1 Changed: 1 Warnings: 0mysql flush privileges;
Query OK, 0 rows affected (0.00 sec)
7时间同步配置
#安装ntp服务
[roothadoop105 ~]# yum install -y ntp
[roothadoop106 ~]# yum install -y ntp
[roothadoop107 ~]# yum install -y ntp#server节点配置ntp服务端
[roothadoop105 ~]# vi /etc/ntp.conf
#注释掉所有的restrict开头的配置
#注释掉所有的server开头的配置
#新增如下配置
restrict 192.168.101.105 mask 255.255.255.0 nomodify notrap
server 127.127.1.0
fudge 127.127.1.0 stratum 10
#启动ntp服务
[roothadoop105 ~]# systemctl start ntpd
[roothadoop105 ~]# netstat -tunlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 1611/master
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1360/sshd
tcp6 0 0 ::1:25 :::* LISTEN 1611/master
tcp6 0 0 :::3306 :::* LISTEN 1801/mysqld
tcp6 0 0 :::22 :::* LISTEN 1360/sshd
udp 0 0 192.168.101.105:123 0.0.0.0:* 1999/ntpd
udp 0 0 127.0.0.1:123 0.0.0.0:* 1999/ntpd
udp 0 0 0.0.0.0:123 0.0.0.0:* 1999/ntpd
udp6 0 0 fe80::35e9:7b14:664:123 :::* 1999/ntpd
udp6 0 0 ::1:123 :::* 1999/ntpd
udp6 0 0 :::123 :::* 1999/ntpd#agent节点配置ntp客户端
[roothadoop105 ~]# vi /etc/ntp.conf
#注释掉所有的restrict开头的配置
#注释掉所有的server开头的配置
#新增如下配置
server 192.168.101.105#进行手动测试ntp
[roothadoop106 ~]# ntpdate 192.168.101.105
24 Feb 17:24:01 ntpdate[1894]: no server suitable for synchronization found#这里no server suitable for synchronization found是因为server端防火墙没有开发ntp服务端口
#ntp服务端开放ntp端口如果前面没有操作关闭防火墙则需要执行如下操作
[roothadoop105 ~]# firewall-cmd --permanent --add-port123/udp
success
[roothadoop105 ~]# firewall-cmd --reload
success#重新测试
[roothadoop106 ~]# ntpdate 192.168.101.105
24 Feb 17:27:17 ntpdate[1897]: adjust time server 192.168.101.105 offset 0.005958 sec
[roothadoop107 ~]# ntpdate 192.168.101.105
24 Feb 17:27:43 ntpdate[1887]: adjust time server 192.168.101.105 offset 0.003158 sec#测试成功启动客户端ntp服务
[roothadoop106 ~]# systemctl start ntpd
[roothadoop107 ~]# systemctl start ntpd#配置ntp服务开机自启
[roothadoop105 ~]# systemctl enable ntpd.service
[roothadoop106 ~]# systemctl enable ntpd.service
[roothadoop107 ~]# systemctl enable ntpd.service
2.安装配置
1CM部署安装
#创建java目录
[roothadoop105 ~]# mkdir /usr/share/java#上传mysql-connector-java-5.1.47.jar到此目录下重命名并同步至其他节点
[roothadoop105 java]# mv mysql-connector-java-5.1.47.jar mysql-connector-java.jar
[roothadoop105 java]# /root/xsync.sh /usr/share/java/mysql-connector-java.jar
fnamemysql-connector-java.jar
pdir/usr/share/java
------------------- hadoop105 --------------
sending incremental file listsent 64 bytes received 12 bytes 152.00 bytes/sec
total size is 1,007,502 speedup is 13,256.61
------------------- hadoop106 --------------
sending incremental file list
mysql-connector-java.jarsent 1,007,849 bytes received 35 bytes 2,015,768.00 bytes/sec
total size is 1,007,502 speedup is 1.00
------------------- hadoop107 --------------
sending incremental file list
mysql-connector-java.jarsent 1,007,849 bytes received 35 bytes 2,015,768.00 bytes/sec
total size is 1,007,502 speedup is 1.00#上传cm服务文件并解压
[roothadoop105 soft]# tar -zxvf cm6.3.1-redhat7.tar.gz
[roothadoop105 soft]# cd ../#创建cm_soft目录存放cm安装文件
[roothadoop105 opt]# mkdir cm_soft
[roothadoop105 opt]# cp /opt/soft/cm6.3.1/RPMS/
noarch/ x86_64/
[roothadoop105 opt]# cp /opt/soft/cm6.3.1/RPMS/x86_64/cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm /opt/cm_soft/
[roothadoop105 opt]# cp /opt/soft/cm6.3.1/RPMS/x86_64/cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm /opt/cm_soft/#将agent和daemons安装包同步至其他两个节点
[roothadoop105 opt]# /root/xsync.sh /opt/cm_soft/
fnamecm_soft
pdir/opt
------------------- hadoop105 --------------
sending incremental file listsent 181 bytes received 17 bytes 132.00 bytes/sec
total size is 1,214,316,032 speedup is 6,132,909.25
------------------- hadoop106 --------------
sending incremental file list
cm_soft/
cm_soft/cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm
cm_soft/cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpmsent 1,214,612,758 bytes received 58 bytes 127,853,980.63 bytes/sec
total size is 1,214,316,032 speedup is 1.00
------------------- hadoop107 --------------
sending incremental file list
cm_soft/
cm_soft/cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm
cm_soft/cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpmsent 1,214,612,758 bytes received 58 bytes 161,948,375.47 bytes/sec
total size is 1,214,316,032 speedup is 1.00
#将server服务也复制到cm目录
[roothadoop105 opt]# cp /opt/soft/cm6.3.1/RPMS/x86_64/cloudera-manager-server-6.3.1-1466458.el7.x86_64.rpm /opt/cm_soft/#每个节点都安装相关依赖服务
[roothadoop105 ~]# yum install -y chkconfig python bind-utils psmisc libxslt zlib sqlite cyrus-sasl-plain cyrus-sasl-gssapi fuse fuse-libs redhat-lsb bind-utils psmisc cyrus-sasl-plain cyrus-sasl-gssapi fuse portmap fuse-libs /lib/lsb/init-functions httpd mod_ssl openssl-devel python-psycopg2 MySQL-python libxslt
[roothadoop106 ~]# yum install -y chkconfig python bind-utils psmisc libxslt zlib sqlite cyrus-sasl-plain cyrus-sasl-gssapi fuse fuse-libs redhat-lsb bind-utils psmisc cyrus-sasl-plain cyrus-sasl-gssapi fuse portmap fuse-libs /lib/lsb/init-functions httpd mod_ssl openssl-devel python-psycopg2 MySQL-python libxslt
[roothadoop106 ~]# yum install -y chkconfig python bind-utils psmisc libxslt zlib sqlite cyrus-sasl-plain cyrus-sasl-gssapi fuse fuse-libs redhat-lsb bind-utils psmisc cyrus-sasl-plain cyrus-sasl-gssapi fuse portmap fuse-libs /lib/lsb/init-functions httpd mod_ssl openssl-devel python-psycopg2 MySQL-python libxslt#每个节点都安装daemon和agent服务
[roothadoop105 ~]# rpm -ivh /opt/cm_soft/cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm
警告/opt/cm_soft/cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm: 头V3 RSA/SHA256 Signature, 密钥 ID b0b19c9f: NOKEY
准备中... ################################# [100%]
正在升级/安装...1:cloudera-manager-daemons-6.3.1-14################################# [100%]
[roothadoop105 ~]# rpm -ivh /opt/cm_soft/cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm
警告/opt/cm_soft/cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm: 头V3 RSA/SHA256 Signature, 密钥 ID b0b19c9f: NOKEY
准备中... ################################# [100%]
正在升级/安装...1:cloudera-manager-agent-6.3.1-1466################################# [100%]
Created symlink from /etc/systemd/system/multi-user.target.wants/cloudera-scm-agent.service to /usr/lib/systemd/system/cloudera-scm-agent.service.
Created symlink from /etc/systemd/system/multi-user.target.wants/supervisord.service to /usr/lib/systemd/system/supervisord.service.#修改server端的agent配置
[roothadoop105 ~]# vi /etc/cloudera-scm-agent/config.ini
#修改该参数
server_hosthadoop105#将该配置同步到另外两个节点
[roothadoop105 ~]# /root/xsync.sh /etc/cloudera-scm-agent/config.ini
fnameconfig.ini
pdir/etc/cloudera-scm-agent
------------------- hadoop105 --------------
sending incremental file listsent 77 bytes received 12 bytes 178.00 bytes/sec
total size is 9,826 speedup is 110.40
------------------- hadoop106 --------------
sending incremental file list
config.inisent 880 bytes received 125 bytes 670.00 bytes/sec
total size is 9,826 speedup is 9.78
------------------- hadoop107 --------------
sending incremental file list
config.inisent 880 bytes received 125 bytes 2,010.00 bytes/sec
total size is 9,826 speedup is 9.78#server端安装cm server服务
[roothadoop105 ~]# rpm -ivh /opt/cm_soft/cloudera-manager-server-6.3.1-1466458.el7.x86_64.rpm
警告/opt/cm_soft/cloudera-manager-server-6.3.1-1466458.el7.x86_64.rpm: 头V3 RSA/SHA256 Signature, 密钥 ID b0b19c9f: NOKEY
准备中... ################################# [100%]
正在升级/安装...1:cloudera-manager-server-6.3.1-146################################# [100%]
Created symlink from /etc/systemd/system/multi-user.target.wants/cloudera-scm-server.service to /usr/lib/systemd/system/cloudera-scm-server.service.#修改cm server服务配置文件
[roothadoop105 ~]# vi /etc/cloudera-scm-server/db.properties
com.cloudera.cmf.db.typemysql
com.cloudera.cmf.db.hosthadoop102:3306
com.cloudera.cmf.db.namescm
com.cloudera.cmf.db.userscm
com.cloudera.cmf.db.password你设置的密码
com.cloudera.cmf.db.setupTypeEXTERNAL#上传CDH包到/opt/cloudera/parcel-repo/目录下
[roothadoop105 ~]# ll /opt/cloudera/parcel-repo/
总用量 2033428
-rw-r--r--. 1 root root 2082186246 2月 8 16:37 CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel
-rw-r--r--. 1 root root 40 2月 8 16:37 CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha
-rw-r--r--. 1 root root 33887 2月 8 16:37 manifest.json#初始化数据库
[roothadoop105 ~]# /opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm scm
Enter SCM password:
JAVA_HOME/usr/java/jdk1.8.0_181-cloudera
Verifying that we can write to /etc/cloudera-scm-server
Creating SCM configuration file in /etc/cloudera-scm-server
Executing: /usr/java/jdk1.8.0_181-cloudera/bin/java -cp /usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/java/postgresql-connector-java.jar:/opt/cloudera/cm/schema/../lib/* com.cloudera.enterprise.dbutil.DbCommandExecutor /etc/cloudera-scm-server/db.properties com.cloudera.cmf.db.
Wed Mar 01 11:30:39 CST 2023 WARN: Establishing SSL connection without servers identity verification is not recommended. According to MySQL 5.5.45, 5.6.26 and 5.7.6 requirements SSL connection must be established by default if explicit option isnt set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to false. You need either to explicitly disable SSL by setting useSSLfalse, or set useSSLtrue and provide truststore for server certificate verification.
[ main] DbCommandExecutor INFO Successfully connected to database.
All done, your SCM database is configured correctly!#如果提示ERROR JDBC Driver com.mysql.jdbc.Driver not found 那一定是上面的mysql-connector-java-5.1.47.jar没有重命名成mysql-connector-java.jar#启动cm server服务并查看状态
[roothadoop105 ~]# systemctl start cloudera-scm-server
[roothadoop105 ~]# systemctl status cloudera-scm-server
● cloudera-scm-server.service - Cloudera CM Server ServiceLoaded: loaded (/usr/lib/systemd/system/cloudera-scm-server.service; enabled; vendor preset: disabled)Active: active (running) since 三 2023-03-01 11:32:51 CST; 14s agoProcess: 12093 ExecStartPre/opt/cloudera/cm/bin/cm-server-pre (codeexited, status0/SUCCESS)Main PID: 12098 (java)CGroup: /system.slice/cloudera-scm-server.service└─12098 /usr/java/jdk1.8.0_181-cloudera/bin/java -cp .:/usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/usr/...3月 01 11:33:05 hadoop105 cm-server[12098]: Wed Mar 01 11:33:05 CST 2023 WARN: Establishing SSL connection without servers identity verificatio...ng applic
3月 01 11:33:05 hadoop105 cm-server[12098]: Wed Mar 01 11:33:05 CST 2023 WARN: Establishing SSL connection without servers identity verificatio...ng applic
3月 01 11:33:05 hadoop105 cm-server[12098]: Wed Mar 01 11:33:05 CST 2023 WARN: Establishing SSL connection without servers identity verificatio...ng applic
3月 01 11:33:05 hadoop105 cm-server[12098]: Wed Mar 01 11:33:05 CST 2023 WARN: Establishing SSL connection without servers identity verificatio...ng applic
3月 01 11:33:06 hadoop105 cm-server[12098]: Wed Mar 01 11:33:06 CST 2023 WARN: Establishing SSL connection without servers identity verificatio...ng applic
3月 01 11:33:06 hadoop105 cm-server[12098]: Wed Mar 01 11:33:06 CST 2023 WARN: Establishing SSL connection without servers identity verificatio...ng applic
3月 01 11:33:06 hadoop105 cm-server[12098]: Wed Mar 01 11:33:06 CST 2023 WARN: Establishing SSL connection without servers identity verificatio...ng applic
3月 01 11:33:06 hadoop105 cm-server[12098]: Wed Mar 01 11:33:06 CST 2023 WARN: Establishing SSL connection without servers identity verificatio...ng applic
3月 01 11:33:06 hadoop105 cm-server[12098]: Wed Mar 01 11:33:06 CST 2023 WARN: Establishing SSL connection without servers identity verificatio...ng applic
3月 01 11:33:06 hadoop105 cm-server[12098]: Wed Mar 01 11:33:06 CST 2023 WARN: Establishing SSL connection without servers identity verificatio...ng applic
Hint: Some lines were ellipsized, use -l to show in full.#如上面没有执行关闭防火墙操作则server端需开放agent连接端口7182和web服务端口7180
[roothadoop105 ~]# firewall-cmd --permanent --add-port7182/tcp --add-port7180/tcp
[roothadoop105 ~]# firewall-cmd --reload#启动agent节点
[roothadoop105 ~]# systemctl start cloudera-scm-agent
[roothadoop106 ~]# systemctl start cloudera-scm-agent
[roothadoop107 ~]# systemctl start cloudera-scm-agent
3.集群部署
1浏览器登录web服务进行机器配置
web连接http://hadoop105:7180 或者 http://ip:7180账户密码默认为 admin。 2选择安装CM服务安装许可版本
使用免费版即可具体看个人需求也可先使用60天。 3进入集群安装
A.欢迎界面直接选继续。 B.集群名称。 C.选择集群主机、或者选择当前管理的主机因为agent服务已经连接到server端所以会显示有三台主机可管理此处我选择当前管理主机勾选并继续。 D.默认选择并继续。 E.Parcels分配安装这里显示有问题先不管。 F.执行网络检查和主机检查点击即可。 G.Inspect Hosts的警告可以忽略直接跳过并安装。 H.也可点击显示检查器结果对相关问题进行修复按下列对每一个节点进行操作即可。
#对每个节点进行操作
[roothadoop105 ~]# echo 10 /proc/sys/vm/swappiness
[roothadoop105 ~]# echo never /sys/kernel/mm/transparent_hugepage/defrag
[roothadoop105 ~]# echo never /sys/kernel/mm/transparent_hugepage/enabled
I.重新检查并继续。 J.选择要安装的服务搭配或者自定义安装 K.选择服务安装的节点这里我为了方便以下部分都是默认 L.最后根据图示警告或者错误点击查看并按相关信息去修正问题即可服务器本身资源不够很难满足系统需求此博客作学习搭建使用提供给有需要的人及自己查阅。 4.错误集锦
错误1数据库测试连接失败 处理mysql服务没有开启或者没有开启远程登录查看 三-1-(6) 进行操作
错误2org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot create directory /tmp. Name node is in safe mode 此处报错查看详细信息得到报错原因是出于hdfs安全模式中 处理解除安全模式之后点击 Resume
#如何解除安全模式#查看运行用户
[roothadoop105 ~]# ps -ef |grep hdfs
hdfs 15275 12499 0 09:08 ? 00:00:00 /usr/bin/python2 /opt/cloudera/cm-agent/bin/cm proc_watcher 15309
hdfs 15276 12499 0 09:08 ? 00:00:00 /usr/bin/python2 /opt/cloudera/cm-agent/bin/cm proc_watcher 15313
hdfs 15309 15275 2 09:08 ? 00:00:19 /usr/java/jdk1.8.0_181-cloudera/bin/java -Dproc_namenode -Dhdfs.audit.loggerINFO,RFAAUDIT -Dsecurity.audit.loggerINFO,RFAS -Djava.net.preferIPv4Stacktrue -Xms2390753280 -Xmx2390753280 -XX:UseParNewGC -XX:UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction70 -XX:CMSParallelRemarkEnabled -XX:HeapDumpOnOutOfMemoryError -XX:HeapDumpPath/tmp/hdfs_hdfs-NAMENODE-9dbc330521cc4b5d25350c6b4cc6294f_pid15309.hprof -XX:OnOutOfMemoryError/opt/cloudera/cm-agent/service/common/killparent.sh -Dyarn.log.dir/var/log/hadoop-hdfs -Dyarn.log.filehadoop-cmf-hdfs-NAMENODE-hadoop105.log.out -Dyarn.home.dir/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hadoop-yarn -Dyarn.root.loggerINFO,console -Djava.library.path/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hadoop/lib/native -Dhadoop.log.dir/var/log/hadoop-hdfs -Dhadoop.log.filehadoop-cmf-hdfs-NAMENODE-hadoop105.log.out -Dhadoop.home.dir/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hadoop -Dhadoop.id.strhdfs -Dhadoop.root.loggerINFO,RFA -Dhadoop.policy.filehadoop-policy.xml -Dhadoop.security.loggerINFO,RFAS org.apache.hadoop.hdfs.server.namenode.NameNode
hdfs 15310 15275 0 09:08 ? 00:00:00 /usr/bin/python2 /opt/cloudera/cm-agent/bin/cm redactor --fds 3 5
hdfs 15313 15276 1 09:08 ? 00:00:11 /usr/java/jdk1.8.0_181-cloudera/bin/java -Dproc_secondarynamenode -Dhdfs.audit.loggerINFO,RFAAUDIT -Dsecurity.audit.loggerINFO,RFAS -Djava.net.preferIPv4Stacktrue -Xms2390753280 -Xmx2390753280 -XX:UseParNewGC -XX:UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction70 -XX:CMSParallelRemarkEnabled -XX:HeapDumpOnOutOfMemoryError -XX:HeapDumpPath/tmp/hdfs_hdfs-SECONDARYNAMENODE-9dbc330521cc4b5d25350c6b4cc6294f_pid15313.hprof -XX:OnOutOfMemoryError/opt/cloudera/cm-agent/service/common/killparent.sh -Dyarn.log.dir/var/log/hadoop-hdfs -Dyarn.log.filehadoop-cmf-hdfs-SECONDARYNAMENODE-hadoop105.log.out -Dyarn.home.dir/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hadoop-yarn -Dyarn.root.loggerINFO,console -Djava.library.path/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hadoop/lib/native -Dhadoop.log.dir/var/log/hadoop-hdfs -Dhadoop.log.filehadoop-cmf-hdfs-SECONDARYNAMENODE-hadoop105.log.out -Dhadoop.home.dir/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hadoop -Dhadoop.id.strhdfs -Dhadoop.root.loggerINFO,RFA -Dhadoop.policy.filehadoop-policy.xml -Dhadoop.security.loggerINFO,RFAS org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode
hdfs 15314 15276 0 09:08 ? 00:00:00 /usr/bin/python2 /opt/cloudera/cm-agent/bin/cm redactor --fds 3 5
root 20784 15132 0 09:23 pts/1 00:00:00 grep --colorauto hdfs#进hdfs用户解除安全模式不能用root用户解除
[roothadoop105 ~]# hadoop dfsadmin -safemode leave
WARNING: Use of this script to execute dfsadmin is deprecated.
WARNING: Attempting to execute replacement hdfs dfsadmin instead.
safemode: Access denied for user root. Superuser privilege is required#切换hdfs用户发现用户不可用
[roothadoop105 ~]# su - hdfs
This account is currently not available.#修改passwd文件将hdfs用户 /sbin/nologin 改为 /bin/bash
[roothadoop105 ~]# vi /etc/passwd
hdfs:x:995:992:Hadoop HDFS:/var/lib/hadoop-hdfs:/bin/bash#执行解除安全模式命令
[roothadoop105 ~]# su - hdfs
上一次登录四 3月 2 09:23:59 CST 2023pts/1 上
[hdfshadoop105 ~]$ hadoop dfsadmin -safemode leave
WARNING: Use of this script to execute dfsadmin is deprecated.
WARNING: Attempting to execute replacement hdfs dfsadmin instead.Safe mode is OFF
其他错误基本上过程遇到了其他安装失败的问题都是server端没有关闭防火墙且未开启相关端口导致所以建议部署过程中直接关闭防火墙如果因为安全问题需要开着防火墙就需要对一些基本端口进行开放操作
