沈阳做网站客户多吗,旺店通app手机企业版下载,wordpress 发表时间,厦门网站建设求职简历目录
一、一些基本概念
二、集群配置图
三、Hadoop高可用集群配置步骤
1.在第一台虚拟机解压hadoop-3.1.3.tar.gz到/opt/soft/目录
2.修改文件名、属主和属组
3.配置windows四台虚拟机的ip映射
4.修改hadoop配置文件
(1)hadoop-env.sh
(2)workers
(3)crore-site.xml …目录
一、一些基本概念
二、集群配置图
三、Hadoop高可用集群配置步骤
1.在第一台虚拟机解压hadoop-3.1.3.tar.gz到/opt/soft/目录
2.修改文件名、属主和属组
3.配置windows四台虚拟机的ip映射
4.修改hadoop配置文件
(1)hadoop-env.sh
(2)workers
(3)crore-site.xml
(4)hdfs-site.xml
(5)mapred-site.xml
(6)yarn-site.xml
5.拷贝hadoop到其他三台虚拟机
6.分配环境变量
7.重启环境变量检验四台虚拟机安装是否成功
四、首次启动hadoop集群
1.高可用启动之前要启动zookeeper
2.三台机器启动JournalNode
3.第一台机器格式化
4.第一台机器启动namenode
5.在第二台机器同步namenode信息
6.第二台机器启动namenode
7.每台机器查看namenode的状态都是standby
8.关闭所有的与dfs有关的服务
9.格式化zookeeper
10.zkCli.sh
11.启动dfs
12.查看namenode节点状态
13.打开网页登录查看
14.每台虚拟机下载主备切换工具
15.启动yarn
16.有resourcemanager的主机名登录8088端口
17.查看resourcemanager节点状态
18.关闭集群 zookeeper集群的安装步骤参考博文《搭建zookeeper高可用集群详细步骤》注意主机名这里换了主机名ant161ant165;ant162ant166;ant163ant167;ant164ant168
一、一些基本概念
JournalNode的作用
Hadoop集群中的DFSZKFailoverController进程的作用
二、集群配置图
ant161ant162ant163ant164NameNodeNameNodeDataNode DataNodeDataNodeDataNodeNodeManagerNodeManagerNodeManagerNodeManagerResourceManagerResourceManagerJournalNode监控NameNode是否同步JournalNodeJournalNode DFSZKFailoverController监控NameNode是否存活 DFSZKFailoverController zookeeper0zookeeper1zookeeper2JobHistory
三、Hadoop高可用集群配置步骤
1.在第一台虚拟机解压hadoop-3.1.3.tar.gz到/opt/soft/目录
[rootant161 install]# tar -zxf ./hadoop-3.1.3.tar.gz -C /opt/soft/2.修改文件名、属主和属组
[rootant161 soft]# mv ./hadoop-3.1.3/ hadoop313[rootant161 soft]# chown -R root:root ./hadoop313/3.配置windows四台虚拟机的ip映射
C:\Windows\System32\drivers\etc目录下的host文件添加以下的主机ip配置 4.修改hadoop配置文件 (1)hadoop-env.sh
# The java implementation to use. By default, this environment
# variable is REQUIRED on ALL platforms except OS X!
export JAVA_HOME/opt/soft/jdk180
export HDFS_NAMENODE_USERroot
export HDFS_DATANODE_USERroot
export HDFS_SECONDARYNAMENODE_USERroot
export HDFS_JOURNALNODE_USERroot
export HDFS_ZKFC_USERroot
export YARN_RESOURCEMANAGER_USERroot
export YARN_NODEMANAGER_USERroot
(2)workers 输入四台虚拟机主机名
ant161
ant162
ant163
ant164(3)crore-site.xml
configurationpropertynamefs.defaultFS/namevaluehdfs://gky/valuedescription逻辑名称必须与hdfs-site.cml中的dfs.nameservices值保持一致/description/propertypropertynamehadoop.tmp.dir/namevalue/opt/soft/hadoop313/tmpdata/valuedescriptionnamenode上本地的hadoop临时文件夹/description/propertypropertynamehadoop.http.staticuser.user/namevalueroot/valuedescription默认用户/description/propertypropertynamehadoop.proxyuser.root.hosts/namevalue*/valuedescription/description/propertypropertynamehadoop.proxyuser.root.groups/namevalue*/valuedescription/description/propertypropertynameio.file.buffer.size/namevalue131072/valuedescription读写文件的buffer大小为128K/description/propertypropertynameha.zookeeper.quorum/namevalueant161:2181,ant162:2181,ant163:2181/valuedescription/description/propertypropertynameha.zookeeper.session-timeout.ms/namevalue10000/valuedescriptionhadoop链接zookeeper的超时时长设置为10s/description/property
/configuration
(4)hdfs-site.xml
configurationpropertynamedfs.replication/namevalue3/valuedescriptionHadoop中每一个block的备份数/description/propertypropertynamedfs.namenode.name.dir/namevalue/opt/soft/hadoop313/data/dfs/name/valuedescriptionnamenode上存储hdfs名字空间元数据目录/description/propertypropertynamedfs.datanode.data.dir/namevalue/opt/soft/hadoop313/data/dfs/data/valuedescriptiondatanode上数据块的物理存储位置/description/propertypropertynamedfs.namenode.secondary.http-address/namevalueant161:9869/valuedescription/description/propertypropertynamedfs.nameservices/namevaluegky/valuedescription指定hdfs的nameservice,需要和core-site.xml中保持一致/description/propertypropertynamedfs.ha.namenodes.gky/namevaluenn1,nn2/valuedescriptiongky为集群的逻辑名称映射两个namenode逻辑名/description/propertypropertynamedfs.namenode.rpc-address.gky.nn1/namevalueant161:9000/valuedescriptionnamenode1的RPC通信地址/description/propertypropertynamedfs.namenode.http-address.gky.nn1/namevalueant161:9870/valuedescriptionnamenode1的http通信地址/description/propertypropertynamedfs.namenode.rpc-address.gky.nn2/namevalueant162:9000/valuedescriptionnamenode2的RPC通信地址/description/propertypropertynamedfs.namenode.http-address.gky.nn2/namevalueant162:9870/valuedescriptionnamenode2的http通信地址/description/propertypropertynamedfs.namenode.shared.edits.dir/namevalueqjournal://ant161:8485;ant162:8485;ant163:8485/gky/valuedescription指定NameNode的edits元数据的共享存储位置(JournalNode列表)/description/propertypropertynamedfs.journalnode.edits.dir/namevalue/opt/soft/hadoop313/data/journaldata/valuedescription指定JournalNode在本地磁盘存放数据的位置/description/property!-- 容错 --propertynamedfs.ha.automatic-failover.enabled/namevaluetrue/valuedescription启用NameNode故障自动切换/description/propertypropertynamedfs.client.failover.proxy.provider.gky/namevalueorg.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider/valuedescription失败后自动切换的实现方式/description/propertypropertynamedfs.ha.fencing.methods/namevaluesshfence/valuedescription防止脑裂的处理/description/propertypropertynamedfs.ha.fencing.ssh.private-key-files/namevalue/root/.ssh/id_rsa/valuedescription使用sshfence隔离机制时需要ssh免登陆/description/propertypropertynamedfs.permissions.enabled/namevaluefalse/valuedescription关闭HDFS操作权限验证/description/propertypropertynamedfs.image.transfer.bandwidthPerSec/namevalue1048576/valuedescription1M/description/propertypropertynamedfs.block.scanner.volume.bytes.per.second/namevalue1048576/valuedescription如果该值为0则DataNode的块扫描程序将被禁用。如果这是正数则这是DataNode的块扫描程序将尝试从每个卷扫描的每秒字节数。/description/property
/configuration
(5)mapred-site.xml
propertynamemapreduce.framework.name/namevalueyarn/valuedescriptionjob执行框架 local, classic or yarn/description
/property
propertynamemapreduce.application.classpath/namevalue/opt/soft/hadoop313/etc/hadoop:/opt/soft/hadoop313/share/hadoop/common/lib/*:/opt/soft/hadoop313/share/hadoop/common/*:/opt/soft/hadoop313/share/hadoop/hdfs/*:/opt/soft/hadoop313/share/hadoop/hdfs/lib/*:/opt/soft/hadoop313/share/hadoop/mapreduce/*:/opt/soft/hadoop313/share/hadoop/mapreduce/lib/*:/opt/soft/hadoop313/share/hadoop/yarn/*:/opt/soft/hadoop313/share/hadoop/yarn/lib/*/valuedescription/description
/property
propertynamemapreduce.jobhistory.address/namevalueant161:10020/value/propertypropertynamemapreduce.jobhistory.webapp.address/namevalueant161:19888/value/property
propertynamemapreduce.map.memory.mb/namevalue1024/valuedescription设置map阶段的task工作内存/description
/property
propertynamemapreduce.reduce.memory.mb/namevalue2048/valuedescription设置reduce阶段的task工作内存/description
/property
(6)yarn-site.xml
configurationpropertynameyarn.resourcemanager.ha.enabled/namevaluetrue/valuedescription开启resourcemanager高可用/description/propertypropertynameyarn.resourcemanager.cluster-id/namevalueyrcabc/valuedescription指定yarn集群中的id/description/propertypropertynameyarn.resourcemanager.ha.rm-ids/namevaluerm1,rm2/valuedescription指定resourcemanager的名字/description/propertypropertynameyarn.resourcemanager.hostname.rm1/namevalueant163/valuedescription设置rm1的名字/description/propertypropertynameyarn.resourcemanager.hostname.rm2/namevalueant164/valuedescription设置rm2的名字/description/propertypropertynameyarn.resourcemanager.webapp.address.rm1/namevalueant163:8088/valuedescription/description/propertypropertynameyarn.resourcemanager.webapp.address.rm2/namevalueant164:8088/valuedescription/description/propertypropertynameyarn.resourcemanager.zk-address/namevalueant161:2181,ant162:2181,ant163:2181/valuedescription指定zookeeper集群地址/description/propertypropertynameyarn.nodemanager.aux-services/namevaluemapreduce_shuffle/valuedescription运行mapreduce程序必须配置的附属服务/description/propertypropertynameyarn.nodemanager.local-dirs/namevalue/opt/soft/hadoop313/tmpdata/yarn/local/valuedescriptionnodemanager本地存储目录/description/propertypropertynameyarn.nodemanager.log-dirs/namevalue/opt/soft/hadoop313/tmpdata/yarn/log/valuedescriptionnodemanager本地日志目录/description/propertypropertynameyarn.nodemanager.resource.memory-mb/namevalue2048/valuedescriptionresource进程的工作内存/description/propertypropertynameyarn.nodemanager.resource.cpu-vcores/namevalue2/valuedescriptionresource工作中所能使用机器的内核数/description/property!--下面三个配置在公司要删除--propertynameyarn.scheduler.minimum-allocation-mb/namevalue256/valuedescription/description/propertypropertynameyarn.log-aggregation-enable/namevaluetrue/valuedescription/description/propertypropertynameyarn.log-aggregation.retain-seconds/namevalue86400/valuedescription日志保留多少秒/description/propertypropertynameyarn.nodemanager.vmem-check-enabled/namevaluefalse/valuedescription/description/propertypropertynameyarn.application.classpath/namevalue/opt/soft/hadoop313/etc/hadoop:/opt/soft/hadoop313/share/hadoop/common/lib/*:/opt/soft/hadoop313/share/hadoop/common/*:/opt/soft/hadoop313/share/hadoop/hdfs/*:/opt/soft/hadoop313/share/hadoop/hdfs/lib/*:/opt/soft/hadoop313/share/hadoop/mapreduce/*:/opt/soft/hadoop313/share/hadoop/mapreduce/lib/*:/opt/soft/hadoop313/share/hadoop/yarn/*:/opt/soft/hadoop313/share/hadoop/yarn/lib/*/value/propertypropertynameyarn.nodemanager.env-whitelist/namevalueJAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME/value/property
/configuration
5.拷贝hadoop到其他三台虚拟机
[rootant161 soft]# scp -r ./hadoop313/ rootant162:/opt/soft/
[rootant161 soft]# scp -r ./hadoop313/ rootant163:/opt/soft/
[rootant161 soft]# scp -r ./hadoop313/ rootant164:/opt/soft/6.分配环境变量
[rootant161 soft]# scp /etc/profile rootant162:/etc/
profile 100% 2202 1.4MB/s 00:00
[rootant161 soft]# scp /etc/profile rootant163:/etc/
profile 100% 2202 1.4MB/s 00:00
[rootant161 soft]# scp /etc/profile rootant164:/etc/
profile
7.重启环境变量检验四台虚拟机安装是否成功
source /etc/profilehadoop hadoop version
四、首次启动hadoop集群
1.高可用启动之前要启动zookeeper
[rootant161 soft]# /opt/shell/zkop.sh start
------------ ant161 zookeeper -----------
JMX enabled by default
Using config: /opt/soft/zk345/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
------------ ant162 zookeeper -----------
JMX enabled by default
Using config: /opt/soft/zk345/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
------------ ant163 zookeeper -----------
JMX enabled by default
Using config: /opt/soft/zk345/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED[rootant161 soft]# /opt/shell/showjps.sh
---------------- ant161 服务启动状态 -----------------
2532 QuorumPeerMain
2582 Jps
---------------- ant162 服务启动状态 -----------------
2283 QuorumPeerMain
2335 Jps
---------------- ant163 服务启动状态 -----------------
2305 Jps
2259 QuorumPeerMain
---------------- ant164 服务启动状态 -----------------
2233 Jps[rootant161 soft]# /opt/shell/zkop.sh status
------------ ant161 zookeeper -----------
JMX enabled by default
Using config: /opt/soft/zk345/bin/../conf/zoo.cfg
Mode: follower
------------ ant162 zookeeper -----------
JMX enabled by default
Using config: /opt/soft/zk345/bin/../conf/zoo.cfg
Mode: leader
------------ ant163 zookeeper -----------
JMX enabled by default
Using config: /opt/soft/zk345/bin/../conf/zoo.cfg
Mode: follower2.三台机器启动JournalNode
[rootant161 soft]# hdfs --daemon start journalnode
WARNING: /opt/soft/hadoop313/logs does not exist. Creating.[rootant162 soft]# hdfs --daemon start journalnode
WARNING: /opt/soft/hadoop313/logs does not exist. Creating.[rootant163 soft]# hdfs --daemon start journalnode
WARNING: /opt/soft/hadoop313/logs does not exist. Creating. 3.第一台机器格式化
[rootant161 soft]# hdfs namenode -format4.第一台机器启动namenode
[rootant161 hadoop]# hdfs --daemon start namenode 5.在第二台机器同步namenode信息
[root ant162 soft]# hdfs namenode -bootstrapStandby 6.第二台机器启动namenode
[root ant162 soft]# hdfs --daemon start namenode 7.每台机器查看namenode的状态都是standby
[root ant161 soft]# hdfs haadmin -getServiceState nn1
standby
[root ant161 soft]# hdfs haadmin -getServiceState nn2
standby
8.关闭所有的与dfs有关的服务
[root ant161 soft]# stop-dfs.sh 9.格式化zookeeper
[root ant161 soft]# hdfs zkfc -formatZK 10.zkCli.sh 11.启动dfs 12.查看namenode节点状态
[rootant161 soft]# hdfs haadmin -getServiceState nn1
standby
[rootant161 soft]# hdfs haadmin -getServiceState nn2
active13.打开网页登录查看 14.每台虚拟机下载主备切换工具
[rootant161 soft]# yum install psmisc -y此时如果停止active那一台的namenode7218是active那一台机器namenode的进程号
[rootant162 soft]# kill -9 7218
此时active那一台网页无法连接另一台没有关闭namenode的机器的网页变为active
再重新启动关闭的namenode网页端的两个网址刷新就会发现刚刚变为active的那一台机器还是active状态而另一台重启的机器就编程了standby等待状态 15.启动yarn 16.有resourcemanager的主机名登录8088端口
哪一台机器是active的状态就会自动跳转到那一台机器的主机名 17.查看resourcemanager节点状态
[rootant161 soft]# yarn rmadmin -getServiceState rm1
active
[rootant161 soft]# yarn rmadmin -getServiceState rm2
standby18.关闭集群
(1)关闭dfs
[rootant161 soft]# stop-dfs.sh(2)关闭yarn
[rootant161 soft]# stop-yarn.sh(3)关闭journalnode
[rootant161 soft]# hdfs --daemon stop journalnode(4)关闭zookeeper
[rootant161 soft]# /opt/shell/zkop.sh stop
