做网站空间会招攻击,网站建设艾瑞市场分析,湖北长欣建设有限公司网站,网站内页做排名1、下载Hadoop安装包
可以到官方网站下载#xff0c;也可以使用网盘下载
官网下载地址#xff1a;Hadoop官网下载地址
网盘地址#xff1a;https://pan.baidu.com/s/1p4BXq2mvby2B76lmpiEjnA?pwdr62r提取码: r62r
2、解压并添加环境变量
# 将安装包移动到指定目录
mv …
1、下载Hadoop安装包
可以到官方网站下载也可以使用网盘下载
官网下载地址Hadoop官网下载地址
网盘地址https://pan.baidu.com/s/1p4BXq2mvby2B76lmpiEjnA?pwdr62r提取码: r62r
2、解压并添加环境变量
# 将安装包移动到指定目录
mv ~/Download/hadoop-3.3.6.tar.gz /opt/module
# 进入目录
cd /opt/module
# 解压安装包
tar -zxvf hadoop-3.3.6.tar.gz
# 修改目录名
mv hadoop-3.3.6 hadoop# 创建目录
mkdir -p /opt/module/hadoop/data/dfs/name
mkdir -p /opt/module/hadoop/data/dfs/data
mkdir -p /opt/module/hadoop/data/hadoop/edits
mkdir -p /opt/module/hadoop/data/hadoop/snn/checkpoint
mkdir -p /opt/module/hadoop/data/hadoop/snn/edits
mkdir -p /opt/module/hadoop/data/hadoop/tmp
mkdir -p /opt/module/hadoop/logs# 添加环境变量
sudo vim /etc/profileexport JAVA8_HOME/Library/Java/JavaVirtualMachines/jdk8/Contents/Home
export MAVEN_HOME/Library/Java/apache-maven-3.8.8
export JRE_HOME${JAVA_HOME}/jre
export CLASSPATH.:${JAVA_HOME}/lib:${JRE_HOME}/lib
export HADOOP_HOME/opt/module/hadoop
export JAVA_LIBRARY_PATH$HADOOP_HOME/lib/native
export HADOOP_COMMON_LIB_NATIVE_DIR$HADOOP_HOME/lib/native
export HADOOP_LOG_DIR$HADOOP_HOME/logs
export HADOOP_MAPRED_HOME$HADOOP_HOME
export HADOOP_COMMON_HOME$HADOOP_HOME
export HADOOP_HDFS_HOME$HADOOP_HOME
export YARN_HOME$HADOOP_HOME
export HADOOP_USER_NAMEhdfsexport PATH$JAVA_HOME/bin:$MYSQL_HOME/bin:$MAVEN_HOME:/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$MAVEN_HOME/bin:$PATH:.
3、修改配置
vim etc/hadoop/core-site.xml
?xml version1.0 encodingUTF-8?
?xml-stylesheet typetext/xsl hrefconfiguration.xsl?
!--Licensed under the Apache License, Version 2.0 (the License);you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an AS IS BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License. See accompanying LICENSE file.
--!-- Put site-specific property overrides in this file. --configurationpropertynamefs.defaultFS/namevaluehdfs://0.0.0.0:9000/value/propertypropertynamehadoop.tmp.dir/namevalue/opt/module/hadoop/data/value/propertypropertynamehadoop.http.staticuser.user/namevaluehdfs/value/property!-- 缓冲区大小根据服务器性能动态调整 --propertynameio.file.buffer.size/namevalue4096/value/property!-- 开启垃圾桶机制删除掉的数据可以从垃圾桶中回收单位分钟 --propertynamefs.trash.interval/namevalue10080/value/propertypropertynamedfs.permissions/namevaluefalse/value/propertypropertynamehadoop.proxyuser.hdfs.hosts/namevalue*/value/propertypropertynamehadoop.proxyuser.hdfs.groups/namevalue*/value/property
/configuration vim etc/hadoop/hdfs-site.xml
?xml version1.0 encodingUTF-8?
?xml-stylesheet typetext/xsl hrefconfiguration.xsl?
!--Licensed under the Apache License, Version 2.0 (the License);you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an AS IS BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License. See accompanying LICENSE file.
--!-- Put site-specific property overrides in this file. --configurationpropertynamedfs.namenode.http-address/namevalue0.0.0.0:9870/value
/propertypropertynamedfs.namenode.secondary.http-address/namevalue0.0.0.0:9868/value
/property!-- 数据存储位置多个目录用英文逗号隔开propertynamedfs.namenode.name.dir/namevalue/opt/module/hadoop/data/dfs/name/value/propertypropertynamedfs.datanode.data.dir/namevalue/opt/module/hadoop/data/dfs/data/value/property --!-- 元数据操作日志、检查点日志存储目录 propertynamedfs.namenode.edits.dir/namevalue/opt/module/hadoop/data/hadoop/edits/value/propertypropertynamedfs.namenode.checkpoint.dir/namevalue/opt/module/hadoop/data/hadoop/snn/checkpoint/value/propertypropertynamedfs.namenode.checkpoint.edits.dir/namevalue/opt/module/hadoop/data/hadoop/snn/edits/value/property --!-- 临时文件目录propertynamedfs.tmp.dir/namevalue/opt/module/hadoop/data/hadoop/tmp/value/property --!-- 文件切片的副本个数 --propertynamedfs.replication/namevalue1/value/property!-- HDFS 的文件权限 --propertynamedfs.permissions.enabled/namevaluefalse/value/property!-- 每个 Block 的大小为128 MB单位KB --propertynamedfs.blocksize/namevalue134217728/value/property
/configurationvim etc/hadoop/mapred-site.xml
?xml version1.0?
?xml-stylesheet typetext/xsl hrefconfiguration.xsl?
!--Licensed under the Apache License, Version 2.0 (the License);you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an AS IS BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License. See accompanying LICENSE file.
--!-- Put site-specific property overrides in this file. --configurationpropertynamemapreduce.framework.name/namevalueyarn/value
/property!-- 设置历史任务的主机和端口0.0.0.0 支持来自服务器外部的访问 --propertynamemapreduce.jobhistory.address/namevalue0.0.0.0:10020/value/property!-- 设置网页端的历史任务的主机和端口 --propertynamemapreduce.jobhistory.webapp.address/namevalue0.0.0.0:19888/value/propertypropertynamemapreduce.application.classpath/namevalue/opt/module/hadoop/etc/hadoop:/opt/module/hadoop/share/hadoop/common/lib/*:/opt/module/hadoop/share/hadoop/common/*:/opt/module/hadoop/share/hadoop/hdfs:/opt/module/hadoop/share/hadoop/hdfs/lib/*:/opt/module/hadoop/share/hadoop/hdfs/*:/opt/module/hadoop/share/hadoop/mapreduce/*:/opt/module/hadoop/share/hadoop/yarn:/opt/module/hadoop/share/hadoop/yarn/lib/*:/opt/module/hadoop/share/hadoop/yarn/*/value/property
/configurationvim etc/hadoop/yarn-site.xml
?xml version1.0?
!--Licensed under the Apache License, Version 2.0 (the License);you may not use this file except in compliance with the License.You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an AS IS BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License. See accompanying LICENSE file.
--
configuration!-- Site specific YARN configuration properties --propertynameyarn.nodemanager.aux-services/namevaluemapreduce_shuffle/value
/property
!-- 支持来自服务器外部的访问 --
propertynameyarn.resourcemanager.hostname/namevalue0.0.0.0/value
/propertypropertynameyarn.resourcemanager.webapp.address/namevalue0.0.0.0:8088/value
/propertypropertynameyarn.log.server.url/namevaluehttp://0.0.0.0:19888/jobhistory/logs/value/propertypropertynameyarn.application.classpath/namevalue/opt/module/hadoop/etc/hadoop:/opt/module/hadoop/share/hadoop/common/lib/*:/opt/module/hadoop/share/hadoop/common/*:/opt/module/hadoop/share/hadoop/hdfs:/opt/module/hadoop/share/hadoop/hdfs/lib/*:/opt/module/hadoop/share/hadoop/hdfs/*:/opt/module/hadoop/share/hadoop/mapreduce/*:/opt/module/hadoop/share/hadoop/yarn:/opt/module/hadoop/share/hadoop/yarn/lib/*:/opt/module/hadoop/share/hadoop/yarn/*/value/propertypropertynameyarn.log-aggregation-enable/namevaluetrue/value/propertypropertynameyarn.log-aggregation.retain-seconds/namevalue604800/value/propertyproperty nameyarn.nodemanager.resource.memory-mb/name value8192/value/propertyproperty nameyarn.scheduler.minimum-allocation-mb/namevalue512/value/propertypropertynameyarn.scheduler.maximum-allocation-mb/namevalue8192/value/propertypropertynameyarn.nodemanager.pmem-check-enabled/namevaluefalse/value/propertypropertynameyarn.nodemanager.vmem-check-enabled/namevaluefalse/value/propertypropertynameyarn.nodemanager.vmem-pmem-ratio/namevalue2.1/value/property
propertynameyarn.nodemanager.env-whitelist/namevalueJAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME/value
/property/configuration4、格式化
bin/hdfs namenode -format
5、启动
./sbin/start-dfs.sh
./sbin/start-yarn.sh6、检查是否启动成功
# 查看是否有相应的进程
jps19494 DataNode
66060 ResourceManager
19631 SecondaryNameNode
19390 NameNode
如果没有启动成功可以打开日志目录按照时间倒序排序 查看最近的日志查看异常情况
日志地址/opt/module/hadoop/logs
7、启动成功后可以打开管理界面
http://127.0.0.1:9870/dfshealth.html#tab-overview yarn控制台
