个人建网站成本,国外可以做网站盈利模式有哪些,交友软件,文化传媒公司名字大全文章目录 一、解压压缩包二、修改配置文件conf/spark-env.sh三、测试提交Spark任务四、Spark on Hive配置4.1 创建hive-site.xml#xff08;spark/conf目录#xff09;4.2 查看hive的hive-site.xml配置与3.1配置的是否一致4.3 测试SparkSQL4.3.1 启动SparkSQL客户端#xff… 文章目录 一、解压压缩包二、修改配置文件conf/spark-env.sh三、测试提交Spark任务四、Spark on Hive配置4.1 创建hive-site.xmlspark/conf目录4.2 查看hive的hive-site.xml配置与3.1配置的是否一致4.3 测试SparkSQL4.3.1 启动SparkSQL客户端Yarn方式4.3.2 启动Hive客户端 五、通过Spark Web-UI分析SQL执行过程TODO六、集群化TODO 一、解压压缩包
[hadoophadoop102 software]$ tar -zxvf spark-3.0.0-bin-hadoop3.2.tgz -C /opt/module二、修改配置文件conf/spark-env.sh
cd /opt/module/spark-3.0.0-bin-hadoop3.2/conf
[hadoophadoop102 conf]$ cp spark-env.sh.template spark-env.sh
[hadoophadoop102 conf]$ vim spark-env.sh内容
export JAVA_HOME/opt/module/jdk1.8.0_291
YARN_CONF_DIR/opt/module/hadoop-3.1.3/etc/hadoop三、测试提交Spark任务
[hadoophadoop102 ~]$ cd /opt/module/spark-3.0.0-bin-hadoop3.2/
[hadoophadoop102 spark-3.0.0-bin-hadoop3.2]$ bin/spark-submit \--class org.apache.spark.examples.SparkPi \--master yarn \--deploy-mode cluster \./examples/jars/spark-examples_2.12-3.0.0.jar \10到YARN WEB页面查看任务提交情况
四、Spark on Hive配置
4.1 创建hive-site.xmlspark/conf目录
[hadoophadoop102 conf]$ cd /opt/module/spark-3.0.0-bin-hadoop3.2/conf/
[hadoophadoop102 conf]$ vim hive-site.xml内容
?xml version1.0?
?xml-stylesheet typetext/xsl hrefconfiguration.xsl?
configuration!--告知Spark创建表存到哪里--propertynamehive.metastore.warehouse.dir/namevalue/warehouse/value/property!-- 不使用spark内置hive存储元数据 --propertynamehive.metastore.local/namevaluefalse/value/property!--告知Spark Hive的MetaStore在哪--propertynamehive.metastore.uris/namevaluethrift://hadoop102:9083/value/property/configuration4.2 查看hive的hive-site.xml配置与3.1配置的是否一致
cd /opt/module/apache-hive-3.1.2-bin/conf
vim hive-site.xml4.3 测试SparkSQL
4.3.1 启动SparkSQL客户端Yarn方式
[hadoophadoop102 spark-3.0.0-bin-hadoop3.2]$ bin/spark-sql --master yarnspark-sql show databases;
spark-sql select count(1) from dw_ods.ods_activity_info_full where dt2023-12-07;4.3.2 启动Hive客户端
[hadoophadoop102 apache-hive-3.1.2-bin]$ bin/hivehive show databases;
hive select count(1) from dw_ods.ods_activity_info_full where dt2023-12-07;五、通过Spark Web-UI分析SQL执行过程TODO 六、集群化TODO 优势在哪里
