江苏建设会计学会网站,泰安房产网签西湖春晓,美橙极速建站系统,青岛网站推广公司SparkML SparkML_lr_train #xff1a;读取py处理后的train表用于训练#xff0c;将训练模型保存好。 SparkML_lr_predict #xff1a;读取训练好的模型#xff0c;读取py处理后的test表用于预测。将预测结果写入normal_data中#xff0c;根据id修改stream_is_normal的值。… SparkML SparkML_lr_train 读取py处理后的train表用于训练将训练模型保存好。 SparkML_lr_predict 读取训练好的模型读取py处理后的test表用于预测。将预测结果写入normal_data中根据id修改stream_is_normal的值。
提交spark任务
bin/spark-submit \
--class SparkML_lr_train \
--master yarn \
--deploy-mode cluster \
./SparkML_lr_train1.jar \
10bin/spark-submit \
--class SparkML_lr_train \
--master yarn \
--deploy-mode client \
./SparkML_lr_train4.jar \
10bin/spark-submit \
--class SparkML_lr_predict \
--master yarn \
--deploy-mode client \
./SparkML_lr_predict.jar \
10bin/spark-submit \
--class lr_train\
--master yarn \
--deploy-mode client \
./lr_train.jar \
10bin/spark-submit \
--class lr_predict\
--master yarn \
--deploy-mode client \
./lr_predict.jar \
10
启动hadoop(启动脚本) hdp.sh start 启动spark(命令行启动) sbin/start-all.sh
bin/spark-submit –class SparkSQL_lr_train –master yarn –deploy-mode client ./SparkSQL_lr_train.jar 10
bin/spark-submit –class lr_train –master yarn –deploy-mode client ./lr_train.jar 10
