网页制作模板的淘宝网站代码,自己的商品链接怎么弄,信用网站一体化建设方案,佛山市招投标交易中心基于hive分析Flask为后端框架echarts为前端框架的招聘网站可视化大屏项目
1. 项目概述
项目目标是构建一个大数据分析系统#xff0c;包含以下核心模块#xff1a; 1、数据爬取#xff1a;通过request请求获取猎聘网的就业数据。 2、数据存储和分析#xff1a;使用 Hive …基于hive分析Flask为后端框架echarts为前端框架的招聘网站可视化大屏项目
1. 项目概述
项目目标是构建一个大数据分析系统包含以下核心模块 1、数据爬取通过request请求获取猎聘网的就业数据。 2、数据存储和分析使用 Hive 进行数据存储和分析。 3、数据迁移使用sqoop将hive数据导入mysql。 4、后端服务使用 Flask 搭建数据接口将分析结果提供给前端。 5、数据可视化使用 ECharts 制作大屏展示实现数据的图形化呈现。
2. 项目环境准备
在开始之前需要搭建如下环境
Hive作为数据仓库用于存储和分析数据。 Flask轻量级 Python Web 框架用于构建后端 RESTful API。 EChartsJavaScript 图表库用于前端数据可视化。 MySQL用于保存一些系统配置或小规模数据。 Sqoop数据同步工具将hive数据同步到mysql。
3、数据爬取
通过python获取猎聘网的照片信息存储到csv文件里
import csv
import timeimport requests
import execjsfrom storage.csv2mysql import sync_data2dbf open(../storage/data.csv, modea, encodingutf-8)
csv_writer csv.DictWriter(f,fieldnames[职位,城市,薪资,经验,标签,公司,公司领域,公司规模])
csv_writer.writeheader()def read_js_code():f open(/Users/shareit/workspace/chart_show/demo.js,encodingutf-8)txt f.read()js_code execjs.compile(txt)ckId js_code.call(r,32)return ckIddef post_data():read_js_code()url https://api-c.liepin.com/api/com.liepin.searchfront4c.pc-search-jobheaders {Accept: application/json, text/plain, */*,Accept-Encoding: gzip, deflate, br,Accept-Language: zh-CN,zh;q0.9,Connection: keep-alive,Sec-Ch-Ua-Platform:macOS,Content-Length: 398,Content-Type: application/json;charsetUTF-8;,Host: api-c.liepin.com,User-Agent:Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36,Origin: https://www.liepin.com,Referer: https://www.liepin.com/,Sec-Ch-Ua: Google Chrome;v119, Chromium;v119, Not?A_Brand;v24,Sec-Ch-Ua-Mobile: ?0,Sec-Fetch-Dest: empty,Sec-Fetch-Mode: cors,Sec-Fetch-Site: same-site,X-Client-Type: web,X-Fscp-Bi-Stat: {location: https://www.liepin.com/zhaopin},X-Fscp-Fe-Version: ,X-Fscp-Std-Info: {client_id: 40108},X-Fscp-Trace-Id: 52262313-e6ca-4cfd-bb67-41b4a32b8bb5,X-Fscp-Version: 1.1,X-Requested-With: XMLHttpRequest,}list [H01$H0001,H01$H0002,H01$H0003,H01$H0004,H01$H0005,H01$H0006,H01$H0007,H01$H0008,H01$H0009,H01$H00010,H02$H0018,H02$H0019,H03$H0022,H03$H0023,H03$H0024,H03$H0025,H04$H0030,H04$H0031,H04$H0032,H05$H05,H06$H06,H07$H07,H08$H08]for name in list:print(-------{}---------.format(name))for i in range(10):print(------------第{}页-----------.format(i))data {data: {mainSearchPcConditionForm:{city: 410, dq: 410, pubTime: , currentPage: i, pageSize: 40, key: ,suggestTag: , workYearCode: 1, compId: , compName: , compTag: ,industry: name, salary: , jobKind: , compScale: , compKind: , compStage: ,eduLevel: },passThroughForm:{scene: page, skId: z33lm3jhwza7k1xjvcyn8lb8e9ghxx1b,fkId: z33lm3jhwza7k1xjvcyn8lb8e9ghxx1b,ckId: read_js_code(),sfrom: search_job_pc}}}response requests.post(urlurl, jsondata, headersheaders)time.sleep(2)parse_data(response)def parse_data(response):try:jobCardList response.json()[data][data][jobCardList]except Exception as e:return4、加载hive数据进行分析
1、将storage下的data.csv上传到虚拟机上 2、创建work_base表并将data.csv数据加载到hive表里
CREATE TABLE work_base (id INT COMMENT id,title STRING COMMENT 标题,city STRING COMMENT 城市,salary STRING COMMENT 薪资,campus_job_kind STRING COMMENT 经验,labels STRING COMMENT 标签,compName STRING COMMENT 公司,compIndustry STRING COMMENT 公司领域,compScale STRING COMMENT 公司规模
) ROW FORMAT DELIMITED FIELDS TERMINATED BY , STORED AS TEXTFILE;LOAD local DATA INPATH ./data.csv OVERWRITE INTO TABLE flask_work.work_base;3、创建hive ads层数仓表进行分析 -- 4. 热门公司分析
CREATE TABLE top_companies (company_name STRING COMMENT 公司名称,job_count INT COMMENT 职位数量
) STORED AS TEXTFILE;INSERT INTO top_companies
SELECT compName, COUNT(*) AS job_count
FROM work_base
GROUP BY compName
ORDER BY job_count DESC
LIMIT 10;-- 5. 岗位分布情况分析
CREATE TABLE job_distribution (job_title STRING COMMENT 岗位名称,job_count INT COMMENT 职位数量
) STORED AS TEXTFILE;INSERT INTO job_distribution
SELECT title, COUNT(*) AS job_count
FROM work_base
GROUP BY title;-- 6. 学历要求分析
CREATE TABLE education_requirements (education_level STRING COMMENT 学历要求,job_count INT COMMENT 职位数量
) STORED AS TEXTFILE;INSERT INTO education_requirements
SELECTCASEWHEN labels LIKE %博士% THEN 博士WHEN labels LIKE %硕士% THEN 硕士WHEN labels LIKE %本科% THEN 本科WHEN labels LIKE %大专% THEN 大专ELSE 其他END AS education_level,COUNT(*) AS job_count
FROM work_base
GROUP BY education_level;-- 7. 薪资待遇分析各个城市的平均薪资
CREATE TABLE city_salary_analysis (city STRING COMMENT 城市,avg_salary DOUBLE COMMENT 平均薪资
) STORED AS TEXTFILE;INSERT INTO city_salary_analysis
SELECT city, AVG(CAST(salary AS DOUBLE)) AS avg_salary
FROM work_base
WHERE salary RLIKE ^[0-9]$
GROUP BY city;
5、将hive分析的结果数据导入mysql
使用sqoop迁移数据
sqoop export \--connect jdbc:mysql://localhost:3306/flask_work \--username root --password 123456 \--table city_job_count \--export-dir /hive/warehouse/flask_work.db/flask_work.city_job_count \--input-fields-terminated-by \001 \--input-lines-terminated-by \n;sqoop export \--connect jdbc:mysql:// localhost:3306/flask_work \--username root --password 123456 \--table job_salary_analysis \--export-dir /user/hive/warehouse/flask_work.db/flask_work.job_salary_analysis \--input-fields-terminated-by \001 \--input-lines-terminated-by \nsqoop export \--connect jdbc:mysql:// localhost:3306/flask_work \--username root --password 123456 \--table top_companies \--export-dir /user/hive/warehouse/flask_work.db/flask_work.top_companies \--input-fields-terminated-by \001 \--input-lines-terminated-by \nsqoop export \--connect jdbc:mysql:// localhost:3306/flask_work \--username root --password 123456 \--table job_distribution \--export-dir /user/hive/warehouse/flask_work.db/flask_work.job_distribution \--input-fields-terminated-by \001 \--input-lines-terminated-by \n6. 后端服务Flask
使用 Flask 构建后端服务编写rest api读取mysql数据提供给前端页面进行展示 app.py
from flask import Flask, render_template, request, flash, redirect, url_for
from data import *
from service.task_service import get_user, register_userapp Flask(__name__)
app.secret_key b6b52fae-5618-4805-b368-501c62c6d1dfapp.after_request
def add_header(response):response.cache_control.max_age 0return responseapp.route(/, methods[GET, POST])
def login():if request.method POST:username request.form[username]password request.form[password]user get_user(username, password)# 检查用户是否存在if user is not None:data SourceData()return render_template(index.html, formdata, titledata.title)else:# 用户名或密码错误显示错误消息flash(用户名或密码错误)return redirect(url_for(login)) # 重定向回登录页面# 如果是 GET 请求则直接返回登录页面return render_template(login.html)app.route(/register, methods[GET, POST])
def register():if request.method POST:username request.form.get(username)password request.form.get(password)if username and password:register_user(username, password) # 确保此函数已定义return 注册成功flash(用户名和密码不能为空)return redirect(url_for(register))return render_template(register.html)if __name__ __main__:app.run(host127.0.0.1, debugFalse)task_service.py
import pymysqldb_config {host: 127.0.0.1,user: root,password: 12345678,database: flask_work,charset: utf8mb4,cursorclass: pymysql.cursors.DictCursor
}
connection pymysql.connect(**db_config)def get_user(username,password):try:with connection.cursor() as cursor:select_query select * from user where username %s and password %scursor.execute(select_query,(username,password))result cursor.fetchall()return result[0]except Exception as e:print(e)return Nonedef get_title_count():try:with connection.cursor() as cursor:select_query select count(distinct(city)) city,count(distinct(compName)) compName from work_base;cursor.execute(select_query)result cursor.fetchall()aresult[0][city]bresult[0][compName]return a,bexcept Exception as e:print(e)return Nonedef work_count_by_city():try:with connection.cursor() as cursor:select_query select city,job_count from city_job_count order by job_count desc limit 10cursor.execute(select_query)result cursor.fetchall()re_list []for re in result:re_list.append({name: re[city], value: re[job_count]})print(re_list)return re_listexcept Exception as e:print(e)return Nonedef work_avg_salary():try:with connection.cursor() as cursor:select_query select job_title,avg_salary from job_salary_analysis order by avg_salary desc limit 10;cursor.execute(select_query)result cursor.fetchall()re_list []for re in result:re_list.append({name: re[job_title][0:8], value: int(re[avg_salary])})print(re_list)return re_listexcept Exception as e:print(e)return Nonedef top_companies():try:with connection.cursor() as cursor:select_query select company_name,job_count from top_companies limit 3;cursor.execute(select_query)result cursor.fetchall()re_list []for re in result:re_list.append({name: re[company_name], value: re[job_count]})print(re_list)return re_listexcept Exception as e:print(e)return Nonedef job_distribution_count():try:with connection.cursor() as cursor:select_query select job_title,job_count from job_distribution order by job_count desc limit 10;cursor.execute(select_query)result cursor.fetchall()re_list []for re in result:re_list.append({name: re[job_title][0:6], value: re[job_count],value2: 20, color: 01, radius: [59%, 70%]})print(re_list)return re_listexcept Exception as e:print(e)return Nonedef register_user(username,password):try:with connection.cursor() as cursor:select_query insert into user(username,password) values(%s,%s)cursor.execute(select_query,(username,password))connection.commit()except Exception as e:print(e)return Nonedef education_requirements():try:with connection.cursor() as cursor:select_query select education_level,job_count from education_requirements;cursor.execute(select_query)result cursor.fetchall()re_list []for re in result:re_list.append({name: re[education_level], value: int(re[job_count])})return re_listexcept Exception as e:print(e)return Nonedef city_salary_analysis():try:with connection.cursor() as cursor:select_query select city,avg_salary from city_salary_analysis order by avg_salary desc limit 10;cursor.execute(select_query)result cursor.fetchall()re_list []for re in result:re_list.append({name: re[city], value: int(re[avg_salary])})return re_listexcept Exception as e:print(e)return Noneif __name__ __main__:print(city_salary_analysis())
7 页面设计
前端采用 ECharts 制作一个招聘网站大数据分析的可视化大屏。 使用 ECharts 渲染数据
!doctype html
html
headmeta charsetutf-8titleindex/titlescript typetext/javascript src../static/js/jquery.js/scriptscript typetext/javascript src../static/js/echarts.min.js/scriptscript typetext/javascript src../static/js/china.js/scriptlink relstylesheet href../static/css/comon0.css
/head
script$(window).load(function(){$(.loading).fadeOut()})/****/
$(document).ready(function(){var whei$(window).width()$(html).css({fontSize:whei/20})$(window).resize(function(){var whei$(window).width()$(html).css({fontSize:whei/20})
});});/script
script typetext/javascript src../static/js/echarts.min.js/script
script typetext/javascript src../static/js/china.js/scriptbody
div classcanvas styleopacity: .2iframe frameborder0 src../static/js/index.html stylewidth: 100%; height: 100%/iframe
/div
div classloadingdiv classloadboximg src../static/picture/loading.gif 页面加载中.../div
/div
div classheadh1{{title}}/h1div classweather!-- img srcpicture/weather.pngspan多云转小雨/span--span idshowTime/span/divscript
var t null;t setTimeout(time,1000);//開始运行function time(){clearTimeout(t);//清除定时器dt new Date();var ydt.getFullYear();var mtdt.getMonth()1;var daydt.getDate();var hdt.getHours();//获取时var mdt.getMinutes();//获取分var sdt.getSeconds();//获取秒document.getElementById(showTime).innerHTML y年mt月day日-h时m分s秒;t setTimeout(time,1000); //设定定时器循环运行}/script/div
div classmainboxul classclearfixlidiv classboxall styleheight: 3.2remdiv classalltitle{{form.echart1.title}}/divdiv classallnav idechart1/divdiv classboxfoot/div/divdiv classboxall styleheight: 3.2remdiv classalltitle{{form.echart2.title}}/divdiv classallnav idechart2/divdiv classboxfoot/div/divdiv classboxall styleheight: 3.2remdiv styleheight:100%; width: 100%;div classalltitle{{form.echart3.title}}/divdiv classallnav idechart3/div/divdiv classboxfoot/div/div/lilidiv classbardiv classbarboxul classclearfixli classpulll_left counter{{form.counter.value}}/lili classpulll_left counter{{form.counter2.value}}/li/ul/divdiv classbarbox2ul classclearfixli classpulll_left{{form.counter.name}}/lili classpulll_left{{form.counter2.name}}/li/ul/div/divdiv classmapdiv classmap1img src../static/picture/lbx.png/divdiv classmap2img src../static/picture/jt.png/divdiv classmap3img src../static/picture/map.png/divdiv classmap4 idmap_1/div/div/lilidiv classboxall styleheight:3.4remdiv classalltitle{{form.echart4.title}}/divdiv classallnav idechart4/divdiv classboxfoot/div/divdiv classboxall styleheight: 3.2remdiv classalltitle{{form.echart5.title}}/divdiv classallnav idechart5/divdiv classboxfoot/div/divdiv classboxall styleheight: 3remdiv classalltitle{{form.echart6.title}}/divdiv classallnav idechart6/divdiv classboxfoot/div/div/li/ul
/div
div classback/div!--echart1--
script
$(function echarts_1() {// 基于准备好的dom初始化echarts实例var myChart echarts.init(document.getElementById(echart1));option {// backgroundColor: #00265f,tooltip: {trigger: axis,axisPointer: {type: shadow}},grid: {left: 0%,top:10px,right: 0%,bottom: 4%,containLabel: true},xAxis: [{type: category,data: {{form.echart1.xAxis|safe}},axisLine: {show: true,lineStyle: {color: rgba(255,255,255,.1),width: 1,type: solid},},axisTick: {show: false,},axisLabel: {interval: 0,// rotate:50,show: true,splitNumber: 15,textStyle: {color: rgba(255,255,255,.6),fontSize: 12,},},}],yAxis: [{type: value,axisLabel: {//formatter: {value} %show:true,textStyle: {color: rgba(255,255,255,.6),fontSize: 12,},},axisTick: {show: false,},axisLine: {show: true,lineStyle: {color: rgba(255,255,255,.1 ),width: 1,type: solid},},splitLine: {lineStyle: {color: rgba(255,255,255,.1),}}}],series: [{type: bar,data: {{form.echart1.series|safe}},barWidth:35%, //柱子宽度// barGap: 1, //柱子之间间距itemStyle: {normal: {color:#2f89cf,opacity: 1,barBorderRadius: 5,}}}]
};// 使用刚指定的配置项和数据显示图表。myChart.setOption(option);window.addEventListener(resize,function(){myChart.resize();});})/script
!--echart2--
script$(function echarts_2() {// 基于准备好的dom初始化echarts实例var myChart echarts.init(document.getElementById(echart2));option {tooltip: {trigger: item // 修改为 item 以适应散点图},grid: {left: 0%,top: 10px,right: 0%,bottom: 4%,containLabel: true},xAxis: [{type: category,data: {{form.echart2.xAxis|safe}},axisLine: {show: true,lineStyle: {color: rgba(255,255,255,.1),width: 1,type: solid},},axisTick: {show: false,},axisLabel: {interval: 0,rotate: 45, // 将标签旋转90度show: true,splitNumber: 15,textStyle: {color: rgba(255,255,255,.6),fontSize: 8,},},}],yAxis: [{type: value,axisLabel: {show: true,textStyle: {color: rgba(255,255,255,.6),fontSize: 12,},},axisTick: {show: false,},axisLine: {show: true,lineStyle: {color: rgba(255,255,255,.1),width: 1,type: solid},},splitLine: {lineStyle: {color: rgba(255,255,255,.1),}}}],series: [{type: scatter, // 将类型更改为 scatterdata: {{form.echart2.series|safe}},symbolSize: 10, // 设置散点的大小itemStyle: {normal: {color: #27d08a,opacity: 1,}}}]};// 使用刚指定的配置项和数据显示图表。myChart.setOption(option);window.addEventListener(resize, function() {myChart.resize();});})
/script!--echart3--
script$(function echarts_3() {// 基于准备好的dom初始化echarts实例var myChart echarts.init(document.getElementById(echart3));// 配置项var option {tooltip: {trigger: item,formatter: {a} br/{b}: {c} ({d}%) // 在悬停提示框中显示公司名称、数值和百分比},legend: {top: 5%,left: center,textStyle: {color: rgba(255,255,255,.6)}},series: [{name: 公司数据, // 扇形图系列名称type: pie,radius: [40%, 70%], // 内外半径形成环形图avoidLabelOverlap: false,itemStyle: {borderRadius: 10,borderColor: #fff,borderWidth: 2},labelLine: {show: true,length: 6,length2: 8,lineStyle: {color: rgba(255,255,255,.6)}},data:{{form.echart3.data|safe}},}]};// 使用刚指定的配置项和数据显示图表。myChart.setOption(option);window.addEventListener(resize, function() {myChart.resize();});});
/script!--echarts4--
script$(function echarts_4() {// 基于准备好的dom初始化echarts实例var myChart echarts.init(document.getElementById(echart4));option {tooltip: {trigger: item // 修改为 item 以适应散点图},grid: {left: 0%,top: 10px,right: 0%,bottom: 4%,containLabel: true},xAxis: [{type: category,data: {{form.echart4.xAxis|safe}},axisLine: {show: true,lineStyle: {color: rgba(255,255,255,.1),width: 1,type: solid},},axisTick: {show: false,},axisLabel: {interval: 0,rotate: 45, // 将标签旋转90度show: true,splitNumber: 15,textStyle: {color: rgba(255,255,255,.6),fontSize: 8,},},}],yAxis: [{type: value,axisLabel: {show: true,textStyle: {color: rgba(255,255,255,.6),fontSize: 12,},},axisTick: {show: false,},axisLine: {show: true,lineStyle: {color: rgba(255,255,255,.1),width: 1,type: solid},},splitLine: {lineStyle: {color: rgba(255,255,255,.1),}}}],series: [{type: scatter, // 将类型更改为 scatterdata: {{form.echart4.series|safe}},symbolSize: 10, // 设置散点的大小itemStyle: {normal: {color: #27d08a,opacity: 1,}}}]};// 使用刚指定的配置项和数据显示图表。myChart.setOption(option);window.addEventListener(resize, function() {myChart.resize();});})
/script!--echarts5--
script
$(function echarts_5() {// 基于准备好的dom初始化echarts实例var myChart echarts.init(document.getElementById(echart5));option {tooltip: {trigger: axis,axisPointer: {type: line // 修改为line以匹配折线图的样式}},grid: {left: 0%,top: 10px,right: 0%,bottom: 2%,containLabel: true},xAxis: [{type: category,data: {{form.echart5.xAxis|safe}},axisLine: {show: true,lineStyle: {color: rgba(255,255,255,.1),width: 1,type: solid},},axisTick: {show: false,},axisLabel: {rotate: 90,interval: 0,show: true,splitNumber: 15,textStyle: {color: rgba(255,255,255,.6),fontSize: 12,},},}],yAxis: [{type: value,axisLabel: {show: true,textStyle: {color: rgba(255,255,255,.6),fontSize: 12,},},axisTick: {show: false,},axisLine: {show: true,lineStyle: {color: rgba(255,255,255,.1),width: 1,type: solid},},splitLine: {lineStyle: {color: rgba(255,255,255,.1),}}}],series: [{type: line, // 修改为line类型data: {{form.echart5.series|safe}},smooth: true, // 可选让折线平滑itemStyle: {color: #2f89cf,opacity: 1,},lineStyle: {width: 2, // 线条宽度type: solid // 线条类型}}]};// 使用刚指定的配置项和数据显示图表。myChart.setOption(option);window.addEventListener(resize, function() {myChart.resize();});
})
/script!--echarts6--
script$(function echarts_6() {var myChart echarts.init(document.getElementById(echart6));var option {tooltip: {},radar: {indicator: {{form.echart6.data|safe}}},series: [{type: radar,data: [{value: [5500, 7000, 11064, 9500, 7469, 6250, 0, 7500, 6250, 7000],name: 城市指标}]}]};myChart.setOption(option);window.addEventListener(resize, function() {myChart.resize();});});
/script!--map_1--
script
$(function map() {// 基于准备好的dom初始化echarts实例var myChart echarts.init(document.getElementById(map_1));
var data {{form.map_1.data|safe}};
var geoCoordMap {海门:[121.15,31.89],鄂尔多斯:[109.781327,39.608266],招远:[120.38,37.35],舟山:[122.207216,29.985295],齐齐哈尔:[123.97,47.33],盐城:[120.13,33.38],赤峰:[118.87,42.28],青岛:[120.33,36.07],乳山:[121.52,36.89],金昌:[102.188043,38.520089],泉州:[118.58,24.93],莱西:[120.53,36.86],日照:[119.46,35.42],胶南:[119.97,35.88],南通:[121.05,32.08],拉萨:[91.11,29.97],云浮:[112.02,22.93],梅州:[116.1,24.55],文登:[122.05,37.2],上海:[121.48,31.22],攀枝花:[101.718637,26.582347],威海:[122.1,37.5],承德:[117.93,40.97],厦门:[118.1,24.46],汕尾:[115.375279,22.786211],潮州:[116.63,23.68],丹东:[124.37,40.13],太仓:[121.1,31.45],曲靖:[103.79,25.51],烟台:[121.39,37.52],福州:[119.3,26.08],瓦房店:[121.979603,39.627114],即墨:[120.45,36.38],抚顺:[123.97,41.97],玉溪:[102.52,24.35],张家口:[114.87,40.82],阳泉:[113.57,37.85],莱州:[119.942327,37.177017],湖州:[120.1,30.86],汕头:[116.69,23.39],昆山:[120.95,31.39],宁波:[121.56,29.86],湛江:[110.359377,21.270708],揭阳:[116.35,23.55],荣成:[122.41,37.16],连云港:[119.16,34.59],葫芦岛:[120.836932,40.711052],常熟:[120.74,31.64],东莞:[113.75,23.04],河源:[114.68,23.73],淮安:[119.15,33.5],泰州:[119.9,32.49],南宁:[108.33,22.84],营口:[122.18,40.65],惠州:[114.4,23.09],江阴:[120.26,31.91],蓬莱:[120.75,37.8],韶关:[113.62,24.84],嘉峪关:[98.289152,39.77313],广州:[113.23,23.16],延安:[109.47,36.6],太原:[112.53,37.87],清远:[113.01,23.7],中山:[113.38,22.52],昆明:[102.73,25.04],寿光:[118.73,36.86],盘锦:[122.070714,41.119997],长治:[113.08,36.18],深圳:[114.07,22.62],珠海:[113.52,22.3],宿迁:[118.3,33.96],咸阳:[108.72,34.36],铜川:[109.11,35.09],平度:[119.97,36.77],佛山:[113.11,23.05],海口:[110.35,20.02],江门:[113.06,22.61],章丘:[117.53,36.72],肇庆:[112.44,23.05],大连:[121.62,38.92],临汾:[111.5,36.08],吴江:[120.63,31.16],石嘴山:[106.39,39.04],沈阳:[123.38,41.8],苏州:[120.62,31.32],茂名:[110.88,21.68],嘉兴:[120.76,30.77],长春:[125.35,43.88],胶州:[120.03336,36.264622],银川:[106.27,38.47],张家港:[120.555821,31.875428],三门峡:[111.19,34.76],锦州:[121.15,41.13],南昌:[115.89,28.68],柳州:[109.4,24.33],三亚:[109.511909,18.252847],自贡:[104.778442,29.33903],吉林:[126.57,43.87],阳江:[111.95,21.85],泸州:[105.39,28.91],西宁:[101.74,36.56],宜宾:[104.56,29.77],呼和浩特:[111.65,40.82],成都:[104.06,30.67],大同:[113.3,40.12],镇江:[119.44,32.2],桂林:[110.28,25.29],张家界:[110.479191,29.117096],宜兴:[119.82,31.36],北海:[109.12,21.49],西安:[108.95,34.27],金坛:[119.56,31.74],东营:[118.49,37.46],牡丹江:[129.58,44.6],遵义:[106.9,27.7],绍兴:[120.58,30.01],扬州:[119.42,32.39],常州:[119.95,31.79],潍坊:[119.1,36.62],重庆:[106.54,29.59],台州:[121.420757,28.656386],南京:[118.78,32.04],滨州:[118.03,37.36],贵阳:[106.71,26.57],无锡:[120.29,31.59],本溪:[123.73,41.3],克拉玛依:[84.77,45.59],渭南:[109.5,34.52],马鞍山:[118.48,31.56],宝鸡:[107.15,34.38],焦作:[113.21,35.24],句容:[119.16,31.95],北京:[116.46,39.92],徐州:[117.2,34.26],衡水:[115.72,37.72],包头:[110,40.58],绵阳:[104.73,31.48],乌鲁木齐:[87.68,43.77],枣庄:[117.57,34.86],杭州:[120.19,30.26],淄博:[118.05,36.78],鞍山:[122.85,41.12],溧阳:[119.48,31.43],库尔勒:[86.06,41.68],安阳:[114.35,36.1],开封:[114.35,34.79],济南:[117,36.65],德阳:[104.37,31.13],温州:[120.65,28.01],九江:[115.97,29.71],邯郸:[114.47,36.6],临安:[119.72,30.23],兰州:[103.73,36.03],沧州:[116.83,38.33],临沂:[118.35,35.05],南充:[106.110698,30.837793],天津:[117.2,39.13],富阳:[119.95,30.07],泰安:[117.13,36.18],诸暨:[120.23,29.71],郑州:[113.65,34.76],哈尔滨:[126.63,45.75],聊城:[115.97,36.45],芜湖:[118.38,31.33],唐山:[118.02,39.63],平顶山:[113.29,33.75],邢台:[114.48,37.05],德州:[116.29,37.45],济宁:[116.59,35.38],荆州:[112.239741,30.335165],宜昌:[111.3,30.7],义乌:[120.06,29.32],丽水:[119.92,28.45],洛阳:[112.44,34.7],秦皇岛:[119.57,39.95],株洲:[113.16,27.83],石家庄:[114.48,38.03],莱芜:[117.67,36.19],常德:[111.69,29.05],保定:[115.48,38.85],湘潭:[112.91,27.87],金华:[119.64,29.12],岳阳:[113.09,29.37],长沙:[113,28.21],衢州:[118.88,28.97],廊坊:[116.7,39.53],菏泽:[115.480656,35.23375],合肥:[117.27,31.86],武汉:[114.31,30.52],大庆:[125.03,46.58]
};
var convertData function (data) {var res [];for (var i 0; i data.length; i) {var geoCoord geoCoordMap[data[i].name];if (geoCoord) {res.push({name: data[i].name,value: geoCoord.concat(data[i].value)});}}return res;
};option {tooltip : {trigger: item,formatter: function (params) {if(typeof(params.value)[2] undefined){return params.name : params.value;}else{return params.name : params.value[2];}}},geo: {map: china,label: {emphasis: {show: false}},roam: false,//禁止其放大缩小itemStyle: {normal: {areaColor: #4c60ff,borderColor: #002097},emphasis: {areaColor: #293fff}}},series : [{name: 消费金额,type: scatter,coordinateSystem: geo,data: convertData(data),symbolSize: function (val) {return val[2] / {{form.map_1.symbolSize}};},label: {normal: {formatter: {b},position: right,show: false},emphasis: {show: true}},itemStyle: {normal: {color: #ffeb7b}}}]
};myChart.setOption(option);window.addEventListener(resize,function(){myChart.resize();});}
)/script/body
/html 8、总结
通过 request爬虫获取数据使用Hive 进行大数据分析、通过sqoop进行数据迁移最后使用Flask 构建后端接口获取mysql数据提供给前端再结合 ECharts 前端可视化能够构建一个完整的大数据展示系统。
如有遇到问题可以找小编沟通交流哦。另外小编帮忙辅导大课作业学生毕设等。不限于MapReduce MySQL, pythonjava大数据模型训练等。 hadoop hdfs yarn spark Django flask flink kafka flume datax sqoop seatunnel echart可视化 机器学习等
