基于hive分析Flask为后端框架echarts为前端框架的招聘网站可视化大屏项目

news2024/11/22 16:12:49

基于hive分析Flask为后端框架echarts为前端框架的招聘网站可视化大屏项目

1. 项目概述

项目目标是构建一个大数据分析系统,包含以下核心模块:
1、数据爬取:通过request请求获取猎聘网的就业数据。
2、数据存储和分析:使用 Hive 进行数据存储和分析。
3、数据迁移:使用sqoop将hive数据导入mysql。
4、后端服务:使用 Flask 搭建数据接口,将分析结果提供给前端。
5、数据可视化:使用 ECharts 制作大屏展示,实现数据的图形化呈现。

2. 项目环境准备

在开始之前,需要搭建如下环境:

Hive:作为数据仓库,用于存储和分析数据。
Flask:轻量级 Python Web 框架,用于构建后端 RESTful API。
ECharts:JavaScript 图表库,用于前端数据可视化。
MySQL:用于保存一些系统配置或小规模数据。
Sqoop:数据同步工具,将hive数据同步到mysql。

3、数据爬取

通过python获取猎聘网的照片信息,存储到csv文件里

import csv
import time

import requests
import execjs

from storage.csv2mysql import sync_data2db


f = open('../storage/data.csv', mode='a', encoding='utf-8')
csv_writer = csv.DictWriter(f,fieldnames=[
        '职位',
        '城市',
        '薪资',
        '经验',
        '标签',
        '公司',
        '公司领域',
        '公司规模'])
csv_writer.writeheader()

def read_js_code():
    f= open('/Users/shareit/workspace/chart_show/demo.js',encoding='utf-8')
    txt = f.read()
    js_code = execjs.compile(txt)
    ckId = js_code.call('r',32)
    return ckId
    
    

def post_data():
    read_js_code()
    url = "https://api-c.liepin.com/api/com.liepin.searchfront4c.pc-search-job"
    headers = {
        'Accept': 'application/json, text/plain, */*',
        'Accept-Encoding': 'gzip, deflate, br',
        'Accept-Language': 'zh-CN,zh;q=0.9',
        'Connection': 'keep-alive',
        'Sec-Ch-Ua-Platform':'macOS',
        'Content-Length': '398',
        'Content-Type': 'application/json;charset=UTF-8;',
        'Host': 'api-c.liepin.com',
        'User-Agent':'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36',
        'Origin': 'https://www.liepin.com',
        'Referer': 'https://www.liepin.com/',
        'Sec-Ch-Ua': '"Google Chrome";v="119", "Chromium";v="119", "Not?A_Brand";v="24"',
        'Sec-Ch-Ua-Mobile': '?0',
        'Sec-Fetch-Dest': 'empty',
        'Sec-Fetch-Mode': 'cors',
        'Sec-Fetch-Site': 'same-site',
        'X-Client-Type': 'web',
        'X-Fscp-Bi-Stat': '{"location": "https://www.liepin.com/zhaopin"}',
        'X-Fscp-Fe-Version': '',
        'X-Fscp-Std-Info': '{"client_id": "40108"}',
        'X-Fscp-Trace-Id': '52262313-e6ca-4cfd-bb67-41b4a32b8bb5',
        'X-Fscp-Version': '1.1',
        'X-Requested-With': 'XMLHttpRequest',
    }
    list = ["H01$H0001","H01$H0002",
            "H01$H0003","H01$H0004","H01$H0005",
            "H01$H0006","H01$H0007","H01$H0008",
            "H01$H0009","H01$H00010","H02$H0018","H02$H0019","H03$H0022",
            "H03$H0023","H03$H0024","H03$H0025","H04$H0030","H04$H0031",
            "H04$H0032","H05$H05","H06$H06","H07$H07","H08$H08"]
    for name in list:
        print("-------{}---------".format(name))
        for i in range(10):
            print("------------第{}页-----------".format(i))
            data = {"data": {"mainSearchPcConditionForm":
                                 {"city": "410", "dq": "410", "pubTime": "", "currentPage": i, "pageSize": 40, "key": "",
                                  "suggestTag": "", "workYearCode": "1", "compId": "", "compName": "", "compTag": "",
                                  "industry": name, "salary": "", "jobKind": "", "compScale": "", "compKind": "", "compStage": "",
                                  "eduLevel": ""},
                             "passThroughForm":
                                 {"scene": "page", "skId": "z33lm3jhwza7k1xjvcyn8lb8e9ghxx1b",
                                  "fkId": "z33lm3jhwza7k1xjvcyn8lb8e9ghxx1b",
                                  "ckId": read_js_code(),
                                  'sfrom': 'search_job_pc'}}}
            response = requests.post(url=url, json=data, headers=headers)
            time.sleep(2)
            parse_data(response)


def parse_data(response):
    try:
        jobCardList = response.json()['data']['data']['jobCardList']
    except Exception as e:
        return

4、加载hive数据进行分析

1、将storage下的data.csv上传到虚拟机上
2、创建work_base表,并将data.csv数据加载到hive表里

CREATE TABLE work_base (
  id INT COMMENT 'id',
  title STRING COMMENT '标题',
  city STRING COMMENT '城市',
  salary STRING COMMENT '薪资',
  campus_job_kind STRING COMMENT '经验',
  labels STRING COMMENT '标签',
  compName STRING COMMENT '公司',
  compIndustry STRING COMMENT '公司领域',
  compScale STRING COMMENT '公司规模'
) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' STORED AS TEXTFILE;


LOAD local DATA INPATH './data.csv' OVERWRITE INTO TABLE flask_work.work_base;

3、创建hive ads层数仓表进行分析


-- 4. 热门公司分析
CREATE TABLE top_companies (
  company_name STRING COMMENT '公司名称',
  job_count INT COMMENT '职位数量'
) STORED AS TEXTFILE;

INSERT INTO top_companies
SELECT compName, COUNT(*) AS job_count
FROM work_base
GROUP BY compName
ORDER BY job_count DESC
LIMIT 10;

-- 5. 岗位分布情况分析
CREATE TABLE job_distribution (
  job_title STRING COMMENT '岗位名称',
  job_count INT COMMENT '职位数量'
) STORED AS TEXTFILE;

INSERT INTO job_distribution
SELECT title, COUNT(*) AS job_count
FROM work_base
GROUP BY title;

-- 6. 学历要求分析
CREATE TABLE education_requirements (
  education_level STRING COMMENT '学历要求',
  job_count INT COMMENT '职位数量'
) STORED AS TEXTFILE;

INSERT INTO education_requirements
SELECT
  CASE
    WHEN labels LIKE '%博士%' THEN '博士'
    WHEN labels LIKE '%硕士%' THEN '硕士'
    WHEN labels LIKE '%本科%' THEN '本科'
    WHEN labels LIKE '%大专%' THEN '大专'
    ELSE '其他'
  END AS education_level,
  COUNT(*) AS job_count
FROM work_base
GROUP BY education_level;

-- 7. 薪资待遇分析(各个城市的平均薪资)
CREATE TABLE city_salary_analysis (
  city STRING COMMENT '城市',
  avg_salary DOUBLE COMMENT '平均薪资'
) STORED AS TEXTFILE;

INSERT INTO city_salary_analysis
SELECT city, AVG(CAST(salary AS DOUBLE)) AS avg_salary
FROM work_base
WHERE salary RLIKE '^[0-9]+$'
GROUP BY city;

5、将hive分析的结果数据导入mysql

使用sqoop迁移数据

sqoop export \
  --connect jdbc:mysql://localhost:3306/flask_work \
  --username root --password '123456' \
  --table city_job_count \
  --export-dir /hive/warehouse/flask_work.db/flask_work.city_job_count \
  --input-fields-terminated-by '\001' \
  --input-lines-terminated-by '\n';

sqoop export \
  --connect jdbc:mysql:// localhost:3306/flask_work \
  --username root --password 123456 \
  --table job_salary_analysis \
  --export-dir /user/hive/warehouse/flask_work.db/flask_work.job_salary_analysis \
  --input-fields-terminated-by '\001' \
  --input-lines-terminated-by '\n'

sqoop export \
  --connect jdbc:mysql:// localhost:3306/flask_work \
  --username root --password 123456 \
  --table top_companies \
  --export-dir /user/hive/warehouse/flask_work.db/flask_work.top_companies \
  --input-fields-terminated-by '\001' \
  --input-lines-terminated-by '\n'

sqoop export \
  --connect jdbc:mysql:// localhost:3306/flask_work \
  --username root --password 123456 \
  --table job_distribution \
  --export-dir /user/hive/warehouse/flask_work.db/flask_work.job_distribution \
  --input-fields-terminated-by '\001' \
  --input-lines-terminated-by '\n'

6. 后端服务(Flask)

使用 Flask 构建后端服务,编写rest api,读取mysql数据提供给前端页面进行展示
app.py

from flask import Flask, render_template, request, flash, redirect, url_for
from data import *
from service.task_service import get_user, register_user

app = Flask(__name__)
app.secret_key = 'b6b52fae-5618-4805-b368-501c62c6d1df'


@app.after_request
def add_header(response):
    response.cache_control.max_age = 0
    return response

@app.route('/', methods=['GET', 'POST'])
def login():
    if request.method == 'POST':
        username = request.form['username']
        password = request.form['password']
        user = get_user(username, password)

        # 检查用户是否存在
        if user is not None:
            data = SourceData()
            return render_template('index.html', form=data, title=data.title)
        else:
            # 用户名或密码错误,显示错误消息
            flash('用户名或密码错误')
            return redirect(url_for('login'))  # 重定向回登录页面

    # 如果是 GET 请求,则直接返回登录页面
    return render_template('login.html')

@app.route('/register', methods=['GET', 'POST'])
def register():
    if request.method == 'POST':
        username = request.form.get('username')
        password = request.form.get('password')
        if username and password:
            register_user(username, password)  # 确保此函数已定义
            return "注册成功!"
        flash('用户名和密码不能为空')
        return redirect(url_for('register'))
    return render_template('register.html')

if __name__ == "__main__":
    app.run(host='127.0.0.1', debug=False)

task_service.py

import pymysql

db_config = {
    'host': '127.0.0.1',
    'user': 'root',
    'password': '12345678',
    'database': 'flask_work',
    'charset': 'utf8mb4',
    'cursorclass': pymysql.cursors.DictCursor
}
connection = pymysql.connect(**db_config)


def get_user(username,password):
    try:
        with connection.cursor() as cursor:
            select_query = "select * from user where username = %s and password = %s"
            cursor.execute(select_query,(username,password))
            result = cursor.fetchall()
            return result[0]
    except Exception as e:
        print(e)
    return None


def get_title_count():
    try:
        with connection.cursor() as cursor:
            select_query = "select count(distinct(city)) city,count(distinct(compName)) compName from work_base;"
            cursor.execute(select_query)
            result = cursor.fetchall()
            a=result[0]['city']
            b=result[0]['compName']
            return a,b
    except Exception as e:
        print(e)
    return None


def work_count_by_city():
    try:
        with connection.cursor() as cursor:
            select_query = "select city,job_count from city_job_count order by job_count desc limit 10"
            cursor.execute(select_query)
            result = cursor.fetchall()
            re_list = []
            for re in result:
                re_list.append({"name": re['city'], "value": re['job_count']})
            print(re_list)
            return re_list
    except Exception as e:
        print(e)
    return None


def work_avg_salary():
    try:
        with connection.cursor() as cursor:
            select_query = "select job_title,avg_salary from job_salary_analysis order by avg_salary desc limit 10;"
            cursor.execute(select_query)
            result = cursor.fetchall()
            re_list = []
            for re in result:
                re_list.append({"name": re['job_title'][0:8], "value": int(re['avg_salary'])})
            print(re_list)
            return re_list
    except Exception as e:
        print(e)
    return None


def top_companies():
    try:
        with connection.cursor() as cursor:
            select_query = "select company_name,job_count from top_companies limit 3;"
            cursor.execute(select_query)
            result = cursor.fetchall()
            re_list = []
            for re in result:
                re_list.append({"name": re['company_name'], "value": re['job_count']})
            print(re_list)
            return re_list
    except Exception as e:
        print(e)
    return None

def job_distribution_count():
    try:
        with connection.cursor() as cursor:
            select_query = "select job_title,job_count from job_distribution order by job_count desc limit 10;"
            cursor.execute(select_query)
            result = cursor.fetchall()
            re_list = []
            for re in result:
                re_list.append({"name": re['job_title'][0:6], "value": re['job_count'],"value2": 20, "color": "01", "radius": ['59%', '70%']})
            print(re_list)
            return re_list
    except Exception as e:
        print(e)
    return None

def register_user(username,password):
    try:
        with connection.cursor() as cursor:
            select_query = "insert into user(username,password) values(%s,%s)"
            cursor.execute(select_query,(username,password))
        connection.commit()
    except Exception as e:
        print(e)
    return None

def education_requirements():
    try:
        with connection.cursor() as cursor:
            select_query = "select education_level,job_count from education_requirements;"
            cursor.execute(select_query)
            result = cursor.fetchall()
            re_list = []
            for re in result:
                re_list.append({"name": re['education_level'], "value": int(re['job_count'])})
            return re_list
    except Exception as e:
        print(e)
    return None


def city_salary_analysis():
    try:
        with connection.cursor() as cursor:
            select_query = "select city,avg_salary from city_salary_analysis order by avg_salary desc limit 10;"
            cursor.execute(select_query)
            result = cursor.fetchall()
            re_list = []
            for re in result:
                re_list.append({"name": re['city'], "value": int(re['avg_salary'])})
            return re_list
    except Exception as e:
        print(e)
    return None


if __name__ == '__main__':
    print(city_salary_analysis())

7 页面设计

前端采用 ECharts 制作一个招聘网站大数据分析的可视化大屏。
使用 ECharts 渲染数据

<!doctype html>
<html>
<head>
    <meta charset="utf-8">
    <title>index</title>
    <script type="text/javascript" src="../static/js/jquery.js"></script>
    <script type="text/javascript" src="../static/js/echarts.min.js"></script>
    <script type="text/javascript" src="../static/js/china.js"></script>
    <link rel="stylesheet" href="../static/css/comon0.css">
</head>
<script>
	$(window).load(function(){
             $(".loading").fadeOut()
            })

/****/
$(document).ready(function(){
	var whei=$(window).width()
	$("html").css({fontSize:whei/20})
	$(window).resize(function(){
		var whei=$(window).width()
	 $("html").css({fontSize:whei/20})
});
	});






</script>
<script type="text/javascript" src="../static/js/echarts.min.js"></script>
<script type="text/javascript" src="../static/js/china.js"></script>

<body>
<div class="canvas" style="opacity: .2">
    <iframe frameborder="0" src="../static/js/index.html" style="width: 100%; height: 100%"></iframe>
</div>
<div class="loading">
    <div class="loadbox"><img src="../static/picture/loading.gif"> 页面加载中...</div>
</div>
<div class="head">
    <h1>{{title}}</h1>
    <div class="weather">
        <!--        <img src="picture/weather.png"><span>多云转小雨</span>-->
        <span id="showTime"></span>
    </div>

    <script>
var t = null;
    t = setTimeout(time,1000);//開始运行
    function time()
    {
       clearTimeout(t);//清除定时器
       dt = new Date();
		var y=dt.getFullYear();
		var mt=dt.getMonth()+1;
		var day=dt.getDate();
       var h=dt.getHours();//获取时
       var m=dt.getMinutes();//获取分
       var s=dt.getSeconds();//获取秒
       document.getElementById("showTime").innerHTML = y+"年"+mt+"月"+day+"日"+"-"+h+"时"+m+"分"+s+"秒";
       t = setTimeout(time,1000); //设定定时器,循环运行
    }



    </script>


</div>
<div class="mainbox">
    <ul class="clearfix">
        <li>
            <div class="boxall" style="height: 3.2rem">
                <div class="alltitle">{{form.echart1.title}}</div>
                <div class="allnav" id="echart1"></div>
                <div class="boxfoot"></div>
            </div>
            <div class="boxall" style="height: 3.2rem">
                <div class="alltitle">{{form.echart2.title}}</div>
                <div class="allnav" id="echart2"></div>
                <div class="boxfoot"></div>
            </div>
            <div class="boxall" style="height: 3.2rem">
                <div style="height:100%; width: 100%;">
                    <div class="alltitle">{{form.echart3.title}}</div>
                    <div class="allnav" id="echart3"></div>
                </div>
                <div class="boxfoot">

                </div>
            </div>
        </li>
        <li>
            <div class="bar">
                <div class="barbox">
                    <ul class="clearfix">
                        <li class="pulll_left counter">{{form.counter.value}}</li>
                        <li class="pulll_left counter">{{form.counter2.value}}</li>
                    </ul>
                </div>
                <div class="barbox2">
                    <ul class="clearfix">
                        <li class="pulll_left">{{form.counter.name}}</li>
                        <li class="pulll_left">{{form.counter2.name}}</li>
                    </ul>
                </div>
            </div>
            <div class="map">
                <div class="map1"><img src="../static/picture/lbx.png"></div>
                <div class="map2"><img src="../static/picture/jt.png"></div>
                <div class="map3"><img src="../static/picture/map.png"></div>
                <div class="map4" id="map_1"></div>
            </div>
        </li>
        <li>
            <div class="boxall" style="height:3.4rem">
                <div class="alltitle">{{form.echart4.title}}</div>
                <div class="allnav" id="echart4"></div>
                <div class="boxfoot"></div>
            </div>
            <div class="boxall" style="height: 3.2rem">
                <div class="alltitle">{{form.echart5.title}}</div>
                <div class="allnav" id="echart5"></div>
                <div class="boxfoot"></div>
            </div>
            <div class="boxall" style="height: 3rem">
                <div class="alltitle">{{form.echart6.title}}</div>
                <div class="allnav" id="echart6"></div>
                <div class="boxfoot"></div>
            </div>
        </li>
    </ul>
</div>
<div class="back"></div>

<!--echart1-->
<script>
$(function echarts_1() {
        // 基于准备好的dom,初始化echarts实例
        var myChart = echarts.init(document.getElementById('echart1'));

       option = {
  //  backgroundColor: '#00265f',
    tooltip: {
        trigger: 'axis',
        axisPointer: {
            type: 'shadow'
        }
    },
    grid: {
        left: '0%',
		top:'10px',
        right: '0%',
        bottom: '4%',
       containLabel: true
    },
    xAxis: [{
        type: 'category',
      		data: {{form.echart1.xAxis|safe}},
        axisLine: {
            show: true,
         lineStyle: {
                color: "rgba(255,255,255,.1)",
                width: 1,
                type: "solid"
            },
        },

        axisTick: {
            show: false,
        },
		axisLabel:  {
                interval: 0,
               // rotate:50,
                show: true,
                splitNumber: 15,
                textStyle: {
 					color: "rgba(255,255,255,.6)",
                    fontSize: '12',
                },
            },
    }],
    yAxis: [{
        type: 'value',
        axisLabel: {
           //formatter: '{value} %'
			show:true,
			 textStyle: {
 					color: "rgba(255,255,255,.6)",
                    fontSize: '12',
                },
        },
        axisTick: {
            show: false,
        },
        axisLine: {
            show: true,
            lineStyle: {
                color: "rgba(255,255,255,.1	)",
                width: 1,
                type: "solid"
            },
        },
        splitLine: {
            lineStyle: {
               color: "rgba(255,255,255,.1)",
            }
        }
    }],
    series: [
		{
        type: 'bar',
        data: {{form.echart1.series|safe}},
        barWidth:'35%', //柱子宽度
       // barGap: 1, //柱子之间间距
        itemStyle: {
            normal: {
                color:'#2f89cf',
                opacity: 1,
				barBorderRadius: 5,
            }
        }
    }

	]
};

        // 使用刚指定的配置项和数据显示图表。
        myChart.setOption(option);
        window.addEventListener("resize",function(){
            myChart.resize();
        });
    })

</script>
<!--echart2-->
<script>
  $(function echarts_2() {
        // 基于准备好的dom,初始化echarts实例
        var myChart = echarts.init(document.getElementById('echart2'));

        option = {
            tooltip: {
                trigger: 'item' // 修改为 'item' 以适应散点图
            },
            grid: {
                left: '0%',
                top: '10px',
                right: '0%',
                bottom: '4%',
                containLabel: true
            },
            xAxis: [{
                type: 'category',
                data: {{form.echart2.xAxis|safe}},
                axisLine: {
                    show: true,
                    lineStyle: {
                        color: "rgba(255,255,255,.1)",
                        width: 1,
                        type: "solid"
                    },
                },
                axisTick: {
                    show: false,
                },
                axisLabel: {
                    interval: 0,
                    rotate: 45, // 将标签旋转90度
                    show: true,
                    splitNumber: 15,
                    textStyle: {
                        color: "rgba(255,255,255,.6)",
                        fontSize: '8',
                    },
                },
            }],
            yAxis: [{
                type: 'value',
                axisLabel: {
                    show: true,
                    textStyle: {
                        color: "rgba(255,255,255,.6)",
                        fontSize: '12',
                    },
                },
                axisTick: {
                    show: false,
                },
                axisLine: {
                    show: true,
                    lineStyle: {
                        color: "rgba(255,255,255,.1)",
                        width: 1,
                        type: "solid"
                    },
                },
                splitLine: {
                    lineStyle: {
                        color: "rgba(255,255,255,.1)",
                    }
                }
            }],
            series: [{
                type: 'scatter', // 将类型更改为 'scatter'
                data: {{form.echart2.series|safe}},
                symbolSize: 10, // 设置散点的大小
                itemStyle: {
                    normal: {
                        color: '#27d08a',
                        opacity: 1,
                    }
                }
            }]
        };

        // 使用刚指定的配置项和数据显示图表。
        myChart.setOption(option);
        window.addEventListener("resize", function() {
            myChart.resize();
        });
    })
</script>


<!--echart3-->
<script>
  $(function echarts_3() {
        // 基于准备好的dom,初始化echarts实例
        var myChart = echarts.init(document.getElementById('echart3'));

        // 配置项
        var option = {
            tooltip: {
                trigger: 'item',
                formatter: '{a} <br/>{b}: {c} ({d}%)'  // 在悬停提示框中显示公司名称、数值和百分比
            },
            legend: {
                top: '5%',
                left: 'center',
                textStyle: {
                    color: "rgba(255,255,255,.6)"
                }
            },
            series: [
                {
                    name: '公司数据',  // 扇形图系列名称
                    type: 'pie',
                    radius: ['40%', '70%'],  // 内外半径,形成环形图
                    avoidLabelOverlap: false,
                    itemStyle: {
                        borderRadius: 10,
                        borderColor: '#fff',
                        borderWidth: 2
                    },
                    labelLine: {
                        show: true,
                        length: 6,
                        length2: 8,
                        lineStyle: {
                            color: "rgba(255,255,255,.6)"
                        }
                    },
                    data:{{form.echart3.data|safe}},
                }
            ]
        };

        // 使用刚指定的配置项和数据显示图表。
        myChart.setOption(option);
        window.addEventListener("resize", function() {
            myChart.resize();
        });
    });
</script>




<!--echarts4-->
<script>
  $(function echarts_4() {
        // 基于准备好的dom,初始化echarts实例
        var myChart = echarts.init(document.getElementById('echart4'));

        option = {
            tooltip: {
                trigger: 'item' // 修改为 'item' 以适应散点图
            },
            grid: {
                left: '0%',
                top: '10px',
                right: '0%',
                bottom: '4%',
                containLabel: true
            },
            xAxis: [{
                type: 'category',
                data: {{form.echart4.xAxis|safe}},
                axisLine: {
                    show: true,
                    lineStyle: {
                        color: "rgba(255,255,255,.1)",
                        width: 1,
                        type: "solid"
                    },
                },
                axisTick: {
                    show: false,
                },
                axisLabel: {
                    interval: 0,
                    rotate: 45, // 将标签旋转90度
                    show: true,
                    splitNumber: 15,
                    textStyle: {
                        color: "rgba(255,255,255,.6)",
                        fontSize: '8',
                    },
                },
            }],
            yAxis: [{
                type: 'value',
                axisLabel: {
                    show: true,
                    textStyle: {
                        color: "rgba(255,255,255,.6)",
                        fontSize: '12',
                    },
                },
                axisTick: {
                    show: false,
                },
                axisLine: {
                    show: true,
                    lineStyle: {
                        color: "rgba(255,255,255,.1)",
                        width: 1,
                        type: "solid"
                    },
                },
                splitLine: {
                    lineStyle: {
                        color: "rgba(255,255,255,.1)",
                    }
                }
            }],
            series: [{
                type: 'scatter', // 将类型更改为 'scatter'
                data: {{form.echart4.series|safe}},
                symbolSize: 10, // 设置散点的大小
                itemStyle: {
                    normal: {
                        color: '#27d08a',
                        opacity: 1,
                    }
                }
            }]
        };

        // 使用刚指定的配置项和数据显示图表。
        myChart.setOption(option);
        window.addEventListener("resize", function() {
            myChart.resize();
        });
    })
</script>






<!--echarts5-->
<script>
$(function echarts_5() {
    // 基于准备好的dom,初始化echarts实例
    var myChart = echarts.init(document.getElementById('echart5'));

    option = {
        tooltip: {
            trigger: 'axis',
            axisPointer: {
                type: 'line' // 修改为'line'以匹配折线图的样式
            }
        },
        grid: {
            left: '0%',
            top: '10px',
            right: '0%',
            bottom: '2%',
            containLabel: true
        },
        xAxis: [{
            type: 'category',
            data: {{form.echart5.xAxis|safe}},
            axisLine: {
                show: true,
                lineStyle: {
                    color: "rgba(255,255,255,.1)",
                    width: 1,
                    type: "solid"
                },
            },
            axisTick: {
                show: false,
            },
            axisLabel: {
                rotate: 90,
                interval: 0,
                show: true,
                splitNumber: 15,
                textStyle: {
                    color: "rgba(255,255,255,.6)",
                    fontSize: '12',
                },
            },
        }],
        yAxis: [{
            type: 'value',
            axisLabel: {
                show: true,
                textStyle: {
                    color: "rgba(255,255,255,.6)",
                    fontSize: '12',
                },
            },
            axisTick: {
                show: false,
            },
            axisLine: {
                show: true,
                lineStyle: {
                    color: "rgba(255,255,255,.1)",
                    width: 1,
                    type: "solid"
                },
            },
            splitLine: {
                lineStyle: {
                    color: "rgba(255,255,255,.1)",
                }
            }
        }],
        series: [{
            type: 'line', // 修改为'line'类型
            data: {{form.echart5.series|safe}},
            smooth: true, // 可选:让折线平滑
            itemStyle: {
                color: '#2f89cf',
                opacity: 1,
            },
            lineStyle: {
                width: 2, // 线条宽度
                type: 'solid' // 线条类型
            }
        }]
    };

    // 使用刚指定的配置项和数据显示图表。
    myChart.setOption(option);
    window.addEventListener("resize", function() {
        myChart.resize();
    });
})
</script>

<!--echarts6-->
<script>
  $(function echarts_6() {
      var myChart = echarts.init(document.getElementById('echart6'));

      var option = {
          tooltip: {},
          radar: {
              indicator: {{form.echart6.data|safe}}
          },
          series: [{
              type: 'radar',
              data: [
                  {
                      value: [5500, 7000, 11064, 9500, 7469, 6250, 0, 7500, 6250, 7000],
                      name: '城市指标'
                  }
              ]
          }]
      };

      myChart.setOption(option);
      window.addEventListener("resize", function() {
          myChart.resize();
      });
  });
</script>






<!--map_1-->
<script>
$(function map() {
        // 基于准备好的dom,初始化echarts实例
        var myChart = echarts.init(document.getElementById('map_1'));
var data = {{form.map_1.data|safe}};
var geoCoordMap = {
    '海门':[121.15,31.89],
    '鄂尔多斯':[109.781327,39.608266],
    '招远':[120.38,37.35],
    '舟山':[122.207216,29.985295],
    '齐齐哈尔':[123.97,47.33],
    '盐城':[120.13,33.38],
    '赤峰':[118.87,42.28],
    '青岛':[120.33,36.07],
    '乳山':[121.52,36.89],
    '金昌':[102.188043,38.520089],
    '泉州':[118.58,24.93],
    '莱西':[120.53,36.86],
    '日照':[119.46,35.42],
    '胶南':[119.97,35.88],
    '南通':[121.05,32.08],
    '拉萨':[91.11,29.97],
    '云浮':[112.02,22.93],
    '梅州':[116.1,24.55],
    '文登':[122.05,37.2],
    '上海':[121.48,31.22],
    '攀枝花':[101.718637,26.582347],
    '威海':[122.1,37.5],
    '承德':[117.93,40.97],
    '厦门':[118.1,24.46],
    '汕尾':[115.375279,22.786211],
    '潮州':[116.63,23.68],
    '丹东':[124.37,40.13],
    '太仓':[121.1,31.45],
    '曲靖':[103.79,25.51],
    '烟台':[121.39,37.52],
    '福州':[119.3,26.08],
    '瓦房店':[121.979603,39.627114],
    '即墨':[120.45,36.38],
    '抚顺':[123.97,41.97],
    '玉溪':[102.52,24.35],
    '张家口':[114.87,40.82],
    '阳泉':[113.57,37.85],
    '莱州':[119.942327,37.177017],
    '湖州':[120.1,30.86],
    '汕头':[116.69,23.39],
    '昆山':[120.95,31.39],
    '宁波':[121.56,29.86],
    '湛江':[110.359377,21.270708],
    '揭阳':[116.35,23.55],
    '荣成':[122.41,37.16],
    '连云港':[119.16,34.59],
    '葫芦岛':[120.836932,40.711052],
    '常熟':[120.74,31.64],
    '东莞':[113.75,23.04],
    '河源':[114.68,23.73],
    '淮安':[119.15,33.5],
    '泰州':[119.9,32.49],
    '南宁':[108.33,22.84],
    '营口':[122.18,40.65],
    '惠州':[114.4,23.09],
    '江阴':[120.26,31.91],
    '蓬莱':[120.75,37.8],
    '韶关':[113.62,24.84],
    '嘉峪关':[98.289152,39.77313],
    '广州':[113.23,23.16],
    '延安':[109.47,36.6],
    '太原':[112.53,37.87],
    '清远':[113.01,23.7],
    '中山':[113.38,22.52],
    '昆明':[102.73,25.04],
    '寿光':[118.73,36.86],
    '盘锦':[122.070714,41.119997],
    '长治':[113.08,36.18],
    '深圳':[114.07,22.62],
    '珠海':[113.52,22.3],
    '宿迁':[118.3,33.96],
    '咸阳':[108.72,34.36],
    '铜川':[109.11,35.09],
    '平度':[119.97,36.77],
    '佛山':[113.11,23.05],
    '海口':[110.35,20.02],
    '江门':[113.06,22.61],
    '章丘':[117.53,36.72],
    '肇庆':[112.44,23.05],
    '大连':[121.62,38.92],
    '临汾':[111.5,36.08],
    '吴江':[120.63,31.16],
    '石嘴山':[106.39,39.04],
    '沈阳':[123.38,41.8],
    '苏州':[120.62,31.32],
    '茂名':[110.88,21.68],
    '嘉兴':[120.76,30.77],
    '长春':[125.35,43.88],
    '胶州':[120.03336,36.264622],
    '银川':[106.27,38.47],
    '张家港':[120.555821,31.875428],
    '三门峡':[111.19,34.76],
    '锦州':[121.15,41.13],
    '南昌':[115.89,28.68],
    '柳州':[109.4,24.33],
    '三亚':[109.511909,18.252847],
    '自贡':[104.778442,29.33903],
    '吉林':[126.57,43.87],
    '阳江':[111.95,21.85],
    '泸州':[105.39,28.91],
    '西宁':[101.74,36.56],
    '宜宾':[104.56,29.77],
    '呼和浩特':[111.65,40.82],
    '成都':[104.06,30.67],
    '大同':[113.3,40.12],
    '镇江':[119.44,32.2],
    '桂林':[110.28,25.29],
    '张家界':[110.479191,29.117096],
    '宜兴':[119.82,31.36],
    '北海':[109.12,21.49],
    '西安':[108.95,34.27],
    '金坛':[119.56,31.74],
    '东营':[118.49,37.46],
    '牡丹江':[129.58,44.6],
    '遵义':[106.9,27.7],
    '绍兴':[120.58,30.01],
    '扬州':[119.42,32.39],
    '常州':[119.95,31.79],
    '潍坊':[119.1,36.62],
    '重庆':[106.54,29.59],
    '台州':[121.420757,28.656386],
    '南京':[118.78,32.04],
    '滨州':[118.03,37.36],
    '贵阳':[106.71,26.57],
    '无锡':[120.29,31.59],
    '本溪':[123.73,41.3],
    '克拉玛依':[84.77,45.59],
    '渭南':[109.5,34.52],
    '马鞍山':[118.48,31.56],
    '宝鸡':[107.15,34.38],
    '焦作':[113.21,35.24],
    '句容':[119.16,31.95],
    '北京':[116.46,39.92],
    '徐州':[117.2,34.26],
    '衡水':[115.72,37.72],
    '包头':[110,40.58],
    '绵阳':[104.73,31.48],
    '乌鲁木齐':[87.68,43.77],
    '枣庄':[117.57,34.86],
    '杭州':[120.19,30.26],
    '淄博':[118.05,36.78],
    '鞍山':[122.85,41.12],
    '溧阳':[119.48,31.43],
    '库尔勒':[86.06,41.68],
    '安阳':[114.35,36.1],
    '开封':[114.35,34.79],
    '济南':[117,36.65],
    '德阳':[104.37,31.13],
    '温州':[120.65,28.01],
    '九江':[115.97,29.71],
    '邯郸':[114.47,36.6],
    '临安':[119.72,30.23],
    '兰州':[103.73,36.03],
    '沧州':[116.83,38.33],
    '临沂':[118.35,35.05],
    '南充':[106.110698,30.837793],
    '天津':[117.2,39.13],
    '富阳':[119.95,30.07],
    '泰安':[117.13,36.18],
    '诸暨':[120.23,29.71],
    '郑州':[113.65,34.76],
    '哈尔滨':[126.63,45.75],
    '聊城':[115.97,36.45],
    '芜湖':[118.38,31.33],
    '唐山':[118.02,39.63],
    '平顶山':[113.29,33.75],
    '邢台':[114.48,37.05],
    '德州':[116.29,37.45],
    '济宁':[116.59,35.38],
    '荆州':[112.239741,30.335165],
    '宜昌':[111.3,30.7],
    '义乌':[120.06,29.32],
    '丽水':[119.92,28.45],
    '洛阳':[112.44,34.7],
    '秦皇岛':[119.57,39.95],
    '株洲':[113.16,27.83],
    '石家庄':[114.48,38.03],
    '莱芜':[117.67,36.19],
    '常德':[111.69,29.05],
    '保定':[115.48,38.85],
    '湘潭':[112.91,27.87],
    '金华':[119.64,29.12],
    '岳阳':[113.09,29.37],
    '长沙':[113,28.21],
    '衢州':[118.88,28.97],
    '廊坊':[116.7,39.53],
    '菏泽':[115.480656,35.23375],
    '合肥':[117.27,31.86],
    '武汉':[114.31,30.52],
    '大庆':[125.03,46.58]
};
var convertData = function (data) {
    var res = [];
    for (var i = 0; i < data.length; i++) {
        var geoCoord = geoCoordMap[data[i].name];
        if (geoCoord) {
            res.push({
                name: data[i].name,
                value: geoCoord.concat(data[i].value)
            });
        }
    }
    return res;
};

option = {
    tooltip : {
        trigger: 'item',
		formatter: function (params) {
              if(typeof(params.value)[2] == "undefined"){
              	return params.name + ' : ' + params.value;
              }else{
              	return params.name + ' : ' + params.value[2];
              }
            }
    },

    geo: {
        map: 'china',
        label: {
            emphasis: {
                show: false
            }
        },
        roam: false,//禁止其放大缩小
        itemStyle: {
            normal: {
                areaColor: '#4c60ff',
                borderColor: '#002097'
            },
            emphasis: {
                areaColor: '#293fff'
            }
        }
    },
    series : [
        {
            name: '消费金额',
            type: 'scatter',
            coordinateSystem: 'geo',
            data: convertData(data),
            symbolSize: function (val) {
                return val[2] / {{form.map_1.symbolSize}};
            },
            label: {
                normal: {
                    formatter: '{b}',
                    position: 'right',
                    show: false
                },
                emphasis: {
                    show: true
                }
            },
            itemStyle: {
                normal: {
                    color: '#ffeb7b'
                }
            }
        }
    ]
};

        myChart.setOption(option);
        window.addEventListener("resize",function(){
            myChart.resize();
        });
    }
)




</script>

</body>
</html>

在这里插入图片描述
在这里插入图片描述
在这里插入图片描述

8、总结

通过 request爬虫获取数据,使用Hive 进行大数据分析、通过sqoop进行数据迁移,最后使用Flask 构建后端接口获取mysql数据提供给前端,再结合 ECharts 前端可视化,能够构建一个完整的大数据展示系统。

如有遇到问题可以找小编沟通交流哦。另外小编帮忙辅导大课作业,学生毕设等。不限于MapReduce, MySQL, python,java,大数据,模型训练等。 hadoop hdfs yarn spark Django flask flink kafka flume datax sqoop seatunnel echart可视化 机器学习等
在这里插入图片描述

本文来自互联网用户投稿,该文观点仅代表作者本人,不代表本站立场。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如若转载,请注明出处:http://www.coloradmin.cn/o/2230188.html

如若内容造成侵权/违法违规/事实不符,请联系多彩编程网进行投诉反馈,一经查实,立即删除!

相关文章

Rust 力扣 - 2841. 几乎唯一子数组的最大和

文章目录 题目描述题解思路题解代码题目链接 题目描述 题解思路 我们遍历长度为k的窗口&#xff0c;用一个哈希表记录窗口内的所有元素&#xff08;用来对窗口内元素去重&#xff09;&#xff0c;我们取哈希表中元素数量大于等于m的窗口总和的最大值 题解代码 use std::coll…

Blender进阶:贴图与UV

9 UV 9.1 贴图与UV UV&#xff0c;指定每个面顶点在贴图上的坐标 演示&#xff1a; 1、添加物体 2、添加贴图&#xff0c;即图片纹理节点 3、进入UV Edit工作区 4、右边&#xff0c;选择一个面 5、左边&#xff0c;选择一个面&#xff0c;移动这个面 9.2 电子表格 电子…

UiPath调用Python脚本的完整示例

一、主要步骤&#xff1a; 1、创建Python脚本 2、安装UiPath.Python.Activities库 3、使用方法&#xff1a; a、添加python作用域 b、加载python脚本 c、调用python方法 d、获取python对象 e、显示Python结果的消息对话框 二、详细步骤 1、安装UiPath.Python.Activities库 …

[pdf,epub]105页《分析模式》漫谈合集01

105页的《分析模式》漫谈合集第1集的pdf、epub文件&#xff0c;已上传到本账号的CSDN资源。 如果无法下载&#xff0c;也可以访问umlchina.com/url/ap.html 已排版成适合手机阅读&#xff0c;pdf的排版更好一些。 ★UMLChina为什么叒要翻译《分析模式》&#xff1f; ★[缝合故…

科技资讯|谷歌Play应用商店有望支持 XR 头显,AR / VR设备有望得到发展

据 Android Authority 报道&#xff0c;谷歌似乎正在为其 Play 商店增加对 XR 头显的支持。该媒体在 Play 商店的代码中发现了相关的线索&#xff0c;包括一个代表头显的小图标以及对“XR 头显”的提及。 谷歌也可能改变了此前拒绝将 Play 商店引入 Meta Quest 头显的决定。今…

ES跟Kafka集成

配合流程 1. Kafka作为分布式流处理平台&#xff0c;能够实时收集和处理不同数据源的数据流&#xff1b; 2. 通过Kafka Connect或者Logstash等中间件&#xff0c;可以将Kafka中的数据流实时推送到Elasticsearch中&#xff1b; 3. Elasticsearch接收到数据后&#xff0c;会根据…

查缺补漏----关于计组两道题辨析

答案&#xff1a;A 指令字由操作码、寻址特征和地址码三个字段组成&#xff0c;寻址特征字段用来指明指令属于哪种寻址方式。若寻址方式是寄存器直接寻址&#xff0c;则地址码所指的通用寄存器中存放的是操作数&#xff0c;若寻址方式是寄存器间接寻址&#xff0c;则对应通用寄…

uniapp学习(010-2 实现抖音小程序上线)

零基础入门uniapp Vue3组合式API版本到咸虾米壁纸项目实战&#xff0c;开发打包微信小程序、抖音小程序、H5、安卓APP客户端等 总时长 23:40:00 共116P 此文章包含第113p的内容 文章目录 抖音小程序下载抖音开发者工具先去开发者工具里进行测试 抖音开放平台配置开始打包上传…

[Approaching any Machine Learning] Supervised vs unsupervised learning - Note

Page 11 single_image pixel_values[1, :].reshape(28, 28) plt.imshow(single_image, cmapgray)用这个代码运行是会出错的&#xff0c;应该改为以下代码&#xff1a; pixel_np pixel_values.iloc[1, :].values single_image pixel_np.reshape(28, 28) plt.imshow(single_…

buu easyRE

这道题目我想写的东西不是很多&#xff0c;前面的部分按常规流程走&#xff0c;第一步我们写逆脚本&#xff0c;推算出数组v15的值&#xff0c;但是输出值却没有啥用&#xff0c;只是告诉我们&#xff0c;the first parts are flag &#xff0c;没多大用&#xff0c;然后后…

qt QProgressBar详解

1、概述 QProgressBar是Qt框架中的一个控件&#xff0c;专门用于显示任务的进度。它提供了一个可视化的进度条&#xff0c;让用户能够直观地了解任务的完成程度。QProgressBar支持水平和垂直两种显示方向&#xff0c;并且可以通过设置最小值和最大值来指定进度条的范围。此外&…

Nginx 实现动态封禁IP,详细教程来了

Nginx 实现动态封禁IP&#xff0c;详细教程来了 需求环境准备设计方案在操作系统层面&#xff0c;配置 iptables&#xff0c;来拦截指定 IP 的网络请求在 Web 服务器层面&#xff0c;通过 Nginx 自身的 deny 选项或者 lua 插件配置 IP 黑名单在应用层面&#xff0c;在处理请求之…

C++入门——“C++11-右值引用和移动语义”

C11相比于C98增加以许多新特性&#xff0c;让C语言更加灵活好用&#xff0c;但是貌似也增加了许多学习的难度&#xff0c;现在先看第一部分。 一、右值引用和移动语义 1.右值引用和左值引用 在C中&#xff0c;值可以大致分为右值和左值&#xff0c;左值大概是哪些已经被定义的变…

Leetcode 64. 最小路径和 动态规划+空间优化

原题链接&#xff1a;Leetcode 64. 最小路径和 二维数据 class Solution { public:int minPathSum(vector<vector<int>>& grid) {int m grid.size();int n grid[0].size();int dp[m][n];dp[0][0] grid[0][0];for (int j 1; j < n; j)dp[0][j] dp[0][…

OpenHarmony、HarmonyOS、HarmonyNext互相兼容吗?

1&#xff0c;三者之间的关系 OpenHarmony&#xff1a;开源底层。HarmonyOS&#xff1a;闭源手机系统&#xff0c;兼容安卓生态。HarmonyOS NEXT&#xff1a;纯血鸿蒙&#xff0c;不兼容安卓。 上一篇文章简单介绍过&#xff0c;就不再多说了&#xff0c;这里说一下HarmonyOS …

定制化视频生成新模范!零样本主体驱动,精确运动控制!复旦阿里等发布DreamVideo-2

文章链接&#xff1a;https://arxiv.org/pdf/2410.13830 项目链接&#xff1a;https://dreamvideo2.github.io/ 亮点直击 DreamVideo-2&#xff0c;首个无需微调&#xff0c;同时支持主体定制和运动控制的零样本视频定制框架&#xff0c;能够通过设计的参考注意力学习主体外观&…

Java中的线程安全问题(如果想知道Java中有关线程安全问题的基本知识,那么只看这一篇就足够了!)

前言&#xff1a;多线程编程已经广泛开始使用&#xff0c;其可以充分利用系统资源来提升效率&#xff0c;但是线程安全问题也随之出现&#xff0c;它直接影响了程序的正确性和稳定性&#xff0c;需要对其进行深入的理解与解决。 ✨✨✨这里是秋刀鱼不做梦的BLOG ✨✨✨想要了解…

Kubernetes Node 节点的生命周期简述

Node 节点是 Kubernetes 的核心组件之一&#xff0c;其生命周期可以简要概括为&#xff1a;注册、运行、下线。本文将简要介绍 Node 生命周期中发生的关键事件。 节点注册 每个 node 节点都需要运行 kubelet&#xff0c;kubelet 启动后会向 kube-apiserver 发起节点的注册请求…

GenAI 生态系统现状:不止大语言模型和向量数据库

自 20 个月前 ChatGPT 革命性的推出以来&#xff0c;生成式人工智能&#xff08;GenAI&#xff09;领域经历了显著的发展和创新。最初&#xff0c;大语言模型&#xff08;LLMs&#xff09;和向量数据库吸引了最多的关注。然而&#xff0c;GenAI 生态系统远不止这两个部分&#…

ctf文件上传题小总结与记录

解题思路&#xff1a;先看中间件&#xff0c;文件上传点&#xff08;字典扫描&#xff0c;会员中心&#xff09;&#xff0c;绕过/验证&#xff08;黑名单&#xff0c;白名单&#xff09;&#xff0c;解析漏洞&#xff0c;cms&#xff0c;编辑器&#xff0c;最新cve 文件上传漏…