一、数据处理

1.爬取数据

我们将使用Python的requests和BeautifulSoup库来爬取数据。在这个示例中，我们将爬取豆瓣电影Top250的数据。

import requests
from bs4 import BeautifulSoup
url = 'https://movie.douban.com/top250'
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3'}
response = requests.get(url, headers=headers)
soup = BeautifulSoup(response.text, 'html.parser')

2.数据清洗

我们需要从爬取的数据中提取出我们需要的信息，例如电影名称、导演、演员、评分等。我们可以使用正则表达式或BeautifulSoup来提取这些信息。

movies = []
for movie in soup.select('.item'):
    title = movie.select('.title')[0].text.strip()
    director = movie.select('.bd p')[0].text.strip().split('\xa0')[0][4:]
    actors = movie.select('.bd p')[0].text.strip().split('\xa0')[1][3:]
    rating = movie.select('.rating_num')[0].text.strip()
    movies.append({'title': title, 'director': director, 'actors': actors, 'rating': rating})

3.数据存储

我们可以将清洗后的数据存储到CSV文件中，以便后续的数据分析处理。

import csv
with open('movies.csv', 'w', newline='', encoding='utf-8') as csvfile:
    fieldnames = ['title', 'director', 'actors', 'rating']
    writer = csv.DictWriter(csvfile, fieldnames=fieldnames)
    writer.writeheader()
    for movie in movies:
        writer.writerow(movie)

完整代码：

import requests
from bs4 import BeautifulSoup

import csv

url = 'https://movie.douban.com/top250'
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3'}
response = requests.get(url, headers=headers)
soup = BeautifulSoup(response.text, 'html.parser')

movies = []
for movie in soup.select('.item'):
    title = movie.select('.title')[0].text.strip()
    director = movie.select('.bd p')[0].text.strip().split('\xa0')[0][4:]
    actors = movie.select('.bd p')[0].text.strip().split('\xa0')[1][3:]
    rating = movie.select('.rating_num')[0].text.strip()
    movies.append({'title': title, 'director': director, 'actors': actors, 'rating': rating})

with open('movies.csv', 'w', newline='', encoding='utf-8') as csvfile:
    fieldnames = ['title', 'director', 'actors', 'rating']
    writer = csv.DictWriter(csvfile, fieldnames=fieldnames)
    writer.writeheader()
    for movie in movies:
        writer.writerow(movie)

二、数据分析

1.数据基本信息

我们可以使用pandas库来读取CSV文件，并查看数据的基本信息。

import pandas as pd
df = pd.read_csv('movies.csv')
print('原始数据量：', len(df))
print('数据内容：')
print(df.head())
print('数据格式：')
print(df.dtypes)

输出结果：

2.数据可视化

我们可以使用matplotlib和wordcloud库来进行数据可视化。下面是一个绘制电影评分分布的柱状图和电影名称的词云图的示例。

from wordcloud import WordCloud, STOPWORDS
import matplotlib.pyplot as plt
# 绘制电影评分分布的柱状图
plt.hist(df['rating'], bins=10, range=(0, 10))
plt.xlabel('Rating')
plt.ylabel('Count')
plt.title('Distribution of Movie Ratings')
plt.show()

# 绘制电影名称的词云图
text = ' '.join(df['title'])
wordcloud = WordCloud(width=800, height=400, background_color='white').generate(text)
plt.imshow(wordcloud, interpolation='bilinear')
plt.axis('off')
plt.show()

完整代码：

from wordcloud import WordCloud, STOPWORDS
import matplotlib.pyplot as plt

import pandas as pd
df = pd.read_csv('movies.csv')
print('原始数据量：', len(df))
print('数据内容：')
print(df.head())
print('数据格式：')
print(df.dtypes)

# 绘制电影评分分布的柱状图
plt.hist(df['rating'], bins=10, range=(0, 10))
plt.xlabel('Rating')
plt.ylabel('Count')
plt.title('Distribution of Movie Ratings')
plt.show()

# 绘制电影名称的词云图
text = ' '.join(df['title'])
wordcloud = WordCloud(width=800, height=400, background_color='white').generate(text)
plt.imshow(wordcloud, interpolation='bilinear')
plt.axis('off')
plt.show()