一、示例的函数说明:
函数processing():用于处理字符串中的空白字符,并拼接字符串。
主函数程序入口:每页显示25部影片,实现循环,共10页。通过format方法替换切换的页码的url地址。然后调用实现爬虫程序的函数get_movie_info(),获取电影信息。
二、示例代码:
from lxml import etree
import time
import random
import requests
headers = {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) '
'Chrome/117.0.0.0 Safari/537.36'
}
def processing(strs):
s = ''
for n in strs:
n = ''.join(n.split())
s = s + n
return s
def get_movie_info(url):
response = requests.get(url, headers=headers)
html = etree.HTML(response.text)
div_all = html.xpath('//div[@class="info"]')
for div in div_all:
names = div.xpath('./div[@class="hd"]/a//span/text()')
name = processing(names)
infos = div.xpath('./div[@class="bd"]/p/text()')
info = processing(infos)
score = div.xpath('./div[@class="bd"]/div/span[2]/text()')
evaluation = div.xpath('./div[@class="bd"]/div/span[4]/text()')
summary = div.xpath('./div[@class="bd"]/p[@class="quote"]/span/text()')
print('电影名称:', name)
print('导演与演员:', info)
print('影片评分:', score)
print('评价人数:', evaluation)
print('影片总结:', summary)
print('-------分割线-------')
if __name__ == '__main__':
for i in range(0, 250, 25):
url = 'https://movie.douban.com/top250?start={page}&filter='.format(page=i)
get_movie_info(url)
time.sleep(random.randint(1, 3))