爬虫-微博个人主页的获取

news2024/11/28 4:49:10

我们在利用爬虫爬取微博个人主页的时候，我们需要获取到个人页面的cookie才能进入到微博的个人主页，否则的话将会是一直跳转到登录页面而导致不能进入个人主页。

import urllib.request
url = ''#自己微博个人主页的源代码
headers = {
    'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36 Edg/114.0.1823.82',
    'Cookie':'XSRF-TOKEN=g0FF-Ky4ISDWHSpw0syth_7Y; login_sid_t=d2f25f9ffe79e4253d810e203384aa7f; cross_origin_proto=SSL; WBStorage=4d96c54e|undefined; _s_tentry=weibo.com; Apache=1422875234826.102.1689914268411; SINAGLOBAL=1422875234826.102.1689914268411; ULV=1689914268412:1:1:1:1422875234826.102.1689914268411:; wb_view_log=1536*8641.25; appkey=; PC_TOKEN=496b26f766; WBtopGlobal_register_version=2023072112; SUB=_2A25JvnzWDeRhGeFJ71cV-CnKyDmIHXVqyukerDV8PUNbmtANLUj3kW9Nf8mRspTt9Vt2jzdjG9A2yOGzQNKZ9NWd; SUBP=0033WrSXqPxfM725Ws9jqgMF55529P9D9WWHEn1u_FrFxcuTWJSni9_45JpX5o275NHD95QNS0BfShnNSoefWs4DqcjMi--NiK.Xi-2Ri--ciKnRi-zNS0MXSKBRS0q0SBtt; ALF=1690519302; SSOLoginState=1689914502; WBPSESS=Ad500zUchVHLxb44p3eM4-2SLSs4GjTsb4vmv-67w2A2W8Wf21lcVI-pGBguoXKjGMf_8tiQ-kKEk_0jobCXrgms98G25KfqMrQYEcJ_VvC7dXNhDOcE7rgCpPAP-ByIXe09gDNr45HbE4Lq6XfYoQ==',
    'Referer':'https://weibo.com/signup/signup.php'
}
#建立请求
request = urllib.request.Request(url = url ,headers = headers)
response = urllib.request.urlopen(request)
content= response.read().decode('utf-8')
with open('weibo.html','w',encoding ='utf-8') as fp:
    fp.write(content)

在爬取完生成的HTML中，我们可以直接点击浏览器标识就可进入页面了。