Tiktok 关键字视频及评论信息爬虫(2) [2025.04.07]

news2025/4/17 10:32:08

🙋‍♀️Tiktok APP的基于关键字检索的视频及评论信息爬虫共分为两期，希望对大家有所帮助。
第一期：基于关键字检索的视频信息爬取
第二期见下文。

1.Node.js环境配置

首先配置 JavaScript 运行环境（如 Node.js），用于执行加密签名代码。
Node.js下载网址：https://nodejs.org/en
Node.js的安装方法（环境配置非常关键，决定了后面的程序是否可以使用）：https://blog.csdn.net/liufeifeihuawei/article/details/132425239

2. Py环境配置

import random
from tqdm import tqdm
import requests
from urllib.parse import urlparse, urlencode
import warnings
from urllib3.exceptions import InsecureRequestWarning
import time

# 忽略 InsecureRequestWarning 警告
warnings.filterwarnings("ignore", category=InsecureRequestWarning)

3. 基于视频URL的评论信息爬取

在上期中，已经给出了如何获得指定视频的URL，下面给出根据URL获得视频评论的信息，允许在爬取的过程中对评论进行翻页。

1. 主程序

爬单个URL的评论信息的方法：

if __name__ == '__main__':
    '''单条数据'''
    req_url = "https://www.tiktok.com/@resep_debm/video/7475545671383174406"
    tiktok_comment = TiktokComment()
    x = tiktok_comment.get_comment_list(req_url)
    print(x)

爬多个URL的评论信息的方法。通过读取videosInfo.json文件中保存的URL信息，将最后的结果保存到videos_comments.json文件中：

if __name__ == '__main__':
    '''多条数据'''
    data = read_json('../results/videosInfo.json')
    print(len(data))
    tiktok_comment = TiktokComment()
    new_data = data.copy()

    for i in tqdm(range(len(data))):
        if 'comments' not in data[i].keys():  #  and i > 1695
            comments = tiktok_comment.get_comment_list(data[i]['video_url'])
            if comments != []:
                new_data[i]['comments'] = comments
        else:
            continue

        if i % 10 == 0:
            write_json('../results/videos_comments.json', new_data)

    # 循环结束后再保存一次，确保所有数据都被写入
    write_json('../results/videos_comments.json', new_data)

2. 定义TiktokComments类
允许获得的评论信息7个字段，包括：
🎰评论ID；
💬评论内容；
🙋评论是否被作者点赞；
😍评论是否热门；
👍评论的点赞数
👀评论的回复数目
⌛评论发布的时间；

class TiktokComments:
    def __init__(self):
        # self.config = read_config()
        self.common_utils = CommonUtils()
        self.cookies = cookie_str_to_dict(read_cookie())
        # self.proxies = self.config.get("proxies", None)  # 代理配置
        self.comment_list_headers = {
            'sec-ch-ua': '"Google Chrome";v="123", "Not:A-Brand";v="8", "Chromium";v="123"',
            'sec-ch-ua-mobile': '?0',
            'User-Agent': self.common_utils.user_agent,
            'sec-ch-ua-platform': '"Windows"',
            'Accept': '*/*',
            'Sec-Fetch-Site': 'same-origin',
            'Sec-Fetch-Mode': 'cors',
            'Sec-Fetch-Dest': 'empty',
            'Accept-Language': 'zh-CN,zh;q=0.9,en;q=0.8',
        }

通过 cursor_num设置翻页，count={comment_num}表示希望获得的评论总数目。

    def get_comment_list(self, video_url, comments_num=100):
        aweme_id = urlparse(video_url).path.split("/")[-1]
        ms_token = self.cookies['msToken']

        req_comments = []
        max_retries = 3  # 最大重试次数
        for i in range(comments_num // 20):
            cursor_num = i * 20
            comment_num = 20
            req_url = f"https://www.tiktok.com/api/comment/list/?WebIdLastTime=1715249710&aid=1988&app_language=ja-JP&app_name=tiktok_web&aweme_id={aweme_id}&browser_language=zh-CN&browser_name=Mozilla&browser_online=true&browser_platform=Win32&browser_version=5.0%20%28Windows%20NT%2010.0%3B%20Win64%3B%20x64%29%20AppleWebKit%2F537.36%20%28KHTML%2C%20like%20Gecko%29%20Chrome%2F123.0.0.0%20Safari%2F537.36&channel=tiktok_web&cookie_enabled=true&" \
                      f"count={comment_num}&current_region=JP&cursor={cursor_num}&device_id=7366941338308609569&device_platform=web_pc&enter_from=tiktok_web&focus_state=true&fromWeb=1&from_page=video&history_len=2&is_fullscreen=false&is_non_personalized=false&is_page_visible=true&odinId=7367172442253296673&os=windows&priority_region=&referer=&region=GB&screen_height=1080&screen_width=1920&tz_name=Asia%2FShanghai&webcast_language=zh-Hans&msToken={ms_token}"
            xbogus = self.common_utils.get_xbogus(req_url, self.common_utils.user_agent)
            req_url += f'&X-Bogus={xbogus}&_signature=_02B4Z6wo000016M20awAAIDAnp.LMKuZmC-jNtUAAI6L17'

            for retry in range(max_retries):
                try:
                    response = requests.request(
                        'GET',
                        req_url,
                        headers=self.comment_list_headers,
                        # cookies=self.cookies,
                        verify=False,
                        timeout=random.randint(3, 7),
                        # proxies=self.proxies
                    )
                    if response.status_code != 200:
                        continue

                    req_json = response.json()
                    comments = req_json.get('comments', [])
                    # print(f"评论数目：{req_json.get('total')}")
                    if not comments:
                        print(f"No comments found for cursor {cursor_num}.")
                        break

                    for comment_item in comments:
                        req_comments.append({
                            "cid": comment_item.get('cid'),
                            "comment": comment_item.get('text'),
                            "comments_is_author_like": comment_item.get('is_author_digged'),
                            "comments_is_hot": comment_item.get('is_comment_translatable'),
                            "comments_like": comment_item.get('digg_count'),
                            "comments_reply": comment_item.get('reply_comment_total'),
                            "comments_time": comment_item.get('create_time')
                        })
                    break  # 成功获取数据，退出重试循环
                except Exception as e:
                    print(f"Error: {e}. Retrying ({retry + 1}/{max_retries})...")
                if retry == max_retries - 1:
                    print("Max retries reached. Skipping this request.")
        return req_comments