selenium模拟浏览器查询导出参考文献

news2024/11/20 4:48:26

通过使用Selenium和BeautifulSoup，在CNKI网站上，以"知识图谱"为关键词，通过自动化工具在搜索页面提取相关文章信息。点击清楚并全选进行文献导出，随后从导出页面和管理导出的页面提取参考文献。

浏览器及WebDriver下载
https://googlechromelabs.github.io/chrome-for-testing/#stable

放到/usr/local/bin （mac的配置）

mv chromedriver /usr/local/bin
cd /usr/local/bin
xattr -d com.apple.quarantine chromedriver

提取文献代码

from selenium import webdriver
from selenium.webdriver.common.by import By
import time
from bs4 import BeautifulSoup

def driver_open(driver, key_word):
    url = "https://www.cnki.net/"
    driver.get(url)
    time.sleep(2)
    driver.find_element(By.CSS_SELECTOR ,'#txt_SearchText').send_keys(key_word)
    time.sleep(2)
    # 点击搜索按钮
    driver.find_element(By.CSS_SELECTOR ,'body > div.wrapper.section1 > div.searchmain > div > div.input-box > input.search-btn').click()
    time.sleep(5)
    driver.find_element(By.CSS_SELECTOR,'#briefBox > div:nth-child(1) > div > div.toolbar-col > div.checkcount > a').click()
    time.sleep(2)
    driver.find_element(By.CSS_SELECTOR,'#selectCheckAll1').click()
    driver.find_element(By.CSS_SELECTOR,'#batchOpsBox > li:nth-child(2) > a').click()
    driver.find_element(By.CSS_SELECTOR,'#batchOpsBox > li:nth-child(2) > ul > li.export > a').click()
    driver.find_element(By.CSS_SELECTOR,'#batchOpsBox > li:nth-child(2) > ul > li.export > ul > li:nth-child(1) > a').click()
    time.sleep(2)
    url1="https://kns.cnki.net/dm8/manage/export.html?language=CHS&uniplatform=NZKPT"
    driver.get(url1)
    time.sleep(5)
    driver.find_element(By.CSS_SELECTOR,'#result > ul').click()
    content = driver.page_source.encode('utf-8')
    soup = BeautifulSoup(content, 'lxml')
    ul = soup.find_all('ul')
    ul = BeautifulSoup(str(ul[5]), 'lxml')
    lis = ul.find_all('li')
    text=""
    for li in lis:
        text = text + li.get_text().strip().replace(" ","")+"\n"
    return text
if __name__ == '__main__':
    chrome_options= webdriver.ChromeOptions()
    chrome_options.add_argument('--headless')
    chrome_options.add_argument('--disable‐gpu')
    path="/usr/local/bin"
	chrome_options.binary_location = path
    driver = webdriver.Chrome(options=chrome_options)
    text = driver_open(driver, '知识图谱') 
    print(text)