基于Python和Spacy的命名实体识别

news2024/9/23 9:30:57

命名实体识别（Named Entity Recognition，简称NER）是一种自然语言处理（NLP）方法，用于检测和分类文本中的命名实体，包括人物、组织、地点、日期、数量和其他可识别的现实世界实体。

Spacy是一个基于Python的开源自然语言处理库，提供广泛的功能，包括标记化、POS标签、句法分析、命名实体识别、文本分类等。它具有预训练模型和灵活的API，受到自然语言处理专业人士和研究人员的广泛青睐。

'''
Prefix the install statements with an exclamation if you are using 
Jupyter Notebook or Google Collab interface.
'''
!pip install -U pip setuptools wheel
# 使用pip安装gpu版本的spacy（有关cpu版本的安装，请参考spacy文档）
!pip install -U 'spacy[cuda-autodetect]'
# 安装基于Transformer的预训练语言模型（英语）
!python -m spacy download en_core_web_trf
# 安装基于CPU优化的管道（英语）
!python -m spacy download en_core_web_sm

如果你正在使用Google Colab或配备GPU的计算机，则后续操作将涉及将运行时类型切换为GPU。否则，如果你使用的是基于CPU的安装，则可以继续使用CPU运行时类型。让我们在以下示例文本上执行NER。

用作示例的输入文本：

“Artificial Intelligence (AI), an ever-evolving field, has witnessed remarkable growth since its inception. Dating back to the Dartmouth Conference in 1956, AI has emerged as a multidisciplinary domain encompassing machine learning, natural language processing (NLP), computer vision, and robotics. Recent breakthroughs, like the introduction of deep learning techniques in the early 2010s, have accelerated AI advancements. Tech giants like Google, IBM, and Microsoft have invested heavily in AI research and development. Significant milestones include the landmark victory of IBM’s Deep Blue over Garry Kasparov in 1997 and the emergence of voice assistants like Apple’s Siri in 2011. AI continues to shape industries across healthcare, finance, and transportation, fueling innovation and transforming the way we live and work.”

我们将使用两个不同的模型，每个模型都有特定的目的。其中一个模型（“en_core_web_sm”）注重效率，另一个模型（“en_core_web_trf”）注重准确性。

# 导入依赖项
import spacy
sample_text = "Artificial Intelligence (AI), an ever-evolving field, has witnessed remarkable growth since its inception. Dating back to the Dartmouth Conference in 1956, AI has emerged as a multidisciplinary domain encompassing machine learning, natural language processing (NLP), computer vision, and robotics. Recent breakthroughs, like the introduction of deep learning techniques in the early 2010s, have accelerated AI advancements. Tech giants like Google, IBM, and Microsoft have invested heavily in AI research and development. Significant milestones include the landmark victory of IBM's Deep Blue over Garry Kasparov in 1997 and the emergence of voice assistants like Apple's Siri in 2011. AI continues to shape industries across healthcare, finance, and transportation, fueling innovation and transforming the way we live and work."
# 加载用于更好效率的普通模型。
nlp = spacy.load("en_core_web_sm")
# 预测实体
text = nlp(sample_text)
# 在jupyter模式下显示文本和突出的实体
spacy.displacy.render(text, style="ent", jupyter=True)

输出：

使用Spacy的可视化工具和en_core_web_sm模型的实体可视化输出

# 使用基于Transformer的模型进行NER
# 使用spaCy加载基于Transformer的管道（roberta-base）
nlp_trf = spacy.load('en_core_web_trf')
# 对模型进行推断
text = nlp_trf(sample_text)
# 使用jupyter模式显示文档
spacy.displacy.render(text, style="ent", jupyter=True)