示例 9-1。导入+过滤数据

>>> import pandas as pd

>>> model_df = pd.read_json('data/mag_papers_0/mag_subset20K.txt', lines=True)
>>> model_df.shape
(20000, 19)
>>> model_df.columns
Index(['abstract', 'authors', 'doc_type', 'doi', 'fos', 'id', 'issue',
       'keywords', 'lang', 'n_citation', 'page_end', 'page_start', 'publisher',
       'references', 'title', 'url', 'venue', 'volume', 'year'],
      dtype='object')

# filter out non-English articles and focus on a few variables
>>> model_df = model_df[model_df.lang == 'en']
...            .drop_duplicates(subset='title', keep='first')
...            .drop(['doc_type', 'doi', 'id', 'issue', 'lang', 'n_citation', 
...                   'page_end', 'page_start', 'publisher', 'references',
...                   'url', 'venue', 'volume'], 
...                  axis=1)
>>> model_df.shape
(10399, 6)

图 9-1。Microsoft Academic Graph 数据集的前两行

示例 9-2。协同过滤第一阶段：构建物品特征矩阵

>>> unique_fos = sorted(list({feature
...                           for paper_row in model_df.fos.fillna('0')
...                           for feature in paper_row }))

>>> unique_year = sorted(model_df['year'].astype('str').unique())
>>> def feature_array(x, var, unique_array):
...     row_dict = {}
...     for i in x.index:
...         var_dict = {}
...         for j in range(len(unique_array)):
...             if type(x[i]) is list:
...                 if unique_array[j] in x[i]:
...                     var_dict.update({var + '_' + unique_array[j]: 1})
...                 else:
...                     var_dict.update({var + '_' + unique_array[j]: 0})
...             else:    
...                 if unique_array[j] == str(x[i]):
...                     var_dict.update({var + '_' + unique_array[j]: 1})
...                 else:
...                     var_dict.update({var + '_' + unique_array[j]: 0})     
...         row_dict.update({i : var_dict})
...     feature_df = pd.DataFrame.from_dict(row_dict, dtype='str').T
...     return feature_df

>>> year_features = feature_array(model_df['year'], unique_year)
>>> fos_features = feature_array(model_df['fos'], unique_fos)

>>> first_features = fos_features.join(year_features).T

>>> from sys import getsizeof
>>> print('Size of first feature array: ', getsizeof(first_features))
Size of first feature array: 2583077234

图 9-2。head of first_features——来自原始数据集的观察（论文）索引是列，特征是行

示例 9-3。协同过滤第 2 阶段：搜索相似项

>>> from scipy.spatial.distance import cosine

>>> def item_collab_filter(features_df):
...     item_similarities = pd.DataFrame(index = features_df.columns, 
...                                      columns = features_df.columns) 
...     for i in features_df.columns:
...         for j in features_df.columns:
...             item_similarities.loc[i][j] = 1 - cosine(features_df[i], 
...                                                      features_df[j])
...     return item_similarities

>>> first_items = item_collab_filter(first_features.loc[:, 0:1000])

为什么我们只使用两个特征来计算项目相似度需要这么长时间？我们使用嵌套for循环对 10,399 × 1,000 矩阵进行点积。随着我们增加添加到模型中的观察数量，每个循环的时间也会增加。请记住，这是全部可用数据集的一个子集，已针对纯英文论文进行过滤。当我们越来越接近“好”结果时，我们需要返回并在更大的集合上进行测试以获得最佳结果。

示例 9-4。论文推荐的热图

>>> import matplotlib.pyplot as plt
>>> import seaborn as sns
>>> import numpy as np
>>> %matplotlib inline
>>> sns.set()
>>> ax = sns.heatmap(first_items.fillna(0), 
...                  vmin=0, vmax=1, 
...                  cmap="YlGnBu", 
...                  xticklabels=250, yticklabels=250)
>>> ax.tick_params(labelsize=12)

较暗的像素表示彼此相似的项目。深色对角线表明余弦相似度正确地表明每篇论文与其自身最相似。然而，因为我们的一个特征有很多 NaN，所以这条线沿着对角线断开了。我们可以看到，虽然大多数项目彼此不相似——即，我们的数据集相当多样化——但还有一些其他的高分候选项目。这些可能是也可能不是定性的好建议，但至少我们可以看到我们的方法没有那么疯狂。

图 9-3。基于两个原始特征的相似论文的热图：年份和研究领域

示例 9-6。固定宽度合并 + 虚拟编码（第 1 部分）

>>> print("Year spread: ", model_df['year'].min()," - ", model_df['year'].max())
>>> print("Quantile spread:\n", model_df['year'].quantile([0.25, 0.5, 0.75]))
Year spread:  1831  -  2017
Quantile spread:
0.25    1990.0
0.50    2005.0
0.75    2012.0
Name: year, dtype: float64

# plot years to see the distribution
>>> fig, ax = plt.subplots()
>>> model_df['year'].hist(ax=ax, 
...                       bins= model_df['year'].max() - model_df['year'].min())
>>> ax.tick_params(labelsize=12)
>>> ax.set_xlabel('Year Count', fontsize=12)
>>> ax.set_ylabel('Occurrence', fontsize=12)

我们可以从偏态分布（图 9-5）中看出这是分箱的绝佳候选。

图 9-5。数据集中 10K+ 学术论文的原始年份分布

示例 9-7。固定宽度分箱 + 虚拟编码（第 2 部分）

# binning here (by 10 years) reduces the year feature space from 156 to 19 
>>> bins = int(round((model_df['year'].max() - model_df['year'].min()) / 10))

>>> temp_df = pd.DataFrame(index = model_df.index)
>>> temp_df['yearBinned'] = pd.cut(model_df['year'].tolist(), bins, precision = 0)
>>> X_yrs = pd.get_dummies(temp_df['yearBinned'])
>>> X_yrs.columns.categories
IntervalIndex([(1831.0, 1841.0], (1841.0, 1851.0], (1851.0, 1860.0],
               (1860.0, 1870.0], (1870.0, 1880.0] ... (1968.0, 1978.0],
               (1978.0, 1988.0], (1988.0, 1997.0], (1997.0, 2007.0],
               (2007.0, 2017.0]]
              closed='right',
              dtype='interval[float64]')

# plot the new distribution
>>> fig, ax = plt.subplots()
>>> X_yrs.sum().plot.bar(ax = ax)
>>> ax.tick_params(labelsize=8)
>>> ax.set_xlabel('Binned Years', fontsize=12)
>>> ax.set_ylabel('Counts', fontsize=12)

几十年来，我们通过分箱保留了原始变量的基本分布。如果我们希望使用一种可以从不同分布中获益的方法，我们可以改变我们的分箱选择来改变这个变量如何呈现给模型。因为我们使用的是余弦相似度，所以这很好。让我们继续讨论我们最初包含在模型中的下一个特征。

图 9-6。新分箱 X_yrs 特征的分布

示例 9-14。基于项目的协同过滤推荐：Take 3

>>> paper_recommender(fourth_features, 2, 3)

Based on the paper:
Paper index =  2
Title : Should endometriosis be an indication for intracytoplasmic sperm inject ...
FOS : nan
Year : 2015
Abstract : nan
Authors : [{'name': 'Jovana P. Lekovich', 'org': 'Weill Cornell Medical College, ...


Top 3 results:
1 . Paper index =  10055
Similarity score:  1.0
Title : [Diagnosis of cerebral tumors; comparative studies on arteriography, ...
FOS : ['Radiology', 'Pathology', 'Surgery']
Year : 1953
Abstract : nan
Authors : [{'name': 'Antoine'}, {'name': 'Lepoire'}, {'name': 'Schoumacker'}]

2 . Paper index =  5601
Similarity score:  1.0
Title : 633 Survival after coronary revascularization, with and without mitral ...
FOS : ['Cardiology']
Year : 2005
Abstract : nan
Authors : [{'name': 'J.B. Le Polain De Waroux'}, {'name': 'Anne-Catherine ...

3 . Paper index =  12256
Similarity score:  1.0
Title : Nucleotide Sequence and Analysis of an Insertion Sequence from Bacillus ...
FOS : ['Biology', 'Molecular biology', 'Insertion sequence', 'Nucleic acid ...
Year : 1994
Abstract : A 5.8-kb DNA fragment encoding the  cryIC  gene from  Bacillus thur...
Authors : [{'name': 'Geoffrey P. Smith'}, {'name': 'David J. Ellar'}, {'name': ...

即使考虑到某些领域的缺失数据，上一轮特征工程的前三名结果也将我们引向了医学领域的其他论文。