示例 2-8。使用对数转换的 Yelp 评论计数来预测平均商业评级

import pandas as pd
import numpy as np
import json
from sklearn import linear_model
from sklearn.model_selection import cross_val_score

# Using the previously loaded Yelp reviews DataFrame,
# compute the log transform of the Yelp review count.
# Note that we add 1 to the raw count to prevent the logarithm from
# exploding into negative infinity in case the count is zero.
biz_df['log_review_count'] = np.log10(biz_df['review_count'] + 1)

# Train linear regression models to predict the average star rating of a business,
# using the review_count feature with and without log transformation.
# Compare the 10-fold cross validation score of the two models.
m_orig = linear_model.LinearRegression()
scores_orig = cross_val_score(m_orig, biz_df[['review_count']], 
                              biz_df['stars'], cv=10)
m_log = linear_model.LinearRegression()
scores_log = cross_val_score(m_log, biz_df[['log_review_count']], 
                             biz_df['stars'], cv=10)
print("R-squared score without log transform: %0.5f (+/- %0.5f)"
      % (scores_orig.mean(), scores_orig.std() * 2))
print("R-squared score with log transform: %0.5f (+/- %0.5f)"
      % (scores_log.mean(), scores_log.std() * 2))

R-squared score without log transform: -0.03683 (+/- 0.07280)R-squared score with log transform: -0.03694 (+/- 0.07650)

示例 2-9。使用在线新闻流行度数据集中的对数转换字数来预测文章流行度

# Download the Online News Popularity dataset from UCI, then use
# Pandas to load the file into a DataFrame.
df = pd.read_csv('OnlineNewsPopularity.csv', delimiter=', ')

# Take the log transform of the 'n_tokens_content' feature, which 
# represents the number of words (tokens) in a news article.
df['log_n_tokens_content'] = np.log10(df['n_tokens_content'] + 1)

# Train two linear regression models to predict the number of shares
# of an article, one using the original feature and the other the
# log transformed version.
m_orig = linear_model.LinearRegression()
scores_orig = cross_val_score(m_orig, df[['n_tokens_content']],
                              df['shares'], cv=10)
m_log = linear_model.LinearRegression()
scores_log = cross_val_score(m_log, df[['log_n_tokens_content']],
                             df['shares'], cv=10)
print("R-squared score without log transform: %0.5f (+/- %0.5f)"
      % (scores_orig.mean(), scores_orig.std() * 2))
print("R-squared score with log transform: %0.5f (+/- %0.5f)"
      % (scores_log.mean(), scores_log.std() * 2))

R-squared score without log transform: -0.00242 (+/- 0.00509)R-squared score with log transform: -0.00114 (+/- 0.00418)

示例 2-10。可视化新闻流行度预测问题中输入和输出之间的相关性

fig2, (ax1, ax2) = plt.subplots(2,1)
ax1.scatter(df['n_tokens_content'], df['shares'])
ax1.tick_params(labelsize=14)
ax1.set_xlabel('Number of Words in Article', fontsize=14)
ax1.set_ylabel('Number of Shares', fontsize=14)

ax2.scatter(df['log_n_tokens_content'], df['shares'])
ax2.tick_params(labelsize=14)
ax2.set_xlabel('Log of the Number of Words in Article', fontsize=14)
ax2.set_ylabel('Number of Shares', fontsize=14)

图 2-9。在线新闻流行度数据集中词数（输入）与分享数（目标）的散点图——上图可视化原始特征，下图显示对数转换后的散点图

示例 2-11。可视化 Yelp 业务评论预测中输入和输出之间的相关性

fig, (ax1, ax2) = plt.subplots(2,1)
ax1.scatter(biz_df['review_count'], biz_df['stars'])
ax1.tick_params(labelsize=14)
ax1.set_xlabel('Review Count', fontsize=14)
ax1.set_ylabel('Average Star Rating', fontsize=14)

ax2.scatter(biz_df['log_review_count'], biz_df['stars'])
ax2.tick_params(labelsize=14)
ax2.set_xlabel('Log of Review Count', fontsize=14)
ax2.set_ylabel('Average Star Rating', fontsize=14)

图 2-10。Yelp 评论数据集中评论数（输入）与平均星级（目标）的散点图——顶部面板绘制原始评论数，底部面板绘制日志转换后的评论数

示例 2-12。Yelp 业务评论计数的 Box-Cox 转换

from scipy import stats

# Continuing from the previous example, assume biz_df contains
# the Yelp business reviews data.
# The Box-Cox transform assumes that input data is positive. 
# Check the min to make sure.
biz_df['review_count'].min()
# 3

# Setting input parameter lmbda to 0 gives us the log transform (without 
# constant offset)
rc_log = stats.boxcox(biz_df['review_count'], lmbda=0)
# By default, the scipy implementation of Box-Cox transform finds the lambda
# parameter that will make the output the closest to a normal distribution
rc_bc, bc_params = stats.boxcox(biz_df['review_count'])
bc_params
# -0.4106510862321085

示例 2-13。可视化原始计数、对数变换计数和 Box-Cox 变换计数的直方图

fig, (ax1, ax2, ax3) = plt.subplots(3,1)
# original review count histogram
biz_df['review_count'].hist(ax=ax1, bins=100)
ax1.set_yscale('log')
ax1.tick_params(labelsize=14)
ax1.set_title('Review Counts Histogram', fontsize=14)
ax1.set_xlabel('')
ax1.set_ylabel('Occurrence', fontsize=14)

# review count after log transform
biz_df['rc_log'].hist(ax=ax2, bins=100)
ax2.set_yscale('log')
ax2.tick_params(labelsize=14)
ax2.set_title('Log Transformed Counts Histogram', fontsize=14)
ax2.set_xlabel('')
ax2.set_ylabel('Occurrence', fontsize=14)

# review count after optimal Box-Cox transform
biz_df['rc_bc'].hist(ax=ax3, bins=100)
ax3.set_yscale('log')
ax3.tick_params(labelsize=14)
ax3.set_title('Box-Cox Transformed Counts Histogram', fontsize=14)
ax3.set_xlabel('')
ax3.set_ylabel('Occurrence', fontsize=14)

图 2-13。Yelp 业务评论计数（底部）的 Box-Cox 转换与原始（顶部）和日志转换（中间）直方图相比

示例 2-14。原始计数和转换计数相对于正态分布的概率图

fig2, (ax1, ax2, ax3) = plt.subplots(3,1)
prob1 = stats.probplot(biz_df['review_count'], dist=stats.norm, plot=ax1)
ax1.set_xlabel('')
ax1.set_title('Probplot against normal distribution')
prob2 = stats.probplot(biz_df['rc_log'], dist=stats.norm, plot=ax2)
ax2.set_xlabel('')
ax2.set_title('Probplot after log transform')
prob3 = stats.probplot(biz_df['rc_bc'], dist=stats.norm, plot=ax3)
ax3.set_xlabel('Theoretical quantiles')
ax3.set_title('Probplot after Box-Cox transform')

图 2-14。比较原始和转换评论计数的分布与正态分布

图 2-15。最小-最大缩放的图示

图 2-16。特征标准化示意图

示例 2-15。特征缩放示例

import pandas as pd
import sklearn.preprocessing as preproc

# Load the Online News Popularity dataset
df = pd.read_csv('OnlineNewsPopularity.csv', delimiter=', ')

# Look at the original data - the number of words in an article
df['n_tokens_content'].as_matrix()
# array([ 219.,  255.,  211., ...,  442.,  682.,  157.])

# Min-max scaling
df['minmax'] = preproc.minmax_scale(df[['n_tokens_content']])
# df['minmax'].as_matrix()
# array([ 0.02584376,  0.03009205,  0.02489969, ...,  0.05215955,
#         0.08048147,  0.01852726])

# Standardization - note that by definition, some outputs will be negative
df['standardized'] = preproc.StandardScaler().fit_transform(df[['n_tokens_content']])
df['standardized'].as_matrix()
# array([-0.69521045, -0.61879381, -0.71219192, ..., -0.2218518 ,
#         0.28759248, -0.82681689])

# L2-normalization
df['l2_normalized'] = preproc.normalize(df[['n_tokens_content']], axis=0)
df['l2_normalized'].as_matrix()
# array([ 0.00152439,  0.00177498,  0.00146871, ...,  0.00307663,
#         0.0047472 ,  0.00109283])

我们还可以使用不同的特征缩放方法可视化数据的分布（图 2-18）。如示例 2-16所示，与对数变换不同，特征缩放不会改变分布的形状；只有数据的规模发生了变化。

示例 2-16。绘制原始数据和缩放数据的直方图

fig, (ax1, ax2, ax3, ax4) = plt.subplots(4,1)
fig.tight_layout()
df['n_tokens_content'].hist(ax=ax1, bins=100)
ax1.tick_params(labelsize=14)
ax1.set_xlabel('Article word count', fontsize=14)
ax1.set_ylabel('Number of articles', fontsize=14)

df['minmax'].hist(ax=ax2, bins=100)
ax2.tick_params(labelsize=14)
ax2.set_xlabel('Min-max scaled word count', fontsize=14)
ax2.set_ylabel('Number of articles', fontsize=14)

df['standardized'].hist(ax=ax3, bins=100)
ax3.tick_params(labelsize=14)
ax3.set_xlabel('Standardized word count', fontsize=14)
ax3.set_ylabel('Number of articles', fontsize=14)

df['l2_normalized'].hist(ax=ax4, bins=100)
ax4.tick_params(labelsize=14)
ax4.set_xlabel('L2-normalized word count', fontsize=14)
ax4.set_ylabel('Number of articles', fontsize=14)

图 2-18。原始和按比例缩放的新闻文章字数——请注意，只有 x 轴的比例发生变化；分布的形状与特征缩放保持一致