scikit-learn实战项目从数据清洗到模型部署的全流程分析与问题解决指南

引言
scikit-learn是Python中最受欢迎的机器学习库之一，它提供了丰富的算法和工具，使得从数据预处理到模型部署的整个流程变得简单高效。本文将通过一个完整的实战项目，详细讲解使用scikit-learn进行机器学习项目的全流程，包括数据清洗、特征工程、模型训练、评估优化以及模型部署。我们将以经典的泰坦尼克号生存预测为例，逐步展示每个环节的操作和常见问题的解决方法。
1. 项目背景与数据理解
1.1 项目目标
泰坦尼克号生存预测是一个经典的二分类问题，目标是根据乘客的个人信息（如年龄、性别、船舱等级等）预测其是否在灾难中幸存。这个项目非常适合初学者，因为它涉及了数据清洗、特征工程、模型训练和评估等关键步骤。
1.2 数据集介绍
数据集包含以下主要字段：
PassengerId
: 乘客ID
Survived
: 是否幸存（0=否，1=是）
Pclass
: 船舱等级（1=头等舱，2=二等舱，3=三等舱）
Name
: 乘客姓名
Sex
: 性别
Age
: 年龄
SibSp
: 同行的兄弟姐妹/配偶数量
Parch
: 同行的父母/子女数量
Ticket
: 船票编号
Fare
: 票价
Cabin
: 船舱号
Embarked
: 登船港口（C=瑟堡，Q=皇后镇，S=南安普顿）
1.3 环境准备
首先，我们需要安装必要的库：
# 安装必要的库
!pip install pandas numpy matplotlib seaborn scikit-learn

# 导入常用库
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.model_selection import train_test_split, cross_val_score, GridSearchCV
from sklearn.preprocessing import StandardScaler, LabelEncoder, OneHotEncoder
from sklearn.impute import SimpleImputer
from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, classification_report, confusion_matrix, roc_auc_score
from sklearn.pipeline import Pipeline
import joblib
import warnings
warnings.filterwarnings(‘ignore’)

# 设置绘图风格
plt.style.use(‘seaborn-v0_8’)
sns.set_palette(“husl”)
2. 数据加载与初步探索
2.1 数据加载
# 加载数据
train_df = pd.read_csv(‘train.csv’)
test_df = pd.read_csv(‘test.csv’)

# 查看数据基本信息
print(“训练集形状:”, train_df.shape)
print(“测试集形状:”, test_df.shape)
print(“\n训练集前5行:”)
print(train_df.head())
print(“\n训练集信息:”)
print(train_df.info())
2.2 数据探索性分析（EDA）
# 查看数值型特征的统计信息
print(train_df.describe())

# 查看缺失值情况
print(“\n训练集缺失值统计:”)
print(train_df.isnull().sum())

# 可视化缺失值
plt.figure(figsize=(10, 6))
sns.heatmap(train_df.isnull(), cbar=False, cmap=’viridis’)
plt.title(‘训练集缺失值分布’)
plt.show()

# 目标变量分布
plt.figure(figsize=(8, 5))
sns.countplot(x=’Survived’, data=train_df)
plt.title(‘幸存者分布’)
plt.show()

# 数值型特征分布
fig, axes = plt.subplots(1, 3, figsize=(15, 5))
sns.histplot(train_df[‘Age’].dropna(), kde=True, ax=axes[0])
axes[0].set_title(‘年龄分布’)
sns.histplot(train_df[‘Fare’], kde=True, ax=axes[1])
axes[1].set_title(‘票价分布’)
sns.countplot(x=’Pclass’, data=train_df, ax=axes[2])
axes[2].set_title(‘船舱等级分布’)
plt.tight_layout()
plt.show()

# 分类特征与目标的关系
fig, axes = plt.subplots(2, 2, figsize=(12, 10))
sns.countplot(x=’Sex’, hue=’Survived’, data=train_df, ax=axes[0, 0])
axes[0, 0].set_title(‘性别与幸存关系’)
sns.countplot(x=’Pclass’, hue=’Survived’, data=train_df, ax=axes[0, 1])
axes[0, 1].set_title(‘船舱等级与幸存关系’)
sns.countplot(x=’Embarked’, hue=’Survived’, data=train_df, ax=axes[1, 0])
axes[1, 0].set_title(‘登船港口与幸存关系’)
sns.boxplot(x=’Survived’, y=’Age’, data=train_df, ax=axes[1, 1])
axes[1, 1].set_title(‘年龄与幸存关系’)
plt.tight_layout()
plt.show()
2.3 常见问题与解决方案
问题1：数据缺失严重
Age
缺失约20%
Cabin
缺失约77%
Embarked
缺失2个
解决方案
：
对于Age，使用中位数或均值填充，或使用更复杂的方法（如基于其他特征的预测）
对于Cabin，由于缺失率太高，可以考虑删除或创建新特征（如是否已知船舱）
对于Embarked，由于只有2个缺失，可以用众数填充
3. 数据清洗与预处理
3.1 处理缺失值
# 合并训练集和测试集以便统一处理
train_df[‘is_train’] = 1
test_df[‘is_train’] = 0
test_df[‘Survived’] = -1 # 测试集没有Survived标签
full_df = pd.concat([train_df, test_df], ignore_index=True)

# 1. 处理Age缺失值 – 使用中位数填充
full_df[‘Age’] = full_df[‘Age’].fillna(full_df[‘Age’].median())

# 2. 处理Embarked缺失值 – 使用众数填充
full_df[‘Embarked’] = full_df[‘Embarked’].fillna(full_df[‘Embarked’].mode()[0])

# 3. 处理Fare缺失值 – 使用中位数填充
full_df[‘Fare’] = full_df[‘Fare’].fillna(full_df[‘Fare’].median())

# 4. 处理Cabin缺失值 – 创建新特征
full_df[‘HasCabin’] = full_df[‘Cabin’].apply(lambda x: 0 if pd.isna(x) else 1)
full_df = full_df.drop(‘Cabin’, axis=1)

# 验证缺失值处理结果
print(“处理后缺失值统计:”)
print(full_df.isnull().sum())
3.2 特征工程
# 1. 从Name中提取Title（称呼）
def extract_title(name):
title_search = re.search(‘ ([A-Za-z]+)\.’, name)
if title_search:
return title_search.group(1)
else:
return “”

import re
full_df[‘Title’] = full_df[‘Name’].apply(extract_title)

# 合并稀有Title
title_mapping = {“Mr”: 0, “Miss”: 1, “Mrs”: 2, “Master”: 3, “Dr”: 4, “Rev”: 4, “Col”: 4, “Major”: 4, “Mlle”: 1, “Countess”: 2, “Ms”: 1, “Lady”: 2, “Jonkheer”: 4, “Don”: 4, “Dona”: 2, “Mme”: 2, “Capt”: 4, “Sir”: 4}
full_df[‘Title’] = full_df[‘Title’].map(title_mapping)

# 2. 创建家庭大小特征
full_df[‘FamilySize’] = full_df[‘SibSp’] + full_df[‘Parch’] + 1
full_df[‘IsAlone’] = (full_df[‘FamilySize’] == 1).astype(int)

# 3. 创建年龄分组特征
full_df[‘AgeGroup’] = pd.cut(full_df[‘Age’], bins=[0, 12, 18, 35, 60, 100], labels=[0, 1, 2, 3, 4])
full_df[‘AgeGroup’] = full_df[‘AgeGroup’].astype(int)

# 4. 创建票价分组特征
full_df[‘FareGroup’] = pd.qcut(full_df[‘Fare’], q=4, labels=[0, 1, 2, 3])
full_df[‘FareGroup’] = full_df[‘FareGroup’].astype(int)

# 5. 创建船舱等级与性别的交互特征
full_df[‘Pclass_Sex’] = full_df[‘Pclass’].astype(str) + ‘_’ + full_df[‘Sex’]

# 6. 删除不需要的原始特征
drop_columns = [‘Name’, ‘Ticket’, ‘PassengerId’]
full_df = full_df.drop(drop_columns, axis=1)

print(“特征工程后的数据形状:”, full_df.shape)
print(“\n特征工程后的数据前5行:”)
print(full_df.head())
3.3 分类特征编码
# 分离训练集和测试集
train_processed = full_df[full_df[‘is_train’] == 1].drop(‘is_train’, axis=1)
test_processed = full_df[full_df[‘is_train’] == 0].drop([‘is_train’, ‘Survived’], axis=1)

# 对分类特征进行编码
categorical_features = [‘Sex’, ‘Embarked’, ‘Pclass_Sex’]
numerical_features = [‘Age’, ‘Fare’, ‘SibSp’, ‘Parch’, ‘FamilySize’, ‘IsAlone’, ‘AgeGroup’, ‘FareGroup’, ‘Title’, ‘HasCabin’]

# 使用OneHotEncoder处理分类特征
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import OneHotEncoder

preprocessor = ColumnTransformer(
transformers=[
(‘num’, StandardScaler(), numerical_features),
(‘cat’, OneHotEncoder(handle_unknown=’ignore’), categorical_features)
])

# 应用预处理
X_train = train_processed.drop(‘Survived’, axis=1)
y_train = train_processed[‘Survived’]
X_test = test_processed

X_train_processed = preprocessor.fit_transform(X_train)
X_test_processed = preprocessor.transform(X_test)

print(“训练集特征形状:”, X_train_processed.shape)
print(“测试集特征形状:”, X_test_processed.shape)
4. 模型训练与评估
4.1 划分训练集和验证集
# 划分训练集和验证集
X_train_split, X_val_split, y_train_split, y_val_split = train_test_split(
X_train_processed, y_train, test_size=0.2, random_state=42, stratify=y_train
)

print(“训练集大小:”, X_train_split.shape)
print(“验证集大小:”, X_val_split.shape)
4.2 基准模型训练
# 定义模型
models = {
‘Logistic Regression’: LogisticRegression(random_state=42, max_iter=1000),
‘Random Forest’: RandomForestClassifier(random_state=42),
‘Gradient Boosting’: GradientBoostingClassifier(random_state=42)
}

# 训练和评估模型
results = {}
for name, model in models.items():
model.fit(X_train_split, y_train_split)
y_pred = model.predict(X_val_split)
accuracy = accuracy_score(y_val_split, y_pred)
results[name] = accuracy
print(f”{name} – 验证集准确率: {accuracy:.4f}”)

# 可视化结果
plt.figure(figsize=(10, 6))
plt.bar(results.keys(), results.values())
plt.title(‘模型准确率比较’)
plt.ylabel(‘准确率’)
plt.ylim(0.7, 0.9)
plt.show()
4.3 模型评估与分析
# 选择最佳模型（假设Random Forest表现最好）
best_model = models[‘Random Forest’]

# 详细评估
y_pred = best_model.predict(X_val_split)
y_pred_proba = best_model.predict_proba(X_val_split)[:, 1]

print(“分类报告:”)
print(classification_report(y_val_split, y_pred))

# 混淆矩阵
cm = confusion_matrix(y_val_split, y_pred)
plt.figure(figsize=(8, 6))
sns.heatmap(cm, annot=True, fmt=’d’, cmap=’Blues’,
xticklabels=[‘未幸存’, ‘幸存’],
yticklabels=[‘未幸存’, ‘幸存’])
plt.title(‘混淆矩阵’)
plt.ylabel(‘真实标签’)
plt.xlabel(‘预测标签’)
plt.show()

# ROC曲线
from sklearn.metrics import roc_curve, auc
fpr, tpr, thresholds = roc_curve(y_val_split, y_pred_proba)
roc_auc = auc(fpr, tpr)

plt.figure(figsize=(8, 6))
plt.plot(fpr, tpr, color=’darkorange’, lw=2, label=f’ROC曲线 (AUC = {roc_auc:.4f})’)
plt.plot([0, 1], [0, 1], color=’navy’, lw=2, linestyle=’–‘)
plt.xlim([0.0, 1.0])
plt.ylim([0.0, 1.05])
plt.xlabel(‘假正率’)
plt.ylabel(‘真正率’)
plt.title(‘ROC曲线’)
plt.legend(loc=”lower right”)
plt.show()
4.4 特征重要性分析
# 获取特征重要性
feature_names = numerical_features + list(preprocessor.named_transformers_[‘cat’].get_feature_names_out(categorical_features))
importances = best_model.feature_importances_

# 创建特征重要性DataFrame
feature_importance_df = pd.DataFrame({
‘Feature’: feature_names,
‘Importance’: importances
}).sort_values(‘Importance’, ascending=False)

# 可视化
plt.figure(figsize=(12, 8))
sns.barplot(x=’Importance’, y=’Feature’, data=feature_importance_df.head(15))
plt.title(‘特征重要性排名（前15）’)
plt.xlabel(‘重要性’)
plt.show()

print(“特征重要性前10:”)
print(feature_importance_df.head(10))
5. 模型优化与调参
5.1 超参数调优
# 定义参数网格
param_grid = {
‘n_estimators’: [100, 200, 300],
‘max_depth’: [None, 10, 20, 30],
‘min_samples_split’: [2, 5, 10],
‘min_samples_leaf’: [1, 2, 4]
}

# 使用GridSearchCV进行调优
rf = RandomForestClassifier(random_state=42)
grid_search = GridSearchCV(
rf,
param_grid,
cv=5,
scoring=’accuracy’,
n_jobs=-1,
verbose=1
)

grid_search.fit(X_train_split, y_train_split)

print(“最佳参数:”, grid_search.best_params_)
print(“最佳交叉验证分数:”, grid_search.best_score_)

# 使用最佳参数训练模型
best_rf = grid_search.best_estimator_
5.2 集成学习
# 使用Stacking集成方法
from sklearn.ensemble import StackingClassifier
from sklearn.svm import SVC

# 定义基础模型
base_models = [
(‘rf’, RandomForestClassifier(n_estimators=200, random_state=42)),
(‘gb’, GradientBoostingClassifier(random_state=42)),
(‘lr’, LogisticRegression(random_state=42, max_iter=1000))
]

# 定义元模型
meta_model = LogisticRegression(random_state=42)

# 创建Stacking分类器
stacking_clf = StackingClassifier(
estimators=base_models,
final_estimator=meta_model,
cv=5
)

# 训练Stacking模型
stacking_clf.fit(X_train_split, y_train_split)

# 评估Stacking模型
y_pred_stacking = stacking_clf.predict(X_val_split)
print(“Stacking模型准确率:”, accuracy_score(y_val_split, y_pred_stacking))
5.3 交叉验证
# 使用交叉验证评估模型
cv_scores = cross_val_score(best_rf, X_train_processed, y_train, cv=5, scoring=’accuracy’)
print(“交叉验证分数:”, cv_scores)
print(“平均准确率:”, cv_scores.mean())
print(“标准差:”, cv_scores.std())
6. 模型部署
6.1 保存模型和预处理管道
# 创建完整的预处理和模型管道
from sklearn.pipeline import Pipeline

# 重新创建预处理步骤
preprocessor = ColumnTransformer(
transformers=[
(‘num’, StandardScaler(), numerical_features),
(‘cat’, OneHotEncoder(handle_unknown=’ignore’), categorical_features)
])

# 创建完整管道
full_pipeline = Pipeline([
(‘preprocessor’, preprocessor),
(‘classifier’, best_rf) # 使用调优后的随机森林
])

# 在完整训练集上训练
full_pipeline.fit(X_train, y_train)

# 保存模型
joblib.dump(full_pipeline, ‘titanic_model.pkl’)
print(“模型已保存为 titanic_model.pkl”)

# 保存特征名称和预处理信息
model_info = {
‘numerical_features’: numerical_features,
‘categorical_features’: categorical_features,
‘preprocessor’: preprocessor,
‘model’: best_rf
}
joblib.dump(model_info, ‘model_info.pkl’)
6.2 创建预测API
# 创建一个简单的预测函数
def predict_survival(passenger_data):
“””
预测乘客是否幸存

参数:
passenger_data: dict, 包含乘客信息的字典

返回:
dict: 包含预测结果和概率的字典
“””
# 加载模型
model = joblib.load(‘titanic_model.pkl’)

# 将字典转换为DataFrame
df = pd.DataFrame([passenger_data])

# 应用相同的特征工程
df[‘HasCabin’] = df[‘Cabin’].apply(lambda x: 0 if pd.isna(x) else 1)
df = df.drop(‘Cabin’, axis=1)

# 提取Title
df[‘Title’] = df[‘Name’].apply(extract_title)
df[‘Title’] = df[‘Title’].map(title_mapping)

# 创建家庭大小特征
df[‘FamilySize’] = df[‘SibSp’] + df[‘Parch’] + 1
df[‘IsAlone’] = (df[‘FamilySize’] == 1).astype(int)

# 年龄分组
df[‘AgeGroup’] = pd.cut(df[‘Age’], bins=[0, 12, 18, 35, 60, 100], labels=[0, 1, 2, 3, 4])
df[‘AgeGroup’] = df[‘AgeGroup’].astype(int)

# 票价分组
df[‘FareGroup’] = pd.qcut(df[‘Fare’], q=4, labels=[0, 1, 2, 3])
df[‘FareGroup’] = df[‘FareGroup’].astype(int)

# 船舱等级与性别交互
df[‘Pclass_Sex’] = df[‘Pclass’].astype(str) + ‘_’ + df[‘Sex’]

# 删除不需要的列
drop_columns = [‘Name’, ‘Ticket’]
df = df.drop(drop_columns, axis=1)

# 预测
prediction = model.predict(df)
probability = model.predict_proba(df)

return {
‘survived’: int(prediction[0]),
‘probability’: float(probability[0][1]),
‘message’: ‘幸存’ if prediction[0] == 1 else ‘未幸存’
}

# 测试预测函数
test_passenger = {
‘Pclass’: 1,
‘Name’: ‘John Smith’,
‘Sex’: ‘male’,
‘Age’: 30,
‘SibSp’: 0,
‘Parch’: 0,
‘Ticket’: ‘A/5 21171’,
‘Fare’: 7.25,
‘Cabin’: ‘C85’,
‘Embarked’: ‘S’
}

result = predict_survival(test_passenger)
print(“预测结果:”, result)
6.3 部署到Web应用（Flask示例）
# 创建Flask应用
from flask import Flask, request, jsonify
import pandas as pd
import joblib
import re

app = Flask(__name__)

# 加载模型
model = joblib.load(‘titanic_model.pkl’)

# 特征工程函数（与之前相同）
def extract_title(name):
title_search = re.search(‘ ([A-Za-z]+)\.’, name)
if title_search:
return title_search.group(1)
else:
return “”

title_mapping = {“Mr”: 0, “Miss”: 1, “Mrs”: 2, “Master”: 3, “Dr”: 4, “Rev”: 4, “Col”: 4, “Major”: 4, “Mlle”: 1, “Countess”: 2, “Ms”: 1, “Lady”: 2, “Jonkheer”: 4, “Don”: 4, “Dona”: 2, “Mme”: 2, “Capt”: 4, “Sir”: 4}

@app.route(‘/predict’, methods=[‘POST’])
def predict():
try:
# 获取JSON数据
data = request.get_json()

# 验证必需的字段
required_fields = [‘Pclass’, ‘Name’, ‘Sex’, ‘Age’, ‘SibSp’, ‘Parch’, ‘Fare’, ‘Embarked’]
for field in required_fields:
if field not in data:
return jsonify({‘error’: f’Missing required field: {field}’}), 400

# 转换为DataFrame
df = pd.DataFrame([data])

# 应用特征工程
df[‘HasCabin’] = df[‘Cabin’].apply(lambda x: 0 if pd.isna(x) else 1)
df = df.drop(‘Cabin’, axis=1)

df[‘Title’] = df[‘Name’].apply(extract_title)
df[‘Title’] = df[‘Title’].map(title_mapping)

df[‘FamilySize’] = df[‘SibSp’] + df[‘Parch’] + 1
df[‘IsAlone’] = (df[‘FamilySize’] == 1).astype(int)

df[‘AgeGroup’] = pd.cut(df[‘Age’], bins=[0, 12, 18, 35, 60, 100], labels=[0, 1, 2, 3, 4])
df[‘AgeGroup’] = df[‘AgeGroup’].astype(int)

df[‘FareGroup’] = pd.qcut(df[‘Fare’], q=4, labels=[0, 1, 2, 3])
df[‘FareGroup’] = df[‘FareGroup’].astype(int)

df[‘Pclass_Sex’] = df[‘Pclass’].astype(str) + ‘_’ + df[‘Sex’]

# 删除不需要的列
drop_columns = [‘Name’, ‘Ticket’]
df = df.drop(drop_columns, axis=1)

# 预测
prediction = model.predict(df)
probability = model.predict_proba(df)

return jsonify({
‘survived’: int(prediction[0]),
‘probability’: float(probability[0][1]),
‘message’: ‘幸存’ if prediction[0] == 1 else ‘未幸存’
})

except Exception as e:
return jsonify({‘error’: str(e)}), 500

if __name__ == ‘__main__’:
app.run(debug=True, port=5000)
6.4 部署到生产环境（Docker示例）
# Dockerfile
FROM python:3.9-slim

WORKDIR /app

# 安装依赖
COPY requirements.txt .
RUN pip install –no-cache-dir -r requirements.txt

# 复制应用代码
COPY . .

# 暴露端口
EXPOSE 5000

# 启动应用
CMD [“python”, “app.py”]
# requirements.txt
pandas==1.5.3
numpy==1.24.3
scikit-learn==1.2.2
flask==2.3.2
joblib==1.2.0
7. 常见问题与解决方案
7.1 数据不平衡问题
问题
：在泰坦尼克号数据集中，幸存者和非幸存者的比例约为38:62，虽然不算严重不平衡，但在实际项目中可能遇到更严重的不平衡。
解决方案
：
# 方法1：使用class_weight参数
from sklearn.utils.class_weight import compute_class_weight

class_weights = compute_class_weight(‘balanced’, classes=np.unique(y_train), y=y_train)
class_weight_dict = dict(enumerate(class_weights))

# 在模型中使用
rf_balanced = RandomForestClassifier(class_weight=class_weight_dict, random_state=42)

# 方法2：使用过采样/欠采样
from imblearn.over_sampling import SMOTE
from imblearn.under_sampling import RandomUnderSampler

# SMOTE过采样
smote = SMOTE(random_state=42)
X_train_smote, y_train_smote = smote.fit_resample(X_train_split, y_train_split)

# 随机欠采样
under = RandomUnderSampler(random_state=42)
X_train_under, y_train_under = under.fit_resample(X_train_split, y_train_split)
7.2 特征选择问题
问题
：特征过多可能导致过拟合，特征过少可能丢失重要信息。
解决方案
：
# 方法1：基于模型的特征选择
from sklearn.feature_selection import SelectFromModel

selector = SelectFromModel(RandomForestClassifier(n_estimators=100, random_state=42))
selector.fit(X_train_split, y_train_split)
X_train_selected = selector.transform(X_train_split)
X_val_selected = selector.transform(X_val_split)

# 方法2：递归特征消除（RFE）
from sklearn.feature_selection import RFE

rfe = RFE(estimator=RandomForestClassifier(n_estimators=100, random_state=42), n_features_to_select=10)
rfe.fit(X_train_split, y_train_split)
X_train_rfe = rfe.transform(X_train_split)
X_val_rfe = rfe.transform(X_val_split)

# 方法3：使用SelectKBest
from sklearn.feature_selection import SelectKBest, f_classif

selector_kbest = SelectKBest(score_func=f_classif, k=10)
X_train_kbest = selector_kbest.fit_transform(X_train_split, y_train_split)
X_val_kbest = selector_kbest.transform(X_val_split)
7.3 模型过拟合问题
问题
：模型在训练集上表现很好，但在验证集上表现差。
解决方案
：
# 方法1：增加正则化
from sklearn.linear_model import LogisticRegression

# 增加L2正则化强度
lr_regularized = LogisticRegression(penalty=’l2′, C=0.1, random_state=42)

# 方法2：使用交叉验证
from sklearn.model_selection import cross_val_score

cv_scores = cross_val_score(RandomForestClassifier(random_state=42),
X_train_processed, y_train, cv=5)
print(“交叉验证分数:”, cv_scores)
print(“平均分数:”, cv_scores.mean())

# 方法3：早停法（适用于梯度提升）
from sklearn.ensemble import GradientBoostingClassifier

gb_early_stop = GradientBoostingClassifier(
n_estimators=1000,
learning_rate=0.1,
random_state=42,
validation_fraction=0.1,
n_iter_no_change=10,
tol=0.001
)
7.4 模型部署问题
问题
：模型在开发环境运行良好，但在生产环境出现错误。
解决方案
：
# 1. 版本控制
import joblib
import json

# 保存模型和元数据
model_version = {
‘model’: model,
‘version’: ‘1.0.0’,
‘timestamp’: ‘2023-10-01’,
‘features’: feature_names,
‘metrics’: {
‘accuracy’: 0.85,
‘auc’: 0.90
}
}

joblib.dump(model_version, ‘model_v1.0.0.pkl’)

# 2. 输入验证
def validate_input(data):
“””验证输入数据是否符合预期格式”””
required_types = {
‘Pclass’: int,
‘Age’: (int, float),
‘Fare’: (int, float),
‘SibSp’: int,
‘Parch’: int
}

for field, expected_type in required_types.items():
if field in data:
if not isinstance(data[field], expected_type):
raise ValueError(f”Field {field} must be of type {expected_type}”)

return True

# 3. 监控和日志
import logging
from datetime import datetime

logging.basicConfig(filename=’model_predictions.log’, level=logging.INFO)

def log_prediction(input_data, prediction, probability):
timestamp = datetime.now().isoformat()
log_entry = {
‘timestamp’: timestamp,
‘input’: input_data,
‘prediction’: prediction,
‘probability’: probability
}
logging.info(json.dumps(log_entry))
8. 性能优化与最佳实践
8.1 代码优化
# 使用NumPy向量化操作代替循环
# 不推荐
def slow_feature_engineering(df):
result = []
for i in range(len(df)):
if df.iloc[i][‘Age’] < 12: result.append(0) elif df.iloc[i]['Age'] < 18: result.append(1) else: result.append(2) return result # 推荐 def fast_feature_engineering(df): conditions = [ df['Age'] < 12, (df['Age'] >= 12) & (df[‘Age’] < 18), df['Age'] >= 18
]
choices = [0, 1, 2]
return np.select(conditions, choices, default=2)
8.2 内存优化
# 使用适当的数据类型减少内存占用
def optimize_memory(df):
# 将float64转换为float32
float_cols = df.select_dtypes(include=[‘float64’]).columns
df[float_cols] = df[float_cols].astype(‘float32’)

# 将int64转换为int32或int8
int_cols = df.select_dtypes(include=[‘int64’]).columns
for col in int_cols:
if df[col].max() < 127: df[col] = df[col].astype('int8') elif df[col].max() < 32767: df[col] = df[col].astype('int16') else: df[col] = df[col].astype('int32') return df # 使用category类型处理分类特征 def optimize_categorical(df): categorical_cols = df.select_dtypes(include=['object']).columns for col in categorical_cols: df[col] = df[col].astype('category') return df 8.3 并行处理 from joblib import Parallel, delayed import time # 并行化模型训练 def train_model_parallel(model, X, y, n_jobs=-1): """并行训练模型""" from sklearn.model_selection import cross_val_score if n_jobs == -1: n_jobs = None # 使用所有CPU核心 scores = cross_val_score(model, X, y, cv=5, n_jobs=n_jobs) return scores.mean() # 并行化特征工程 def parallel_feature_engineering(df, feature_funcs): """并行执行多个特征工程函数""" results = Parallel(n_jobs=-1)( delayed(func)(df) for func in feature_funcs ) return results 9. 项目总结与扩展 9.1 项目成果 通过本项目，我们完成了： 数据清洗 ：处理了缺失值，创建了新特征 特征工程 ：从原始数据中提取了有意义的特征 模型训练 ：训练了多个模型并进行了比较 模型优化 ：通过调参和集成学习提高了性能 模型部署 ：创建了预测API并部署到Web应用 9.2 性能指标 基准模型准确率 ：约82% 调优后模型准确率 ：约86% Stacking集成模型准确率 ：约87% 9.3 扩展方向 更多特征工程 ：尝试更复杂的特征组合 深度学习 ：使用神经网络进行预测 自动化机器学习 ：使用AutoML工具（如Auto-sklearn） 实时预测 ：部署到云服务（如AWS SageMaker） 模型监控 ：建立模型性能监控系统 9.4 最佳实践总结 始终从数据探索开始 ：理解数据分布和缺失情况 特征工程是关键 ：好的特征比复杂的模型更重要 使用交叉验证 ：避免过拟合，获得可靠的性能估计 保存完整管道 ：确保训练和预测的一致性 考虑生产环境 ：包括错误处理、日志记录和监控 10. 完整代码示例 # 完整的端到端项目代码 import pandas as pd import numpy as np import re import joblib from sklearn.model_selection import train_test_split, GridSearchCV from sklearn.preprocessing import StandardScaler, OneHotEncoder from sklearn.compose import ColumnTransformer from sklearn.pipeline import Pipeline from sklearn.ensemble import RandomForestClassifier from sklearn.metrics import accuracy_score, classification_report class TitanicPredictor: """泰坦尼克号生存预测器""" def __init__(self): self.model = None self.preprocessor = None self.feature_names = None def load_data(self, train_path, test_path): """加载数据""" self.train_df = pd.read_csv(train_path) self.test_df = pd.read_csv(test_path) def preprocess_data(self): """数据预处理""" # 合并数据 self.train_df['is_train'] = 1 self.test_df['is_train'] = 0 self.test_df['Survived'] = -1 full_df = pd.concat([self.train_df, self.test_df], ignore_index=True) # 处理缺失值 full_df['Age'] = full_df['Age'].fillna(full_df['Age'].median()) full_df['Embarked'] = full_df['Embarked'].fillna(full_df['Embarked'].mode()[0]) full_df['Fare'] = full_df['Fare'].fillna(full_df['Fare'].median()) full_df['HasCabin'] = full_df['Cabin'].apply(lambda x: 0 if pd.isna(x) else 1) full_df = full_df.drop('Cabin', axis=1) # 特征工程 full_df['Title'] = full_df['Name'].apply(self.extract_title) title_mapping = {"Mr": 0, "Miss": 1, "Mrs": 2, "Master": 3, "Dr": 4, "Rev": 4, "Col": 4, "Major": 4, "Mlle": 1, "Countess": 2, "Ms": 1, "Lady": 2, "Jonkheer": 4, "Don": 4, "Dona": 2, "Mme": 2, "Capt": 4, "Sir": 4} full_df['Title'] = full_df['Title'].map(title_mapping) full_df['FamilySize'] = full_df['SibSp'] + full_df['Parch'] + 1 full_df['IsAlone'] = (full_df['FamilySize'] == 1).astype(int) full_df['AgeGroup'] = pd.cut(full_df['Age'], bins=[0, 12, 18, 35, 60, 100], labels=[0, 1, 2, 3, 4]) full_df['AgeGroup'] = full_df['AgeGroup'].astype(int) full_df['FareGroup'] = pd.qcut(full_df['Fare'], q=4, labels=[0, 1, 2, 3]) full_df['FareGroup'] = full_df['FareGroup'].astype(int) full_df['Pclass_Sex'] = full_df['Pclass'].astype(str) + '_' + full_df['Sex'] # 分离数据集 self.train_processed = full_df[full_df['is_train'] == 1].drop('is_train', axis=1) self.test_processed = full_df[full_df['is_train'] == 0].drop(['is_train', 'Survived'], axis=1) # 定义特征 self.numerical_features = ['Age', 'Fare', 'SibSp', 'Parch', 'FamilySize', 'IsAlone', 'AgeGroup', 'FareGroup', 'Title', 'HasCabin'] self.categorical_features = ['Sex', 'Embarked', 'Pclass_Sex'] def extract_title(self, name): """从姓名中提取Title""" title_search = re.search(' ([A-Za-z]+)\.', name) if title_search: return title_search.group(1) else: return "" def train_model(self): """训练模型""" X = self.train_processed.drop('Survived', axis=1) y = self.train_processed['Survived'] # 创建预处理管道 self.preprocessor = ColumnTransformer( transformers=[ ('num', StandardScaler(), self.numerical_features), ('cat', OneHotEncoder(handle_unknown='ignore'), self.categorical_features) ]) # 创建完整管道 pipeline = Pipeline([ ('preprocessor', self.preprocessor), ('classifier', RandomForestClassifier(random_state=42)) ]) # 划分训练集和验证集 X_train, X_val, y_train, y_val = train_test_split( X, y, test_size=0.2, random_state=42, stratify=y ) # 训练模型 pipeline.fit(X_train, y_train) # 评估模型 y_pred = pipeline.predict(X_val) accuracy = accuracy_score(y_val, y_pred) print(f"验证集准确率: {accuracy:.4f}") print("\n分类报告:") print(classification_report(y_val, y_pred)) # 保存模型 self.model = pipeline joblib.dump(pipeline, 'titanic_model_final.pkl') print("模型已保存") def predict(self, passenger_data): """预测单个乘客""" if self.model is None: self.model = joblib.load('titanic_model_final.pkl') df = pd.DataFrame([passenger_data]) # 应用相同的特征工程 df['HasCabin'] = df['Cabin'].apply(lambda x: 0 if pd.isna(x) else 1) df = df.drop('Cabin', axis=1) df['Title'] = df['Name'].apply(self.extract_title) title_mapping = {"Mr": 0, "Miss": 1, "Mrs": 2, "Master": 3, "Dr": 4, "Rev": 4, "Col": 4, "Major": 4, "Mlle": 1, "Countess": 2, "Ms": 1, "Lady": 2, "Jonkheer": 4, "Don": 4, "Dona": 2, "Mme": 2, "Capt": 4, "Sir": 4} df['Title'] = df['Title'].map(title_mapping) df['FamilySize'] = df['SibSp'] + df['Parch'] + 1 df['IsAlone'] = (df['FamilySize'] == 1).astype(int) df['AgeGroup'] = pd.cut(df['Age'], bins=[0, 12, 18, 35, 60, 100], labels=[0, 1, 2, 3, 4]) df['AgeGroup'] = df['AgeGroup'].astype(int) df['FareGroup'] = pd.qcut(df['Fare'], q=4, labels=[0, 1, 2, 3]) df['FareGroup'] = df['FareGroup'].astype(int) df['Pclass_Sex'] = df['Pclass'].astype(str) + '_' + df['Sex'] # 删除不需要的列 drop_columns = ['Name', 'Ticket'] df = df.drop(drop_columns, axis=1) # 预测 prediction = self.model.predict(df) probability = self.model.predict_proba(df) return { 'survived': int(prediction[0]), 'probability': float(probability[0][1]), 'message': '幸存' if prediction[0] == 1 else '未幸存' } # 使用示例 if __name__ == "__main__": # 创建预测器实例 predictor = TitanicPredictor() # 加载数据 predictor.load_data('train.csv', 'test.csv') # 预处理数据 predictor.preprocess_data() # 训练模型 predictor.train_model() # 测试预测 test_passenger = { 'Pclass': 1, 'Name': 'John Smith', 'Sex': 'male', 'Age': 30, 'SibSp': 0, 'Parch': 0, 'Ticket': 'A/5 21171', 'Fare': 7.25, 'Cabin': 'C85', 'Embarked': 'S' } result = predictor.predict(test_passenger) print("\n测试预测结果:", result) 结语 通过本指南，我们详细介绍了使用scikit-learn进行机器学习项目的完整流程。从数据清洗、特征工程到模型训练、评估和部署，每个环节都提供了详细的代码示例和问题解决方案。泰坦尼克号项目虽然简单，但涵盖了机器学习项目的核心步骤，是学习和实践的绝佳起点。 记住，机器学习项目是一个迭代的过程，需要不断尝试、评估和优化。希望本指南能帮助你更好地理解和应用scikit-learn，构建出高质量的机器学习模型。