{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 结课项目\n",
    "\n",
    "在此项目中，你将扮演侦探，运用你的机器学习技能构建一个算法，通过公开的安然财务和邮件数据集，找出有欺诈嫌疑的安然雇员。\n",
    "\n",
    "## 我要如何完成此项目？\n",
    "\n",
    "在开始之前，你应该注意，此迷你项目需要大量数据点才能给出直观的结果，并且良好地运行起来。 **此项目更为棘手的原因在于，我们使用了真实的数据，这些数据可以是杂乱无章的，而且在进行机器学习时不具有我们所希望的大量数据点。** \n",
    "\n",
    "不要失去信心——作为数据分析师，你只需要习惯不完美的数据！如果你遇到之前没有见过的事物，请退后一步想想聪明的解决之道。要相信自己！\n",
    "\n",
    "## 项目概述\n",
    "\n",
    "安然曾是 2000 年美国最大的公司之一。2002 年，由于其存在大量的企业欺诈行为，这个昔日的大集团土崩瓦解。 在随后联邦进行的调查过程中，大量有代表性的保密信息进入了公众的视线，包括成千上万涉及高管的邮件和详细的财务数据。 \n",
    "\n",
    "你将在此项目中扮演侦探，运用你的新技能，**根据安然丑闻中公开的财务和邮件数据来构建相关人士识别符。** 为了协助你进行侦查工作，我们已将数据与手动整理出来的欺诈案涉案人员列表进行了合并， 这意味着被起诉的人员要么达成和解，要么向政府签署认罪协议，再或者出庭作证以获得免受起诉的豁免权。\n",
    "\n",
    "### 相关文件如下所示：\n",
    "\n",
    "*poi_id.py*：用于 POI 识别符的初始代码，你将在此处撰写你的分析报告。你也将提交此文件的副本，用于评估人员检验你的算法和结果。\n",
    "\n",
    "*final_project_dataset.pkl*：项目数据集，详情如下。\n",
    "\n",
    "*tester.py*：在你提交供优达学城评估的分析报告时，你将随附算法、数据集和你使用的特征列表（这些是在 poi_id.py 中自动创建的）。 评估人员将在此后使用这一代码来测试你的结果，以确保性能与你在报告中所述类似。你无需处理这一代码，我们只是将它呈现出来供你参考。\n",
    "\n",
    "*emails_by_address*：该目录包含许多文本文件，每个文件又包含特定邮箱的往来邮件。 你可以进行参考，并且可以根据邮件数据集的详细信息创建更多的高级特征。你无需处理电子邮件语料库来完成项目。\n",
    "\n",
    "## 迈向成功\n",
    "\n",
    "我们将给予你可读入数据的初始代码，将你选择的特征放入 numpy 数组中，该数组是大多数 sklearn 函数假定的输入表单。 **你要做的就是设计特征，选择并调整算法，用以测试和评估识别符。** 我们在设计数个迷你项目之初就想到了这个最终的项目，因此请记得借助你已完成的工作成果。\n",
    "\n",
    "在预处理此项目时，我们已将安然邮件和财务数据与字典结合在一起，字典中的每对键值对应一个人。 字典键是人名，值是另一个字典（包含此人的所有特征名和对应的值）。 数据中的特征分为三大类，即财务特征、邮件特征和 POI 标签。\n",
    "\n",
    "**财务特征**: ['salary', 'deferral_payments', 'total_payments', 'loan_advances', 'bonus', 'restricted_stock_deferred', 'deferred_income', 'total_stock_value', 'expenses', 'exercised_stock_options', 'other', 'long_term_incentive', 'restricted_stock', 'director_fees'] (单位均是美元）\n",
    "\n",
    "**邮件特征**: ['to_messages', 'email_address', 'from_poi_to_this_person', 'from_messages', 'from_this_person_to_poi', 'shared_receipt_with_poi'] (单位通常是电子邮件的数量，明显的例外是 ‘email_address’，这是一个字符串）\n",
    "\n",
    "**POI 标签**: [‘poi’] (boolean，整数)\n",
    "\n",
    "我们鼓励你在启动器功能中制作，转换或重新调整新函数功能。如果这样做，你应该把新功能存储到my_dataset，如果你想在最终算法中使用新功能，你还应该将功能名称添加到 my_feature_list，以便于你的评估者可以在测试期间访问它。关于如何在数据集中添加具体的新要素的例子，可以参考“特征选择”这一课。\n",
    "\n",
    "此外，我们还建议你可以在完成项目过程中做一些记号。你可以写出系列问题的答案（在下一页），将这个作为提交的项目的一部分，以便于评估者了解到你对于不同方面分析的方法。你的思维过程在很大程度上比你的最终项目更重要，我们将通过你在这些问题的解答中了解你的思维过程。\n",
    "\n",
    "# 开始\n",
    "\n",
    "首先把所有需要导入的模块都放在第一个 cell，只需要导入一次。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-09-15T08:57:42.573006Z",
     "start_time": "2019-09-15T08:57:40.517186Z"
    }
   },
   "outputs": [],
   "source": [
    "#!/usr/bin/python\n",
    "\n",
    "# poi_id.py\n",
    "import sys\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.naive_bayes import GaussianNB\n",
    "from tester import dump_classifier_and_data\n",
    "sys.path.append(\"../tools/\")\n",
    "from feature_format import featureFormat, targetFeatureSplit\n",
    "import pickle\n",
    "from poi_email_addresses import poiEmails\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Task1：确定需要使用的特征\n",
    "\n",
    "这个部分我初步认为欺诈案嫌疑人的特征首先会体现在收入水平上，所以先使用薪资、奖金和总股票持有量这三个财务特征。稍后再看邮件特征。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-09-15T08:57:54.296374Z",
     "start_time": "2019-09-15T08:57:54.287372Z"
    }
   },
   "outputs": [],
   "source": [
    "# Task 1: Select what features you'll use.\n",
    "# features_list is a list of strings, each of which is a feature name.\n",
    "# The first feature must be \"poi\".\n",
    "\n",
    "features_list = ['poi', 'salary', 'bonus', 'total_payments'] # You will need to use more features\n",
    "\n",
    "# Load the dictionary containing the dataset\n",
    "with open(\"final_project_dataset.pkl\", \"rb\") as data_file:\n",
    "    data_dict = pickle.load(data_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Task2：移除数据中的异常值"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-09-15T08:59:49.667494Z",
     "start_time": "2019-09-15T08:59:48.491166Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 6348 characters]\n",
      "text/plain": [
       "<Figure size 1440x72 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 6916 characters]\n",
      "text/plain": [
       "<Figure size 1440x72 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 6408 characters]\n",
      "text/plain": [
       "<Figure size 1440x72 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Task 2: Remove outliers\n",
    "\n",
    "\n",
    "#这里定义了一个可以画出某个数据一维的分布情况的函数，方便观察一下异常值\n",
    "def draw_1d(*,dataset, feature_name, style = \"r.\", figsize = 20):\n",
    "    plt.figure(figsize=(figsize,1))\n",
    "    for name in dataset.keys():\n",
    "        f1 = dataset[name][feature_name]\n",
    "        if f1 != \"NaN\":\n",
    "            plt.plot(f1,0,style)\n",
    "    plt.title(feature_name)\n",
    "    plt.xlabel(\"Value\")\n",
    "    plt.show()\n",
    "\n",
    "draw_1d(dataset = data_dict,feature_name=\"salary\",style=\"bx\")\n",
    "draw_1d(dataset = data_dict,feature_name='total_payments')\n",
    "draw_1d(dataset = data_dict,feature_name='total_stock_value')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "首先遍历了整个字典中的人名，然后画出各种数据的一维图像，发现有一个点远超所有的数据，我要把他找出来看看是谁的工资和其他人差这么多个数量级。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-09-07T07:33:59.237214Z",
     "start_time": "2019-09-07T07:33:59.212915Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The biggest data: SKILLING JEFFREY K\n",
      "The people who has taken the highest amount: ['SKILLING JEFFREY K', 'LAY KENNETH L', 'FREVERT MARK A', 'PICKERING MARK R', 'WHALLEY LAWRENCE G', 'DERRICK JR. JAMES V', 'FASTOW ANDREW S', 'SHERRIFF JOHN R', 'RICE KENNETH D', 'CAUSEY RICHARD A', 'KEAN STEVEN J', 'HAEDICKE MARK E', 'MCMAHON JEFFREY', 'METTS MARK']\n"
     ]
    }
   ],
   "source": [
    "def find_max(ori_dataset, *, features, out_type=\"r\", ratio=0.1, num=1):\n",
    "\n",
    "    from copy import deepcopy\n",
    "    # 这里使用深层拷贝来防止原来的字典数据被这里的 NaN 值修改语句修改\n",
    "    dataset = deepcopy(ori_dataset)\n",
    "\n",
    "    for key in dataset.keys():\n",
    "        if dataset[key][features] == \"NaN\":\n",
    "            dataset[key][features] = 0.0\n",
    "\n",
    "    sort = sorted(dataset, reverse=True,\n",
    "                  key=lambda name: dataset[name][features])\n",
    "\n",
    "    if out_type == \"r\":\n",
    "        data_point_num = int(len(sort)*ratio)\n",
    "        #print(\"Should return %d data point(s)\" % data_point_num)\n",
    "    elif out_type == \"n\":\n",
    "        data_point_num = num\n",
    "\n",
    "    return sort[:data_point_num]\n",
    "\n",
    "\n",
    "print(\"The biggest data:\", find_max(\n",
    "    data_dict, features=\"salary\", out_type=\"n\", num=1)[0])\n",
    "\n",
    "print(\"The people who has taken the highest amount:\",find_max(data_dict, features=\"salary\", out_type=\"r\", ratio=0.1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "发现最大的那个异常值是名字为 TOTAL 的字典键，也就是说其实是数据库中求和过后的结果，我不太需要这个值，所以直接删掉，并且把删除这个值之后的数据保存为 dataset_processed.pkl"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-09-06T12:14:20.726454Z",
     "start_time": "2019-09-06T12:14:20.717299Z"
    }
   },
   "outputs": [],
   "source": [
    "# 数据只需要删除一次并且保存一下\n",
    "# data_dict.pop(\"TOTAL\")\n",
    "\n",
    "# with open('dataset_processed.pkl', 'wb') as dataset:\n",
    "#    pickle.dump(data_dict, dataset, protocol=pickle.HIGHEST_PROTOCOL)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-09-15T09:00:09.512022Z",
     "start_time": "2019-09-15T09:00:09.504954Z"
    }
   },
   "outputs": [],
   "source": [
    "#读入之前保存的删除了 TOTAL 的数据集\n",
    "with open(\"dataset_processed.pkl\", \"rb\") as data_file:\n",
    "    data_dict = pickle.load(data_file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-09-15T09:00:37.410984Z",
     "start_time": "2019-09-15T09:00:37.082927Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 9304 characters]\n",
      "text/plain": [
       "<Figure size 1440x72 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "draw_1d(dataset = data_dict,feature_name=\"salary\",style=\".\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-09-15T09:01:25.221720Z",
     "start_time": "2019-09-15T09:01:24.632969Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 10588 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "x_data = []\n",
    "for name in data_dict.keys():\n",
    "    f1 = data_dict[name]['salary']\n",
    "    f2 = data_dict[name]['total_stock_value']\n",
    "    if f1 != \"NaN\" and f2 != \"NaN\":\n",
    "        if f1 > 6e5 or f2 > 1e7:\n",
    "            x_data.append([name,f1, f2])\n",
    "            plt.scatter(f1,f2,marker=\"x\",c=\"r\")\n",
    "        else:\n",
    "            plt.scatter(f1,f2,marker=\".\",c=\"g\")\n",
    "plt.title(\"Features Visualization\")\n",
    "plt.ylabel(\"F1\")\n",
    "plt.xlabel(\"F2\")\n",
    "plt.show()\n",
    "# print(\"People who has taken the Highest amount:\")\n",
    "# for x in x_data:\n",
    "#    print(\"\\nName: %s\\nSalary: %d\\nStock: %d\" % (x[0],x[1],x[2]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-09-15T09:02:10.424615Z",
     "start_time": "2019-09-15T09:02:10.257014Z"
    }
   },
   "outputs": [],
   "source": [
    "# Task 3: Create new feature(s)\n",
    "\n",
    "# 这里想使用PCA对于财务方面的数据做一个分析，然后看看效果如何。\n",
    "# 所以可能要输入全部的特征给PCA才比较有效果\n",
    "from sklearn.decomposition import PCA\n",
    "# \n",
    "\n",
    "\n",
    "all_financial_features = ['poi', 'deferral_payments', 'total_payments', 'loan_advances',\n",
    "                          'bonus', 'restricted_stock_deferred', 'deferred_income',\n",
    "                          'total_stock_value', 'expenses', 'exercised_stock_options', \n",
    "                          'other', 'long_term_incentive', 'restricted_stock', 'director_fees']\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "finance_data = featureFormat(\n",
    "    data_dict, all_financial_features, remove_NaN=True)\n",
    "poi_labels, finance_features = targetFeatureSplit(finance_data)\n",
    "\n",
    "features_train, features_test ,labels_train, labels_test = train_test_split(finance_features, poi_labels, test_size = 0.3, random_state = 42)\n",
    "features_train = np.array(features_train)\n",
    "features_test = np.array(features_test)\n",
    "finance_features = np.array(finance_features)\n",
    "\n",
    "all_features_pca = np.array(finance_features)\n",
    "\n",
    "finance_pca = PCA(n_components=3, whiten=True)\n",
    "all_finance_pca = PCA(n_components=2, whiten=True)\n",
    "#all_pca.fit_\n",
    "all_features_pca = all_pca.fit_transform(finance_features)\n",
    "finance_features_train = pca.fit_transform(features_train)\n",
    "finance_features_test = pca.transform(features_test)\n",
    "\n",
    "\n",
    "print(pca.explained_variance_)\n",
    "print(pca.explained_variance_ratio_)\n",
    "# Store to my_dataset for easy export below.\n",
    "#my_dataset = data_dict\n",
    "\n",
    "\n",
    "# Extract features and labels from dataset for local testing\n",
    "#data = featureFormat(my_dataset, features_list, sort_keys=True)\n",
    "#labels, features = targetFeatureSplit(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 看看邮件数据里有什么有价值的东西\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-09-15T09:49:12.665613Z",
     "start_time": "2019-09-15T09:49:12.649203Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[11270511.41941995  3322099.12922766]\n",
      "[0.7550185  0.22254946]\n"
     ]
    }
   ],
   "source": [
    "from sklearn.decomposition import PCA\n",
    "\n",
    "all_email_features =  ['poi','to_messages', 'from_poi_to_this_person', 'from_messages', \n",
    "                       'from_this_person_to_poi', 'shared_receipt_with_poi'] \n",
    "\n",
    "email_data = featureFormat(\n",
    "    data_dict, all_email_features, remove_NaN=True)\n",
    "poi_labels_email, email_features = targetFeatureSplit(email_data)\n",
    "\n",
    "email_features_train, email_features_test ,email_labels_train, email_labels_test \\\n",
    "    = train_test_split(email_features, poi_labels_email, test_size = 0.3, random_state = 42)\n",
    "email_features_train = np.array(email_features_train)\n",
    "email_features_test = np.array(email_features_test)\n",
    "\n",
    "\n",
    "email_pca = PCA(n_components=2,whiten=True)\n",
    "train_email_features_pca = email_pca.fit_transform(email_features_train)\n",
    "print(email_pca.explained_variance_)\n",
    "print(email_pca.explained_variance_ratio_)\n",
    "test_email_features_pca = email_pca.transform(email_features_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-09-15T09:49:14.636440Z",
     "start_time": "2019-09-15T09:49:14.622153Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Before PCA: 0.9230769230769231\n",
      "After PCA 0.9230769230769231\n"
     ]
    }
   ],
   "source": [
    "from sklearn.svm import SVC\n",
    "\n",
    "email_svm = SVC(kernel=\"rbf\",gamma=\"auto\")\n",
    "email_svm.fit(email_features_train, email_labels_train)\n",
    "print(\"Before PCA:\",email_svm.score(email_features_test, email_labels_test))\n",
    "\n",
    "email_svm_pca = SVC(kernel=\"rbf\",gamma=\"auto\")\n",
    "email_svm_pca.fit(train_email_features_pca, email_labels_train)\n",
    "print(\"After PCA\",email_svm_pca.score(test_email_features_pca, email_labels_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-09-15T09:19:53.592451Z",
     "start_time": "2019-09-15T09:19:53.578858Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Before PCA: 0.22727272727272727\n",
      "After PCA: 0.8636363636363636\n"
     ]
    }
   ],
   "source": [
    "# Task 4: Try a varity of classifiers\n",
    "# Please name your classifier clf for easy export below.\n",
    "# Note that if you want to do PCA or other multi-stage operations,\n",
    "# you'll need to use Pipelines. For more info:\n",
    "# http://scikit-learn.org/stable/modules/pipeline.html\n",
    "\n",
    "# Provided to give you a starting point. Try a variety of classifiers.\n",
    "# 使用朴素贝叶斯对原始数据（未进行PCA处理）进行拟合\n",
    "clf = GaussianNB()\n",
    "clf.fit(features_train, labels_train)\n",
    "print(\"Before PCA:\",clf.score(features_test, labels_test))\n",
    "\n",
    "# 使用朴素贝叶斯对经过主成分分析的数据进行拟合\n",
    "clf_pca = GaussianNB()\n",
    "clf_pca.fit(finance_features_train, labels_train)\n",
    "print(\"After PCA:\",clf_pca.score(finance_features_test, labels_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**PCA无敌！！！！**我的妈耶，经过PCA以后的数据用最简单的分类器，准确率提升也太夸张了。因为一开始就采用了PCA，所以看到Task5说要把分类器的表现从0.3提升起来，感觉比较奇怪，因为我的分类器已经0.93的准确率了。\n",
    "所以才做了这个对比。可以看到，即使只选择第一主成分，对于准确率的提升也是很大了。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-09-15T09:02:34.636528Z",
     "start_time": "2019-09-15T09:02:34.622874Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8863636363636364"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 测试支持向量机算法\n",
    "from sklearn.svm import SVC\n",
    "\n",
    "svm = SVC(kernel=\"rbf\",gamma=\"auto\")\n",
    "svm.fit(finance_features_train, labels_train)\n",
    "svm.score(finance_features_test, labels_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-09-15T09:02:37.491152Z",
     "start_time": "2019-09-15T09:02:37.312285Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8181818181818182"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 测试 Adaboost 算法\n",
    "from sklearn.ensemble import AdaBoostClassifier\n",
    "\n",
    "ada = AdaBoostClassifier(n_estimators=80)\n",
    "ada.fit(finance_features_train, labels_train)\n",
    "ada.score(finance_features_test, labels_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-09-15T09:02:40.718031Z",
     "start_time": "2019-09-15T09:02:40.213094Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 10616 characters]\n",
      "text/plain": [
       "<Figure size 864x432 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(12,6))\n",
    "for f in range(len(finance_features_train)):\n",
    "    if labels_train[f]:\n",
    "        plt.scatter(finance_features_train[f][0],finance_features_train[f][1],c = \"r\", marker=\"x\")\n",
    "    else:\n",
    "        plt.scatter(finance_features_train[f][0],finance_features_train[f][1],c = \"b\", marker=\".\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Task 5: Tune your classifier to achieve better than .3 precision and recall\n",
    "# using our testing script. Check the tester.py script in the final project\n",
    "# folder for details on the evaluation method, especially the test_classifier\n",
    "# function. Because of the small size of the dataset, the script uses\n",
    "# stratified shuffle split cross validation. For more info:\n",
    "# http://scikit-learn.org/stable/modules/generated/sklearn.cross_validation.StratifiedShuffleSplit.html\n",
    "\n",
    "# Example starting point. Try investigating other evaluation techniques!\n",
    "features_train, features_test, labels_train, labels_test = \\\n",
    "    train_test_split(features, labels, test_size=0.3, random_state=42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-09-15T09:50:20.596483Z",
     "start_time": "2019-09-15T09:50:20.588076Z"
    }
   },
   "outputs": [],
   "source": [
    "# Task 6: Dump your classifier, dataset, and features_list so anyone can\n",
    "# check your results. You do not need to change anything below, but make sure\n",
    "# that the version of poi_id.py that you submit can be run on its own and\n",
    "# generates the necessary .pkl files for validating your results.\n",
    "\n",
    "dump_classifier_and_data(email_svm_pca, data_dict, all_email_features)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-09-15T09:50:24.883604Z",
     "start_time": "2019-09-15T09:50:23.467614Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Got a divide by zero when trying out: SVC(C=1.0, cache_size=200, class_weight=None, coef0=0.0,\n",
      "    decision_function_shape='ovr', degree=3, gamma='auto', kernel='rbf',\n",
      "    max_iter=-1, probability=False, random_state=None, shrinking=True,\n",
      "    tol=0.001, verbose=False)\n",
      "Precision or recall may be undefined due to a lack of true positive predicitons.\n"
     ]
    }
   ],
   "source": [
    "#!/usr/bin/pickle\n",
    "\n",
    "\"\"\" a basic script for importing student's POI identifier,\n",
    "    and checking the results that they get from it \n",
    " \n",
    "    requires that the algorithm, dataset, and features list\n",
    "    be written to my_classifier.pkl, my_dataset.pkl, and\n",
    "    my_feature_list.pkl, respectively\n",
    "\n",
    "    that process should happen at the end of poi_id.py\n",
    "\"\"\"\n",
    "sys.path.append(\"../tools/\")\n",
    "from feature_format import featureFormat, targetFeatureSplit\n",
    "import pickle\n",
    "import sys\n",
    "from sklearn.model_selection import StratifiedShuffleSplit\n",
    "\n",
    "\n",
    "PERF_FORMAT_STRING = \"\\\n",
    "\\tAccuracy: {:>0.{display_precision}f}\\tPrecision: {:>0.{display_precision}f}\\t\\\n",
    "Recall: {:>0.{display_precision}f}\\tF1: {:>0.{display_precision}f}\\tF2: {:>0.{display_precision}f}\"\n",
    "RESULTS_FORMAT_STRING = \"\\tTotal predictions: {:4d}\\tTrue positives: {:4d}\\tFalse positives: {:4d}\\\n",
    "\\tFalse negatives: {:4d}\\tTrue negatives: {:4d}\"\n",
    "\n",
    "\n",
    "def test_classifier(clf, dataset, feature_list, folds=1000):\n",
    "    data = featureFormat(dataset, feature_list, sort_keys=True)\n",
    "    labels, features = targetFeatureSplit(data)\n",
    "    #cv = StratifiedShuffleSplit(labels, folds, random_state=42)\n",
    "    cv = StratifiedShuffleSplit(n_splits=folds, random_state=42)\n",
    "    true_negatives = 0\n",
    "    false_negatives = 0\n",
    "    true_positives = 0\n",
    "    false_positives = 0\n",
    "    for train_idx, test_idx in cv.split(features, labels):\n",
    "        features_train = []\n",
    "        features_test = []\n",
    "        labels_train = []\n",
    "        labels_test = []\n",
    "        for ii in train_idx:\n",
    "            features_train.append(features[ii])\n",
    "            labels_train.append(labels[ii])\n",
    "        for jj in test_idx:\n",
    "            features_test.append(features[jj])\n",
    "            labels_test.append(labels[jj])\n",
    "\n",
    "        # fit the classifier using training set, and test on test set\n",
    "        clf.fit(features_train, labels_train)\n",
    "        predictions = clf.predict(features_test)\n",
    "        for prediction, truth in zip(predictions, labels_test):\n",
    "            if prediction == 0 and truth == 0:\n",
    "                true_negatives += 1\n",
    "            elif prediction == 0 and truth == 1:\n",
    "                false_negatives += 1\n",
    "            elif prediction == 1 and truth == 0:\n",
    "                false_positives += 1\n",
    "            elif prediction == 1 and truth == 1:\n",
    "                true_positives += 1\n",
    "            else:\n",
    "                print(\"Warning: Found a predicted label not == 0 or 1.\")\n",
    "                print(\"All predictions should take value 0 or 1.\")\n",
    "                print(\"Evaluating performance for processed predictions:\")\n",
    "                break\n",
    "    try:\n",
    "        total_predictions = true_negatives + \\\n",
    "            false_negatives + false_positives + true_positives\n",
    "        accuracy = 1.0*(true_positives + true_negatives)/total_predictions\n",
    "        precision = 1.0*true_positives/(true_positives+false_positives)\n",
    "        recall = 1.0*true_positives/(true_positives+false_negatives)\n",
    "        f1 = 2.0 * true_positives / \\\n",
    "            (2*true_positives + false_positives+false_negatives)\n",
    "        f2 = (1+2.0*2.0) * precision*recall/(4*precision + recall)\n",
    "        print(clf)\n",
    "        print(PERF_FORMAT_STRING.format(accuracy, precision,\n",
    "                                        recall, f1, f2, display_precision=5))\n",
    "        print(RESULTS_FORMAT_STRING.format(total_predictions,\n",
    "                                           true_positives, false_positives, false_negatives, true_negatives))\n",
    "        print(\"\")\n",
    "    except:\n",
    "        print(\"Got a divide by zero when trying out:\", clf)\n",
    "        print(\"Precision or recall may be undefined due to a lack of true positive predicitons.\")\n",
    "\n",
    "\n",
    "CLF_PICKLE_FILENAME = \"my_classifier.pkl\"\n",
    "DATASET_PICKLE_FILENAME = \"my_dataset.pkl\"\n",
    "FEATURE_LIST_FILENAME = \"my_feature_list.pkl\"\n",
    "\n",
    "\n",
    "def dump_classifier_and_data(clf, dataset, feature_list):\n",
    "    with open(CLF_PICKLE_FILENAME, \"wb\") as clf_outfile:\n",
    "        pickle.dump(clf, clf_outfile)\n",
    "    with open(DATASET_PICKLE_FILENAME, \"wb\") as dataset_outfile:\n",
    "        pickle.dump(dataset, dataset_outfile)\n",
    "    with open(FEATURE_LIST_FILENAME, \"wb\") as featurelist_outfile:\n",
    "        pickle.dump(feature_list, featurelist_outfile)\n",
    "\n",
    "\n",
    "def load_classifier_and_data():\n",
    "    with open(CLF_PICKLE_FILENAME, \"rb\") as clf_infile:\n",
    "        clf = pickle.load(clf_infile)\n",
    "    with open(DATASET_PICKLE_FILENAME, \"rb\") as dataset_infile:\n",
    "        dataset = pickle.load(dataset_infile)\n",
    "    with open(FEATURE_LIST_FILENAME, \"rb\") as featurelist_infile:\n",
    "        feature_list = pickle.load(featurelist_infile)\n",
    "    return clf, dataset, feature_list\n",
    "\n",
    "\n",
    "def main():\n",
    "    # load up student's classifier, dataset, and feature_list\n",
    "    clf, dataset, feature_list = load_classifier_and_data()\n",
    "    # Run testing script\n",
    "    test_classifier(clf, dataset, feature_list)\n",
    "\n",
    "\n",
    "main()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
