![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:1500\/1*Y_329i21ja4m0RPskuwUKw.png\")
<\/figure>In this article, we\u2019ll build a simple neural network using Keras<\/a>. We\u2019ll assume you have prior knowledge of machine learning packages such as scikit-learn<\/a>and other scientific packages such as Pandas<\/a> and Numpy<\/a>.<\/p>\n Training an artificial neural network involves the following steps:<\/p>\n Now let\u2019s proceed to solve a real business problem. An insurance company has approached you with a dataset of previous claims of their clients. The insurance company wants you to develop a model to help them predict which claims look fraudulent. By doing so you hope to save the company millions of dollars annually. This is a classification<\/a> problem. These are the columns in our dataset.<\/p>\n As in many business problems, the data provided will not be processed for us. We therefore have to prepare it in a way that our algorithm will accept it. We see from the dataset that we have some categorical columns. We need to convert these to zeros and ones so that our deep learning model will be able to understand them. Another thing to note is that we have to feed our dataset to the model as numpy arrays. Below we import the necessary packages and then load in our dataset<\/a><\/mark>.<\/p>\n We then to convert the categorical columns to dummy variables.<\/p>\n In this case we use We use sklearn\u2019s<\/a> Next we make sure to drop the column we\u2019re predicting to prevent it from leaking into the training set and the test set. We must avoid using the same dataset to train and test the model. We set We then split the data into a training and test set. We use 0.7 of the data for training and 0.3 for testing.<\/p>\n Next we have to scale our dataset using Sklearn\u2019s The first thing we need to do is import Keras. By default, Keras will use TensorFlow<\/a> as its backend.<\/p>\n Next we need to import a few modules from Keras. The Sequential module is required to initialize the ANN, and the Dense module is required to build the layers of our ANN.<\/p>\n Next we need to initialize our ANN by creating an instance of Sequential. The Sequential function initializes a linear stack of layers. This allows us to add more layers later using the Dense module.<\/p>\n We use the add method to add different layers to our ANN. The first parameter is the number of nodes you want to add to this layer. There is no rule of thumb as to how many nodes you should add. However, a common strategy is to choose the number of nodes as the average of nodes in the input layer and the number of nodes in the output layer.<\/p>\n Say for example you had five independent variables and one output. Then you would take the sum of that and divide by two, which is three. You can also decide to experiment with a technique called parameter tuning. The second parameter, In this case, it will use a uniform distribution to make sure that the weights are small numbers close to zero. The next parameter is the activation function. We use the Rectifier function<\/a>, shortened as ReLU. We mostly use this function for the hidden layer in ANN. The final parameter is Adding the second hidden layer is similar to adding the first hidden layer.<\/p>\n We don\u2019t need to specify the We change the first parameter because in our output node we expect one node. This is because we are only interested in knowing whether a claim was fraudulent or not. We change the activation function because we want to get the probabilities that a claim is fraudulent. We do this by using the Sigmoid activation function<\/a>.<\/p>\n In case you\u2019re dealing with a classification problem that has more than two classes (i.e. classifying cats, dogs, and monkeys) we\u2019d need to change two things. We\u2019d change the first parameter to 3 and change the activation function to softmax. Softmax is a sigmoid function applied to an independent variable with more than two categories.<\/p>\n Compiling is basically applying a stochastic gradient descent<\/a> to the whole neural network. The first parameter is the algorithm you want to use to get the optimal set of weights in the neural network. The algorithm used here is a stochastic gradient algorithm.<\/p>\n There are many variants of this. A very efficient one to use is Adam<\/a>. The second parameter is the loss function within the stochastic gradient algorithm. Since our categories are binary, we use the This will show us the probability of a claim being fraudulent. We then set a threshold of 50% for classifying a claim as fraudulent. This means that any claim with a probability of 0.5 or more will be classified as fraudulent.<\/p>\n This way the insurance firm can be able to first track claims that are not suspicious and then take more time evaluating claims flagged as fraudulent.<\/p>\n The confusion matrix can be interpreted as follows. Out of 2000 observations, 1550 + 175 observations were correctly predicted while 230 + 45 were incorrectly predicted. You can calculate the accuracy by dividing the number of correct predictions by the total number of predictions. In this case (1550+175) \/ 2000, which gives you 86%.<\/p>\n Let\u2019s say the insurance company gives you a single claim. They\u2019d like to know if the claim is fraudulent. What would you do to find out?<\/p>\n where a,b,c,d represents the features you have.<\/p>\n Since our classifier expects numpy arrays, we have to transform the single observation into a numpy array and use the standard scaler to scale it.<\/p>\n After training the model one or two times, you\u2019ll notice that you keep getting different accuracies. So you\u2019re not quite sure which one is the right one. This introduces the bias variance trade off. In essence, we\u2019re trying to train a model that will be accurate and not have too much variance of accuracy when trained several times.<\/p>\n To solve this problem we use the K-fold cross validation with K equal to 10. This will split the training set into 10 folds. We\u2019ll then train our model on 9 folds and test it on the remaining fold. Since we have 10 folds, we\u2019re going to do this iteratively through 10 combinations. Each iteration will gives us its accuracy. We\u2019ll then find the mean of all accuracies and use that as our model accuracy. We also calculate the variance to ensure that it\u2019s minimal.<\/p>\n Keras has a scikit learn wrapper ( Next we import the k-fold cross validation function from scikit_learn<\/p>\n The This function will build the classifier and return it for use in the next step. The only thing we have done here is wrap our previous ANN architecture in a function and return the classifier.<\/p>\n We then create a new classifier using K-fold cross validation and pass the parameter To apply the k-fold cross validation function we can use scikit-learn\u2019s To obtain the relative accuracies we get the mean of the accuracies.<\/p>\n The variance can be obtained as follows:<\/p>\n The goal is to have a small variance between the accuracies.<\/p>\n Overfitting<\/a> in machine learning is what happens when a model learns the details and noise in the training set such that it performs poorly on the test set. This can be observed when we have huge differences between the accuracies of the test set and training set, or when you observe a high variance when applying k-fold cross validation.<\/p>\n In artificial neural networks, we counteract this using a technique called dropout regularization<\/a>. Dropout regularization works by randomly disabling some neurons at each iteration of the training to prevent them from being too dependent on each other.<\/p>\n In this case we apply the dropout after the first hidden layer and after the second hidden layer. Using a rate of 0.1 means that 1% of the neurons will be disabled at each iteration. It is advisable to start with a rate of 0.1. However you should never go beyond 0.4 because you will now start to get underfitting.<\/p>\n Once you obtain your accuracy you can tune the parameters to get a higher accuracy. Grid Search enables us to test different parameters in order to obtain the best parameters.<\/p>\n The first step here is to import the We also need to modify our We\u2019ll still use the The next step is to create a dictionary with the parameters we\u2019d like to tune \u2014 in this case the batch size, the number of epochs, and the optimizer function. We still use Adam as an optimizer and add a new one called rmsprop<\/a>. The Keras documentation recommends rmsprop when dealing with Recurrent Neural Networks<\/a>. However we can try it for this ANN to see if it gives us a better result.<\/p>\n We then use Grid Search to test these parameters. The grid search function expects our estimator, the parameters we just defined, the scoring metric and the number of k-folds.<\/p>\n Like in previous objects we need to fit our training set.<\/p>\n We can get the best selection of parameters using It\u2019s important to note that this process will take a while as it searches for the best parameters.<\/p>\n Artificial Neural Networks are just one type of deep neural network. There are other networks such Recurrent Neural Networks (RNN), Convolutional Neural Networks<\/a>(CNN), and Boltzmann machines<\/a>.<\/p>\n RNNs can predict if the price of a stock<\/a> will go up or down in the future. CNNs are used in computer vision<\/a> \u2014 recognizing cats and dogs in a set of images or recognizing the presence of cancer cells in a brain image. Boltzmann machines are used in programming recommender systems<\/a>. Maybe we can cover one of these neural networks in the future.<\/p>\n<\/div>\n<\/div>\n<\/div>\n","protected":false},"excerpt":{"rendered":"Training an Artificial Neural Network<\/strong><\/h1>\n
\n
Business Problem<\/strong><\/h1>\n
<\/figure>
<\/figure>
<\/figure>
Data Preprocessing<\/strong><\/h2>\n
import pandas as pd<\/em><\/span>import numpy as np<\/em><\/span>df = pd.read_csv(\u2018Datasets\/claims\/insurance_claims.csv\u2019)<\/em><\/span><\/pre>\nfeats = [\u2018policy_state\u2019,\u2019insured_sex\u2019,\u2019insured_education_level\u2019,\u2019insured_occupation\u2019,\u2019insured_hobbies\u2019,\u2019insured_relationship\u2019,\u2019collision_type\u2019,\u2019incident_severity\u2019,\u2019authorities_contacted\u2019,\u2019incident_state\u2019,\u2019incident_city\u2019,\u2019incident_location\u2019,\u2019property_damage\u2019,\u2019police_report_available\u2019,\u2019auto_make\u2019,\u2019auto_model\u2019,\u2019fraud_reported\u2019,\u2019incident_type\u2019]\ndf_final = pd.get_dummies(df,columns=feats,drop_first=True)<\/pre>\n
drop_first=True<\/code> to avoid the dummy variable trap. For example, if you have a, b, c, d as categories then you can drop d as a dummy variable. This is because if something does not fall into either a, b, or c then it\u2019s definitely in d. This is referred to as multicollinearity.<\/p>\ntrain_test_split<\/code> to split the data into a training set and a test set.<\/p>\nfrom sklearn.model_selection import train_test_split<\/em><\/span><\/pre>\n.values<\/code> at the end of the dataset in order to get the numpy arrays. This is the way our deep learning model will accept the data. This step is important because our machine learning model expects the data in form of arrays.<\/p>\nX = df_final.drop([\u2018fraud_reported_Y\u2019,\u2019policy_csl\u2019,\u2019policy_bind_date\u2019,\u2019incident_date\u2019],axis=1).values\ny = df_final[\u2018fraud_reported_Y\u2019].values<\/pre>\n
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3)<\/em><\/span><\/pre>\nStandardScaler<\/code>. <\/em>Due to the massive amounts of computations taking place in deep learning, feature scaling is compulsory. Feature scaling standardizes the range of our independent variables.<\/p>\nfrom sklearn.preprocessing import StandardScaler<\/em><\/span>sc = StandardScaler()<\/em><\/span>X_train = sc.fit_transform(X_train)<\/em><\/span>X_test = sc.transform(X_test)<\/em><\/span><\/pre>\n<\/div>\n<\/div>\n<\/div>\n\n\n\nBuilding the Artificial Neural Network(ANN)<\/strong><\/h2>\n
import keras<\/em><\/span><\/pre>\nfrom keras.models import Sequential<\/em><\/span>from keras.layers import Dense<\/em><\/span><\/pre>\nclassifier = Sequential()<\/em><\/span><\/pre>\nAdding input layer (First Hidden Layer)<\/h2>\n
kernel_initializer<\/code>, is the function that will be used to initialize the weights.<\/p>\ninput_dim<\/code>, which is the number of nodes in the input layer. It represents the number of independent variables.<\/p>\nclassiifier.add(\n Dense(3, kernel_initializer = \u2018uniform\u2019,\n activation = \u2018relu\u2019, input_dim=5))<\/em><\/span><\/pre>\nAdding Second Hidden Layer<\/strong><\/h2>\n
classiifier.add(\n Dense(3, kernel_initializer = \u2018uniform\u2019,\n activation = \u2018relu\u2019))<\/em><\/span><\/pre>\ninput_dim<\/code> parameter because we have already specified it in the first hidden layer. In the first hidden layer, we specified this in order to let the layer know how many input nodes to expect. In the second hidden layer the ANN already knows how many input nodes to expect so we don\u2019t need to repeat ourselves.<\/p>\nAdding the output layer<\/strong><\/h2>\n
classifier.add(\n Dense(1, kernel_initializer = \u2018uniform\u2019,\n activation = \u2018sigmoid\u2019))<\/em><\/span><\/pre>\nCompiling the ANN<\/strong><\/h2>\n
classifier.compile(optimizer= \u2018adam\u2019,\n loss = \u2018binary_crossentropy\u2019,\n metrics = [\u2018accuracy\u2019])<\/em><\/span><\/pre>\nbinary_crossentropy<\/code>loss function. Otherwise we would have used categorical_crossentopy<\/code>. <\/em>The final argument is the criterion we\u2019ll use to evaluate our model. In this case we use the accuracy.<\/p>\nFitting our ANN to the training set<\/strong><\/h2>\n
classifier.fit(X_train, y_train, batch_size = 10, epochs = 100)<\/em><\/span><\/pre>\nX_train<\/code> represents the independent variables we\u2019re using to train our ANN, and y_train<\/code> represents the column we\u2019re predicting. Epochs represents the number of times we\u2019re going to pass our full dataset through the ANN. Batch_size<\/code> is the number of observations after which the weights will be updated.<\/p>\nPredicting using the training set<\/strong><\/h2>\n
y_pred = classifier.predict(X_test)<\/em><\/span><\/pre>\ny_pred = (y_pred > 0.5)<\/em><\/span><\/pre>\nChecking the confusion matrix<\/strong><\/h2>\n
from sklearn.metrics import confusion_matrix<\/em><\/span>cm = confusion_matrix(y_test, y_pred)<\/em><\/span><\/pre>\n<\/figure>
Making a single Prediction<\/strong><\/h2>\n
new_pred = classifier.predict(sc.transform(np.array([[a,b,c,d]])))<\/em><\/span><\/pre>\nnew_pred = (new_prediction > 0.5)<\/em><\/span><\/pre>\nEvaluating our ANN<\/strong><\/h2>\n
KerasClassifier<\/code>) that enables us to include K-fold cross validation in our Keras code.<\/p>\nfrom keras.wrappers.scikit_learn import KerasClassifier<\/em><\/span><\/pre>\nfrom sklearn.model_selection import cross_val_score<\/em><\/span><\/pre>\nKerasClassifier<\/code> expects one of its arguments to be a function, so we need to build that function.The purpose of this function is to build the architecture of our ANN.<\/p>\ndef make_classifier():\n classifier = Sequential()\n classiifier.add(Dense(3, kernel_initializer = \u2018uniform\u2019, activation = \u2018relu\u2019, input_dim=5))\n classiifier.add(Dense(3, kernel_initializer = \u2018uniform\u2019, activation = \u2018relu\u2019))\n classifier.add(Dense(1, kernel_initializer = \u2018uniform\u2019, activation = \u2018sigmoid\u2019))\n classifier.compile(optimizer= \u2018adam\u2019,loss = \u2018binary_crossentropy\u2019,metrics = [\u2018accuracy\u2019])\n return classifier<\/pre>\n
build_fn<\/code> as the function we just created above. Next we pass the batch size and the number of epochs, just like we did in the previous classifier.<\/p>\nclassiifier = KerasClassifier(build_fn = make_classifier,\n batch_size=10, nb_epoch=100)<\/em><\/span><\/pre>\ncross_val_score<\/code> function. The estimator is the classifier we just built with make_classifier<\/code> and n_jobs=-1<\/code> will make use of all available CPUs. cv is the number of folds and 10 is a typical choice. The cross_val_score<\/code> will return the ten accuracies of the ten test folds used in the computation.<\/p>\naccuracies = cross_val_score(estimator = classifier,\n X = X_train,\n y = y_train,\n cv = 10,\n n_jobs = -1)<\/em><\/span><\/pre>\nmean = accuracies.mean()<\/em><\/span><\/pre>\nvariance = accuracies.var()<\/em><\/span><\/pre>\nFighting Overfitting<\/strong><\/h2>\n
from keras.layers import Dropout\n\nclassifier = Sequential()\nclassiifier.add(Dense(3, kernel_initializer = \u2018uniform\u2019, activation = \u2018relu\u2019, input_dim=5))\n\n# Notice the dropouts\nclassifier.add(Dropout(rate = 0.1))\nclassiifier.add(Dense(6, kernel_initializer = \u2018uniform\u2019, activation = \u2018relu\u2019))\nclassifier.add(Dropout(rate = 0.1))\n\nclassifier.add(Dense(1, kernel_initializer = \u2018uniform\u2019, activation = \u2018sigmoid\u2019))\nclassifier.compile(optimizer= \u2018adam\u2019,loss = \u2018binary_crossentropy\u2019,metrics = [\u2018accuracy\u2019])<\/pre>\n
Parameter Tuning<\/strong><\/h2>\n
GridSearchCV<\/code> module from sklearn.<\/p>\nfrom sklearn.model_selection import GridSearchCV<\/em><\/span><\/pre>\nmake_classifier<\/code> function as follows. We create a new variable called optimizer<\/em><\/code> that will allow us to add more than one optimizer in our params variable.<\/p>\ndef make_classifier(optimizer):\n classifier = Sequential()\n classiifier.add(Dense(6, kernel_initializer = \u2018uniform\u2019, activation = \u2018relu\u2019, input_dim=11))\n classiifier.add(Dense(6, kernel_initializer = \u2018uniform\u2019, activation = \u2018relu\u2019))\n classifier.add(Dense(1, kernel_initializer = \u2018uniform\u2019, activation = \u2018sigmoid\u2019))\n classifier.compile(optimizer= optimizer,loss = \u2018binary_crossentropy\u2019,metrics = [\u2018accuracy\u2019])\n return classifier<\/pre>\n
KerasClassifier<\/code>, but we won\u2019t pass the batch size and number of epochs since these are the parameters we want to tune.<\/p>\nclassifier = KerasClassifier(build_fn = make_classifier)<\/em><\/span><\/pre>\nparams = {\n 'batch_size':[20,35],<\/em><\/span> 'nb_epoch':[150,500],<\/em><\/span> 'Optimizer':['adam','rmsprop'<\/em><\/strong>]\n}<\/em><\/span><\/pre>\ngrid_search = GridSearchCV(estimator=classifier,\n param_grid=params,\n scoring=\u2019accuracy\u2019,\n cv=10)<\/em><\/span><\/pre>\ngrid_search = grid_search.fit(X_train,y_train)<\/em><\/span><\/pre>\nbest_params<\/code> from the grid search object. Likewise we use the best_score_<\/code> to get the best score.<\/p>\nbest_param = grid_search.best_params_<\/em><\/span>best_accuracy = grid_search.best_score_<\/em><\/span><\/pre>\nConclusion<\/strong><\/h1>\n