![\"\"](\"https:\/\/www.comet.com\/site\/wp-content\/uploads\/2023\/06\/1NUil-wyDtAKmoD-HjWXBLw-1024x614.webp\")
<\/figure>\n\n\n\n<\/p>\n\n\n\n
In the last post, you created a 2-layer neural network from scratch and now have a better understanding of how neural networks work.<\/p>\n
In this second part, you\u2019ll use your network to make predictions, and also compare its performance to two standard libraries (scikit-learn and Keras).<\/p>\n
Specifically, you\u2019ll learn how to:<\/p>\n
- \n
- Train and test the neural network<\/li>\n
- Build a 2-layer neural network using scikit-learn<\/li>\n
- Build a 2-layer neural network using Keras<\/li>\n<\/ul>\n
\ud83d\udc49\ud83c\udffd Full code in Google Colab here.<\/a><\/p>\n
Training and Testing the Neural Network<\/h1>\n
In case you\u2019ve forgotten what your network looks like, here\u2019s the code:<\/p>\n
class NeuralNet():\n '''\n A two layer neural network\n '''\n\n def __init__(self, layers=[13,8,1], learning_rate=0.001, iterations=100):\n self.params = {}\n self.learning_rate = learning_rate\n self.iterations = iterations\n self.loss = []\n self.sample_size = None\n self.layers = layers\n self.X = None\n self.y = None\n\n def init_weights(self):\n '''\n Initialize the weights from a random normal distribution\n '''\n np.random.seed(1) # Seed the random number generator\n self.params[\"W1\"] = np.random.randn(self.layers[0], self.layers[1])\n self.params['b1'] =np.random.randn(self.layers[1],)\n self.params['W2'] = np.random.randn(self.layers[1],self.layers[2])\n self.params['b2'] = np.random.randn(self.layers[2],)\n\n def relu(self,Z):\n '''\n The ReLu activation function is to performs a threshold\n operation to each input element where values less\n than zero are set to zero.\n '''\n return np.maximum(0,Z)\n\n def dRelu(self, x):\n x[x<=0] = 0\n x[x>0] = 1\n return x\n\n def eta(self, x):\n ETA = 0.0000000001\n return np.maximum(x, ETA)\n\n\n def sigmoid(self,Z):\n '''\n The sigmoid function takes in real numbers in any range and\n squashes it to a real-valued output between 0 and 1.\n '''\n return 1\/(1+np.exp(-Z))\n\n def entropy_loss(self,y, yhat):\n nsample = len(y)\n yhat_inv = 1.0 - yhat\n y_inv = 1.0 - y\n yhat = self.eta(yhat) ## clips value to avoid NaNs in log\n yhat_inv = self.eta(yhat_inv)\n loss = -1\/nsample * (np.sum(np.multiply(np.log(yhat), y) + np.multiply((y_inv), np.log(yhat_inv))))\n return loss\n\n def forward_propagation(self):\n '''\n Performs the forward propagation\n '''\n\n Z1 = self.X.dot(self.params['W1']) + self.params['b1']\n A1 = self.relu(Z1)\n Z2 = A1.dot(self.params['W2']) + self.params['b2']\n yhat = self.sigmoid(Z2)\n loss = self.entropy_loss(self.y,yhat)\n\n # save calculated parameters\n self.params['Z1'] = Z1\n self.params['Z2'] = Z2\n self.params['A1'] = A1\n\n return yhat,loss\n\n def back_propagation(self,yhat):\n '''\n Computes the derivatives and update weights and bias according.\n '''\n y_inv = 1 - self.y\n yhat_inv = 1 - yhat\n\n dl_wrt_yhat = np.divide(y_inv, self.eta(yhat_inv)) - np.divide(self.y, self.eta(yhat))\n dl_wrt_sig = yhat * (yhat_inv)\n dl_wrt_z2 = dl_wrt_yhat * dl_wrt_sig\n\n dl_wrt_A1 = dl_wrt_z2.dot(self.params['W2'].T)\n dl_wrt_w2 = self.params['A1'].T.dot(dl_wrt_z2)\n dl_wrt_b2 = np.sum(dl_wrt_z2, axis=0, keepdims=True)\n\n dl_wrt_z1 = dl_wrt_A1 * self.dRelu(self.params['Z1'])\n dl_wrt_w1 = self.X.T.dot(dl_wrt_z1)\n dl_wrt_b1 = np.sum(dl_wrt_z1, axis=0, keepdims=True)\n\n #update the weights and bias\n self.params['W1'] = self.params['W1'] - self.learning_rate * dl_wrt_w1\n self.params['W2'] = self.params['W2'] - self.learning_rate * dl_wrt_w2\n self.params['b1'] = self.params['b1'] - self.learning_rate * dl_wrt_b1\n self.params['b2'] = self.params['b2'] - self.learning_rate * dl_wrt_b2\n\n def fit(self, X, y):\n '''\n Trains the neural network using the specified data and labels\n '''\n self.X = X\n self.y = y\n self.init_weights() #initialize weights and bias\n\n\n for i in range(self.iterations):\n yhat, loss = self.forward_propagation()\n self.back_propagation(yhat)\n self.loss.append(loss)\n\n def predict(self, X):\n '''\n Predicts on a test data\n '''\n Z1 = X.dot(self.params['W1']) + self.params['b1']\n A1 = self.relu(Z1)\n Z2 = A1.dot(self.params['W2']) + self.params['b2']\n pred = self.sigmoid(Z2)\n return np.round(pred)\n\n def acc(self, y, yhat):\n '''\n Calculates the accutacy between the predicted valuea and the truth labels\n '''\n acc = int(sum(y == yhat) \/ len(y) * 100)\n return acc\n\n\n def plot_loss(self):\n '''\n Plots the loss curve\n '''\n plt.plot(self.loss)\n plt.xlabel(\"Iteration\")\n plt.ylabel(\"logloss\")\n plt.title(\"Loss curve for training\")\n plt.show()<\/pre>\nTo train and test the network, you\u2019ll read in the heart disease dataset, preprocess it like you would in any other machine learning task, specifically, you\u2019ll:<\/p>\n
- \n
- Split the data into train and test sets,<\/li>\n
- Standardize it using StandardScaler,<\/li>\n
- Initialize a model from your
NeuralNetwork<\/code> class, and finally,<\/li>\n- Train and predict with the Model<\/li>\n<\/ul>\n
import numpy as np\nimport warnings\nwarnings.filterwarnings(\"ignore\") #suppress warnings\nimport matplotlib.pyplot as plt\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.preprocessing import StandardScaler\n\n#convert imput to numpy arrays\nX = heart_df.drop(columns=['heart_disease'])\n\n#replace target class with 0 and 1\n#1 means \"have heart disease\" and 0 means \"do not have heart disease\"\nheart_df['heart_disease'] = heart_df['heart_disease'].replace(1, 0)\nheart_df['heart_disease'] = heart_df['heart_disease'].replace(2, 1)\n\ny_label = heart_df['heart_disease'].values.reshape(X.shape[0], 1)\n\n#split data into train and test set\nXtrain, Xtest, ytrain, ytest = train_test_split(X, y_label, test_size=0.2, random_state=2)\n\n#standardize the dataset\nsc = StandardScaler()\nsc.fit(Xtrain)\nXtrain = sc.transform(Xtrain)\nXtest = sc.transform(Xtest)\n\nprint(f\"Shape of train set is {Xtrain.shape}\")\nprint(f\"Shape of test set is {Xtest.shape}\")\nprint(f\"Shape of train label is {ytrain.shape}\")\nprint(f\"Shape of test labels is {ytest.shape}\")<\/pre>\n<\/figure>\n \n ![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:346\/0*oHD_uni106zhl3Si.png\")
<\/figure><\/picture><\/div>\n<\/figure>\n In the code block above, first, you get the training data, excluding the label\u2014this is done with the
drop<\/code> function. Thedrop<\/code> function removes the specified column from the dataset and returns the remaining features. Next, you saved the label to a variable calledy_label<\/code>.<\/p>\nNext, you split the data into train and test set, with the test set taking 10 percent of the overall data. Then, you standardize the dataset. Standardization is important when working with neural networks, as it has a serious effect on the training time and network performance.<\/p>\n
In the next code section, you\u2019ll initialize the neural network and train it using the fit function.<\/p>\n
nn = NeuralNet() # create the NN model\nnn.fit(Xtrain, ytrain) #train the model<\/pre>\n
Here, you initialize the neural network with the default parameters:<\/p>\n
- \n
- layers ==> [13,8,1]<\/li>\n
- learning_rate ==> 0.001<\/li>\n
- iterations ==> 100<\/li>\n<\/ul>\n
Then, you passed in the training data (
Xtrain<\/code>,ytrain<\/code>) to the fit method. This is the training phase of the model.<\/p>\nNow that you’re done training, let\u2019s plot the loss using the handy
plot_loss<\/code>function in theNeuralNetwork<\/code> class:<\/p>\nnn.plot_loss()<\/span><\/pre>\n\n ![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:432\/1*eJQyOCpEDyQRna__cH4SfA.png\")
<\/figure><\/picture><\/div>\n<\/figure>\n You can see that the loss starts pretty high, then quickly goes down, and starts approaching zero. This gives you an insight into how the network learns. Now, let\u2019s try increasing the learning rate and the size of the hidden layer, and see how it affects the loss.<\/p>\n
nn = NeuralNet(layers=[13,10,1], learning_rate=0.01, iterations=500) # create the NN model\nnn.fit(Xtrain, ytrain) #train the model\nnn.plot_loss()<\/pre>\n
\n ![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:432\/1*iAo3OaiZwkIv-lEk-N3JdA.png\")
<\/figure><\/picture><\/div>\n<\/figure>\n You can clearly see that the loss is lower than the previous one. So these sets of parameters might actually be doing better than the first. To confirm this, let\u2019s show the accuracy on both the train and test set.<\/p>\n
For the first Architecture, we have the following accuracies:<\/p>\n
train_pred = nn.predict(Xtrain)\ntest_pred = nn.predict(Xtest)\n\nprint(\"Train accuracy is {}\".format(nn.acc(ytrain, train_pred)))\nprint(\"Test accuracy is {}\".format(nn.acc(ytest, test_pred)))<\/pre>\n\n ![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:314\/1*VRoIogc4eZx5fyM9UMWkFg.png\")
<\/figure><\/picture><\/div>\n<\/figure>\n For the second network, I had the same set of accuracies. This suggests that the second model is overfitting the data and the first model is actually better.<\/p>\n
And just so you know, those are pretty great accuracies for a Neural Network you built from scratch!<\/p>\n
Working with Python Libraries<\/h1>\n
Just to spice it up, let\u2019s compare your implementation with standard libraries like Keras<\/a> and scikit-learn. If you don\u2019t have these libraries installed on your machine, you either have to install them or use Google Colab<\/a> for this section.<\/p>\n
A 2-Layer Neural Network with scikit-learn<\/h2>\n
The MLPClassifier<\/a> in the scikit-learn package contains an implementation of a neural network. To use it, first, you have to import it from the
sklearn.neural_network<\/code> class, and initialize the architecture:<\/p>\nfrom sklearn.neural_network import MLPClassifier\nfrom sklearn.metrics import accuracy_score\n\nsknet = MLPClassifier(hidden_layer_sizes=(8), learning_rate_init=0.001, max_iter=100)<\/pre>\n
Notice that you specify just the number of hidden nodes when using the MLPClassifier\u2014this is because the size of the input feature is inferred from the dimension of the input data. Also, you can specify the learning rate and the number of iterations. To keep things fair, you\u2019ll use the same parameters you used for your own Neural Net. That is 8 hidden nodes, a learning rate of 0.001 and 100 iterations.<\/p>\n
Let\u2019s train this network and calculate its accuracy:<\/p>\n
sknet.fit(Xtrain, ytrain)\npreds_train = sknet.predict(Xtrain)\npreds_test = sknet.predict(Xtest)\n\nprint(\"Train accuracy of sklearn neural network: {}\".format(round(accuracy_score(preds_train, ytrain),2)*100))\nprint(\"Test accuracy of sklearn neural network: {}\".format(round(accuracy_score(preds_test, ytest),2)*100))<\/pre>\n\n ![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:570\/1*N7jlT8BqDfXeYlvwii4P-A.png\")
<\/figure><\/picture><\/div>\n<\/figure>\n Well, can you believe it! Your accuracy is actually higher than that of scikit-learn. This does not mean that you can use your Neural Network in production though. The catch here is that many standard implementations of neural networks use different optimization strategies regarding weight initialization, model training, gradient updates, adaptive learning rates, regularization, and so on, and these are important to consider when using models in industry.<\/p>\n
While we can implement most of these optimizations in our implementation, we\u2019ll refrain from doing that in this post. If interested, check out these amazing articles:<\/p>\n
\n- \n
- Setting the learning rate of your neural network<\/a><\/li>\n
- A Gentle Introduction to Mini-Batch Gradient Descent and How to Configure Batch Size<\/a><\/li>\n
- ML | Mini-Batch Gradient Descent with Python<\/a><\/li>\n
- Weight Initialization Techniques in Neural Networks<\/a><\/li>\n
- All the Backpropagation derivatives<\/a><\/li>\n<\/ul>\n<\/div>\n
Next, let\u2019s see how to create a neural network with Keras.<\/p>\n
A 2-Layer Neural Network with Keras<\/h2>\n
Keras<\/a> is an open-source deep-learning library written in Python. It was designed to make experimentation with deep learning libraries faster and easier. In this article, you’ll be using the Tensorflow Implementation of Keras (tf.keras). This implementation is similar to normal Keras. The only difference is the way it is imported.<\/p>\n
To create a neural network model in tf.keras, you have to import the Sequential, Layers and Dense modules. The
Sequential<\/code> module can accept a series of layers stacked on top of each other.<\/p>\nLet\u2019s demonstrate this below:<\/p>\n
from tensorflow.keras import Sequential\nfrom tensorflow.keras.layers import Dense\nfrom tensorflow.keras.optimizers import Adam\n\n# define the model\nmodel = Sequential()\nmodel.add(Dense(8, input_shape=(13,)))\nmodel.add(Dense(1, activation='sigmoid'))<\/pre>\n
In the code section above, you created a
Sequential<\/code> model. This tells Keras that you want to create stacks of layers. Next, you add twoDense<\/code> layers. The firstDense<\/code> layer has an input shape of 13 and 8 hidden nodes, while the secondDense<\/code> layer, which is your output, has a single node and uses the sigmoid activation function.<\/p>\nNext, compile the model by passing in the loss function, an optimizer that tells the network how to learn, and a metric to be calculated:<\/p>\n
# compile the model\nopt = Adam(learning_rate=0.001)\nmodel.compile(optimizer=opt, loss='binary_crossentropy', metrics=['accuracy'])<\/pre>\n
There are many types of optimizers you can choose from, and each may affect the network\u2019s performance. Start with the Adam optimizer here, but feel free to try other ones here<\/a>. Also, use a binary cross-entropy loss function, given that you’re working on a binary classification task.<\/p>\n
After compiling, you\u2019ll train the network and evaluate it:<\/p>\n
model.fit(Xtrain, ytrain, epochs=100, verbose=0)\ntrain_acc = model.evaluate(Xtrain, ytrain, verbose=0)[1]\ntest_acc = model.evaluate(Xtest, ytest, verbose=0)[1]\n\nprint(\"Train accuracy of keras neural network: {}\".format(round((train_acc * 100), 2)))\nprint(\"Test accuracy of keras neural network: {}\".format(round((test_acc * 100),2)))<\/pre>\n\n ![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:570\/1*aSRQJSHGesxHTndzAFnIBA.png\")
<\/figure><\/picture><\/div>\n<\/figure>\n And that’s it! The accuracies of Keras on both the train and test sets are similar to what you have in your own Neural Network.<\/p>\n
Summary<\/h1>\n
That\u2019s it\u2026 You\u2019ve built, trained, and tested a neural network from scratch, and also compared the performance with 2 standard deep learning libraries.<\/p>\n
In summary, to create a neural network from scratch, you have to perform the following:<\/p>\n
- \n
- Get training data and target variable<\/li>\n
- Initialize the weights and biases<\/li>\n
- Compute forward propagation<\/li>\n
- Compute backpropagation<\/li>\n
- Update weights and bias<\/li>\n
- Repeat steps 2-4 for n times<\/li>\n<\/ol>\n
Conclusion<\/h1>\n
I hope this has been an effective introduction to Neural Networks, AI and deep learning in general. More importantly, I hope you\u2019ve learned the steps and challenges in creating a Neural Network from scratch, using just Python and Numpy. While your network is not state-of-art, I\u2019m sure this post has helped you understand how neural network works.<\/p>\n
There are lots of other things that go into effectively optimizing a neural network for production. This is the reason why you won\u2019t need to use a \u201cbuilt-from-scratch\u201d Neural Network in the industry, instead, you\u2019ll use existing libraries that have been efficiently optimized.<\/p>\n
Congratulations, you\u2019re well on your way to becoming a great AI engineer. If you have any questions, suggestions, or feedback, don\u2019t hesitate to use the comment section below.<\/p>\n
\n ![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:640\/1*FCiTcgvIHxdfEUurUWMsfQ.jpeg\")
<\/figure><\/picture><\/div>\n<\/figure>\n Connect with me on <\/em>Twitter<\/em><\/strong><\/a>.<\/em><\/strong><\/p>\n
- A Gentle Introduction to Mini-Batch Gradient Descent and How to Configure Batch Size<\/a><\/li>\n
- Setting the learning rate of your neural network<\/a><\/li>\n
- Train and predict with the Model<\/li>\n<\/ul>\n