DropOut in DEEP LEARNING

An artificial Neural Network is a network of different inputs and hidden layers.

Neural Network is a very powerful model when it comes to solving very complex problems like Image Recognition, Speech Recognition. So, whenever the Artificial Neural Network is very deep then there is one problem. The problem is that the deeper the neural network has and the more the weights and bias it has, Artificial Neural Network tends to overfit the model. Overfitted model is a kind of problem as we may be confused from it if we did not analyze it properly. At first glance in the training phase, it may seem that the model has performed very well, however, while testing the model its accuracy drops significantly leading to a decrease in the generalization of our model. Our model, no matter it may be Machine Learning problem, or Deep Neural Network problem, it must be generalized model meaning that whenever it sees a new unseen data it must be able to detect and find pattern accurately.

So, now let me introduce the term DropOut, a technique that as written as a thesis at the University of Toronto by Nitish Srivastav under the supervision of Geoffery Hinton, the God of Deep Learning who gave us the concept of MultiLayer Neural Network.

The working principle of Dropout is similar to RandomForest. In RandomForest, we randomly select the subset of features and in each and every iteration this subset of features keeps on changing. Similarly, in Dropout also we will select the subset of hidden layers, activation functions in each iteration and we will drop certain hidden layers and input layers in every iteration. With this conceptualization or technique on dropping certain layers and inputs, network layers adapt to correct mistakes from the prior layer which in turn makes the model more powerful and robust.

We can see from the figure that I have crossed out certain neurons meaning that those neurons and layers are dropped for that iteration. But one must careful that these same layers won’t be dropped next time it will choose randomly another layer in every iteration to avoid an overfitting problem. What we are doing here is that we are just activating and deactivating neurons to solve the overfitting problem.

Dropout can be used with most of the types of neural networks like Artificial Neural Network(ANN), Convolutional Neural Network(CNN). or Recurrent Neural Network(RNN).

Similarly, dropout can be implemented on any or all hidden layers as well as invisible layers or input layers but never on the output layer.