By looking both loss function and accuracy graphics it seems that the overfitting was strongly reduced. There is a general trend of train loss being slightly lower than validation loss and train accuracy being slightly higher than validation accuracy. This is expected to happen, since the model is fitted on the training set.
However in our case the metrics become too wiggly and this is an evidence that the model cannot learn properly, requiring better hyperparameters tunning. We maintain the same number of epochs as before, for comparisons purpose.

Hyperparameters optimization¶

A neural network have many different configurations and hyperparameters and setting them right is an important step, in order to achieve the best model for the problem.
Here we will investigate some of the most important hyperparameters, each one independently of the others, for an overall illustration of the best solution for this digit classification problem.

For a first approach on hyperparameters analysis, it is recommended to start with a smaller model, by applying part of the training set or a few classes of the entire outputs. This way the parametrization analysis can be performed much faster and guide towards the direction each hyperparameter should have.

Learning rate $\eta$¶

We start by changing the learning rate $\eta$. Our reference will be $\eta = 0.5$, the first value used.
The wiggly behavior we saw on previous model is mainly due to $\eta$ being too high and the model not converging to optimal value. This similar behavior can be seen for any higher value of $\eta$. We demonstrate this with $\eta = 1$. For this value of $\eta$ the metrics get worse than reference.

In [19]:

model.append(tf.keras.models.Sequential([
    tf.keras.layers.Dense(30, kernel_regularizer=tf.keras.regularizers.l2(0.01),
                          activation=tf.nn.sigmoid, input_shape=(784,), name='hidden_1_layer'),
    tf.keras.layers.Dense(10, activation='softmax', name='output_layer')
]))

params.append([tf.keras.losses.SparseCategoricalCrossentropy(),
               tf.optimizers.SGD(learning_rate=1),
               tf.keras.metrics.SparseCategoricalAccuracy()])

nn_compile(model[-1], params[-1])