fig, (ax1, ax2) = plt.subplots(1, 2, sharey=True, sharex=True, figsize=(16, 5))
nn_plot(hist[21], 100, subplot=ax1, loss=True, title="Reference: full training set / lambda 0.01")
nn_plot(hist[24], 100, subplot=ax2, loss=True, title="full training set / lambda 0.005")

fig, (ax1, ax2) = plt.subplots(1, 2, sharey=True, sharex=True, figsize=(16, 5))
nn_plot(hist[21], 100, subplot=ax1, acc=True, title="Reference: full training set / lambda 0.01")
nn_plot(hist[24], 100, subplot=ax2, acc=True, title="full training set / lambda 0.005")

# Evaluate the model on the test data using 'evaluate'
print("Evaluate on test data")
results.append(model[-1].evaluate(x_test, y_test, batch_size=64))
print("test loss, test acc:", results[4])

Evaluate on test data
157/157 [==============================] - 0s 3ms/step - loss: 0.2522 - sparse_categorical_accuracy: 0.9686
test loss, test acc: [0.2521584928035736, 0.9685999751091003]

By evaluating the graphics it looks like our model got slightly better. The accuracy over the test set was slightly better as well, going from 96.61% to 96.86%.

Dropout

There is still another technique, commonly used, which applies regularization very effectively over the model. The dropout technique hiddens random nodes each time the network is trained, so each time, the network is a different network, with different nodes, weights and data. The final model will consider all nodes, but they will be more independent and less heavily correlated, thus robust to overfitting. We will apply a dropout rate of 50%.

model.append(tf.keras.models.Sequential([
    tf.keras.layers.Dense(100, kernel_regularizer=tf.keras.regularizers.l2(0.01),
                          activation='relu', input_shape=(784,), name='hidden_1_layer'),
    tf.keras.layers.Dropout(0.5, name='dropout'),
    tf.keras.layers.Dense(10, activation='softmax', name='output_layer')
]))

nn_compile(model[-1], params[3])