fig, (ax1, ax2) = plt.subplots(1, 2, sharey=True, figsize=(16, 5))
nn_plot(hist[5], 100, subplot=ax1, loss=True, title="Reference: full training set / activation sigmoidal")
nn_plot(hist[15], 100, subplot=ax2, loss=True, title="full training set / activation relu")

fig, (ax1, ax2) = plt.subplots(1, 2, sharey=True, figsize=(16, 5))
nn_plot(hist[5], 100, subplot=ax1, acc=True, title="Reference: full training set / activation sigmoidal")
nn_plot(hist[15], 100, subplot=ax2, acc=True, title="full training set / activation relu")

# Evaluate the model on the test data using 'evaluate'
print("Evaluate on test data")
results.append(model[-1].evaluate(x_test, y_test, batch_size=64))
print("test loss, test acc:", results[1])

Evaluate on test data
157/157 [==============================] - 1s 3ms/step - loss: 0.2414 - sparse_categorical_accuracy: 0.9529
test loss, test acc: [0.24137741327285767, 0.9528999924659729]

As ilustrated by the graphics, our model improved by applying function ReLU. It did not reach overfitting since now the model has more samples to learn from and test. The accuracy over the test set increased significantly, from 91.82% to 95.29%.

Number of nodes and layers

Until now we used the same topology for all our analysis, by applying one hidden layer with 30 nodes. We chose the number 30 randomly, but the best number depends on each problem, its predictors and complexity and requires some attention and testing. We will analyze other topologies now, for our neural network.

We start by rerunning our last model, but for a training set of 2,000 samples. We change from 1,000 to 2,000 mainly due to overfitting happening after 100 epochs, with the ReLU activation function, and we want to avoid reaching 100% accuracy on training set. With 2,000 samples we still can analyze faster than by applying the entire set.

model.append(tf.keras.models.Sequential([
    tf.keras.layers.Dense(30, kernel_regularizer=tf.keras.regularizers.l2(0.01),
                          activation='relu', input_shape=(784,), name='hidden_1_layer'),
    tf.keras.layers.Dense(10, activation='softmax', name='output_layer')
]))

nn_compile(model[-1], params[3])