fig, (ax1, ax2) = plt.subplots(1, 2, sharey=True, figsize=(16, 5))
nn_plot(hist[4], 100, subplot=ax1, loss=True, title="Reference: eta 0.025 / 1,000 training set")
nn_plot(hist[5], 100, subplot=ax2, loss=True, title="Reference: eta 0.025 / full training set")

fig, (ax1, ax2) = plt.subplots(1, 2, sharey=True, figsize=(16, 5))
nn_plot(hist[4], 100, subplot=ax1, acc=True, title="Reference: eta 0.025 / 1,000 training set")
nn_plot(hist[5], 100, subplot=ax2, acc=True, title="Reference: eta 0.025 / full training set")

With the entire training set, the convergence is faster and the metrics are even better than the small training set, since the model now has more data to learn from. We can calculate the accuracy over the test set.

# Evaluate the model on the test data using 'evaluate'
print("Evaluate on test data")
results.append(model[-1].evaluate(x_test, y_test, batch_size=64))
print("test loss, test acc:", results[0])

Evaluate on test data
157/157 [==============================] - 0s 2ms/step - loss: 0.3884 - sparse_categorical_accuracy: 0.9182
test loss, test acc: [0.3883625268936157, 0.9182000160217285]

With this model, we achieved an accuracy of 91.82% over the test set. We will use $\eta = 0.025$ and analyze other hyperparameters now.

Size of minibatch

We are using as our optimizer the Stochastic Gradient Descent (SGD). For the smaller training data we used minibatches of 10 samples and for the entire training set we used 64 samples. Now we will investigate these values.
Our aim is to find the best values, which compromises the trade-off between memory / speed usage and model efficiency.

The time spent for modelling, with minibatch of 10 samples was:

print("Execution time to fit 500 epochs: %s seconds" % (end_time[0] - start_time[0]))

Execution time to fit 500 epochs: 111.62148904800415 seconds

We start by analyzing the small training set, with the equivalent to the Gradient Descent, using single batch method.

model.append(tf.keras.models.Sequential([
    tf.keras.layers.Dense(30, kernel_regularizer=tf.keras.regularizers.l2(0.01),
                          activation=tf.nn.sigmoid, input_shape=(784,), name='hidden_1_layer'),
    tf.keras.layers.Dense(10, activation='softmax', name='output_layer')
]))

nn_compile(model[-1], params[-1])