print("Execution time to fit 100 epochs: %s seconds" % (end_time[2] - start_time[2]))

Execution time to fit 100 epochs: 178.35433721542358 seconds

fig, (ax1, ax2) = plt.subplots(1, 2, sharey=True, figsize=(16, 5))
nn_plot(hist[4], 100, subplot=ax1, loss=True, title="Reference: minibatch 10 samples")
nn_plot(hist[6], 100, subplot=ax2, loss=True, title="batch (each sample)")

fig, (ax1, ax2) = plt.subplots(1, 2, sharey=True, figsize=(16, 5))
nn_plot(hist[4], 100, subplot=ax1, acc=True, title="Reference: minibatch 10 samples")
nn_plot(hist[6], 100, subplot=ax2, acc=True, title="batch (each sample)")

The gradient descent method, by using each sample is not recommended. It can be seen through the graphics that the model becomes too sensitive to individual data and has difficulties to converge, when applied the learning rate of $\eta = 0.025$. The model can converge with smaller $\eta$ values, however this requires even more time for model fitting.

We will proceed and compare stochastic gradient descent with minibatches bigger than 10. It is recommended to use batch sizes multiple of 2, so we will compare 10 with 32, 64 and 128.

Minibatch: 32 samples

model.append(tf.keras.models.Sequential([
    tf.keras.layers.Dense(30, kernel_regularizer=tf.keras.regularizers.l2(0.01),
                          activation=tf.nn.sigmoid, input_shape=(784,), name='hidden_1_layer'),
    tf.keras.layers.Dense(10, activation='softmax', name='output_layer')
]))

nn_compile(model[-1], params[-1])