print("Execution time to fit 500 epochs: %s seconds" % (end_time[5] - start_time[5]))

Execution time to fit 500 epochs: 46.76651573181152 seconds

fig, (ax1, ax2) = plt.subplots(1, 2, sharey=True, sharex=True, figsize=(16, 5))
nn_plot(hist[4], 500, subplot=ax1, loss=True, title="Reference: minibatch 10 samples")
nn_plot(hist[9], 500, subplot=ax2, loss=True, title="minibatch 128 samples")

fig, (ax1, ax2) = plt.subplots(1, 2, sharey=True, sharex=True, figsize=(16, 5))
nn_plot(hist[4], 500, subplot=ax1, acc=True, title="Reference: minibatch 10 samples")
nn_plot(hist[9], 500, subplot=ax2, acc=True, title="minibatch 128 samples")

It is important to notice some points. First, when the batch size increases, the runtime of each epoch becomes faster, since there are less batches per epoch to calculate. However the bigger the batch size, more epochs it will require to converge. Thus our execution time ranged between 45 and 58 seconds for 500 epochs. Except for the gradient descent method, which calculates for each individual sample, that required around 3 minutes for 100 epochs.

One way to understand the impact of bigger batch sizes is that each batch will have more samples influencing the same modelling, thus the model loses precision. In order to compensate this loss in precision, we can increase the learning rate. However when learning rate increases, the model can become unstable or unable to converge to the optimal value.

We can see from the graphics that batches of size 32 and 64 showed better metrics convergence and time execution. Minibatch 32 is the default value of keras implementation. We will compare increased learning rate for these two batches.

Minibatch: 32 samples / higher learning rate

model.append(tf.keras.models.Sequential([
    tf.keras.layers.Dense(30, kernel_regularizer=tf.keras.regularizers.l2(0.01),
                          activation=tf.nn.sigmoid, input_shape=(784,), name='hidden_1_layer'),
    tf.keras.layers.Dense(10, activation='softmax', name='output_layer')
]))

params.append([tf.keras.losses.SparseCategoricalCrossentropy(),
               tf.optimizers.SGD(learning_rate=0.050),
               tf.keras.metrics.SparseCategoricalAccuracy()])

nn_compile(model[-1], params[-1])