Training & evaluation with the built-in methods

Author: fchollet Date created: 2019/03/01 Last modified: 2023/06/25 Description: Complete guide to training & evaluation with fit() and evaluate().
View in Colab • GitHub source
This guide covers training, evaluation, and prediction (inference) models when using built-in APIs for training & validation (such as Model.fit(), Model.evaluate() and Model.predict()).
If you are interested in leveraging fit() while specifying your own training step function, see the guides on customizing what happens in fit():
If you are interested in writing your own training & evaluation loops from scratch, see the guides on writing training loops:
In general, whether you are using built-in loops or writing your own, model training & evaluation works strictly in the same way across every kind of Keras model – Sequential models, models built with the Functional API, and models written from scratch via model subclassing.
When passing data to the built-in training loops of a model, you should either use:
In the next few paragraphs, we'll use the MNIST dataset as NumPy arrays, in order to demonstrate how to use optimizers, losses, and metrics. Afterwards, we'll take a close look at each of the other options.
Let's consider the following model (here, we build in with the Functional API, but it could be a Sequential model or a subclassed model as well):
Here's what the typical end-to-end workflow looks like, consisting of:
We'll use MNIST data for this example.
We specify the training configuration (optimizer, loss, metrics):
We call fit(), which will train the model by slicing the data into "batches" of size batch_size, and repeatedly iterating over the entire dataset for a given number of epochs.
The returned history object holds a record of the loss values and metric values during training:
We evaluate the model on the test data via evaluate():
Now, let's review each piece of this workflow in detail.
To train a model with fit(), you need to specify a loss function, an optimizer, and optionally, some metrics to monitor.
You pass these to the model as arguments to the compile() method:
The metrics argument should be a list – your model can have any number of metrics.
If your model has multiple outputs, you can specify different losses and metrics for each output, and you can modulate the contribution of each output to the total loss of the model. You will find more details about this in the Passing data to multi-input, multi-output models section.
Note that if you're satisfied with the default settings, in many cases the optimizer, loss, and metrics can be specified via string identifiers as a shortcut:
For later reuse, let's put our model definition and compile step in functions; we will call them several times across different examples in this guide.
In general, you won't have to create your own losses, metrics, or optimizers from scratch, because what you need is likely to be already part of the Keras API:
If you need to create a custom loss, Keras provides three ways to do so.
The first method involves creating a function that accepts inputs y_true and y_pred. The following example shows a loss function that computes the mean squared error between the real data and the predictions:
If you need a loss function that takes in parameters beside y_true and y_pred, you can subclass the keras.losses.Loss class and implement the following two methods:
Let's say you want to use mean squared error, but with an added term that will de-incentivize prediction values far from 0.5 (we assume that the categorical targets are one-hot encoded and take values between 0 and 1). This creates an incentive for the model not to be too confident, which may help reduce overfitting (we won't know if it works until we try!).
Here's how you would do it:
If you need a metric that isn't part of the API, you can easily create custom metrics by subclassing the keras.metrics.Metric class. You will need to implement 4 methods:
State update and results computation are kept separate (in update_state() and result(), respectively) because in some cases, the results computation might be very expensive and would only be done periodically.
Here's a simple example showing how to implement a CategoricalTruePositives metric that counts how many samples were correctly classified as belonging to a given class:
The overwhelming majority of losses and metrics can be computed from y_true and y_pred, where y_pred is an output of your model – but not all of them. For instance, a regularization loss may only require the activation of a layer (there are no targets in this case), and this activation may not be a model output.
In such cases, you can call self.add_loss(loss_value) from inside the call method of a custom layer. Losses added in this way get added to the "main" loss during training (the one passed to compile()). Here's a simple example that adds activity regularization (note that activity regularization is built-in in all Keras layers – this layer is just for the sake of providing a concrete example):
Note that when you pass losses via add_loss(), it becomes possible to call compile() without a loss function, since the model already has a loss to minimize.
Consider the following LogisticEndpoint layer: it takes as inputs targets & logits, and it tracks a crossentropy loss via add_loss().
You can use it in a model with two inputs (input data & targets), compiled without a loss argument, like this:
For more information about training multi-input models, see the section Passing data to multi-input, multi-output models.
In the first end-to-end example you saw, we used the validation_data argument to pass a tuple of NumPy arrays (x_val, y_val) to the model for evaluating a validation loss and validation metrics at the end of each epoch.
Here's another option: the argument validation_split allows you to automatically reserve part of your training data for validation. The argument value represents the fraction of the data to be reserved for validation, so it