1. Custom loss functions
When we're creating models for certain situations, we may have special requirements. Custom loss functions can help guide our neural nets towards meeting those needs.
2. Direction is important
Direction is important for stock price change predictions. If a price change was in a positive direction, we want to predict something also in the positive direction. Cases, where the prediction direction is wrong, could end poorly.
3. MSE with directional penalty
One way to guide our neural net to predict correct directionality is to apply a penalty to incorrect prediction direction. We can still use the mean squared error penalty, but if the direction is wrong, we'll multiply the loss by a penalty factor.
4. Implementing custom loss functions
To implement custom loss functions, we need to import a few things. One is our backend, which in this case is tensorflow. There are other backends like theano, which will have slightly different function names for what we do here.
5. Creating a function
Next, we create the actual loss function as a Python function. This starts with def, then has the function name -- we'll call it mean_squared_error. Then we give the arguments, which for keras loss functions should be y_true and y_pred for the actual and predicted values.
6. Mean squared error loss
Let's first look at what the mean squared error loss would look like. We would take the true values minus the predicted values, then use tf-dot-square to square that difference. Then we take the average of that value with tf-dot-reduce_mean. We tell it to take the average across the -1 axis, which is the last axis -- this will take the average of all the training samples our net just trained on. Then we return that value as the loss.
7. Add custom loss to keras
Lastly, we need to import keras-dot-losses and set this function as part of keras-dot-losses. This makes it available for use in our models. We can then specify our loss function name for the loss argument in model-dot-compile, and fit the model.
8. Checking for correct direction
We can now add a penalty for wrong prediction directions. We first check if the sign of the prediction and actual values differ. To do this we'll multiply true values by predictions, and check where it's negative using tf-dot-less. Two negative or two positive values, which yields a positive value when multiplied, means we predicted the correct direction. Opposite signs will give a negative value.
9. Using tf.where()
Now we create our custom loss function called sign_penalty, and use a penalty value of 10. We then use tf-dot-where, which takes 3 arguments: a matrix of booleans, or True/False values, code that will be run where the boolean matrix has True values, and code that will be run when the matrix has False values. We check with tf-dot-less if prediction direction is correct. When the direction is wrong, we apply a penalty to the square of actual minus predicted values. When the direction is correct, we don't use the penalty.
10. Tying it together
Now that we have our conditional statement, we return the average of our squared errors as we did before. We also add our loss function to keras-dot-losses so we can use it.
11. Using the custom loss
To use our loss, we specify the function name when compiling. Here, we've created our same model as in the last lesson, but are using the sign_penalty loss function instead of mse in model-dot-compile().
12. The bow-tie shape
Once we've fit our model, we again plot the predictions versus actual values. This results in an interesting shape that looks like a bow-tie, at least for the training samples. The penalty for misclassifying is strong enough to force the model to mostly match signs of predictions and actual values.
13. Create your own loss function!
Now let's make a custom loss function and try it out!