Code examples and use cases

First order extensions with a ResNet

Mini-batch sub-sampling

Extension hook example

Diagonal Gauss-Newton Second order optimizer

Custom module example

Saving memory in convolutions

Gradient of backpropagated quantities

Hutchinson Trace Estimation

Residual networks

Differentially Private SGD.

Recurrent networks

Matrix-free second-order optimization