:orphan:

Code examples and use cases
============================



.. raw:: html

    <div class="sphx-glr-thumbnails">

.. thumbnail-parent-div-open

.. raw:: html

    <div class="sphx-glr-thumbcontainer" tooltip="First order extensions with a ResNet">

.. only:: html

  .. image:: /use_cases/images/thumb/sphx_glr_example_first_order_resnet_thumb.png
    :alt:

  :ref:`sphx_glr_use_cases_example_first_order_resnet.py`

.. raw:: html

      <div class="sphx-glr-thumbnail-title">First order extensions with a ResNet</div>
    </div>


.. raw:: html

    <div class="sphx-glr-thumbcontainer" tooltip="By default, BackPACK&#x27;s extensions consider all samples in a mini-batch. Some extensions support limiting the computations to a subset of samples. This example shows how to restrict the computations to such a subset of samples.">

.. only:: html

  .. image:: /use_cases/images/thumb/sphx_glr_example_subsampling_thumb.png
    :alt:

  :ref:`sphx_glr_use_cases_example_subsampling.py`

.. raw:: html

      <div class="sphx-glr-thumbnail-title">Mini-batch sub-sampling</div>
    </div>


.. raw:: html

    <div class="sphx-glr-thumbcontainer" tooltip="The extension hook is a function called on each module after the BackPACK extensions have run. It can be used to reduce memory overhead if the goal is to compute transformations of BackPACK quantities. Information can be compacted during a backward pass and obsolete tensors be freed manually.">

.. only:: html

  .. image:: /use_cases/images/thumb/sphx_glr_example_extension_hook_thumb.png
    :alt:

  :ref:`sphx_glr_use_cases_example_extension_hook.py`

.. raw:: html

      <div class="sphx-glr-thumbnail-title">Extension hook example</div>
    </div>


.. raw:: html

    <div class="sphx-glr-thumbcontainer" tooltip="A simple second-order optimizer with BackPACK on the classic MNIST example from PyTorch. The optimizer we implement uses uses the diagonal of the GGN/Fisher matrix as a preconditioner, with a constant damping parameter;">

.. only:: html

  .. image:: /use_cases/images/thumb/sphx_glr_example_diag_ggn_optimizer_thumb.png
    :alt:

  :ref:`sphx_glr_use_cases_example_diag_ggn_optimizer.py`

.. raw:: html

      <div class="sphx-glr-thumbnail-title">Diagonal Gauss-Newton Second order optimizer</div>
    </div>


.. raw:: html

    <div class="sphx-glr-thumbcontainer" tooltip="In PyTorch, you can easily compute derivatives of a scalar-valued variable f w.r.t. to a variable param by calling f.backward(). This computes the Jacobian ∂(f) / ∂(param) that has shape [1, *param.shape].">

.. only:: html

  .. image:: /use_cases/images/thumb/sphx_glr_example_batched_jacobians_thumb.png
    :alt:

  :ref:`sphx_glr_use_cases_example_batched_jacobians.py`

.. raw:: html

      <div class="sphx-glr-thumbnail-title">Batched Jacobians</div>
    </div>


.. raw:: html

    <div class="sphx-glr-thumbcontainer" tooltip="There are different approaches to apply the Jacobian with respect to the kernel of a convolution. They exhibit a non-trivial trade-off between run time and memory consumption (see more details below). The default choice in BackPACK is a memory- intensive implementation. This can lead to out-of-memory errors.">

.. only:: html

  .. image:: /use_cases/images/thumb/sphx_glr_example_save_memory_convolutions_thumb.png
    :alt:

  :ref:`sphx_glr_use_cases_example_save_memory_convolutions.py`

.. raw:: html

      <div class="sphx-glr-thumbnail-title">Saving memory in convolutions</div>
    </div>


.. raw:: html

    <div class="sphx-glr-thumbcontainer" tooltip="This tutorial demonstrates how to perform multiple backward passes through the same computation graph with BackPACK. This option can be useful if you run into out-of-memory errors. If your computation can be chunked, you might consider distributing it onto multiple backward passes to reduce peak memory.">

.. only:: html

  .. image:: /use_cases/images/thumb/sphx_glr_example_retain_graph_thumb.png
    :alt:

  :ref:`sphx_glr_use_cases_example_retain_graph.py`

.. raw:: html

      <div class="sphx-glr-thumbnail-title">BackPACK's retain_graph option</div>
    </div>


.. raw:: html

    <div class="sphx-glr-thumbcontainer" tooltip="If :pybackward() is called with create_graph=True, PyTorch creates the computation graph of the outputs of the backward pass, including quantities computed by BackPACK. This makes it possible to compute higher order derivatives with PyTorch, even if BackPACK&#x27;s extensions no longer apply.">

.. only:: html

  .. image:: /use_cases/images/thumb/sphx_glr_example_gradient_of_variance_thumb.png
    :alt:

  :ref:`sphx_glr_use_cases_example_gradient_of_variance.py`

.. raw:: html

      <div class="sphx-glr-thumbnail-title">Gradient of backpropagated quantities</div>
    </div>


.. raw:: html

    <div class="sphx-glr-thumbcontainer" tooltip="This example illustrates the estimation the Hessian trace of a neural network using Hutchinson&#x27;s method [Hutchinson, 1990], which is an algorithm to obtain such an an estimate from matrix-vector products:">

.. only:: html

  .. image:: /use_cases/images/thumb/sphx_glr_example_trace_estimation_thumb.png
    :alt:

  :ref:`sphx_glr_use_cases_example_trace_estimation.py`

.. raw:: html

      <div class="sphx-glr-thumbnail-title">Hutchinson Trace Estimation</div>
    </div>


.. raw:: html

    <div class="sphx-glr-thumbcontainer" tooltip="Residual networks">

.. only:: html

  .. image:: /use_cases/images/thumb/sphx_glr_example_resnet_all_in_one_thumb.png
    :alt:

  :ref:`sphx_glr_use_cases_example_resnet_all_in_one.py`

.. raw:: html

      <div class="sphx-glr-thumbnail-title">Residual networks</div>
    </div>


.. raw:: html

    <div class="sphx-glr-thumbcontainer" tooltip=" Differentially Private Stochastic Gradient Descent (DP-SGD) [Abadi et al., 2016] is a rather simple idea. Instead of doing a typical SGD update, with the sum of all gradients in a minibatch,">

.. only:: html

  .. image:: /use_cases/images/thumb/sphx_glr_example_differential_privacy_thumb.png
    :alt:

  :ref:`sphx_glr_use_cases_example_differential_privacy.py`

.. raw:: html

      <div class="sphx-glr-thumbnail-title">Differentially Private SGD.</div>
    </div>


.. raw:: html

    <div class="sphx-glr-thumbcontainer" tooltip="Recurrent networks">

.. only:: html

  .. image:: /use_cases/images/thumb/sphx_glr_example_rnn_thumb.png
    :alt:

  :ref:`sphx_glr_use_cases_example_rnn.py`

.. raw:: html

      <div class="sphx-glr-thumbnail-title">Recurrent networks</div>
    </div>


.. raw:: html

    <div class="sphx-glr-thumbcontainer" tooltip="This example walks you through a second-order optimizer that uses the conjugate gradient (CG) method and matrix-free multiplication with the block diagonal of different curvature matrices to solve for the Newton step.">

.. only:: html

  .. image:: /use_cases/images/thumb/sphx_glr_example_cg_newton_thumb.png
    :alt:

  :ref:`sphx_glr_use_cases_example_cg_newton.py`

.. raw:: html

      <div class="sphx-glr-thumbnail-title">Matrix-free second-order optimization</div>
    </div>


.. raw:: html

    <div class="sphx-glr-thumbcontainer" tooltip="This tutorial explains how to support new layers in BackPACK.">

.. only:: html

  .. image:: /use_cases/images/thumb/sphx_glr_example_custom_module_thumb.png
    :alt:

  :ref:`sphx_glr_use_cases_example_custom_module.py`

.. raw:: html

      <div class="sphx-glr-thumbnail-title">Custom module example</div>
    </div>


.. thumbnail-parent-div-close

.. raw:: html

    </div>


.. toctree::
   :hidden:

   /use_cases/example_first_order_resnet
   /use_cases/example_subsampling
   /use_cases/example_extension_hook
   /use_cases/example_diag_ggn_optimizer
   /use_cases/example_batched_jacobians
   /use_cases/example_save_memory_convolutions
   /use_cases/example_retain_graph
   /use_cases/example_gradient_of_variance
   /use_cases/example_trace_estimation
   /use_cases/example_resnet_all_in_one
   /use_cases/example_differential_privacy
   /use_cases/example_rnn
   /use_cases/example_cg_newton
   /use_cases/example_custom_module


.. only:: html

  .. container:: sphx-glr-footer sphx-glr-footer-gallery

    .. container:: sphx-glr-download sphx-glr-download-python

      :download:`Download all examples in Python source code: use_cases_python.zip </use_cases/use_cases_python.zip>`

    .. container:: sphx-glr-download sphx-glr-download-jupyter

      :download:`Download all examples in Jupyter notebooks: use_cases_jupyter.zip </use_cases/use_cases_jupyter.zip>`


.. only:: html

 .. rst-class:: sphx-glr-signature

    `Gallery generated by Sphinx-Gallery <https://sphinx-gallery.github.io>`_