.. _testing-label:

Running and writing tests
=========================


Write tests
^^^^^^^^^^^

Every new function that is written in to HyperSpy needs to be tested and
documented.

Tests are short functions found in hyperspy/tests that call your functions
under some known conditions and check the outputs against known values. They
should depend on as few other features as possible so that when they break
we know exactly what caused it. Ideally, the tests should be written at the
same time than the code itself, as they are very convenient to run to check
outputs when coding. Writing tests can seem laborious but you'll probably
soon find that they're very important as they force you to sanity check all
you do. For details on running and writing HyperSpy test see :ref:`dev_tests`



HyperSpy uses the `pytest <http://doc.pytest.org/>`_ library for testing. The
tests reside in the ``hyperspy.tests`` module. 

First ensure pytest and its plugins are installed by:

.. code:: bash
   
   # If using a standard hyperspy install
   pip install hyperspy[test]
   # Or, from a hyperspy local development directory
   pip install -e .[test]
   # Or just installing the dependencies using conda
   conda install -c conda-forge pytest pytest-mpl

To run them:

.. code:: bash

   pytest --mpl --pyargs hyperspy

Or, from HyperSpy's project folder simply:

.. code:: bash

   pytest


Useful hints on testing:

* When comparing integers, it's fine to use ``==``. When comparing floats, be
  sure to use ``numpy.testing.assert_almost_equal()`` or
  ``numpy.testing.assert_allclose()``.
* ``numpy.testing.assert_equal()`` is convenient to compare numpy arrays.
* The ``hyperspy.misc.test_utils.py`` contains a few useful functions for
  testing.
* ``@pytest.mark.parametrize()`` is a very convenient decorator to test several
  parameters of the same function without having to write to much repetitive
  code, which is often error-prone. See `pytest documentation for more details
  <http://doc.pytest.org/en/latest/parametrize.html>`_.
* It is good to check that the tests does not use too much of memory after
  creating new tests. If you need to explicitly delete your objects and free
  memory, you can do the following to release the memory associated to the
  ``s`` object, for example:

.. code:: python

    del s
    gc.collect()


* Once, you have pushed your PR to the official HyperSpy repository, it can be
  useful to check the coverage of your tests using the coveralls.io check of
  your PR. There should be a link to it at the bottom of your PR on the github
  PR page. This service can help you to find how well your code is being tested
  and exactly which part is not currently tested.
* `pytest-sugar <https://pypi.python.org/pypi/pytest-sugar>`_ can be installed
  to have a nicer look and feel of pytest in the console (encoding issue have
  been reported in the Windows console).


.. _plot-test-label:

Plot testing
^^^^^^^^^^^^
Plotting is tested using the ``@pytest.mark.mpl_image_compare`` decorator of
the `pytest mpl plugin <https://pypi.python.org/pypi/pytest-mpl>`_.  This
decorator uses reference images to compare with the generated output during the
tests. The references images are located in the folder defined by the argument
``baseline_dir`` of the ``@pytest.mark.mpl_image_compare`` decorator.

To run plotting tests, you simply need to add the option ``--mpl``:
::

    pytest --mpl

If you don't use the ``--mpl``, the code of the tests will be executed but the
images will not be compared to the references images.

If you need to add or change some plots, follow the workflow below:

    1. Write the tests using appropriate decorator such as
       ``@pytest.mark.mpl_image_compare``.
    2. If you need to generate new reference image in the folder
       ``plot_test_dir``, for example, run: ``pytest
       --mpl-generate-path=plot_test_dir``
    3. Run again the tests and this time they should pass.
    4. Use ``git add`` to put the new file in the git repository.

When the plotting tests are failling, it is possible to download the figure 
comparison images generated by pytest-mpl in the `artifacts tabs 
<https://ci.appveyor.com/project/hyperspy/hyperspy/build/1.0.2500/job/2c2qccaktd90po2q/artifacts>`_ 
of the corresponding build.


The plotting tests need matplotlib > 3.0.0, since small changes in the way 
 matplotlib generates the figure can make the tests fail.

In travis and appveyor, the matplotlib backend is set to ``agg`` by setting
the ``MPLBACKEND`` environment variable to ``agg``. At the first import of
``matplotlib.pyplot``, matplotlib will look at the ``MPLBACKEND`` environment
variable and set accordingly the backend.

See `pytest-mpl <https://pypi.python.org/pypi/pytest-mpl>`_ for more details.

Exporting pytest results as HTML
^^^^^^^^^^^^
With ``pytest-html`` it is possible to export the results of running pytest 
for easier viewing. I can be installed by conda:

.. code:: bash

   conda install pytest-html
   
and run by:

.. code:: bash

   pytest --mpl --html=report.html