Writing packages that extend HyperSpy

Added in version 1.5: External packages can extend HyperSpy by registering signals, components and widgets.

Warning

The mechanism to register extensions is in beta state. This means that it can change between minor and patch versions. Therefore, if you maintain a package that registers HyperSpy extensions, please verify that it works properly with any future HyperSpy release. We expect it to reach maturity with the release of HyperSpy 2.0.

External packages can extend HyperSpy by registering signals, components and widgets. Objects registered by external packages are “first-class citizens” i.e. they can be used, saved and loaded like any of those objects shipped with HyperSpy. Because of HyperSpy’s structure, we anticipate that most packages registering HyperSpy extensions will provide support for specific sorts of data.

Models can also be provided by external packages, but don’t need to be registered. Instead, they are returned by the create_model method of the relevant BaseSignal subclass, see for example, the exspy.signals.EDSTEMSpectrum.create_model() of the exspy.signals.EDSTEMSpectrum.

It is good practice to add all packages that extend HyperSpy to the list of known extensions regardless of their maturity level. In this way, we can avoid duplication of efforts and issues arising from naming conflicts. This repository also runs an integration test suite daily, which runs the test suite of all extensions to check the status of the ecosystem. See the corresponding section for more details.

At this point, it is worth noting that HyperSpy’s main strength is its amazing community of users and developers. We trust that the developers of packages that extend HyperSpy will play by the same rules that have made the Python scientific ecosystem successful. In particular, avoiding duplication of efforts and being good community players by contributing code to the best matching project are essential for the sustainability of our open software ecosystem.

Registering extensions

In order to register HyperSpy extensions, you need to:

1. Add the following line to your package’s setup.py:

   entry_points={'hyperspy.extensions': 'your_package_name = your_package_name'},
   

2. Create a hyperspy_extension.yaml configuration file in your module’s root directory.

3. Declare all new HyperSpy objects provided by your package in the hyperspy_extension.yaml file.

For a full example on how to create a package that extends HyperSpy, see the HyperSpy Sample Extension package.

Creating new HyperSpy BaseSignal subclasses

When and where to create a new BaseSignal subclass

HyperSpy provides most of its functionality through the different BaseSignal subclasses. A HyperSpy “signal” is a class that contains data for analysis and functions to perform the analysis in the form of class methods. Functions that are useful for the analysis of most datasets are in the BaseSignal class. All other functions are in specialized subclasses.

The flowchart below can help you decide where to add a new data analysis function. Notice that only if no suitable package exists for your function, you should consider creating your own.

Registering a new BaseSignal subclass

To register a new BaseSignal subclass you must add it to the hyperspy_extension.yaml file, as in the following example:

signals:
    MySignal:
        signal_type: "MySignal"
        signal_type_aliases:
        - MS
        - ThisIsMySignal
        # The dimension of the signal subspace. For example, 2 for images, 1 for
        # spectra. If the signal can take any signal dimension, set it to -1.
        signal_dimension: 1
        # The data type, "real" or "complex".
        dtype: real
        # True for LazySignal subclasses
        lazy: False
        # The module where the signal is located.
        module: my_package.signal

Note that HyperSpy uses signal_type to determine which class is the most appropriate to deal with a particular sort of data. Therefore, the signal type must be specific enough for HyperSpy to find a single signal subclass match for each sort of data.

Warning

HyperSpy assumes that only one signal subclass exists for a particular signal_type. It is up to external package developers to avoid signal_type clashes, typically by collaborating in developing a single package per data type.

The optional signal_type_aliases are used to determine the most appropriate signal subclass when using set_signal_type(). For example, if the signal_type Electron Energy Loss Spectroscopy has an EELS alias, setting the signal type to EELS will correctly assign the signal subclass with Electron Energy Loss Spectroscopy signal type. It is good practice to choose a very explicit signal_type while leaving acronyms for signal_type_aliases.

Additionally, the optional key hidden: True can be defined if a signal should be registered with HyperSpy, but not listed by hyperspy.api.print_known_signal_types(). This option can be used if a signal subclass is needed for certain functionalities, such as casting to a different signal subclass, but should usually not be set directly by the user.

Creating new HyperSpy model components

When and where to create a new component

HyperSpy provides the hyperspy._components.expression.Expression component that enables easy creation of 1D and 2D components from mathematical expressions. Therefore, strictly speaking, we only need to create new components when they cannot be expressed as simple mathematical equations. However, HyperSpy is all about simplifying the interactive data processing workflow. Therefore, we consider that functions that are commonly used for model fitting, in general or specific domains, are worth adding to HyperSpy itself (if they are of common interest) or to specialized external packages extending HyperSpy.

The flowchart below can help you decide when and where to add a new hyperspy model hyperspy.component.Component for your function, should you consider creating your own.

Registering new components

All new components must be a subclass of hyperspy._components.expression.Expression. To register a new 1D component add it to the hyperspy_extension.yaml file as in the following example:

components1D:
  # _id_name of the component. It must be a UUID4. This can be generated
  # using ``uuid.uuid4()``. Also, many editors can automatically generate
  # UUIDs. The same UUID must be stored in the components ``_id_name`` attribute.
  fc731a2c-0a05-4acb-91df-d15743b531c3:
    # The module where the component class is located.
    module: my_package.components
    # The actual class of the component
    class: MyComponent1DClass

Equivalently, to add a new component 2D:

components2D:
  # _id_name of the component. It must be a UUID4. This can be generated
  # using ``uuid.uuid4()``. Also, many editors can automatically generate
  # UUIDs. The same UUID must be stored in the components ``_id_name`` attribute.
  2ffbe0b5-a991-4fc5-a089-d2818a80a7e0:
    # The module where the component is located.
    module: my_package.components
    class: MyComponent2DClass

Note

HyperSpy’s legacy components use their class name instead of a UUID as _id_name. This is for compatibility with old versions of the software. New components (including those provided through the extension mechanism) must use a UUID4 in order to i) avoid name clashes ii) make it easy to find the component online if e.g. the package is renamed or the component relocated.

Creating and registering new widgets and toolkeys

To generate GUIs of specific methods and functions, HyperSpy uses widgets and toolkeys:

• widgets (typically ipywidgets or traitsui objects) generate GUIs,

• toolkeys are functions which associate widgets to a signal method or to a module function.

An extension can declare new toolkeys and widgets. For example, the hyperspy-gui-traitsui and hyperspy-gui-ipywidgets provide widgets for toolkeys declared in HyperSpy.

Registering toolkeys

To register a new toolkey:

1. Declare a new toolkey, e. g. by adding the hyperspy.ui_registry.add_gui_method decorator to the function you want to assign a widget to.

2. Register a new toolkey that you have declared in your package by adding it to the hyperspy_extension.yaml file, as in the following example:

GUI:
  # In order to assign a widget to a function, that function must declare
  # a `toolkey`. The `toolkeys` list contains a list of all the toolkeys
  # provided by extensions. In order to avoid name clashes, by convention,
  # toolkeys must start with the name of the package that provides them.
  toolkeys:
    - my_package.MyComponent

Registering widgets

In the example below, we register a new ipywidget widget for the my_package.MyComponent toolkey of the previous example. The function simply returns the widget to display. The key module defines where the functions resides.

GUI:
  widgets:
    ipywidgets:
      # Each widget is declared using a dictionary with two keys, `module` and `function`.
      my_package.MyComponent:
        # The function that creates the widget
        function: get_mycomponent_widget
        # The module where the function resides.
        module: my_package.widgets

Integration test suite

The integration test suite runs the test suite of HyperSpy and of all registered HyperSpy extensions on a daily basis against both the release and development versions. The build matrix is as follows:

Build matrix of the integration test suite

HyperSpy	Extension	Dependencies
Release	Release	Release
Release	Development	Release
RELEASE_next_patch	Release	Release
RELEASE_next_patch	Development	Release
RELEASE_next_minor	Release	Release
RELEASE_next_minor	Development	Release
RELEASE_next_minor	Development	Development
RELEASE_next_minor	Development	Pre-release

The development packages of the dependencies are provided by the scipy-wheels-nightly repository, which provides scipy, numpy, scikit-learn and scikit-image at the time of writing. The pre-release packages are obtained from PyPI and these will be used for any dependency which provides a pre-release package on PyPI.

A similar Integration test workflow can run from pull requests (PR) to the hyperspy repository when the label run-extension-tests is added to a PR or when a PR review is edited.