hyperspy.samfire_utils.samfire_pool module
- class hyperspy.samfire_utils.samfire_pool.SamfirePool(**kwargs)
Bases:
hyperspy.utils.parallel_pool.ParallelPoolCreates and manages a pool of SAMFire workers. For based on ParallelPool - either creates processes using multiprocessing, or connects and sets up ipyparallel load_balanced_view.
Ipyparallel is managed directly, but multiprocessing pool is managed via three of queues:
Shared by all (master and workers) for distributing “load-balanced” work.
Shared by all (master and workers) for sending results back to the master
Individual queues from master to each worker. For setting up and addressing individual workers in general. This one is checked with higher priority in workers.
- pool
The pool object
- Type
{ipyparallel.load_balanced_view, multiprocessing.Pool}
- num_workers
The number of workers actually created (may be less than requested, but can’t be more)
- Type
- timestep
The timestep between “ticks” that the result queues are checked. Higher timestep means less frequent checking, which may reduce CPU load for difficult fits that take a long time to finish.
- Type
Creates a ParallelPool with additional methods for SAMFire. All arguments are passed to ParallelPool
- property _not_too_long
Returns bool if it has been too long after receiving the last result, probably meaning some of the workers timed out or hung.
- add_jobs(needed_number=None)
Adds jobs to the job queue that is consumed by the workers.
- Parameters
needed_number ({None, int}) – The number of jobs to add. If None (default), adds need_pixels
- collect_results(timeout=None)
Collects and parses all results, currently not processed due to being in the queue.
- Parameters
timeout ({None, flaot}) – the time to wait when collecting results. If None, the default timeout is used
- property need_pixels
Returns the number of pixels that should be added to the processing queue. At most is equal to the number of workers.
- parse(value)
Parse the value returned from the workers.
- Parameters
value (tuple of the form (keyword, the_rest)) –
Keyword currently can be one of [‘pong’, ‘Error’, ‘result’]. For each of the keywords, “the_rest” is a tuple of different elements, but generally the first one is always the worker_id that the result came from. In particular:
(‘pong’, (worker_id, pid, pong_time, optional_message_str))
(‘Error’, (worker_id, error_message_string))
(‘result’, (worker_id, pixel_index, result_dict, bool_if_result_converged))
- ping_workers(timeout=None)
Ping the workers and record one-way trip time and the process_id pid of each worker if available.
- Parameters
timeout ({None, flaot}) – the time to wait when collecting results after sending out the ping. If None, the default timeout is used
- prepare_workers(samfire)
Given SAMFire object, populate the workers with the required information. In case of multiprocessing, start worker listening to the queues.
- Parameters
samfire (samfire) – the SAMFire object that will be using the pool
- run()
Run the full process of adding jobs to the processing queue, listening to the results and updating SAMFire as needed. Stops when timed out or no pixels are left to run.
Run the full procedure until no more pixels are left to run in the SAMFire.
- stop()
Stops the appropriate pool and (if ipyparallel) clears the memory and history.
- update_parameters()
Updates various worker parameters.
- Currently updates:
Optional components (that can be switched off by the worker)
Parameter boundaries
Goodness test