ParallelLab
: Running experiments locally in parallel¶
-
class
epyc.
ParallelLab
(notebook: epyc.labnotebook.LabNotebook = None, cores: int = 0)¶ A
Lab
that uses local parallelism.Unlike a basic
Lab
, this class runs multiple experiments in parallel to accelerate throughput. Unlike aClusterLab
it runs all jobs synchronously and locally, and so can’t make use of a larger compute cluster infrastructure and can’t run tasks in the background to be collected later. This does however mean thatepyc
can make full use of a multicore machine quite trivially.The optional
cores
parameter selects the number of cores to use:- a value of 1 uses 1 core (sequential mode);
- a value of +n uses n cores;
- a value of 0 uses all available cores; and
- a value of -n uses (available - n) cores.
So a value of
cores=-1
will run on 1 fewer cores than the total number of physical cores available on the machine.Important
This behaviour is slightly different to that of
joblib
as described here.Note that you can specify more cores to use than there are physical cores on the machine: this will have no positive effects. Note also that using all the cores on a machine may result in you being locked out of the user interface as your experiments consume all available computational resources, and may also be regarded as an unfriendly act by any other users with whom you share the machine.
Parameters: - notebook – (optional) the notebook used to store results
- cores – (optional) number of cores to use (defaults to all available)
-
ParallelLab.
numberOfCores
() → int¶ Return the number of cores we will use to run experiments.
Returns: maximum number of concurrent experiments
Running experiments¶
As with the sequential Lab
class, experiments run on a
ParallelLab
will be run synchronously: the calling thread
will block until all the experiments have completed.
Note
If you need asynchronous behaviour than you need to use a ClusterLab
.
-
ParallelLab.
runExperiment
(e: epyc.experiment.Experiment)¶ Run the experiment across the parameter space in parallel using the allowed cores. The experiments are all run synchronously.
Parameters: e – the experiment
Warning
ParallelLab
uses Python’s joblib
internally to create
parallelism, and joblib
in turn creates sub-processes in which to
run experiments. This means that the experiment is running in a
different process than the lab, and hence in a different address
space. The upshot of this is that any changes made to variables in
an experiment will only be visible to that experiment, and won’t be
seen by either other experiments or the lab. You can’t, for
example, have a class variable that’s accessed and updated by all
instances of the same experiment: this would work in a “normal”
Lab
, but won’t work on a ParallelLab
(or indeed
on a ClusterLab
).
The way to avoid any issues with this is to only communicate via
the Experiment
API, accepting parameters to set the
experiment up and returning them through a results dict.
Any updates to experimental parameters or metadata are also
communicated correctly (see Advanced experimental parameter handling).