ParallelLab: Running experiments locally in parallel

class epyc.ParallelLab(notebook: epyc.labnotebook.LabNotebook = None, cores: int = 0)

A Lab that uses local parallelism.

Unlike a basic Lab, this class runs multiple experiments in parallel to accelerate throughput. Unlike a ClusterLab it runs all jobs synchronously and locally, and so can’t make use of a larger compute cluster infrastructure and can’t run tasks in the background to be collected later. This does however mean that epyc can make full use of a multicore machine quite trivially.

The optional cores parameter selects the number of cores to use:

  • a value of 1 uses 1 core (sequential mode);
  • a value of +n uses n cores;
  • a value of 0 uses all available cores; and
  • a value of -n uses (available - n) cores.

So a value of cores=-1 will run on 1 fewer cores than the total number of physical cores available on the machine.

Important

This behaviour is slightly different to that of joblib as described here.

Note that you can specify more cores to use than there are physical cores on the machine: this will have no positive effects. Note also that using all the cores on a machine may result in you being locked out of the user interface as your experiments consume all available computational resources, and may also be regarded as an unfriendly act by any other users with whom you share the machine.

Parameters:
  • notebook – (optional) the notebook used to store results
  • cores – (optional) number of cores to use (defaults to all available)
ParallelLab.numberOfCores() → int

Return the number of cores we will use to run experiments.

Returns:maximum number of concurrent experiments

Running experiments

As with the sequential Lab class, experiments run on a ParallelLab will be run synchronously: the calling thread will block until all the experiments have completed.

Note

If you need asynchronous behaviour than you need to use a ClusterLab.

ParallelLab.runExperiment(e: epyc.experiment.Experiment)

Run the experiment across the parameter space in parallel using the allowed cores. The experiments are all run synchronously.

Parameters:e – the experiment

Warning

ParallelLab uses Python’s joblib internally to create parallelism, and joblib in turn creates sub-processes in which to run experiments. This means that the experiment is running in a different process than the lab, and hence in a different address space. The upshot of this is that any changes made to variables in an experiment will only be visible to that experiment, and won’t be seen by either other experiments or the lab. You can’t, for example, have a class variable that’s accessed and updated by all instances of the same experiment: this would work in a “normal” Lab, but won’t work on a ParallelLab (or indeed on a ClusterLab).

The way to avoid any issues with this is to only communicate via the Experiment API, accepting parameters to set the experiment up and returning them through a results dict. Any updates to experimental parameters or metadata are also communicated correctly (see Advanced experimental parameter handling).