Experiment: A single computational experiment

class epyc.Experiment

Base class for an experiment conducted in a lab.

An Experiment defines a computational experiment that can be run independently or (more usually) controlled from an instamnce of the Lab class. Experiments should be long-lasting, able to conduct repeated runs at several different parameter points.

From an experimenter’s (or a lab’s) perspective, an experiment has public methods set() and run(). The former sets the parameters for the experiment; the latter runs the experiment. producing a set of results that include direct experimental results and metadata on how the experiment ran. A single run may produce a list of result dicts if desired, each filled-in with the correct metadata.

Experimental results, parameters, and metadata can be access directly from the Experiment object. The class also exposes an indexing interface to access experimental results by name.

Important

Experiments have quite a detailed lifcycle that it is important to understand when writing any but the simplest experiments. See The lifecycle of an experiment for a detailed description.

Creating the results dict

The results dict is the structure returned from running an Experiment. They are simply nested Python dicts which can be created using a static method.

static Experiment.resultsdict() → Dict[str, Dict[str, Any]]

Create an empty results dict, structured correctly.

Returns:an empty results dict

The ResultsDict type is an alias for this structure. The dict has three top-level keys:

Experiment.PARAMETERS

Results dict key for describing the point in the parameter space the experiment ran on.

Experiment.RESULTS

Results dict key for the experimental results generated at the experiment’s parameter point.

Experiment.METADATA

Results dict key for metadata values, mainly timing.

The contents of the parameters and results dicts are defined by the Experiment designer. The metadata dict includes a number of standard elements.

Standard metadata elements

The metadata elements include:

Experiment.STATUS

Metadata element that will be True if experiment completed successfully, False otherwise.

Experiment.EXCEPTION

Metadata element containing the exception thrown if experiment failed.

Experiment.TRACEBACK

Metadata element containing the traceback from the exception (as a string).

Experiment.START_TIME

Metadata element for the datetime experiment started.

Experiment.END_TIME

Metadata element for the datetime experiment ended.

Experiment.SETUP_TIME

Metadata element for the time spent on setup in seconds.

Experiment.EXPERIMENT_TIME

Metadata element for the time spent on experiment itself in seconds.

Experiment.TEARDOWN_TIME

Metadata element for the time spent on teardown in seconds.

Experiment sub-classes may add other metata elements as required.

Note

Since metadata can come from many sources, it’s important to consider the names given to the different values. epyc uses structured names based on the class names to avoid collisions.

If the Experiment has run successfully, the Experiment.STATUS key will be True; if not, it will be False and the Experiment.EXCEPTION key will contain the exception that was raised to cause it to fail and the Experiment.TRACEBACK key will hold the traceback for that exception.

Warning

The exception traceback, if present, is a string, not a traceback object, since these do not work well in a distributed environment.

Configuring the experiment

An Experiment is given its parameters, a “point” in the parameter space being explored, by called Experiment.set(). This takes a dict of named parameters and returns the Experiment itself.

Experiment.set(params: Dict[str, Any]) → epyc.experiment.Experiment

Set the parameters for the experiment, returning the now-configured experiment.

Parameters:params – the parameters
Returns:the experiment
Experiment.configure(params: Dict[str, Any])

Configure the experiment for the given parameters. The default stores the parameters for later use. Be sure to call this base method when overriding.

Parameters:params – the parameters
Experiment.deconfigure()

De-configure the experiment prior to setting new parameters. The default removes the parameters. Be sure to call this base method when overriding.

Important

Be sure to call the base methods when overriding Experiment.configure() and Experiment.deconfigure(). (There should be no need to override Experiment.set().)

Running the experiment

To run the experiment, a call to Experiment.run() will run the experiment at the given parameter point.

The dict of experimental results returned by Experiment.do() is formed into a results dict by the private Experiment.report() method. Note the division of responsibilities here: Experiment.do() returns the results of the experiment (as a dict), which are then wrapped in a further dict by Experiment.report().

If the experiment returns a list of results dicts instead of just a single set, then by default they are each wrapped in the same parameters and metadata and returned as a list of results dicts.

Experiment.setUp(params: Dict[str, Any])

Set up the experiment. Default does nothing.

Parameters:params – the parameters of the experiment
Experiment.run(fatal: bool = False) → Dict[str, Dict[str, Any]]

Run the experiment, using the parameters set using set(). A “run” consists of calling setUp(), do(), and tearDown(), followed by collecting and storing (and returning) the experiment’s results. If running the experiment raises an exception, that will be returned in the metadata along with its traceback to help with experiment debugging. If fatal is True it will also be raised from this method: the default prints the exception but doesn’t raise it.

Parameters:fatal – (optional) raise any exceptions (default suppresses them into the results dict)
Returns:a results dict
Experiment.do(params: Dict[str, Any]) → Union[Dict[str, Any], List[Dict[str, Dict[str, Any]]]]

Do the body of the experiment. This should be overridden by sub-classes. Default does nothing.

An experiment can return two types of results:

  • a dict mapping names to values for experimental results; or
  • a results dict list, each of which represents a fully-formed experiment
Parameters:params – a dict of parameters for the experiment
Returns:the experimental results
Experiment.tearDown()

Tear down the experiment. Default does nothing.

Experiment.report(params: Dict[str, Any], meta: Dict[str, Any], res: Union[Dict[str, Any], List[Dict[str, Dict[str, Any]]]]) → Dict[str, Dict[str, Any]]

Return a properly-structured dict of results. The default returns a dict with results keyed by Experiment.RESULTS, the data point in the parameter space keyed by Experiment.PARAMETERS, and timing and other metadata keyed by Experiment.METADATA. Overriding this method can be used to record extra metadata values, but be sure to call the base method as well.

If the experimental results are a list of results dicts, then we report a results dict whose results are a list of results dicts. This is used by RepeatedExperiment amd other experiments that want to report multiple sets of results.

Parameters:
  • params – the parameters we ran under
  • meta – the metadata for this run
  • res – the direct experimental results from do()
Returns:

a results dict

Important

Again, if you override any of these methods, be sure to call the base class to get the default management functionality. (There’s no such basic functionality for Experiment.do(), though, so it can be overridden freely.)

Note

You can update the parameters controlling the experiment from within Experiment.setUp() and Experiment.do(), and these changes will be saved in the results dict eventually returned by Experiment.run().

Accessing results

The easiest way to access an Experiment’s results is to store the results dict returned by Experiment.run(). It is also possible to access the results post facto from the Experiment object itself, or using a dict-like interface keyed by name. These operations only make sense on a newly-run Experiment.

Experiment.success() → bool

Test whether the experiment has been run successfully. This will be False if the experiment hasn’t been run, or if it’s been run and failed.

Returns:True if the experiment has been run successfully
Experiment.failed() → bool

Test whether an experiment failed. This will be True if the experiment has been run and has failed, which means that there will be an exception and traceback information stored in the metadata. It will be False if the experiment hasn’t been run.

Returns:True if the experiment has failed
Experiment.results() → Union[Dict[str, Dict[str, Any]], List[Dict[str, Dict[str, Any]]]]

Return a complete results dict. Only really makes sense for recently-executed experimental runs.

Returns:the results dict, or a list of them
Experiment.experimentalResults() → Union[Dict[str, Any], List[Dict[str, Any]]]

Return the experimental results from our last run. This will be None if we haven’t been run, or if we ran and failed.

Returns:the experimental results dict, which may be empty, and may be a list of dicts
Experiment.__getitem__(k: str) → Any

Return the given element of the experimental results. This only gives access to the experimental results, not to the parameters or metadata.

Parameters:k – the result key
Returns:the value
Raises:KeyError if there is no such result
Experiment.parameters() → Dict[str, Any]

Return the current experimental parameters, which will be None if none have been given by a call to set()

Returns:the parameters, which may be empty
Experiment.metadata() → Dict[str, Any]

Return the metadata we collected at out last execution, which will be None if we’ve not been executed and an empty dict if we’re mid-run (i.e., if this method is called from do() for whatever reason).

Returns:the metadata, which may be empty