Experiment
: A single computational experiment¶
-
class
epyc.
Experiment
¶ Base class for an experiment conducted in a lab.
An
Experiment
defines a computational experiment that can be run independently or (more usually) controlled from an instamnce of theLab
class. Experiments should be long-lasting, able to conduct repeated runs at several different parameter points.From an experimenter’s (or a lab’s) perspective, an experiment has public methods
set()
andrun()
. The former sets the parameters for the experiment; the latter runs the experiment. producing a set of results that include direct experimental results and metadata on how the experiment ran. A single run may produce a list of result dicts if desired, each filled-in with the correct metadata.Experimental results, parameters, and metadata can be access directly from the
Experiment
object. The class also exposes an indexing interface to access experimental results by name.
Important
Experiments have quite a detailed lifcycle that it is important to understand when writing any but the simplest experiments. See The lifecycle of an experiment for a detailed description.
Creating the results dict¶
The results dict is the structure returned from running an
Experiment
. They are simply nested Python dicts which can be
created using a static method.
-
static
Experiment.
resultsdict
() → Dict[str, Dict[str, Any]]¶ Create an empty results dict, structured correctly.
Returns: an empty results dict
The ResultsDict
type is an alias for this structure. The dict has three top-level keys:
-
Experiment.
PARAMETERS
¶ Results dict key for describing the point in the parameter space the experiment ran on.
-
Experiment.
RESULTS
¶ Results dict key for the experimental results generated at the experiment’s parameter point.
-
Experiment.
METADATA
¶ Results dict key for metadata values, mainly timing.
The contents of the parameters and results dicts are defined by the Experiment
designer. The metadata dict includes a number of standard elements.
Standard metadata elements¶
The metadata elements include:
-
Experiment.
STATUS
¶ Metadata element that will be True if experiment completed successfully, False otherwise.
-
Experiment.
EXCEPTION
¶ Metadata element containing the exception thrown if experiment failed.
-
Experiment.
TRACEBACK
¶ Metadata element containing the traceback from the exception (as a string).
-
Experiment.
START_TIME
¶ Metadata element for the datetime experiment started.
-
Experiment.
END_TIME
¶ Metadata element for the datetime experiment ended.
-
Experiment.
SETUP_TIME
¶ Metadata element for the time spent on setup in seconds.
-
Experiment.
EXPERIMENT_TIME
¶ Metadata element for the time spent on experiment itself in seconds.
-
Experiment.
TEARDOWN_TIME
¶ Metadata element for the time spent on teardown in seconds.
Experiment
sub-classes may add other metata elements as required.
Note
Since metadata can come from many sources, it’s important to consider the names given to the different values. epyc uses structured names based on the class names to avoid collisions.
If the Experiment
has run successfully, the
Experiment.STATUS
key will be True
; if not, it will be
False
and the Experiment.EXCEPTION
key will contain the
exception that was raised to cause it to fail and the Experiment.TRACEBACK
key will hold the traceback for that exception.
Warning
The exception traceback, if present, is a string, not a traceback
object, since these
do not work well in a distributed environment.
Configuring the experiment¶
An Experiment
is given its parameters, a “point” in the
parameter space being explored, by called Experiment.set()
. This
takes a dict of named parameters and returns the Experiment
itself.
-
Experiment.
set
(params: Dict[str, Any]) → epyc.experiment.Experiment¶ Set the parameters for the experiment, returning the now-configured experiment.
Parameters: params – the parameters Returns: the experiment
-
Experiment.
configure
(params: Dict[str, Any])¶ Configure the experiment for the given parameters. The default stores the parameters for later use. Be sure to call this base method when overriding.
Parameters: params – the parameters
-
Experiment.
deconfigure
()¶ De-configure the experiment prior to setting new parameters. The default removes the parameters. Be sure to call this base method when overriding.
Important
Be sure to call the base methods when overriding Experiment.configure()
and
Experiment.deconfigure()
. (There should be no need to override Experiment.set()
.)
Running the experiment¶
To run the experiment, a call to Experiment.run()
will run the experiment
at the given parameter point.
The dict of experimental results returned by Experiment.do()
is
formed into a results dict by the private Experiment.report()
method. Note the division of responsibilities here: Experiment.do()
returns the results
of the experiment (as a dict), which are then wrapped in a further dict by
Experiment.report()
.
If the experiment returns a list of results dicts instead of just a single set, then by default they are each wrapped in the same parameters and metadata and returned as a list of results dicts.
-
Experiment.
setUp
(params: Dict[str, Any])¶ Set up the experiment. Default does nothing.
Parameters: params – the parameters of the experiment
-
Experiment.
run
(fatal: bool = False) → Dict[str, Dict[str, Any]]¶ Run the experiment, using the parameters set using
set()
. A “run” consists of callingsetUp()
,do()
, andtearDown()
, followed by collecting and storing (and returning) the experiment’s results. If running the experiment raises an exception, that will be returned in the metadata along with its traceback to help with experiment debugging. If fatal is True it will also be raised from this method: the default prints the exception but doesn’t raise it.Parameters: fatal – (optional) raise any exceptions (default suppresses them into the results dict) Returns: a results dict
-
Experiment.
do
(params: Dict[str, Any]) → Union[Dict[str, Any], List[Dict[str, Dict[str, Any]]]]¶ Do the body of the experiment. This should be overridden by sub-classes. Default does nothing.
An experiment can return two types of results:
- a dict mapping names to values for experimental results; or
- a results dict list, each of which represents a fully-formed experiment
Parameters: params – a dict of parameters for the experiment Returns: the experimental results
-
Experiment.
tearDown
()¶ Tear down the experiment. Default does nothing.
-
Experiment.
report
(params: Dict[str, Any], meta: Dict[str, Any], res: Union[Dict[str, Any], List[Dict[str, Dict[str, Any]]]]) → Dict[str, Dict[str, Any]]¶ Return a properly-structured dict of results. The default returns a dict with results keyed by
Experiment.RESULTS
, the data point in the parameter space keyed byExperiment.PARAMETERS
, and timing and other metadata keyed byExperiment.METADATA
. Overriding this method can be used to record extra metadata values, but be sure to call the base method as well.If the experimental results are a list of results dicts, then we report a results dict whose results are a list of results dicts. This is used by
RepeatedExperiment
amd other experiments that want to report multiple sets of results.Parameters: - params – the parameters we ran under
- meta – the metadata for this run
- res – the direct experimental results from do()
Returns:
Important
Again, if you override any of these methods, be sure to call the base class
to get the default management functionality. (There’s no such basic functionality
for Experiment.do()
, though, so it can be overridden freely.)
Note
You can update the parameters controlling the experiment from
within Experiment.setUp()
and Experiment.do()
, and
these changes will be saved in the results dict eventually
returned by Experiment.run()
.
Accessing results¶
The easiest way to access an Experiment
’s results is to store
the results dict returned by Experiment.run()
. It is also
possible to access the results post facto from the
Experiment
object itself, or using a dict-like interface keyed
by name. These operations only make sense on a newly-run Experiment
.
-
Experiment.
success
() → bool¶ Test whether the experiment has been run successfully. This will be False if the experiment hasn’t been run, or if it’s been run and failed.
Returns: True if the experiment has been run successfully
-
Experiment.
failed
() → bool¶ Test whether an experiment failed. This will be True if the experiment has been run and has failed, which means that there will be an exception and traceback information stored in the metadata. It will be False if the experiment hasn’t been run.
Returns: True if the experiment has failed
-
Experiment.
results
() → Union[Dict[str, Dict[str, Any]], List[Dict[str, Dict[str, Any]]]]¶ Return a complete results dict. Only really makes sense for recently-executed experimental runs.
Returns: the results dict, or a list of them
-
Experiment.
experimentalResults
() → Union[Dict[str, Any], List[Dict[str, Any]]]¶ Return the experimental results from our last run. This will be None if we haven’t been run, or if we ran and failed.
Returns: the experimental results dict, which may be empty, and may be a list of dicts
-
Experiment.
__getitem__
(k: str) → Any¶ Return the given element of the experimental results. This only gives access to the experimental results, not to the parameters or metadata.
Parameters: k – the result key Returns: the value Raises: KeyError if there is no such result
-
Experiment.
parameters
() → Dict[str, Any]¶ Return the current experimental parameters, which will be None if none have been given by a call to set()
Returns: the parameters, which may be empty
-
Experiment.
metadata
() → Dict[str, Any]¶ Return the metadata we collected at out last execution, which will be None if we’ve not been executed and an empty dict if we’re mid-run (i.e., if this method is called from do() for whatever reason).
Returns: the metadata, which may be empty