SummaryExperiment: Statistical summaries of experiments

class epyc.SummaryExperiment(ex: epyc.experiment.Experiment, summarised_results: List[str] = None)

Bases: epyc.experimentcombinator.ExperimentCombinator

An experiment combinator that takes an underlying experiment and returns summary statistics for some of its results. This only really makes sense for experiments that return lists of results, such as those conducted using RepeatedExperiment, but it works with any experiment.

When run, a summary experiment summarises the experimental results, creating a new set of results that include the mean and variance for each result that the underyling experiments generated. (You can also select which results to summarise.) The raw results are discarded. The new results have the names of the raw results with suffices for mean, median, variance, and extrema.

The summarisation obviously only works on result keys coming from the underlying experiments that are numeric. The default behaviour is to try to summarise all keys: you can restrict this by providing a list of keys to the constructor in the summarised_results keyword argument. Trying to summarise non-numeric results will be ignored (with a warining).

The summary calculations only include those experimental runs that succeeded, that is that have their status set to True. Failed runs are ignored.

Extra metadata elements in the results dict

Summarisation removes the raw results of the various experiments from the results dict and replaces them with summary values. Each summarised value is replaced by five derived values for the mean, median, variance, and extrema, with standard suffices.

SummaryExperiment.MEAN_SUFFIX

Suffix for the mean of the underlying values.

SummaryExperiment.MEDIAN_SUFFIX

Suffix for the median of the underlying values.

SummaryExperiment.VARIANCE_SUFFIX

Suffix for the variance of the underlying values.

SummaryExperiment.MIN_SUFFIX

Suffix for the minimum of the underlying values.

SummaryExperiment.MAX_SUFFIX

Suffix for the maximum of the underlying values.

The metadata also enumerates the number of experiments performed, the number summarised (since unsuccessful experiments are omitted), and any exceptions raised.

SummaryExperiment.UNDERLYING_RESULTS

Metadata element for the number of results that were obtained.

SummaryExperiment.UNDERLYING_SUCCESSFUL_RESULTS

Metadata elements for the number of results that were summarised.

Running the experiment

SummaryExperiment.do(params: Dict[str, Any]) → Dict[str, Any]

Perform the underlying experiment and summarise its results. Our results are the summary statistics extracted from the results of the instances of the underlying experiment that we performed.

We drop from the calculations any experiments whose completion status was False, indicating an error. Our own completion status will be True unless we had an error summarising a field (usually caused by trying to summarise non-numeric data).

We record the exceptions generated by any experiment we summarise under the metadata key SummaryExperiment.UNDERLYING_EXCEPTIONS

Parameters:params – the parameters to the underlying experiment Returns:the summary statistics of the underlying results

Creating and changing the summary statistics

SummaryExperiment.summarise(results: List[Dict[str, Dict[str, Any]]]) → Dict[str, Dict[str, Any]]

Generate a summary of results from a list of experimental results dicts returned by running the underlying experiment. By default we generate mean, median, variance, and extrema for each value recorded.

Override this method to create different or extra summary statistics.

Parameters:results – an array of experimental results dicts Returns:a dict of summary statistics