cloudmesh.ee package

Subpackages

Submodules

cloudmesh.ee.experimentexecutor module

class ExperimentExecutor(verbose=False)[source]

Bases: object

Initialize the ExperimentExecutor Object

Parameters:: verbose (bool) – If true prints additional infromation when ExperimentExecutor methods are called

static count_word_occurrences(list_of_dicts)[source]

generate(script=None, variables=None, fences=('{', '}'))[source]

Expands the script template given the passed configuration.

Parameters:

script (str) – The string contents of the script file.
variables (dict) – The variables to be replaced. If omitted, uses the internal variables found.
fences ((str, str)) – A 2-position tuple that encloses template variables (start and end).

Returns:

The expanded script and a dict of replaced values: based on data.

Return type:

tuple[str, dict]

generate_experiment_batch_scripts(out_mode=None, replace_all=True)[source]

Utility method to genrerate either hierarchical or flat outputs; or debug.

NOte the falt mode is no longer supported

Parameters:: out_mode (string) – The mode of operation. One of: “debug”, “flat”, “hierarchical”
Returns:: generates the batch scripts
Return type:: None

generate_experiment_permutations(variable_str)[source]

Generates experiment permutations based on the passed string and appends it to the current instance.

Parameters:: variable_str (str) – A Parameter.expand string (such as epoch=[1-3] x=[1,4] y=[10,11])
Returns:: list with permutations over the experiment variables
Return type:: list

generate_setup_from_configuration(configuration, replace_all=True)[source]

generates a setup directory from the configuration parameters

Parameters:: configuration (dict) – the configuration dict

Returns:

generate_submit(name=None, job_type='slurm')[source]

Generates a list of commands based on the permutations for submission.

Parameters:

name (str) – Name of the experiments.
job_type (str) – Name of the job type used at submission such as ee, slurm, jsrun, mpirun, sh, bash.

Returns:

Prepares the internal data for the experiments. If verbose: is set, prints them.

Return type:

None

get_data(flat=False)[source]

converts the data from the yaml file with the flatdict

Parameters:: flat (boolen) – if set to true uses flatdict
Returns:: result of flatdict without the seperator
Return type:: dict

info(verbose=None)[source]

Prints information about the ExperimentExecutor object for debugging purposes

Parameters:: verbose (bool) – if True prints even more information
Returns:: None
Return type:: None

list(directory='project', config='config.yaml', log='*.out', debug=False, verbose=True, table=True)[source]

Lists all experiments

Returns:: prints the experiments
Return type:: None

load_source_template(script)[source]

Registers and reads the template script in for processing

This method must be run at least once prior to generating the batch script output.

Parameters:: script (str) – A string that is the path to the template script.
Returns:: The text of the template file unaltered.
Return type:: str

property now

The time of now in the format “%Y-m-%d”

Returns:: “%Y-m-%d”
Return type:: str

static permutation_generator(exp_dict)[source]

Creates a cartesian product of a {key: list, ...} object.

Parameters:

exp_dict (dict) – The dictionary to process.

Returns:

A list of dictionaries containing the resulting: cartesian product.

Return type:

list[dict]

Example::: my_dict = {“key1”: [“value1”, “value2”], “key2”: [“value3”, “value4”]} out = ExperimentExecutor.permutation_generator(my_dict) # out == [ # {“key1”: “value1”, “key2”: “value3”}, # {“key1”: “value1”, “key2”: “value4”}, # {“key1”: “value2”, “key2”: “value3”}, # {“key1”: “value2”, “key2”: “value4”}, # ]

static progress_to_dict(input_string)[source]

save_experiment_configuration(name=None)[source]

Saves the experiment configuration in a json file

Parameters:: name (str) – name of the configuration file
Returns:: writes into the file with given name the json content
Return type:: None

shutil = <module 'shutil' from '/opt/hostedtoolcache/Python/3.10.4/x64/lib/python3.10/shutil.py'>

spec_replace(spec)[source]

given a spec in yaml format, replaces all values in the yaml file that are of the form “{a.b}” with the value of

a:: b: value

if it is defined in the yaml file

Parameters:: spec (str) – yaml string
Returns:: replaced yaml file
Return type:: str

update_from_attributes(attributes)[source]

attributes are of the form “a=1,b=3”

Parameters:: attributes – A string to expand into key-value pairs
Returns:: self.data with updated dict
Return type:: dict

update_from_cm_variables(load=True)[source]

Adds Cloudmesh variables to the class’s data parameter as a flat dict.

Parameters:: load (bool) – Toggles execution; if false, method does nothing.
Returns:: self.data with updated cloudmesh variables
Return type:: dict

update_from_dict(d)[source]

Add a dict to self. data

Parameters:: d (dict) – dictionary
Returns:: self.data with updated dict
Return type:: dict

update_from_file(filename)[source]

Updates the configuration self.data with the data within the passed file.

Parameters:: filename (str) – The path to the configuration file (yaml, json, py, ipynb)
Returns:: self.data with updated cloudmesh variables from the specified file
Return type:: dict

update_from_os(variables)[source]

LOads all variables from os.environ into self.data with os.name

Parameters:: variables ([str]) – the name of the variables such as “HOME”
Returns:: self.data with all variaples added with os.name: value
Return type:: dict

update_from_os_environ()[source]

Updates the config file output to include OS environment variables

Returns:: The current value of the data configuration variable
Return type:: dict

static update_with_directory(directory, filename)[source]

prefix with the directory if the filename is not starting with . / ~

Parameters:

directory (str) – the string value of the directory
filename (str) – the filename

Returns:

directory/filename

Return type:

str

cloudmesh.ee.slurm module

Management of Slurm on the localhost.

This command is useful if you like to develop advances APIs using slurm, but do not want to do it on a production computer.

Curerntly onlu ubuntu 20.04 is supported.

The service can either be used as a commandline tool or as API. If you use the commandline YOU wnat to do the following. First, you need to start the service with:

cms ee slurm start

Next, you can test the status with:

cms ee slurm status

To submit jobs you can now use the regular slurm commands such as:

srun
ee
squeue

To stop the service please use:

cms ee slurm stop

The API reflects what we do on the commandline and we have the methods:

Slurm.start()
Slurm.stop()
Slurm.status()

class Slurm[source]

Bases: object

static start()[source]

Starts the Slurm service oon your local machine.

Returns:: On stdout prints the action information

static status()[source]

Shows the status of the SLurm service oon your local machine.

Returns:: On stdout prints the action information

static stop()[source]

Stops the SLurm service oon your local machine.

Returns:: On stdout prints the action information

Module contents

Management of custom batch script for experiments while exploring parameter sets.