cloudmesh.sbatch package
Subpackages
Submodules
cloudmesh.sbatch.sbatch module
- class SBatch(verbose=False)[source]
Bases:
objectInitialize the SBatch Object
- Parameters:
verbose (bool) – If true prints additional infromation when SBatch methods are called
- generate(script=None, variables=None, fences=('{', '}'))[source]
Expands the script template given the passed configuration.
- Parameters:
script (str) – The string contents of the script file.
variables (dict) – the variables to be replaced, if ommitted uses the internal variables found
fences ((str,str)) – A 2 position tuple, that encloses template variables (start and end).
- Returns:
The script that has expanded its values based on data.
- Return type:
str
- generate_experiment_batch_scripts(out_mode=None, replace_all=True)[source]
Utility method to genrerate either hierarchical or flat outputs; or debug.
NOte the falt mode is no longer supported
- Parameters:
out_mode (string) – The mode of operation. One of: “debug”, “flat”, “hierarchical”
- Returns:
generates the batch scripts
- Return type:
None
- generate_experiment_permutations(variable_str)[source]
Generates experiment permutations based on the passed string and appends it to the current instance.
- Parameters:
variable_str (str) – A Parameter.expand string (such as epoch=[1-3] x=[1,4] y=[10,11])
- Returns:
list with permutations over the experiment variables
- Return type:
list
- generate_setup_from_configuration(configuration, replace_all=True)[source]
generates a setup directory from the configuration parameters :type configuration: :param configuration: the configuration dict :type configuration: dict :return: :rtype:
- generate_submit(name=None, job_type='slurm')[source]
Generates a list of commands based on the permutations for submission
- Parameters:
name (str) – Name of the experiments
job_type (str) – name of the job type used at submission such as sbatch, slurm, jsrun, mpirun, sh, bash
- Returns:
prepars the internal data for the experiments, if set to verbose, prints them
- Return type:
None
- get_data(flat=False)[source]
converts the data from the yaml file with the flatdict
- Parameters:
flat (boolen) – if set to true uses flatdict
- Returns:
result of flatdict without the seperator
- Return type:
dict
- info(verbose=None)[source]
Prints information about the SBatch object for debugging purposes
- Parameters:
verbose (bool) – if True prints even more information
- Returns:
None
- Return type:
None
- load_source_template(script)[source]
Registers and reads the template script in for processing
This method must be run at least once prior to generating the batch script output.
- Parameters:
script (str) – A string that is the path to the template script.
- Returns:
The text of the template file unaltered.
- Return type:
str
- property now
The time of now in the format “%Y-m-%d” :return: “%Y-m-%d” :rtype: str
- static permutation_generator(exp_dict)[source]
Creates a cartisian product of a {key: list, …} object.
- Parameters:
exp_dict (dict) – The dictionary to process
- Returns:
A list of dictionaries containing the resulting cartisian product.
- Return type:
list
- For example
my_dict = {“key1”: [“value1”, “value2”], “key2”: [“value3”, “value4”]} out = permutation_generator(my_dict) out # [{“key1”: “value1”, “key2”: ‘value3”},
# {“key1”: “value1”, “key2”: “value4”}, # {“key1”: “value2”, “key2”: “value3”}, # {“key1”: “value2”, “key2”: “value4”}
- save_experiment_configuration(name=None)[source]
Saves the experiment configuration in a json file
- Parameters:
name (str) – name of the configuration file
- Returns:
writes into the file with given name the json content
- Return type:
None
- spec_replace(spec)[source]
given a spec in yaml format, replaces all values in the yaml file that are of the form “{a.b}” with the value of
- a:
b: value
if it is defined in the yaml file
- Parameters:
spec (str) – yaml string
- Returns:
replaced yaml file
- Return type:
str
- update_from_attributes(attributes)[source]
attributes are of the form “a=1,b=3”
- Parameters:
attributes (
str) – A string to expand into key-value pairs- Returns:
self.data with updated dict
- Return type:
dict
- update_from_cm_variables(load=True)[source]
Adds Cloudmesh variables to the class’s data parameter as a flat dict.
- Parameters:
load (bool) – Toggles execution; if false, method does nothing.
- Returns:
self.data with updated cloudmesh variables
- Return type:
dict
- update_from_dict(d)[source]
Add a dict to self. data
- Parameters:
d (dict) – dictionary
- Returns:
self.data with updated dict
- Return type:
dict
- update_from_file(filename)[source]
Updates the configuration self.data with the data within the passed file.
- Parameters:
filename (str) – The path to the configuration file (yaml, json, py, ipynb)
- Returns:
self.data with updated cloudmesh variables from the specified file
- Return type:
dict
- update_from_os(variables)[source]
LOads all variables from os.environ into self.data with os.name
- Parameters:
variables ([str]) – the name of the variables such as “HOME”
- Returns:
self.data with all variaples added with os.name: value
- Return type:
dict
cloudmesh.sbatch.slurm module
Management of Slurm on the localhost.
This command is useful if you like to develop advances APIs using slurm, but do not want to do it on a production computer.
Curerntly onlu ubuntu 20.04 is supported.
The service can either be used as a commandline tool or as API. If you use the commandline YOU wnat to do the following. First, you need to start the service with:
cms sbatch slurm start
Next, you can test the status with:
cms sbatch slurm status
To submit jobs you can now use the regular slurm commands such as:
srun
sbatch
squeue
To stop the service please use:
cms sbatch slurm stop
The API reflects what we do on the commandline and we have the methods:
Slurm.start()
Slurm.stop()
Slurm.status()
Module contents
Management of custom batch script for experiments while exploring parameter sets.