cloudmesh_client.common package¶
Subpackages¶
Submodules¶
cloudmesh_client.common.BaseConfigDict module¶
Some simple yaml file reader
-
class
cloudmesh_client.common.BaseConfigDict.
BaseConfigDict
(*args, **kwargs)[source]¶ Bases:
collections.OrderedDict
A class to obtain an OrderedDict from a yaml file.
-
error_keys_not_found
(keys)[source]¶ Check if the requested keys are found in the dict.
Parameters: keys – keys to be looked for
-
get
(*keys)[source]¶ returns the dict of the information as read from the yaml file. To access the file safely, you can use the keys in the order of the access. Example: get(“provisioner”,”policy”) will return the value of config[“provisioner”][“policy”] from the yaml file if it does not exists an error will be printing that the value does not exists. Alternatively you can use the . notation e.g. get(“provisioner.policy”)
-
load
(filename)[source]¶ Loads the yaml file with the given filename.
Parameters: filename – the name of the yaml file
-
make_a_copy
(location=None)[source]¶ Creates a backup of the file specified in the location. The backup filename appends a .bak.NO where number is a number that is not yet used in the backup directory.
TODO: This function should be moved to another file maybe XShell
Parameters: location – the location of the file to be backed up
-
read
(filename)[source]¶ Loads the information in the yaml file. It is the same as load and is used for compatibility reasons.
Parameters: filename – the name of the yaml file
-
set
(value, *keys)[source]¶ Sets the dict of the information as read from the yaml file. To access the file safely, you can use the keys in the order of the access. Example: set(“{‘project’:{‘fg82’:[i0-i10]}}”, “provisioner”,”policy”) will set the value of config[“provisioner”][“policy”] in the yaml file if it does not exists an error will be printing that the value does not exists. Alternatively you can use the . notation e.g. set(“{‘project’:{‘fg82’:[i0-i10]}}”, “provisioner.policy”)
-
write
(filename=None, output='dict', attribute_indent=4)[source]¶ This method writes the dict into various outout formats. This includes a dict, json, and yaml
Parameters: - filename – the file in which the dict is written
- output – is a string that is either “dict”, “json”, “yaml”
- attribute_indent – character indentation of nested attributes in
-
-
class
cloudmesh_client.common.BaseConfigDict.
OrderedJsonEncoder
(skipkeys=False, ensure_ascii=True, check_circular=True, allow_nan=True, sort_keys=False, indent=None, separators=None, encoding='utf-8', default=None, use_decimal=True, namedtuple_as_object=True, tuple_as_array=True, bigint_as_string=False, item_sort_key=None, for_json=False, ignore_nan=False, int_as_string_bitcount=None, iterable_as_array=False)[source]¶ Bases:
simplejson.encoder.JSONEncoder
-
indent
= 4¶
-
-
cloudmesh_client.common.BaseConfigDict.
check_file_for_tabs
(filename, verbose=True)[source]¶ identifies if the file contains tabs and returns True if it does. It also prints the location of the lines and columns. If verbose is set to False, the location is not printed.
Parameters: filename – the filename Return type: True if there are tabs in the file
-
cloudmesh_client.common.BaseConfigDict.
ordered_dump
(data, stream=None, Dumper=<class 'yaml.dumper.Dumper'>, **kwds)[source]¶ writes the dict into an ordered yaml.
Parameters: - data – The ordered dict
- stream – the stream
- Dumper – the dumper such as yaml.SafeDumper
-
cloudmesh_client.common.BaseConfigDict.
ordered_load
(stream, Loader=<class 'yaml.loader.Loader'>, object_pairs_hook=<class 'collections.OrderedDict'>)[source]¶ Loads an ordered dict into a yaml while preserving the order
Parameters: - stream – the name of the stream
- Loader – the yam loader (such as yaml.SafeLoader)
- object_pairs_hook – the ordered dict
-
cloudmesh_client.common.BaseConfigDict.
read_yaml_config
(filename, check=True, osreplace=True, exit=True)[source]¶ reads in a yaml file from the specified filename. If check is set to true the code will fail if the file does not exist. However if it is set to false and the file does not exist, None is returned.
Parameters: - filename – the file name
- check – if True fails if the file does not exist, if False and the file does not exist return will be None
cloudmesh_client.common.ConfigDict module¶
-
class
cloudmesh_client.common.ConfigDict.
Config
[source]¶ Bases:
object
-
classmethod
check_file_for_tabs
(filename, verbose=True)[source]¶ identifies if the file contains tabs and returns True if it does. It also prints the location of the lines and columns. If verbose is set to False, the location is not printed. :param filename: the filename :type filename: str :rtype: True if there are tabs in the file
-
classmethod
find_file
(filename, load_order=None, verbose=False)[source]¶ find the specified file in the list of directories that are given in the array load_order :param filename: the file name :type filename: str :param load_order: an array with path names in with the filename is looked for. :type load_order: list of str :param verbose: :type verbose: bool :return: file name if successful :rtype: string if the file exists or None otherwise
-
classmethod
-
class
cloudmesh_client.common.ConfigDict.
ConfigDict
(filename, load_order=None, verbose=False, etc=False, reload=False)[source]¶ Bases:
object
-
classmethod
check
(filename)[source]¶ checks the filename if it is syntactically correct and does not include tabs :param filename: :type filename: string :return:
-
data
= {}¶
-
filename
= None¶
-
json
¶ Parameters: start – start key in dot notation returns the dict in json format :return: json string version :rtype: string
-
load
(filename)[source]¶ loads the configuration from the yaml filename :param filename: :type filename: string :return:
-
make_a_copy
(location=None)[source]¶ Creates a backup of the file specified in the location. The backup filename appends a .bak.NO where number is a number that is not yet used in the backup directory. TODO: This function should be moved to another file maybe XShell :param location: the location of the file to be backed up
-
save
(filename=None)[source]¶ saves the configuration in the given filename, if it is none the filename at load time is used. :param filename: the file name :type filename: string :return:
-
versions
= ['4.1']¶
-
write
(filename=None, output='dict', attribute_indent=4)[source]¶ This method writes the dict into various outout formats. This includes a dict, json, and yaml :param filename: the file in which the dict is written :param output: is a string that is either “dict”, “json”, “yaml” :param attribute_indent: character indentation of nested attributes in
-
yaml
¶ returns the dict in yaml format :return: returns the yaml output of the dict :rtype: string:
-
classmethod
cloudmesh_client.common.DictDB module¶
-
class
cloudmesh_client.common.DictDB.
DictDB
(filename, d=None, attributes=None)[source]¶ Bases:
dict
Implements a persistent dictionary using sqlite
Usage:
- element = {“name”:, “abc”,
- “address”: “here”, “email”: “abc@example.com”}
The name in the dict is identifies where to store it.
d = PersistentData(os.path.join(“~”, “.cloudmesh/cloudmesh.db”))
d.add(element)
- d.delete(“name”: “abc”)
- # deletes all elements from the db where ethe name is matching abc
- d.json
- # exports to json
- d.yaml
- # exports a yaml as a string
- d.get(“name”: “abc”)
- gets the first match
- d.find(“name”: “abc”)
- finds the first match
-
add
(d)[source]¶ adds the dict to the database. The attribute “name” is used to define a unique name for the object in the database
Parameters: d – the dict that at minimum must contain a name attribute
-
attributes
(attribute_names)[source]¶ defines the attributes that each record can have. If a database has previously defined and new attributes are added the database records are updated.
Parameters: attribute_names – The attribute names
-
backup
(filename=None)[source]¶ backs up the current database. If the filename is omitted, the backup will be created in the same directory as the database with the postfix .bak.# where # is the largest number that has not yet been assigned. For example., lets assume a backup exists with the name file.bak.9, than the next higher number is used (e.g. 10) and the backup file file.back.10 is used.
Parameters: filename – the backup filename. Returns:
-
delete
(operator='and', **kwargs)[source]¶ deletes all elements in that match the query formulated by kwargs and the operator operators that are allowed are = and, or
Parameters: - kwargs – the attributes that we look for
- operator – the operator and / or
-
find
(operator='and', **kwargs)[source]¶ finds all elements in that match the query formulated by kwargs and the operator operators that are allowed are = and, or
Parameters: - kwargs – the attributes that we look for
- operator – the operator and / or
-
get
(operator='and', **kwargs)[source]¶ finds the first elements in that match the query formulated by kwargs and the operator operators that are allowed are = and, or
Parameters: - kwargs – the attributes that we look for
- operator – the operator and / or
-
json
¶ Returns: the json object of the database
-
yaml
¶ Returns: a yaml string of the database
cloudmesh_client.common.DynamicClass module¶
cloudmesh_client.common.Error module¶
cloudmesh_client.common.FlatDict module¶
-
class
cloudmesh_client.common.FlatDict.
FlatDict
(d)[source]¶ Bases:
dict
A data structure to manage a flattened dict. It is initialized by passing the dict at time of initialization.
-
dict
¶
-
-
cloudmesh_client.common.FlatDict.
flatten
(d, parent_key='', sep='__')[source]¶ flattens the dict into a one dimensional dictionary
Parameters: - d – multidimensional dict
- parent_key – replaces from the parent key
- sep – the separation character used when fattening. the default is __
Returns: the flattened dict
-
cloudmesh_client.common.FlatDict.
key_prefix_replace
(d, prefix, new_prefix='')[source]¶ replaces the list of prefix in keys of a flattened dict
Parameters: - d – the flattened dict
- prefix (list of str) – a list of prefixes that are replaced with a new prefix. Typically this will be “”
- new_prefix – The new prefix. By default it is set to “”
Returns: the dict with the keys replaced as specified
cloudmesh_client.common.FlatDict2 module¶
-
class
cloudmesh_client.common.FlatDict2.
FlatDict2
[source]¶ Bases:
object
-
classmethod
convert
(obj, flatten=True)[source]¶ This function converts object into a Dict optionally Flattening it :param obj: Object to be converted :param flatten: boolean to specify if the dict has to be flattened :return dict: the dict of the object (Flattened or Un-flattened)
-
primitive
= (<type 'int'>, <type 'str'>, <type 'bool'>, <type 'unicode'>, <type 'dict'>, <type 'list'>)¶
-
classmethod
cloudmesh_client.common.GitInfo module¶
managing information from GIT
-
class
cloudmesh_client.common.GitInfo.
GitInfo
[source]¶ Bases:
object
This class can be used to return some elementary information about the git hub directory. This includes the:
- version number of the code
- the list of people contributing to the code
- emails of the people contributing
- a statistic about canged code by person
Example:
gitinfo = GitInfo() print gitinfo.authors() pprint(gitinfo.authors("dict")) pprint(gitinfo.emails()) pprint(gitinfo.emails("dict")) pprint(gitinfo.info()) print gitinfo.stat("laszewski@gmail.com") stats = gitinfo.compute() print stats
returns the authors of the authors either as a text or as a dict. The format is specified as an argument.
Parameters: format_arg – if “dict” is specified a dict will be returned
-
compute
()[source]¶ computes the statistic of all authors in a git repository and returns the result as a dict. :rtype: a dict with the values
-
emails
(format_arg=None)[source]¶ returns the emails of the authors either as a text or as a dict. The format is specified as an argument.
Parameters: format_arg – if “dict” is specified a dict will be returned Return type: dict or arry of e-mails dependent on format_arg
-
info
()[source]¶ returns the information about authors, emails, and commits in a dict. :rtype: a dict with the information
cloudmesh_client.common.LibcloudDict module¶
cloudmesh_client.common.LogUtil module¶
-
class
cloudmesh_client.common.LogUtil.
LogUtil
[source]¶ Bases:
object
-
DEFAULT_LOG_LEVEL
= 'ERROR'¶
-
FORMAT
= '%(asctime)s [%(levelname)s] %(filename)s:%(lineno)s %(funcName)s() %(message)s'¶
-
LOG_LEVEL_KEY
= 'log_level'¶
-
category
= 'general'¶
-
static
get_level
()[source]¶ get the log level from database :param cloudname: The name of the cloud :return: the log level
-
static
get_level_obj
(log_level)[source]¶ gets the log level when passing a string :param log_level: case insensitive string. Valid values are debug,
info, warning, critical, errorReturns: a logging level
-
static
get_logger
()[source]¶ get caller file name :return: file name based on the context where the logger is caller
-
cloudmesh_client.common.Printer module¶
Convenient methods and classes to print tables
-
class
cloudmesh_client.common.Printer.
Printer
[source]¶ Bases:
object
-
classmethod
csv
(d, order=None, header=None, sort_keys=True)[source]¶ prints a table in csv format
Parameters: d (dict) – A a dict with dicts of the same type. - :param order:The order in which the columns are printed.
- The order is specified by the key names of the dict.
Parameters: - header (list or tuple of field names) – The Header of each of the columns
- sort_keys (bool) – TODO: not yet implemented
Returns: a string representing the table in csv format
-
classmethod
dict
(d, order=None, header=None, output='table', sort_keys=True, show_none='')[source]¶ TODO :param d: A a dict with dicts of the same type. :type d: dict :param order:The order in which the columns are printed.
The order is specified by the key names of the dict.Parameters: - header (list or tuple of field names) – The Header of each of the columns
- output (string) – type of output (table, csv, json, yaml or dict)
- sort_keys (bool) –
Returns:
-
classmethod
dict_table
(d, order=None, header=None, sort_keys=True, show_none='', max_width=40)[source]¶ prints a pretty table from an dict of dicts :param d: A a dict with dicts of the same type.
Each key will be a columnParameters: - order – The order in which the columns are printed. The order is specified by the key names of the dict.
- header (A list of string) – The Header of each of the columns
- sort_keys (string or a tuple of string (for sorting with multiple columns)) – Key(s) of the dict to be used for sorting. This specify the column(s) in the table for sorting.
-
classmethod
list
(l, order=None, header=None, output='table', sort_keys=True, show_none='')[source]¶ Parameters: - l – l is a lsit not a dict
- order –
- header –
- output –
- sort_keys –
- show_none –
- key –
Returns:
-
classmethod
cloudmesh_client.common.SSHkey module¶
cloudmesh_client.common.Shell module¶
-
class
cloudmesh_client.common.Shell.
Shell
[source]¶ Bases:
object
-
command
= {'windows': {}, 'darwin': {}, 'linux': {}}¶ big question for badi and others
how do we now define dynamically functions based on a list that we want to support
what we want is where args are multiple unlimited parameters to the function
- def f(args…):
name = get the name from f a = list of args…
cls.execute(cmd, arguments=a, capture=True, verbose=False)
commands = [‘ps’, ‘ls’, ….. ] for c in commands:
generate this command and add to this class dynamicallyor do something more simple
ls = cls.execute(‘cmd’, args…)
-
cygwin_path
= 'bin'¶
-
classmethod
execute
(cmd, arguments='', shell=False, cwd=None, traceflag=True, witherror=True)[source]¶ Run Shell command
Parameters: - cmd – command to run
- arguments – we dont know yet
- capture – if true returns the output
Returns:
-
classmethod
mkdir
(newdir)[source]¶ works the way a good mkdir should :) - already exists, silently complete - regular file in the way, raise an exception - parent directory(ies) does not exist, make them as well
-
-
class
cloudmesh_client.common.Shell.
Subprocess
(cmd, cwd=None, stderr=-1, stdout=-1, env=None)[source]¶ Bases:
object
cloudmesh_client.common.StopWatch module¶
Class for starting and stopping named timers.
This class is based on a similar java class in cyberaide, and java cog kit.
-
class
cloudmesh_client.common.StopWatch.
StopWatch
[source]¶ Bases:
object
A class to measure times between events.
-
classmethod
get
(name)[source]¶ returns the time of the timer.
Parameters: name (string) – the name of the timer Return type: the elapsed time
-
classmethod
start
(name)[source]¶ starts a timer with the given name.
Parameters: name (string) – the name of the timer
-
classmethod
stop
(name)[source]¶ stops the timer with a given name.
Parameters: name (string) – the name of the timer
-
timer_end
= {}¶
-
timer_start
= {}¶
-
classmethod
cloudmesh_client.common.TableParser module¶
-
class
cloudmesh_client.common.TableParser.
TableParser
(output='dict', header=True, index=None, change=None, strip=True, lower=True, strip_seperator=True, seperator='|', comment_chars='+#')[source]¶ Bases:
object
-
classmethod
convert
(table=None, output='dict', header=True, index=None, change=None, strip=True, lower=True, strip_seperator=True, seperator='|', comment_chars='+#')[source]¶
-
classmethod
cloudmesh_client.common.dotdict module¶
cloudmesh_client.common.hostlist module¶
Handle hostlist expressions. This module provides operations to expand and collect hostlist expressions. The hostlist expression syntax is the same as in several programs developed at LLNL (https://computing.llnl.gov/linux/). However in corner cases the behaviour of this module have not been compared for compatibility with pdsh/dshbak/SLURM et al.
-
cloudmesh_client.common.hostlist.
collect_hostlist
(hosts, silently_discard_bad=False)[source]¶ Collect a hostlist string from a Python list of hosts. We start grouping from the rightmost numerical part. Duplicates are removed. A bad hostname raises an exception (unless silently_discard_bad is true causing the bad hostname to be silently discarded instead).
-
cloudmesh_client.common.hostlist.
collect_hostlist_1
(left_right)[source]¶ Collect a hostlist string from a list of hosts (left+right). The input is a list of tuples (left, right). The left part is analyzed, while the right part is just passed along (it can contain already collected range expressions).
-
cloudmesh_client.common.hostlist.
expand_hostlist
(hostlist, allow_duplicates=False, sort=False)[source]¶ Expand a hostlist expression string to a Python list. Example: expand_hostlist(“n[9-11],d[01-02]”) ==>
[‘n9’, ‘n10’, ‘n11’, ‘d01’, ‘d02’]Unless allow_duplicates is true, duplicates will be purged from the results. If sort is true, the output will be sorted.
-
cloudmesh_client.common.hostlist.
expand_part
(s)[source]¶ Expand a part (e.g. “x[1-2]y[1-3][1-3]”) (no outer level commas).
-
cloudmesh_client.common.hostlist.
expand_range
(prefix, range_)[source]¶ Expand a range (e.g. 1-10 or 14), putting a prefix before.
-
cloudmesh_client.common.hostlist.
expand_rangelist
(prefix, rangelist)[source]¶ Expand a rangelist (e.g. “1-10,14”), putting a prefix before.
-
cloudmesh_client.common.hostlist.
format_range
(low, high, width)[source]¶ Format a range from low to high inclusively, with a certain width.
cloudmesh_client.common.ssh_config module¶
cloudmesh_client.common.todo module¶
cloudmesh_client.common.type module¶
cloudmesh_client.common.util module¶
-
cloudmesh_client.common.util.
HEADING
(txt=None)[source]¶ Prints a message to stdout with #### surrounding it. This is useful for nosetests to better distinguish them.
Parameters: txt (string) – a text message to be printed
-
cloudmesh_client.common.util.
auto_create_requirements
(requirements)[source]¶ creates a requirement.txt file form the requirements in the list. If the file exists, it get changed only if the requirements in the list are different from the existing file
Parameters: requirements – the requirements in a list
-
cloudmesh_client.common.util.
auto_create_version
(class_name, version, filename='__init__.py')[source]¶
-
cloudmesh_client.common.util.
backup_name
(filename)[source]¶ Parameters: filename (string) – given a filename creates a backup name of the form filename.bak.1. If the filename already exists the number will be increased as much as needed so the file does not exist in the given location. The filename can consists a path and is expanded with ~ and environment variables. Return type: string
prints a banner of the form with a frame of # arround the txt:
############################ # txt ############################
.
Parameters: - txt (string) – a text message to be printed
- c (character) – the character used instead of c
-
cloudmesh_client.common.util.
copy_files
(files_glob, source_dir, dest_dir)[source]¶ Parameters: - files_glob – *.yaml
- source_dir – source directiry
- dest_dir – destination directory
Returns:
-
cloudmesh_client.common.util.
exponential_backoff
(fn, sleeptime_s_max=1800)[source]¶ Calls fn until it returns True, with an exponentially increasing wait time between calls
-
cloudmesh_client.common.util.
generate_password
(length=8, lower=True, upper=True, number=True)[source]¶
-
cloudmesh_client.common.util.
grep
(pattern, filename)[source]¶ Very simple grep that returns the first matching line in a file. String matching only, does not do REs as currently implemented.
-
cloudmesh_client.common.util.
path_expand
(text)[source]¶ returns a string with expanded variable.
Parameters: - text – the path to be expanded, which can include ~ and $ variables
- text – string
prints a banner of the form with a frame of # arround the txt:
############################ # txt ############################
.
Parameters: - txt (string) – a text message to be printed
- c (character) – the character used instead of c
cloudmesh_client.common.vmname module¶
-
class
cloudmesh_client.common.vmname.
VMName
[source]¶ Bases:
object
-
static
format
(name=None)[source]¶ given a name that returns the following information as a tuple.
prefix - the prefix of the vm name index - the current index padding - the padding factor to fill the index with leading 0The input name format is as follows prefix-0011 upon return you will get
(prefix, 11, 4)if None is specified the current os user name will be used, the index will start as 1, and we will use four digits all in all for the padding.
Parameters: name – the name that derives a format for Returns: prefix, index, padding
-
static
get
(prefix=None, idx=None, user=None)[source]¶ Return a vm name to use next time. prefix or index can be given to update a vm name (optional)
- Args:
- prefix (str, optional): the name of prefix idx (int, str, optional): the index to increment. This can be a digit or arithmetic e.g. +5 or -3 can be used
-
static