Cloudmesh Comet CLI

We developed a command line based client tool for users to access Comet VC easily and securely. The CLI is part of a larger effort called Cloudmesh Client toolkit, which provides command line and shell interface of accessing heterogeneous clouds, clusters, and workstations right from the users’ computers. By utilizing this client tool, Comet VC users can access all capabilities to manage their virtual cluster, while bypassing the security concerns and the complexity of programming with the Comet nucleus API.

Teminology

We use in this section the following terminology:

computeset:

A group of compute nodes started together and being in some state (submitted, started, finished, failed). Each compute node can only belong to 1 computesets in submitted or active state.

frontend:

A node with limited computational resources used to manage a virtual cluster. Frontends run 24/7, have a public interface and a private interface. The public interface provides outside access to the virtual cluster while the private interface is used to manage/install the compute nodes.

image:

A file containing the contents and structure (ISO9660) of a disk volume which can be attached as a cdrom to a node.

console:

An interactive representation of the screen of a virtual cluster node (text or graphical) provided to assist with node installation and management.

virtual cluster:

A virtual cluster is a loosely or tightly connected network of virtual computers managed together by a virtual cluster administrator.

node:

The term node is used to refer to individual computers in a virtual cluster.

image attach:

Attach is an action applied to a node / image pair whereby the contents of the image are made available to a node on the next power on.

image detach:

Detach is an action applied to a node / image pair whereby the contents of the image are made unavailable to the node on the next power on.

Installation

Cloudmesh Comet CLI is developed using Python. The installation is starightforward and easy:

pip install cloudmesh_client

Both Python 2.7 and 3.5 is supported. You are HIGHLY recommended to use a virtualenv based environment to install and use this toolkit.

Configuration

The configuration of the cloudmesh client is done semi automatically for you. All you have to do after the installation is to call cloudmesh client once.

This is done best with the command:

cm help

This will automatically generate a template configuration file at:

~/.cloudmesh/cloudmesh.yaml

Next you can initiate the configuration file:

cm comet init

Follow the prompt to configure and select endpoint, and initialize the auth settings by provide your username and password. Once you are successfully authenticated it will retrieve the api key and secret and configure the cloudmesh.yaml with proper api auth information. Then you will be able to use the commands to manage your comet virtual cluster.

For the use of the Comet VC production system, you need to use the OTP generated by your assigned YubiKey. You need to have the key plugged in, and upon the request of your password, type your PIN first, and then tap your key.

Console access of nodes requires 2-factor Auth as well, so please have your YubiKey ready for console access

Commands

Comet CLI supports accessing multiple nucleus service endpoint, e.g., DEV system or Production system. You will know which system to use after your VC is setup. To check which system you are currently accessing, run this:

cm comet active

Or change to another system to interact with, e.g.:

cm comet active production

Next we list a number of important commands from the CLI that will help you managing your comet virtual clusters.

Getting information of your cluster(s); nodes; computesets; etc.

List all clusters owned by the authenticated identity (summarized format):

cm comet ll

List all clusters owned by the authenticated identity (detailed list):

cm comet cluster

List a cluster by name (we use here vc2 as example):

cm comet cluster vc2

Cluster listing view also supports other output format, e.g. json, csv, etc.:

cm comet cluster vc2 --format=csv

This can be useful for scripting.

List all defined computesets:

cm comet computeset

List one computeset:

cm comet computeset 63

To obtain the detailed information of a node:

cm comet node info vc2

Gives your the information of the vc2 frontnode. This includes essential network configuration information if you want to rebuild the frontend node.

cm comet node info vc2 vm-vc2-0

Gives the detailed information of the specified compute node.

By default it returned information in a table format. It also accepts the –format=FROMAT parameter so the output can be easily parsed while being used in scripting (FORMAT could be one of csv, json, yaml).

Power management of frontend node:

Power on the front end node of the specified cluster:

cm comet power on vc2

To power if off:

cm comet power off vc2

You can also reboot/reset/shutdown the fronend using the same syntax, e.g., to reboot:

cm comet power reboot vc2

Please note running frontend node itself would not charge you any allocations.

Resource allocation and tear down:

The requested cluster/VMs has to be started by requesting physical allocations. This will charge your allocation based on number of nodes and the walltime you requested.

Please note that each VM you requested corresponds to 24 cores (You have the exclusive access to the whole physical node), thus one VM node hour is equivalent to 24 SUs. So please TERMINATE the resources (see below) if you don’t actively use them. Keeping them idle (even SHUT OFF) will still cost your allocation.

Currently for the DEV resources you are not charged for any usage. Please use the DEV resource to develop/test your application/workflow, and move to the production for the actual usage. Please terminate the requested compute set(s) if you are not actively using them to free the resources so other users can use them as well.

Start a set of compute nodes in cluster vc2:

cm comet start vc2 vm-vc2-[0-3]

This will request the nodes for a default period of time and power on them (currently default as 2 days).

To request for a customized time period, use –walltime parameter. E.g., 100m (100 minutes), 6h (6 hours), 2d (2 days) , 1w (1 week):

cm comet start vc2 vm-vc2-[0-3] --walltime=6h

The above will put the request under the one allocation associated with the cluster. If your cluster have more than one allocations, use –allocation parameter:

cm comet start vc2 vm-vc2-[0-3] --allocation=YOUR_ALLOCATION

If you have more allocations, but does not specify via CLI, you will see a list of allocations to choose from to use.

You can also request a group of N nodes, if there is enough resource:

cm comet start vc2 --count=4

The comet system will find 4 available nodes from the specified cluster and start them as one computeset.

You can also start a single-node computeset:

cm comet start vc2 vm-vc2-[7]

or simply:

cm comet start vc2 vm-vc2-7

To power down all running nodes from an active computeset and also free the allocated resources:

cm comet terminate 123

This will gracefully shutdown the nodes in the group identified by computeset 123, and also free the unused allocations. A computeset will be teared down automatically when it reaches its requested walltime.

Please note you have to terminate the requested resources as such so it stops charging your account for the usage.

Power management of compute nodes:

You can power off and back on individual nodes from a cluster, without affecting other running nodes in the computesets. E.g.:

cm comet power off vc2 vm-vc2-[0-7]

and then:

cm comet power on vc2 vm-vc2-[0-7]

or shutdown gracefully a group of nodes:

cm comet power shutdown vc2 vm-vc2-[0-3]

Please note even if you powered off all nodes from an active computeset, the computeset itself along with the associated physical resource is still active and can only be accessed exclusively by you till the requested walltime is reached. During this time you can freely power the nodes back on and off.

Please be reminded that your account will still be charged in this case until you terminate the computeset as described earlier.

Please note that the node disks are persisted silently on the background periodically, or when you shut them off gracefully (‘cm comet terminate’ or ‘cm comet power shutdown’). So the next time you start the node, or powered if on, you can start with where you have left.

Getting Console access

Get console of the frontend:

cm comet console vc2

Get console of a running compute node:

cm comet console vc2 vm-vc2-0

Please note the console access requires 2-factor auth. The commands will ask your comet nucleus username (if not set previously) and password, and you will need your Yubikey for the access.

Once you have passed this extra step of auth, it will print out a shortlived URL for accessing the console of the node requested. It will also pop open a browser window if a supported OS/browser is detected. In OSX it will use the system default browser to display the console; and in Linux environment it will try to use firefox.

You can also get only the URL directly by calling:

cm comet console --link vc2

System image management

Get the list of images that are available to you:

cm comet iso list

Upload an image to the public shared folder:

cm comet iso upload /path/to/your/image.iso

Or with a specified new image name:

cm comet iso upload /path/to/your/image.iso --imagename=newimagename.iso

Attach an image to the frontend, by specifying the exact image name, or the index of the image in the iso list:

cm comet iso attach newimagename.iso vc2
cm comet iso attach 6 vc2

Or to a compute node:

cm comet iso attach newimagename.iso vc2 vm-vc2-0

To detach the attached iso image from frontend node:

cm comet iso detach vc2

Or from a compute node:

cm comet iso detach vc2 vm-vc2-0

Image attaching/detaching also works on compute nodes in bulk:

cm comet iso attach newimagename.iso vc2 vm-vc2-[0-4]

cm comet iso detach vc2 vm-vc2-[0-4]

Please note image attaching/detaching will only take effect AFTER you hard reboot the node (power off and then power on).

Other commands:

You can rename a compute node, or a list of nodes in batch:

cm comet node rename vc2 vm-vc2-0 mynode0

cm comet node rename vc2 vm-vc2-[0-7] newname-[0-7]

Please use hostlist format to specify the list of OLDNAMES AND NEWNAMES