.. _comet_command:

Cloudmesh Comet CLI
======================================================================

We developed a command line based client tool for users to access Comet
VC easily and securely. The CLI is part of a larger effort called
Cloudmesh Client toolkit, which provides command line and shell
interface of accessing heterogeneous clouds, clusters, and workstations
right from the users' computers. By utilizing this client tool, Comet VC
users can access all capabilities to manage their virtual cluster, while
bypassing the security concerns and the complexity of programming with
the Comet nucleus API.

Teminology
-----------

We use in this section the following terminology:

computeset:
    A group of compute nodes started together and being in some state
    (submitted, started, finished, failed). Each compute node can only
    belong to 1 computesets in submitted or active state.

frontend:
    A node with limited computational resources used to manage a
    virtual cluster. Frontends run 24/7, have a public interface and a
    private interface. The public interface provides outside access to
    the virtual cluster while the private interface is used to
    manage/install the compute nodes.

image:
    A file containing the contents and structure (ISO9660) of a disk
    volume which can be attached as a cdrom to a node.

console:
    An interactive representation of the screen of a virtual cluster
    node (text or graphical) provided to assist with node installation
    and management.

virtual cluster:
    A virtual cluster is a loosely or tightly connected network of virtual
    computers managed together by a virtual cluster administrator.

node:
    The term node is used to refer to individual computers in a
    virtual cluster.

image attach:
    Attach is an action applied to a node / image pair whereby the
    contents of the image are made available to a node on the next
    power on.

image detach:
    Detach is an action applied to a node / image pair whereby the
    contents of the image are made unavailable to the node on the next
    power on.

Installation
--------------

Cloudmesh Comet CLI is developed using Python. The installation is
starightforward and easy:

.. prompt:: bash

	    pip install cloudmesh_client

Both Python 2.7 and 3.5 is supported. You are HIGHLY recommended to use
a virtualenv based environment to install and use this toolkit.

Configuration
--------------

The configuration of the cloudmesh client is done semi automatically
for you.  All you have to do after the installation is to call
cloudmesh client once.

This is done best with the command:

.. prompt:: bash
	    
	    cm help

This will automatically generate a template configuration file at::

    ~/.cloudmesh/cloudmesh.yaml

Next you can initiate the configuration file:

.. prompt:: bash
	    
	    cm comet init

Follow the prompt to configure and select endpoint, and initialize the
auth settings by provide your username and password. Once you are
successfully authenticated it will retrieve the api key and secret and
configure the cloudmesh.yaml with proper api auth information. Then you
will be able to use the commands to manage your comet virtual cluster.

**For the use of the Comet VC production system, you need to use the OTP 
generated by your assigned YubiKey. You need to have the key plugged in, 
and upon the request of your password, type your PIN first, and then tap 
your key.**

**Console access of nodes requires 2-factor Auth as well, so please have
your YubiKey ready for console access**

Commands
--------------

Comet CLI supports accessing multiple nucleus service endpoint, e.g.,
DEV system or Production system. You will know which system to use after
your VC is setup. To check which system you are currently accessing,
run this:

.. prompt:: bash

    cm comet active

Or change to another system to interact with, e.g.:

.. prompt:: bash

    cm comet active production

Next we list a number of important commands from the CLI that will
help you managing your comet virtual clusters.

Getting information of your cluster(s); nodes; computesets; etc.
~~~~

List all clusters owned by the authenticated identity (summarized
format):

.. prompt:: bash
  
  cm comet ll

List all clusters owned by the authenticated identity (detailed
list):

.. prompt:: bash
  
  cm comet cluster
    
List a cluster by name (we use here vc2 as example):

.. prompt:: bash
  
  cm comet cluster vc2

Cluster listing view also supports other output format, e.g. json,
csv, etc.:

.. prompt:: bash

  cm comet cluster vc2 --format=csv

This can be useful for scripting.
    
List all defined computesets:

.. prompt:: bash

  cm comet computeset
    
List one computeset:

.. prompt:: bash
  
   cm comet computeset 63

To obtain the detailed information of a node:

.. prompt:: bash

    cm comet node info vc2

Gives your the information of the vc2 frontnode. This includes essential
network configuration information if you want to rebuild the frontend
node.

.. prompt:: bash

    cm comet node info vc2 vm-vc2-0

Gives the detailed information of the specified compute node.

By default it returned information in a table format. It also accepts
the --format=FROMAT parameter so the output can be easily parsed while
being used in scripting (FORMAT could be one of csv, json, yaml).

Power management of frontend node:
~~~~
Power on the front end node of the specified cluster:

.. prompt:: bash

    cm comet power on vc2

To power if off:

.. prompt:: bash

    cm comet power off vc2

You can also reboot/reset/shutdown the fronend using the same 
syntax, e.g., to reboot:

.. prompt:: bash

    cm comet power reboot vc2

Please note running frontend node itself would not charge you any
allocations.

Resource allocation and tear down:
~~~~

The requested cluster/VMs has to be started by requesting physical
allocations. This will charge your allocation based on number of 
nodes and the walltime you requested.

**Please note that each VM you requested corresponds to 24 cores (You 
have the exclusive access to the whole physical node), thus one VM node 
hour is equivalent to 24 SUs. So please TERMINATE the resources (see 
below) if you don't actively use them. Keeping them idle (even SHUT OFF)
will still cost your allocation.**

**Currently for the DEV resources you are not charged for any usage. 
Please use the DEV resource to develop/test your application/workflow, 
and move to the production for the actual usage. Please terminate the 
requested compute set(s) if you are not actively using them to free the 
resources so other users can use them as well.**

Start a set of compute nodes in cluster vc2:

.. prompt:: bash
  
    cm comet start vc2 vm-vc2-[0-3]
    
This will request the nodes for a default period of time and power on
them (currently default as 2 days).

To request for a customized time period, use --walltime parameter. 
E.g., 100m (100 minutes), 6h (6 hours), 2d (2 days) , 1w (1 week):

.. prompt:: bash

    cm comet start vc2 vm-vc2-[0-3] --walltime=6h

The above will put the request under the one allocation associated
with the cluster.  If your cluster have more than one allocations,
use --allocation parameter:

.. prompt:: bash

    cm comet start vc2 vm-vc2-[0-3] --allocation=YOUR_ALLOCATION

If you have more allocations, but does not specify via CLI, you will
see a list of allocations to choose from to use.

You can also request a group of N nodes, if there is enough resource:

.. prompt:: bash

    cm comet start vc2 --count=4

The comet system will find 4 available nodes from the specified
cluster and start them as one computeset.

You can also start a single-node computeset:

.. prompt:: bash

    cm comet start vc2 vm-vc2-[7]

or simply:

.. prompt:: bash

    cm comet start vc2 vm-vc2-7

To power down all running nodes from an active computeset and also
free the allocated resources:

.. prompt:: bash

    cm comet terminate 123

This will gracefully shutdown the nodes in the group identified by
computeset 123, and also free the unused allocations. A computeset
will be teared down automatically when it reaches its requested
walltime.

**Please note you have to terminate the requested resources 
as such so it stops charging your account for the usage.**

Power management of compute nodes:
~~~~

You can power off and back on individual nodes from a cluster, 
without affecting other running nodes in the computesets. E.g.:

.. prompt:: bash

    cm comet power off vc2 vm-vc2-[0-7]

and then:

.. prompt:: bash

    cm comet power on vc2 vm-vc2-[0-7]

or shutdown gracefully a group of nodes:

.. prompt:: bash

    cm comet power shutdown vc2 vm-vc2-[0-3]

Please note even if you powered off all nodes from an active
computeset, the computeset itself along with the associated physical
resource is still active and can only be accessed exclusively by you
till the requested walltime is reached. During this time you can 
freely power the nodes back on and off.

**Please be reminded that your account will still be charged in this
case until you terminate the computeset as described earlier.**

**Please note that the node disks are persisted silently on the background
periodically, or when you shut them off gracefully ('cm comet terminate' 
or 'cm comet power shutdown'). So the next time you start the node, or 
powered if on, you can start with where you have left.**

Getting Console access
~~~~
Get console of the frontend:

.. prompt:: bash
  
    cm comet console vc2

Get console of a running compute node:

.. prompt:: bash
  
    cm comet console vc2 vm-vc2-0

**Please note the console access requires 2-factor auth.**
The commands will ask your comet nucleus username (if not set previously) 
and password, and you will need your Yubikey for the access.

Once you have passed this extra step of auth, it will print out a shortlived
URL for accessing the console of the node requested. It will also pop open a
browser window if a supported OS/browser is detected. In OSX it will use the
system default browser to display the console; and in Linux environment it
will try to use firefox.

You can also get only the URL directly by calling:

.. prompt:: bash

    cm comet console --link vc2

System image management
~~~~
Get the list of images that are available to you:

.. prompt:: bash

    cm comet iso list

Upload an image to the public shared folder:

.. prompt:: bash

    cm comet iso upload /path/to/your/image.iso

Or with a specified new image name:

.. prompt:: bash

    cm comet iso upload /path/to/your/image.iso --imagename=newimagename.iso

Attach an image to the frontend, by specifying the exact image name, or the index
of the image in the iso list:

.. prompt:: bash

    cm comet iso attach newimagename.iso vc2
    cm comet iso attach 6 vc2

Or to a compute node:

.. prompt:: bash

    cm comet iso attach newimagename.iso vc2 vm-vc2-0

To detach the attached iso image from frontend node:

.. prompt:: bash

    cm comet iso detach vc2

Or from a compute node:

.. prompt:: bash

    cm comet iso detach vc2 vm-vc2-0

Image attaching/detaching also works on compute nodes in bulk:

.. prompt:: bash

    cm comet iso attach newimagename.iso vc2 vm-vc2-[0-4]

.. prompt:: bash

    cm comet iso detach vc2 vm-vc2-[0-4]

Please note image attaching/detaching will only take effect **AFTER you
hard reboot the node (power off and then power on)**.

Other commands:
~~~~

You can rename a compute node, or a list of nodes in batch:

.. prompt:: bash

    cm comet node rename vc2 vm-vc2-0 mynode0

.. prompt:: bash

    cm comet node rename vc2 vm-vc2-[0-7] newname-[0-7]

Please use hostlist format to specify the list of OLDNAMES AND
NEWNAMES