Modules

This is a detailed documentation of all basic modules

mist.io

Routes and wsgi app creation

mist.io.add_routes(config)

This function defines pyramid routes.

Takes a Configurator instance as argument and changes it’s configuration. Any return value is ignored. This was put in a separate function so that it can easily be imported and extended upon. Just use: config.include(add_routes)

mist.io.main(global_config, **settings)

This function returns a Pyramid WSGI application.

mist.io.views

mist.io.views

Here we define the HTTP API of the app. The view functions here are responsible for taking parameters from the web requests, passing them on to functions defined in methods and properly formatting the output. This is the only source file where we import things from pyramid. View functions should only check that all required params are provided. Any further checking should be performed inside the corresponding method functions.

mist.io.views.add_backend(request)

Adds a new backend.

mist.io.views.add_key(request)
mist.io.views.associate_key(request)
mist.io.views.check_auth(request)

Check on the mist.core service if authenticated

mist.io.views.check_monitoring(request)

Ask the mist.io service if monitoring is enabled for this machine.

mist.io.views.create_machine(request)

Creates a new virtual machine on the specified backend.

mist.io.views.delete_backend(request)

Deletes a backend.

Note

It assumes the user may re-add it later so it does not remove any key associations.

mist.io.views.delete_key(request)

Delete key.

When a keypair gets deleted, it takes its asociations with it so just need to remove from the server too.

If the default key gets deleted, it sets the next one as default, provided that at least another key exists. It returns the list of all keys after the deletion, excluding the private keys (check also list_keys).

mist.io.views.delete_machine_metadata(request)

Deletes metadata for a machine, given the machine id and the tag to be deleted.

mist.io.views.delete_rule(request)

Deletes a rule.

mist.io.views.disassociate_key(request)
mist.io.views.edit_key(request)
mist.io.views.exception_handler_mist(exc, request)

Here we catch exceptions and transform them to proper http responses

This is a special pyramid view that gets triggered whenever an exception is raised from any other view. It catches all exceptions exc where isinstance(exc, context) is True.

mist.io.views.generate_keypair(request)
mist.io.views.get_loadavg(request, action=None)

Get the loadavg png displayed in the machines list view.

mist.io.views.get_private_key(request)

Gets private key from keypair name.

It is used in single key view when the user clicks the display private key button.

mist.io.views.get_public_key(request)
mist.io.views.get_stats(request)
mist.io.views.home(request)

Home page view

mist.io.views.list_backends(request)

Gets the available backends.

Note

Currently, this is only used by the backend controller in js.

mist.io.views.list_images(request)

List images from each backend. Furthermore if a search_term is provided, we loop through each backend and search for that term in the ids and the names of the community images

mist.io.views.list_keys(request)

List keys.

List all key pairs that are configured on this server. Only the public keys are returned.

mist.io.views.list_locations(request)

List locations from each backend.

mist.io.views.list_machines(request)

Gets machines and their metadata from a backend.

mist.io.views.list_sizes(request)

List sizes (aka flavors) from each backend.

mist.io.views.list_specific_images(request)
mist.io.views.list_supported_providers(request)

@param request: A simple GET request @return: Return all of our SUPPORTED PROVIDERS

mist.io.views.machine_actions(request)
mist.io.views.probe(request)

Probes a machine using ping and ssh to collect metrics.

Note

Used for getting uptime and a list of deployed keys.

mist.io.views.rename_backend(request)

Renames a backend.

mist.io.views.set_default_key(request)
mist.io.views.set_machine_metadata(request)

Sets metadata for a machine, given the backend and machine id.

mist.io.views.shell_stream(request)

Execute command via SSH and stream output

Streams output using the hidden iframe technique.

mist.io.views.star_image(request)

Toggle image as starred.

mist.io.views.toggle_backend(request)
mist.io.views.update_monitoring(request)

Enable/disable monitoring for this machine using the hosted mist.io service.

mist.io.views.update_rule(request)

Creates or updates a rule.

mist.io.views.update_user_settings(request)

try free plan, by communicating to the mist.core service

mist.io.methods

mist.io.methods.connect_provider(backend)

Establishes backend connection using the credentials specified.

It has been tested with:

  • EC2, and the alternative providers like EC2_EU,
  • Rackspace, old style and the new Nova powered one,
  • Openstack Diablo through Trystack, should also try Essex,
  • Linode

Backend is expected to be a mist.io.model.Backend

mist.io.methods.delete_machine_metadata(user, backend_id, machine_id, tag)

Deletes metadata for a machine, given the machine id and the tag to be deleted.

Libcloud handles this differently for each provider. Linode and Rackspace, at least the old Rackspace providers, don’t support metadata updating. In EC2 you can delete just the tag you like. In Openstack you can only set a new list and not delete from the existing.

Mist.io client knows only the value of the tag and not it’s key so it has to loop through the machine list in order to find it.

Don’t forget to check string encoding before using them in ifs. u’f’ is ‘f’ returns false.

mist.io.methods.disable_monitoring(user, backend_id, machine_id)

Disable monitoring for a machine.

mist.io.methods.edit_key(user, new_key, old_key)

Edits a given key’s name from old_key —> new_key @param user: The User @param new_key: The new Key name (id) @param old_key: The old key name (id) @return: Nothing, only raises exceptions if needed

mist.io.methods.enable_monitoring(user, backend_id, machine_id, name='', dns_name='', public_ips=None)

Enable monitoring for a machine.

mist.io.methods.get_machine_actions(machine_from_api, conn)

Returns available machine actions based on backend type.

Rackspace, Linode and openstack support the same options, but EC2 also supports start/stop.

The available actions are based on the machine state. The state codes supported by mist.io are those of libcloud, check config.py.

mist.io.methods.list_images(user, backend_id, term=None)

List images from each backend.

Furthermore if a search_term is provided, we loop through each backend and search for that term in the ids and the names of the community images

mist.io.methods.list_locations(user, backend_id)

List locations from each backend.

Locations mean different things in each backend. e.g. EC2 uses it as a datacenter in a given availability zone, whereas Linode lists availability zones. However all responses share id, name and country eventhough in some cases might be empty, e.g. Openstack.

In EC2 all locations by a provider have the same name, so the availability zones are listed instead of name.

mist.io.methods.list_machines(user, backend_id)

List all machines in this backend via API call to the provider.

mist.io.methods.list_sizes(user, backend_id)

List sizes (aka flavors) from each backend.

mist.io.methods.probe(user, backend_id, machine_id, host, key_id='', ssh_user='')

Ping and SSH to machine and collect various metrics.

mist.io.methods.reboot_machine(user, backend_id, machine_id)

Reboots a machine on a certain backend.

mist.io.methods.rename_backend(user, backend_id, new_name)

Renames backend with given backend_id.

mist.io.methods.set_default_key(user, key_id)

Sets a new default key

@param user: The user @param key_id: The id of the key we want to set as default @return: Nothing. Raises only exceptions if needed.

mist.io.methods.set_machine_metadata(user, backend_id, machine_id, tag)

Sets metadata for a machine, given the backend and machine id.

Libcloud handles this differently for each provider. Linode and Rackspace, at least the old Rackspace providers, don’t support metadata adding.

machine_id comes as u’...’ but the rest are plain strings so use == when comparing in ifs. u’f’ is ‘f’ returns false and ‘in’ is too broad.

mist.io.methods.ssh_command(user, backend_id, machine_id, host, command, key_id=None, username=None, password=None)

We initialize a Shell instant (for mist.io.shell).

Autoconfigures shell and returns command’s output as string. Raises MachineUnauthorizedError if it doesn’t manage to connect.

mist.io.methods.start_machine(user, backend_id, machine_id)

Starts a machine on backends that support it.

Currently only EC2 supports that. Normally try won’t get an AttributeError exception because this action is not allowed for machines that don’t support it. Check helpers.get_machine_actions.

mist.io.methods.stop_machine(user, backend_id, machine_id)

Stops a machine on backends that support it.

Currently only EC2 supports that. Normally try won’t get an AttributeError exception because this action is not allowed for machines that don’t support it. Check helpers.get_machine_actions.

mist.io.dal

Mist Io DAL

DAL can stand for ‘Database Abstraction Layer’, ‘Data Access Layer’ or any of several more similar combinations.

The role of this DAL is to take control over all persistence related operations like reading from and writing to some storage. The rest of the application knows nothing about how the storage is implemented and is only presented with a simple, object oriented API.

Mist io uses yaml files as its storage backend. We store data using a main dict consisting of nested dicts, lists, ints, strs etc. This module provides an object oriented interface on those dicts.

A basic class here is OODict that defines a dict to object mapper. Classes that inherit OODict are initiated using a dict. Dict keys are on the fly transformed to object attributes, based on predefined fields. These fields (subclasses of Field) can be of a certain type and may have a default value.

From here you should import: StrField, IntField, FloatField, BoolField, ListField, DictField, getOODictField, getFieldsListField, getFieldsDictField, OODict, UserEngine

class mist.io.dal.BoolField(val=None)

Sets a boolean field. Default value: False

back_types = [<type 'bool'>]
front_types = [<type 'bool'>]
class mist.io.dal.DictField(val=None)

Sets a dictionary field. Default value: {}

back_types = [<type 'dict'>]
front_types = [<type 'dict'>]
class mist.io.dal.Field(val=None)

Field is an abstract base class meant to be subclassed.

All field elements must inherit it so that they can be distinguished from other instance attributes.

Front refers to the python side, back refers to the storage side. front_types and back_types must be lists of new style classes. They represent the valid types a field may have. Values will always be casted based on the first type. The rest just declare that they are also accepted inputs and should not raise errors.

What is a field? A field object doesn’t actually hold the value corresponding to that field. It just contains the information of how this value should look like, what type it should have in the storage backend and what in the python frontend plus a default value, should there be no corresponding value set. It is used by OODict’s and FieldSequence’s to appropriately handle the values fetched from storage.

back_types

Must be a list of new style classes. These classes represent the valid types a field may have in the storage side. Values will always be casted based on the first type. The rest just declare that they are also accepted inputs and should not raise errors.

cast2back(front_value=None)

Take a value from the frontend and cast to backend, replacing with default if None.

cast2front(back_value=None)

Take a value from the backend and cast to frontend, replacing with default if None.

front_types

Must be a list of new style classes. These classes represent the valid types a field may have in the python side. Values will always be casted based on the first type. The rest just declare that they are also accepted inputs and should not raise errors.

class mist.io.dal.FieldsDict(*args, **kwargs)

This defines a dict like container object that parses the real dict in the backend by treating the items as fields. It inherits basic container methods getitem, setitem, delitem and len from FieldsSequence and adds the iter method. Based on these basic container methods, the MutableMapping ABC provides the rest of the dict api, so you can use this container just as you would do with a dict(clear, setdefault etc).

class mist.io.dal.FieldsList(*args, **kwargs)

This defines a list like container object that parses the real list in the backend by treating the items as fields. It inherits basic container methods getitem, setitem, delitem and len from FieldsSequence and adds the insert method. Based on these basic container methods, the MutableSequence ABC provides the rest of the list api, so you can use this container just as you would do with a list (append, pop etc).

insert(index, value)
class mist.io.dal.FieldsSequence(*args, **kwargs)

Abstract Base Class providing part of the required methods for a custom container type to work.

Sets up a basic Sequence field, whose items are being parsed by some Field subclass. That means you can have a list or dict with str values, or int, or bool, or etc. This is an abstract base class. It interfaces upon an existing sequence without copying it and treats its items based on a field type.

Here we provide getitem, setitem, delitem and len methods for the containers. These methods are complemented by others in the subclasses.

get_raw()
class mist.io.dal.FileLock(lock_file)

This class implements a basic locking mechanism in the filesystem.

It uses the atomic operation of softlinks creation in UNIX-like systems.

When the lock is acquired, a softlink (with a .lock suffix) is created pointing to some arbitrary path. The link command will fail if such a link already exists. Acquire will sleep and retry to create the link, up to a specified timeout where it’ll give up and break the lock.

The lock is initialized given a lock filepath. When the lock has been acquired, any FileLock instance with the same filepath won’t be able to acquire the lock. Once however a certain FileLock instance has acquired a lock, its acquire() method can be called again. You have to release as many times as you lock. This doesn’t implement the reentrant property the way usual Rlocks do. For example, it doesn’t enforce that the releases are in exact reverse order of acquires. That however is enforced by the use of the with statement higher up in other classes using this one.

acquire()
break_after = 10
check()
isset()
release()
reset(lock_file)
sleep = 0.05
class mist.io.dal.FloatField(val=None)

Sets a floating point number field. Default value: 0.0

back_types = [<type 'float'>]
front_types = [<type 'float'>]
class mist.io.dal.HtmlSafeStrField(val=None)

Escapes < and > when reading field.

cast2front(back_value=None)
class mist.io.dal.IntField(val=None)

Sets an integer field. Default value: 0

back_types = [<type 'int'>, <type 'float'>]
front_types = [<type 'int'>, <type 'float'>]
class mist.io.dal.ListField(val=None)

Sets a list field. Default value: []

back_types = [<type 'list'>, <type 'tuple'>]
front_types = [<type 'list'>, <type 'tuple'>]
class mist.io.dal.OODict(_dict=None)

OODict is an abstract base class that defines a dict to object mapper. It is instantiated given a dict, as obtained by mongo for example. By defining (in OODict subclasses) properties that are instances of subclasses of Field, we define the dict schema, the names of the fields, the types of their values as well as default values.

This class interfaces an existing dict. So when you give it a dict, as obtained by mongo or yaml or memcache or whatever, it sees which properties that are instances of subclasses of Field are defined. When you try to access that property, the call is intercepted and the values are read from and to the dict directly. If they are not of the correct type, they’ll be casted. If the values are missing on the dict, they will be set to the default. This aproach doesn’t copy data and store it seperately. It just interfaces.

as_dict()
get_raw()
keys()
class mist.io.dal.OODictYaml(yaml_rel_path)

This takes care of all storage related operations.

refresh()
save()

Save data to yaml file.

class mist.io.dal.OODictYamlLock(yaml_rel_path)
lock_n_load(*args, **kwds)

File lock context manager.

It must be used with a ‘with’ statement as follows: with user.lock_n_load(): # edit user user.save() Lock is automatically released after exiting the ‘with’ block. Attempting to save without first acquiring the lock will raise an exception.

save()

Save user data to storage. Raises exception if not in a “with user.lock_n_load():” code block.

class mist.io.dal.ObjectField(val=None)

This is an abstract base class that inherits from Field. It assumes that front type is a subclass of BaseObject and changes the way values are casted to backend accordingly.

cast2back(front_value=None)
class mist.io.dal.StrField(val=None)

Sets a string field. Default value: ‘’

back_types = [<type 'str'>, <type 'unicode'>]
front_types = [<type 'str'>, <type 'unicode'>]
class mist.io.dal.User
mist.io.dal.make_field(obj_type)

Create a Field subclass out of some obj_type.

This works for any obj_type that subclasses OODict or FieldsSequence.

mist.io.model

Mist Io Model

Here we define the schema of our data structure in an object oriented way.

Simple, low level, helper functions can also be added to the following classes. (eg user.get_num_mon_machines(), user.keys.unused()). It is recommended that only pure functions (no side-effects) are used as class methods.

How this works: The basic class is the OODict. This defines a dict to object mapper. When we need a new data structure, we define a new subclass of OODict. Class properties that are instances of Field subclasses are considered to be OODict fields. These are the keys in the underlying dict that will be used. There is a large variety of standard type fields. One can create an OODict that has a field which is also parsed by some OODict. To do so, you define a field on the outer OODict that is created by make_field. Finally, list or dict like collections can be created by subclassing FieldsList and FieldsDict. The items of these collections will be parsed according to the field type defined in the class. This collection can be used as a field in some OODict by use of make_field. If it sounds too complicated, just look the code below, it should be pretty self-explanatory.

class mist.io.model.Backend(_dict=None)

A cloud vm provider backend

apikey = <class 'mist.io.dal.HtmlSafeStrField'>('')
apisecret = <class 'mist.io.dal.StrField'>('')
apiurl = <class 'mist.io.dal.StrField'>('')
auth_version = <class 'mist.io.dal.HtmlSafeStrField'>('')
enabled = <class 'mist.io.dal.BoolField'>(False)
get_id()
machine_count = <class 'mist.io.dal.IntField'>(0)
machines = <class 'mist.io.dal.FieldsDictField'>(<class 'mist.io.model.Machines'>: {})
poll_interval = <class 'mist.io.dal.IntField'>(10000)
provider = <class 'mist.io.dal.HtmlSafeStrField'>('')
region = <class 'mist.io.dal.HtmlSafeStrField'>('')
starred = <class 'mist.io.dal.ListField'>([])
tenant_name = <class 'mist.io.dal.HtmlSafeStrField'>('')
title = <class 'mist.io.dal.HtmlSafeStrField'>('')
class mist.io.model.Backends(*args, **kwargs)
class mist.io.model.Keypair(_dict=None)

An ssh keypair.

construct_public_from_private()

Constructs pub key from self.private and assignes to self.public. Only works for RSA.

default = <class 'mist.io.dal.BoolField'>(False)
generate()

Generates a new RSA keypair and assignes to self.

isvalid()

Checks if self is a valid RSA keypair.

machines = <class 'mist.io.dal.ListField'>([])
private = <class 'mist.io.dal.StrField'>('')
public = <class 'mist.io.dal.StrField'>('')
class mist.io.model.Keypairs(*args, **kwargs)
class mist.io.model.Machine(_dict=None)

A saved machine in the machines list of some backend.

For the time being, only bare metal machines are saved, for API backends we get the machine list from the provider.

dns_name = <class 'mist.io.dal.HtmlSafeStrField'>('')
name = <class 'mist.io.dal.HtmlSafeStrField'>('')
public_ips = <class 'mist.io.dal.ListField'>([])
ssh_port = <class 'mist.io.dal.IntField'>(22)
uuid = <class 'mist.io.dal.StrField'>('')
class mist.io.model.Machines(*args, **kwargs)

Collection of machines of a certain backend.

For the time being, only bare metal machines are saved, for API backends we get the machine list from the provider.

class mist.io.model.User

The basic model class is User. It contains all the methods necessary to find and save users in memcache and in mongo. It transforms the user dict into an object with consistent attributes instead of inconsistent dict with missing keys.

backends = <class 'mist.io.dal.FieldsDictField'>(<class 'mist.io.model.Backends'>: {})
email = <class 'mist.io.dal.StrField'>('')
keypairs = <class 'mist.io.dal.FieldsDictField'>(<class 'mist.io.model.Keypairs'>: {})
mist_api_token = <class 'mist.io.dal.StrField'>('')

mist.io.shell

mist.io.shell

This module contains everything that is need to communicate with machines via SSH.

class mist.io.shell.Shell(host, username=None, key=None, password=None, port=22)

sHell

This class takes care of all SSH related issues. It initiates a connection to a given host and can send commands whose output can be treated in different ways. It can search a user’s data and autoconfigure itself for a given machine by finding the right private key and username. Under the hood it uses paramiko.

Use it like: shell = Shell(‘localhost’, username=’root’, password=‘123’) print shell.command(‘uptime’)

Or: shell = Shell(‘localhost’) shell.autoconfigure(user, backend_id, machine_id) for line in shell.command_stream(‘ps -fe’): print line

autoconfigure(user, backend_id, machine_id, key_id=None, username=None, password=None)

Autoconfigure SSH client.

This will do its best effort to find a suitable keypair and username and will try to connect. If it fails it raises MachineUnauthorizedError, otherwise it initializes self and returns a (key_id, ssh_user) tupple. If connection succeeds, it updates the association information in the key with the current timestamp and the username used to connect.

check_sudo()

Checks if sudo is installed.

In case it is self.sudo = True, else self.sudo = False

command(cmd, pty=True)

Run command and return output.

If pty is True, then it returns a string object that contains the combined streams of stdout and stderr, like they would appear in a pty.

If pty is False, then it returns a two string tupple, consisting of stdout and stderr.

command_stream(cmd)

Run command and stream output line by line.

This function is a generator that returns the commands output line by line. Use like: for line in command_stream(cmd): print line.

connect(username, key=None, password=None, port=22)

Initialize an SSH connection.

Tries to connect and configure self. If only password is provided, it will be used for authentication. If key is provided, it is treated as and OpenSSH private RSA key and used for authentication. If both key and password are provided, password is used as a passphrase to unlock the private key.

Raises MachineUnauthorizedError if it fails to connect.

disconnect()

Close the SSH connection.