The dip.io module serves two purposes. Firstly it provides the support to applications to enable the reading and writing of models. Secondly it provides support for implementing new formats and forms of storage so that they can be used by existing applications without needing to make any changes to those applications. The basis of all this support is an i/o manager which is an object that implements or can be adapted to the IIoManager interface.
dip considers storage to be either streaming storage or structured storage. Streaming storage stores data as a byte stream. A filesystem is the most common example of streaming storage. Structured storage stores data in a storage specific structure. An SQL database would be an example of structured storage.
A model is stored according to a data format. When a model is written it is encoded according to the format. When it is read it is decoded from the format. A model will normally have a native format which is used when reading and writing the model. It may be possible to export a model to other formats and to import a model from other formats. Each format has a unique string identifier.
A codec is an object that implements an encoder and/or a decoder for a particular format. Streaming storage has no implicit format and so it is used in conjunction with an explicit codec. The codec effectively imposes the structure of the data on top of the byte stream. Structured storage has, by definition, an implicit format and, therefore, an implicit codec.
A codec specifies encoder and decoder interfaces that a model must implement, or be able to be adapted to, in order to be encoded and decoded.
A model is stored at a storage location that is unique within a particular piece of storage. For example, the location of a model stored in a filesystem is the absolute path name of the file containing the encoded model. A storage location may be implicit. This means that where a model is stored is determined by the value of the model and is not specified by the user.
In Getting Started with dip.shell we briefly described the steps to be taken to ensure that a model could be written to and read from storage. Here we will reiterate those steps.
Note that some codecs may only support decoding or encoding but not both. Such codecs would not be used as a model’s native format but they would still be useful when importing from or exporting to non-native formats.
dip provides two codecs that between them cover many common cases.
The need to implement a new type of storage arises less often than the need to implement a new codec. When the need does arise it is typically as a result of some new technology or service becoming available that can be used by many applications rather than something that is application specific. For example, an organisation may subscribe to a cloud based file service. A new type of storage would then be implemented to provide access to it. All existing applications could then use it without making changes to those applications.
The other situation that would require a new type of storage to be implemented is when a database is being used as structured storage.
In this section we describe the high level steps taken to implement a new type of storage, including the interfaces and classes that need to be written.
IStorage also defines the ui attribute which is an implementation of the IStorageUi interface. This interface defines methods that create the necessary user interfaces that the user will use to select a storage location. For example, the filesystem storage type included with dip provides access (assuming the default PyQt4 toolkit) to QFileDialog using this mechanism. A storage type that handled a database may implement a database browser.
Sometimes you may have a situation where a model can be read from or written to a particular storage type, but you want to place restrictions on that access and the options presented to the user. For example, a certain type of user may only be able to read from the storage, or access to the storage may be limited to certain times of the day, or you simply wish to prevent a certain type of model from ever being written a certain type of storage.
The i/o manager will consult a list of storage policies to determine if a model using a particular format should actually be allowed to be read from or written to a particular storage instance. A policy is a callable that is passed the format and the storage instance and should return True if the access is permitted. If any policy returns False then the access is denied.