dip.io module serves two purposes. Firstly it provides the support
to applications to enable the reading and writing of models. Secondly it
provides support for implementing new formats and forms of
storage so that they can be used by existing applications without
needing to make any changes to those applications. The basis of all this
support is an i/o manager which is an object that implements or can be
adapted to the
dip considers storage to be either streaming storage or structured storage. Streaming storage stores data as a byte stream. A filesystem is the most common example of streaming storage. Structured storage stores data in a storage specific structure. An SQL database would be an example of structured storage.
A model is stored according to a data format. When a model is written it is encoded according to the format. When it is read it is decoded from the format. A model will normally have a native format which is used when reading and writing the model. It may be possible to export a model to other formats and to import a model from other formats. Each format has a unique string identifier.
A codec is an object that implements an encoder and/or a decoder for a particular format. Streaming storage has no implicit format and so it is used in conjunction with an explicit codec. The codec effectively imposes the structure of the data on top of the byte stream. Structured storage has, by definition, an implicit format and, therefore, an implicit codec.
A codec specifies encoder and decoder interfaces that a model must implement, or be able to be adapted to, in order to be encoded and decoded.
A model is stored at a storage location that is unique within a particular piece of storage. For example, the location of a model stored in a filesystem is the absolute path name of the file containing the encoded model. A storage location may be implicit. This means that where a model is stored is determined by the value of the model and is not specified by the user.
Reading and Writing a Model¶
In Getting Started with dip.shell we briefly described the steps to be taken to ensure that a model could be written to and read from storage. Here we will reiterate those steps.
- Unless one already exists, create a codec, i.e. an object that implements the
ICodecinterface, for the model’s native format. A codec will specify decoder and encoder interfaces that a model must implement or be able to be adapted to.
- Ensure that the model implements the decoder and encoder interfaces. This is normally done by creating appropriate adapters. Whether or not a single adapter is used for both interfaces is a matter of personal programming style.
- Create an instance of the codec and add it to the i/o manager’s list of codecs.
Note that some codecs may only support decoding or encoding but not both. Such codecs would not be used as a model’s native format but they would still be useful when importing from or exporting to non-native formats.
dip provides two codecs that between them cover many common cases.
UnicodeCodeccodec, with its
IUnicodeDecoderdecoder interface and
IUnicodeEncoderencoder interface, stores a model as a Unicode byte stream. By default it uses the UTF-8 encoding.
XmlCodeccodec, with its
IXmlDecoderdecoder interface and
IXmlEncoderencoder interface, stores a
Modelinstance as XML.
Implementing a New Type of Storage¶
The need to implement a new type of storage arises less often than the need to implement a new codec. When the need does arise it is typically as a result of some new technology or service becoming available that can be used by many applications rather than something that is application specific. For example, an organisation may subscribe to a cloud based file service. A new type of storage would then be implemented to provide access to it. All existing applications could then use it without making changes to those applications.
The other situation that would require a new type of storage to be implemented is when a database is being used as structured storage.
In this section we describe the high level steps taken to implement a new type of storage, including the interfaces and classes that need to be written.
IStorage also defines the
attribute which is an implementation of the
interface. This interface defines methods that create the necessary user
interfaces that the user will use to select a storage location. For example,
the filesystem storage type included with dip provides access (assuming the
default PyQt5 toolkit) to
QFileDialog using this
mechanism. A storage type that handled a database may implement a database
Defining a Storage Policy¶
Sometimes you may have a situation where a model can be read from or written to a particular storage type, but you want to place restrictions on that access and the options presented to the user. For example, a certain type of user may only be able to read from the storage, or access to the storage may be limited to certain times of the day, or you simply wish to prevent a certain type of model from ever being written a certain type of storage.
The i/o manager will consult a list of storage policies
to determine if a model using a particular format should actually be allowed to
be read from or written to a particular storage instance. A policy is a
callable that is passed the format and the storage instance and should return
True if the access is permitted. If any policy returns
False then the
access is denied.