Download version 1.4.8: Windows or Linux

Persistency Library: Eco-DB (Eiffel CHAIN Orientated Database)

This library has 26 classes.

DEPENDS EIFFEL-LOOP

INTRODUCTION

Eco-DB is an acronym for Eiffel CHAIN Orientated Database, so called because it allows the extension by inheritance of a container conforming to CHAIN to assume many of the characteristics of a relational database table. Eco-DB leverages many of the facilities of the reflection cluster from the Eiffel-Loop base library, allowing class attributes to be managed as data table fields.

Some of the main features of this database system are as follows:

1. Perform the normal database CRUD operations.

2. Table joins, meaning a field in one CHAIN [G] table can be used to look up a row-item in another using a hash table index. A supporting feature is the ability to generate indexed primary keys in an automated fashion.

3. Option to store data securely using AES encryption.

4. Database fields are defined as class attributes and are managed reflectively, but there is also a manual method for writing and reading.

5. A simple centralised method to specify which fields should maintain a hash index for fast row look-ups by field value. A caveat is it only useful for unique id fields, like email addresses for examples.

6. Powerful Eiffel-orientated query facility that leverages the Eiffel conjunctive, disjunctive and negated keywords. Can also be used with PREDICATE agents.

7. Leverages a feature of the reflection cluster to link selected fields of an expanded type to a textual representation, as defined for example by a DATE or implementation of EL_ENUMERATION [N -> NUMERIC]. This is important when it comes to importing or exporting tables to/from a human readable form, or for setting the fields reflectively from a string based source.

8. Contract support to protect database integrity by the inclusion of a CRC check-sum for each CHAIN table. This guards against accidental changes of field type, field order, field name or textual representation.

9. Facility to export a meta-data record of the precise definition of the persistent data structure as a pseudo-Eiffel class. See for example: myching.software-meta-data.tar.gz

10. Fully automated import/export of CHAIN tables in either CSV or Pyxis format (an XML analogue with superior readability). This can be used as a very safe form of backup allowing data to be re-imported even if the field order has changed. The Pyxis format is very compact and readable allowing easy manual inspection of data. The gedit text editor has syntax highlighting for this format. See for example: payment.pyx recording Paypal transactions.

11. Unlike a relational database, the data rows of a CHAIN table do not have to be flat, since class attributes in a store-able item, can themselves be declared to be store-able. For example class EL_UUID (inheriting UUID) can be a storable attribute, which itself is reflectively stored as 5 integer attributes of various types.

12. Application version awareness allows data to be migrated from a data table created by an earlier software version.

13. Has been used in production for an online shop to store details of software subscription sales via Paypal. It is also used very reliably in the My Ching software application to manage a journal of I Ching readings and store localization information. In fact My Ching was one of the main drivers for development of this library.

PERSISTENCE

Of course this is the fundamental property of any database. Eco-DB offers 2 kinds of persistence:

1. CHAIN level persistence

This type of persistence involves storing the entire chain to a file in one operation. This is useful for data that is more or less static, like for example the localisation table EL_TRANSLATION_ITEMS_LIST.

See class ECD_CHAIN.

2. item level persistence

Item level, or "incremental persistence" is where the effects of any of the basic CHAIN operations (extend/replace/delete**) are recorded as they happen in a separate editions file. When the chain is loaded during object initialisation, a chain level store is loaded first, and then the stored editions are applied to bring the chain to it's final state.

See class ECD_RECOVERABLE_CHAIN for more details.

TABLE JOINS

Being able to join*** tables via a common field is the essence of a relational database. Eco-DB offers a number of features that support the joining of chains.

1. Field Indexing

For a large number of chain items, performing joins can be slow without the use of field indices. Eco-DB offers an easy way to maintain field indices with very little code via the implementing class ECD_ARRAYED_LIST [EL_STORABLE] which does all the work of maintaining the index. To index selected fields you just need to redefine the function new_index_by found in ECD_ARRAYED_LIST as in this example:

class
   SUBSCRIPTION_LIST

inherit
   ECD_ARRAYED_LIST [SUBSCRIPTION]
      rename
         item as subscription_item
      redefine
         new_index_by
      end

feature {NONE} -- Factory

   new_index_by: TUPLE [machine_id: like new_index_by_string_8; activation_code: like new_index_by_uuid]
      do
         create Result
         Result.machine_id := new_index_by_string_8 (agent machine_id_index)
         Result.activation_code := new_index_by_uuid (agent {SUBSCRIPTION}.activation_code)
      end

feature {NONE} -- Implementation

   machine_id_index (subsription: SUBSCRIPTION): STRING
      do
         if subsription.is_expired then
            create Result.make_empty
         else
            Result := subsription.machine_id
         end
      end

And here is an example showing how to use the created index:

class SUBSCRIPTION_LIST

feature -- Status query

   is_subscription_current (activation_code: EL_UUID): BOOLEAN
      do
         if attached index_by.activation_code as table then
            table.search (activation_code)
            if table.found then
               Result := table.found_item.is_current
            end
         end
      end

2. Primary Keys

Being able to assign a unique identifier to each item in a chain is essential to creating many kinds of data-joins. Eco-DB offers a convenient way to both generate primary keys and maintain an index for it. This is achieved with the auxiliary class ECD_PRIMARY_KEY_INDEXABLE [EL_KEY_IDENTIFIABLE_STORABLE] when used in conjunction with ECD_ARRAYED_LIST [EL_STORABLE]. The class parameter implies that the storable item must conform to EL_KEY_IDENTIFIABLE_STORABLE. Generation of primary key values is automatic when the list is extended, as is maintenance of the primary key hash-table index.

QUERY LANGUAGE

Of course the Eiffel language itself can be used to query any CHAIN list, but sometimes the meaning of the query is obscured in implementation details. What is needed is a slightly more abstract way of expressing queries that makes the meaning more apparent. This is provided by the class EL_QUERYABLE_CHAIN and it's helper EL_QUERY_CONDITION_FACTORY. The implementing class ECD_ARRAYED_LIST inherits EL_QUERYABLE_CHAIN.

Conditions can be combined using the logical operators: and, or and not as in this example from class COLLATE_SONGS_TASK found in project Eiffel-Loop/example/manage-mp3.

apply
   -- sort mp3 files into directories according to genre and artist set in Rhythmbox music library Database.
   -- Playlist locations will be updated to match new locations.
   local
      new_mp3_path: EL_FILE_PATH; song: RBOX_SONG; query_result: LIST [RBOX_SONG]
   do
      query_result := Database.existing_songs_query (not (song_is_cortina or song_has_normalized_mp3_path))
      if query_result.is_empty then
         lio.put_line ("All songs are normalized")
      else
         across query_result as query loop
            song := query.item
            ..
         end
      end
   end

The routine existing_songs_query passes a modified form of the query to songs list.

existing_songs_query (condition: EL_QUERY_CONDITION [RBOX_SONG]): like songs.query
   do
      Result := songs.query (not song_is_hidden and condition)
   end

songs: EL_QUERYABLE_ARRAYED_LIST [RBOX_SONG]

The query atoms song_is_cortina and song_has_normalized_mp3_path are defined in class SONG_QUERY_CONDITIONS which is defined as follows

class
   SONG_QUERY_CONDITIONS

inherit
   EL_QUERY_CONDITION_FACTORY [RBOX_SONG]
      rename
         any as any_song
      export
         {NONE} all
      end

META-DATA EXPORT

The routine export_meta_data in class ECD_REFLECTIVE_RECOVERABLE_CHAIN stores in a versioned directory the precise specification of the data layout, including the correct order, field types and names. The specification is formatted as pseudo Eiffel code so it can be easily viewed in an editor equipped with Eiffel syntax highlighting.

See for example: myching.software-meta-data.tar.gz (missing the version directory)

IMPORT/EXPORT

It is important to have a way to backup data that offer some degree of independence from the precise binary data structure for the purpose of replacing data with data from another software version, which may have fields stored in a different order, or types etc. Eco-DB supports two export formats:

CSV or Comma Separated Values if the data is flat, i.e. all the fields are basic types and are not compound types conforming to either EL_STORABLE or TUPLE.
Pyxis format which is very readable and compact. Shorter fields are grouped together as attributes on separate lines. See for example: payment.pyx which is a record of Paypal transactions.

The relevant class for importing or exporting is ECD_REFLECTIVE_RECOVERABLE_CHAIN

VERSION MANAGEMENT

A record of the software version is stored in each table. By defining procedure read_version from class EL_STORABLE

read_version (a_reader: EL_MEMORY_READER_WRITER; version: NATURAL)
   deferred
   end

it is possible to migrate data written by an earlier version of the software. If this is not required, this routine can be renamed to read_default_version in the inheritance section.

A DATABASE ABSTRACTION

There is work in progress to create an abstraction representing the concept of a database i.e. a collection of related tables. Currently this exists only in the form of an application library for the myching.software shop server. More work is needed to create useful abstractions that can be added to the Eco-DB library.

A preview of classes which will form the basis of reusable abstractions are as follows:

DATABASE contains fields conforming to DATA_TABLE and various routines that operate on all the tables at once. For example: make_open, close, delete, backup, export_as_pyxis etc
DATABASE_CONFIGURATION for storing/reading the database credentials etc.
DATA_TABLE is an abstraction allowing thread safe table operations in a concurrent environment.
DATABASE_MANAGER_SHELL is a menu driven shell for managing the database and performing operations like import, export, backup etc.

Foot Notes

** delete is a routine from ECD_CHAIN and not from CHAIN.

*** We are using the term join somewhat loosely and mean only that if you have two chains CHAIN [A] and CHAIN [B], you can produce a subchain of CHAIN [B] where each B item has a matching field value with an item from CHAIN [A].

Directory: library/persistency/database/eco-db

chain

ECD_ARRAYED_LIST

Provides the features below when used in conjunction with either of these 2 classes:

from the Eco-DB library.

An Eiffel-orientated data query language via the features of EL_CHAIN and EL_QUERYABLE_CHAIN. The class EL_QUERYABLE_ARRAYED_LIST has links to some examples in the mp3-manager project.

Automatically maintained field indexes accessible via the tuple attribute index_by

Class ECD_PRIMARY_KEY_INDEXABLE [EL_KEY_IDENTIFIABLE_STORABLE] allows automatic maintenance of a primary key index

Further Information

Click on class link to see instructions and client examples.

ECD_ARRAYED_TABLE_PAIR_LIST

ECD_ARRAYED_TUPLE_LIST that can be initialized from a HASH_TABLE

Further Information

Click on class link to see client examples.

ECD_ARRAYED_TUPLE_LIST

A storable arrayed list of types conforming to TUPLE

ECD_CHAIN

A CHAIN with items that can be stored and retrieved from memory buffer reader/writer conforming to EL_MEMORY_READER_WRITER

ECD_RECOVERABLE_CHAIN

A ECD_STORABLE_CHAIN that is recoverable despite write any errors. The former class can store and load the complete state of all chain items, while this class immediately stores any of the following chain editions: extend, replace, remove, delete.

When doing a file retrieval, the last complete state is loaded from file_path and then all the recent editions are loaded and applied from a separate file: editions_file_path. The routine safe_store stores the complete chain in a temporary file and then does a quick check on the integrity of the save by checking all the item headers. Only then is the stored file substituted for the previously stored file.

Further Information

Click on class link to see instructions.

ECD_REFLECTIVE_RECOVERABLE_CHAIN

Adds features to ECD_RECOVERABLE_CHAIN to do reflective CSV and Pyxis format exports By 'reflective' is meant that the exported CSV field names match the fields name of the class implementing EL_REFLECTIVELY_SETTABLE_STORABLE.

Further Information

Click on class link to see client examples.

ECD_STORABLE_ARRAYED_LIST

Storable arrayed list of objects conforming to EL_STORABLE

Further Information

Click on class link to see client examples.

ECD_STORABLE_CHAIN

Chain of storable items which can be saved to and read from a file. The chain has the following features:

Support for AES encryption
Ability to mark items for deletion without actually having to remove them immediately. This allows implementations like class ECD_REFLECTIVE_RECOVERABLE_CHAIN to support field indexing.
Ability to store software version information which is available to the item implementing EL_STORABLE.

Further Information

Click on class link to see descendants, notes and client examples.

Click on class link to see notes.

ECD_INDEX_TABLE_REPRESENTATION

A table index for assignment to a reflected field conforming to HASHABLE

Further Information

Click on class link to see notes and client examples.

ECD_KEY_INDEX

A field index conforming to ECD_INDEX_TABLE [EL_STORABLE, HASHABLE] that indexes a primary key of type NATURAL_32

ECD_PRIMARY_KEY_INDEXABLE

Assigns values to storable items conforming to EL_KEY_IDENTIFIABLE_STORABLE, and augments classes ECD_ARRAYED_LIST and ECD_REFLECTIVE_RECOVERABLE_CHAIN with a primary key index.

Further Information

Click on class link to see instructions.

ECD_REFLECTIVE_INDEX_TABLE

A field index conforming to ECD_INDEX_TABLE [EL_STORABLE, HASHABLE] that uses a EL_REFLECTED_FIELD instance field to obtain the value of the indexed field.

Further Information

Click on class link to see client examples.

ECD_MULTI_TYPE_READER_WRITER

Eco-DB file reader/writer for storing types conforming to types EL_STORABLE

ECD_READER_WRITER

Eco-DB file reader writer

Further Information

Click on class link to see descendants and client examples.

Persistency Library: Eco-DB (Eiffel CHAIN Orientated Database)

Directory: library/persistency/database/eco-db

chain

ECD_ARRAYED_LIST

ECD_ARRAYED_TABLE_PAIR_LIST

ECD_ARRAYED_TUPLE_LIST

ECD_CHAIN

ECD_RECOVERABLE_CHAIN

ECD_REFLECTIVE_RECOVERABLE_CHAIN

ECD_STORABLE_ARRAYED_LIST

ECD_STORABLE_CHAIN

editions

ECD_EDITIONS_FILE

ECD_ENCRYPTABLE_EDITIONS_FILE

index

ECD_AGENT_INDEX_TABLE

ECD_GROUP_TABLE

ECD_INDEX

ECD_INDEX_TABLE

ECD_INDEX_TABLE_REPRESENTATION

ECD_KEY_INDEX

ECD_PRIMARY_KEY_INDEXABLE

ECD_REFLECTIVE_INDEX_TABLE

reader-writer

ECD_ENCRYPTABLE_MULTI_TYPE_READER_WRITER

ECD_ENCRYPTABLE_READER_WRITER

ECD_MULTI_TYPE_READER_WRITER

ECD_READER_WRITER

ECD_TUPLE_READER_WRITER

support

ECD_CHAIN_HEADER

ECD_CONSTANTS

ECD_RECOVERABLE_CHAIN_HEADER