 | |  |  |  |  |
| | | | | |
MemCom is a data management system for handling engineering and scientific data. Symbolic data manipulations as well as transparent input/output operations replace data accessing by records, keys etc. MemCom offers a collection of tools that provide for efficiency of code development, data structuring, and execution speed:
Modular Programming MemCom is particularly well suited for modular applications on mono-processor systems or in distributed processing environments. The modular design of MemCom allows great flexibility by providing techniques adapted to different hardware and software environments.
Efficiency The overhead, which is introduced by a data classification system such as MemCom, is small. In some cases the programs are sped up by MemCom, due to the sophisticated input/output-methods involved. A great effort has been made to minimise overhead due to data descriptors and search procedures.
Data Portability The contents of MemCom databases can easily be transported from one computer system to another in a way that is completely transparent to the user.
Flexibility MemCom within an application may be adapted to the hardware environment in order to optimise its performance. Input/output operations can be performed in direct mode, in buffered mode through MemCom's own paging system, or by directly mapping data in the virtual memory management system of the hardware.
Monitoring and debugging The interactive Python module pymemcom allows for checking, modifying and manipulating data, as well as for process monitoring. The module supports all of the MemCom functions.
All modules are compatible through the databases. This means that existing software can be adapted to MemCom and made compatible with new developments. In addition, MemCom can be attached to other data management systems by exchanging the input- output module. MemCom evolved from the designer's own experience with data handling in technical environments like numerical analysis and graphics. Thus, data access methods such as sequential access by record number or by key or random access by record or by key have been replaced by consistent, portable data manipulation facilities. Operations on the databases are either executed directly between the user area and the databases, through the optional paging system, or through the Dynamic Memory Manager. User programs or host languages, which refer to MemCom, will interact with different MemCom modules. The figure below illustrates how the basic MemCom modules are linked to the user program and to the databases.
MemCom contains several data access and data manipulation modules, each of them providing mechanisms adapted to specific problem types:
The Database Manager (DB) performs the basic MemCom data access operations.
The Relational Table system (TB) provides for basic operations on relational tables and descriptors. Data are stored in position-independent format, i.e elements of the tables may be stored, retrieved, added or deleted in any order according to the keywords.
The Dynamic Memory Manager (DMM) provides for run-time dynamic memory management and, combined to the Database Manager, for input/output management (Fortran 77 only).
The Array Table system (AT) allows for storing and retrieving of scalar and array entries in tables addressed by line numbers and column numbers (or names).
The Sparse Table system (ST) allows for storing and retrieving of scalar and array entries in sparse tables addressed by line numbers and column numbers (or names).
The Neutral Data Format Manager (NM) encodes and decodes MemCom databases.
All MemCom modules are compatible through the MemCom databases, and different MemCom modules may be accessed within the same applications. While the database manager (DB) always operates directly between the user buffers and the databases, the Relational Table Manager (RT), the Dynamic Memory Manager (DMM), the Array Table system (AT), and the Sparse Table system (ST) operate with internal buffers. Relational tables and relational datasets are accessed through the table buffers, the Dynamic Memory is accessed with the Dynamic Memory Manager pointer mechanism, Array Tables are accessed via the Array Table paging system, and Sparse Tables are accessed via the Sparse Table paging system.
The exchange of data between the user buffers, the Relational Table Manager, or the Dynamic Memory Manager buffers can be direct or buffered (paged): MemCom disk input/output operations may be performed in buffered or in unbuffered mode without modifying the contents of the databases. Buffered mode puts in input/output paging mode. The actual input/output operations may be performed in direct mode or in buffered mode (See also the figure below):
Direct: Data are exchanged between the user buffers or the Relational Table Manager and the Dynamic Memory Manager system buffers on the one hand, and the databases on the other hand, without any intermediate buffers.
Paged: Data are exchanged between the user buffers or the Relational Table Manager and the Dynamic Memory Manager system buffers on the one hand, and the databases on the other hand, with a paging system in between.
The choice of either method depends on the application, few large data accesses being efficient in direct mode, and many small randomly distributed data accesses being more efficient in paged mode. The array table subsystem (AT) and the sparse table subsystem (ST) always use paged disk input/output.
MemCom classifies data according to the following hierarchy:
Databases
Datasets
Subsets
Dataset elements
Datasets and subsets are collections of data managed by MemCom. Datasets are identified by their name field or by their address and the corresponding database handle. The database manager and the Dynamic Memory Manager keep tables pointing to the datasets on the respective datasets. The physical input/output may be performed in buffered and unbuffered mode.
