Developing modules for moodss (Modular Object Oriented Dynamic SpreadSheet)

Contents:

Note: all examples are drawn from the random Tcl and Random Perl sample modules.

1. source

A module is a package, it must have a name and a version.
Note: version number management is important as the module major version number is taken into account when storing data cells history in a database (can be done from both the moodss GUI application and the moomps daemon). Modules developers must insure that no major changes occur between minor releases of the module. In other words, if there is a change in the module data structure, such as adding a new column, changing the order of columns, changing column data, such as label or type (message and anchor can be altered without problems), then a new major version number must be used, to preserve database integrity (a new major version number results in the new data to be stored with new, thus different identifiers).

Module names can be any combination of the following characters (minus the characters not allowed by the specific module language, of course):

1.1. namespace

All module procedures and data are kept in a specific namespace bearing the module name.

The update function is not needed when the module is asynchronous.

Note: at this time, it is required for Perl and Python modules, as they cannot be asynchronous.

1.2. configuration

The module configuration defines the data table column headers, help information, ... This data never changes during the lifetime of the application.

The updates member is a counter used to keep track of the number of times that the module data was updated, and is also used by the core to detect when module data display should be updated (see variable data for more information).

The label members ($data(n,label) in Tcl, $data{columns}[n]{label} in Perl, form['columns'][n]['label'] in Python, with n the column number), define the text to be displayed as column titles. There must be as many label members as they are columns. The titles must contain no ? character.

The type members ($data(n,type) in Tcl, $data{columns}[n]{type} in Perl, form['columns'][n]['type'] in Python) define the type of the corresponding column data. Valid types are simply those that the Tcl lsort command can handle: ascii, dictionary, integer and real, plus the clock type, which accepts any format that the Tcl clock format command can handle (see the trace module for an example). There must be as many type members as they are columns.

The message members ($data(n,message) in Tcl, $data{columns}[n]{message} in Perl, form['columns'][n]['message'] in Python) define the text of the help message to be displayed in a floating yellow window (widget tip, balloon, also see User Interface) as the user moves the mouse pointer over column titles. It can be composed of only a few words or multiple formatted lines. There must be as many message members as they are columns.

The anchor member ($data(n,anchor) in Tcl, $data{columns}[n]{anchor} in Perl, form['columns'][n]['anchor'] in Python) is optional. Column data is either centered by default, tucked to the left or right side of the column. Valid values are center, left or right.

Note that column numbers start at 0. There must be no hole in the column numbers sequence.

The pollTimes member is a list of valid poll times (in seconds) for the module. The list is not ordered, as its first element represents the default poll time value to be used when the moodss application starts. This value may be overridden by a command line argument. The smallest value in the list is used by the core as the lowest possible poll time and checked against when the user enters a new value through the poll time dialog box. The list must not be empty.
Note that the list is also used by moodss as a set of possible choices in the dialog box used to set the new poll time. The user may still directly input any value as long as it is greater than or equal to the minimum value.

If the module is asynchronous (data can be updated at any time and not in response to update procedure invocations (no polling required)), the pollTimes member must be a single negative integer value representing the preferred time interval for viewers that require one (only graphs at this point). For example, if you wish graph viewers to have a display interval of 10 seconds, use:

In this case, the graph viewers time range (knowing that they feature 100 time points) would be 1000 seconds. I guess that the value that you should specify as the pollTimes member should be the expected average update interval for your asynchronous data. Note that the graphical viewers x axis always display properly labeled absolute time ticks in any case.

When several asynchronous modules are loaded with no synchronous modules, the interval used for all relevant viewers is the average (in absolute value) of all module intervals. For example, if you load 2 asynchronous modules, one with a pollTimes member of -10 and the other of -20, then a 15 seconds interval value is retained. Note that the interval can be forced through the --poll-time command line argument.

If at least one synchronous module is loaded concurrently with any number of asynchronous modules, the actual application poll time (the one that can be set with the then available poll time dialog box) is used.

The indices member is an optional list that specifies the table columns that should be displayed. If not specified, all the table columns are visible.

The sort entry is optional. It defines the index of the column which should be initially used as a reference for sorting the data table rows, and in which order (increasing or decreasing) the rows should be sorted. The column index for sorting works like the -index Tcl lsort command option, that is rows are sorted so that that specific column appears sorted in the specified order. The specified column must be visible (see indices member documentation above).

The indexColumns list specifies the columns required to uniquely identify a row in the table. In database talk, it represents the table key. To maintain backward compatibility, it is optional and defaults to 0, the leftmost column. The index columns are used when creating data viewer elements: their label is built by concatenating the key value for the cell row with the cell column title. The key value is the concatenation of the index column values for the cell. When specified, all the columns in the list must be visible (see indices member documentation above).

The helpText member specifies a text of any length, to be displayed when the user requests help information on the current module from within the help menu. The text can be HTML formatted or plain. HTML formatted help requires the <HTML> and <BODY> tags to be present, while tables and frames are not supported (and many other tags: stick to formatted text at the moment). The core will render HTML formatted or plain text in the module help window according to the help text contents.

The views member is optional. If specified, it defines one or more views to be used in place of the default view. One table will be displayed per view. For each view, 1 member must be defined: indices, the sort member being optional (syntax and usage are identical to the default table members). A swap optional member may be used if the table data is to be displayed with columns and rows swapped, which is generally the case when the data table has 1 or 2 rows, or a fixed number of rows (see memstats module for an example). The swap value is a boolean and must be either 0 or 1.

The switches member is optional. A switch is a single letter or a string with the - or + sign as header. The boolean value (0 or 1) specifies whether the switch takes an argument. If the switches member exists, an appropriate initialize procedure must be provided by the module (see Initialization). The core will take care of parsing the command line and reject any invalid switch / value combination for the module. The switches value may not be changed in the initialize procedure.

Note: if you wish to pass a sensitive password as an option to the module, use a special option name (-passwd or -password or --passwd or --password) so that the core knows when not to display or store readable characters, for example when entering a password in a dialog box, or storing password value in the moomps daemon data cells history database.

For example, a recent random module configuration is as follows:

In this case, data is presented in 2 tables: one for the CPU and memory usage, the other for disk usage.

The identifier member is optional. It is string that uniquely identifies this module. If set, it is displayed by the core in the initial data tables title area and data cell labels in viewers. This feature can be used for example in modules that gather data from a remote host: in such a case, the identifier could be set to dataType(hostName) (the ps module uses ps(host) string). The allowed character set for the identifier is identical to the allowed set for a module name.
Note that the identifier member is usually set in the module initialize procedure, as it usually depends on the module options.

The resizableColumns is optional. It is a boolean value (0 or 1) which specifies whether displayed data table(s) columns can be manually resized by the user with the mouse. Not available on a per view basis. If not present or set to 0, all the module data columns are automatically sized according to their content.

1.3. Initialization

The initialize procedure, if it exists, is invoked by the core before any update occurs (when the update procedure is invoked if the module is synchronous).

In order for a module to accept command line arguments, the initialize procedure must exist (more information below). Otherwise, if the module can take no arguments, it stays optional. In that case, it is rather redundant with in-line module code (outside of any module function or procedure). When the module takes no arguments, so does the initialize procedure if it exists.

The initialize procedure is mandatory when the module supports command line arguments, and in such a case takes option values as arguments.

Let us use the following command line as example:
  $ moodss random --asynchronous --other-option value -x 1234

In all cases, data members other than updates and switches can be set or updated in the initialize procedure, and successfully taken into account by the core.

Example for a module that take arguments, where a module identifier is generated according to the -i or --identify command line switches:

The core waits for the initialize procedure to be completed before initializing the next module. For modules likely to initialize slowly and/or susceptible to initialization failure, it is advised to allow the user interface to be updated in the meantime:

Similar techniques must be used in such cases within the module update procedure, so that the user interface and any modules running concurrently are not prevented to update. The remote capable moodss modules contain various coding techniques achieving that functionality, which I am sure you can improve on.

1.4. Termination

If a procedure named terminate exists in the module namespace, it is invoked when the module is unloaded dynamically. That procedure must take no arguments. For example:

1.5. Variable data

The tabular data (variable data) that the module code must update is stored in the data array (same as the module configuration data in Tcl, named after the module configuration data hash in Perl, and after the module configuration data list in Python).

In case of a synchronous module, the core invokes the module update procedure (which obviously must exist) when it is time to refresh the data display (tables and possibly graphical viewers). At this time, the update procedure may update the tabular data straight away (synchronous operation) or launch a request for later data update (asynchronous operation (only available in Tcl)).

In case of an asynchronous module, variable data may be updated at any time. The update procedure may not exist.

For all module types, it actually does not matter when the data is updated. The core will know that fresh data is available when the updates array member is set (actually incremented as it also serves as a counter for the number of updates so far).
It is the module programmer's responsibility to increment this counter right after all tabular data has been updated.

For example, retrieving information for the processes running on a machine is a local operation that can be achieved in a reasonably small amount of time. In such a case, data would be updated immediately and the updates variable incremented at the same time.
But if the data has to be retrieved from across a network, waiting for it to come back would cause a delay that the user would certainly notice, as the application would not respond to mouse or keyboard input during the whole time that it would take to fetch the whole data. In such cases, it is easier to let the update procedure return immediately without setting the updates variable, which would be incremented at a later time, only when the data would become available.

For example, in Tcl, when waiting for data to come across a network connection, the fileevent command could be used on a non blocking channel, where the script to be evaluated when the channel becomes readable would increment the updates array member.

In Perl, this is not yet possible, since the Perl interpreter is dormant outside of the update() function call (a solution to this problem is being studied).

In Python, it is possible to use threads, but at this time, there is no way to pass the information back to the Tcl interpreter outside of the Tcl interpreter driven call of the update function (a solution to this problem is being studied).

The column number must start from 0 up to the total number of columns minus 1 (no holes are allowed in the column sequence).
The row number can take any positive integer value (between 0 and 2147483647) and be defined in any order, as long as it is unique during the lifetime of the module data. If a new row is created, it must take a value that was never used: the index of a row that has disappeared cannot be reused. Row numbers need not be consecutive.

When all rows (or only those table cells that have changed) have been updated, the updates member array must be incremented so that the core knows that it can update the table data display.

The Tcl random module source code can be made to function asynchronously: please look into the random.tcl file.

2. installation

A module is a package in the Tcl sense. When writing a module, you must then provide a pkgIndex.tcl file along with the module code file, placed in the module directory. The pkgIndex.tcl file is very simple, as the following example shows:

Modules can be installed at any valid place that the Tcl core allows (look at the pkg_mkIndex manual page for more information).

When you unpack moodss, you will find the sample modules in sub directories. The current directory (.) is appended to the auto_load global list variable so that sample modules can be found when moodss is run from the unpacking directory.

For example, if you unpacked moodss in /home/joe/moodss-X.x/, you will find the random module package in /home/joe/moodss-X.x/random/ so that the following will work:
  $ cd /home/joe/moodss-X.x/
  $ wish moodss random

You can install your new modules in the default location: /usr/local/lib/ on UNIX. For example, if you move the files in /home/joe/moodss-X.x/random/ to /usr/local/lib/random/, moodss will still be able to find the random module (again, look at the pkg_mkIndex manual page for more information).

Please take a look at the INSTALL file for the latest information on how to install the moodss application itself.

3. Displaying messages

Note: this functionality is not yet available for Perl and Python modules.

You may want to inform the user of the module activity. You may use the message area (called the messenger, across the bottom of the application main window) through the following API:
  pushMessage "error: message..."
  popMessage
  flashMessage "warning: message..." numberOfSeconds

Your message can be any kind of string (1 line only: it should fit in a reasonably wide main window), and numberOfSeconds being optional and defaulting to 1. Note that messages sent to the message area are also displayed in the trace module table(s) if it(they) exist(s) (see below). In such cases, popping messages has no effect on trace module tables.

One should use the message area facilities with discretion, as it is already used by the application core for informing the user of modules loading, initialization, updates, context sensitive help, ... My advice is to use it for important messages or errors pertinent to the module itself.

The module name should not appear at the beginning of the messages as it is automatically prepended internally (see pushMessage above).
I suggest using the importance level (as in thresholds) as the header. Possible values are, in increasing importance order: debug, info, notice, warning, error, critical, alert, emergency. Look for examples in included modules.

You also have the option of using trace tables (also see trace module) through the following API:
  traceMessage "critical: message..."

The difference is that messages from the module are displayed in the trace module tables if they exist (see trace, there can be more than 1 trace module loaded), remain visible to the user for a longer period of time, and can be multi-line.

Finally, displaying separate (toplevel) message windows using Tk is of course always possible (if you load Tk in the module). Keep in mind that the core, not being aware of the module event that has taken place, will continue to invoke the module update procedure (unless the module is asynchronous, of course). In such a case, you may want to use an internal (to the module) busy flag so that the update procedure immediately returns until the user acknowledges the informational message. Other strategies are of course possible (let me know if you have one that you think should appear here as an example).

4. Error handling

In order to allow clean error handling with module loading either by command line arguments or dynamically while the application is running, the following rules need be followed:

Specifying the module name in the messages is not necessary as the core handles it.

5. Daemon specifics

Before moodss was packaged with a daemon (moomps), it was natural for such a GUI application to immediately react on module errors: when there is an error in a module initialization stage, the error is displayed in a dialog box and loading the module is aborted. Since the user is in front of the screen, he can try to fix the problem right away.

Now if the same module is used by the moomps daemon, in case of initialization error when the daemon is started unattended, the error message is logged but the module is never loaded. This is a problem if the error cause is a temporary communication breakage with a remote host, for example.

One solution would be for the moomps daemon to keep retrying loading a module until successful, but that may cause more problems than it solves, especially if the module loads a binary library or triggers some other mechanism which may not accept such forceful treatments (possibly when using secure channels, encryption, ...), or if the module contains bugs that prevents clean unloading, which may result in moomps hanging or crashing.

The module knows best if, when and how to retry its initialization, so if the module supports the --daemon option (no arguments), it is automatically set by moomps for the purpose of letting the module know whether it should keep trying to reinitialize on failures, or more generally do something different in daemon mode.

6. Perl

Moodss modules can also be written in the Perl language.

An embedded Perl interpreter is used for each loaded module, thus achieving complete independence between modules.

The tclperl library is required (available on my homepage).

Complete documentation for programming Perl modules can be found in the module development section, and as comments inside the Random.pm module source file.

7. Python

Moodss modules can also be written in the Python language.

An embedded Python interpreter is used for each loaded module, thus achieving complete independence between modules.

The tclpython library (version 3.0 or above, for Python 2.2 or above) is required (available on my homepage as source and Red Hat rpms).

Complete documentation for programming Python modules can be found in the module development section, and as comments inside the randpy.py module source file.

8. Threads and events programming

The Tcl language allows non blocking implementations in 2 ways:

Non blocking code is very useful when coding modules, as it prevents the user interface (the moodss core) from being hung when a module, for example, cannot reach a remote host. Such coding techniques are therefore recommended for a graphical user interface, which should stay responsive at all times.
Unfortunately, handling all the different cases and errors is not easy, and coding using threads or events in Tcl involves 2 different techniques.

That is why the object oriented line task class (linetask.tcl file) was created and is included in the moodss source code. It allows, via a common interface, the transparent handling of communication with a data pipe, uses threads if available, or asynchronous event handling otherwise.
It is used in several modules, and especially in the pci module (documentation), where all the code is thoroughly commented and can be used as a reference for all remote capable modules that need to communicate with a remote machine via a pipe.

Note that threads are used in priority if available, as they are guaranteed to never block the main process, which is still not the case using events, as they can hang, for example, on a non responding DNS query.

9. Tips and tricks

9.1. Single row tables

When data to be displayed is an array of unrelated values, the only solution is to organize the data in a table with a single row and 1 column per value (which does not prevent the use of several views for displaying the data).

In such a case, no index column is required and as a matter of fact rather gets in the way. The trick is to use an empty column 0, use as index by the core by default, and to prevent it from being displayed by using 1 or more views.

Please look at the cpustats module code for a working example.

9.2. Void values as numbers

In the module data array, you cannot leave or set a numeric cell empty when no data is available. Use the ? character (also used in statistics tables) instead.