redFOX

Programmer's Manual

Version 0.93

12 November, 2005

Copyright © 2004-2005, Red Plain Technology

1 Introduction: What is redFOX?

2 Installation

3 Getting Started

4 Programmer's Model

5 Threads

6 The Node Manager and Routing

7 Node Configuration

8 Exceptions

9 Class Reference

10 Platform Notes

11 Porting Guide

12 Diagnostic Aids

13 Limitations and Known Bugs

1 Introduction: What is redFOX?

redFOX is a high performance, lightweight, distributed object technology for embedded systems. It is designed from the ground up for real-time systems requiring fast and predictable communication between objects on different processors. It provides a fast, efficient implementation of the distributed object paradigm from which to build high performance, but flexible, distributed applications.

redFOX makes C++ objects remotely accessible by generating middleware directly from C++ class definitions. No Interface Definition Language (IDL) is needed. It runs remote invocations at consistent priorities across multiple processors and Operating Systems. It runs on heterogeneous processors, operating systems and message transports.

redFOX minimizes software overhead of remote invocation to match the performance of hand coded message passing. It brings the abstraction benefits of the distributed object paradigm without a performance penalty.

Local object invocation has the overhead of a virtual function call. This makes it practical to build a distributed application from many small, reusable objects, rather a few large, inflexible objects. It allows system partitioning to be deferred until the application behavior and the target environment are well known. It also makes it easier to port the application to a new environment.

This manual is for software developers using redFOX. It describes installation, using the object model and system configuration.

2 Installation

A redFOX distribution package contains all the components required to use redFOX distributed objects on one development platform and one target system. For this release of redFOX, two systems, Linux and Cygwin (on MS Windows) are supported and in each case the development and the target platforms are the same.

The installed system supplies a modified version of the GCC preprocessor which is called via a new command, rcc, and a link library, redfox.a, which is required by all programs which use redFOX. Sample prorgrams and documentation in the form of a manual (this document) and a set of HTML pages which document the redFOX API are also installed.

2.1 Pre-requisites

To build applications with redFOX, you will need a suitably configured Linux or Microsoft Windows development platform. In the case of Windows, cygwin and g++ are also needed. The following table shows what software is needed for the different target platforms supported by redFOX. 

Software	Description	Version(s)	From	Linux Target	OSE Target	Windows Target
Microsoft Windows^1	Operating System	2000, XP	www.microsoft.com		X	X
Red Hat Linux	Operating System	7.2+	www.redhat.com	X
Fedora Core Linux	Operating System	2+	fedora.redhat.com	X
OSE ^2	Real Time Operating System Development Kit	5.1+	www.enea.com		X
Cygwin	GNU sub-system for MS Windows	1.1.0+	www.cygwin.com		X	X
GCC g++	C++ Compiler	3.2+	included with Cygwin,  Linux and OSE	X	X	X
Boost	C++ Class Library	1.30.2+	www.boost.org, included with recent versions of Linux.	X	X	X

1This release of redFOX requires the Cygwin GNU environment for development.

2 Enea OSE is not supported in this release.

2.2 Distribution Files

redFOX is distributed as a single tar file which can be down downloaded from www.redplain.com. Tw o versions of the file are available:

• Windows/Cygwin: redplain_?_??_cygwin-ia32.tar.gz
• Linux/Fedora Core: redplain_?_??_linux-ia32.tar.gz

(where ?_?? represents the version number)

Both files have the same internal structure and are installed in the same way. Each file contains the following files:

File	Purpose	Shared Root
INSTALL	An installation script which performs all tasks needed to install redFOX
License.txt	Text of the redFOX license agreement
Gccxml-*	A tar file containing the GCCXML utility which is required by redFOX.	/usr/local /usr/local/share
runtime_folders*	A tar file containing the essential redFOX components	/usr/local
share_folders*	A tar file containing the program samples and the documentation	/usr/local/share

2.3 Installation Procedure

Download the distribution file and save it in a directory (eg /tmp) which will become the parent of the temporary installation folder.

The installation can be "shared" which means that it will be available in a standard system-wide location for all users or it can be "personal" which means that it will be installed in a location of the users choosing. The first case normally requires root or administrator privilege while the second is available to any user with write access to at least one directory.

The first step is to unpack the distribution:

$ cd /tmp

$ tar -xzf redplain_0_92_linux-ia32.tar.gz

This creates a new installation folder called redplain_redFOX_0_92. Change to this directory:

$ cd redplain_redFOX_0_92

Now execute the INSTALL script which will guide you through the remaining steps:

$ ./INSTALL

The questions posed by the script can be answered in most cases with a single keystroke such as "y" for "yes", etc. The license text can be viewed with the less viewer if available.

The script will offer you a choice of installation locations (as noted above). Select the one which suits your requirements.

This is an annotated example of an installation

$ ./INSTALL

=================================================================

Redplain Technology Ltd - RedFOX Middleware Installation

INSTALLATION SCRIPT

Version 0.1 - Alpha Release 1 October 2005

=================================================================

This script installs redFOX, Redplain's distributed object system

for C++. The installation can be for the current user or for all

users (requires root privilege).

redFOX requires the GCCXML utility and the Boost library to be

installed also. This script will check that they are accessible

and GCCXML will be installed, if required. Boost must be installed

separately

This version of redFOX only works with g++ V3.n on Linux and

MS Windows (with Cygwin) platforms. Future releases will work on

a wider range of platforms.

Some of this software is not in the public domain and may be used

only in accordance with the licence displayed below.

Redplain Technologies Ltd, Ashbourne, Co Meath, Ireland

email: info@redplain.com web: www.redplain.com

-----------------------------------------------------------------

Step 1 - read the licence

Type ENTER to view or L to view with 'less' (q to quit)? ➊

SOFTWARE EVALUATION AGREEMENT

1.Introduction and Acceptance.

This Software License Agreement (the "Agreement") is a legal

agreement between you (either an individual or an entity)

("You") and Red Plain Technology Ltd. ("Red Plain") regarding

[...missing sections of license...]

14.Questions.

Should you have any questions relating to this Agreement, or

if you desire to contact Red Plain for any reason, please call

(USA) +1-919-602-3865; or (Europe) +353-87-2208233.

S

I accept and agree the license ? [y/n] y ➋

Step 2 - decide where the package should be installed

Please select the location where the package will be installed

1: standard location, shared by all users ()

2: your home directory (/home/john/redplain)

3: other location (to be defined)

enter a number from the list above:2 ➌

Linux_ia32

Location /home/john/redplain does not exist -- create it ? [y/n]y ➍

GCCXML is already installed

Do you want to install a new copy in /home/john/redplain ? [y/n]y ➎

[...list of installed files...]

Delete the installation folder /tmp/redplain_redFOX_0_92/. [y/n] n ➏

Redplain middleware redFOX has been successfully installed

The numbered notes below refer to the labels in the example above.

➊ Enter return to view the license agreement directly or "L" to use the less viewer. If you use less, type "q" to end viewing and return to the INSTALL script.

➋ Enter "y" to accept the agreement. If you enter anything else the script terminates.

➌ At this point you select the location for the installed files. Option 1 gives the shared installation in standard locations; option 2 places all the files in a sub-directory of your home directory while option 3 allows you to choose the location. In the example, option 2 is chosen. Options 2 and 3 require that the target directory does NOT exist. If it does the script will terminate.

➍ This question allows you to check that the correct target has been chosen. Answer "y" to continue.

➎ At this point a check is made to determine if the GCCXML utility is already installed on your system. If it is you do not need to install it again. However, the version supplied with redFOX may be more recent.

➏ At this point the installation is complete and you are offered the choice of retaining or deleting the temporary installation folder. Answer "y" or "n". (In the current release the folder is never deleted).

3 Getting Started

The redFOX distribution includes a simple demo application, which you can build and run under Linux or Windows. Use it to check that everything is working, to familiarize yourself with the tools and to serve as a starting template for your own applications.

The demo application has one source file, counter.cpp and a Makefile.

1. Open two console windows with the current directory at $(REDPLAIN)/samples/counter.
2. In one window, build the application by typing make.
3. In the same window, run counter, taking standard input from counter0.cfg
4. In the other window, run counter, taking standard input from counter1.cfg.

If everything worked, you have run your first distributed application using redFOX, albeit as two processes on a single processor. You can copy this simple application and use it to experiment with distributed processing.

The files counter0.cfg and counter1.cfg contain the node configuration information for the redFOX node manager. The information includes the node number, the total number of nodes and details of connections to other nodes. In this case, there are two nodes and they communicate by TCP/IP on the local host.

The application on each node starts by calling main(). When the first call is made to redplain::node_manager::instance(), the configuration information is read and the node is initialized. The node can now send, receive and run remote invocations.

In the counter application, a simple counter class (1) has a constructor (2), a destructor (3), increment (4) and value (5) methods. The class is accessed remotely using class redplain::dref<Counter>, which is similar in nature to a C++ rerefence and is automatically generated from directives (6). Node 0 merely initializes (7) and waits for shutdown (14). All activity on node 0 is a result of remote invocations from node 1. Node 1 waits for node 0 to come up (8), remotely creates a counter object on node 0 (9), increments it 10 times (10) and reads its value to confirm that it is correct (11). Node 1 then destroys the counter (12) shuts down node 0 (13) and terminates itself.

// redFOX sample application – remote counter

#include <iostream>
#include <redplain/dref.h>
#include <redplain/node_manager.h>
#include <redplain/thread.h> // for redplain::thread::sleep
using namespace std;
using namespace redplain;

// A simple class to demonstrate remotely accessible objects ///////////////

class Counter { // (1)
private:
uint32_t n_;
public:
inline Counter() : n_(0) { // (2)
cout << "Counter::Counter() called.\n";
}

inline ~Counter() { // (3)
cout << "Counter::~Counter() called.\n";
}

inline void increment() { // (4)
cout << "Counter::increment() called.\n";
++n_;
}

inline uint32_t value() const { // (5)
cout << "Counter::value() called. Returning " << n_ << "\n";
return n_;
}
};

// Directives to generate dref<Counter>, making Counters remotely accessible ///

#pragma redplain dref<::Counter> interface // (6)
#pragma redplain dref<::Counter> implementation

// Application /////////////////////////////////////////////////////////////

int main() {

redplain::node_id_t node = node_manager::instance().local_node(); // (7)
cout << "redFOX Remote Counter Demo starting on node " << node << ".\n";
if (node == 1) {
while (!node_manager::instance().ping(0)) // (8)
redplain::thread::sleep(1);
cout << "Creating remote counter ...\n";
dref<Counter> remoteCounter(0); // (9)

const unsigned loops = 10;
for (uint32_t n=0; n != loops; ++n) {
cout << "Incrementing remote counter ...\n";
remoteCounter.increment(); // (10)
}

if (remoteCounter.value() == loops) // (11)
cout << "Remote Counter working.\n";
else
cout << "Remote Counter faulty.\n";

cout << "Deleting remote counter ...\n";
redplain::delete_object(remoteCounter); // (12)

// Tell other node manager to terminate
node_manager::instance().shutdown_all(); // (13)

} else {
// Main thread on node 0 just waits for termination.
node_manager::instance().wait_for_shutdown(); // (14)
}

cout << "Node " << node << " terminating.\n";
return 0;
}

4 Programmer's Model

4.1 Terminology

In the redFOX programmer's model, computation is performed in nodes which contain execution threads and objects. Typically, a node comprises a processor, a memory, an operating system and I/O interfaces. Nodes are connected directly and indirectly by networks. This is illustrated below.

Computation threads invoke object methods which, in turn, invoke other object methods. Invocations may be local (intra-node) or remote (inter-node), synchronous or asynchronous.

Nodes may be heterogeneous in processor architecture, operating system, programming language and tool chain. These differences are hidden in remote invocations.

Threads are prioritised and pre-emptable subject to the limitations of the underlying operating systems.

4.2 Objects and drefs

A key concept in redFOX is the dref, a distributed object reference that is valid on any node. The developer uses drefs in a distributed application like references are used in a non-distributed application.

In C++, a dref for an application class is created using a template and the application class definition. Drefs are small (about the size of two pointers) and fixed size. They can be read, copied and transmitted with a computational cost not much higher than that of a C++ reference.

In C++, drefs are modelled as specialisations of the redplain::dref template class. For each application class T, there is a class redplain::dref<T> whose methods mirror those of T, but these methods invoke (locally or remotely) a particular object of class T.

The diagram below shows the relationship between a dref class and an application class.

A dref<T> can be created from an existing T, from another dref<T> or using a constructor that remotely creates an object and refers to it. For every constructor of T, dref<T> has a constructor with a node identifier prepended to the argument list. Such a dref constructor creates an object on the specified node using operator new and initializes itself to access to the newly created object. An object created in this way can be destroyed using the function template redplain::delete_object(dref<T>);

4.3 rcc

To compile a redFOX-based application, run the redFOX pre-processor, rcc, instead of the C++ compiler. rcc parses application classes and generates the appropriate dref and other classes. It passes the generated code along with the application code to the C++ compiler. The diagram below illustrates the generation and incorporation of code into a distributed application. (In the diagram, it is assumed for simplicity that all nodes run the same application image.)

4.3.1 Directives to Generate a dref Class

rcc does not generate drefs for all application classes. To do so would add unnecessarily to compile time and code size. To cause rcc to generate code for a particular class, two pragmas must be added to the source code at appropriate places.

For each dref class, rcc must generate source code for both interface and implementation. This code is read by the compiler and included in the application. The interface consists of the dref class definition and some inline functions. The implementation consists of statically allocated data and the bodies of the larger functions.

4.3.1.1 Interface Directive

After a class declaration, in the same file (normally a header file), but outside any namespace or class scope, an interface pragma should be inserted as follows.

#include <redplain/dref.h>

//...

class Counter {

//...

};

#pragma redplain dref<::Counter> interface

This causes rcc to insert a class definition for class redplain::dref<::Counter> at the point where the pragma is found.

The interface pragma should be inserted

• after inclusion of redplain/dref.h
• after the definition of the class
• outside any namespace or class scope
• in every compilation unit that uses the dref

This is usually done by inserting it at the end of the file containing the class definition.

The interface pragma is specified in section 4.3.1.4 below.

4.3.1.2 Implementation Directive

The non-inline functions and static data for a dref class must be compiled and linked exactly once in the application. This is done by inserting an implementation pragma into one compilation unit, usually the source file for the application class, after inclusion of the application class header file.

#include "Counter.h"

#pragma redplain dref<::Counter> implementation

// Non-inline Counter methods, etc, follow...

The implementation pragma is specified in section 4.3.1.4 below.

4.3.1.3 Remote Only Classes

Distributed applications often involve objects that only exist on some nodes and whose classes are not compiled for other nodes. If we remotely invoke such objects from nodes that do not include the application class, we must make sure that rcc does not generate code (such as local invocation) that references the application class. To specify to rcc that a class is not implemented locally, add the option remote_only to the interface and implementation pragmas. For example:

#include "redplain/dref.h"

//...

class Counter {

//...

};

#pragma redplain dref<::Counter> interface (remote_only)

Even if a class is "remote_only" for a particular node, the application image must still contain an implementation pragma for the class. Otherwise, a link error will occur. To prevent this, ensure that the implementation pragma is in one compilation unit of every node that uses the dref class.

4.3.1.4 Pragma Specifications

The following pragma directives are recognized by rcc.

#pragma redplain dref<CLASS> interface [([OPTIONS)]

#pragma redplain dref<CLASS> implementation [(OPTIONS)]

where

CLASS is the fully qualified name of the application class (including namespace and class nesting).

OPTIONS is a comma-delimited set of options that effect code generation as follows:

remote_only specifies that the class is not implemented on the local node, so code referring to the class should not be generated.

4.4 Const objects

The constness of methods of dref<T> reflect the constness of the corresponding methods of T. (None of the methods of dref<T> actually alter the state of the dref<T>.)

In addition to dref<T>, rcc generates a separate class dref<const T>, which can be used to access const methods only of a class T. A dref<T> may be converted into a dref<const T>, but not vice versa.

4.5 dptrs

Just as a dref is designed to be a generalization of a reference, so is a dptr (pronounced "d-pointer") a generalization of a pointer. Like a pointer, a dptr can have a null value and can be assigned and compared. It does not, however, support arithmetic operations. dptr<T>::operator-> yields a dref<T> if the dptr is not null and throws an exception if it is null.

4.6 Link Errors and Unimplemented Methods

It is legal in C++ to declare a method in a class and not implement it. This will not usually cause problems until some code calls the method. Then a link error occurs. The code generated by rcc contains calls to every remotely invocable method in a class, so if a method is not implemented, a link error will occur when the generated code is linked. Therefore, when using redFOX, all remotely invocable methods must be implemented.

4.7 Local and Remote Invocation

When a dref method is called, the first thing that happens is that the dref determines if the object is local. If it is, the dref retrieves its object pointer and makes the invocation directly. With compiler optimization, a local invocation can have less runtime overhead than a virtual function call.

If the object is remote, the dref method constructs a message encoding the parameters and sends it to the node where the object is stored. If the invocation is synchronous (see 4.8), the dref method waits for a response. The response normally encodes a return value and modified parameters. If the invocation throws an exception or suffers node or communications failure, the dref method throws an exception.

4.8 Synchronous and Asynchronous Invocation

A synchronous remote invocation is one that waits for and retrieves a response. The response explicitly conveys information of return value and modified parameters. It also implicitly conveys the information that the invocation has completed. An asynchronous invocation is one that sends an invocation message with instructions not to respond and that does not wait for or receive a response. This means that modifications to parameters are lost and that the caller does not know when, how or if the invocation completes.

An asynchronous invocation has a number of uses, despite its limitations. It allows the invoking thread to resume work immediately when the invocation message is sent. In some systems, message sending may be much slower than computation, so it may not be efficient to keep a thread blocked waiting for a response that will take a long time to come.

Another use of an asynchronous invocation is to start an additional thread of execution. A distributed system has at least one processor per node and it needs many more threads to keep it busy than a single node. An asynchronous invocation is a convenient way to start a thread.

redFOX drefs provide synchronous methods to invoke all application object methods. For application methods that have a void return type, they also provide asynchronous methods. To call a method asynchronously, put the method name in the macro REDPLAIN_ASYNC().

Consider for example our Counter class, with its increment() method returning void and its value() method returning a uint32_t. dref<Counter > has synchronous methods to invoke both Counter methods and an asynchronous method to invoke increment(). Usage as follows:

dCounter.increment(); // Synchronous invocation

dCounter.REDPLAIN_ASYNC(increment)(); // Asynchronous invocation

In the case of local objects, synchronous dref methods do direct calls, and asynchronous dref methods delegate the invocation to a different thread. This may result in an asynchronous invocation being slower than the synchronous invocation when the object is local. So asynchronous isn't always faster.

The following code fragment shows how to invoke a local object synchronously and a remote object asynchronously.

redplain::dref<Counter> dCounter(/*…*/);

// call dcounter.increment() synchronously is local, asynchronously if remote.

If (redplain::is_local(dCounter))

dCounter.increment();

else

dCounter.REDPLAIN_ASYNC(increment)();

4.9 Parameters and Return Values

redFOX is designed to make method invocation efficient for both local and remote objects. It is also designed to be conceptually similar to object invocation in C++. But to use redFOX successfully, we must bear in mind how parameters are handled. In particular, the consequences of remote invocation should be considered when designing interfaces.

If an invocation is local, parameters are passed directly to the method, so invoking a local object through a dref in almost indistinguishable from invoking the method using a C++ reference. redFOX tries to make remote invocation functionally identical to local invocation in most cases without sacrificing efficiency.

redFOX treats different categories of parameters differently for remote invocation.

4.9.1 Parameter Passing Conventions

A value parameter is sent in a message. A copy of the value is passed to the method of the remote object and discarded when the invocation is finished.

A const reference parameter is processed similarly. A reference to a copy of the referenced value is passed to the method of the remote object. The copy is discarded when the invocation is complete.

A const pointer parameter is processed like a const reference parameter unless the pointer is a null pointer. A non-null pointer is assumed to point to a valid object. The method of the remote object is called with a pointer to a copy of the value. The copy is discarded when the invocation is complete. If the pointer is null, the method of the remote object is called with a null pointer.

A non-const reference parameter is processed like a const reference parameter, except that the copied (and possibly modified) value is copied back when the invocation is complete and the original value is overwritten with the copy. This makes passing a reference parameter to a remote object have the same effect as passing a reference to an ordinary C++ method.

A non-const pointer parameter is processed like a non-const reference parameter except that a null pointer causes the method on the remote object to be called with a null pointer.

A return value is returned by the method call on the remote object. It is copied into the response of the invocation and discarded. The copy of the return value is returned by the dref method. Pointers and references returned by a method would be invalid if they were the result of a remote invocation. Therefore, application class methods returning pointers and references do not have corresponding dref methods.

Passing drefs and dptrs has special significance in redFOX. Drefs and dptrs are small enough to pass by value, but it may be necessary to pass a reference instead if the automatically generated dref class definition does not precede its use as a parameter.

These conventions can be illustrated with an example.

class A; // Unknown class A

class B {

public:

void f1(int, const std::string& input, std::string& output);

A f2(A*, const A*, redplain::dref<A>&); // A unknown. Pass dref<A>& instead of dref<A>.

B* f3(); // Not accessible via dref because it returns a pointer

redplain::dref<B> f4(); // Use instead of f3() because it is remotely accessible

};

#pragma redplain dref<B> interface

4.9.2 Consequences

Some consequences of these conventions need to be pointed out.

4.9.2.1 Consider using drefs and dptrs instead of references and pointers.

drefs and dptrs may be passed between nodes with little runtime overhead. They work like C++ references and pointers and passing them is key to fully exploiting distributed objects.

4.9.2.2 Avoid pointer access to array parameters.

Pointers must not be used to access arrays in a remotely accessed method. This will cause memory corruption in a remote invocation because only one value is allocated. Alternatives to accessing arrays include

1. In the case of a fixed size array, encapsulate it in a struct and pass a pointer or reference to the struct

2. Replace the array with an STL container and pass a reference to the container.

3. Replace the array with an object with methods and pass a dref to the object.

4.9.2.3 Consider remote invocation overhead of an output parameter

A common idiom in C and C++ is to use a pointer or reference as an output parameter. The invoked method does not read the value, but only writes a result to it. redFOX cannot determine if a referenced value will be read by a method or not, so it must assume that it will be and it will therefore copy the value in the invocation. This is inefficient if the copied value is large and is not read before being overwritten. There are a number of ways of dealing with this inefficiency.

1. Ignore it. If the object is small (e.g. an int32_t), the overhead of copying an extra few bytes in the invocation message is unlikely to have a significant impact on performance.

2. Minimize it. Most objects that are not small are variable sized (e.g. STL containers). Passing a reference to an empty STL container or std::string adds just one byte to a redFOX invocation.

3. Return it. The return value of a method is not copied in the invocation message. An output parameter may be eliminated by aggregating an additional value with the return value.

Example:

class Big {

// This class has a lot of state.

};

class A {

public:

void f1(Big* output); // Bad: redFOX unnecessarily copies the Big pointed to in invocation

void f2(int32_t* output); // Not too bad because overhead for copying an int32_t is small

void f3(std::string* output); // OK if string is empty before call

Big f4(); // Good: State of Big object is only in response, not invocation

};

#ifdef REDPLAIN_RCC_PASS1

REDPLAIN_GENERATE_DREF(::B, dref_B)

#else

#include "dref_B.rfh"

#endif

4.10 Data Representation

When a method is invoked remotely, compatibility of data representation in the two processors must be addressed. Compatibility issues include endian-ness, sizes of data types and variety of floating point formats.

4.10.1 Endian-ness

Endian-ness is handled transparently by redFOX. Data in messages is endian-converted as necessary.

4.10.2 Data Type Size

Different processors may have different data size types. An int may be 16 bits, 32 bits or 64 bits, depending on processor, compiler and compiler switches. redFOX handles data size compatibility by taking it into account when selecting remote methods. If a distributed application runs on processors with different sized data types, methods must be defined using the size-specific typedefs of <stdint.h>.

So if the application attempts to call Widget::f(int) from a node with 32 bit ints to a node with 64 bit ints, the invocation will throw an exception because the requested method is unavailable. If, however, the method is defined as Widget::f (int32_t), the invocation will succeed because an int32_t is the same size on both nodes.

This approach has the following consequences.

1. redFOX does not silently truncate or extend data passing between nodes with different sized data types.
2. If processors have different sized types, the developer must be explicit about the size of data used in method parameters.
3. If all processors use the same sized types, the developer can avoid addressing the issue. It may, however, be wise to use sized data types from the start to keep the software ‘64 bit clean' and avoid problems when the hardware environment changes.

4.10.3 Floating Point Formats

Different processors support different floating point formats. redFOX does not attempt to distinguish between them. It assumes that floats and doubles are the same on all nodes.

4.11Encoding and Decoding - Object Serialization

redFOX uses template function specializations to copy parameters of different types between nodes.

4.11.1 Pre-defined Serialization Functions

redFOX includes library functions and templates to serialize fundamental types, std::strings, STL containers and its own classes and templates, including drefs and dptrs.

4.11.2 Serialization Functions for Application Classes

If you pass or return objects of any other class (whether by value, pointer or reference) to a dref method, you must supply two template specialization functions for that class. One function, redplain::encode<T>( redplain::encoder& e, const T& rhs) encodes an object of class T for transmission. The other function is a constructor T::T(redplain::decoder& d) that reads the object state from a decoder.

For a typical class, these functions are straightforward. They simply encode or decode each member variable and base class of the object. They do this by calling other encode and decode functions.

Example:

class Widget : public Tool {

private:

int32_t x;

std::tring y;

std::map<std::string, std::vector<Tool> > lookup;

friend redplain::encode(redplain::encoder&, const Widget&);

public:

Widget (redplain::decoder d) : Bar(d), x(d), y(d), lookup(d) {

};

};

namespace redplain {

template<> void encode<Widget>(encoder& e, const Widget& rhs) {

encode<Tool>(e, rhs);

encode(e, rhs.x);

encode(e, rhs.y);

encode(e, rhs.lookup);

}

};

In the example above, Widget inherits from Tool and contains three member variables; an integer, a string and a lookup table mapping a string to a vector<Tool>. To encode a Widget, the Tool base class is encoded first, then the members x, y and lookup in turn. The order in which these are encoded is unimportant, but it is essential that they are decoded in the same order.

For this example to work, class Tool must also have serialization functions defined. The writer of class Widget doesn't need to know anything about serialization of class Tool, except that the functions exist.

WARNING: Because redFOX is designed for performance, it does not check whether serialized objects decode all the data they encode. Failure to decode all the encoded data and only all the encoded data will have consequences beyond an object in an incorrect state. It will disrupt communications between the nodes and may cause undefined behavior in multiple nodes.

4.11.3 Objects containing Pointers and References

Objects that contain pointers or references require more consideration. A copy of a pointer would be invalid if transferred to a different node. If the object owns the data pointed to and if that data is on the heap, it is reasonable to copy the data pointed to and create a copy of it remotely on the heap. Of course, the heap pointer must be deleted when the remote copy of the object is destroyed.

If the data is not owned by the object, the situation is less clear cut. If the cost of a remote invocation to access the data is not too important, if it is accessed using method call, it may be appropriate to replace the reference or pointer with a dref or dptr.

4.11.4 Objects containing drefs and dptrs

redFOX contains pre-defined functions for serializing drefs and dptrs. These must be called when objects containing drefs and dptrs are serialized.

4.11.5 Non-copyable objects

Some objects are not suitable for copying remotely. An object that may not be copied locally (which is often enforced by a private unimplemented copy constructor) is unlikely to behave correctly if copied remotely. Objects that own local resources including dynamically allocated memory, open files, etc.) may also be unsuitable. Classes whose objects will not work correctly when copied remotely or which lack an assignment operator (used to replace a local object with its modified remote copy) should not have encode and decode functions defined. This will cause a compile error when such objects are passed to a dref method, whether by value, pointer or reference.

4.12 Object Lifecycle

An application object may be created locally or remotely. In either case, drefs and dptrs to the application object may be created and copied to other nodes. Application objects are created locally using standard C++ mechanisms.

Remote application objects are created using special dref constructors that result in objects being created on the remote heap.

The application is responsible for object destruction and for ensuring that drefs to destroyed objects are not used. Remote (or local) objects may be deleted using the function template redplain::delete_object(dref<T>)..

4.13 Application Startup and Termination

When a redFOX application calls main(), the node_manager singleton is inactive. The first time redplain::node_manager::instance() is called, the node_manager starts threads to receive and execute remote invocations from other nodes. An easy way to get distributed processing started is for a thread on one node to create an object on another node and use the dref to that object to invoke it. Different threads of execution can be spawned by making asynchronous invocations on remote objects.

redplain::node_manager::instance().shutdown() should be called on each before the application terminates to stop the processing of remote invocations in an orderly manner. To coordinate termination on all nodes, call redplain::node_manager::instance().shutdown_all() on one node before returning from main() and call redplain::node_manager::instance().wait_for_shutdown() on all other nodes.

4.14 Classes, Inheritance and Polymorphism

Method calls on objects with virtual and non-virtual functions behave in redFOX largely as experience with C++ would lead us to expect. If A is a base class of B, then a dref<B> can be converted to a dref<A>. A method invocation on a dref<A> or a dref<B> will execute the same methods as if an A& or a B& were used in place of the drefs.

4.15 Operator Overloading

Assume we have a class Widget with two overloads for operator+. One is a member function that takes a Widget parameter and returns a Widget&. The other is a global function that takes a const Widget& and an int32_t as parameters.

Neither of these will be reflected in member functions of dref<Widget>. The first won't have a member function in dref<Widget> because it returns a Widget&. The second won't result in a member function because it is a global function and not a member of Widget.

Example:

class Widget {

/* ... */

Widget& operator+(const Widget&) const; // Member overload

};

Widget& operator+(const Widget&, int); // Global function overload

Widget w, x;

dref<Widget> dw(w);

Widget y = dw + x; // Error! Won't compile.

Widget z = dw + 1; // Error! Won't compile.

To make the addition of Widgets and the addition of int32_t's to Widgets accessible via drefs, both must be implemented as member functions that don't return pointers or references.

Revised Example:

class Widget {

/* ... */

Widget operator+(const Widget&) const; // Member overload

Widget operator+(int32_t); // Member overload

};

Widget operator+(const Widget&, int); // Global function overload

Widget w, x;

dref<Widget> dw(w);

Widget y = dw + x; // O.K.

Widget z = dw + 1; // O.K.

The fact that some operator overloads are remotely accessible, but others (that are syntactically identical) aren't may impact portability and reusability. When a third party class library is upgraded or replaced, it may be changed in a way that doesn't normally affect usage of the class, but that unexpectedly prevents invocation via a dref.

4.16 Templates

It is possible to use dref and dptr classes for template instantiations, just like any other class. It is, however, necessary to write dref class generation directives (see 4.3.1) for each instantiation of the template.

Example:

template<typename T> class Widget {

/* ... */

};

// Remotely access Widget<int32_t> and Widget<char>

#pragma redplain dref<::Widget<int32_t> > interface

#pragma redplain dref<::Widget<char> > interface

// Following lines must be in one implementation file.

#pragma redplain dref<::Widget<int32_t> > implementation

#pragma redplain dref<::Widget<char> > implementation

5 Threads

redFOX supports consistent thread prioritization. Threads run at different priorities on a node will, where possible, have remote invocations run at the same relative priorities. redFOX provides a thread class to allow the user to control thread priorities across nodes. It does not address scheduling policies, etc., which are left to the underlying operating systems.

5.1 redFOX Threads

redplain::thread is an abstract base class whose run() method must be implemented in derived classes. This method is called when the thread is started. Between creating the thread object and calling its start method, the set_priority method may be called to alter the priority at which the thread is run.

The wait method can be called to wait until the thread has finished running. When a thread is destroyed, wait is called by the destructor, so it is not possible to destroy a running thread.

5.2 Server Threads

redFOX runs a set of server threads on each node. These threads are run at different priorities and their role is to execute incoming remote invocations at the appropriate priority.

5.3 Priority

redFOX threads run at a priority in the range between thread::lowest_priority and thread::highest_priority inclusive, and at thread::normal_priority by default. These redplain::thread priorities map to underlying operating system thread priorities in a way that ensures that the operating system enforces the prioritization. If there is not an operating system priority for each possible thread::priority_t, it will not be possible to set a thread to some priorities. Attempting to set a thread to an unavailable priority will result in its being set to the next lowest available priority by default. To select the next highest available priority instead, set the second parameter in thread::set_priority to true. The following code samples show how thread priority may be manipulated.

redplain::thread* my_thread = redplain::thread::my_thread();

assert(my_thread != 0);

priority_t my_priority = my_thread->get_priority();

class background_thread : public redplain::thread { /* ... */ };

background_thread bt;

bt.set_priority(my_priority – 1);

// bt will run at next lowest priority.

class urgent_thread : public redplain::thread { /* ... */ };

urgent _thread ut;

ut.set_priority(my_priority + 1, true);

// ut will run at next highest priority

bt.start();

ut.start();

5.4 Thread Configuration

The node configuration information controls the mapping of thread::priority_t to OS priority and the number of server threads at each priority.

Two entries in the configuration information achieve this. A "priorities" entry specifies a numerical OS priority for each priority_t. Where a priority_t has no corresponding OS priority, the next lowest OS priority is entered.

This is illustrated with the following example for a mapping for Windows. Windows allows a total of seven thread priorities, with numerical values -15, -2, -1, 0, 1, 2, 15, with 0 as a default. In this example, we map thread::lowest_priority to -15, thread::highest_priority to 15 and cluster other values around thread::lnormal_priority, which maps to 0.

priorities=-15,-15,-15,-15,-15,-15,-15,-15,-15,-15,-15,-15,-15,-15,-15,-15,-15,-15,-2,-1,0,1,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,15

A "threads" entry in the configuration information specifies the number of server threads to run at each priority. The number for each unavailable priority must be 0. (??? Must there be at least one thread at each available priority?)

threads=1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1

It is important for the correct operation of remote invocations to ensure that the priority of the invoking thread is available on the invoked node and that adequate server threads are available. To decide which priorities to map and how to many server threads to run at each priority, it is useful to think in terms of "Distributed Threads".

5.5 Distributed Threads

When considering synchronous remote invocations, it is helpful to think in terms of distributed threads. A distributed thread initially consists of one thread running on one node. When a synchronous remote invocation occurs, the initial thread blocks and execution of the distributed thread is taken up by a redFOX server thread on another node. A distributed thread may visit many nodes as it executes and is blocked on all nodes waiting for a remote invocation return.

The message sequence diagram above illustrates the execution of a distributed thread.

· The thread starts on Node 1.

· It invokes an object on Node 2, blocking on Node 1.

· It invokes an object on Node 3, leaving it blocked on Node 1 and Node 2.

· It invokes an object on Node 1. redFOX can identify this type of call back, so it re-uses the original calling thread to execute the incoming invocation. The thread is now running on Node 1, but blocked on Node 2 and Node 3.

· It returns to Node 3, Node 2 and Node 1 and finally it is running on Node 1 only.

5.6 Semaphores and Locks

A semaphore class redplain::semaphore is implemented using the underlying Operating System's equivalent. redFOX also includes a redplain::lock class that holds a semaphore for its lifetime and a redplain::possible_lock class that may acquire a semaphore during its lifetime and releases it if it does. Semaphores may be remotely accessed using their dref and dptr classes.

6 The Node Manager and Routing

Each node in a redFOX application has a node manager. It is a singleton and is accessed using redplain::node_manager::instance(). The first time that redplain::node_manager::instance() is called, the node manager is created. It reads the node configuration from the redplain::in() ipipe.and sets up server threads to process incoming invocations and dispatcher threads to process messages from neighboring nodes.

The node manager has methods to

· return the node_id_t of the current node

· return the number of nodes in the system

· return drefs to managers of other nodes

· terminate processing of incoming invocations

· terminate all nodes in the system

· read and write the routing table for messages destined for other nodes

It also forwards messages (invocations and responses) destined for other nodes. If the routing table is set up correctly, a message will be forwarded to neighboring nodes until it reaches its destination. This means that a node does not need to establish communications to all nodes in the system, but just those to which it has direct access. A node can gain access to a distributed system (and vice versa) with just a serial cable and no protocol stack.

7 Node Configuration

Node configuration is read as ASCII text from redplain::in(). It consists of a number of lines, the last of which consists of the string "end". Other lines are either comment lines, which begin with "//" or name=value pairs. A name is an alphanumeric string. A value may be a string, a decimal integer or a comma-delimited sequence of values. The order in which names appear is significant and some must appear before others.

The table below lists the names of configuration items, their types, the number needed in the configuration text, their pre-requisites and notes.

Name	Type	Number needed in configuration	Pre- requisites	Notes
Node	integer	1		Assigns node id. Each node must have a different node between 0 and nodes-1 inclusive. Must occur once in configuration.
Nodes	integer	1	node	Number of nodes running the application. Must occur once in configuration.
Priorities	integer sequence	0-1		Operating System priority of each redplain::thread priority (see 5.4) Must occur once in configuration.
Routes	integer sequence	0-1	node, nodes	For each node, the node to which messages destined for it should be forwarded. Must occur once in configuration.
Threads	integer sequence	0-40	priorities	Controls creation of server threads at a given priority. Parameters are four numbers specifying · thread priority · stack size in bytes · initial number of threads to run · maximum number of threads to run Must occur once per supported priority in configuration.
connection	string sequence	0-	node, nodes	The first string is connection type. Subsequent strings are interpreted depending on the type. May occur any number of times in configuration.

The table below list connection types and their parameters.

Type	Parameters	Notes
Tcp_server	integer sequence	Two integers specifying · IP port number · Number of tcp_clients to connect to
tcp_client	string sequence	Two strings specifying · IP host name of server to connect to · IP port name of server to connect to
Rs232	string sequence	Operating System dependent sequence of strings controlling baud rate, handshake, etc.

This is a sample node configuration string for a redFOX node.

// redFOX Configuration string

// for node 1 of 6 running Windows

// running 1-6 server threads at priorities 0, 18, 19, 20 and 21

// with three connections to other nodes (nodes 0, 2 and 4)

// and forwarding messages for nodes 3 and 5 to node 4.

// node id

node=1

// Number of nodes

nodes=6

// priorities maps redFOX priorities 0-39 to OS priorities.

// Where a redFOX priority does not map to an OS priority,

// repeat the mapping of the previous redFOX priority.

// This example maps priorities

// 0 to -15, 18 to -2, 19 to -1, 20 to 0, 21 to 1, 22 to 2 and 23 to 15.

priorities=-15,-15,-15,-15,-15,-15,-15,-15,-15,-15,-15,-15,-15,-15,-15,-15,-15,-15,-2,-1,0,1,2,15,15,15,15,15,15,15,15,15,15,15,15,15,15,15,15,15

// threads defines the minimum and maximum number of thread to run at each

// priority. Parameters are:

// priority, stack size (bytes), minimum threads, maximum threads.

threads=0,4096,1,6

threads=18,4096,1,6

threads=19,4096,1,6

threads=20,4096,1,6

threads=21,4096,1,6

// routes defines the "next hop" node to which messages for each node

// should be sent.

// In this example, node 1 sends messages directly to nodes 0,2 and 4

// and sends messages for nodes 3 and 5 to node 4.

routes=0,1,2,4,4,4

// 2130706433 = 0x7f000001 = 127.0.0.1 = localhost

connection=tcp_client,2130706433,54321

connection=tcp_server,54322,1

connection=rs232,com1,baud=115200 parity=n data=8 stop=1 to=on xon=off odsr=off octs=off dtr=on rts=on idsr=off

end

8 Exceptions

In redFOX, node failures, communication failures and some internal failures result in C++ exceptions derived from class redplain::red_exception. Application exceptions may also inherit from redplain::red_exception. Uncaught red_exceptions are propagated between nodes. If software on node A invokes an object on node B, and if this object in turn invokes an object on node C and if a red_exception is thrown on node C, an uncaught red_exception will be propagated to node B and then to node A.

Exception propagation has the consequence that exception-safe C++ classes behave correctly in the presence of distributed system failures, even if these classes are not specifically designed for distributed systems.

8.1 The red_exception Class Hierarchy

redplain::red_exception is a base class for exceptions that are propagated by redFOX. A red_exception records the node on which the exception occurred and this can be read using the node() member function.

Instances of the template redplain::red_exception_with<T> inherit from redplain::red_exception and contain a T member variable that can be read using the value() member function.

8.2 Pre-defined Exceptions

The table below lists the instantiations of red_exception_with<T> that are included in the redFOX llibrary.

Exception Class	Usage
bool, signed char, unsigned char, signed short, unsigned short, signed int, unsigned int, signed long, unsigned long, signed long long, unsigned long long	Encapsulations of Plain Old Data types. Pointers are not supported.
std::exception, std::runtime_error, std::logic_error, std::domain_error, std::length_error, std::ios_base::failure, std::range_error, std::underflow_error, std::overflow_error, std::invalid_argument, std::out_of_range, std::bad_exception, std::bad_typeid, std::bad_cast, std::bad_alloc	Encapsulations of exceptions defined in the Standard C++ Library. May be thrown by application code, Standard C++ Library or redFOX.
redplain::unknown_exception	An application exception was thrown that is not an instance of red_exception_with and that is not in this table.
redplain::communication_exception	Communication with another node has failed or a node that is running a remote invocation cannot be contacted.
redplain::no_method	An attempt has been made to run a method on a node that doesn't implement that method. Name() member function returns the name of the affected method.
redplain::invalid_dref	An attempt has been made to invoke an object with an invalid dref.
redplain::invalid_object	An attempt has been made to invoke an object in an invalid state (e.g. an object that is being destroyed).
redplain::no_thread	No thread is available to run the invocation on the node.
redplain::io_truncated	Message transmission or reception was truncated. io_truncated::value() returns number of bytes missing.

8.3 User-defined Exceptions

Application exceptions can be converted into red_exceptions using the following procedure. Exceptions of class T can be propagated if the following steps are taken.

1. Ensure T is serializable.

Make sure that functions to encode and decode T are provided as described in 4.10

2. Register red_exception_with<T>.

Include the following lines in one C++ compiled file (not a header file) at file scope.

#include <red_exception.h>

//...

REDPLAIN_EXCEPTION_REGISTER(T);

3. Throw red_exception_with<T> instead of T.

Instead of throwing T objects, throw red_exception_with<T>s. If T's are thrown in code that cannot be changed, they may be caught and converted as follows.

try {

// ...

} catch(T t) {

throw redplain::red_exception_with<T>(t);

};

Where multiple catch statements are used for classes that are related by inheritance, ensure that catch statements for derived classes occur before the catch statements for their base classes.

8.4 Exception Inheritance Hierarchies

It is common to organize exceptions into inheritance hierarchies. This allows groups of exception types to be caught. Unfortunately, this strategy is frustrated by converting Ts to red_exception_with<T>s. The difficulty is illustrated as follows.

class T { /* ... */ };

class U : public T { /* ... */ };

//...

try {

//...

} catch (T& t) {

// Catches Ts and Us.

};

One catch statement catches all subclasses of T. If we replace T with red_exception_with<T> and red_exception_with<U>, the inheritance relationship is lost.

class T { /* ... */ };

class U : public T { /* ... */ };

//...

try {

//...

} catch (red_exception_with<T>& t) {

// Catches red_exception_with<T>, but not red_exception_with<U>

};

This difficulty can be overcome if a red_exception_with<T> can be converted back into a T exception. The function red_exception::throw_value() does that. In class red_exception, the function throws a red_exception, but in red_exception_with<T>, it throws a T. So we can recover the inheritance information with nested try…catch statements as follows.

class T { /* ... */ };

class U : public T { /* ... */ };

//...

try {

try {

//...

} catch (red_exception& e) {

// Catches red_exception_with<T>, for all T

e.throw_value();

}

} catch(T t) {

// Catch Ts and Us.

};

The inner catch catches the red_exception and throws a T. The outer catch catches Ts and Us.

8.5 Exception Example

The use of propagated exceptions is best summarised with an example. Consider an application that throws exceptions derived from app_exception. It uses two classes, app_exception and app_derived_exception. To propagate app_exceptions and app_derived_exceptions, we must provide serialization functions, throw red_exception_with<>s and register red_exception_with<>s.

//

// Pre-existing exception classes

//

class app_exception { /* ... */ };

class app_derived_exception : public app_exception { /* ... */ };

//

// Serialization functions

//

template<> void encode<app_exception>(

encoder& e, const app_exception& a) {

/* ... */

}

template<> app_exception decode<app_exception>(

decoder& e, const app_exception& a) {

/* ... */

}

template<> encode<app_derived_exception>(

encoder& e, const app_derived_exception & a) {

/* ... */

}

template<> app_derived_exception decode<app_derived_exception>(

decoder& e, const app_derived_exception & a) {

/* ... */

}

//

// Register red_exception_with<app_exception>

// and red_exception_with<app_derived_exception>

//

REDPLAIN_EXCEPTION_REGISTER(app_exception);

REDPLAIN_EXCEPTION_REGISTER(app_derived_exception);

Then, where app_exceptions or app_derived-exceptions are thrown, we must throw the appropriate red_exception_with<>s instead.

9 Class Reference

The classes and functions accessible in redFOX are documented here.

10 Platform Notes

10.1Windows

redFOX has been developed on Windows XP and Windows 2000 using gcc version 3.xx. It is regularly regression tested on Windows XP SP2. It uses the cygwin environment, but its thread, semaphore and pipe implementations use the Windows API directly. It has been tested, but is not currently supported with the Mingw environment. Support for other compilers such as Visual C++ is anticipated, but not yet scheduled.

10.2Linux

redFOX has been developed on Linux using gcc version 3.xx. It is regularly regression tested on Fedora Core 3 (kernel version 2.6). Threads are implemented using the clone library call. Semaphores are implemented using the sem_* library calls. TCP/IP pipes are implemented using the BSD socket library.

11 Porting Guide

redFOX is designed to run on a variety of processors and Operating Systems. To port it to a new platform, its abstractions of threads, semaphores, communication channels and console must be implemented on that platform. These abstractions are encapsulated in the classes and functions redplain::thread, redplain::semaphore, redplain::ipipe, redplain::opipe, redplain::in() and redplain:out().

11.1Pipes

redplain::ipipe and redplain::opipe are abstract base classes that abstract point to point communication channels. redFOX includes implementations of these abstractions using TCP/IP connections, RS232 ports, shared memory and RTOS communication services. It also includes filters that implement the ipipe or opipe interfaces using other ipipes or opipes, and provide additional functionality such as encryption, authentication and diagnostics.

redFOX uses ipipes and opipes for inter-node communications and for console i/o.

Users with custom console or communications facilities may provide ipipe and opipe interfaces to their facilities.

11.2The Console

redplain::in() is an ipipe that redFOX uses for input. In particular, it reads node configuration information (see 7) from it during initialization. This is implemented by default as the libc standard input, stdin.

redplain:out() is an opipe that redFOX uses for output. It is used by redplain::print(…) and other diagnostics.

12 Diagnostic Aids

Diagnostic aids include the functions redplain::print(…), which prints data of different types on the console if NDEBUG is defined, redplain::puts(), which outputs a string to the console, redplain_assert(bool), which outputs a console message similar to the libc assert macro, and redplain_test(bool) which makes the test regardless of NDEBUG.

The NDEBUG macro is #defined in debug builds of the redfox library, but not in non-debug builds.

13 Limitations and Known Bugs

In this version of redFOX, the following limitations and restrictions apply. Some of these will be removed or relaxed in later versions.

1. Limitation: Base classes of remotely accessible classes must themselves be remotely accessible. Otherwise, base class methods cannot be remotely invoked.

2. Limitation: Base class pointer and reference arguments to dref methods cause only the base class part of the object to be transferred.

3. Limitation: Methods returning pointers or references are not supported. To make remote invocation of such methods behave the same as local invocation would require the referenced object to be copied from the remote node. This creates two problems. First, changes to such an object would not be propagated back to the original. Second, the lifetime of the replicated object is unknown.