The Python xstruct module

Introduction

The xstruct module is an extension of the standard Python struct module that provides a more convenient way of creating and manipulating packed binary data structures. Besides offering the pack, unpack and calcsize functions inherited from the struct module, the xstruct module offers a new structdef function that you can use to define packed binary data structures. From these structure definitions, you can create the actual structure objects (see demonstration below).

Downloading

You can download the following: For questions and remarks, please drop me (Robin Boerdijk) an e-mail at

boerdijk@my-deja.com.

A demonstration

For demonstration purposes, let's say you want to implement a simple client/server communication protocol with a binary message format defined by the following C struct: 

/* XDSP (XStruct Demonstration Protocol) message format */

typedef unsigned char octet;

typedef struct {
  char magic[4];           /* must be "XSDP" */
  octet version[2];        /* major version, minor version */
  octet byte_order;        /* 0 = big endian, 1 = little endian */
  octet message_type;      /* 0 = request, 1 = reply */
  unsigned long correl_id; /* correlation id, links replies to requests */
  octet data[16];          /* request or reply data */
} XsdpMessage;

Defining the structure

You can use the structdef function from the xstruct module to define the XSDP message format in Python: 

Python 1.5.2 (#0, Apr 13 1999, 10:51:12) [MSC 32 bit (Intel)] on win32
Copyright 1991-1995 Stichting Mathematisch Centrum, Amsterdam

>>> import xstruct

>>> XsdpMessage = xstruct.structdef(xstruct.big_endian, [
  ("magic",        (xstruct.string, 4),   "XSDP",   xstruct.readonly),
  ("version",      (xstruct.octet,  2),   (1, 0)), 
  ("byte_order",   (xstruct.octet,  1),    0,       xstruct.readonly), 
  ("message_type", (xstruct.octet,  1)), 
  ("correl_id",    (xstruct.unsigned_long, 1)),
  ("data",         (xstruct.string, 16))
])

The structdef function takes two parameters. The first parameter specifies the physical layout (i.e, byte order and alignment) of the structure in memory. The second parameter is a list of tuples that defines the fields of the structure. A single field tuple has the following format:

( field name, ( field type, repeat count ) , initial value , flags )

where initial value and flags are optional.

The structdef function returns a structdef object: 

>>> XsdpMessage
<structdef object at 79ad60>

The structdef object can be used to create actual structure objects.

Creating and manipulating structure objects

To create an actual XSDP message structure object, you call the XSDP message structure definition object: 

>>> msg = XsdpMessage()

To see fields and their values, evaluate or print the structure object: 

>>> print msg
magic: XSDP
version: (1, 0)
byte_order: 0
message_type: 0
correl_id: 0L
data:

Individual fields can be accessed as attributes of the structure object: 

>>> msg.correl_id = 0x01020304
>>> msg.correl_id
16909060L

Alternatively, you can use the mapping interface of the structure object: 

>>> msg['data'] = "Hello, World !"
>>> msg['data']
'Hello, World !\000\000'

If you try to change the value of a field that has been marked as readonly in the structure definition, an exception will be raised: 

>>> msg.magic = "XXXX"
Traceback (innermost last):
  File "<stdin>", line 1, in ?
xstruct.error: field is not changeable

Accessing the packed binary format

The purpose of the xstruct module is to enable you to conveniently create and manipulate packed binary data structures. Therefore, sooner or later, you probably want to get hold of the physical byte stream. One way to do this is to make a copy of the internal data buffer of the structure object using the Python str operator: 

>>> buf = str(msg)
>>> buf
'XSDP\001\000\000\000\000\001\002\003\004Hello, World !\000\000'

A more efficient way to manipulate the internal data buffer is to use the new buffer interface of Python 1.5.2. For this to work, however, the client code has to use this buffer interface as well. The standard Python file object already does this, as you can see from: 

>>> open("tmp", "w").write(msg)
>>> open("tmp", "r").read()
'XSDP\001\000\000\000\000\001\002\003\004Hello, World !\000\000'

Using this method, the packed binary data structure is written directly to the file, without the need of making a string copy first.

From a packed binary format, you can also create a structure object: 

>>> msg2 = XsdpMessage(buf)
>>> msg2
magic: XSDP
version: (1, 0)
byte_order: 0
message_type: 0
correl_id: 16909060L
data: Hello, World !

Or using the buffer interface: 

>>> msg3 = XsdpMessage()
>>> open("tmp", "r").readinto(msg3)
28
>>> msg3
magic: XSDP
version: (1, 0)
byte_order: 0
message_type: 0
correl_id: 16909060L
data: Hello, World !