.NET/Link Example: BZip2 Compression

This example demonstrates calling an "unmanaged" Windows DLL (that is, an old C-style DLL, not a .NET assembly) from Mathematica using DefineDLLFunction. The DLL we use is libbz2-1.0.0.dll, which performs compression and decompression using the bzip2 algorithm. The DLL is bundled with this example file, so you can run the program immediately. The code for the DLL is open source, and the home page, which includes documentation, can be found at http://sources.redhat.com/bzip2/. 

We define two Mathematica functions, Compress and Uncompress, that write and read compressed data.

To try this example, evaluate all the initialization cells (you can do this with the menu command Kernel/Evaluation/Evaluate Initialization). Then go to the Example section.

Code and Discussion

Needs["NETLink`"];

If the DLL was placed in a Mathematica application's library directory (for example, <Mathematica dir>\AddOns\Applications\SomeApp\Libraries\Windows), or anywhere on the system's PATH, then we could specify just the name of the dll, libbz2-1.0.0.dll, and it would automatically be found. Because this DLL is in a non-standard location, however, we must supply the full path, which we obtain by noting that the DLL is in the same directory as the current notebook:

fullDllPath = ToFileName[First["FileName" /. NotebookInformation[InputNotebook[]]], "libbz2-1.0.0.dll"];

Now we use DefineDLLFunction to create Mathematica functions whose action is to call into the desired functions from the DLL. We need functions for opening and closing files, reading and writing compressed data, and checking the error state.

bzopen

Here is the C-language prototype for the BZ2_bzopen function:

BZFILE * BZ2_bzopen  ( const char *path, const char *mode );

Now define the Mathematica function that calls this DLL function. Note that pointers to arbitrary structures like BZFILE* are mapped to IntPtr, the .NET class that represents a generic pointer. DefineDLLFunction will accept syntax for type names in C, VB, or C# notation. Typically you are looking at a C-language prototype for a DLL function, so that is the most natural form to use:

bzopen = DefineDLLFunction["BZ2_bzopen", fullDllPath, "IntPtr", {"const char*", "const char*"}];

There are many alternative ways to define this function. Here are some examples:

(* Using const is unnecessary: *)
DefineDLLFunction["BZ2_bzopen", fullDllPath, "IntPtr", {"char*", "char*"}];
(* This uses C# syntax for string args: *)
DefineDLLFunction["BZ2_bzopen", fullDllPath, "IntPtr", {"string", "string"}];
(* This uses VB syntax for string args: *)
DefineDLLFunction["BZ2_bzopen", fullDllPath, "IntPtr", {"ByVal path As String", "ByVal mode As String"}];

bzwrite

Here is the C-language prototype for the BZ2_bzwrite function:

int BZ2_bzwrite ( BZFILE* b, void* buf, int len );

Once again we use the generic .NET IntPtr type for the BZFILE* argument. The void* buf argument is the address of a block of memory containing the data to compress and write out to the file. You could think of it as a byte* or, even better, as a byte[]. .NET/Link lets you pass a list to a pointer or array argument, and we want to pass a list of bytes from Mathematica for this argument, so it is clearest to declare it as a byte[]. Note that we could also use char[] to mean the same thing. .NET/Link lets you use C or C# syntax for these types, but it gives preference to C syntax, so even though char in C and C# are different sizes, .NET/Link assumes you mean the C char type (a single byte).

bzwrite = DefineDLLFunction["BZ2_bzwrite", fullDllPath, "int", {"IntPtr", "byte[]", "int"}];

bzread

Here is the C-language prototype for the BZ2_bzread function:

int BZ2_bzread  ( BZFILE* b, void* buf, int len );

Compare this to the definition for bzwrite. The void* buf argument represents the address of a block of memory into which data will be read. You can specify void* as the type and it will be automatically mapped to IntPtr, but for our uses we want it to be typed specifically as a byte* or byte[] (either works fine).

bzread = DefineDLLFunction["BZ2_bzread", fullDllPath, "int", {"IntPtr", "byte[]", "int"}];

bzclose

Here is the C-language prototype for the BZ2_bzclose function:

void BZ2_bzclose ( BZFILE* b );

bzclose = DefineDLLFunction["BZ2_bzclose", fullDllPath, "void", {"IntPtr"}];

bzerror

Here is the C-language prototype for the BZ2_bzerror function:

char* BZ2_bzerror ( BZFILE* b, int* errCode );

The last argument is an int* that gets filled in with an error code by the function. We could declare this as an int*, but there is a better way, for the following reasons. .NET/Link lets you pass a literal integer, like 42, for an int* parameter, but that is only suitable when the argument is "in" only, because there is no way of communicating back out any changes made to the integer. To solve this, you can pass a symbol like x to an int* slot, and x will be assigned the value of the integer written into the int* address by the function call. But .NET/Link cannot know that this is effectively an out-only parameter, so .NET/Link requires that a legitimate integer value be passed in. This means that even though the value of the integer is not inspected by the function, you need to initialize the value by setting x to, say, 42 before passing x to the function. A slicker trick is to tell .NET/Link that this is an out-only parameter by declaring it in C#-style syntax as out int. Note that the pointer asterisk is gone from this--an out int, like a ref int or ByRef As Integer (in VB syntax), is effectively already the address of an int. When we declare the parameter as out int, .NET/Link knows to ignore its value going in to the function, but propagate a value back out. This means that we can paass in an uninitialized symbol like x to the function and then inspect the value of x after the call to see what integer value it was assigned during the call.

bzerror = DefineDLLFunction["BZ2_bzerror", fullDllPath, "char*", {"IntPtr", "out int"}];

The Compress function

Compress::fopen = "Failed to open file `1` for writing.";
Compress::bzerr = "An error was reported by a bzip2 library routine. The error code was `1` and the message was `2`.";

The main argument for Compress is a Mathematica string, but we want to pass the data into the DLL as an array (List) of bytes, so we convert strings to byte arrays. Mathematica strings consist of 2-byte Unicode characters, so to properly handle non-ASCII characters like α we have to either convert every character to 2 bytes before compression (not a very good idea to double the size of the data before starting compression!) or encode in an efficient format like UTF8 that lets ASCII characters remain as single bytes.

Compress[file_String, data_String] := Compress[file, ToCharacterCode[data, "UTF8"]]

Compress[file_String, data:{___Integer}] :=
	NETBlock[
		Module[{bzPtr, numBytesWritten, errString, errCode},
			bzPtr = bzopen[file, "w"];
			(* bzPtr is a pointer (an instance of the IntPtr class). Here we test to see if it was NULL: *)
			If[bzPtr@ToInt32[] == 0,
				Message[Compress::fopen, file];
				Return[$Failed]
			];
			numBytesWritten = bzwrite[bzPtr, data, Length[data]];
			If[numBytesWritten == 0,
				errString = bzerror[bzPtr, errCode];
				Message[Compress::bzerr, errCode, errString];
				numBytesWritten = $Failed
			];
			bzclose[bzPtr];
			numBytesWritten
		]
	]

The Uncompress function

Uncompress::fopen = "Failed to open file `1` for reading.";
Uncompress::bzerr = "An error was reported by a bzip2 library routine. The error code was `1` and the message was `2`.";

The Uncompress function is slightly trickier. There are two ways to do it, and we will examine each in turn. The two techniques differ in how they handle the second argument to BZ2_bzread, which is the address of a buffer into which the function will write uncompressed data.

The first technique is the simplest. It types the second argument to BZ2_bzread as a byte[] (this was done earlier in the definition for bzread) and makes use of .NET's default marshaling for arays. When a managed array of bytes is sent to a DLL function, it is marshaled as a a native C-style array of bytes, which is just a contiguous block of memory, exactly what the BZ2_bzread function wants. When the DLL call returns, .NET marshals the data from the block of memory back into the managed array. Therefore we can allocate an array of bytes in .NET, pass this object as the second argument to bzread, and then when the function returns we can simply read the contents of that array to get the compressed data.

Uncompress[file_String] :=
	NETBlock[
		Module[{bzPtr, bufPtr, cnt, result, managedArray, errString, errCode},
			bzPtr = bzopen[file, "r"];
			If[bzPtr@ToInt32[] == 0,
				Message[Uncompress::fopen, file];
				Return[$Failed]
			];
			(* Here we allocate a .NET array of 20000 bytes. The actual class name of the byte type
			   in .NET is System.Byte, so a 1-dimensional array of these is called System.Byte[].
			   NETNew always returns an object reference, so we get back the object, not a list of
			   20000 zeros.
			*)
			managedArray = NETNew["System.Byte[]", 20000];
			result = "";
			While[(cnt = bzread[bzPtr, managedArray, 20000]) > 0,
				(* NETObjectToExpression takes the array object and forces it to be returned to
				   Mathematica as a list of numbers instead of an object reference. We encoded
				   the data using the UTF8 format in the Compress function, so we must use that
				   format when we decode from an array of bytes back into a Mathematica string.
				*)
				result = result <> FromCharacterCode[Take[NETObjectToExpression[managedArray], cnt], "UTF8"]
			];
			If[cnt == -1,
				(* In this call, errCode is an "out int" parameter. It doesn't matter what value it
				   has going in, and it will be assigned an integer value during the call.
				*)
				errString = bzerror[bzPtr, errCode];
				Message[Uncompress::bzerr, errCode, errString];
				result = $Failed
			];
			bzclose[bzPtr];
			result
		]
	]

Here is the second tehnique for dealing with the block of memory argument that BZ2_bzread requires. This time we make this argument an IntPtr that holds the address of a block of memory. We must manually allocate and later deallocate this block of memory. This technique is more complicated, but it is presented because it demonstrates some useful techniques for dealing directly with memory and pointers in Mathematica code. Several useful methods are found in the System.Runtime.InteropServices.Marshal class. We use the Marshal.AllocHGlobal() function to allocate a block of memory and Marshal.FreeHGlobal() to free it. These names will be familiar to Windows programmers as simple wrappers for the Windows API functions GlobalAlloc and GlobalFree. Marshal.AllocHGlobal() returns an IntPtr that points to the allocated memory. This IntPtr is passed to bzread which writes into it. The final step is to copy the data in that raw memory block into a managed array (that is, a real .NET array) that can be returned to Mathematica as a list.

(* Here is a new definition for calling the BZ2_bzread DLL function. The second argument is different
   from the one used in the bzread definition.
*)
bzread2 = DefineDLLFunction["BZ2_bzread", fullDllPath, "int", {"IntPtr", "IntPtr", "int"}];

Uncompress2[file_String] :=
	NETBlock[
		Module[{bzPtr, bufPtr, cnt, result, managedArray, errString, errCode},
			LoadNETType["System.Runtime.InteropServices.Marshal"];
			bzPtr = bzopen[file, "r"];
			(* Test if the IntPtr was NULL. *)
			If[bzPtr@ToInt32[] == 0,
				Message[Uncompress::fopen, file];
				Return[$Failed]
			];
			(* Allocate a block of memory 20000 bytes long and get an IntPtr to it. This IntPtr is what
			   we will pass in to bzread2.
			*)
			bufPtr = Marshal`AllocHGlobal[20000];
			(* To return the data to Mathematica, we need to copy the raw byte data from memory into
			   a managed array. This array is used for that purpose.
			*)
			managedArray = NETNew["System.Byte[]", 20000];
			result = "";
			While[(cnt = bzread2[bzPtr, bufPtr, 20000]) > 0,
				(* Here we copy the data from the block of memory into the managed array. *)
				Marshal`Copy[bufPtr, managedArray, 0, cnt];
				(* See the notes for this line in the original Uncompress function above. *)
				result = result <> FromCharacterCode[Take[NETObjectToExpression[managedArray], cnt], "UTF8"]
			];
			(* Deallocate the block we allocated earlier. *)
			Marshal`FreeHGlobal[bufPtr];
			If[cnt == -1,
				errString = bzerror[bzPtr, errCode];
				Message[Uncompress::bzerr, errCode, errString];
				result = $Failed
			];
			bzclose[bzPtr];
			result
		]
	]

Example

Let's compress and uncompress a file from the Mathematica distribution.

strm = OpenRead[ToFileName[{$TopDirectory, "Documentation", "English", "MainBook"}, "Contents.nb"], DOSTextFormat->False];
origFileData = Import[strm, "Text"];
Close[strm];

StringLength[origFileData]

Now we compress that data to a temporary file. Calling Close[OpenTemporary[]] is just a trick for generating an unused filename in the temporary directory.

tempFileName = Close[OpenTemporary[]];
Compress[tempFileName, origFileData]

Check the size of the compressed version:

ByteCount /. FileInformation[tempFileName]

That's about 10:1 compression! Now check that we can uncompress and get the original data back:

Uncompress[tempFileName] == origFileData