Today was a milestone. We demonstrated the Linux version of FieldWorks TE to a group of people at the office chapel time. It worked well enough for the demo, in that it wasn’t slow and didn’t crash, and the people were impressed

Progress has been very rapid in the past month or two, for several reasons, but today was the first time that we felt TE was ready to show to others. It was helpful to have the extra incentive of a scheduled demo for the final push to get things working on a non-developer machine.

We plan to make a binary version available for the Windows TE developers at Jaars to look at, and the work we did for the demo will help with this. Soon, they’ll also be able to build copies from source for themselves.

Some of the reasons for the rapid progress are:

1. All the hard work that went into porting the infrastructure over the past two years.
2. A decision to work with SWF for the UI for the time being, even though it has some shortcomings.
3. The use of a Windows-hosted database across a network connection.

Some of the big things that are still missing are:

1. A local database.
2. Proper text rendering using Graphite and Pango.
3. A real side-bar implementation (awaiting development of a DotNetBar replacement).

This is still pre-alpha software, and depends on a pre-release version of Mono built from source, but we feel we have come a long way and that it’s time to begin reviewing our progress with other FieldWorks developers.

Links to screen shots:

I get this question from time to time, especially when I go to our Computer Technical Conference (CTC), which comes around every two years.  It is a fair question, and since CTC is just a month away, I’m taking the opportunity to remember why we did what we did. I had to ask the people that made the decision, because I had a hard time remembering. It’s been a few years.

FieldWorks started out as a Smalltalk project. When the Smalltalk vendor went under (as I have heard), someone mapped the object data to SQL Server. This was SQL Server 7.0, I think, back before the turn of the century. (Literally.) We have since upgraded to SQL Server 2000, then 2005. (We don’t have any plans currently to move to 2008.) The application code was ported to C++, then newer code was done in C#. You’ve seen various entries from Mark and others about porting that.

I have made at least two studies over the years about which RDBMS to port to. I’m not sure MySQL even existed during the first study, but I know it wasn’t mature enough to do what we wanted at the second study. By the time MySQL had matured–how much may still be a matter for debate–we had already had work underway porting to Firebird.

At one time I had recommended we go with PostgreSQL, and we did head in that direction for a while. However, PostgreSQL needed Cygwin at the time, and Firebird already had a native Windows implementation. Also in Firebird’s favor was the fact that it could be embedded into the application, and this has always had a strong pull for some on the team. Also, it was thought that it would be easier to port the SQL that SQL Server uses to Firebird than it would be to port it to PostgreSQL. Finally, we had a volunteer who knew Firebird and was able to give a significant amount of time. We are in fact indebted to him for laying the foundation of the database port to Firebird.

So the team dropped PostrgeSQL in favor of Firebird.

We have looked at object-oriented datababses as well. None were open source during most of time we made our decisions. We have looked db4objects, but have decided to continue with Firebird for now. Perhaps the biggest reason is that the jump to another SQL RDBMS has already shown to be difficult. Jumping to an object-oriented database would be compound the difficulty.

We have tens of thousands of lines of SQL code for FieldWorks now. Most of it is now ported to Firebird. We still need to put it through unit tests and regression testing.

We’ve ported the core C++ parts which are the foundation of all the FieldWorks applications. A big part of the remaining work is in C# to get Translation Editor and Language Explorer to build and run in Linux.

But tackling Translation Editor would be quite a big step and open up a lot of difficulties all at once. So first we’re working to port WorldPad. Although WorldPad won’t be as widely used as Translation Editor, our work on WorldPad is a stepping stone on the way to completing the Translation Editor because many pieces of WorldPad are shared by Translation Editor. And completing the Translation Editor will be easier if we first work on WorldPad.

Because WorldPad is older than Translation Editor, we’ve first changed some of the underlying technology to put us in a better position to transfer our work to Translation Editor. WorldPad in Windows is a C++ application wrapping views, a library in the FieldWorks code that allows fast display and editing of writing systems with complex scripts and other unique features required by minority language writing systems.

We’re writing another WorldPad, WorldPadGTK, using C# and embedded the view in a GTK window. WorldPad in Linux will be a C# GTK application wrapping views.

Since we’re now using areas of the code that have had a lot of changes during that time, there was an increasing need to update our local VCS to reflect the latest from Dallas.

First I migrated our VCS from CVS to SVN - a welcome change. (I can rename directories now!)

Then we merged in the new code from Dallas, resolved the conflicting files (only 93), and made things still compile in Linux.

After having approached this with apprehension, I was thankful that it was easier than I had anticipated and only took two and a half weeks.

And now we’re up to date!

Microsoft’s ADO.NET Entity Framework does just that. One of the components is LINQ. Here is a beginner’s application to show how to do it with Firebird.

Download and run the latest Firebird Data Provider for .NET and Mono.

Open Visual Studio 2008 (VS). You can try using 2005, but I couldn’t get it to work. Create a new project. (File/New/Project…) Under “Project types” select Windows. Under Templates select Console Application. Give it a name. (I used LinqToFirebird.) Give the project a location. I prefer to leave the checkbox “Create directory for solution” unchecked, to avoid a deeper directory tree than I want.

In VS, go to the Solution Explorer. We’re going to add a reference to the Firebird .NET Data Provider. Right-click on References. A dialog will appear. Select the Browse tab. Go to the directory where you installed the provider and select FirebirdSqlData.FirebirdClient.dll

Now we get to code. In Program.cs, add the following above namespace at the top:

using System.Data;
using FirebirdSql.Data.FirebirdClient;

In the Main method, use the following code. You will have to change the connection string to your own, of course. (The best site for database connection strings is ConnectionStrings.com.) Microsoft has a number of examples for LINQ.

FbConnection con = new FbConnection(
“ServerType=0;User=SYSDBA;Password=masterkey;Dialect=3;” +
“Database=C:\\Firebird\\Data\\FieldWorks\\FW01.FDB;”);

FbDataAdapter fbadapt = new FbDataAdapter(
“SELECT * FROM RDB$CHARACTER_SETS;”, con);

// Table mappings
fbadapt.TableMappings.Add(”Table0″, “CharSets”);
DataSet dataset = new DataSet();

// Fill the dataset with a call to the database
fbadapt.Fill(dataset);

// Query
DataTable CharSets = dataset.Tables[0];
var queryCharSets =
from rCharSets in CharSets.AsEnumerable()
select rCharSets;

Console.WriteLine(”RDB$CHARACTER_SETS:”);
foreach (DataRow rCharSets in queryCharSets)
{
Console.WriteLine(rCharSets.Field(”RDB$CHARACTER_SET_NAME”));
}

// pause
Console.ReadLine();

CloseConnection(con);

HelloView is really just a wrapper for a FieldWorks view containing a simple string. It’s the minimal FW program, although it relies on a significant proportion of the infrastructure. That’s why getting it working represents such a significant milestone. It’s the culmination of many man-months of work porting C++ code, work which had no visible results until now.

In parallel with the C++ porting, we have been working on the framework of a more significant FW application: a GTK and C# version of WorldPad. This uses the same underlying technology as HelloView, so it takes a relatively small additional step to make the same capability work in WorldPad, and thus demonstrate something rather more interesting.

WorldPadGTK will be the subject of another post.

The .Net specification provides a way for managed code (eg C#) to instantiate COM objects (eg C++) and call methods on them. This is very important in FieldWorks because the majority of the application code is C#, but makes use of some very powerful infrastructure code written in C++. The seamless joining of C# and C++ via COM in .Net is a huge advantage.

The Mono developers have thoughtfully provided the same capabilities in the Mono runtime as are available in .Net. This means that, on Windows, Mono works exactly as expected with Microsoft’s COM implementation. Surprisingly, the same capability also exists in the Linux version of Mono, except that there’s no COM implementation there. However, they have arranged things such that if a library exists with the right name, and provides exactly the right functions, then Mono will use that library to do COM on Linux.

This has been used by others with a commercial Windows- to- Unix porting library, but never with an open-source library. We developed libcom to be just such a library, designed to be lightweight, with very few dependencies, and doing just one thing well. Because libcom implements the COM APIs exactly according to spec, and Mono uses them exactly according to the same spec, it turns out that Mono works perfectly with libcom. We are able to create and use COM objects implemented in C++ from C#, and vice versa. But only in small test programs. Until now.

In order for the managed runtime to know what interfaces and methods a given COM object supports, there needs to be a special .Net assembly, called an interop, that defines .Net classes with special attributes that mark them as “stand-ins” for their COM counterparts. The application uses these classes, and the runtime behind the scenes makes all the right magic happen (using Runtime-Callable Wrappers).

These interops are normally generated by a Microsoft tool called TlbImp, which works from a type library (TLB) that is produced from IDL compiled by the MIDL compiler. Unfortunately, many of the interfaces and methods used by FieldWorks are too complicated for MIDL/TLB, so SIL created a custom tool called IDLImp that compiles IDL into C# source that is then compiled with the normal C# compiler (Microsoft or Mono) to produce an interop DLL.

We were fairly sure that the interops that IDLImp was producing were not 100% compatible with Mono. Perhaps Mono was interpreting the .Net spec differently from Microsoft? More strictly, perhaps?

To find out, Tom and Brent took simple interops generated by both TlbImp and IDLImp and compared them. After a lot of detective work, they found that the TlbImp-produced interops had some extra attributes in them, and after adding these attributes by hand to the results produced by IDLImp, they were able to run our problem code successfully, without a crash.

This is a huge breakthrough! We have been roadblocked by this problem since before Christmas, and have had to work on other things pending a solution. We had lots of good things lined up ready to go, as soon as we could solve the COM problem. Well, now we can proceed at last. We still need to tweak IDLImp to produce the extra attributes in the interops, but that should be fairly easy.

Watch this space for further developments!

libcom implements a subset of Microsoft COM (Component Object Model) and supports both C++ and C# (via Mono) COM clients and servers on Linux. libcom is similar to ole32.dll in Windows.

libcom can be downloaded from http://linux.lsdev.sil.org/wiki/index.php/Downloads.

Update: libcom now has its own web page at http://linux.lsdev.sil.org/wiki/index.php/Libcom.