SIL LSDev Linux Development

David asked, “I’m curious to know why Firebird has been chosen as a database engine [for FieldWorks] out of the many other RDBMSs out there, such as MySQL.”

I get this question from time to time, especially when I go to our Computer Technical Conference (CTC), which comes around every two years.  It is a fair question, and since CTC is just a month away, I’m taking the opportunity to remember why we did what we did. I had to ask the people that made the decision, because I had a hard time remembering. It’s been a few years.

FieldWorks started out as a Smalltalk project. When the Smalltalk vendor went under (as I have heard), someone mapped the object data to SQL Server. This was SQL Server 7.0, I think, back before the turn of the century. (Literally.) We have since upgraded to SQL Server 2000, then 2005. (We don’t have any plans currently to move to 2008.) The application code was ported to C++, then newer code was done in C#. You’ve seen various entries from Mark and others about porting that.

I have made at least two studies over the years about which RDBMS to port to. I’m not sure MySQL even existed during the first study, but I know it wasn’t mature enough to do what we wanted at the second study. By the time MySQL had matured–how much may still be a matter for debate–we had already had work underway porting to Firebird.

At one time I had recommended we go with PostgreSQL, and we did head in that direction for a while. However, PostgreSQL needed Cygwin at the time, and Firebird already had a native Windows implementation. Also in Firebird’s favor was the fact that it could be embedded into the application, and this has always had a strong pull for some on the team. Also, it was thought that it would be easier to port the SQL that SQL Server uses to Firebird than it would be to port it to PostgreSQL. Finally, we had a volunteer who knew Firebird and was able to give a significant amount of time. We are in fact indebted to him for laying the foundation of the database port to Firebird.

So the team dropped PostrgeSQL in favor of Firebird.

We have looked at object-oriented datababses as well. None were open source during most of time we made our decisions. We have looked db4objects, but have decided to continue with Firebird for now. Perhaps the biggest reason is that the jump to another SQL RDBMS has already shown to be difficult. Jumping to an object-oriented database would be compound the difficulty.

We have tens of thousands of lines of SQL code for FieldWorks now. Most of it is now ported to Firebird. We still need to put it through unit tests and regression testing.

Since we’re porting from SQL Server to Firebird, code that works on both RDBMSs is highly valuable. We want to connect to either using C#.

Microsoft’s ADO.NET Entity Framework does just that. One of the components is LINQ. Here is a beginner’s application to show how to do it with Firebird.

Download and run the latest Firebird Data Provider for .NET and Mono.

Open Visual Studio 2008 (VS). You can try using 2005, but I couldn’t get it to work. Create a new project. (File/New/Project…) Under “Project types” select Windows. Under Templates select Console Application. Give it a name. (I used LinqToFirebird.) Give the project a location. I prefer to leave the checkbox “Create directory for solution” unchecked, to avoid a deeper directory tree than I want.

In VS, go to the Solution Explorer. We’re going to add a reference to the Firebird .NET Data Provider. Right-click on References. A dialog will appear. Select the Browse tab. Go to the directory where you installed the provider and select FirebirdSqlData.FirebirdClient.dll

Now we get to code. In Program.cs, add the following above namespace at the top:

using System.Data;
using FirebirdSql.Data.FirebirdClient;

In the Main method, use the following code. You will have to change the connection string to your own, of course. (The best site for database connection strings is ConnectionStrings.com.) Microsoft has a number of examples for LINQ.

FbConnection con = new FbConnection(
“ServerType=0;User=SYSDBA;Password=masterkey;Dialect=3;” +
“Database=C:\\Firebird\\Data\\FieldWorks\\FW01.FDB;”);

FbDataAdapter fbadapt = new FbDataAdapter(
“SELECT * FROM RDB$CHARACTER_SETS;”, con);

// Table mappings
fbadapt.TableMappings.Add(”Table0″, “CharSets”);
DataSet dataset = new DataSet();

// Fill the dataset with a call to the database
fbadapt.Fill(dataset);

// Query
DataTable CharSets = dataset.Tables[0];
var queryCharSets =
from rCharSets in CharSets.AsEnumerable()
select rCharSets;

Console.WriteLine(”RDB$CHARACTER_SETS:”);
foreach (DataRow rCharSets in queryCharSets)
{
Console.WriteLine(rCharSets.Field(”RDB$CHARACTER_SET_NAME”));
}

// pause
Console.ReadLine();

CloseConnection(con);

The libcom COM Support Library version 0.4.0 has been released. This is the first release of libcom. libcom is licensed under the LGPL.

libcom implements a subset of Microsoft COM (Component Object Model) and supports both C++ and C# (via Mono) COM clients and servers on Linux. libcom is similar to ole32.dll in Windows.

libcom can be downloaded from http://linux.lsdev.sil.org/wiki/index.php/Downloads.

Update: libcom now has its own web page at http://linux.lsdev.sil.org/wiki/index.php/Libcom.

Recently we had a couple developers from other offices join us for LSDev Linux Brainstorming Week to learn from one another, talk about technology choices, and do some pair programming.

Recognizing that parts of our C++ code need to be able to access and work with C# objects through COM (Component Object Model), Eberhard and I set out to test this in Mono on Linux.

We were successful in making a C# application create a regular C# object and a C++ object (via libcom and RCW), pass the C# object to the C++ object, have the C++ object modify the C# object from native code, and then back in C# code observe that our C# object was indeed manipulated successfully from C++.

This capability to call back and forth between C# and C++ using COM is very important for porting several SIL programs to Linux, and we were concerned about whether it would even be possible with Mono. We found out several months ago that C# calling into C++ using libcom and Mono RCWs worked, and I was pleased to find that going the other direction, C++ calling C#, elegantly Just Works.

This new test works thanks to Jonathan Chambers’s implementation of COM Callable Wrappers (CCWs) in Mono, which was announced here.

Eberhard and I largely reused existing libcom test code and just made some additions. The steps to perform this test and an explanation of what is happening can be found on our wiki here.

There are a number of SIL projects which would benefit from an organized way of being packaged for multiple Linux distributions. In addition to supporting several distributions, we would like to support the current and previous distribution releases as well as both 32- and 64-bit architectures.

The way we have chosen to do this is to host each kind of system as a vserver guest all on one physical machine. VServer is more like an advanced chroot than a virtual machine, and is faster and closer to the hardware than other kinds of virtualization. A script could build packages for each platform by invoking each guest’s package-building process.
Developers could also log in to experience and test their software on any of the supported platforms without having to set up such a machine themselves.

Hydra, our multi-headed monster, named for its many guests on one base machine, is being set up to host these guests.

For more information, see the wiki page for Packaging and testing.

We’ve just received our own OLPC hardware, through the beta programme. It is an XO B2. The hardware is really nifty, although surprisingly heavy. The software is still a little rough in places, but continues to improve at an amazing rate.

We’ll be experimenting with it in all kinds of ways, but the real reason for having it is so that we can try porting various pieces of our software to it and see how well they work on it. In particular, we’d like to see how Graphite performs there.

Most of this comes from the Mono project documentation roadmap available on the Mono site here.

Microsoft has published three versions of the .NET Framework.
1.0 came first, next was 1.1 which incrementally updated the previous version, then 2.0 was a considerable expansion from 1.1.

To the core of .NET an addition, called .NET 3.0 has been released. It is a set of new API’s that run on top of the .NET 2.0 installation. The Mono project has been keeping track of the improvements made in each release, some of the implementations that they have kept up with are:

• Core: mscorlib, System, System.Security and System.XML assemblies.
• ADO.NET: System.Data and various other database providers.
• ASP.NET: WebForms and Web Services are supported. Work on WSE1/WSE2 has also started.
• Compilers: C#, VB.NET and various command line tools that are part of the SDK.
• Open Source, Unix and Gnome specific libraries.

Note: The description of the releases is not complete. I am showing the parts of .NET that have been implemented in each release and some other classes. for a complete list they are available on the Mono web site. Read the rest of this post…

The core of SIL FieldWorks was written in C++. New development in FieldWorks is done in C#, and ties into the C++ core through COM.

After much looking all over the Internet in January 2005, I basically concluded that not only does no one use COM in Linux, but no one wanted to (outside of big enterprise application ports to UNIX).

When both porting FieldWorks to Linux and making it cross-platform, we lose access to Microsoft’s COM implementation in Windows. We were originally going to use SWIG to generate C# wrappers around the C++ core libraries. All C++ methods would then be P/Invoked, and COM would be bypassed.

Then I found that Jonathan Chambers had put a lot of good work into Mono’s COM support. I was able to have a C# COM client access a C++ COM server in Windows using Mono.

Read the rest of this post…

Someone just asked me,

What’s the best way to do a cross-platform product that has the native XXX look and feel?

It depends on exactly what you mean by native look and feel. There are several GUI toolkits that produce the right pixels, so that in a screen shot it would look just like an XXX application, but which still don’t produce an application that feels right. For example, on Mac, if you select a range of text and then press right-arrow, what happens? The insertion point should move to the right boundary of the previous selection. However, on Windows, it typically moves one character beyond that point. I find this inconsistency very frustrating in several of the cross-platform apps that I use regularly.

Also, applications may have the right pixels for all the individual controls, but the overall layout of their UI doesn’t feel comfortable for an XXX user. The classic example is the relative ordering of OK and Cancel buttons in dialogs. Most toolkits now take care of this transparently, but there are other things, such as the extent to which toolbars are used, that can be uncomfortable for users. For example, Mac applications typically use toolbars very sparingly, and Windows apps use them extensively.

Read the rest of this post…

Today I read the Mono 1.2 release notes. A few things stuck out to me:

• full implementation of 1.0 and 2.0 APIs
• full SWF 1.1 API
• MONO_IOMAP setting may help with some potential crossplatform issues
• Bundles. If our COM implementation ends up being rather Frankenstein (or really cutting-edge :)), Bundles would let us distribute our own custom Mono along with the application. I don’t really see this as a desirable solution, but it’s nice to know it’s there.
• “Support for the registry is now provided on Linux and Windows.”