OddThinking

This post describes the internals of my D600 synch application.

I am going to start with a reveal of the how the trick works, for those readers who just want to see how such a miraculous thing could be possible.

The trick

The D600 phone, like many phones, has a serial interface to the PC, which appears as another COM device, just like an old modem.

Just like an old modem, it accepts AT commands (they used to be called “Hayes-compatible” commands, back in the 2400 baud days). For example, you can send the modem/phone the string ATE0 and it turns off Echo (i.e. whether every key you type should be sent back to you, like the old terminal days.)

It normally replies with OK to indicate success.

More complicated commands are possible: AT+ORGR=43 means read event #43 from the Appointment Book. It is returned as a comma-separated set of fields.

I imagine that’s enough information for those “skilled in the art” to see how a synch application is possible. The rest of this article is just filler for the rest of us.

Architecture

The application is written in Python.

The architecture is, to a first order of approximation, based on a number of layers. Each layer hid the layer below it, and offered a higher level of abstraction to the one above.

PySerial Layer

Starting at the bottom, the lowest layer is the PySerial module which takes care of dealing with the serial port. I have used PySerial before, and been burned – 32-byte slabs of read buffer would be occasionally lost – but I didn’t notice any such issues this time.

SerialPhoneConnection

Above that layer is the SerialPhoneConnection.

It has the responsibility of reading and writing bytes/characters to the phone.

It manages a read cache and also takes care of timeouts (by translating them to null strings).

PhoneConnection

Ostensibly, there is a layer above SerialPhoneConnection just called PhoneConnection.

Is it really a layer? It is really just an interface offered by a base class of SerialPhoneConnection.

The concept was that other sub-class phone connections could be created. I had two in mind.

The first was Bluetooth, which I assume appears as another serial connection, somehow.

The second is a file-based system to enabling high-quality unit-testing against a known stream of input. This was never implemented. I get a pat on the head for trying to be good, and a kick up the bum for not actually following through.

Split Layer

The next layers that sit above the PhoneConnection are actually split into two – the parts responsible for reading tokens from the phone, and the parts responsible for writing commands from the phone.

Lexer

The lexer reads the stream of bytes and delimits tokens using a simple context-free grammar^*.

It provides a stream of tokens for the layer above. Tokens are simple objects representing words, quoted strings, commas, whitespace, OK and ERROR responses, and the like.

It supports lookahead, but in a slightly unconventional way – tokens can be read from the stream or “unread” back into the stream, where they are put at the front of the cache for the next read call. (I thought more than one token lookahead might be required, but I was wrong – the responses form an LL(1) language.)

^* For the most part, the grammar is context-free, and can work out what each token is without assistance. There turned out to be one exception – there are some “Pascal strings” used in some commands – the string as a length, following by a space, followed by that many bytes. The lexer needs to be told when to expect such a string. (I suspect file transfer is another situation that requires such a lexer feature.)

Lexer Expect

On top of the lexer (and still on the Read side of the architecture) was the Lexer Expect layer. I really want to rename this layer, but the right term hasn’t hit me yet.

This layer provides a set of operations to the parsing code (part of the Command Protocol layer, see below), which allows the parser’s indicate the token that is expected to be received next.

For example, receiveOK() can be called by the parser to say an OK token is expected. Lex Expect calls the lexer, and if it receives a different token, it raises an exception with a useful description (e.g. “Parsing Error: ‘ERROR’ was received, ‘OK’ expected.”)

Receive methods for tokens representing strings values will return the provided values.

This layer also allows a lookahead (e.g. isNextTokenComma()).

Message Builder

Sitting next to Lexer/Lexer Expect, and on top of Phone Connection is Message Builder. I would like to say that this is another nice clean abstraction layer, like everything else that I have described until now, but this layer didn’t get implemented as cleanly as I would like.

There are many flaws to this layer of abstraction.

Flaw # 1: It isn’t really a layer. Message Builder is just a bunch of utilities that have no state.

The utilities take Python records (more accurately, Python dictionaries, mapping field name to value) and generate D600 commands, ready to be written.

It enforces some very basic type-checking (that mandatory fields are present in the dictionary, that numeric types are in range, that strings are the correct length).

Flaw #2: It has a half-baked, over-engineered and under-implemented solution to dealing with bad values.

Should it assert, because it is a bug in the code? Should it raise an exception, to let the higher levels deal with it? Should it truncate strings/use defaults? Should it try to send the bad value anyway (useful in reverse-engineering the protocol) I gave up trying to come up with an overall solution. I believe it now does any and all of these, based on whims.

Flaw #3: It fails to completely hide the lower layer – in particular the PhoneConnection.write() command is called directly by the layer above. MessageBuilder should probably be refactored into a true layer, called CommandSender.

Flaw #4: It attempts to completely encapsulate the details of the individual AT commands from the higher layers. There is little point in achieving this, as the next layer up is still tightly coupled to the details of the command. Furthermore, there is leakage, as the phone message often contains a “Heard You” string (my terminology), where it repeats back part of the command. So even if the AT commands was completely abstracted, the next layer up will still see part of it. There is a heroic, but ill-fated attempt to make this Heard You string a cookie/blob that the upper layer can detect, without understanding, but the abstraction leakage is so great, this abstraction should be abandoned.

CommandProtocol

CommandProtocol sits on top of the LexerExpect and MessageBuilder layers. (As discussed just above, it has a tentacle that reaches down to the PhoneConnection layer, but that’s not ideal.)

Despite its partly-fragile foundation, it does a reasonable job of abstracting the to-and-fro details of the phone’s protocols from the layers above.

It offers a bunch of commands like getSIMMemoryContact(43), which will construct the appropriate command to send to the phone, send it to the phone, lex the result, and return a dictionary of field-names and values.

It is worth noting that while the protocol is encapsulated, the presence and meanings of the fields is not – it remains a problem for the higher layers to work out what a “specialAvatarTypeId” of 255 means.

This layer has many gaps where the meaning of a particular fields is simply unknown to me.

Phone Modes

The phone has a number of different kinds of mode.

A simple example is the Echo mode – whether text is echoed back to the computer when it is received by the phone.

More complicated examples include whether the Phone Book commands operate on the contacts stored in the SIM or in the the Phone’s memory.

Many of the commands built by the Message Builder rely on being in a particular set of modes.

However, the MessageBuilder has no state, so it doesn’t know whether to change modes. (Flaw #5: this mode state should be incorporated into the suggested CommandSender layer)

Rather than have every command set up all the modes that it requires required, the Phone Modes object tracks and manages the modes of the phone.

The CommandProtocol object can simply issue a request on the PhoneModes layer to ensureEchoOff() and the PhoneModes object will only send the necessary codes if the Echo mode isn’t already known to be off.

There is a interdependency between the CommandProtocol and the PhoneModes object. The more complex commands in CommandProtocol call PhoneModes to ensure the mode is correct. PhoneModes may call some of the simpler commands in CommandProtocol to achieve its goal.

From the perspective of the higher layer, the PhoneModes is an implementation detail of the CommandProtocol, and is not called directly.

Phone DOM (Domain Object Model)

The Phone DOM sits at the top of the re-usable stack, on top of Command Protocol and introduces the first sign of real civilisation.

It has some nice and clean Python classes, representing real concepts like Contacts, Events, Schedule items, To Do items.

Each supports a constructor to make your own, a readFactory to grab them from the phone and a write method to send them down to the phone.

The objects have real attributes that are guaranteed to be there when you want them (unlike dictionaries). Attributes are given meaningful identifiers (recurrence can be “Yearly” rather than 4) and pythonic types (e.g. Datetimes) . Values with similar lifetimes are grouped together (e.g. Alarm settings) and there is even a bit of inheritance here and there.

Civilised!

Type-checking isn’t as extensive as it could be, but at least this layer provides the right place to do it.

There is a collection type offered (in an attempt to abstract some of the complexities of location identifiers), and methods to read in or write out (to the phone) an entire table at a time.

This layer has many gaps where I haven’t got around to providing interfaces – e.g. there is no SMS Message class, despite support at the lower layers. There is no File class (which isn’t supported at the lower layers).

Synch Application

As mentioned before, all the layers below the application layer have deliberately been kept clean of the influences of Outlook (which is, after all, only one PIM). While there are large gaps where I am not interested in a particular piece of functionality, the code that is present should be generically useful – and not tied to any odd syching predilections I may have.

That leaves the application layer to hold all the weird stuff!

The basic algorithm is:


Fetch all the contacts from the phone.
Fetch all the contacts from Outlook.
For each Outlook contact:
    Look for an equivalent one in the phone list.
    If present, remove it from the phone list.
    If absent, add it to the To Be Sent list.
Delete all the remaining items in the phone list from the phone.
Add all the items in the To Be Sent list.

Repeat, but with events, instead of contacts. (Events include tasks.)


There are some slight complications – for example, I get a list of Birthday events from Outlook at the same time as the Contacts – but this is basically the algorithm.

Why the Equivalence Check?

I could have started by simply deleting all the items from the phone, and not bothered with the equivalence check.

However, to delete, you must know the ids of the items. To know the ids of the items you must read them all in first.

Once they have been read it, it is faster to eliminate the ones that don’t need deleting and re-sending than it is to simply delete and resend them.

It also reduces the time window of the phone having bad data on it, and the amount of badness to the data, which is important if you want to grab the phone mid-synch.

How are Outlook Calendar Items mapped to D600 events?

After bitterly complaining about how poorly recurring items are mapped to D600 events by the Samsung software, I knew I had to do a better job.

I looked back at the D600 Event fields chart I produced some time ago. I realised I wasn’t interested in producing To Do events (because of the poor UI, described previously here and here). I also realised that there was no point in producing Miscellaneous events, because Schedule events could do everything an Miscellaneous event could do.

That left Anniversary and Schedule events (ignoring recurrence options for the moment) the big difference between the two is whether the event is an “All Day Event” (in Outlook terms).

So the mapping to types was much easier than I anticipated – All Day Events mapped to anniversaries, other events to Schedule.

Then, I started work on understanding the Outlook object model for Recurring events and their exceptions. How could I work out which events would be happening in a given window?

A flash of inspiration had me madly searching the MSDN web-site and then dancing around the keyboard with joy. Rather than trying internalise the rules in my code, I could simply ask Outlook to do it for me. It handles a simple query language (via a COM interface). I asked it to give me all the events that occurred in a given date range, sorted by Start date. Then I walked through it, and created one Phone event per occurrence.

This isn’t ideal. For example, if there is an non-All-Day event, that occurs once a week, without any exceptions, it would be better to record it as a single recurring schedule event in the D600, rather than dozens of single-shot events. Perhaps one day I will add such refinements.

However, in one fell swoop and about 14 lines of code, I implemented a much more practical solution than that offered by PC Studio.

Date Window

There is limited space for Event items on the phone (400).

To deal with this, I implemented a date window, which is supported by Mobile Master, but I haven’t seen it in PC Studio.

The idea is to only store events from the short-term past and on into the future as far as you can get. For me, that means any event that finished more than 6 weeks ago can be deleted from the phone.

On the other hand, if you have less appointments, the synch software can get bogged down sending events far into the future. I put a 2 year limit on the events that shold be sent to the phone.The latest (non-recurring) event sent to the phone is a warning that the calendar is inaccurate after this point.

I don’t fill the phone completely, but leave some space for manually added events as well.

Summary

Once you know the trick, and have worked out what the commands are, the implementation is fairly straight-forward. (Then again, I have written similar hardware drivers for modems, electronic wallboards, digital cameras and marine equipment in my past, so maybe I am familar with the concepts.

The layered model generally worked well, and my biggest mess occurred at exactly the point that I strayed from it.

Each layer has noticeably less coverage of the phone features than the layer below; by the time you get to the PhoneDOM, I am only handling two types of contacts and four types of events. By the time you get to the application, I am only handling one type of contact and two types of events.

The result is limited in power and flexibility, but it makes me happy!

Code is provided unsupported. I haven’t even testing the distribution contains all the files needed. It is provided for study purposes only. I retain copyright; speak to me before you use it in anger – I am likely to let you have it for free once I hear the purpose.

The Code