Apology

I owe my new Nokia 3710 fold mobile telephone an apology. A few days ago I was moaning about how difficult it is to match a telephone number with a new contact. I discovered today that if I look at the log of callers, I can create a new contact with the unrecognised telephone number directly instead of using the old method (create a new contact then connect the number to the contact). Well done, Nokia: this method is much better.

I also received a usb cable for the telephone, so I could upload my personal ringtones and pictures which I had saved from my old phone. On the old phone, if a contact whose picture I had would call, the entire screen would fill with the contact's picture (thus I could see who was phoning when I wasn't wearing my reading glasses). Now, it seems that the contact's picture is displayed on about a quarter of the phone's screen so it's a bit more difficult to recognise.

Kate Bush - The Director's Cut

A message appeared a week ago on one of my music mailing lists about the existence of a new Kate Bush recording that can legally be played, until its release. Dutiful fan of Kate that I am (or was), I went to the site, played the album in its entirety and at the same time, recorded it to my hard disk, courtesy of some software which can record whatever the soundcard plays.

I listened on and off to the recording and eventually burnt it to a cd so that I could listen to it in the car. I played the cd the other night on my way to my MBA studies but after a few tracks I had to replace it with the Unthank sisters. There were technical problems with the burnt cd, but that wasn't the only reason: the songs simply don't interest me.

The album consists of eleven songs taken from Kate's sixth and seventh albums, "The Sensual World" and "The Red Shoes". I certainly bought the first on vinyl but probably didn't buy the second; not having a record player, I haven't heard these songs in nearly thirty years. This time around, Kate has revised the recordings: they have been thinned out and new vocals recorded. Whilst this makes the songs less bombastic than they used to be, they're still not the sort of thing which I would want to hear (that said, I like the opening "Flower of the mountain" and "Deeper understanding").

On the other hand, listening to the Unthanks was like coming home. Since having written about 'The Bairns', I've also bought its follow-up, "Here's the tender coming". This is somewhat less bizarre than its predecessor, primarily due to the lack of piano (the pianist had to leave the group for reasons that I have yet to ascertain, but the parting was amicable), but whilst becoming more mainstream, it loses some of its charm. There are even guitars and drums used (gasp), albeit occasionally, and the arrangements are generally more 'normal' as opposed to the more abstract arrangements of 'The Bairns'.

The centerpiece of the album belongs to a beautiful cover of Anne Brigg's "Living by the water", with a sumptuous and shimmering vocal by Becky Unthank (I notice that she always sings lead on the modern material) and exciting chord changes. There's even a trombone in the background, which at times sound incongruous and at other times sounds perfectly languid, invoking a warm summer's evening.

One thing which irks about both records is that in both cases, the penultimate track is anthemic ("Fareweel Regality" and "Here's the tender coming"), which would be a good way to close the record. Instead they chose to place something entirely different at the end, which spoils the mood created by the previous song.

It seems that I can't totally free myself of my musical likes and loves from the 70s (and some of them in retrospect have turned out to have been pearls); whilst most music from the current century leaves me cold, there are gems to be found.

Mobile computer update

For the last ten days or so, I've been carrying around a heavy IBM R61 (?) mobile computer which has no battery. As a result, I was unable to use the computer on any of the train journeys which I made during that time, which partially negates the whole point of having a mobile computer.

I am pleased to report that the lemon has been replaced with a neat IBM X61s (with battery), which is the same as I had before I was given the lemon. Actually, this computer is slightly better than my previous computer, as the sound card on this computer works.

I should point out that the IT personnel have mobile computers which are 'lent' to them; if the company suddenly needs a mobile computer for some reason and none are available, then our computers are seconded to that need. That's why this new computer (to me) is actually the fifth different mobile computer which I've had in the past few years. Most of my work I do with a desktop computer in my cubby-hole, so these changes don't affect me that much.

New mobile phone

Way back in April 2008, I wrote that I had received a new mobile phone, a Nokia 6288. Slightly more than three years later, my phone got damaged somehow; as a result, the screen and keyboard no longer work, meaning that the phone is next to useless. I  could still receive calls, but I couldn't see who was calling me and I certainly couldn't phone anyone. Cue new phone.

The company distributed new mobile phones about two months ago; when I went to exchange my phone, it transpired that there was no phone for me. The person responsible for mobile phones in the company was embarrassed at first, but later remembered why there was no phone: IT people were receiving smart phones.

Not being a gadget driven person, I received this news with mixed emotions. I began dropping hints each time I met this person, but no smartphone materialised. With the death of my mobile, there was no alternative but to demand a new phone. It was not a smartphone, but rather a Nokia 3710.

I tried working with it a little, but found it not very conducive to my needs. I wrote a letter to my supervisor pointing out that I had been promised a smartphone, that my IT colleagues have smartphones and that I am being discriminated against. The fact that a few days before, my mobile computer had been taken for someone else and I had been given a heavier model which has no battery (which would have made it even heavier) - a non-mobile mobile - hadn't exactly improved my mood regarding equipment supplied by the company.

I was told that only sales people received smartphones, which is stretching the truth. The other IT people received smartphones so that they could learn how to use them and help the others who had received such phones. There are others who also have smartphones, but on the other hand, I saw several managers with lesser phones. 'Twas a shame about the promise.

The psychological aspects of this incident are intriguing: one's disappointment is matched by one's expectations. Had the man responsible for the phones not said that I would be getting a smartphone, then I wouldn't have been so disappointed. To be honest, I would prefer to continue with the Nokia 6288 instead of a smartphone, and certainly continue with the old model in preference to the new model.

I can barely hear the new phone ringing, even though I have increased its volume and placed it in my shirt's breast pocket. There is no USB cable supplied so that I can transfer ringtones and pictures saved from my previous phone. But aside from these carpings, there are some very strange design issues which make it hard for me to like the phone. For example, I frequently need to see the call log, especially calls which I have missed (probably because I didn't hear the phone). On the old phone, the log was quickly accessible, but not so on the new phone.

Another intriguing problem raised its head yesterday: someone phoned me and I wanted to store his number as a new contact. As on the old phone, this is not possible to do via the call log; one has to create a new contact and attach the telephone number to the contact. On the old phone, a contact is created by defining the surname and forename; the contact can exist without a telephone number. Once the contact has been created, one goes to the call log and attaches the number to the newly created contact. But on the new phone, one creates a new contact by first entering the telephone number and then adding the name. But I don't have the number! 

I can't attach the number to the contact because there is no contact and I can't create the contact because there is no number! Maybe the way out of this conundrum is to enter a dummy number, create the contact, attach the number then delete the dummy number. Too much work! In the end, I wrote the number on a piece of paper, then entered it when creating the new contact. Is there no such thing as copy/paste?

Thinking about the problems last night when returning from my MBA studies, I realised that the phone had been designed for a new generation of users who are much more interested in accessing the Internet and using the mobile as a general purpose device (phone, music player, internet browser) than old fogies like me who see the phone as a phone. That's why functionality which is connected to the Internet is easy to access, whereas more traditional functions have been shunted aside.

As I say, I am not a gadget driven person but neither am I a Luddite. Give me my old phone back (or at least, the same model which works)!

Another database query in a background thread

In the last blog, I wrote about running a database query that automates Word in a background thread, no mean technical feat, but not necessarily useful. After discussing the subject with my client, the redoubtable Occupation Psychologist, we saw that there wouldn't be much need for this technique, at least for the moment.

On the other hand, every now and then we force our flagship product to recalculate averages and standard deviations for thirty plus scales. Whilst calculating the average is not difficult, calculating the standard deviation and updating users' scores against the new standard deviations is very time consuming. This is the sort of task that would be speeded up by calculating in a separate thread.

As the program is a MDI (multiple document interface) program, the user can display several forms at once, in the same way that Windows displays several programs at once. Unlike Windows, the program is only single threaded, meaning that although several forms can be displayed at once, only one is actually 'working'. Using a background thread means that more than one form can be 'working' at the same time.

With regard to the example form that we chose: the user is presented with a list of scales, a grid and a progress bar. Upon choosing a scale, the form begins running a query and updates the progress bar in order to let the user know that something is happening. When the query is complete, the grid fills with the calculated data. The new scenario which I envisage for this form would be:

• The user chooses a scale to be calculated
• The form activates the background thread (query) and minimises itself
• When the thread (query) has completed its work, it notifies the form that it has finished
• The form reads the result of the calculation from a database table and returns to its original size (not maximising, which would fill the entire screen)

Originally I thought that it would be possible to hide the form while it was calculating then show it upon completion, but it seems that it is not possible to hide MDI child forms; instead I have to minimise it and then restore. In terms of programming, this doesn't require any extra work, but I think that the end result would have been slightly less confusing had I been able to hide the form.

The last post showed how to create a thread in a simple way without resort to Delphi's TThread object via the Windows API. A limitation to this approach is that the thread cannot access any of the form variables, especially not any of the visible components of the form. In the previous post, I got around this limitation by outputting the query's result to Word; here I get around the limitation by outputting the query's result to a database table. The TThread object has a 'synchronize' method which allows for the background thread to access the form's visible components, but it seemed like overkill here.

Another limitation of the API approach is that only one variable can be passed to the thread (in the Word example, no variable was passed). The problem here is that several variables have to be passed to the thread and several variables have to be passed back to the form. The thread has to know which scale has been chosen for calculation and it has to know what the handle of the form is, in order that the thread can send a message to the form's handle upon completion. Ideally, the thread should also be told the location of the database. The thread has to inform the form how many records have been processed: this was done by using one of the parameters which the thread sends to the form upon completion.

Whilst the thread cannot access any of the form variables, it can access any of the variables defined as global to the form. In other words, if the form definition looks like this

interface

uses ...

const
 TH_MESSAGE = WM_USER + 12;
 TH_Finished = 217; // random number

type
 TScaleAnswers = class(TForm)
  lb: TListBox;
  DBGrid1: TDBGrid;
  DataSource1: TDataSource;
  Label1: TLabel;
  Label2: TLabel;
  qScales: TSQLQuery;
  XLBtn: TBitBtn;
  qScaleAnswers: TSimpleDataSet;
  qClear: TSQLQuery;
private
 working: boolean;
 procedure ThreadMessage (var Message: TMessage); message TH_MESSAGE;
public
end;

implementation

var
 scale: longint;
 thread: dword

Then the thread can access scale and thread but not working. An invocation specific number (not surprisingly, the thread's id) is stored in thread whilst the chosen scale's internal number is stored in scale. The calling form's handle is passed as the parameter to the thread. Thread is used as a key when inserting data into the database table in order to facilitate its retrieval and eventual deletion.

The thread is created in the following manner

procedure TScaleAnswers.lbDblClick(Sender: TObject);
begin
 if not working then
  begin
   working:= true;
   scale:= sendmessage (lb.handle, lb_getitemdata, lb.ItemIndex, 0);
   CreateThread (nil, 0, @DoQuery, pointer (self.handle), 0, thread);
   windowstate:= wsMinimized;
  end
end;

After the thread has finished its calculation, it sends a message to the form:

sendmessage (formhandle, th_message, th_finished, pcount);

This is handled in the form by the following code:

procedure TScaleAnswers.ThreadMessage (var Message: TMessage);
begin
 if Message.WParam = TH_FINISHED then
  begin
   with qScaleAnswers do
     begin
      connection:= dm.sqlconnection1;
      dataset.params[0].asinteger:= thread;
      open
     end;
    label2.caption:= inttostr (message.lparam);
    windowstate:= wsNormal;
    working:= false;
   end;
end;

How does this code 'know' to execute? Referring back to the form's definition, there is a procedure defined thus: ThreadMessage (var Message: TMessage); message TH_MESSAGE. This means that the above procedure will be invoked whenever the form is sent a message with the TH_MESSAGE parameter. A sanity check (probably unnecessary) is made against the wparam parameter. This code activates a SimpleDataSet to retrieve the correct data (utilising thread in a 'where' clause) which is then displayed. Note how label2 receives its data from the message's lparam parameter, which is equivalent to the thread's pcount variable.

Conceivably, this form could be invoked simultaneously with several different scales being calculated. In practice, the calculation occurs so quickly on my computer (the database server is local) so that I never get a chance to do anything else, although I suspect that on the client's network it won't be quite so fast. It transpires that updating the original form's progress bar was taking a good proportion of the calculation time, so removing it improved up the calculation's apparent speed. But even so, the main program is unresponsive until the calculation has completed.

Running a database query that automates Word in a background thread

Now that Word seems to be automating properly, I thought that I would devote some time to implementing a technique which has eluded me so far. Several of my programs run database queries which take minutes to run, and during this time the programs are unresponsive. The solution is to run the query in a separate thread from the main program, but this is easier said than done. Some programs run database queries which output their results to Word - no user intervention is required, and again it would make the program more responsive if the Word code were handled as a background task.

A few years ago, I read this article which seemed to make implementing background queries easy. Although I tried several times to get this code to work, I was never successful - until last night. Handling such a task requires two special techniques to make it work: neither of these are difficult in themselves, but they both have to be present

• All database components have to be declared in the thread procedure
• If the thread automates Word, then CoInitialize and CoUnInitialize have to be invoked in the thread

I had tried to use the dbExpress database components but discovered that the TSQLConnection component requires many properties to be defined and I was missing a few. So for a change, I decided to go with the Interbase database components which come with Delphi (IBDatabase, et al.). As I have never used these components before, I wrote a small test program which uses these components in a normal manner; this way I could see how the components needed to be connected and which properties have to be defined. Once I knew this, I could transfer the visual definitions to a text definition.

The thread cannot access any of the components defined on the same form; I'm not sure whether this applies to the program's datamodule as well. Just to be on the safe side, I retrieved  the database's physical location from where I store it in the registry, although it may be possible to retrieve this value from the datamodule's TSQLConnection.

Here is part of the code;

function DoWord (p: pointer): longint; stdcall; // see the linked article
 var
  ibdb: TIBDatabase;
  qDefaults: TIBQuery;
  trans: TIBTransaction;
  wrdApp, wrdDoc, wrdSel: variant;

begin
 CoInitialize (nil);
 wrdApp:= GetWordObject;
 wrdApp.visible:= false;
 wrdDoc:= wrdApp.Documents.Add;
 wrdSel:= wrdApp.Selection;

 ibdb:= TIBDatabase.Create (nil);
 with TRegIniFile.create (regpath) do
  begin
   ibdb.databasename:= ReadString ('firebird', progname, '');
   free
  end;

with ibdb do
 begin
  loginprompt:= false;
  params.add ('password=masterkey');
  params.add ('user_name=sysdba');
  sqldialect:= 1;
  connected:= true;
 end;

trans:= TIBTransaction.create (nil);
trans.defaultdatabase:= ibdb;

qDefaults:= TIBQuery.create (nil);
with qDefaults do
 begin
  database:= ibdb;
  transaction:= trans;
  sql.Add ('select * from defaults');
  active:= true;
  while not eof do
   begin
    wrdSel.typetext (fieldbyname ('id').asstring + #13);
    next;
   end;
  close;
  free
 end;

 wrdSel.homekey (wdStory);
 wrdApp.visible:= true;
 wrdApp:= unassigned;
 wrdDoc:= unassigned;
 wrdSel:= unassigned;
 trans.free;
 ibdb.free;
 CoUnInitialize;
 result:= 0
end;

Obviously, I've left out most of the Word code, because that's not relevant to the subject. Now that the technique has been implemented successfully, I have to decide where I am going to use this! It may be that the technique is useful from an academic point of view, but anyway it is good to have another tool in my toolbox.

More facts your mother never told you about Word automation

It transpires that I was mistaken when I wrote in my previous blog that the problems in automating Word 2010 had been solved. Even when defining the complete path to the template, I was receiving 'invalid variant operation' messages. After having wasted many hours trying to solve the problem - and trying to isolate the problem, I escalated the problem to the excellent Stack Overflow site.

I first thought that the problem was with the template, but when I saw that one of my programs worked with a certain template but that another didn't with a different template, I was inclined to believe that the problem was with accessing the bookmarks within the template. It turns out that I was wrong on all accounts.

Here is the answer which solved the problem; I quote it verbatim.

Word 2010 has a bug that is related to loading Normal.dotm (and maybe plugins too, who knows?). When you start Word 2010 as you would normally do, you see a splashscreen and Word performs some initialization, including loading Normal.dotm. When you start Word via automation - CreateOleObject('Word.Application') - it doesn't wait till Normal.dotm is loaded and returns immediately. But performing operations when Normal.dotm is still being loaded seems to crash Word. What I did to solve this problem is to make a loop that just waits for the template to load. You can also choose for a delay to give Word the time to initialize, but so far, the loop works.


Something like this:

wrdapp := CreateOleObject('Word.Application');

// loop that waits for the normal template to load

while wrdapp.Templates.Count = 0 do Sleep(200);

// continue operations

I added the 'sleep' command to one incalcitrant program, and the fix works! Now I have to go over all the programs and fix wherever Word is invoked, but this shouldn't be too difficult.

In retrospect, the 'invalid variant operation' message now becomes clear: I was trying to access something which hadn't yet been created, so of course the operation was invalid.

Now perhaps I can get back to creative programming instead of holding fingers in the dam and getting frustrated (never a good idea when debugging).