The birds

Before I became embroiled with the quest of writing an HTML enabled Hebrew mail client, I had some spare time which I devoted to musical pursuits. Still involved in my series of Van der Graaf Generator/Peter Hammill covers, I decided to record a version of "The birds". This is a song which appears on Peter's first solo album, "Fool's mate", coming at the end of a wonderful sequence.

Listening to the album once again in order to refresh my memory, I was struck by how beautiful Hugh Banton's piano playing was on this song; Robert Fripp's guitar work also impressed me but less than it used to (and I've been listening to this record for 40 years). For the first time, I got the impression that Fripp might have added his contribution after the basic track had already been recorded.

I should also point out that whilst Banton's incredible piano work is all over the record (check out "Vision"), he very rarely played the piano in the context of VdGG, leaving that to Hammill. I suspect that his delicate and filigree playing would have been lost in the wider context of the group, which is a shame.

Anyway, "The birds". This song harmonically is very simple, having the same line F/C/Dm/C repeated three or four times during the verse. Banton's playing disguises this simplicity but I would have to suffer it. After working out a brief swung version which didn't get past the demo stage, I started work in earnest with a pad and bass; I found a lovely guitar patch which was playing harmonics and utilised this with one of Fripp's licks. I balanced these harmonics with vibes, which at one stage play a lick from the Doors' "Riders on the storm". A solo was played on a slightly distorted electric guitar; after a few days, I improved the solo and added an arppegiated organ to fill out the background.

Recording the vocal went very quickly, and in no time I had a completed track ready. It's  fairly good, although also fairly basic. It's certainly lighter than some of the other epic tracks which I have recorded, although I deliberately aimed for some light contrast.

Relaying the email

I finished my last blog on a cliff-hanger: after getting HTML text, attachments and UTF-8 subject encodings to work properly, I discovered that there was a major problem with my program. As long as mail was being sent to an address with the same Internet provider as the SMTP definition - and the OP and I share an Internet provider - everything was ok, but as soon as I tried to send email to an address outside of the provider's domain, I received an error message about relaying and the mail was not sent.

It transpires that a few years ago, Internet providers tightened up their email protocols to prevent spam mail. I wasn't aware of this and blissfully continued to use code which worked in the past but not now. My first action was to switch providers and send email via Gmail, but this too required some changes. After more feverish googling, I discovered that almost certainly I wouldn't be able to send email via Gmail with Indy 9 (which comes with Delphi 7), but that changing to Indy 10 would solve my problem.

Trying to find a site from which I could download of Indy 10 was amazingly difficult; the Indy project is very coy about the downloads and I had to negotiate several screens before I eventually found what I was looking for. Very strange: I would have thought that they would have placed a link to the latest version on their home page. After downloading the new version and updating Delphi so it would use this new version (a procedure which took about an hour to my further surprise), I eventually got an Indy 10/Gmail demo to work! Hurrah! One 'gotcha' was that two units have to be linked manually into the project and no one told me this explicitly.

But then I had to convert the subject line to UTF-8, the body to HTML and add the attachments. Finally I received an answer to my query about encoding the subject line: the TIDMessage component which exists in both Indy 9 and Indy 10 has an event called 'OnInitializeISO', which exposes two fields, VTransferHeader and VHeaderEncoding. By setting one to 'UTF-8' and the other to 'B', the subject line is automatically encoded as UTF-8/Base64. Unfortunately, there is no help regarding this field, so as Randy Newman sang correctly, "There ain't no book you can read". Fortunately, there was "someone to tell you", which is how I learnt about this. This little tip would have saved me at least three hours of screwing around had I known about it beforehand.

I still haven't got 'non-connected' attachments working correctly but I think that I know what to do. The only problem left - and it's no small problem - is that the email which I sent from home to work arrived correctly but with no Hebrew text contained within. It looks like I'm going to have to encode all of the internal text (what would be the body of the email) to UTF8/Base64 before storing this in the email. This isn't really a problem, more of an annoyance.

As I've been writing the majority of this email series a couple of days after the events described, I always knew when writing how I solved the problems. Now I don't know whether I will solve the two remaining problems easily, but I am optimistic. There is a tutorial but I think that it could be improved; a complete demo program would certainly help. Fingers crossed.

I've been very busy during the day for the past few days which means that I've had to work hard at night on these email problems. One night I dreamt about encoding subject headers and last night I dreamt about setting up web sites for a Chinese girl and having problems with the encoding. Hopefully I'll be able to dream about more regular subjects in the near future.

Creating an email with a subject line in Hebrew

There ain't no book you can read
There ain't nobody to tell you
But I don't think I'm getting
What everybody's getting
Maybe I'm doing it wrong

 (Randy Newman, "Maybe I'm doing it wrong")

After having successfully created emails which display embedded pictures, we then discovered that the subject line of those emails was not always displayed. The test emails which I sent to myself displayed correctly (naturally), but the subject was not readable (only question marks) when the OP accessed the mail on her iPhone.

After rooting around on the Internet, I discovered that the subject line can have its own encoding. Whatever the coding, the line cannot contain characters whose value is over 127. It turns out that most email subject lines in Israel are encoded with the Windows 1255 code page, which is not surprising as this the code page for Israel.

For every encoding (such as Windows-1255 or ISO-8859), there are two possibilities: the line can be 'Quotable' or it can be 'Base64' encoded (Base64 encoding is a system which turns translates byte streams (there may be values over 127) into readable characters (A-Z, a-z, 0-9, +, =).

The above picture show the headers in an Email which I was sent. The part which interests me at the moment is the second line, Subject: =?windows-1255?B?Uk.......?=. Ignoring the 'subject:' part, the line consists of three sections: a prefix (=?windows-1255?B?), a suffix (?=) and a payload, which is everything inbetween.

The prefix tells the email client which type of encoding was used; in this case, it is base64 encoding of Windows-1255. The email client of the person who sent me the email encoded the subject into Base64 ; my email client reads the prefix and then decodes the subject. It's not clear to me at the moment why the majority of emails use B encoding as opposed to Q encoding.

Having learnt this, I 'instructed' my email program to send letters encoded as Windows-1255-Q; this requires taking a Hebrew letter and displaying it as the character equivalent in hex (for example: the letter 'aleph' is represented as character 224 in the code page. 224 is E0 in hex, so the encoding would be E0=). The subjects of letters sent like this were correctly decoded by computers running Outlook, but the iPhone remained stubborn. Presumably the iPhone doesn't know how to decode emails sent in windows-1255 format; emails sent from the iPhone are encoded in UTF-8?B format.

In order to send subjects encoded in UTF8?B format, I have to convert the subject line into UTF-8 encoding and then encode it again to Base-64. After no small amount of fiddling about, I discovered that the Delphi system function AnsiToUTF8 performs the first step correctly (at one stage, I assumed that there was a mistake as the Unicode coding for the letter aleph was not the same as the UTF-8 coding; it transpires that by definition they are not the same), but the Base-64 encoding was producing incorrect results.

After pulling out what is left of my hair and ruining the keyboard with a multitude of google queries, I discovered that there was a bug in the Base-64 implementation that I was using, which came from this page (which is the same as this page). The 'codes64' string begins with the digits, then capital letters, then lower case letters and then punctuation. But this page shows that the capital letters should come first, followed by the lower case letters, followed by the digits, followed by the punctuation. Once I had made this correction, the strings began to be encoded 'correctly' - I could read them properly in my email client, which I am taking to be the definition of 'correct'.

Why the epigram from Randy Newman which opens this posting? Because it seems that the information which I needed does not exist in one place; I had to ferret it out piece by piece and deal with misinformation. At the beginning, almost certainly I was doing it wrong; there was no book to read and no one to tell me. Obviously I'm not getting what every other email client is getting, so clearly I'm doing it wrong.

So: we have slain yet another problem in the path to sending Hebrew emails to a mailing list. Unfortunately, there is still another problem waiting in the wings....

Election day duty driver

On my kibbutz, like many others, there exists the function called 'duty driver'. This means that every day during the week, a kibbutz member takes a large car and drives people from the kibbutz to the nearest town then picks people up returning by train or by bus from the town to the kibbutz. The hours are from 5-10pm, after work, and it's a voluntary - meaning unpaid - job. 

In the old days, "everybody used to take a turn at doing everything" - there would be a rota for working on Saturday, a rota for working in the dining room in the evenings, a rota for guard duty, a rota for duty driver, etc. Over the years, most of these rotas have disappeared and the only ones left are duty driver and guard duty. Once men used to do both, but now they do either one or the other. I opted for duty driver as it gives me some sense of social reward (the hours are better too).

In the past few years, more and more kibbutz members have acquired their own form of transport and so the number of people who take advantage of the duty driver has steadily decreased. In turn, the scope of the duty driver has been reduced, meaning that regular journeys are made between 5-7:30pm, after which the driver sits at home and waits for people to telephone.

The last time that a general election was held in Israel was four years ago, on a rainy Tuesday in January. People don't work on the day elections are held, ostensibly to allow them to travel to wherever they have to be in order to vote, but for most people it is just a day off (shops are closed but public transport is available). As it happened, I was duty driver that day: along with the holiday and the rain, I don't think I gave anybody a lift. I simply drove through the empty and wet streets of Bet Shemesh.

There will be a general election again next Tuesday, and although it doesn't look like rain (we had enough last week), once again I will be duty driver on election day. Coincidence or favouritism? What are the chances of being duty driver on consecutive election days?

At first, I thought that it was a strange coincidence (well, all coincidences are strange by definition), but then I realised that the odds of this happening are far from astronomical. As I do this duty once every two months, there are about 45 people on the rota and so the probability of being duty driver on any given day is thus 1 in 45. As there is no connection between the rota from four years ago and the current rota, the two probabilities should be multiplied together meaning that there is a 1 in 2025 chance of being duty driver for two consecutive election days. 

Better odds of being duty driver on election day than winning the national lottery.

Embedding pictures in HTML emails

Following on from my  post from a few days ago about previewing HTML emails, it seems that I prematurely congratulated myself about including pictures in the email. Whilst I could see the pictures in the emails (not only in the preview), other recipients of the emails could not see the pictures.

This was because the pictures were referenced by the "img src" command which assumes that the picture exists on the computer which is reading the HTML code. This works when one is reading an Internet page with a browser, but doesn't work when reading emails - because the picture is not embedded in the email.

In order to find out what command was necessary to do this, I first sent myself an email from Outlook which contained an embedded picture; I soon discovered that the necessary command is still "img src", but instead of accessing/naming a file, it uses a "cid" (content id). But how does one set this value?

There is a very informative page here, but this mainly discusses Indy 10, and I am (of course) using Indy 9. After asking the question on Stack Overflow, along with a bit of to and fro, I saw what I needed to do.

An HTML letter is built of several sections; one of these sections deals with embedded pictures.

if laPicture.text <> '' then
 with TIdAttachment.Create (email.MessageParts, laPicture.text) do
  begin
   ContentDisposition:= 'inline';
   ContentType:= 'image/jpeg';
   DisplayName:= ExtractFileName (laPicture.text);
   filename:= ExtractFileName (laPicture.text);
   ExtraHeaders.Values['Content-ID'] := '<12345>';
 end;

The key line is the one which starts with 'ExtraHeaders' - this assigns a content-ID value to the picture. The text quoted is arbitrary - it could have been the file name or the digit 1; anything will do as long as it is enclosed in angle brackets.

The HTML code for accessing this then becomes

img src="cid:12345"

Several tests confirmed that this was the key to embedding pictures in HTML mails.

The next problem to be overcome is the fact that the subject line of the email sometimes comes out in gibberish. I will address this is a future blog entry.

Play it again, Sam

I went to the theatre the other night. To some of my readers, this may seem an everyday, almost mundane, thing to do, but as far as I can recall, I don't think that I've been once to the theatre in the last 34 years, ever since I emigrated... probably something to do with the language (and why should I go to see Shakespeare translated into Hebrew?).

So what dragged me out on a cold and windy (but thankfully not rainy) night? My old friend, Woody Allen's first play, "Play it again, Sam". As I related several years ago, this is probably my favourite film and I traveled many miles in order to see it. What I didn't write in that blog entry was that I also saw the play from which the film was derived - in a theatre in Hampstead, early 1978. I think that it was a student company that put on the play so one did not expect much.

Anyway, the play has been translated into Hebrew and is now being presented on the stages of Israel, courtesy of the Beer Sheva theatre company. The character of Alan Felix (aka Woody Allen) is played by either Guri Alfi or Eidan Alterman, both top Israeli comedians. The show which I saw had Guri Alfi, who is definitely more visible these days than Alterman, appearing twice a week on television; he seems a very good match for Allen.

Everything was excellent: the play, of course (the translation was fine and covered maybe 90% of the film), the set, the acting and especially the direction. I had wondered how they would deal with Bogart and Nancy (the ex-wife) coming in and out of the play; there were three video screens placed stage left, stage right and stage rear upon which Bogart appeared and performed his dialogs with perfect timing. Nancy appeared sometimes on the screens and sometimes in the flesh.

The actress who played Nancy also played several of the dates, most noticeably Sharon. This scene was played out almost identically to the film (although the track medal which Woody Allen displayed ostentatiously was replaced by a similar medal for 10km speed walking), including throwing the record across the room. This scene had an addition to the dialogue: the record (not played) was by Thelonious Monk, and Alfi got many laughs by pronouncing this name as unclearly as possible (this might be one of his personal party pieces). We had the Jackson Pollock dialogue in the museum complete, although they missed the joke at the end by the girl walking off without looking at Alfi after he asks what she's doing on Friday night (on Saturday night, she's committing suicide).

On stage was a pianist who accompanied the acting, along with playing the odd song (especially, of course, "As time goes by"). There was a joke possibly taken from Randy Newman in his recording of the song 'Shame' in which he starts talking to the background singers (who sing 'Shame') - "could you please keep quiet". Alfi also talks at one point to the pianist, asking him to be more quiet.

What surprised me the most was how they managed to encompass the variety of styles presented in the film - especially the third act - on one stage in a continuous performance. Indeed, very little was missed out; the entire Julie sequence when Alan, Dick and Linda go to the beach house for the weekend was unsurprisingly cut, but made very little difference.

The above picture shows Guri Alfi as Alan Felix and Efrat Boimold as Linda Christie. I should point out that Boimold didn't appear at the show we attended; her part was played by Yael Grobglass, who is both taller and slimmer than Boimold (her picture appears below).

If you get the chance, go and see this show.

Previewing an HTML email

Following on from my previous post about creating emails and sending them to a mailing list, the Occupational Psychologist asked me whether it is possible to add pictures to the emails. Whilst this is not an area in which I have great expertise, I was able to answer in the affirmative because the original HTML code which I found included a picture.

But my problem then became - how does the program know where to insert the picture (top, middle, bottom)? I solved this problem - or at least, postponed it - by adding a radiogroup to the form which allows the user to choose either top and bottom, and then simply add the HTML code necessary in the correct place:

if rg.itemindex = 0 then add ('!img src="' + laPicture.Text + '"!br!');
for ml:= 1 to mem.lines.count do add (mem.lines[ml - 1] + '!br!');
if rg.itemindex = 1 then add ('!img src="' + laPicture.Text + '"'!br!');

The above snippet again has substituted an exclamation mark for an angle bracket as the editor here does not like HTML code, even if it's within a blockquote passage.

I sent myself an email and discovered that the picture's path had best not contain any Hebrew text; it's best to store such a picture in the root directory of C or c:\windows\temp.

I then tried to second guess the OP, wondering what she would ask for next. As the email is now no longer WYSIWYG because of the picture, she would probably want to see a preview. No problem (or so I naively thought)! All I need to do is save the HTML code in a file, fire up the default Internet browser and tell it to read the file which I have just created.

First problem: the HTML code was only created in response to clicking the 'email' button, which would actually send the email. Thus I need to create the HTML code once for preview, storing it in a file, and once for sending. This actually is not too bad, as the user could change the text of the email, requiring the HTML to be recreated. Once I got past this stage, I then discovered that my Hebrew text was appearing as garbage letters.

I started googling Unicode and similar matters, only to discover that if I did try creating a unicode file (not necessarily a simple matter in Delphi 7), the result would still not appear correctly. Then I remembered that browsers have an 'encoding' option; once I used the correct encoding on my displayed page, the correct text was displayed. How can I avoid the user having to choose the correct encoding every time? By putting the instruction in the HTML file, of course! This requires inserting some code directly after the !html! line in the header - !meta http-equiv="Content-Type" content="text/html; charset=windows-1255"!. Windows-1255 is the default page for Hebrew. Once this instruction was inserted, the Hebrew text appeared correctly.

Last flourish: instead of my programming calling the default browser to display the code, why not include a minimal browser in my own program? This would allow me to delete the temporary file which I had created every time the preview screen closes. As this is Delphi, it was no problem to find the TWebBrowser component and insert it into a form. The browser is told to display the file by calling its Navigate method. Simple as pie.

What's left? Well, it would be nice to display a picture in an arbitrary place in the letter. I could do this by replacing the radiogroup with an edit box, asking the user in which line to place the picture (0=top, 999=bottom, 5=after the fifth line, etc) but this may be too much. I'll find out on Friday.

Another possibility would be always adding a picture, such as the OP's logo, at the bottom of the letter. This would require storing the name of the picture in the database, but is simple to implement.

Recurring actors in long running TV shows

It sometimes happens in a long running tv show that an actor will appear over the years in a few supporting roles, each time as someone different. This happened a fair amount in the Star Trek franchise, although they had it easy as someone could appear as a human one week, as a type 1 alien another week and as a type 2 alien the following week.

There are two long running British series that I watch every week: Casualty (now in series 26, Israel running about three months behind Britain) and Silent Witness (now in series 12, running about four years behind Britain). There have apparently been several actors who have played multiple characters in Casualty although I haven't been aware of this. As we only see one episode a week, it's easy to forget the extras.

But Silent Witness is shown every night (Sun-Thu), so we get through the episodes at a cracking pace. I've noticed two supporting actors who have appeared twice: Ruth Gemmell played a detective constable in all six episodes of the first series, but was 'killed' at the end of that series. Despite this, she made a remarkable recovery and appeared as a Detective Inspector in series nine or ten (not listed on IMDb). I know that dying (especially early) is a good way to sell records but apparently it also helps one's police career.

A rather more egregious example is that of Ciaran McMenamin, who played the part of Prof Sam Ryan's hitherto unknown son in episodes 1 and 2 of series 8, the final episodes of Ryan (Amanda Burton). In these episodes, he played a surly and unpleasant Irishman. Lo and behold, in episodes 5 and 6 of series 12, he once again plays a fairly major part, also as a surly and unpleasant Irishman.

In the first few series of SW, each series featured an unvarying police team (hence the multiple appearances of Ruth Gemmell as a dc) but since then, there have always been new policemen every week ... except once, when Paul Panting played the part (alliteration running wild for a minute) of DCI Mumford in a few different episodes. How this policeman rose to the exalted rank of Detective Chief Inspector is beyond me as he doesn't seem to be a very good policeman. The same actor apparently played the part of 'Gordon' in one of the earliest episodes but didn't trouble my memory then.

Sending mail from programs

I've been working in a desultory fashion for the past two weeks on adding mailing lists to the management program which I write and maintain for the Occupational Psychologist. Whilst the concept is obvious (define a list which holds the email addresses of people), the actual implementation is slightly more complicated, as there are at least two types of people with email addresses defined in the program - therapists and customer contacts - and their details are stored in different tables. But once this problem is out of the way, all that remains is to create a form which receives a mailing list, a subject, an optional file and text (one's ordinary email message) and then send the subject, file and text to the mailing list.

There are a few caveats which need to be considered before one starts:

1. Not everyone in the mailing list may have an email address
2. The people in the mailing list should appear in the Bcc field, so all the recipients are oblivious to who also appears in the mailing list
3. The number of recipients who appear in a given mail should not exceed 25, in order not to arouse the mail server's spam filter

As there are other parts of the program which send mails via Outlook, I thought that I would start with the same general idea (open Outlook, connect to the Inbox, create a mail, insert the data). I noticed that in all the other cases, the program displays the email on the screen but does not send it, leaving this to Outlook. This is because these cases insert a recipient's address but do not include any text.

This meant that I had to add the final necessary stages - add text, send, close Outlook - to the mailing list code. I wrote general code to do this a few years ago so it wasn't a problem to add the code. Having written that, I should point out that this old code was written before Outlook adopted a security patch which prevents unauthorised mails to be sent. The way to overcome this security patch is to use the Redemption software package. 

Whilst using this package allows my code to create and send emails, it also causes my code to crash with an Access Violation message. I noticed that the address in the AV message to always be the same, which is a give away that there is probably a bug in the clean-up code of the library. After I googled the issue, I came across a mail which I myself had posted to Stack Overflow, presenting this exact same issue (with the same address in the AV message) and asking for a solution. Whilst several people had posted answers, none of them helped.

I then realised that I must have given up on writing Outlook code at that point and had moved to sending emails via an SMTP server (what might be termed, "the classical way") and the Indy components which come with Delphi. Once I had this problem solved, sending the emails - along with all the caveats listed above - became easy.

For icing on the cake, I also discovered how I could send the emails as right to left displaying HTML text as opposed to left to right plain text (problematic when the text is in Hebrew). I would like to be able to take the output of a rich text component and use this directly but apparently rich text and HTML are different and there is no simple way to convert.

There is one problem which comes with using SMTP code: the component sending the mail has to know the address of the SMTP server, the account name and password. This is not the sort of information which I would like to embed in a program as a hacker could get the program, decipher the information and then hack the email account. In the past, I have stored this information (encoded or otherwise) in the computer's registry, but here I preferred to store the information in the database itself. 

And after all that typing, here's the code to a simple SMTP program

procedure TForm1.EmailBtnClick(Sender: TObject);
var
 i: integer;
 html: TStrings;
 htmpart, txtpart: TIdText;

begin
 html:= TStringList.Create;
 with html do
  begin
  Add ('!html!');
  Add ('!head!');
  Add ('!/head!');
  add ('!body!!div align="right"!');
  for i:= 1 to mem.lines.count do
   add (mem.lines[i-1] + '!br!');
  Add ('!/div!!/body!!/html!');
  end;

 with IdSMTP1 do
  begin
   host:= .......;
   username:= .......;
   password:= .......;
  end;

 with email do
  begin
   From.Address:= 'somewhere@somecompany.com';
   bcclist.add.address:= 'user@user.com';
    subject:= edSubject.Text;
   ContentType := 'multipart/mixed';
   Body.Assign (html);
  end;

 txtpart:= TIdText.Create(email.MessageParts);
 txtpart.ContentType:= 'text/plain';
 txtpart.Body.Text:= '';
 htmpart:= TIdText.Create (email.MessageParts, html)
 htmpart.ContentType := 'text/html';

// TIdAttachment.Create (email.MessageParts, fn);

 try
  try
   IdSMTP1.Connect (1000);
   IdSMTP1.Send (email);
  except
  end;
 finally
  if IdSMTP1.Connected
   then IdSMTP1.Disconnect;
 end;
 html.free;
 close;
end;

This would be the code, except that there's a bit missing - possibly the most important part. The formatter here doesn't seem to allow me to enter html code as actual text, so I've had to alter the code slightly. In the lines which follow 'with html do...', the exclamation mark (!) should be replaced by an angle bracket.

Being a tourist in Israel: Ein Karem

Ein Karem is a village nestling in at the bottom of one of the hills leading to Jerusalem. I mentioned this village several years ago when I walked through it with my son's class on the way to Jerusalem. Yesterday we visited in order to celebrate my wife's birthday; we were far from alone - the village is a popular attraction for people looking for somewhere to eat. Abroad, such places seem to be two a penny, but here they are somewhat rare.

Here's a picture of the restaurant in which we ate: the name is 'Karma' but for some reason they chose an unusual spelling of the word.

One can't really see from the photograph but the restaurant (on two levels) is packed. We had the foresight to order a table in advance, but not everyone did so; as we entered, someone asked for a table for ten. "Have you ordered?" asked the maitre d'. "I didn't know we have to", answered the lady. "Then you'll have to wait...." 

There isn't that much to see in the village: there's a main "strip" with three of four restaurants, and a pedestrian lane running at right angles with small shops and cafes. Last time we walked along that lane, but as my father wasn't feeling too well yesterday, we had to cut our visit short.

In the distance, facing the restaurant, can be seen the mosque pictured below. There is also an imposing church in the village but I couldn't take a picture of it.

Visitors: beware: whilst there is parking space in the village, there are so many visitors on a Saturday lunchtime that those places get filled quickly. It is better to visit the village during the week when it is far less crowded.

The drive from the kibbutz to the village is maybe only 20km, but the scenery along the way is stunning. Who needs to travel abroad when there is this beauty in one's backyard? Indeed, there were plenty of cyclists and motorcyclists on the road, with many cars parked along the way.

A visitor from Britain is supposed to be coming to visit at the end of January: I hope that we will have the time and suitable weather in order to make the trip to Ein Karem.

Sequencing "House with no door"

I see that blogging is a habit which has to be maintained: if one neglects it for a few days, it becomes harder and harder to return. This December has been a fairly barren month in terms of blogging probably because I've actually been more than busy at work although not with subjects which I can blog about (not because they're secret but rather they're either mundane or esoteric, requiring too much explanation). It's true that I did suffer a period of emptiness after the first doctoral exam which was at the beginning of the month, and it's true that for the past few days I've been suffering from some unspecified viral infection which has left me lethargic and suffering pain in my lower legs, but these are just excuses....

When looking for something to occupy my time, I decided to return to my continuing project of recording covers of Van der Graaf Generator songs. This project was curtailed in the summer when first I had great difficulty in recording vocals to my version of 'Pilgrims' and then almost permanently mothballed when it seemed that I would never sing again after the pertussis episode. But now, a few months later, my voice has returned and so has my appetite for recording. 

There has always been one song which I knew that I would probably record - 'House with no door'. After a few false starts (one promising version began with marimba arpeggios, a device which unfortunately I have used too often, especially in 'Pilgrims'), I plumped for an electric piano playing simple chords as the basis for the recording. I ended up with a version which sounds just like one of my songs, only even simpler, as the chord sequence is very scalar (this caused me problems with the solos - I like complicated chord sequences!).

During the sequencing process, my headphones stopped working properly so I had to buy a new pair. Perusing the headphones on sale in the local mall (two different shops), I discovered that the kind of 'phones (or 'cans' as Peter Hammill and the boys used to call them) which I had previously (closed with a generous amount of foam) were no longer being sold; there are closed 'phones (but not with a perfect seal) but they all come with microphones attached. This actually is not necessarily a bad thing. After connecting the 'phones and microphone to the computer and making a simple test recording, I was pleased to discover that the microphone actually has a very good frequency response and is very clean - no background hum or clicks.

Once I had my backing track ready, I commenced singing. Although I know the song very well, it still took a few takes to get a good recording - there was one line in particular with which I had trouble phrasing. After about an hour, I comped together three separate takes, performed the necessary note correction and then commenced the aurally exhausting process of mixing, with most of the time spent on getting a good vocal sound. I discovered two things about the headphones/microphone combination: very little equalisation (tone control) is necessary but unfortunately the insufficient closure of the headphones along with the adjacency of the microphone to the 'phones had caused a great deal of leakage - the microphone was recording the instrumental track along with the vocal. Normally this isn't too much of a problem, but there were many rim shots from the drums being picked up and these were causing phasing in the final mix. I cleaned as many of these as I could and made a final mix.

Listening to the track over the next few days whilst wearing my producer's hat, I became aware that the instrumental section was limping; in order to correct this, I recorded a new solo over the second half and also spiced up the drumming. This version was fairly intimate with an 'in the face' vocal (very little reverb or gimmickry) and was then deemed to be a winner.

Or maybe not: after a few more days of listening, I realised that there was a cognitive mismatch between the lyrics (somewhat sad and certainly detached) and the music (somewhat happy). A new, spacier, version was called for. I spent most of yesterday creating that version. I didn't have to add a single note (although I probably removed a few); instead I used different instrumentation and moved the parts around. I found a rather strange pad sound to be the basis of the sound and worked from there. 

The beauty of computer recording is that I don't have to record a new vocal track: as long as the new version is in the same key, at the same speed and of the same structure, the existing vocal will automatically sit correctly on top of the music. When mixing, I added a 2kb boost to my vocal along with a hefty amount of reverb. Unfortunately, the leakage which I mentioned previously came back in spades: although there was no drum track with which the phantom rim shots could phase, these phantom shots appeared out of nowhere, coming and going seemingly at random.

Last night I wasn't too convinced about the new version, but having listened to it several times this morning, I think that this will be 'the one'. Of course, I'll keep the original version - it will be a 'bonus track'.

Doppelganger

Out of curiosity, today I googled my name (I used to do this quite frequently) and discovered that there's another No'am Newman living in Bet Shemesh, which is the town just across the road. There's even a film of my double playing the piano! Fortunately, I don't think that anyone is going to confuse us.

Guitar stand

In lieu of something better to write, here are two pictures of the guitar stand which I bought the other day. Due to poor lighting, I took the pictures in the kitchen - not the normal location of the guitar. I admit that I was dubious about a stand which does not support from below, but this stand holds the guitar by its neck; the body rests against foam covered legs.

Post mortem on the Research Proposal exam

'The deductive approach is based on developing an understanding of how a logical chain of events can combine to produce a result, whereas the inductive approach is based on observing events, then explaining them'. This is the sentence which I memorised this morning whilst riding on the train to Tel Aviv, in order to take the Introduction to Business Research 1 (aka "The Research Proposal") exam. I felt that I had most of the material down cold, but last night I was looking at hypotheses and de/inductive approaches, thus making the above sentence as important as 'A paradigm is a series of beliefs and values about research'.

As I wrote before, the structure of the exam is very consistent, and there appear to be only a few questions which can be asked. The first question, 50% of the marks, is critiquing a research proposal; almost certainly the second question (25%) will be about research paradigms and how they relate to the case study and the third question (25%) would be about something else - managing time, methodology or hypotheses.

The case study which appeared in the exam was a very badly written proposal about success in new businesses. Although it took just over an hour to fill six pages of A4 paper, this question was technically very simple (provided that one knows the material, of course!). The proposal is composed of fourteen different sections, so one has to go through each section and offer comments. In a sense, I was fortunate that there was so much which was obviously missing as it made it easier to write the answers. Some of what I wrote was stock sentences (for example, the sections about ethics, deliverables, time needed after the viva and appendices were all stock answers) and some was pre-prepared, like noting that the number of companies to be questioned was not noted and that no sample questionnaire was appended.

The second question was, as I had guessed, about research paradigms and how they affect the case study. In the proposal, the candidate had written about quantitative analysis and standard statistical methods, meaning that he was writing about using the paradigm of positivism (although this word specifically did not appear in the text). My answer first explained what a research paradigm is, then showed the advantages and disadvantages of the two paradigms, positivism and phenomenology. This material was almost word for word the same essay which I wrote for practice the other night, although more concentrated and lacking all the anecdotal material which I thought was unnecessary. I then wrote that in my humble opinion, the research question called for the paradigm of phenomenology - and of course, explained why I thought so. I even wrote a little on how this would change the research.

The third question was how a candidate moves from a wide research area to a specific research question and hypotheses. This isn't material which I had actively revised, although by chance, I had skimmed over some of this material last night so it was fresh enough to enable me to describe competently what programmers call stepwise refinement (the text refers to this as building a work breakdown structure). Once I had built what should have been the candidate's research question (much more focused than the original vague statement), I then explained about null and alternative hypotheses, and even wrote down a few possible pairs (the null hypothesis is what the researcher is trying to prove, such as 'good cash-flow is an indication for success', whereas the alternative hypothesis is the negation, 'good cash-flow is not an indication for success'). At this point, I felt it was time to throw in the sentence about deduction and induction, pointing out the candidate was using induction to prove his case. This may well not have been in the scope of the question, but I had the time.

I finished the exam after two and a half hours of furious writing (interrupted only by a few toilet breaks). I think that I did very well in the exam. I'll probably only get the results towards the end of January - until then, I'm on "holiday": I'd like to know that I passed the first course before I start the second, which will only be examined  in June (hopefully before the graduation ceremony in Edinburgh!).

There were only four students in the examination room (including myself), but three invigilators. Each student was taking a different exam, so we couldn't have copied even had we tried.

It had rained heavily yesterday, so I came prepared with a thick coat and a large umbrella. Fortunately, I did not rain while I was in Tel Aviv, so I was able to walk from the hotel where the exam took place to the Dizengoff centre, where I bought a guitar stand in the music shop. I then walked from there to the train station, cursing my heavy coat and umbrella. But when I got to my local train station, it was raining heavily, so then I was pleased that I had the coat and umbrella. My wife was supposed to pick me up, but she had to go and help her brother and his wife look after their twins (they came home from hospital yesterday). I had to find a taxi.

Almost ready for my first Doctoral exam

One of the movie channels broadcast the newish Richard Gere film "The Double" the other night. I don't think that it's a good film, but today I'm not writing film criticism. At one stage, there is an FBI agent who starts rattling off a series of sentences in rapid fire about null hypotheses (he spoke so fast that I couldn't really understand what he was saying), that "Paul is not Cassius" (you'd have to watch the film to understand). My ears pricked up at the word 'hypothesis', because formulating null and alternative hypotheses is part of any research project and at the moment, my brain is almost obsessed with research projects.

The exam for the course 'Introduction to Business Research 1' is being held tomorrow (Wednesday), so obviously I am up to my ears in revision. Relaxing for an hour by watching 'The Double' reminds me that I have skimmed over the chapter about hypotheses, induction and deduction; these subjects don't seem to have appeared in recent exam questions.

I spent yesterday evening's revision session by writing an essay - without preparation - about the research paradigms of positivism and phenomenology. After writing for about half an hour and covering two pages of A4 paper, I checked my answer against the table which lists advantages of disadvantages of the two paradigms. I could see that my essay tended to the anecdotal and was also missing a few points. I then wrote another, shorter, essay which was straight to the point and covered all the points. By writing such essays, I can get the material into my mind, along with the correct English phrasing. This will be similar to the preparation for the Negotiation course, in which I had stock phrases stored in my mind.

I intend tonight to cover the hypotheses material, just to make sure that it's clear. I'll also check the previous exams to which I have access in order to see whether there were questions about this material and which angle was taken. Otherwise, I intend to have a quiet night. I am not one of the people who sleep during the course and then take the coursebook a week before the exam and burn the information into their brain; I prefer to read the coursebook from the very beginning, letting the information sink in slowly and then reinforcing the absorbed information by reading the material again and again. 

No last minute revision for me: if the material is not in my brain now, it never will be. What is important in these final hours is getting the mental material into a form in which it can be written effectively.

The exam will be held in the same hotel as the MBA exams were; I have been told that sitting with me (but not taking the same exam) will be students repeating MBA exams. I wonder whether I will see any familiar faces. Pyschologists recommend that students imagine the physical setting of the exam; this exercise is intended to prevent anxiety appearing in the critical minutes before and during the exam. The same exercise is recommended for athletes preparing for a race. Whilst I have no problem in visualising the setting, there are a few exams which I don't remember how they finished. For example, I have a memory of leaving the Economics exam and riding back to the bus station in the evening; after the Marketing exam, I walked along the sea front and then all the way to the bus station (quite a long walk!), but I don't have any memory of the Project Management exam. No, wait a minute: part of that memory of the Economics exam is actually the PM exam! Yes, I was sitting in a public taxi and figuring out alternative methods of crashing a project.

Probably some of the above is a bit more muddled than it need be, but that doesn't bother me. All I need is some more revision this evening, a good night's sleep and some light revision tomorrow morning - and then I'll be fit to take on the world!

Uncle once more!

The last few weeks haven't been too good on a personal level; I seem to develop a dull headache on a daily basis, and the hour that the headache starts seems to be earlier every day. Coincidentally, I was asked to monitor my blood pressure a few days before the headaches started, and the high levels that I am seeing may have something to do with the headaches. Acupuncture helps relieve the ache but only for a few hours at most. My family doctor can be very elusive: I was supposed to see him today but now will only see him on Tuesday.

But there is good news - my sister in law gave birth to twins (a boy and a girl) two days ago! Here is a picture of the girl who weighed about 2.8 kg at birth. So far she hasn't shown much

enthusiasm for eating but hopefully that will change shortly. Her brother weighed only 1.9kg at birth and is currently with the premature/under-weight babies, although he's not in an incubator.

This is the sister-in-law which I wrote about five years ago when she gave birth for the first time to a premature baby who died after a week. The odds seem much better this time around! In the mean time, she has also married my brother in law (my wife's brother).

Warming up for DBA exam

I see that I've hardly written anything about the first course in the DBA degree - Introduction to Business Research 1. This course gives an introduction to the structure of the doctoral degree then concentrates on the first piece of work that the candidate (we are no longer students!) has to produce - the research proposal. Chapters in the course discuss the background of the proposal (including a very interesting chapter on the philosophy of research), culminating in a chapter on the structure of the research proposal itself.

I had been reading the course material with great interest, always considering how the material applies to my chosen area (what is my research question? what are my hypotheses?). But as there are only six chapters to read, I found that I had read the whole course twice two months before the exam. As a result, I almost totally ignored the course for a month. A week ago, I awoke with a start and realised that soon I am to be examined on this material. It's time to get hard core.

The most important thought which I had a few days ago was that I am not to be examined on my research proposal but rather on how well I have learned the course: two completely different things. My first step, then, was to download a few previous exams and check both the examiners' solutions and selected students' solutions. After reading a few of these, I went back in time to the final stages of the Strategic Planning course, in which the lecturer would hand us an old exam and ask us for the solution, which was according to a template that he devised.

After examining a few papers, it became clear that the exam always consists of three questions: the first (50%) presents a research proposal and the examinee is to critique it. The other two questions (25% each) require essays on subjects such as research methodology, philosophy and time planning. The research proposal question can be answered by creating a template and then following it; the proposal consists of fourteen different sections (abstract, research question, research methodology, timetable, ethics, deliverables et al.) and most of these sections have a pre-defined structure. Thus it should be fairly easy to deliver a critique after memorising which points should be referenced in which section (the ethics, deliverables and appendix sections are particularly easy).

I spent most of my study time yesterday creating such a template and then answering - writing out in full - one such exam question according to the template. I intend to answer a few more questions in the next few days in order to fix the template in my mind. Writing the answer out in full hand is a very important aid to fixing the answer; I very rarely write anything by hand these days and certainly not two page essays. As opposed to the MBA courses, I don't attend lectures and write notes on the coursework, so at the moment I have no muscle memory; I consider this to be very important.

When I need some variety, I will look at the other questions in the exams and try to prepare answers in advance - for example, the course text devotes a few pages to the advantages and disadvantages of the positivism and phenomenology approaches (I will have to practice writing phenomenology and methodology!) and I imagine that there will be a need to discuss them at some point in the exam.

That said, there are some subjects which seem fairly obvious: I am sure that I could write a competent essay about time planning (Gantt charts, etc) without having to read the material again; I did achieve a high mark in Project Management, after all!

Inside the DOCU program (3) - Saving screenshots

Whilst discussing the DOCU program last week, the Occupational Psychologist asked why she can't store screenshots along with the written documentation. I wasn't about to tell her that the program was implemented with a RichEdit component, which implies very strongly that only rich text can be saved, so I said that I would consider the options.

The simple part of the option which I found was to add a field in the database of 'blob' type; such a field can store anything as a sequence of bytes. Theoretically I know how to display jpg images stored in a database as one of our programs displays images, and even more theoretically I know how to save such an image into the database.

The solution actually required several items

• adding a blob field to the database
• learning how to save an image from the clipboard to a picture component in Delphi
• saving that picture to the database
• clearing a picture from the database
• loading the contents of a blob field to a picture
• creating a suitable form from the 'entry' form and closing it appropriately

From the outset, I decided that each entry in the database could have only one picture assigned to it; otherwise this would add complexity (Should the user choose which picture to display? Should all pictures be displayed for this entry? How to manage the pictures?). Thus I was able to add one icon to the toolbar (see yesterday's blog) instead of having to add an image choosing dialog.

My logic was as follows: if the entry has an assigned picture, then open a son form and display the picture. Otherwise, open a son form, copy into it whatever graphic is in the clipboard's memory, display it and save it to the database. There also should be an option to clear the picture in case a wrong graphic was saved.

Here's the code which does all of the above.

Procedure TDoPicture.Execute (const s: string; aleft, atop: longint);
var
 empty: boolean;
 bmp: TBitmap;
 ms: TMemoryStream;
 j: TJPEGImage;

begin
 caption:= s;
 left:= aleft;
 top:= atop;
 with qGetPicture do
  begin
   params[0].asinteger:= myentry;
   open;
   empty:= isempty;
   if not empty then
    begin
     j:= TJPEGImage.Create;
     try
      j.Assign (qGetPicturePIC);
      OnAssign (j)
     finally
      j.Free;
     end;
    end;
   close
  end;

 if empty and Clipboard.HasFormat (CF_PICTURE) then
  begin
   bmp:= TBitmap.Create;
   j:= TJPEGImage.Create;
   try
    bmp.Assign (Clipboard);
    j.Assign (bmp);
    OnAssign (j);
   finally
    bmp.Free;
    j.free
   end;

   ms:= TMemoryStream.Create;
   try
    Image1.Picture.Graphic.SaveToStream (ms);
    ms.Position:= 0;
    with qSavePicture do
     begin
      parambyname ('p1').asinteger:= myentry;
      ParamByName('p2').LoadFromStream (ms, ftGraphic);
      execsql
     end;
   finally
    ms.Free;
   end
  end;
 show
end;

procedure TDoPicture.OnAssign (j: TJPEGImage);
begin
 Image1.picture.assign (j);
 image1.autosize:= true;
 // resize the form to be slightly larger than the stored image
 width:= image1.Width + 16;
 height:= image1.Height + 48;
end;

Looking at the above now, it all seems so obvious, but it wasn't when I was writing the program! Taking it bit by bit:

• the first few lines are concerned with setting up the form with the correct caption; the form is placed at the same height and to the right of the calling form
• a query is then opened which checks whether there is an assigned picture. If the query is empty, then the 'empty' variable becomes true. If not empty, then the next few lines show how to read a blob from the database, store it in a temporary variable and then cause the blob to be displayed as a jpg image. The 'OnAssign' procedure resizes the form to be slightly larger than the image.
• If there is no assigned picture, then the program checks to see whether the clipboard holds graphic data. If so, the image is first saved in a temporary bitmap (bmp) and then converted into jpg format, which is then displayed. Afterwards, the image is saved to the database, an action which requires the use of a stream.

As I wanted to have the user interface as simple as possible (ie no user interface at all), I was stumped at first how I was going to allow the user to remove the stored graphic. Then I remembered that I have been using the system menu frequently for such actions; the code became very simple to write.

At the beginning of the form, the following adds an option to the system menu

const
 SC_Original = WM_USER + 2;

procedure TDoPicture.FormCreate (Sender: TObject);
var
 SysMenu: HMenu;

begin
 SysMenu:= GetSystemMenu (Handle, FALSE);
 AppendMenu (SysMenu, MF_STRING, SC_Original, 'Clear picture');
end;

And this is how a click on that menu option is handled

procedure TDoPicture.WMSysCommand (var Msg: TWMSysCommand);
begin
 if Msg.CmdType = SC_Original then
  begin
   qClear.params[0].asinteger:= myentry;
   qClear.ExecSQL;
   image1.picture:= nil;
   close
  end
 else inherited;
end;

In an MDI application, the program 'knows' about its child forms because they are 'stored' in the MDIChildren array. As DOCU is not an MDI program, another solution is required; Delphi stores all on screen forms in the 'forms' property of the 'screen' variable. So closing the picture form when its parent form is closed requires the following code

 for i:= 1 to screen.FormCount do
  if screen.forms[i-1] is TDoPicture
   then if TDoPicture (screen.forms[i-1]).myentry = entry
    then screen.forms[i-1].close;

Here's a pretty useless screenshot of an entry form along with its graphic; what would seem to be WinAmp playing in a window is actually a screenshot of WinAmp captured and stored in Docu.

Inside the DOCU program (2) - saving and loading rich text

Yesterday I discussed the gross structure of the DOCU program; today I am going to concentrate on the 'entry' form.

Originally, the form contained a dbMemo component whose font was fixed at compile time. Obviously at some stage I received a request to change the component so that coloured and or bolded text could be stored. The dbMemo component doesn't allow this but the dbRichEdit component does. Fortunately for me, one of the demo programs which comes with Delphi is an editor based around a rich edit component, so I was able to extract from this demo most of the necessary interface code. With the use of the tool buttons, the user is able to choose type face, font size, bolding, italics, underlining and colour as well as numeration.

If everything were as easy as this, then I would be out of a job. The problems began when I wrote the code to store the text in the database; when the saved text was restored from the database, all the richness (ie fonts and colours) had been lost!

The answer is to be found here; unfortunately, there is no real explanation of why the code succeeds whereas the naive code fails. 

For loading the rtf text:

//Get the data from the database as AnsiString
rtfString:= sql.FieldByName ('rtftext').AsAnsiString;

//Write the string into a stream
stream:= TMemoryStream.Create;
stream.Clear;
stream.Write (PAnsiChar(rtfString)^, Length(rtfString));
stream.Position:= 0;

//Load the stream into the RichEdit
RichEdit.PlainText:= False;
RichEdit.Lines.LoadFromStream (stream);

stream.Free;

For saving the rtf text:
//Save to stream
stream:= TMemoryStream.Create;
stream.Clear;

RichEdit.Lines.SaveToStream (stream);
stream.Position:= 0;

//Read from the stream into an AnsiString (rtfString)
if (stream.Size >= 0) then
 begin
   SetLength(rtfString, stream.Size);
   if (stream.Read(rtfString[1], stream.Size) <= 0) then
   raise EStreamError.CreateFmt('End of stream reached with %d bytes left to read.', [stream.Size]);
end;

stream.Free;

//Save to database
sql.FieldByName('rtftext').AsAnsiString := rtfString;

Inside the DOCU program (1)

I haven't abandoned this blog; I've simply been going through a few weeks of nothing much interesting happening so there's been little to write about. Over the weekend, I added an interesting twist to an existing program of mine, so I've going to devote a few blog entries to this program.

A year or so ago, the Occupational Psychologist (OP) with whom I work and I had a discussion about program documentation, probably after she asked for some feature and I had told her that I had already added that feature and explained to her how it worked (this happened again a few weeks ago when I was asked to add a feature which I had added in March). Out of this discussion was born the DOCU (as in documentation) program.

From a bottom up point of view, this program can store 'entries' which are connected to 'subjects' which are connected to 'programs'. From the top down point of view, a program - such as the management program - might have several subjects (such as customer calls), and each subject can have several entries (printing the text of a call, creating a linked call, etc). The main form of the program is simply a dialog box with three different grids (one grid per table) which are of course synchronised. There is nothing revolutionary about this so far.

The first interesting twist in the program came as an attempt to find a solution which utilised the Stack Overflow gurus' dislike of MDI programs (such as the management program). It turns out that this was simpler than I had expected. Whilst the main program has two modal sub-forms (one for adding programs to the database and one for adding subjects; under the hood, they're the same form), the 'add entry' form is non-modal. This means that any number of such forms can be displayed on the screen at the same time, when each form can be derived from a different subject. The main form's size is fixed (because it is built on three grids); had I used MDI, then the 'add entry' forms would have been constrained to the area of the main window (which in practical terms means that the main window has to be maximised). Using a non-MDI main form, the non-modal 'add entry' forms can appear anywhere on the screen.

As per the maxim that one picture is worth a thousand words, here is a compressed screenshot of my computer screen with the program running and three 'add entry' forms displayed. As most readers won't be able to read Hebrew, it may be difficult to recognise which is the main form and which are the child forms; the form with three grids is the main form.