Busy morning

Despite today being Saturday, and thus my one day off from work, I was up and awake at 6:45 am. The first item on my agenda was to watch something which I had taped yesterday on the video. During a few idle moments on Friday, I had turned on the tv and was pleasantly rewarded by the sight of Paul Simon; judging by the mustache and the song he sang ("Loves me like a rock"), I had tuned into a repeat of "Saturday Night Live" hosted by PS circa 1975.

After that song, there was a comic interlude with Simon playing one on one basketball with some pro from the NBA who was about two feet taller. Amusing but predictable. Then cut to Simon singing a song which seemed familiar but wasn't one of his ... wait a minute, it's "Marie", by Randy Newman. After one verse of this, Simon says that it is indeed "Marie", written by one of his favourite songwriters, and he so much liked Newman that he (PS) was able to persuade him (RN) to appear. Cue to Randy Newman playing "Sail away"!

Continuing this level of surprise, next on was Art Garfunkel for two duets, "The Boxer" and "Scarborough Fair". Simon put in a few "friendly jabs", like asking whether Garfunkel was done with his film career and whether he would be upto singing a few songs. I've never been too fond of "The Boxer", probably because of its over-dramatic arrangement, but with just the two of them and one acoustic guitar, the song was much better. It was very easy to hear what each was singing, which was a revelation as I've never heard the harmonies before.

The next item on the agenda was to master a live recording of Randy Newman made only a few days ago. RN is currently on tour in North America, featuring some new songs, and someone had uploaded a recording of his appearance in Toronto a week ago (14 Oct). I had downloaded it and intended to check volume levels before burning some audio copies. It's been my experience that such recordings, if not edited, tend to have very loud applause and very quiet music. This was indeed the case: the applause was approximately four times as loud as the music, and overloading the audio.

After a few experiments, I decided to boost the non-applause sections by 100% (ie make them twice as loud) and compress the applause sections to 50%. Whilst there are some small patches with audible volume changes, these are very short and don't distract from the performance.

New songs! One is called "I'll Never Get Over Losing You", which has nice lyrics but not an outstanding tune. The other is called "(A few words) In Defence of My Country", which is very topical and very funny. It accepts that America's current leaders may not be much, but they're nothing compared to some of the kings, dicatators and leaders that have existed over the past thousand years. Supposedly the new album will be released sometime during 2007, although no one knows quite when.

Finishing up an early morning full of activity, I set to work programming, adding a new feature/report to a program which I have developed to aid a clinical psychologist. This new report used some nice SQL features and looks very nifty! I imagine that the general public doesn't appreciate the niceties of programming, but I certainly enjoyed myself, and the results are pleasing.

That's enough for one day. Now I can rest and shore up some energy for the coming week.

Algorithms, part 2

My original idea didn't work too well, but I've thought of another way to solve the problem, which again divides the work into three:

first query: parts which have different values in both months

second query: for all parts which had a value in the first month, see whether they have a value in the second month

third query: for all parts which had a value in the second month, see whether they have a value in the first month

The second and third queries are similar in nature to my original method, but this way should be faster as the program is not iterating over all the parts, but merely a subset.

Last week was holiday week in Israel. I spent two days learning a new ERP program (and only scratching the surface), and not doing much else with the rest of the time. So there's nothing much to report here.

Strange dream leads to improved algorithm

People don't normally remember their dreams for long, so I'm setting this one down before I forget it.

I was with a group of people (I think that we were students on a course together) who had to run around a ship (some structure with at least two floors or decks) and improve our time for the course. Normally it was taking around five minutes, but I managed to run the course one time in four minutes. Speaking to the group (rather pompously, I admit), I started drawing parallels between how I had managed to improve my time by 20% and tuning computer algorithms. Small increases in performance can be achieved by looking at each part of the algorithm and applying local optimisation. In the dream, an example of local optimisation was the way I was descending a ladder between decks; normally I was climbing down the ladder 'properly', putting my foot on each rung. The local optimisation was to slide down the ladder, holding on with my hands but not using my feet.

The problem with local optimisation, I explained to the group, is that there is limited scope for improvement. True, it may be possible to end up with a time which is maybe five times as fast as the original, but in order to gain true improvement, one has to find a better algorithm. In computer terms, this is (for example) one which works in logarithmic relation instead of linear relation.

Then I woke up. Still thinking about changing algorithms, I started thinking about a problem at work. I started programming over twenty years ago, when computers were relatively slow, lack of memory was a major restraint, and it was very important to tune one's algorithms in order to obtain maximum performance. I don't work in a field which still requires high performance from the cpu, so these days, such matters seem unimportant. But I do know one report in one program which could do with some improvement, and not surprisingly I was working on it with no success the other day.

I have a database which maintains data about dead stock at work (defined as parts which haven't been bought for the last six months but the stock amount is more than twelve months usage), and one of the reports which I produce is a list of changes in the last month - what's been added (not good) and what's been used (good). At the moment, I'm using a simple, straight forward and brute strength approach to this problem: for every part in the database, I get the amount for one month and the amount for the previous month. If the amounts are different and not zero, I display the result.

This works, but it is incredibly slow. I've been trying to find a masterful SQL statement which will allow me to do what I want, but so far I've been unsuccessful. This morning I woke up with a new approach which involves three queries and may seem more complicated, but should be much faster than the simple approach, because the number of parts being looked at is much smaller. First of all, get data for the parts which had a value in both months where the values are not the same and not zero (I have a SQL query which will do this). Then get data for the parts which have a value for this month but not for the previous month. Finally get data for the parts which had a value last month but not this month. The resulting combined list will be the result set which I require.

Of course, I will check that the results which I get match the results which I was getting via the simple approach. I will also time both methods and see which one is faster, even though I'm sure that this new approach will be better. What might be construed as funny - or counter instructive - is that I am spending hours on finding a better approach for a report which will save maybe five minutes per month. Looked at in this light, it's not worth the bother to improve the time.

But it's an intellectual challenge, and that's what's important. Keep burning new synaptic connections in the brain. Thinking about thinking can sometimes be more valuable than simply thinking.

I was thinking about thinking, but it didn't really get me very far - Peter Hammill ("Black room")