Extending the auxiliary program

Pretty keen - yes, my hobby keeps me busy
and if I talk to myself, what's the crime?
- Peter Hammill, "Last frame"

After a good night's sleep, my right brain came up with an interesting idea that extends yesterday's solution ¹ of an external program to calculate correlations. What would happen if I had another calculation that required heavy processing? Would I create a new external program with its dedicated messages?

No: all I need to do is create a general, auxiliary, program to which is passed a string of parameters, where the first parameter is the function number within the auxiliary program to be executed. When that function completes, it sends a message back to the main program, where one parameter is the function number and the second parameter is the data instance number. This reminds me vaguely of the old DOS Int 21h interface.

As this auxiliary program is going to support a variety of calculations, it makes sense to have certain portions global to the program - the initialisation of the database along with the NewInstance and CloseInstance function/procedure. Then all the specific correlation code can go into one procedure with its local variables.

As I mentioned yesterday, I am using the InterBase components; this makes initialising the database easier especially as I am writing a text-only unit. One major difference between the InterBase components and the dbExpress set is that the first set uses transactions. I discovered in a painful manner yesterday that in order for a query to insert or update data in the database, the transaction has to be committed, otherwise nothing happens. Not only that, the transaction has to be set as non-active after the commit so that it can be reused. This flag setting seems to be required after every database action, regardless of whether its an insertion or a selection. This has tripped me up several times, and I'm not sure if I am yet capable of writing a sequence of SQL commands without getting an error message about the transaction.

As this blog's epigraph puts, "yes, my hobby keeps me busy". I enjoy programming - or at least, defining what has to be programmed. As code rarely works the way it is intended first time, I have to debug a certain amount/great deal, and this can be tedious.

Internal links
[1] 1839

---

This day in history:

Blog #	Date	Title	Tags
294	17/10/2010	Sweet and sour chicken	Cooking, Slow cooker
295	17/10/2010	Project Management course	MBA, Project management
417	17/10/2011	Getting ready for the Kindle	Kindle, E-book
983	17/10/2016	Who's watching who?	Grandfather
1349	17/10/2020	Completing the story of porting an application to Windows 10/Delphi 10.2	Delphi, Unicode

Removing the thread code and finding a different solution

After several hours of testing, I came to the reluctant conclusion that nothing works in the program when this type of thread is executing - somewhat negating the whole point of having a thread! I've restored the program to how it was on Saturday night: the interface form calculates the correlations and saves the data in an instance of the temp table. A secondary form is called to display the data.

I've been experimenting on a separate copy of the program units, and it appears that something connected with the database code in the thread is causing the problem. I'm using completely different database components and not accessing any units whatsoever but there the problem still exists.

So, I wondered, what else can I do? An old solution popped into my head: execute a separate program to do the work. This program can be passed whatever parameters are necessary, and can send a message to the main program that data is waiting to be displayed. But it's not always necessary to execute a separate program: if there aren't that many data points to be calculated then it's quicker to do this within the existing program. I determined that - on my computer - 2500 records is approximately the tipping point.

The first part of the code - entering into the temp table the records that will act as the input for the correlation code - remains in the interface unit. Instead of having to issue a separate query in order to determine how many tuples have been inserted, I simply store the return value of the ExecSQL function. Almost always I ignore the return value, which is the number of tuples affected by the query. In this case, the number of tuples will be the number of tuples inserted which is the total number of examinees. 

If this number is less than 2500 then the original code executes within the interface unit; at the end, the son form is created directly to display the data. Should there be more than 2500 tuples, then first a string is built with all the parameters that need to be passed, eg '21 15 12 01/01/22'. Then the external program is executed, receiving this string. It turns out that almost all of the thread code that I wrote could be transferred without change into the external program, including all the database code. In order to simplify matters for myself, I am using the InterBase components as opposed to the dbExpress components that I normally use. Again, this program does not include any module from the main program.

The final line in this program is 'sendmessage (hwnd_broadcast, RegisterWindowMessage ('Show correl'), rinstance, 0)'; this sends a message to every open program on the computer (because the program doesn't know who executed it) with a message number that is not necessarily constant; also using the RegisterWindowMessage technique means that the message number is known only to this pair of programs. I discussed¹ this Windows technique way way back in 2009. The call arrives at the overridden DefaultMessage handler that calls the secondary form to display the data.

Finally, success.

Internal links
[1] 176

---

This day in history:

Blog #	Date	Title	Tags
1348	16/10/2020	PrioXRef 3 - insert without parameters	Programming
1538	16/10/2022	Indexing blogs	Programming, Meta-blogging, Blog manager program
1678	16/10/2023	The running grave	Cormoran Strike

Terminating threads

Yesterday¹ I wrote about threads, noting that the only problem that I've found seems to be that if the window with the user interface (which calls the thread) is closed before the thread finishes its work, the thread appears to die. I searched the web for a description of this problem and how to solve it. 

This tutorial holds the answer. Although it's written describing the case of a main program and a thread, it appears to be suitable for a 'son' form that is calling a thread. The first thing to do is to maintain a count of threads invoked by the form

Constructor TCorrel2Scales.Create;
begin
.
.
.
threadcount:= 0;
end;

Whenever a new thread is created ('DoCorrelThread (id1, id2, fdt ...'), the variable 'threadcount' is incremented. Two procedures have to be added: one is private ('HandleTerminate') and one is an event handler ('FormCloseQuery') - as follows

procedure TCorrel2Scales.HandleTerminate(Sender: TObject);
begin
 Dec (threadcount);
end;

procedure TCorrel2Scales.FormCloseQuery(Sender: TObject; var CanClose: Boolean);
begin
 if threadcount = 0
  then canclose:= true
  else canclose:= false;
end;

No change was necessary in the thread unit. I tested this by first running a correlation of two scales from 2018 - many data points - before I added the above code (actually I had added it but neglected to connect the 'FormCloseQuery' handler to the 'OnFormCloseQuery' event) then closing the form (TCorrel2Scales) before the thread terminated. This caused the program to crash. After I connected the event to its handler, the same test worked - the form would not close until the thread had terminated.

So now I know.

Addition from a few hours later: it seems that I missed something in the tutorial. The thread has its OnTerminate event point to HandleTerminate, but in this example the thread is created within the calling form and so this works. In my case, the thread is created in a separate unit and so it would seem that there is no value that can be passed to the thread's OnTerminate event. 

This is solved by defining a new type, TMyProc = Procedure (sender: TObject) of object. Then I can pass the HandleTerminate 'variable' to the thread code. In non-Delphi environments, this is known as a callback and occurs quite frequently in Windows code; even I've used it albeit very rarely.

While all of the above prevents closing the interface form while there is still a thread in existence, it doesn't solve - in fact, dodges - the original problem: how can I close the interface form while there is still a thread in existence. I suspect that the solution is to have the main form create the thread. I tried this but it appeared not to work: I still could not close the interface form. I shall have to work on this some more.

Internal links
[1] 1837

---

This day in history:

Blog #	Date	Title	Tags
140	14/10/2008	Where have I been all these months?	Literature, Peter Robinson, David Lodge
415	14/10/2011	Dieting has a number of destructive side effects	Martin Seligman, Diet, Acupuncture
516	14/10/2012	Friday is cooking day	Cooking
765	14/10/2014	Some days you're the pigeon and some days you're the statue	DBA, Films
1083	14/10/2017	The seven stage model for developing enhancements	DBA
1266	14/10/2019	Priority: LIKE cannot accept a variable	Priority tips
1430	14/10/2021	Potassium levels in food	Health, Food science, CPAP, Blood pressure
1536	14/10/2022	Gillian McPherson	1971

Correlations and threads

I have a non-specific memory of sitting at my desk when I was a student in London, calculating correlations that I needed for graphing data. Correlations measure the connection between two non-connected variables; if one increases as does the other by the same percentage, then there is a perfect correlation between the two. The two variables can have no correlation whatsoever or they also could have a negative correlation (one goes up while the other goes down). I don't recall how I learned about correlations - presumably we had a course in statistics during the degree studies, but I have no memory of this whatsoever. Correlation is another arrow in the statistics quiver*.

On Friday, in the context of our flagship database that stores data originating in psychological questionnaires, the OP wanted to know the correlation factors between two scales. This is done by first finding all the clients from a given date who have values for both scales, then calculating the mean and standard deviation for each scale. Then one sums (the difference between the observed value for one scale and its mean) multiplied by (the difference between the observed value for the other scale and its mean); this value is divided first by the product of the two standard deviations and then by number of data points (i.e. clients) less one. A good explanation of this can be found here. I wasn't fazed by her request as I knew about correlation.

Enough mathematics. Two years ago, I added a correlation function in the program for clients who had answered the questionnaires twice, in order to see if there was any change in their attitude to life (no change would mean perfect (1.0) correlations between the two values). I gave a brief look at what I had written there and shuddered as the code seemed over-complicated. These days I have temporary files¹, so the task was somewhat simplified. I wanted to write a query that would return the sum of the multiplied differences, but I messed this up and instead used a cursor that would iterate over the population (i.e. clients) and calculate the total line by line. This worked.

In the afternoon, I took the dog for her afternoon walk. As usual, my mind considered² the code that I had written in the morning and I realised where my mistake had been in the query (brackets in the wrong place). I then realised that I could use the idea that originated in the blog manager and then transferred to the documentation program of showing the results in a separate window. This way, one can run several correlations and see the differences between them (the original version showed the values calculated by the current parameters; using different parameters would cause the earlier values to be 'lost'). Good idea.

As the Yom Kippur fast was approaching, I didn't have any time to implement these changes, but I wrote the ideas down for later. During the fast, with nothing very much to do (I don't listen to music, work on the computer or watch the television; I either read or rest), I considered the ideas for future implementation. I noted to myself that the correlation function requires quite heavy calculating that grows according to the number of data points (a perfect correlation!) and during this time, the program is not responsive. Could I create a separate thread and have the calculation performed there? This would free up the user interface, making it more responsive. Assuming that this is possible, how would the thread tell the main program that it had finished and that the results could be displayed? Could the thread send a message³ to the main program that would cause a window to open with the newly calculated data?

I have written thread code before with Word⁴ and with Excel⁵, but in these cases, the database 'work' was done in the main program and the data extracted was passed to the thread. There is no display problem as either Word or Excel open with the data. I don't know enough about threads to know in advance whether my idea would work.

When the fast had finished and I had drunk and eaten sufficiently, I got to work. First I corrected the correlation code then I added the use of another temporary file and the separate windows showing results. This worked perfectly. Then I created a new unit that had the interface code of the previous unit, but passed the parameters from this unit onto a thread. All the computation code went into the thread; this also means that queries have to be created manually and their SQL code added in the text. I was initially dubious as to whether the queries would need an SQL connection defined within the thread, but using the connection defined in the data module worked fine.

Once I got all the syntax problems sorted, I added a final line to this unit:

sendmessage (mainhandle, WM_ShowCorrel, rinstance, 0);

This is meant to cause the main program to open up a specific window and pass to it the data contained in the temporary table with instance 'rinstance'. To my surprise, this almost worked correctly the first time; in the main program I was checking the value of 'wparam' instead of 'lparam'. Once this was fixed, all the pieces worked together perfectly.

The only problem that I've found seems to be that if the window with the user interface (which calls the thread) is closed before the thread finishes its work, the thread appears to die. I don't know enough about threads to know whether this is standard behaviour or some kind of bug; the thread doesn't have to 'know' who created it, only how it was created (i.e. the parameters). I'll have to find some sources to read about this.

Now I can look for other places that have heavy calculations (mainly those calculating means and standard deviations for a few thousand values) and see how I can use the thread and message method to ease the user interface.

* When I was thinking about writing this blog, the word 'quiver' had disappeared from my brain. It's as if the left hemisphere of my brain created a thread to find the word, as about fifteen minutes later the word 'quiver' popped into my mind when I was thinking about something else. Very appropriate.

Internal links
[1] 1548
[2] 1829
[3] 1310
[4] 1436
[5] 1443

---

This day in history:

Blog #	Date	Title	Tags
139	13/10/2008	Fotheringay 2	Sandy Denny, Fotheringay
414	13/10/2011	Kindle	Kindle
640	13/10/2013	Tasks	Programming, Delphi
1265	13/10/2019	Acting like an MBA (a 'suit')	Personal, MBA
1677	13/10/2023	Time out	Personal, Song writing

Is this a sign that I'm getting old?

In a dream that I had a few days ago, I was trying to remember the word (in English) that is used sometimes in financial terms, despite the fact that its normal meaning is 'non-religious'. After spending some time in bed trying to remember, I eventually got up and went to the computer, at 2 am. The word that I was looking for was SECULAR.

When I awoke, I checked my memory - and the word was gone. I went back to the computer, checked my browsing history and found the word again. This time I wrote it down on a pad by the computer. 

Over the next few days I would check my memory and most times I could not think of the word. Eventually I used a mnemonic - Dracula, although 'regular' would be better - that was supposed to remind me of this word. I could remember the mnemonic but about half of the time I couldn't make the connection between Dracula and secular. It's as if the word has completely disappeared from my brain, or as if my brain is not prepared to store it.

And it's not as if I've never used the word: I'm sure that in my youth I used it frequently (when describing my youth movement) and a quick check shows that four blog entries over the years also contain this word (the last time was just over a year ago).

This is somewhat frightening as I am a person whose livelihood is dependent by no small amount on a good memory. This short-term and very specific amnesia may have been caused by the viral infection that has been plaguing me for the past few days - at least I'm hoping so.

I shall try and find sentences containing 'secular' (like "This person was born secular but turned into an observant Jew several years ago") in order to constantly remind me of this word. A song lyric would be best but I can't remember any song containing this word - even the erudite Peter Hammill has yet to use 'secular'.

This reminds me (a poor choice of verb) of a story by Robert Silverberg, 'How it was when the past went away': terrorists placed amnesifacients in San Francisco's water supply, causing people to forget. But the drug affected people differently: one person forgot who he was, one person remembered everything except for his wife, one person lost the previous nine months, and one - the example that I want to use - is a professional memory artist, the sort of person who can recall the seventh page in the eighth chapter of some book. Some of his memories are intact, but others ... are gone. In the end, he commits suicide (for some reason, I can't locate an e-copy of this story; I certainly have it in print).

That's how I felt: almost all of memory was intact but a certain dictionary entry had simply disappeared and I couldn't restore it. 

---

This day in history:

Blog #	Date	Title	Tags
17	08/10/2005	Bar mitzva	Bar mitzva
205	08/10/2009	The girl who played with fire	Literature, Steig Larsson
292	08/10/2010	Misinformation	ERP
890	08/10/2015	How the mind works	Psychology, David Lodge
1429	08/10/2021	DBA: Pilot study 2	DBA
1674	08/10/2023	The blackest day for 50 years - notes from a peripheral participant	Israel

A year after the massacre

After nine months of needless internal conflict that only encouraged them, our despised enemies took to the stage a year ago and changed our world for ever. We will not forget. We will not forget those who were slaughtered, we will not forget those who were captured and tortured, and we will not forget those for whose return we pray daily.

 

---

This day in history:

Blog #	Date	Title	Tags
56	07/10/2006	Strange dream leads to improved algorithm	Programming, The brain
204	07/10/2009	Pressure cooker	Cooking
1343	07/10/2020	Completing the second version of PrioXRef	Programming

The swimming season finished rather abruptly

Formally the swimming season was to have ended last Saturday, 28 September, but the powers that be decided that there would be three extra sessions: on the morning of the New Year (Wednesday), Friday (i.e. yesterday) and Saturday (i.e. today). 

On Wednesday morning I walked dutifully to the pool and swam the mandatory 20 lengths in fairly cold water. We spent the evening of the New Year with my daughter in law's brother and family, and the second night of New Year with my family. Towards the end of that Thursday evening, I felt a familiar tickle in my throat signifying that I probably had 'a viral infection of the upper respiratory system'. I slept over ten hours that night which is very unusual for me; it was clear that I had no strength to go swimming on Friday morning. I barely had the strength to take the dog for her morning walk that I shortened considerably. Ditto yesterday evening and this morning.

So the swimming season finished more abruptly than I would have liked. With a little SQL mojo, here are the data for the past few years, since I started documenting each swim. In 2019, I was swimming only once a week, but since 2020 (the Covid year), I've been swimming twice a week.

Year	Number of swims	Total lengths
2019	16	438
2020	24	562
2021	30	548
2022	29	567
2023	31	561
2024	34	710

My nose is running and my eyes are hurting, so I assume that's the end of this blog entry.

---

This day in history:

Blog #	Date	Title	Tags
16	05/10/2005	Sudoko	Programming
203	05/10/2009	Old computer in new case	Computer
411	05/10/2011	Firebird DB management tool (3) - Bells and whistles	Programming, SQL, dbExpress
979	05/10/2016	DBA: Entering the final third of the doctorate	DBA
1079	05/10/2017	A kibbutz day	Jewish holidays, Obituary, Kibbutz
1178	05/10/2018	Geoff Emerick, 1945-2018	Obituary, Beatles
1342	05/10/2020	Continuing the development of the Priority Cross Referencer (PrioXRef)	Programming
1428	05/10/2021	Continuing the BP saga	Health, CPAP, Blood pressure, Aldosterone
1672	05/10/2023	The neuroendocrine system and my blood tests	Health, Nutrition

Slow cooking with the barbecue mat

A few weeks ago, I wrote¹ about ordering the 'Reusable Non-Stick BBQ Grill Mat' that finally arrived on Tuesday evening whilst I was rehearsing for the New Year Ceremony. I managed to extract the parcel from the kibbutz 'Post Office', and yesterday evening I had the chance to use it in the slow cooker. 

Below are two pictures: one with meat and mat, and one with vegetables on top of the mat.

The mat can be seen quite clearly in the picture without meat (left) and less clearly in the picture with vegetables - obviously because the vegetables are hiding the mat. It can be seen in the top right hand corner of the picture.

Did the mat do its job? Definitely yes! When cooking finished, I was able to scrape the cooked vegetables from the mat without them getting entangled with the meat. I then pulled the mat out of the slow cooker (this is slightly difficult as the sides of the cooker are very hot) then scooped the meat out. I think that next time I will add some string (or similar) to the mat to facilitate pulling it out of the cooker; if I put string on opposite sides then maybe I'll be able to pull the mat out with the vegetables still on it, making it even easier to decant them into a dish.

Both the meat and vegetables were cooked to perfection, so the mat had no effect on this, as I suspected. I think I know why sometimes one or two lumps of meat are cooked but are not tender: the amount of meat that I bought (1.6kg), when cut, filled out the bottom of the slow cooker, but had there been more meat, this would have been placed on top of the bottom layer and would not necessarily be covered in liquid. Something to bear in mind for next time.

Internal links
[1] 1817

---

This day in history:

Blog #	Date	Title	Tags
410	04/10/2011	Watching the weight	Diet, Acupuncture
514	04/10/2012	Blues for S	Jewish holidays, ClientDataSet, Reason, Slow cooker, Acupuncture
978	04/10/2016	New Year greetings	Jewish holidays
1078	04/10/2017	Another end to headaches	Health, Migraine

Apples, honey and missiles

For the past few years, the kibbutz has been holding its celebration of the Jewish New Year a day in advance, so allowing everybody to celebrate without inconveniencing those who will usher in the New Year with relatives, either on or off the kibbutz.

So yesterday I finished work just after 4:30 pm, took the dog for a short walk, showered (the day had been overcast but humid) then went down to the dining room to set up equipment and have a short rehearsal. The 'show' was due to start at 6:30 pm; we may have actually started on time (my phone was turned off so I couldn't check). The hall was filled with people of all ages - from new born children (they even got their own dance, whilst being held by one of their parents) to great grandparents.

 

Although it's very hard to tell, this picture comes from the final song of the evening, entitled with one of those Hebrew words that is very hard to translate; Google Translate says that it means "I wish", which is OK but doesn't go far enough in expressing the wish for a better outcome (or future, as you wish). I can tell, because if I magnify the picture, I can see a capo on the first fret. All the songs went down well and I'm sure that everybody present enjoyed themselves. 

We had just finished packing away the equipment when suddenly we were told - go to the air raid shelters! There are missiles from Iran in the air! I have to admit that the Iranians were very considerate: had they sent their missiles fifteen or twenty minutes earlier then our entire evening would have been ruined. Had they waited another ten minutes, then I would have been at home in our security room. Of course, the best option would have been no missiles at all. Not only that, Iran seems to have had a failure in intelligence: had they wanted to inconvenience the population, it would have been much better to send the missiles on the eve of the New Year (i.e. this evening) when half the population is traveling to the other half and so is out of the shelters, not on the evening before.

There are four (actually five) shelters within easy reach of the dining room so nobody panicked. I went down to the music shelter that I had left a minute or two earlier after having returned equipment; I was accompanied by maybe one hundred people. The air shelter is quite large, but having so many people meant that we had to stand. There is an air conditioner in the shelter so at least we had fresh air, but it was quite humid and I broke out into a sweat for the third time that day. At some stage, one of the teenagers asked if I wanted some water; he came back shortly after with a cup. Later on, the emergency crew came in with about twenty big bottles of water and someone gave me one, not that I drank that much.

Eventually (after maybe an hour) the all clear was given and we piled out of the shelter and into the fresh evening air. I went back into the dining room to pick up my guitar and pedal board and then it was home. There I found a few people in our bedroom that doubles as the security room watching the news. I received a brief update, and as the attack was over, went into the lounge where I could breathe some fresh air.

Here's a video of us accompanying the children who entered first grade a few weeks ago: each child is hold a placard with the name of the Hebrew month on one side and a picture of the season on the other. By chance, my second grand-daughter is standing in front of me.

---

This day in history:

Blog #	Date	Title	Tags
409	02/10/2011	Holiday cooking	Cooking
1341	02/10/2020	Weight at the beginning of October	Health