Wait states

I migrated another database to unicode over the weekend. Updating the migrator was fairly straight-forward, although there was one problematic table with something like 35,000 that was defined as WIN1251 (Cyrillic) although the characters were actually encoded in WIN1255 (Hebrew). That table took some time to define its migration route. Although this database has fewer tables than the first database that I converted and most of those tables contain only numbers with much less text, it took much longer to migrate the database - because some of the tables are huge (relatively speaking). There are a few tables with over two million rows and one with over ten million rows. 

Let us say that I had problems migrating those tables because of their size. I started the migration program yesterday evening only to discover that it had crashed whilst handling one of those humungous tables. I restarted the program from the table where it had crashed, but I could see that soon it too would crash. So what could I do?

I thought that the crashing might be due to the migration program overwhelming the database manager who simply could not keep up with the pace at which records were being entered. 30 years ago, this would be called 'adding wait states': a wait state is a deliberate, one-or-more clock cycle delay introduced by a processor to synchronize with slower external components, such as RAM or I/O devices. It prevents the CPU from processing data before it is ready, ensuring data integrity at the cost of overall system performance (thank you Chrome AI). 

The first thing that the migrator does for each table is delete any data that may exist in the table; I added the command 'sleep (15000)' after the deletion to allow the data to be deleted; this command stops the migrator for 15 seconds. I then added a command that there would be 10 second breaks after each 2048 records have been read. This was initially 15 seconds and 1024 records, but I could see that this would be too slow. For one table, with just over 2 million records with the sleep time was 10 seconds every 2048 records; it took 3 hours and 13 minutes to migrate the table.

I'm now migrating the really large table and don't expect it to finish within the next few hours - but I do expect it to complete.

---

This day in blog history:

Blog #	Date	Title	Tags
826	12/04/2015	How not to display data in graphs	Statistics
1121	12/04/2018	Apology to Israel Railways	Trains
1602	12/04/2023	More bells and whistles for the 'Blog' program	Programming, Delphi, Blog manager program
1741	12/04/2024	Update on our spring performance	Musical group
1921	12/04/2025	Passover begins tonight	Jewish holidays, Song writing