Blog word counts

Yesterday afternoon, a new metric for these blogs popped into my head: word count. Obviously I'm not a writer who is paid by the number of words that I write per blog, nor do I have to fill a quota of words for a newspaper or magazine article, but nevertheless, word count is a parameter that is worth measuring.

What would be the results/reports deriving from adding word counts?

• Number of words per month
• Number of words per year
• Top n blogs per length

These are all trivial to program and the first two aren't particularly valuable as they are dependent on the number of blogs in the time period. A more useful metric would be number of words per 100 blogs. Word count per tag isn't a good idea, as a blog can have several tags: sometimes the tags are dependent upon each other, eg 'holiday' and 'Italy', and anyway there's no way of noting how many words within a blog are devoted to a specific tag when the tags aren't dependent.

As I wrote to CoPilot when proposing this idea: this is a far from simple proposition: consider the attached text file.

1. Everything upto and including <body> should be ignored.

2. Contents between tags should be ignored (eg <div class="separator" style="clear: both; text-align: center;">).

3. In the case of this example blog, the counting should stop at the line <hr width="80%"><br /><div>, but this is not necessarily the end of the blog text. This line could be preceeded with <p>Internal links that definitely marks the end of the current blog, but as can be seen, that line doesn't appear here. The blog text may include a horizontal line, that so doesn't definitely mark the end of the text. The blog text normally has the 'This day in blog history' section that definitely marks the end; this should not be counted. But sometimes this doesn't appear (and it certainly doesn't appear in old blogs). I supposed that one way of marking the end of the text to be counted would be to use a special tag (one that I don't use; there are probably several) but this would have to be added manually.

A little analysis showed that the first blog with 'this day in history' was numbered 1554, meaning about 70% of the blogs until now are WITHOUT that 'This day in blog history' marker, nor the horizontal line nor the 'Internal links'. There's no way to fix this: I will manually have to go over each and every blog, adding a specific marker or sentinel, <!--END-BLOG-TEXT--> that shows where the end of blog text is. This can be handled automatically in the future by adding it to the 'this day in blog history' code that I paste into each blog.

The actual mechanics of word counting were far easier than I expected. This boiled down to the following steps

1. Get blog text from database
2. Determine the positions of the opening marker (<body>) and the sentinel, extracting everything inbetween for further handling.
3. Strip the HTML tags from that remaining text.
4. Replace certain HTML constructs (such as '& nbsp;') with spaces or appropriate punctuation.
5. Remove all punctuation.
6. Save the text to a string list and have each word appear on a different line
7. Count the number of lines in the string list - that's the word count!
8. Save the word count to a field in the blog's tuple within the database.

All of these steps were fairly trivial and CoPilot put the code together almost immediately. I originally created a form that would ask for a blog number and then would determine the word count; this almost immediately became superceded by the idea of passing a blog number to the form, and eventually I realised that I don't need a visual form at all but a simple text unit that does all the work. The only minor complication is that the two necessary queries (steps 1 and 8 above) will have to be defined in text and not as visual components but that's not a real problem. I also wrote a short routine that allows a range of blogs to be counted, although of course this is done by iterating over the list of blogs. The only thing that I left out and will have to be added is to calculate the word count automatically when the text of a blog is added to the database.

A small amount of thought went into answering the question 'how would the list of top n blogs be displayed'; should I use the existing 'display blog list' form? I don't think so as this form has a tremendous amount of functionality, all of which is not relevant. I think that a simple query and grid - like showing the number of blogs per tag - is what is required, plus the ability to show a given blog's text (but without the bells and whistles that have been added to the browser form).

All this thinking has unfortunately prevented me from falling back to sleep in the middle of the night. It's now almost 5 am and I may as well get up as I'm not going to get any more sleep this morning. I'll need an afternoon nap.

Incidentally, the word count for this blog is 914.

---

This day in blog history:

Blog #	Date	Title	Tags
107	27/10/2007	When Sally met David (Friday night is sequencing night)	MIDI, Folktronix
298	27/10/2010	Calculating possibilities correctly	Probability
644	27/10/2013	Dates	Personal, King Crimson
1269	27/10/2019	Agent running in the field	John Le Carre
1540	27/10/2022	Emergency surgery	Computers