Monday, January 21, 2019

Validity styles

Over the past six months, one of the psychologists employed by the OP for whom I consult has been working on developing new models, which explain inter alia the 'validity style' of an examinee's responses. These can vary from 'random' (in which the examinee hits one key repeatedly without reading the questions) to 'honest' and on to 'normal' as a default value. The psychologist sent me a list of rules which enable the program which displays an examinee's results to determine the style. He wrote the rules in pseudo-code:
If (INF == extreme && IR == extreme) then return RANDOM. If (INF == extreme) then return INFREQUENT. If (IR == extreme) then return INCONSISTENT. If (DEF == extreme || GI == extreme || (DEF == borderline && GI == borderline)) then return FAKE_GOOD. If (DEF == low) then return HONEST. If (CM == extreme) then return UNCOMMON. (Else) return UNCLASSIFIED.
I have implemented similar rule mechanisms in the past, but in each case they were always 'and' statements, like the first rule shown above. 'Or' statements seem to be more complicated, and 'short-circuit evaluation' should be used throughout. There was one rule which had both 'and' and 'or' statements which had me literally scratching my head how to implement this.

At one stage, I half seriously considered embedding a Prolog interpreter into the program, as these rules could easily be rephrased as Prolog rules, such as
random:- inf (extreme), it (extreme). infrequent:- inf (extreme). inconsistent:- ir (extreme). fake_good:- def (extreme). fake_good:- gi (extreme). fake_good:- def (borderline), gi (borderline). honest:- def (low). uncommon:- cm (extreme). unclassified:- true.
but I quickly backed down from this idea. The above mental exercise reminded me that in Prolog, 'or' statements are handled by having two similar rules. After this insight, it became easier to translate the rules into data stored in SQL tables.

Writing the interface code to store the rules was complicated as I identified three different possibilities: creating a rule with a clause, editing that clause and adding a new clause to an existing rule. Thus for 'or' statements, there would be two (or more) rules whereas an 'and' statement would require one rule with two (or more) clauses. Once I worked this out in my head, the implementation simplified.

Writing the code to evaluate the rules (the equivalent of a Prolog interpreter) took some time, primarily because I confused the short circuit evaluation with the value returned by the evaluation function as to whether a rule was proven. After debugging this code live several times, I finally figured out how it should be written. Testing the code on real data seemed to work, but in almost every case, the result was 'normal', which didn't really test anything.

So I added a 'validity style' field to the 'people' table and wrote an external program based on my code which would evaluate for all people who have been tested since 01/01/17 (about 4500 people) their validity style and store this value in the table. I ran the program then queried the table to find people who did not have a normal style. The result was no one, which I knew was incorrect. I went over the code line by line which seemed to be correct, but then I noticed that one query was not closed. This wouldn't make any difference if only one person is being tested (as in the production code), but for my testing code this was a bug. So I added the 'close' statement, ran the test again, and discovered the many people who have non-normal validity styles.

The next step is to check the complete output for these people in order to see whether the raw data supports the validity style. This isn't my problem.

No comments: