fraudulence still unproven

Now here’s a misleading headline:

Lie detector finds benefit cheats

Apparently for the past few months Edinburgh City Council has been trialling some sort of automatic voice analysis software to detect whether people are lying on the phone.

The report states that phonecalls with 75 people have been analysed, and the package has flagged 25 of them as requiring further investigation – ie, a third of the sample.

The problem is that the outcome of this further investigation is not reported in the article (if indeed it has been carried out). So we have no idea whether the system is actually accurate. Perhaps it’s too sensitive, and gives more false alarms than correct identification of suspect speakers. All that the package can do is identify acoustic patterns which may indicate that the speaker is under stress of some sort – it can’t tell if the speaker is actually telling a lie, or intending to cheat the system – so it’s clearly important to establish the actual sensitivity and specificity of its outputs.

(An earlier article claims that by comparing measurements with the recording made of the speaker’s voice at the start of the phonecall, it will be possible to ensure that “nervousness or shyness” can be excluded as sources of the acoustic patterns. But ‘ensure’ is somewhat optimistic; there’s no real guarantee that the speaker’s first utterance of the phonecall will be representative of their typical unstressed speech. It’s also odd that the 75 speakers mentioned in the article were called by the council, not callers to the council, and the basis for the council selecting these particular benefits claimants is not stated. The acoustic patterns of people answering an unexpected phonecall would presumably be much more variable than those of people who have sat down to call up the council in their own time, making comparisons even harder to interpret.)

Just as odd is the fact that none of these suspect 25 people have actually been found to be cheating the system. The article mentions a case in which “one Edinburgh resident was recently sentenced to 150 hours community service for claiming more than £11,000 in benefits over five years while she was employed.” But since there’s no claim whatsoever that she was caught as a result of the ‘phone lie detector’ system, the relevance of this anecdote is pretty unclear.

Perhaps people are deterred from making false benefit claims on the basis that a mysterious voice analysis program might catch them telling lies, and perhaps some of the 25 flagged speakers are genuinely suspicious, but what the council has failed to demonstrate (at least going by what the article reports) is that the technology has actually made any concrete contribution to identifying benefit cheats. “Voice change detector seems to be identifying voice changes” might have been a more accurate way of putting it.

Advertisements

3 thoughts on “fraudulence still unproven

  1. Hmm, who wouldn’t be slightly stressed if the City Council were to call? I’m the one calling them instead of the other way around though. Is this system really going to help them much I wonder?

    Like

  2. exactly :)

    And the other funny thing is that, by their own admission, only a tiny number of people who claim benefits are attempting to cheat the system. So instead of proclaiming the technology as a success, they should really have been surprised at how much it was clearly over-estimating the problem – and wondering if it was really worth it after all :)

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s