Data, the high-processing android in Star Trek: The Next Generation, had an evil twin brother named Lore. Data was rational; Lore was irrational. Data solved problems; Lore
created them. Data was focussed on progress; Lore made mischief.

Data and lore are the two ways of knowing. Modern society escapes from the ignorance of earlier eras inasmuch as it turns to data.

When I started working on medical journals, a new approach called “evidence-based medicine” was being fostered. Physicians were quick to point out that medicine had been based on evidence in the past, but this qualification was a little disingenuous. Even in the 1990s, many therapies were decided by the consensus of experts rather than being based on information from scientific studies. In fact, what physicians and researchers were discovering, to their chagrin, was that well-conducted studies turned many common medical practices on their ear.

As a patient, I was alarmed to discover how many approaches were based on lore. Expert consensus was often no better than a “groupthink” exercise in which physicians reinforced each other’s misconceptions and mouthed platitudes based on few anecdotal cases.

Physicians are not alone in doing this, of course. It’s human nature.

The book and new film Moneyball, by Michael Lewis, shows how baseball scouts and sports commentators in the media repeat all the old, inaccurate homilies and saws about baseball players that are belied by statistics on athletes’ performance.

In my own life and work experience, I have seen people make decisions based on nebulous, value-laden premises. There’s some schadenfreude when these decisions go awry, but how much damage is caused in the meantime?

The most serious recent example is the transformation of the mandatory long-form census to a voluntary household survey. Without detailed, statistically valid data, society is unable to make informed decisions. Banks, municipalities, school boards, and businesses used these data to create systems that work for people.

Cancelling the long-form census appears to come out of libertarian lore that mandatory gathering of personal information is coercive. The reasons given for the cancellation ― that there had been complaints, that data could be misused ― were not supported by any facts. There had been few complaints from citizens, and Statistics Canada is very careful to maintain
confidentiality of all data; there has never been a breach.

But the decision went forward. As my husband said, “Who needs data when you’ve got dogma?” (This comment has circulated widely, often attributed to me.)

A ray of light is the recent Supreme Court of Canada decision concerning the safe-injection site in Vancouver. In this decision, information concerning the benefit of harm reduction in avoiding overdoses, preventing illnesses transmitted by needles, and even in guiding addicts to health and social services, outweighed anti-drug policies based on judgement and values. As such, the decision represented a triumph of data over dogma.

More on data and lore in my next post.