Tuesday, December 13, 2005

What's wrong with documentaries and journalism

Just saw a documentary last night: Rize. It's about the culture of clowning and krumping in LA, which is basically street kids dressing up like clowns and having these energetic dance competitions. And though I enjoyed the movie, it re-affirmed the view I walked in with: Don't trust documentaries.

It's not just documentaries I don't trust-- it's also journalism and, most recently, many books.

The basis of this reasoning is that truth is difficult to get at. It requires careful study, and personally, I believe that science is the only trustworthy way to determine what it's rational to believe about the world. It's not that science always brings up the truth; it doesn't. It's not that non-science can't get lucky once in a while; it does. It's just that looking at current scientific research gives you what is most rational to believe right now, given the world's state of knowledge and theory.

Key to this is peer review, and that's where journalism, documentaries, and many books get into trouble. Peer review is the process of getting published through approval of your scientific peers. If you've made a mistake, it's the reviewer's job to see that. You might have not controlled for a variable. You might have ignored some other force that has been discovered that might render your hypothesis faulty. When you read a paper in a scientific journal, or see it at a scientific conference, you can trust to some degree that it's been screened by about three scientists who can vouch for its quality. That means that if you want to walk away believing what the article concludes, you can feel fairly fuzzy inside with respect to your justification in doing so.

The problem with documentaries and journalism is not that they can't get it right, but since there's no peer review you can't tell if it was done right or not. The fact that you can create polemical, one-sided documentaries means that you can't tell, as a naive observer, whether the documentary you're watching is one of those or not. So when I watch documentaries I have this uncomfortable conflict in my head: believe it or don't? I'd rather not have the struggle and just watch fiction. Another problem with documentaries is that the visual medium is not condusive to reporting quantitative data which is often so key in understanding anything. Here's an example from Rize: They showed a long sequence in which the krumping people were dancing, and similar-looking dances from African tribes were interspersed. What am I to make of this? Did these inner-city kids watch a lot of discovery channel, tape these dances and run with them? Is the filmmaker trying to say there is something preserved through black culture running all the way back to Africa? Or even more ridiculously, that there is something innate about black people to make them dance this way? Of course, this question is not answered in the film. All we have is this sequence that suggests a connection, but doesn't come out and say what it is, let alone try to justify it. So what is the observer rationally supposed to take from this? The verbal nature of writing allows very precise statements, and when you have precise statements, the required evaluation of them becomes much clearer.

[From my friend Daniel Saunders after reading my essay (LaChappelle is the director):
From a salon.com interview with David LaChappelle,

"They create family where there was none and they create art where there
wasn't any. I was given every art program when I was a kid. They've had
none. They've had no African studies. They've never seen African dance.
They've never seen African face paint. It's in their blood."


What's my problem with journalism? Well, journalism has one up on documentaries because at least they have fact-checkers. So I think it's fine to trust journalism with getting the facts right. However, there is the problem of which facts are reported, and in what context. Like a documentary, the presentation of facts usually makes a subtle, implied point. It's this point, which is often implicit, that is not a fact that can be "checked." It is, whether they know it or not, a scientific theory that needs empirical evaluation. Even worse, sometimes this theory is not implicit. There's a respected magazine called The Economist. I was reading an article about vehicle safety and in that article they wrote that Americans accept accidents as inevitable. The facts they drew on were that they purchased big cars that had poor maneuverability and bulk. This is in spite of the fact that you're safer in a Jetta, statistically, than in an SUV. But to say a culture accepts accidents as inevitable is a major claim that is underconstrained by the facts. Why on earth did they publish such a thing? It's a plausible explanation, but there are a hundered roads called "Plausible" on the way to truth. It could be that they are buying SUVs purely for style reasons. It could be they like being higher up, and their reasons for buying them has nothing to do with safety. It could be that they don't rellly understand how smaller cars can avoid accidents. As you can see, the Economist's statement assumes a lot. Figuring this out is the job of a scientist.

Now we get to books. The problem with books, even "scientific" books, is that, believe it or not they are not peer reviewed either. Gladwell's awful "The Tipping Point" is a great case in point. He's a great writer, and the book is full of interesting information, but it's terribly unscientific, even though he's arguing for a scientific theory. Yesterday I bought a book for my father for Christmas (you can keep reading Dad; I'm taking it back today) called "The Skeptical Environmentalist," by Lomborg. It looked great! Full of charts, one quarter of the book was endnotes and a bibliography, written by a statistician, quotes of praise from Matt Ridley (author of Gemone, a great book), the Wall Street Journal, and the Times. It looked like a great, data-driven review of the state of the environment. I got on the web later and found out that there was a huge outcry from scientists blasting this book. I could not find a single scientist who praised it (Matt Ridley is a science writer, but not a scientist). Like Diamond's "Collapse," (which is on the opposite side of the environmental adgenda), the science in books is often either poorly reported (Collapse) or poorly done (Lomborg).

So now I'm even skeptical of books. For me to trust a non-fiction book now, it needs to be written by a scientist I trust, with a heavy publication record in the field as the book. See what I mean about getting crotchety?

A caveat: Autobiographies and travel stories have their value. In this essay I am talking primarily about big truths worth knowing. Big things about the world we live in. Little details about the lives of individuals perhaps need not be scientific in their reporting.

Anyway, this is the long-winded reason that if you invite me over to watch a documentary, I might just stay home and watch "Spirited Away." If I'm watching untrustworthy material, it might as well have a good story.


Anonymous said...

You will have to make 2006 the year of no documentaries as your resolution.

I hear your frustration but is the problem that film-makers don't know the definition of documentary, or is it that they only produce what they know the market will buy? The market, print and film, seems to demand crap. Evidence can come down the road later, as in "see, I knew Jessica and Nick were going to break up".

Anyway, I'm just throwing this together quickly. I'm across the street from you at KGH, where my patients are not that interested in peer-reviewed evidence either. Found your name last week when I clicked on the Kingston link on my profile page.

James Aach said...

My frustration with the media is a bit different. I work in the controversial field of nuclear power. I see things all the time in print or on TV that are factually wrong or grossly misleading. (However, I'm not implying nuclear is a perfect energy source.)

There is not a single book on the market written by someone who has worked in the industry and understands how things operate (other than books by disgruntled ex-employees). So I wrote one - I spent years at it, hired outside editing help, etc. I made the book a novel so that it would be more accessible. It's the right size, follows the basic format, etc...

But.... my book has not sparked the interest of the publishing community at all, despite endorsements I've had from some prominent people. I don't claim a conspiracy or evil intentions in all this, but it is peculiar to me that there is no room in the market for such a work. The market seems to be driven by forces I don't quite understand.

I've since published my novel free on the net at www.RadDecision.blogspot.com, and had good reviews from readers so far.

Rafael Guetta said...

Your problem is not with documentary but with the monetary system. Everything that is done for profit should be taken with a grain of salt. You are being very naive in thinking you can only believe in research. Scientific research is backed by mega corporation who has university professors sitting at their board meetings on Mondays and conducting research on Thursday.

Are you saying that at the forties through to sixties you would only trust the proven scientific research that showed that cigarettes are harmless? or would you trust most doctors who smoked Camel cause it does not irritate your throat? or maybe you would march your kids to a shower of DTD powder because back at the time it was proven scientifically DTD isn't toxic and was sprayed on migrant children. Or maybe you would love agent orange sprayed on your food cause their was a scientific research it is harmless to people. Half of the things you put in your body today will be proven in a couple of generations to be toxic. By then you will be dead and your money went to very rich people who became even richer. Your children will be sold a new scientific truth that will kill them just the same. Very bleak, I know, but if you think the world is any different, you are like a race horse with blinders. You might get to the finish line first, but you will be whipped all through it, you will hate your fellow horse for trying to walk in your path, and eventually, the man who was riding on your back the all time takes most of the glory you thought belongs to you.

I rather see an artistic representation of reality knowing their is an agenda behind it.

1. You should read "trust us we are experts". It will deflate your trust in research.
2. Watch the "inside job" to see a bit of the conflict of interest between researcher, teachers and the market, as an example of an academic system gone wrong.
3. Please tell me what is it about viewers and looking for answers in films (like your comments about the connection between blacks dancing in the US and the ones in Africa)? The filmmaker gave you an image which brought some questionsto your mind, and your mind started exercising what it is designed for, thinking, and contemplated on answers for you. And guess what? Quite quickly your brain gave you three very good answers that were based on your experience and knowledge of life. Now why would you prefer some scientist give you "temporal answers" and cripple your mind from debating the truth to someone who shows you an image and asks you for your truth?

Jim Davies said...

Rafael: The problem is this: if you're not going to believe science, then what are you going to believe? You can't just trust your own rationality and ability to figure things out with scientific data analysis.

So unless you have specific reasons to believe that the scientists are biased, yes, you should believe them. Sometimes that means dealing with products that turn out to be actually dangerous in the light of new findings.

However, the alternative is worse: believing things based on intuition or cultural myth.

See also my blog post on why reason isn't enough: http://jimdavies.blogspot.ca/2012/01/when-reason-isnt-enough-teaching.html