How to Read a Review

I occasionally wander around the Internet, reading what other people say about us on various bulletin boards. This is generally not a good use of time, but I can’t resist (especially when a helpful reader sends me an e-mail saying, “Someone’s lambasting you on www.gocheckitout.com!”). Recently I’ve noticed a number of discussions on different boards, all covering the same general topic …

“Boy, I’ve met too many home schoolers who won’t buy anything that Susan Wise Bauer didn’t review positively! They’re really missing some good stuff!”

“I wonder if Susan Wise Bauer has stock in the materials she reviews?”

“I bet Susan Wise Bauer changes her mind about most of her recommendations when her children get older.”

“I don’t like most of Susan Wise Bauer’s reviews. I’d rather stick with stuff that real home school moms are using.” [Indignant snort from Susan, who was reading this with her four real children raising havoc in the next room.]

These comments reflect a very proper suspicion of Expertise. After all, we home schoolers and afterschoolers are generally an independent lot. If we’re home schooling, we’ve already bucked the educational experts who told us not to put our kids in school. And if we’re afterschooling, we’re already convinced that our kids need more than the “expert” in the classroom is giving.

Yet despite our independence, we’re also insecure. We worry that we’re missing something absolutely foundational, and that we won’t realize it until our children are in high school (and by then it will be TOO LATE!). We worry that we’re using an inferior curriculum but aren’t well-informed enough to realize it. We worry that our children are going to hate learning instead of loving it, and it will be OUR FAULT. We have, in fact, all of the normal worries of any conscientious parent; we worry over emotional, physical, social, and emotional development … and then add educational development on top of all those other concerns.

No wonder we long for some help in choosing curricula!

But the home school publishing scene is still young, which means that there’s no established procedure for home school curricula reviews. In “trade publishing” — publishing of mass market titles for consumers — the review process has a long tradition. Publishers send out “bound galleys” (pre-publication copies of books with temporary paper covers) to journals, magazines, and newspapers, one to three months before the publication date. The book review editor at each journal, magazine, or newspaper decides which books will be reviewed and assign them to a writer. The publisher hopes for good reviews — but it is considered bad form for a publisher to ask a book review editor when a review might appear, or to complain about a bad review. And it’s superlatively bad form for an author to complain about negative reviews. After all, the publisher and author are asking people to pay money for a book. They have offered their work to the public, and so the public is entitled to express any reasonable opinion about it.

On the reader’s side, book reviews are generally recognized to be a useful but not infallible guide to a book’s quality. Readers of book reviews don’t generally agonize over whether or not to buy the book. Instead, they want to know: What’s the book about? Is it worth reading? If so, should I buy it in hardcover or wait for the paperback?

But the home school market has none of these established traditions. Home schooling and afterschooling parents have a lot at stake when they buy curricula, and when they read reviews, they tend to look for a clear green light (Buy it and your child will thrive!) or an unambiguous red light (Waste of money! Errors of fact! Heresy! Stop!). And publishers of home school materials are often unfamiliar with the conventions of the book-review world.

When I first started doing online reviews of home school materials, I had been writing book reviews for journals and magazines for five years or so. So I treated curricula reviews the way I treated any other review materials: I summarized the content, pointed out positive aspects, discussed any problems, and finished up with a recommendation to the consumer as to whether or not the product was worth the money.

However, this trade-publishing attitude didn’t sit well with a number of curricula publishers. Often these were small presses, publishing only one or two works. They objected to sending review copies (something I wasn’t expecting — free books to reviewers is such a standard part of trade publishing!). They complained that I didn’t acknowledge receipt of materials or return them when I was finished (something I had never done before). They objected to any negative remarks, and wanted to know why I hadn’t allowed them to preview my comments first. (Any book reviewer who did this would never write again.) Most of all, they resented criticism. After all, they had published their curricula as a ministry, and it deserved to be treated like a ministry, not a product.

It’s time for both readers and publishers to rethink the purpose of home school reviews — and to begin to establish some customs to govern the publication of (and reaction to) reviews. So let me propose a few elementary principles for readers and publishers alike.

For Parents: How Should You Read A Review?

Before you start, know what you’re looking for. Shopping for curricula can be a little like shopping at the local discount superstore; you go in looking for a lamp and before you know it you’ve got a cartful of items you didn’t know you needed. More than ever, curricula are being published for (and targeted at) home schoolers, and home school catalogs now look more like Target than like the cozy little mom-and-pop stores they once resembled. Recognize that shopping for curricula, like shopping for household items, is an activity that creates a need where none existed before. Capitalism is built on the necessity of convincing consumers that they never own enough and that they have to continually upgrade, in a circle of improvement that never ends. (Car manufacturers have mastered this principle.) For curricula buyers, this is related to Principle #2:

Avoid curricula envy. If your child is making steady progress and isn’t frustrated, your curriculum is probably a good match for him. Does this mean you should never look at other options? No; it’s always a good thing to examine other curricula and consider different approaches. But when do you switch curricula? When your child cries at the prospect of math or spelling, or when you can see that she’s not using the skills she’s supposed to be learning. Don’t feel pressured to switch because everyone in your home school group has. Switch because you can see real benefits to a different approach.

Look at the source. When you read a laudatory review, first look at the place where it appears. There are two kinds of reviews: objective and profit-based. Profit-based reviews are done by people who have a financial interest in the curricula doing well. Publishers’ websites provide wonderful reviews of their own products, but you should take these with (the appropriate numbers of) grains of salt. Catalog/websites that sell programs are a tricky case. Generally you should take store descriptions with a similar dose of salt. But many websites (my own Well-Trained Mind Press included) praise the products they sell, but only carry those products that they think are worth using.

Home schooling magazines are not free from taint; the advertisements they carry help to pay the bills, and publishers generally will not advertise in a magazine that publishes a negative review of their product. So magazine reviews are sometimes affected by the prospect of advertising dollars.

This is equally true of awards, which are often given to publishers and then followed by enormous press packs inviting the publishers to spend large amounts of money advertising on/for, and otherwise promoting, the award-giver’s website/business. (“Don’t you want to put this badge on your product showing that it was #1? This badge which coincidentally bears the name of my company/website/magazine?”)

The bottom line: Objective reviews are done by people with no financial interest in the curricula’s success or failure.

Evaluate the author’s experience and bias. Is the author a home schooler? Many programs which work very well in a classroom are not as good for home use. What methods does she support? What are her priorities? Does she hold to classical methods, to relaxed schooling, to “school-at-home” techniques? Do those methods match your own? If a negative review of (for example) a grammar program reflects the author’s belief that grammar isn’t really necessary, you should take this into account. How much experience has the author had? In what areas? Has she had the chance to evaluate older children who have used the program or method under study? Do you know how well those children have mastered the skills or material being taught? What religious or philosophical considerations should you take into account? A science curriculum might be praised because it teaches (or doesn’t teach) a certain view of beginnings, not because of its other aspects. A literature program might be exalted or condemned because of its use (or avoidance of) certain books. Know the reviewer’s bias before you buy.

It isn’t always possible to get answers to these questions, but you should at least ask them — particularly for reviews published on websites, because they haven’t gone through the winnowing process provided by editors and publishing houses. Anyone with an e-mail account can publish reviews to the Internet.

Read more than one review. In his Rulebook for Arguments, Anthony Weston writes that an “expert” must be informed (see Principle #4), impartial (see Principle #3), and largely in agreement with other experts. If we’re the only people praising or condemning a program, you should be careful! (And apply this test to other reviewers as well.) But think carefully about who is an “expert” in the field of home school curricula reviews. Many classical materials, in particular, have been developed for the classroom. Praise from a well-regarded classical school might give a curriculum the stamp of academic excellence, but how about affordability and ease of use for the home school parent? If those materials have been geared primarily towards the classroom, and towards the teacher who goes to do specialized training each summer between semesters, they might not suit your needs. So make sure you consider homeschool as well as classical expertise when reading reviews.

Realize that a partly good review is a good review. Many home school readers think that one negative comment in a review sinks the ship; obviously this curriculum is not worth using. Try to avoid this “all-or-nothing” mentality. Learning is an immensely complicated process, and even the simplest curriculum has to function on a number of different levels and accomplish a number of different purposes simultaneously. Most (perhaps all!) curricula do some tasks well, and other tasks not so well. There is no such thing as the perfect curriculum. Every curriculum needs tweaking, supplementing, and modification. Criticisms are meant to help parents tweak at home — NOT to suggest that a program is not worth using.

Remember that logic is limited. No program, no matter how great, is great for everyone. No curricula, no matter how well designed, suits every child. Don’t allow reviews to enslave you. If the curricula everyone else is using doesn’t do the trick for your child, switch. If (for example) I say that a program isn’t thorough, but your child is flourishing with it, smile forgivingly at my review and keep right on using it.

For Home School Publishers: How Do You React To Reviews?

Don’t react. I speak as a writer who has been both praised and vilified. It is pointless to complain about the content of a review. While it is always appropriate to correct errors of fact, objecting to the opinions of reviewers is always counterproductive. I have several times broken this rule and have always regretted it. If your program is a good and useful one, it will speak for itself.

If you are selling a product, accept criticism as part of the process. Many home schooler curricula writers seem to see their products as a ministry, not a product. But all of those parents who are investing money (sometimes significant amounts of it!) in a program deserve to know its strengths and weaknesses.

Remember that the best way to make money is to give something away. Review copies ought to be considered as advertising costs. The best way to create enthusiasm about a good program is to make it visible.

Realize that a partly good review is a good review. Authors and publishers often take criticism very personally indeed; saying that a curriculum has gaps is like saying that someone’s child is ugly. I know this feeling well. (But see point number 6 above.)