And so it is that whenever I have to read and evaluate applications (to grad school, for scholarships, jobs, whatever), I invariably find that the reference letters have slightly less practical utility than my own (male) nipples. They are the vestigial nubs of something that might have served a real function in a very different set of circumstances (and usually there's two, but sometimes you get a third).
As I sat pondering this, in the outgoing water of my bath, it suddenly hit me - get reference letters for the reference letters! It's so simple. Each person who supplies a letter of reference shall submit two (occasionally three) letters from colleagues familiar with their reference letter writing, which can be used to evaluate the letter of reference.
I am sure the administration will enthusiastically support my initiative. It costs no money, builds on existing administrative infrastructure, leaves a documented paper trail and has the patina of objectivity. The evaluation of our reference letters could even be metricized for use in promotion and tenure decisions. What's for an admin not to love?
I offer here are a few humble examples to illustrate:
Professor Plotz has a real eye for talent. I have had the good fortune to read dozens of his recommendations and each student is better than the one before. I can tell, because he describes each student as 'literally the best student I have ever taught.' Plotz's remarkable ability to recruit ever better and better students into his lab is a true asset to our department.
Dr. Dunkleplunk is an absolute saint. He has a kind word for all, and sees the good in everyone. His commitment to the students' 'self of steam' is exemplary. All of his students pass with flying colours. I am certain that without his tireless efforts, many of our students would have tragically stopped believing in the power of their dreams long ago.
Sam Straydhup is a brilliant scholar and an inspiring teacher. Were I to voice any criticism it is that Sam is a perfectionist, who tends to damn students with faint praise. Stingy with grades, Sam tends to withold A+ from students simply because their work 'wasn't perfect'. Fortunately, Sam has been assigned to take a module offered by our campus professional re-education office, and I am confident that she can improve her commitment to student success.
.... and so ad infinitum.