I have been declared persona non grata at Pannostrum,
which is most unfair. All I made was a modest proposal. The marketing
department are in a panic. One of the ‘prestigious’ medical journals,
Journal of Advanced Clinical and Surgical Science, an oxymoron if ever
there was one, and known to all as JACASS has decreed that no
publications will be accepted from the pharmaceutical industry unless the
analysis has been repeated by an independent academic-based statistician.
Marketing have demanded that we immediately identify a number of university-based
statisticians to form a panel to which we can turn. My proposal was much more
radical. ‘Let’s stop sending papers to this rag’, I said. ‘It’s full of crap
anyway. Why don’t we just publish our reports on the web?’
This went down like the proverbial lead balloon. I had
overlooked the fact that several members of the medical department and even
some of biostatistics count their publications in JACASS as being amongst the
jewels in their scientific record. I can’t understand it myself since I always
make damn sure that my name never gets stuck on any paper heading for a journal
with innumerate readers. By the time the moronic editors and referees have
hacked it to bits there is nothing of any value left anyway. Anyway, I don’t
think I have been so unpopular since I refused to validate the spreadsheet of
one of our biochemistry units on the grounds that the program clearly didn’t
work. ‘How can you tell?’, they said, ‘since you haven’t even looked at it’. ‘I
don’t need to look at it. It’s obviously wrong and I am just applying some
Bayesian inference, and a very informative prior’, I replied. ‘It’s the
combination of numerically challenged scientists, moderately complex
application and spreadsheets that has probability zero of producing a valid
result’. They were outraged and the powers that be forced me to look at the
damn thing. Luckily I found the negative sum of squares in the first 30
seconds.
Anyway, to get back to JACASS, my punishment from on high,
for giving too much lip, was to force me into academic interaction. I was sent
to see Professor Percy Vere, a physician in the Department of Evidence Based
Medicine at the
I can’t claim that our first meeting was much of a success.
I don’t know who had briefed Vere but he seemed to think that I has been sent
to learn how to conduct a meta-analysis according to the procedures of the
Archie Association, about which I couldn’t give a damn, whereas the real reason
I was there, surely, was to bring the light of statistical reason into the
heathen darkness of medical research.
‘You see, Mr McPearson’, he said, ‘the important thing is
that any meta-analysis should be produced in such a way that it would satisfy
the MVT.’
‘The MVT, Mr Vere?’ I replied.
‘That’s Professor Vere, Mr McPearson’ he said, ‘and
MVT stands for Meta-analysis Validation Tool.’
‘That’s Dr McPearson, Professor Vere’ I said, ‘and I
presume that you are referring to the index proposed by Gaucho and Roper in JOG.’
(JOG is the Journal of Overviewing and Generalization)
‘There’s no need to stand on your dignity,’ he said, ‘and
yes it is the Gaucho and Roper1 index and if you are acquainted with
it I am sure that you will see its value.’
‘Very valuable as far as it goes,’ I replied, ‘but
unfortunately not far enough.’
‘How do you mean?!’ said Vere in outraged tones.
‘Well, I don’t see any of the following. Did the analysis
respect the structure of the original trials? Was double counting
avoided? Was data imputation necessary? Was pooling carried out
in a way that was relevant to the question put?
‘I am afraid you will have to explain the relevance of these
points, Mr McPearson.’
‘Right. Here goes. You see this meta-analysis,’ I said
picking up a paper from my briefcase that had been published recently in the
Albion Physcian’s Enquirer (APE) and holding it at arm’s length suspended from
one corner before pacing it on the table, carefully, and opening it. ‘You will
note that it synthesises a mixture of cross-over and parallel group trials. Yet
you will see, however that the data are summarised in terms of means in each
group and standard deviations. Yet this information could not allow you to
analyse a single cross-over trial adequately, so it beats me as to how it could
be used to summarise results from cross-over trials adequately.’
‘What a silly objection. This is just the way that the
software package we use, MetOrg, requires the data.’
‘So much the worse for it. Now have a look at the standard
deviations from those cross-over trials that I have highlighted. You will note
that they are all equal to 10.0 for each arm and that there appear to be 16
such values. How do you explain this extraordinary coincidence?’
‘Oh that’s very easy. We are always having to do this sort
of thing. You see very often people who publish cross-over trials fail to give
the appropriate information. They will insist on producing estimates of
the treatment difference and the standard error or confidence intervals rather
than the treatment means and standard deviations, so that we have to use
plausible values instead. But after all, all things are related, we can use
values estimated from the other trials. No man is an island etc’ and then
adding with barely concealed mirth, ‘although some appear to be a
‘Very revealing,’ I replied with barely concealed distaste,
‘I suppose that you could call this an example of Vere-say evidence.’
‘I do think that making puns on people’s names is
very cheap,’ he replied with glacial hauteur. ‘What were your other points?’
‘Well you will note’, I said, ‘that even for the parallel
group trials the same means and standard deviations appeared twice in the
control group arm in many places.’
‘That’s an obvious point. We often find this. These must be
three-arm trials so the control group has been entered twice, once for each
experimental arm. This is very common.’
‘Yes. So it appears and this brings me to my final point.
Why is it necessary to pool the results from two experimental arms from one
trial into a meta-analysis? After all if there are different arms, the
treatments are different.’
‘This is also very common. Why, we did it in our
meta-analysis of Detense. I think you will find that it is merely that results
for different doses have been pooled.’
‘Call me old-fashioned,’ I said, ‘but as someone who
actually works in drug development, I consider that differences between doses
are important. A lot of what we do is geared to finding appropriate doses.’
‘Well this is a very minor consideration. The important
thing is to find all the evidence. Nothing you have shown me from this paper
seems inappropriate. Let me have a look… This is outrageous!!!!. This is
our meta-analysis of Detense. Surely you are not going to argue with this? It’s
pear reviewed, carried out according to Archie Association guidelines, designed
by distinguished academic experts and published in a prestigious journal. It’s
beyond reproach. How would you go about doing such a meta-analysis?’
‘Well, we have gone about it,’ I said, ‘which is why I am
here. We have our own meta-analysis in which we have not pooled different
doses, have not counted control arms twice, have used standard errors and
treatment contrasts from given trials, thus enabling us to include cross-over
trials properly and have not imputed any values, since we have access to all
relevant data.’
‘I would be very interested to know how you have managed to
do this using MetOrg,’ said DeVere.
‘We haven’t,’ I said, ‘we have programmed it ourselves’.
‘Programmed it yourselves! That can’t be a valid approach.
What hubris. What makes you competent to carry out this sort on scientific
innovation? Who will accept that? Are you aware that published analyses in APE
show that the standard of pharmaceutical industry meta-analyses is inferior to
Archie Association ones.’
‘Would this be research carried out by members of the Archie
Association using the Gaucho and Roper quality index?’
‘Yes. So you need have no fear about the validity of the
result.’
Well that was it. My best attempts at diplomacy and
persuasion had failed. It was impossible to get the man to see reason. In the
end, he proposed that we get an independent assessment by a third party to see
who was right and I, unwisely, agreed.
Some weeks later I passed Dr Angina Cutter in the corridor.
‘
‘Oh,’ I replied, ‘Vere and I agreed that we would get a
third party to review our and his analyses. He was going to get back to me with
a proposal for a name.’
‘He’s done better than that,’ trilled Angina delightfully,
her eyes shining , ‘He’s gone ahead and had the review done. I was talking to
Sir Lancelot just now (what a charming man) and he was telling me that Pannostrum
and the Archie Association are now singing from the same hymn sheet as he put
it and that we have agreed the results. Isn’t that great? Another problem
solved.’
‘I presume,’ I said gloomily, ‘that that means we are
accepting Vere’s results.’
‘Yes, apparently. Not our fault. It seems we were using the
wrong software. It seems that we should have been using MetOrg. Had we been
doing so, we would have had to do it properly. I proposed to Sir Lancelotthat
we should make it company policy to always use MetOrg in future and the good
news for you is that he agreed that this was a good idea and we will be
obtaining copies of MetOrg.’
‘And this independent statistician,’ I said gloomily, ‘is
not perchance someone who works at the
‘Yes. How did you guess?’
‘Something to do with a fundamental statistical concept Vere
was having difficulty, with’ I said.
‘Oh. What was that?’
‘
Reference
1. Gaucho, M and Roper, Z. (1992) A peer-validated valid
quality assessment tool for valid meta-analysis. Journal of Overviewing and
Generalization, 5, 23-45.
Back to G McP's prose selection