Balancing Act: Guernsey Scrapes the Barrel
I am afraid that I have to confess to scraping the barrel. I who generally rely on wit, penetrating logic, intelligence and flair to win my arguments have been reduced to using the lowest, most despicable and contemptible weapon in the controversialist’s armoury: authority. Let me explain.
IDEA: ‘Innovation in Design, Excellence in Analysis’ is the company’s latest initiative and as attentive readers will know already, nobody at Panacea escapes, not even scientists with the brilliance of yours truly. As part of IDEA I have been assigned to the RARA (Radical Approaches to Random Allocation) working party. Look, I refuse to accept any blame for these emetic acronyms – the company will insist on employing morons who have graduated in media studies, marketing or some other similar non-subject from the ex-polytechnics masquerading as universities in this country and they then have to justify their existence. Coming up with stupid acronyms is how they spend that part of the day they manage to spend ‘working’ between expense-account meals.
Not surprisingly with random allocation as the theme, the working party is made up of a bunch of tossers: we have Harvey Puffer and the admittedly pulchritudinous Angina Cutter as the wastes of space from the medical department, Clive Viper from Panacea Investor Support Section (see SPIN passim) and Di Muddle from Company Regulatory Affairs at Panacea Pharmaceuticals. In fact, I’m the only member of the team with any scientific brains.
Well, anyway, our meeting the other day didn’t start well. Harvey had had an idea. Or rather he thought he had an idea or perhaps even an IDEA. Harvey doesn’t do original thought; come to think of it, he doesn’t do much of the other sort either, however he had been reading a report of a clinical trial funded by CRIC the Clinical Research and Innovation Council and published in the Speculum. Harvey was very excited by this. “I’m very excited by this,’ he told everybody, ‘very excited indeed. Look what it says here. I’ll read it out “A minimisation algorithm was used to balance the treatment groups with respect to eligibility criteria and other major prognostic factors and implemented via a central telephone randomisation system”. Clever stuff, what? They didn’t just randomise but they actually balanced as well using this minimisation algorithm. This could be just what we are looking for.’
Di immediately looked rather worried, as if fearful that she might be required to make a regulatory pronouncement as to whether minimisation algorithms (whatever they might be) were allowed by the Agency or not.
‘Pathetic!’ I opined, ‘They can’t even describe what they did correctly’.
‘Guernsey, I am disappointed in you’ trilled Angina trying (and succeeding) to be winning and admonitory at the same time, ‘do you have to be so negative?’
‘Humph. Negativity is what his lot have in spades’, chipped in Viper, ‘they all suffer from not-invented-here-syndrome.’
‘I do think, Guernsey,’ said Harvey in a rather pained voice, ‘that you could take the time to consider this carefully. After all this is published in the Speculum’, he added in tones of near-religious awe.
I did consider drawing his attention modestly to the rather large number of letters to the editor I have had published in the Speculum pointing to errors of analysis and interpretation in that supposed oracle but thought better of it. ‘Well, give me an example of an eligibility criterion.’ I said.
Di perked up immediately. ‘The most important eligibility criteria is informed consent,’ she pronounced sententiously.
I rolled my eyes at this solecism. ‘How singular,’ I said, ‘or perhaps not, apparently.’ I carried on, ‘So let me get this straight. They check to see whether a patient on the trial has signed the informed consent form and balance accordingly. But I must confess this is rather puzzling because I would have thought that if they had done their job correctly all the patients on the trial would have signed consent forms so that the simple stratagem of making sure that there were equal numbers of patients on both arms would have balanced for this rather important criterion. Or were they also recruiting those who hadn’t signed and so needed to balance the proportion of illegals on each arm. Similarly’, I said, warming to my theme, ‘I understand that the patients were to be excluded if suffering from liver disease, so I am at a loss to understand why they thought that it was necessary to balance on this factor, since, presumably, 100% of the patients entered onto the trial were judged free of liver disease.’
Viper and Angina got the point right away. I could see Harvey struggling. Di chipped in, ‘ I am sure that a trial sponsored by CRIC and published in the Speculum would not involve violation of consent,’ she said in a rather shocked voice.
‘No,’ I added, ‘Just violation of good statistical principles.’
Harvey had been thinking and had finally grasped the point. ‘Age’ he announced triumphantly, ‘criteria like age. They are referring to things like age you know thingummy variables, what-do-you call em.’
‘Continuous variables, Harvey?’
‘Yes, continuous variables.’
‘Well, that would be a bit of a problem, Harvey. You see, minimisation can only balance categories and it doesn’t even do that sensibly. It has to take continuous variables and turn them into a series of categories and then add together apples and pears to form a balancing score.’
‘Well what’s the problem with that?’ chipped in Viper, ‘aren’t you clinical research people always dichotomising continuous variables?’
‘Not on my watch’, I replied, ‘but to return to my theme, the one thing that the authors of this paper did not do is minimise on the eligibility criteria, or at least to the extent that they did so, it was completely superfluous since all that they needed to do was make sure that all the patients entered onto the trial satisfied the eligibility criteria and then balance total numbers.’
‘Oh come on,’ Guernsey, ‘that’s nitpicking,’ said Viper, ‘what they meant to say was that they minimised on some of the continuous variables used to judge eligibility having turned them into categories. Between you and I that seems pretty sensible.’
‘Well, between you and me it seems pretty pathetic,’ I replied. ‘It is known that perfect balance can only expect to save about one patient per factor for equivalent power to a perfectly randomised design and minimisation doesn’t even use sound design theory for its balancing act.’
‘But saving a few patients could be terribly important,’ said Angina doing her passionate, earnest, caring, concerned medic bit.
‘There were more than 20,000 on this trial,’ I replied. ‘Saving half a dozen must have been terribly important.’
‘Well what would sound design theory be?’ sneered Viper.
‘You write down the model for analysis and then using regression theory and working with the design matrix you allocate the next patient so as to optimise the treatment variance,’ I replied. (Were I addressing the wet-behind-the-ears statistics graduates we recruit into Panacea I would then have proceeded to remind them that for a linear model the variance of in a regression is proportional to and you can manipulate your allocation to make the relevant element of this inverse as small as possible.)
‘Ah,’ said Harvey perking up, ‘that’s where you’re going wrong. You see these people in the Speculum just used a log-rank test. So they didn’t model,’ he added triumphantly.
‘Yes,’ I said. ‘I am not surprised. That’s a typical CRIC trick. Tell me, Harvey, can you remember from your undergraduate days when you did your introductory stats course for medical students at St Vitus’s the difference between a matched pairs t-test and a two sample t-test?’
‘Yes. We all know that,’ interrupted Angina. Clive showed irritation at the intrusion of such elementary stuff into the conversation and even Di’s face had a glimmer of understanding flicker across it momentarily, ‘You must never mistake the two’, continued Angina. ‘If you have designed your trial to match, then you must analyse the differences as a single sample. That’s what the matched pairs t-test is. You must never analyse the two groups independently if they are not.’
‘Very nicely put I said.’ ‘This girl’s definitely got potential,’ I thought. ‘Yes, well,’ I continued, ‘fitting a linear model with subject as a factor in addition to treatment is formally equivalent to carrying out a matched pairs t-test.’ Harvey looked shocked.
‘Do you mean to say, Guernsey’ he said, ‘that a matched pairs t-test is equivalent to some linear model?’
‘Yes,’ I replied, and ‘you can tell the sort of contempt in which I hold the CRIC lot, the Speculum referees and minimisers in general in that they must be the sort of persons who think that balancing removes the obligation to model. Take their philosophy to its logical conclusion and you analyse the matched pairs design as if it were a completely randomised design. In fact,’ I carried on, ‘modelling important prognostic factors makes a far bigger contribution to estimating treatment effects accurately than balancing for covariates. If CRIC had done exactly the opposite of what they did, randomised rather than minimised but fitted a proportional hazards model rather than the log-rank test, they could have made a far bigger contribution to reducing patient numbers.’
Harvey looked unconvinced. Angina cut in, ‘But how can that be, Guernsey? I have actually heard that your variances increase when using proportional hazards?’
I was flabbergasted. What on earth had Angina been up to? Had she been swatting up on statistics? This was an unexpectedly sophisticated point to come out of the medical department.
‘That’s correct, I replied. Unlike ordinary linear models, which we use for continuous measurements such as blood pressure and forced expiratory volume in one second, survival analysis models and also logistic regression models give higher variances of the treatment effect if you condition on covariates. However that doesn’t mean that you lose power, since if you fail to condition on a prognostic covariate you bias the treatment estimates downwards. In fact, generally you gain power by fitting prognostic variables.’
‘It’s as clear as mud,’ said Harvey. ‘But,’ he added, ‘suppose the hazard isn’t proportional. What then? Aha, that’s what they were up to in the Speculum. They were making sure that their analysis was robust to that assumption.’
‘They were doing no such thing,’ I said. ‘The log-rank test also requires proportional hazards. In fact it needs it at least as much if not more than the regression model. If there are no ties then the log-rank test gives the same answer as fitting the proportional hazards model without covariates and using the score test. Furthermore, whereas it is quite conceivable that the hazard might be approximately proportional given a more elaborate model but not so given a simpler one, the reverse is very unlikely.’
‘Well, I still don’t see it,’ said Harvey, ‘can you give me an argument I can understand as to why we shouldn’t minimise?’
It is here, gentle reader, that I must confess to having scraped the barrel. I had just wiped the floor with Harvey and he was too dim to see it. In exasperation I resorted to the Finally Decisive Argument. ‘The Agencies don’t like it,’ I said.
Reference to the Agencies galvanised Di out of her state of near catatonia. ‘The Agencies don’t like it?’
‘Yes,’ I replied, ‘there is even a Points to Ponder document arguing against it.’
‘Oh,’ said Harvey, ‘that’s a blow’ and he looked quite downcast, however Di was looking more and more confident and cleared her throat.
‘This debate has been very interesting and illuminating,’ she said, ‘however, before we get too carried away I think that we should consider the position of Company Regulatory Affairs at Panacea Pharmaceuticals and that is very clear.’
Good grief, I thought, if CRAPP is clear about something that will be a first. Don’t tell me that this perpetual fence-sitter has actually read this points to ponder document and understood it. I don’t often find myself agreeing with Clive Viper but I could see from the expression on his face that he too thought the same. ‘Do you mean to say that CRAPP has a position on minimisation?,’ he said.
‘Well not on minimisation per se?’ said Di.
‘Well what then?’ said Harvey.
‘We do most definitely have a position,’ replied Di, somewhat nettled.
‘On what?’ shrieked Angina, most aggressively for her.
On points to ponder documents. Our position in Company Regulatory Affairs, in fact it’s our considered advice, is that Panacea Pharmaceuticals should always follow exactly any recommendations in Points to Ponder documents.’
‘I don’t know what your problem is, Guernsey?’ Angina said to me afterwards. ‘It seems you got your way.’
How could I explain my disgust? I don’t give a damn for the opinion of CRAPP, still less for the Agency; it is only the truth that matters. When one has the brilliant shining light of statistical reasoning to guide one, one does not care to have to rely on the white stick and guide dog of regulatory advice.
Return to Guernsey McPearson Prose