On Why Published Clinical Trial Data Should be Made Public

[frances]

This article from an Australian website series April, 2012 argues that clinical trial results, notably those for targeted therapies
in cancer, are not designed with meaningful parameters and are not independently evaluated or
released to the public in a departure from the early days of cancer medicine.

Ian Haines makes an argument for research data from published papers to be made public.

It’s time for medical journals to mandate data release and independent audits for all clinical trials submitted for publication. And here’s why.

Over the past few years, editorials and commentaries in some high-profile medical journals and magazines have heralded the exciting revolution in personalised cancer therapy. They talk of treatments involving combinations of antibodies and enzyme inhibitors specifically designed and tailored for each individual’s cancer.

But most of these many new treatments cost between $40,000 and $120,000 per patient per year. And results of clinical trials so far show miniscule benefits that require extraordinary statistical gymnastics to achieve the magical “significance” level.

The problem is that these usually commercially-sponsored studies are designed by the sponsor and the statistical endpoints are no longer one or more of the traditional “gold standards”.

Those gold standards include:

•extending overall average survival;

•improving “quality of life” measures;

•reducing toxicity of treatment and;

•reducing the cost of treatment.

So many contemporary clinical trials have stopped measuring things that matter, such as quality of life and overall survival. Instead, they measure things that are easier to achieve positive results for, such as how long a patient’s X-ray stays stable in the very subjective opinion of an investigator.

These measures are clinically inconsequential, often unverifiable because of unavailability of the “raw data” and are far too subjective.

In addition, the raw data from these studies are protected – seen and analysed only by the sponsor’s data team. The results are then usually written by “medical writers” employed by the sponsor together with clinicians who often have significant financial conflicts of interest with the sponsor.

The final analyses may have little relation to the originally stated primary and secondary endpoints of the study and may emphasise other, often inconsequential, positives. And the abstract, which is a summary of the results, often emphasises positives that are not even supported by the data.

This is akin to a large public company producing its end-of-year financial results but keeping the primary data hidden, and then doing an audit on itself.

So despite hyperbole of medical “revolutions”, the US Food and Drug Administration (FDA) recently withdrew its previous approval of the world’s biggest selling multi-billion dollar breast cancer drug, bevacizumab. This step is almost unprecedented.

And the National Institute for Health and Clinical Excellence (NICE), the equivalent body in the United Kingdom, issued the shock announcement that it didn’t consider the use of any of the three “blockbuster” new targeted therapies for bowel cancer – cetuximab, bevacizumab and panitumumab – to be a cost-effective use of National Health Service (NHS) funds.

In fact, when the FDA examined the raw data of the key trial that had produced the initial approval for bevacizumab, its independent reviewers had to reverse the initial assessment in 51% of the cases.

When I completed my medical training in 1978, the first cures of solid tumour cancers by chemotherapy (as distinct from acute blood cancers or acute leukaemias) had just been reported in the medical literature. Hodgkin’s disease, aggressive non-Hodgkins lymphomas and advanced testicular cancer, which had hitherto usually been fatal, had disappeared in many patients with combinations of new and older drugs.

This was a watershed moment for cancer treatment in the twentieth century and generated great optimism for the future.

Clinical trials needed to prove the benefits of new treatments could easily be done by well-organised research groups. We didn’t need statisticians to dissect the p-values and hazard ratios of multiple opaque statistical analyses to tell us that these were significant advances in treatment.

And the drugs were affordable.

The breakthroughs of the time spawned the new speciality of medical oncology even though we still didn’t have effective treatments for the common and usually fatal advanced solid cancers such as bowel, lung, breast, melanoma or prostate.

And as we did our daily ward rounds, seeing the many referrals with these conditions, our most common note in the medical record was that “chemotherapy had nothing meaningful to offer”.

To prevent us returning to those days, it’s time to mandate data release and independent audits for all clinical trials. This will increase the chance of trials that are designed to measure relevant endpoints, of producing results that are verifiable, and of generating analyses and conclusions that match the original design endpoints.

To avoid mirages in our quest for cancer treatment oases, our ongoing need for innovation, hard work and entrepreneurship will need to be carefully balanced with more transparency and accountability to independent reviewers and assessors.

++

[rp1954]

Effective combinations of old drugs and tests have been given short shrift on a number of occasions. Trials without overall survival reflect less robust end points, easier to fudge. There is a lot of systemic bias for expensive drugs, whether they work well or not.

[Laurettas]

Thank you for this article, Frances. I agree with it. I have read some of the trial results and they will call something significant if it lengthens life by say 1.4 months but another drug will be labeled as not helpful if its increased life stat is 1.2 months. Never understood that. I think we have an elephant in the room that no one wants to talk about as far as these new drugs and this needs to be addressed. The advances are far to miniscule in my opinion, especially considering the huge cost.

[Bob_Weiss]

Many of the new so-called miracle drugs for CR cancer turn out to have rather minimal benefits and work for only some of the patients who meet the medical criteria for the drug. Yet their cost is always very expensive. I don't have the expertise to know whether this is largely due to drug companies supporting the low risk research projects that will have only minimal value for patients; or whether the lack of major chemo advances in CR cancer are due to lack of researchers' knowledge of what is going on. Fortunately surgery in CR cancer can have great value, and radiation can have substantial benefit also.

[frances]

hi guys,

Like you say rp, I definitely read how drugs that were under investigation for cancer got tossed aside when something sexier came along, that being the targeted therapies.

Laurettas, hi there, yes, amazing you say 1.4 equalled 1.2 months in study OS value. Doesnt surprise me after reading this opinion article. Then the media gets hold of something and just touts it. Perfect example was that bevacizumab story out of ASCO just this month. I guess if you think one month translates to major benefit in a study re CRC you can tout the drug into such headlines as Avastin Non Stop! Ok, slight exaggeration, but you get me.

Bob, great reminder that surgery and other treatments are making advances, too!

Frances

[Bev G]

If they're published, aren't they by definition public? Results of studies often don't make it into practice protocols for all sorts of reasons, some of them good, and some of them bad, but if they're published studies, they should be available for review, to anyone willing to research things a bit.

Yes, I wondered and was confused by that myself when I read the OP. Glad you asked.

[frances]

Even though papers on clinical trials are accepted for publication by highly cited journals, the raw data from those trials is by no means made public. Sometimes it is only partially supplied to the publishing journal, which may not require any review of for example micro assay data in its editorial guidelines.

Data sharing and transparency in clinical research are quite an issue among competitive research scientists.

There is no legislation in the US to ensure data availability. There's a few papers on this issue on the web, incuding a study of public availability of published research data in high impact journals:

http://www.plosone.org/article/info:doi ... ne.0024357


Frances

[Bev G]

Data handling and interpretation is very strictly regulated and reviewed at every step, both by academic statisticians and by whomever is sponsoring the clinical trial. Of what benefit would it be to provide raw data to the public? What would anyone do with it? Collaboration does exist between scientists, in a wide variety of settings, IMO, the public wouldn't have any idea what to do with the information about all of the machinations of this collaboration. I would be fascinated to know what Jeremy thinks about this.

[Gaelen]

Well, I'm not Jeremy, but I can tell you after watching the members of this and several other forums for the last eight years, that in my experience, little/no good comes of providing deep scientific data (raw data) to non-medically-trained patients in any environment where you aren't also prepared to teach them what they just read. And the teaching process is WAY longer than a minute or two.

Heck, we've all seen dozens of examples where providing data basic to their own reports and scans has caused confusion levels and conclusion leaping that would impress a salmon swimming home. People aren't willing to look up the language on their scan and blood reports, the side effects of their chemo drugs - relatively simple, easy-to-read stuff. Goddess help up whenever we try to discuss statistics, their relevance, etc. When people can't even accept the responsibility for looking up side effects on their own, when they can't/won't even take the time to read an abstract of a clinical trial study, then WTH good would it do to overwhelm them by adding to that publication the volumes of raw data that was collected to get the final results.

So what do they do?
They come to forums like this one to ask people who aren't doctors (aka total strangers), to give them an interpretation of their scan report, blood draw results, side effects to expect, etc.
All because they won't take the responsibility to look up and learn the BASIC medical language involved in these kinds of reports.

IMO, The last thing 99 per cent of patients, families and caregivers need is MORE confusing information...like raw data. For the rest of the people who DO sufficiently understand (or think they understand) raw data and stats well enough to draw intelligent conclusions, find a clinical researcher who likely has access to what you want to read, and beg/borrow/bribe your way into getting the raw data.

Or read online published peer-reviewed journals and pay extra for the access to the raw data.

[frances]

Let's stay on topic. This is not about literally making data 'public'. And it's not about patients. By making public what is meant is that a kind of institutional governance is being discussed, not a two toned brochure for the doctors waiting room.

This starts with an opinion article that claims the actual science in clinical trials is being distorted by a publishing and review process that is flawed.

The fact is the highly esteemed peer review journals are not requiring the teams whose studies they publish to provide complete data to them. They're not auditing what they publish. The study link I cited in my last post has a table of top science journals that are not requiring researchers to be able to back their hypotheses. One often cited oncology journal has no constraints in place.

So the argument goes that in the absence of an incentive or should I say disincentive or constraint for transparency, flawed science is inevitable. The end point example Laurettas gave, etc etc. The Avastin case. So the author calls for independent auditing. Especially since this industry is hardly exempt from profit, greed and skulduggery.

I personally am disturbed by these findings.

Frances

[Bev G]

I'll just say this, then drop it. Having personal knowledge of what it takes to get a paper published (many of them) in a scientific, peer-reviewed journal (and very particularly ANYTHING having to do with a study funded by the government or a pharmaceutical company) I have complete confidence in any and all statistics being thoroughly vetted and interpreted prior to publication. To go back to the raw data, begin again, re-do what has already been done would be not just unduly cumbersome, but prohibitively expensive, further increasing costs and delaying getting information to those who are in a position to use this information. Like it or not, scientific, peer-reviewed journals are by and large produced for the scientists (physicians and researchers) who will take others work, and hopefully, run with it, incrementally heading towards the cure. These raw data ARE available, and in my experience, authors are willing to share it when legitimate reasons exist (it make take a spare hard-drive to send it off, though )

[jscho]

Bev, you stated my opinion very nicely.

Each scientific field has its own peculiarities, but in my field it would be a waste of time to validate all information in a submission when refereeing. Typically, I will check mathematical derivations but stop short at reproducing data (from modeling or simulation) or verifying statistical analysis. In mathematics, proofs are checked for their rigor and this slows the rate of publication. Mathematicians typically publish few papers, so the time a manuscript is under review isn't important. I have both asked and been asked for data, and this is the norm in the scientific community.

Scientists are not immune to human faults, and there are incidents of data falsification. In my field, anything truly interesting is often verified explicitly and reproduced, so there are implicit checks and balances in the system. The reputation of a researcher is of critical importance and has an influence on how his research is perceived. It would be quite foolish to cheat or mislead in any way. I think the temptations are much greater in biomedical researcher where more grant money and prestige are at play.

Best,
Jeremy

[frances]

Yes controls in science publishing are rigorous -- and so they should be. The entire field is under review right now w open access another issue that promises enormous impact.

Jeremy thank you for your input on math. However, as you note, math isn't driven by science dollars. Clinical research in drug discovery and development does not hinge on trust and reputation alone. Not in today's world. Today's GFC world. Or the one where world class Aids researchers fought tooth and nail over who discovered HIV.

Clearly the opinion of this respected oncologist and others in the field is that a laxity in a world where let's face it, transparency is demanded in every industry, is eroding the science itself. This same science is endorsed by journals, taken up by the media and fuels the stock exchange. And on and on.

If too many studies with a whole fruit basket of end points and parameters are being published in oncology without the maximum standards that leaders in the field are calling for, but by a bunch of corporate owned publishing houses with their own agenda, then I'm going to get concerned as a patient. Ultimately. If I was a breast cancer patient on Avastin it's possibly affected me directly that half the published research on that drug and its approval for me has been withdrawn.

I personally can't see the argument or that there is one for less accountability, when that's all that's being called for by the author here. Like do you want less fluoride in your water? So it takes them time to assemble their data? In the digital age their data should be compact and available, as I'm sure it is when it's shared across the world, lab to lab. I'm happy to improve my knowledge on how this is done these days.

Frances

[Gaelen]

Let's stay on topic. This is not about literally making data 'public'. And it's not about patients. By making public what is meant is that a kind of institutional governance is being discussed, not a two toned brochure for the doctors waiting room.
...So the argument goes that in the absence of an incentive or should I say disincentive or constraint for transparency, flawed science is inevitable. The end point example Laurettas gave, etc etc. The Avastin case. So the author calls for independent auditing. Especially since this industry is hardly exempt from profit, greed and skulduggery.
I personally am disturbed by these findings.

"Let's stay on topic." ???
Seriously, Frances, if you had wanted the conversation to go a certain way, you might have explained your personal definition of "public" in the OP or what you were disturbed about (no, your OP wasn't clear about either of those things.)
In light of the limits of your OP, every reply has been completly on topic.
When will people recognize that once they put a post out there, they don't actually control the flow or direction of a thread?

[frances]