Can we trust health economic evaluations of new pharmaceuticals? Examining the impact of the researcher’s degrees of freedom using a many-analysts approach

Research problem and specific questionsHealth economic evaluations of pharmaceutical drug treatments (pharmacoeconomic evaluations)—the formal modeling and comparison of benefits and costs of a new drug—play an increasingly important role in decisions regarding which new drugs to reimburse and introduce in many healthcare systems around the world.

Pharmacoeconomic models are often based on complex simulation models that extrapolate long-term benefits and costs.

Hence, the design choices necessary to build and analyze these models are numerous and can be considered as the researcher´s degrees of freedom.

If the researcher´s degrees of freedom substantially impact modeling results, the validity of these models as an input to decision-making can be questioned.

Using a cancer drug context, we aim to (1) identify to what extent current pharmacoeconomic models are transparent, and as a consequence reproducible, with regards to all the design, value and analysis assumptions required to construct them.

Further, we aim to (2) investigate how much the researcher’s degrees of freedom in design and analysis choices affect pharmacoeconomic-modeling results.Data and methodTo accomplish Aim 1, we will create and apply a novel reproducibility checklist to published pharmacoeconomic cancer drug models, perform data extraction, and synthesize and categorize information.

To accomplish Aim 2, we will apply an innovative many-analysts approach and instruct four pharmacoeconomic research teams to carry out independent model analyses of two novel cancer drugs.

Differences in end results will be extracted, and the primary causes for variation in the modeling results will be identified based on formal statistical comparison of models as well as on qualitative evaluation of research logs.Societal relevance and utilizationGiven the large economic and public health consequences of reimbursement decisions, it is essential that the pharmacoeconomic evidence provide valid input to decision-makers.

Our long-term goal is to develop a framework and guidelines for improving reproducibility and reducing the impact of researcher degrees of freedom in pharmacoeconomic models.Plan for project realizationThe project team consists of four health economics research groups in Sweden with a broad and extensive experience in pharmacoeconomic modeling.

Potential threats to implementation have been identified, as well as proposed solutions to overcome these threats.