NJR Data Quality Audit - a progress update from NJR Medical Director, Mr Martyn Porter
September 2017
Please note: This article is an update to the Data Quality Audit report on pages 18-20 in the NJR 14th Annual Report and as featured in the September 2017 issue of the Journal of Trauma & Orthopaedics
Data quality and validation are essential components of any audit or scientific research. As such, improving data quality remains the number one overall strategic priority for the National Joint Registry (NJR). Now entering its fifteenth year since its inception and with around 2.3 million records on the NJR database, the registry is a uniquely powerful resource. Not only to monitor the performance of implants, hospitals and surgical technique but to also drive quality improvement in the orthopaedic sector as a whole.
As such, for surgeons and patients alike, the necessity for having accurate and complete data is an absolute requirement. It is critical that we are collecting the most relevant, high quality data in order to provide robust evidence to support decision-making in regard to patient safety, standards in quality of care and overall cost effectiveness in joint replacement surgery. Quite simply, if the data is incomplete or incorrect, then false conclusions may be drawn from any analysis.
Furthermore, with our ever maturing dataset the NJR’s remit has naturally broadened and the ability for the dataset to drive forward change in other areas has grown – from patient recorded outcome measures (PROMs) to implant price-benchmarking, from research to service re-design. As well as also being an important source of evidence for regulators, such as the Care Quality Commission (CQC), to inform their judgements about services.
Undoubtedly many hospitals work hard in implementing an NJR process to ensure that all eligible primary and revision joint replacement operations are recorded on the database and put forward for analysis. However, the completeness of data within the NJR is reliant on the input at the local level, which is subject to variation across hospitals.
The NJR’s programme of work to improve the quality of our data has therefore been a key focus in recent years, driven through the NJR’s ‘Supporting Data Quality Strategy’. This has involved implementing and developing a series of initiatives specifically aiming to:
1. Support quality hospital data collection, entry and submission
2. Monitor data accuracy and data completeness
3. Facilitate validation through routine NJR reporting
4. Work collaboratively to promote the benefits of data validation
5. Provide appropriate external review to ensure compliance
The data quality audit
Key to this strategy has been the NJR’s national programme aimed at assessing data completeness and quality within the registry. Known more simply as the Data Quality Audit, the programme has allowed the NJR to compare patient record for record procedures recorded in local hospitals’ databases (Patient Administration System) to the registry, with the aim of investigating the accurate number of arthroplasty procedures submitted compared to the number carried out.
The importance of the Data Quality Audit is clear when considered through the context of the NJR’s primary aims. Unlike many other national audits, there are two principal outcomes of interest to the NJR: 1) mortality, and 2) revision surgery. Similar to other registries, data on mortality is collected via the Office of National Statistics (ONS) and linked to individuals within the NJR. However, it is important to remember that joint replacement is a very successful operation and – at the time of writing – all surgeons and hospitals have outcomes in line with the expected mortality rate.
More consistent with the primary aims of the NJR is revision surgery, an indication of implant failure or surgical performance. This is determined by linking a primary arthroplasty procedure to a secondary procedure, which typically occurs a number of years after the primary procedure. Therefore, compliance with reporting revision surgical procedures is essential to estimate implant failure rates and surgical performance more accurately.
What have we learnt so far?
The Data Quality Audit has recently completed its second year. In the first year, the audit was introduced to NHS Trusts and Health Boards only; making sure the initial rollout was not over reaching the NJR’s resource. I’m pleased to now be able to report on the audit’s findings from year one in detail below.
In the programme’s second year (2016/17), we were able to expand the Data Quality Audit to also check the quality of data in independent healthcare organisations, giving all hospitals – now in both the NHS and independent sector – the opportunity to demonstrate the highest possible standards of clinical governance, which all are striving to achieve.
It is too early to usefully comment on any analysis from the audit’s second year, but on the whole the local feedback has been positive. The engagement from the independent sector has been well received and it is clear that many NHS hospitals have improved their NJR process by taking on board feedback given the previous year. However, although any provider carrying out hip, knee, ankle, elbow or shoulder surgery is mandated to submit all eligible primary and revision procedures to the NJR since 2011, the Data Quality Audit has highlighted that many providers continue to face barriers in achieving this.
The audit’s first year highlights this. Completing the audit’s first year has been a huge achievement; the ambitious programme began in July 2015 and by the end of August 2017, 145 of 149 eligible NHS Trusts and Health Boards had fully completed the audit. Achieving such a high rate of completion offers an incredibly powerful snapshot of NJR data quality within the NHS, and doing so is thanks to the continued hard work of the NJR’s data quality audit team and colleagues on the NJR’s Data Quality Sub-committee.
Beginning in July 2015, each CEO received correspondence from the NJR inviting them to join the NJR’s data checking programme for the previous financial year (2014/15) and to identify a data quality lead to help hospitals complete the audit.
Clearly achieving this level of completion has taken time, particularly in terms of assessing consistency between the NJR and hospitals’ PAS. The process involved Trusts and Health Boards returning a file of patients for whom OPCS4 codes had been locally recorded; in other words, patient records indicating a primary or revision hip or knee replacement procedure in the financial year 2014/15. This was matched by the NJR against all the joint replacement procedures that the organisation had submitted to the NJR for the same timeframe. The possible outcomes for each record were:
a) A full match by patient ID, operation date and procedure (recorded OPCS4 codes and in the NJR)
b) Recorded OPCS4 codes but no NJR record identified
c) NJR record identified but no corresponding record or OPCS4 codes in PAS
Of the 145 Trusts and Health Boards that have completed their audit, a total of 117,115 (94%) procedures were matched between the NJR and local PAS extract (outcome (a) above).
19,489 procedures were found in the hospital PAS system but not on the NJR (outcome (b)). On further investigation, these procedures were found to have been outsourced (24%); identified as not being an NJR procedure (15%); an incorrect patient identifier had been used (12%); or another reason (10%). The remaining 7,536 (39%) records were eligible for NJR entry.
Regarding outcome (c), a total of 9,921 procedures were found to be on the NJR but not on the hospital PAS database. This is of particular significance in the current financial climate. As a simplified example, with each operation worth around £5,000 per procedure, this equates to over £49.5million in potential lost revenue across the 141 Trusts and Health Boards who completed the audit.
In summary, a total of 119,426 (58,204 hips, 58,911 knees, and 2,311 which could not be defined) procedures were identified between the NJR and local PAS extract. Of these, 106,030 (51,294 hips, 54,736 knees) were indicated to be primary procedures, and 11,085 (6,910 hips, 4,175 knees) were revisions.
95.69% (96.04% hips, 95.34% knees) and 90.27% (90.43% hips, 90.11% knees) of primary and revision procedures were recorded in the NJR respectively.
What does this represent?
The takeaway message from the audit’s first year is that whilst the overall scale of missing records was found to be low (5.82%), the proportion of missing revision records was found to be higher than that for primary procedures (9.47% for hips and 9.87% for knees). Put another way, the observed differences suggest potentially systematic under-reporting of revision procedures.
The failure of hospitals to upload revision procedures into the NJR is concerning, as linked revision procedures form the basis of analyses which investigate implant failure and surgical performance – which fundamentally underpins the core purpose of the NJR.
If systematic under-reporting of revision procedures is occurring, this is likely to bias results and reduce the statistical power of the NJR to quickly detect failing implants at higher than expected rates. The large size of the NJR will somewhat compensate for this when assessing failure rates at a national level. However, when attempting to subdivide data, for example by hospital or by surgeon, the reduction in statistical power and systematic under-reporting of revisions may be misrepresentative.
To put the importance of the audit and data quality into context, if data is missing at random, then comparisons of the NJR data at the level of the implant may still be valid, but comparisons of sub-samples of the NJR, such as surgeon or hospitals, are much more problematic.
Next steps for the NJR’s data quality work
Further investigation is required to ascertain whether these are random events or a systematic under-reporting of revision procedures. Analysis of the audit’s results in year two will help this and that work is currently underway.
Positively, we have already seen a number of NHS hospitals demonstrate a marked improvement in data quality following the previous year’s audit process. However, we are conscious that some hospitals continue to struggle with systemic process and resource problems, whilst others have not benefitted from clinical leadership on use of national audit and data quality improvement techniques.
To help providers overcome these barriers, the NJR’s Data Quality Strategy will be strengthened through the introduction of a dedicated NJR Data Quality Assurance Team. This team will provide the NJR with additional capacity to work closely with hospitals to identify and resolve data quality issues; review underlying data entry processes; and support process changes through examples of good practice. Perhaps most importantly though, the team will increase the automation of data quality reporting and feedback, where possible, thus making this area of work more timely and effective to providers.
The Data Quality Audit has lifted the lid on the NJR’s dataset. The overall scale of missing records in the registry is low, dispelling previous critique about the quality of the data in recent years. Yet, clearly more work is required to investigate why revision procedures are missing at a higher proportion, and then the registry can proudly declare its position as both the largest and the most robust arthroplasty register in the world.
Finally, I know that engaging in the Data Quality Audit has required a strong departmental effort, I would like to offer my thanks on behalf of the NJR to all staff who have worked to complete this important audit and who I hope will enact upon its findings going forward.
I look forward to keeping you abreast of further analyses and developments in this area of work for the National Joint Registry.