NJR Data Quality Audit - a progress update from NJR Data Quality Chairman, Mr Matthew Porteous
Last year we included in the Annual Report a detailed update on progress of the Data Quality Audit programme. This year we report the further progress of this initiative.
Improving data quality remains the number one overall strategic priority for the National Joint Registry (NJR). The NJR’s ‘Supporting Data Quality Strategy’ has involved implementing and developing a series of initiatives specifically aiming to:
• Support quality hospital data collection, entry and submission
• Monitor data accuracy and data completeness
• Facilitate validation through routine NJR reporting
• Work collaboratively to promote the benefits of data validation
• Provide appropriate external review to ensure compliance
The Data Quality Audit
The NJR’s Data Quality Audit programme is designed to assess the completeness and quality of the data submitted to the NJR. It enables the NJR to compare patient records for procedures recorded in local hospitals’ databases (e.g. Patient Administration Systems) to the registry, with the aim of investigating the accuracy of the number of arthroplasty procedures submitted to the NJR, compared to the number carried out.
The importance of the Data Quality Audit is clear when considered in the context of the NJR’s primary aims. To achieve those aims, there are two principal outcomes of interest to the NJR: revision surgery and mortality.
Revision surgery is an indicator of implant failure or surgical performance. This is determined by linking a primary arthroplasty procedure to a subsequent procedure which typically occurs a number of years after the primary procedure. Therefore, compliance with reporting revision surgical procedures is essential to estimate implant failure rates and surgical performance more accurately.
Mortality data is collected from Civil Registration data obtained from NHS Digital and linked to individuals within the NJR. However, it is important to remember that joint replacement is a very successful operation with a very low mortality rate and, currently, all surgeons and hospitals have outcomes in line with the expected range.
The audit process
For each year of audit, the NJR produces a ‘data template’ which is sent to hospitals for completion. The hospital populates the template with the codes of all locally recorded procedures. The NJR then compares the completed file returned by each hospital to records submitted to the NJR. There are three possible outcomes:
• A full match
• Locally recorded procedures which have not been submitted to the NJR
• An NJR record identified but no corresponding local record
The results of the comparison are then forwarded to each hospital as queries. Once the discrepancies have been accounted for, a final data quality audit report is produced for each hospital. Hospitals are then expected to submit the details of missing procedures to the NJR.
The first year of the audit (2014/5), was confined to 149 eligible NHS Trusts and Health Boards all of whom completed the audit
The second year of the audit (2015/16) is not directly comparable as in the light of experience it was performed by hospital rather than trust (some trusts comprise more than one hospital) and for the first time included the private sector. The third year (2016/17) is directly comparable to the second.
Audit progress
We have now fully completed audits for 2014/15 (NHS Trusts only) and 2015/16 (NHS and Private hospitals). 2016/17 is almost completed with only one hospital failing to do so at the time of writing. The 2017/18 audit is now well underway and has been completed by 48% of all units though this percentage rises on a weekly basis.
It is noticeable that as organisations become familiar with the audit process the turnaround time by units has reduced, though there remain a few hospitals with significant resource and compliance issues.
The time taken to complete each year of audit inevitably results in some overlap between years. In order to avoid over-loading hospital staff, the next year’s audit tool is not sent to a hospital until the previous year is completed, i.e. audit tools for 2017/18 were not sent until the audit for 2016/17 had been completed.
What does the audit show?
The primary aim of the audit is to ensure that we now have close to 100% accuracy in recording primary and revision hip and knee replacement since 2014, albeit with some lag for records that are identified as missing by the audit and have to be entered at a much later date.
We have looked at the number of missing records at the time the audit was conducted and looked at the balance of missing primary and revision cases. These are shown in the table below for the three completed years and the incomplete 2017/8 year.
While the overall number of cases missing (5 to 8%) is relatively low the higher proportion of missing revisions (9 to 12%) is of concern because it is only by knowing that a case has been revised that we are able to calculate failure rates of implants, units and surgeons.
We were concerned that there was a deterioration in submission rates for revisions between the 2014/15 and 2015/16 audits. There are however possible explanations. First, the second audit included private units for the first time. These tend to better at completing their audit forms than NHS hospitals, but perform fewer revisions and this may have had an impact on the proportion of revisions missing (in comparison to primaries). Secondly there was considerable delay in completing the first audit as it was a new concept. This meant that all the cases for 2015/16 had been entered before any changes had been made as a result of lessons learnt when completing the 2014/15 audit. By the 2016/17 audit many hospitals had made changes to their processes and we are now beginning to see some improvement in initial submission rates, which we hope to be confirmed when we have a full picture from 2017/18.
Not too much should be inferred from the percentages of missing data in the 2017/18 audit as we know that trusts that complete the audit cycle fastest also tend to be those with the best internal processes, and therefore tend to have fewer missing cases.
While the cases identified by audit itself are eventually entered, it is likely that historically (before 2014) we have been underreporting revision rates at a higher level than those of primaries. We have conducted some analysis on the data we hold for the three completed years and there is some evidence to support the hypothesis that this is because revisions are more likely to be done as emergencies (some infections and dislocations and most periprosthetic fractures) and we know that these are more likely to be missed than those done on routine lists. With continued Data Quality Audits these cases have all been identified since 2014 and looking forward this will cease to be an issue.
Data accuracy and potential ‘Never Events’
In addition to the Data Quality Audit which has focused on ensuring that we do not miss any records we have also been trying to improve the quality of the data submitted.
We are conducting a retrospective review of cases from the entirety of the database where there appear to be data entry errors including:
• Implant incompatibility (size or side)
• Operation performed after date of death
• Revision performed before primary joint replacement
• Duplicate primary joint replacements on the same side
• Revision on day of primary
• Patients with ASA 5
• Hemiarthroplasty revision submitted on H2 form
Where the patient is still alive, and the case has not been revised, hospitals have been sent a list of such cases and asked to review and correct them. Almost 18,000 of these historical cases were identified and two thirds of these have now been resolved and where appropriate corrected.
From August 2017, the rules of data submission have been tightened to highlight to hospitals where data is either incomplete or unlikely to be correct (e.g. more than one femoral stem used). We also highlight where incompatible implants have been submitted (e.g. a left knee implant inserted into a right knee) and follow these up closely as potential Never Events. While most of these cases turn out to be administrative errors, we have identified 33 genuine never events (of 723 potential cases) involving hip head/cup size mismatches and 12 genuine wrong sided knee replacements (of 3,257 potential cases). These are notified to the hospitals for further investigation.
Future developments
The current audit process has validated the method of checking for missing records but has proved time and labour intensive for both NJR and hospital staff. We are now trialling a semi-automated process which allows units to check their data quality on a monthly or quarterly basis. This will greatly reduce the number of mismatches that have to be checked each time it is run, and the pilot suggests this rapidly becomes part of the normal workflow. We are planning to roll this out fully by March 2020.
We are currently concluding a pilot audit of shoulder replacement entries in the North East of England as we are planning to include shoulders in the automated audit process once it is running smoothly for hips and knees, with a target date of 2021.
Eventually we plan to include ankle and elbow replacement in the audits, although issues with variability in the use of OPCS codes for these procedures will require further adjustment to the auditing method.
Summary
We are making steady progress in improving both the quality and quantity of the submissions to the NJR particularly for hip and knee replacement (which make up 95% of all submissions).
We believe that we now have processes in place to ensure virtually all hip and knee replacements performed since 2014 have been submitted; and are actively working to make the way we do this less resource-intensive and timelier for both hospitals and the NJR. We will be rolling this process out to include shoulders and ultimately ankles and elbows.
At the same time, we have set systems in place to identify potential data entry errors and have those corrected, as well as identifying and notifying potential Never Events to the hospitals in real time.
Most hospitals have responded positively to the lessons learnt from the data audit, and changes made in the light of recommendations have meant improved initial compliance and less work for those units. There do however remain a small number of hospitals with very significant compliance and resource issues.