NJR Data Quality Audit - a progress update from NJR Data Quality Chairman, Mr Matthew Porteous
Last year, the NJR’s Medical Director, Mr Martyn Porter, provided a comprehensive progress report on the NJR’s Data Quality programme. That report outlined why a comprehensive data quality audit was needed and highlighted the results of the first year’s audit which was based on data submitted by NHS Trusts and Health Boards. This report includes extracts from Mr Porter’s original report and provides an update on progress.
Improving data quality remains the number one overall strategic priority for the National Joint Registry (NJR). The NJR’s ‘Supporting Data Quality Strategy’ has involved implementing and developing a series of initiatives specifically aiming to:
1. Support quality hospital data collection, entry and submission
2. Monitor data accuracy and data completeness
3. Facilitate validation through routine NJR reporting
4. Work collaboratively to promote the benefits of data validation
5. Provide appropriate external review to ensure compliance
The data quality audit
The NJR’s Data Quality Audit programme is designed to assess the completeness and quality of the data submitted to the NJR. It enables the NJR to compare patient records for procedures recorded in local hospitals’ databases (e.g. Patient Administration Systems) to the registry, with the aim of investigating the accurate number of arthroplasty procedures submitted, compared to the number carried out.
The importance of the Data Quality Audit is clear when considered through the context of the NJR’s primary aims. To achieve those aims, here are two principal outcomes of interest to the NJR: revision surgery and mortality.
Revision surgery is an indicator of implant failure or surgical performance. This is determined by linking a primary arthroplasty procedure to a subsequent procedure which typically occurs a number of years after the primary procedure. Therefore, compliance with reporting revision surgical procedures is essential to estimate implant failure rates and surgical performance more accurately.
Mortality data is collected via the Office of National Statistics (ONS) and linked to individuals within the NJR. However, it is important to remember that joint replacement is a very successful operation and, currently, all surgeons and hospitals have outcomes in line with the expected mortality rate.
What have we learnt so far?
For the first year of the audit, all of the 149 eligible NHS trusts and health boards had completed the audit. It showed that, whilst the number of missing records was low (5.82%), the proportion of missing revision procedures was higher than that for primary procedures (9.47% for hips and 9.87% for knees).
It has not been possible to determine the number of missing records from the second year of the audit (which included independent healthcare sector providers for the first time) because, at the time of writing, only 63% of audits undertaken in NHS trusts and health boards had been completed and, in the independent sector, only 59% had been completed. The results of the second year of the audit will be published later in the year.
In terms of process, it was also decided to contact hospitals directly, rather than via trusts and health boards. This has resulted in earlier engagement and improved response rates.
The audit process
This year the process has undergone some automation in order to reduce the workload on the Data Quality Assurance Team and it is expected that that automation will continue into 2018/19 to provide assistance to hospital staff.
For each year of audit, the NJR produces an ‘audit tool’ which is sent to hospitals for completion. The hospital populates the tool with the codes of all locally recorded procedures. The NJR then compares the completed file returned by each hospital to records submitted to the NJR. There are three possible outcomes:
• a full match.
• locally recorded procedures which have not been submitted to the NJR.
• an NJR record identified but no corresponding, local record.
The results of the comparison are then forwarded to each hospital as queries. Once the discrepancies have been accounted for, a final data quality audit report is produced for each hospital. Hospitals are then expected to submit the details of missing procedures to the NJR.
Progress for each year of auditThe time taken to complete each year of audit inevitably results in some overlap between years. In order to avoid over-loading hospital staff, the next year’s audit tool will not be sent to a hospital until the previous year is completed, i.e. audit tools for 2016/17 will not be sent until such time as the audit for 2015/16 is complete. As at the 1st April 2018, the progress of each audit was as follows:
2014/15 - One NHS trust had yet to complete the audit for 2014/15 but were expected to do so.
2015/16 - The progress of the 2015/16 audit is shown in the pie charts below:
2016/17 - The progress of the 2016/17 audit, for both the NHS and the independent sector is shown in the pie chart below. Note that audit tools for the 2016/17 audit will not be sent to hospitals until they have completed the 2015/16 audit.
Summary
Although the Data Quality Audit process is still being embedded in some hospitals, it is clear that resources remain an issue for many, despite the appointment of an NJR Data Quality lead. This has led to an increase in the number of requests for visits by the NJR’s Data Quality Assurance Team, despite the provision of online training aids, which will inevitably slow progress.
However, the timeliness and quality of responses have improved since the 2014/15 audit and, with greater automation and additional developments to help hospitals deal with queries about discrepancies, the data quality audit will continue to become increasingly embedded in each hospital. Awareness of the need for ongoing data quality will ensure that fewer queries are raised in subsequent annual audits and that the audit is, therefore, significantly quicker to complete.