Monitoring quality indicators for the Xpert MTB/RIF molecular assay in Ethiopia

Summary

Introduction

In Ethiopia, >300 GeneXpert instruments have been deployed for tuberculosis (TB) testing using the Xpert MTB/RIF cartridge. Implementing quality indicators is necessary for monitoring and evaluating the quality of Xpert MTB/RIF diagnostic services.

Objective

To assess the use of quality indicators for the Xpert MTB/RIF molecular assay in Ethiopia and to compare the findings with the predefined targets described in the literature.

Methods

Clinical specimens collected from patients with suspected TB were subjected to Xpert MTB/RIF testing at the National TB Reference Laboratory (NTRL) between January and December 2018. Data were collected from GeneXpert software and Laboratory Information System (LIS) databases. Quality indicators were calculated and analyzed. Bivariate and multivariate analyses were performed using SPSS software version 20 (SPSS Inc., Chicago, Illinois, USA).

Results

Of the 2515 specimens tested, 2274 (90.4%) had successful test results; 18.2% were positive for Mycobacterium tuberculosis (MTB). Among MTB positives (n = 413), 4.8% and 1.0% were rifampicin (RIF)-resistant and RIF-indeterminate cases, respectively. Unsuccessful results were 241 (9.6%); 8.9% of the total number of tests were errors, 0.04% had invalid results and 0.6% ‘no result’. The most frequent error was probe check failure (error 5007). Instrument module A4, B2, B3, C3, and D3 (p<0.05) and tester experience (p<0.05) had a statistically significant association with errors in multivariate analysis. Additional 42 MTB cases (9.2% of the total cases) were detected among unsuccessful results by follow-up tests. Sixty-four percent of the initial test results were released within the turnaround time (TAT) ≤24 hours.

Conclusion

Most of the quality indicators for the Xpert MTB/RIF molecular assay were maintained within the targets. However, the error rate and TAT were out of the targets. Defective modules and lacking experience were the factors affecting successful test outcomes.

Keywords:

tuberculosis – Polymerase chain reaction – Specimen preparation and treatment – Ethiopia – Sputum – Government laboratories

Introduction

The World Health Organization (WHO) has endorsed use of the Xpert MTB/RIF assay (Cepheid, Sunnyvale, CA) for the detection of Mycobacterium tuberculosis (MTB) and associated rifampicin resistance near the point of care, facilitating rapid diagnosis of tuberculosis (TB) and drug resistant TB (DR-TB) in adults and children with presumptive pulmonary and extrapulmonary TB [1–3]. As a result, Xpert MTB/RIF testing is being scaled up to all over the world. In Ethiopia, over 300 GeneXpert instruments have been deployed in different health facilities since 2012. The national Xpert MTB/RIF implementation guideline-recommended applying the Xpert MTB/RIF technology in high-risk groups for DR-TB, HIV seropositive individuals, children (<14 years of age), and patients with presumed extrapulmonary TB [4]. In August 2018, the Ethiopian National TB Control Program recommended that the Xpert MTB/RIF assay be used for testing on specimens from all presumptive TB patients irrespective of risk for DR-TB, HIV status, and age of the patient if Xpert MTB/RIF is accessible [5]. However, the quality of Xpert MTB/RIF testing has to be ensured in order to maximize the benefits for patient care and rapid diagnosis.

Xpert MTB/RIF is an automated molecular assay that simultaneously detects MTB and its resistance to rifampicin in less than two hours, uses heminested real-time polymerase chain reaction (PCR) assay to amplify MTB specific rpoB gene sequence [6]. The assay uses five different probes (A, B, C, and D) with molecular beacons in detecting mutations within the rifampicin-resistance determining region. The test integrates sample processing and PCR in a disposable plastic cartridge containing all reagents required for mycobacterial lysis, DNA extraction, amplification, and detection [2].

Quality indicators measure the degree to which a set of inherent characteristics fulfill performance requirements [7]. Moreover, the indicators validate how well the laboratory meets the requirements of the quality of the testing processes (pre-analytical, analytical, and post-analytical phases). According to the ISO 15189 standard, the laboratory should establish quality indicators for systematically monitoring and evaluating the laboratory’s contribution to patient care [7]. The quality indicators should be periodically reviewed to ensure their continued appropriateness.

For TB culture test, the quality indicators have been comprehensively reported in different mycobacteriology laboratories [8–10]; however, quality indicators for Xpert MTB/RIF assessment was made in Ethiopia and thus was not yet reported. We implemented the quality indicators recommended for Xpert MTB/RIF at the National TB Reference Laboratory (NTRL) of Ethiopia. Targets were set for the quality indicators from guidelines [11, 12] or literature [13–18] (Table 1). Any observed changes outside of the targets require an investigation for identifying potential causes. Therefore, the objective of this study was to assess the quality indicators for Xpert MTB/RIF molecular assay on the basis of the predefined targets described in the literature.

Materials and methods

Sample collection and processing

For patients with suspected pulmonary TB, a single spot sputum specimen (a minimum of 1.0ml) was collected using a sterile 50ml Falcon tubes following proper patient instruction at the reception unit of Ethiopian Public Health Institute (EPHI). Non-respiratory specimens were collected aseptically using an appropriate procedure by specially trained clinicians and the specimens transferred into a sterile 50ml Falcon tube and sent to NTRL of EPHI. Specimens were processed as previously described and as per manufacturer’s recommendations [2, 19]. In the case of unsuccessful (error, invalid, and no result) and RIF resistance indeterminate test results, repeating a test was carried out using the leftover Sample Reagent (SR)-treated sample within 12 hours (if kept in a refrigerator at 2–8°C) or from a newly collected specimen. Sixteen modules GeneXpert instrument was utilized for sample testing during the testing period. The GeneXpert® Dx Version 4.7b Software was used for Xpert MTB/RIF testing.

Test related data collection and analysis

For each specimen, the following information was collected: laboratory identification number, referring health facility, specimen type, specimen quality (in case of sputum), specimen volume, Xpert MTB/RIF test result, error code, reagent lot, dates and times of specimen collected, tested and reported, dates and times of retesting (in the initial test was unsuccessful; error, no result and invalid, and indeterminate), tester identifier, and tester experience (<2yrs., 2–3 yrs., and >3 yrs.) in Xpert MTB/RIF testing. The details of test-specific errors were collected from the Errors tab of the View Results window. Each error codes were further defined based on code definitions in the GeneXpert Dx System Operator Manual[20]. The quality indicators; percentage of samples reported as MTB detected (Indicator 1), RIF resistant MTB (Indicator 2), RIF indeterminate MTB (Indicator 3), error (Indicator 4), invalid (Indicator 5), and ‘no result’ (Indicator 6), were calculated and analyzed as defined in Table 1. Turnaround time (the period between the specimen receipt and the test report released from the laboratory) (Indicator 7) was also calculated as one of the quality indicators. We analyzed the quality indicators by considering the initial test outcome only, but not the retesting results. The calculated value of the indicators was compared against the targets. Bivariate analysis was performed using SPSS version 20 (SPSS Inc., Chicago, Illinois, USA) to identify the associated causes among the possible factors for the indicators out of the limit or target. Multivariate analysis was performed using models that included a variable that was significate in the bivariate analysis (p ≤ 0.2). A p-value <0.05 was considered statistically significant. As this was a retrospective study using anonymous data, ethics approval not sought. All data were fully anonymized before accession.

Results

Demographic and clinical characteristics

A total of 2515 clinical specimens were collected from 2441 presumptive TB patients during the period between January 01, 2018 through December 31, 2018; 1895 (75.3%) were respiratory and 620 (24.7%) were non-respiratory specimens. The majority of the patients from whom specimens collected were male (57.9%); the median age of patients was 38 years (IQR, 27–54) (Table 2). The specimens were collected from patients found in the Addis Ababa City, the Capital of Ethiopia. The large majority of patients (96.1%) provided a single specimen; 66(2.7%) patients provided two specimens and 4 (0.2%) patients provide three specimens. Twenty-five patients (1.0%) submitted two or three clinical specimens from different sources or anatomical sites.

Xpert test results

A total of 2515 samples were analyzed using the Xpert MTB/RIF molecular assay over the one year study period; the mean number of samples processed each month was 210 (range 36 to 296). Of these, 2274 (90.4%) had successful Xpert test results. Among the successful test results, MTB positivity was 18.2% (95% CI: 16.8–19.7). Of the latter, the percentages of RIF resistance and RIF resistance indeterminate were 4.8% (95% CI: 2.9–7.0%) and 1.0% (95% CI: 0.2–2.2%), respectively. The proportion of unsuccessful test results of the initial testing was 9.6% (Table 3).

Xpert retesting results

Among the 241 unsuccessful test results (error, invalid, and no result), 232 (96.3%) were retested using leftover SR-treated or newly collected samples. The median time to perform the retesting was 122.5 (IQR, 88–279.8) minutes. In the retested group, the percentage of MTB positivity was 17.9%. Thus, additional 35 MTB cases were detected by retesting initially unsuccessful tests. Two (5.7%) of the retested positives were RIF resistant while one was RIF indeterminate. Thirty-six (15.5%) of the retested specimens did not yield any result (Table 4). Out of the 36 unsuccessful primary retests, a secondary retest was conducted on 27 samples (75%) and results available for 22 (81.5%) of them. The secondary retesting identified five additional MTB cases; all of them were RIF-susceptible and 17 negative results. Thus, in the secondary retest, five additional errors were recorded. Two of the five errors were retested for the third time; one of them was RIF susceptible TB and the other was negative for TB. Therefore, overall 42 MTB cases (9.2% of the total detected cases) were detected by various levels of retesting.

A total of 261 retests were performed because of unsuccessful tests and unsuccessful retests. Considering the direct reagent cost of the manufacturer (9.98 USD per Xpert MTB/RIF cartridge; the negotiated public sector pricing) [21], about 2,604.78 USD (261*9.98) was per annum. The cost of unsuccessful tests was estimated to be 2,315.36 USD (232*9.98) whereas the cost of unsuccessful retests was estimated to be 289.42 USD (29*9.98) per annum.

Quality indicators of Xpert MTB/RIF molecular assay

Four hundred fifteen MTB cases were detected from the 2274 specimens tested successfully using Xpert MTB/RIF assay. Therefore, the percentage of annual MTB positivity (Indicator 1) was 18.2%, ranging from 14.3–25.5% across months. Similarly, the percentages of RIF resistant MTB (Indicator 2) and RIF indeterminate MTB (Indicator 3) were 4.8% (ranging 0.0–8.8%) and 1.0% (ranging 0.0–20.0%), respectively (Table 5).

A total of 225 tested specimens were with an error result. The annual error rate (Indicator 4) was 8.9%, varying from 2.2–15.8% across the months. The error rate was higher than the target (<3%) in all months of the year, excluding October and December (Table 5). Overall, 234 total error codes were recorded from 225 error test results. Six error test results had multiple types of error codes; two or three error codes occurred in combination. All happened in combination with error code 5006 i.e. 5006|1001|1002 (n = 3), 5006|5007 (n = 2), and 5006|5017 (n = 1). The error codes or messages were categorized by error types (Table 6). The most common error type was post-run analysis error (92.7%, 217/232). Of the latter, the predominant error code was 5007 (92.6%, 201/217) due to Probe Check failure. Operation terminated errors (2008 and 2014) and run-time errors (1001 and 1002) were also recorded by 4.7% and 2.6%, respectively (Table 6). However, there was no error associated with cartridge loading and self-test.

Only a single invalid result case was reported in the month of September 2018, which made the annual invalid rate (Indicator 5) 0.04% and it was within the target (<1%). Also, the annual “no result” rate (Indicator 6) was 0.6%, varying from 0.0–2.9% across the months. The ‘no result’ rate (Indicator 7) was higher than the target (<1%) in the month of July 2018 (2.9%). The percentage of test results that were reported within TAT (≤24 hours) of the assay (Indicator 7) was 64.1%, varying from 32.2–100% across the months. However, 98.1% of the tests were reported within 48 hours of TAT, varying from 95.2–100% depending on the months (Table 5).

Factors associated with Xpert MTB/RIF error results

In bivariate analysis, site of the specimen, instrument module, tester, and tester experience were associated with a high error rate. Respiratory specimens were 1.9 times more likely to have an error result than non-respiratory specimens (p = 0.001). Samples tested on instrument module A4, B2, B3, C3, and D3 had a statistically significant association with error result (p<0.05). When results were stratified by tester experience (<2yrs., 2 – 3yrs., and >3yrs.), samples tested by personnel with 2–3 years of experience were 2.3 times more likely to have an error test result than those with >3 years of experience (p = 0.002). In multivariate analysis, independent risk factors for an error result included instrument module A4 (AOR 64.7; 95%CI: 4.5–435.2, p = 0.002), B2(AOR 42.8; 95%CI: 3.4–447.9, p = 0.004), B3 (AOR 13.7; 95%CI: 1.7–407.5, p = 0.013), C3 (AOR 13.0; 95%CI: 1.7–48.3, p = 0.013), and D3 (AOR 14.3; 95%CI: 1.9–407.5, p = 0.010), and tester experience; <2yrs.(AOR 2.1; 95%CI: 1.1–3.7, p = 0.019) and 2-3yrs.(AOR 2.4; 95%CI: 1.3–4.4, p = 0.003) (Table 7).

Discussion

The study presented the use of monitoring quality indicators of Xpert MTB/RIF in initial tests and unsuccessful result retests. Moreover, it demonstrated a method of investigating potential causes of indicators being out of the acceptable limits or targets. All quality indicators were within their targets, with the exception of error rate (Indicator 4) and TAT (Indicator 7). Error rate and TAT were away from the targets; <3% of error rate and 90% of test results report within 24 hours, throughout the year, excluding the months October and December. In these two months, the test statistic was relatively lower than the others.

The success rate of the initial test was 90.4%. The overall MTB positivity rate (Indicator 1) was 18.2%. However, an additional 42 MTB cases were detected following the various level of retesting due to unsuccessful test and retest results. Considering the additional cases, 456 MTB cases detected from the 2495 presumptive TB/DR-TB patients. This figure (Indicator 1, i.e 18.3%) was within the range of MTB positivity rate (13.42–24.61%) reported by various studies in different areas of Ethiopia [13–18]. The observed variation in MTB positivity rate among reports might be linked with the difference in HIV acquisition, health-seeking behavior, geographic location, and TB control effort in the study settings. Additionally, the knowledge status of health care workers (HCWs) towards the diagnostic tool and the clinical practices could affect the positivity rate of the test [22, 23]. The MTB positivity rate (18.3%) recorded by the present study was better than those reported previously by community-based TB prevalence studies [24–26].

The overall initial RIF resistance rate (Indicator 2) was 4.8%. Two RIF resistant cases were detected in 42 MTB cases, which were detected by retesting of unsuccessful test and retest results (test failures). There was a slight variation in RIF resistance rate (4.8%) following retesting of test failures although the difference was not statistically significant. This observation (4.8%) was similar with the WHO estimate for Ethiopia; 5.2% (95% CI: 2.8–8.4) [27] and those reported by Geleta et al [13] and Gelalcha et al [18]. However, several other studies in Ethiopia had reported higher RIF resistance rate [14–17]. The inconsistency could be due to the difference in a group of patients subjected to Xpert testing and the enrollment of a large number of previously treated TB patients [28]. The current RIF indeterminate rate (Indicator 3) (1.0%) was lower than those reported earlier in Ethiopia [13, 17]. This shows the bacilli load in most clinical specimens was sufficient in yielding adequate DNA for determining RIF resistance.

In agreement with the result of the study, Creswell et al [29] reported a 10.6% unsuccessful rate. Furthermore, even higher unsuccessful rate reported by Gidado et al [30] and Agizew et al [31]. Xpert data source could be a reason for a higher rate than our study. The studies conducted by Gidado et al [30] and Agizew et al [31] used GxAlert and GeneXpert software(.gxx file format) as the Xpert data source, respectively. These data sources do not differentiate the retest results of test failures rather they consider them as the initial test of a different sample. For example, in our report, combining unsuccessful results of the initial test and retest all together increases the overall unsuccessful results rate to 10.1%, which is similar with the rate reported by Gidado et al [30]. We propose quality indicators for a retest to be analyzed separately so that the actual figure of quality indicators for the initial test can be determined. Also, the cost implication and delay in providing valid test results because of test failures should be assessed. Because of unsuccessful results, we lost 2,604.78 USD per annum by considering the direct reagent cost of the manufacturer (9.98 USD per Xpert MTB/RIF cartridge, which is a negotiated pricing). The reagent cost required for 2515 samples test is 25,099.7 USD ($9.98 per a test); however, the test failures increased the required cost to 27,704.8 USD ($11.02 per a test) i.e. 1.04 USD an increment per a test due to test failures. Thus high unsuccessful rate has an impact on the cost of a test and needs to be maintained within a limit. In addition to data sources, factors such as defective modules, staff experience, and cartridge version (G3 vs. G4) could affect the occurrence of unsuccessful results [31, 32]. On the other hand, relatively lower unsuccessful rate reported by Ardizzoni et al [32] and Mustapha et al [33]. However, laboratory register used as a sole data source for Xpert data and may not capture the initial test outcomes in case of test failures. This could lead to underreporting of unsuccessful results, or else regular supervision may be provided for Xpert facilities under the project.

In our report, the unsuccessful test results were mainly due to error results (93.4%). The annual error rate (Indicator 4) was 8.9%, which is higher than the target (<3%). The high error rate was not identified and resolved timely as the quality indicators have been analyzed using the data only from the LIS, which captures only the final or reported Xpert results. The tester may have done retest from leftover processed samples in case of unsuccessful results until a positive or negative result is achieved, but only the final result was documented on laboratory register and reported via LIS. This caused underreporting of the error rate in routine monitoring of indicators and falsely led to unnecessary confidence about the assay quality. On the basis of this observation, it can be suggested that the data from the GeneXpert instrument software (.gxx file format) along with the LIS or laboratory register could be utilized for analyzing the indicators for the purpose of discovering unreported unsuccessful results.

The most prevalent error was 5007, which is mainly related to the technical issues, i.e., human errors due to non-adherence to manufacturer-recommended procedure during sample processing such as filling reaction tubes with viscous sputum or incorrect sample volumes, and reagent storage condition [20, 30]. This requires improvement on the technical capability of the testers and the storage condition of cartridges. Similarly, a high percentage of 5006, 5007 and 5008 errors were observed from different resource-limited settings [30, 31]. Previous studies revealed that the G3 of the cartridge is associated with high occurrence of unsuccessful results mainly by the signal loss detection error due to loss of tube pressure (Error 5011) [29, 34]; however, Cepheid improved the cartridge deficiency (G4) to reduce errors mainly caused by signal loss error (Error 5011) and G4 version widely available in March 2013. As we used G4 version of the cartridge for the entire tests, the proportion of Error 5011 was low (3.0%) and it looks that the improvement (G3→G4) has limited the incidence of Error 5011 as previously reported [31, 32, 34].

In contrast to published studies [30][31], in our study, the invalid results occurred at a rate of 0.04% (Indicator 5). This shows that there was no specimen associated inhibition of real-time PCR [19, 35]. It also further indicates that the blood cells in specimens tested were not at the level of interfering PCR amplification. Hemoglobin and lactoferrin were reported as PCR-inhibitor in previous studies [36, 37].

Xpert ‘no result’ is commonly associated with the interruption of power supply or lack of the basics of computer use [20]. In this study, ‘no result’ rate (Indicator 6) was 0.6%, which is below the target (<1%). This shows that the Xpert facility has been continuously provided with stable power supply; the power supply backup in case of interruption functions well. In contrary to our finding, Gidado et al [30] reported a relatively higher (2.2%) rate of ‘no result’. The difference was probably due to the level of the diagnostic centers in TB laboratories network. In the present case, the laboratory being central or national probably benefited from having a lower incidence of “no results”. However, power interruption remains a challenge at the lower level of the diagnostic centers in resource-limited countries like Ethiopia.

Sixty-four percent of Xpert test results released within the TAT (≤24 hours), but the laboratory targeted 90% of test results within ≤24 hours TAT (Indicator 7). Therefore, the laboratory failed to meet its target. When the TAT extended to ≤48 hours, 98% was attained. Recently, Shiferaw and Yismaw reported 46.2% Xpert tests within targeted TAT in Ethiopia [38]; however, they used shorter TAT (2 hours).

In conclusion, 90.4% of the initial tests were successful. The unsuccessful results rate was high (9.6%); error result was the main contributor. However, the follow-up tests usually resolved the errors and an additional 42 MTB cases detected through retest of failures. Probe check failure was the most frequent error and related to technical and cartridge issues. Instrument modules and tester experience associated with a high error rate. The test results released within TAT was below the target. Hence the present study showed that error rate (Indicator 4) and TAT (Indicator 7) were the two quality indicators that require improvement and continuous assessment. In addition, we illustrated that LIS database or laboratory register along with GeneXpert instrument database (.gxx file format) as the right data source for analyzing the quality indicators in order to avoid underreporting of unsuccessful results. The indicators should be monitored on a monthly basis to identify areas that could compromise quality, investigate possible causes and institute corrective actions in a timely manner. We further proposed the indicators for retesting to be analyzed separately so that the indicators of the initial tests can be determined appropriately. Therefore, the findings of the study can give a good insight into monitoring quality indicators of the assay for other Xpert MTB/RIF laboratories in TB laboratory network of the country.