VERIFICATION OF CLINICAL ACCURACY OF AUTOMATED NON-INVASIVE SPHYGMOMANOMETERS: IS IT APPROPRIATE TO USE BLOOD PRESSURE SIMULATORS?

Overview

Cardiovascular diseases are the most common cause of death in developed countries. Blood measurement is an integral part of the diagnosis of these diseases. With the development of oscillometric blood pressure monitors, the question of regular monitoring of their clinical accuracy (overall error) has arisen. This paper deals with the overall accuracy of two commercial tonometers (Hartmann Digital HG 160 comfort and HuBDIC HBP–1520), using two calibrated blood pressure simulators (Fluke BP Pump 2 and Fluke ProSim). Using the Wilcoxon rank-sum test, significant differences between the simulators have been proved for all measurements—both for SBP and DBP measurements and both for Hartmann Digital HG 160 and HuBDIC HBP–1520 tonometers (p < 0.001). Therefore, without the precise knowledge of the relationship between the blood pressure monitor and the simulator used, it is not appropriate to use simulators to determine the overall error. On the other hand, the tested devices had a very good repeatability of the measurements at all presets, with both simulators. From this point of view, it is suitable to use simulators to determine the stability of measurement by a given tonometer rather than its clinical accuracy.

Keywords:

blood pressure – automated non-invasive sphygnomanometers accuracy – blood pressure simulators

Introduction

Cardiovascular diseases are well known as one of the leading causes of death in developed countries. The risk of coronary, cerebral or peripheral artery diseases closely relates to creeping changes in the cardiovascular system such as atherosclerosis or hypertension [1]. At the same time, the prevalence of arterial hypertension is about 44% in European countries and about 28% in the United States [2]. Together with the ageing of the popu-lation in developed countries, which is well-documented [3], the hypertension is a significant danger and should be monitored carefully. Furthermore, many of the hyper-tension patients are not aware of their disease [4], and the numbers could be higher in fact. In all cases, the therapy of hypertension requires an accurate and reliable measurement of blood pressure.

In 1987, a protocol for evaluating the clinical accuracy of devices for the blood pressure measurement was pub-lished by the American Association for the Advance-ment of Medical Instrumentation (AAMI) [5]. Conse-quently, a protocol for validating the blood pressure monitors was also published by the British Hypertension Society (BHS) in 1990 [6]. The revised version of the BHS protocol was published in 1993 [7]. In 2010, the European Society of Hypertension (ESH) published International Protocol revision 2010 for the validation of blood pressure measuring devices in adults which combines and simplifies the AAMI and BHS require-ments [8]. Finally, the AAMI, ESH and International Organization for Standardization (ISO) declared a con-sensus on preparing of Universal Standard for the Vali-dation of Blood Pressure Measuring Devices in 2018 [9]. A blood pressure monitor could be recommended for clinical use if the AAMI criteria both for systolic (SBP) and diastolic (DBP) pressures are met (the differ-ence between the device and the mercury standard is lower than 5 mmHg) and if the device received a grade of A or B based on the BHS protocol. These protocols reacted to the rapid development of so-called automated non-invasive blood pressure monitors (within the mean-ing of this article, blood pressure monitors and sphyg-momanometers are understood to be the same) and to the need of determining the clinical accuracy of these devices that the most commonly use the oscillometric method [10].

The basic safety requirements and the necessary func-tionality of the automated non-invasive blood pressure monitors are described in EN 80601–2–30 + A1 [11]. In this standard, there is a requirement for a maximum cuff pressure error that cannot be greater than ±3 mmHg or 2% of the readings (greater of these values) over the entire measuring range (for temperature between 10 °C and 40 °C and relative humidity between 15% to 85% without condensation) (see paragraph 201.12.1.102). The overall clinical accuracy must then be in accordance with EN 81060–2 (see 201.106) [12].

In addition to production itself, regular checks of these monitors are also important. According to EN 80601–2–30 + A1 [11], regular verification of the accuracy of automated non-invasive sphygmomanometers is always recommended every two years, or after maintenance or repair (see paragraph 201.7.9.2.13). This verification is carried out in a manometric mode (see paragraph 201.12.1.107), which allows the static pressure to be measured across the whole range of the device. However, there is no requirement in this standard to verify clinical accuracy, i.e. especially the accuracy of the DBP and SBP measurements.

There are different approaches to regular check and verification of automated electronic non-invasive sphygmomanometers. In many countries, these devices are ranked in terms of their importance for health protec-tion in the category of “legally controlled measuring instruments” (e.g. the Czech Republic and Slovakia), for which it is necessary by law to perform regular verification of their metrological characteristics (usually every two years). The overall verification process consists of a series of steps and is based on harmonized technical standards (EN 81060–1 [13], EN 1060–3 + A2 [14] etc.) or, where appropriate, manufacturer's infor-mation. However, the scope of the verification itself is different in some countries. Whereas, for example, in the Czech Republic, regular verification of measuring accu-racy is limited to the accuracy test of the cuff pressure indication and the maximum error of the indication of the cuff pressure is checked (must be within ±3 mmHg), in Slovakia, besides the cuff pressure indication, the testing of the total error of the monitor (clinical accuracy) is performed by simultaneous comparative measurement with a reference auscultation sphygmoma-nometer in selected individuals (on the same arm ac-cording to N1 test method EN 1060–4 [15]; this standard has already been replaced by EN 81060–2 [12]) and/or by comparative measurement with a calibrated blood pressure simulator.

The goal of the study was to assess the suitability of using blood pressure simulators to determine the total error (clinical accuracy) of automated noninvasive sphygmomanometers.

Material and Methods

Evaluated Devices

The devices available in the global market with the different type of oscillometric method for blood pressure measurement have been selected:

1. Hartmann Digital HG 160 comfort—an automatic device intended for clinical and self-measurement in home conditions. The device uses the oscillometric method during the deflation of the cuff for measurement of blood pressure in the range of 30 to 280 mmHg with a declared accuracy of ±3 mmHg [16].
2. HuBDIC HBP–1520—an automatic device for measurement of NIBP during inflation of the cuff. The manufacturer declares that the device measures in the range of 40 to 230 mmHg with the accuracy of ±3 mmHg [17].

Performed Tests

These devices were tested according to the Czech metrological regulation MP 017 [18], which determines the testing procedure for blood pressure measuring devices during their verification and corresponds to the standard EN 81060–1 [13] and EN 1060–3 + A2 [14]. Specifically, the tests were carried out according to Chapter 6.3 concerning the process of metrological verification of electronic automated blood pressure devices. The following checks and functional tests were carried out:

1. Zero setting control—at zero overpressure in the pneumatic system of the device, the display must show zero after the device has been switched on.
2. Rapid exhaust test—in the pneumatic system of the device, a positive pressure close to the upper limit of the measuring range shall be generated and the quick-release valve shall be actuated. The time taken for the pressure reduction from 260 mmHg to 15 mmHg is measured and shall not exceed 10 s.
3. Air leakage test—in the pneumatic system a positive pressure close to the upper limit of the mea-suring range is generated. After the pressure has stabilized for five minutes, the value of the pressure drop per minute is measured using a stopwatch. The decrease shall not exceed 6 mmHg/min. The test is performed with a cuff wrapped around a cylinder of appropriate size.
4. Cuff pressure indication accuracy test—direct comparison of pressure standard reading with tonometer reading. Measurements are made in 50 mmHg incre-ments between 0 mmHg and the upper end of the scale range, at both rising and falling pressure (at the same pressure points). The maximum permissible error of cuff pressure indication over the entire range is 3 mmHg.
5. In the framework of this article, the total error (i.e. overall system accuracy according EN 1060–3 [19],
   
   clinical accuracy) of the devices under investigation was determined according to paragraph 4.2.2.3 of the Slovak Decree No. 210/2000 Coll [20]. The total error is deter-mined according to this Decree:

• simultaneous repetitive measurement per-formed by the test and reference auscultation meter on the upper arm of the same hand of the selected physi-cal persons and at the same time,
• a test using a calibrated blood pressure simu-lator.

In the method referred to in point (a) the persons must be selected so that at least one systolic blood pressure value and one diastolic blood pressure value is in the high pressure range (>160/>100) mmHg, one in the normal pressure range (90–160)/(80–100) mmHg and one value in the low pressure range (<100/<80) mmHg.

In the method referred to in point (b) the minimum number of test points is five, with at least one test point in each band (high, normal and low pressure).

According to point 3.3 of the Decree [20], the arith-metic mean total error of measurements for DBP and SBP must be within ±5 mmHg and sample standard deviation of measurement must be within ±8 mmHg which complies with the requirements of EN 1060–3 (part 7.9) [19]. If a blood pressure simulator is used, the standard deviation must be within ±3 mmHg. The results are evaluated according to EN 1060–4 (part 4.9) [15].

First of all, blood pressure differences from all paired measurements are calculated according to the following formulas:

where pSBPt is systolic blood pressure value indicated by the tonometer under test, pSBPs is systolic blood pressure value set on the blood pressure simulator, pDBPt is diastolic blood pressure value indicated by the tonometer under test, pDBPs is diastolic blood pressure value set on the blood pressure simulator.

The arithmetic mean error and sample standard deviation for SBP and DBP are then calculated as

where p_SBPt is systolic blood pressure value indicated by the tonometer under test, p_SBPs is systolic blood pressure value set on the blood pressure simulator, p_DBPt is dias-tolic blood pressure value indicated by the tonometer under test, p_DBPs is diastolic blood pressure value set on the blood pressure simulator.

In the case of our tested devices, we did not perform the test on individuals according to point (a), because it was not possible to measure the auscultation method (HuBDIC HBP–1520 measures during cuff inflation, Hartmann Digital HG 160 has too rapid deflation). That is why we have focused to the test using a calibrated blood pressure simulator.The use of only one of these methods is allowed if the properties of the test instrument do not allow the tests to be performed by both methods. In that case, this fact shall be indicated in the test records. If both methods are used, the meter must pass the test according to both methods in order to be verified.

Calibrated Blood Pressure Simulators

We tested the total error using two calibrated blood pressure simulators, both from FLUKE:

1. FLUKE BP Pump 2—the device is a blood pressure simulator intended for testing of adult and neonatal blood pressure monitors. The simulator allows to simulate the systolic blood pressure (SBP) in range of 20 to 250 mmHg, diastolic blood pressure (DBP) in range of 10 to 200 mmHg, heart rate in the range of 30 to 250 bpm (beats per minute) and pulse volume in the range of 0.1 to 2.4 ml. The volume of the inner adult cuff is 310 ml and 20 ml of the neonatal cuff. The simulator provides 22 different presets for simulating blood pressure such as a healthy heathy patient, geriatric patient, obese patient, a patient with tachycardia, brady-cardia or atrial fibrillation etc. The preset for healthy patient was used for testing of blood pressure measure-ment repeatability [21].
2. FLUKE ProSim 8—the device is a vital signs simulator which can also simulate the oscillometric pulsations for testing of NIBP measurement devices. The device can generate the pressure in the range of 20 to 400 mmHg; the maximal pulse volume is 1.25 ml. Similarly, as BP Pump 2 the simulator provides different presets. Identically as for FLUKE BP Pump 2, the preset for healthy patient was used for testing of blood pressure measurement repeatability [22].

Both simulators had metrological traceability to pressure standards of the Czech Metrology Institute. The static accuracy of both simulators compared to the PMI 40K (Cressto, CZ) reference pressure gauge with accuracy ±0.3 mmHg, was better than ±1 mmHg in the range of measurement 0 to 300 mmHg. The maximum expanded standard uncertainty of calibration was ±0.13 mmHg.

Statistical Analysis

The total errors obtained by both simulators has been statistically analysed to prove the difference between the results acquired by FLUKE BP Pump 2 and FLUKE ProSim 8.

First, the normality of data has been tested by Lilliefors test (the hypothesis H₀: the data comes from a distribution in the normal family) [23].

Consequently, Wilcoxon rank-sum test (Mann-Whitney U–test) has been used to test the null hypoth-esis H₀ that data obtained using FLUKE BP Pump 2 and FLUKE ProSim 8 are samples from continuous distribu-tions with equal medians, against the alternative that they are not [24].

Results

Total Errors

All measurements were carried out under laboratory conditions meeting the requirements of MP 017 [18].

Both selected devices met requirements a) to d) according to section 2.2. The air leakage was repeatedly tested in the whole range. All values for both investi-gated devices were lower than 6 mmHg/min, which is the required value by the regulation [18]. The cuff pressure indication accuracy for both devices was better than ±3 mmHg in the whole range from 50 mmHg to 300 mmHg with steps of 50 mmHg, both during inflation and deflation of the cuff.

The total error was evaluated according to the regu-lation [20]. Both evaluated devices were tested both with FLUKE BP Pump 2 [21] and FLUKE Prosim 8 [22] simulators. Measurement for each preset was repeated five times, and consequently, the arithmetic mean error of the measurements and the sample standard deviations for seven different presets and for both devices evalu-ated on each simulator were calculated, see Table 1 to Table 4.

Finally, the total mean arithmetic error values of the meters, including sample standard deviations, were calculated, see Table 5 and Table 6.

Results of statistical analysis

The complete set of the data has been used for the statistical analysis. Both for SBP and DBP, and both for FLUKE BP Pump 2 [21] and FLUKE Prosim 8 [22], all five measurements for all seven presets has been collect-ed. It means that each data collection was a set of 35 values.

Using the Lilliefors test, the null hypotheses H₀: the data comes from a normal distribution, have been rejected for all datasets (p < 0.05). Consequently, using the Wilcoxon rank-sum test, the significant differences between the simulators have been proved for all mea-surements – both for SBP and DBP measurements and both for Hartmann Digital HG 160 and HuBDIC HBP–1520 tonometers (p < 0.001).

Discussion

The results show that both tested blood pressure monitors met the requirements of the regulation MP 017 [18] and can be used for providing health care in the Czech Republic. However, the results of the total error test (according to Slovak Decree No. 210/2000 Coll. [20]) of devices using calibrated blood pressure simu-lators are ambiguous. The mean differences obtained using the FLUKE BP Pump 2 are typically lower than the corresponding values obtained using FLUKE Prosim 8. Only HuBDIC HBP–1520 met the criteria, and only when tested with the Fluke BP Pump 2 blood pressure simulator. In all other cases, one or both required criteria were not met. The result of the metrological control is thus greatly influenced by the choice of a blood pressure simulator. The differences between the two simulators are statistically significant. This observation is in line with the conjecture on about the practical applicability of blood pressure simulators to determine clinical accuracy. Even in the EN 1060–3 [19] is note under point 3.6 Patient simulator: „The devices are not used for testing accuracy but is required in assessing stability of performance.” This is also reported by some manu-facturers of the patient simulators (Fluke) [25]: „The NIBP simulator is used to evaluate the performance of NIBP monitors. NIBP accuracy is determined by static pressures, whether generated by the NIBP simulator or other pressure source that is traceable to national metrology standards. A well-designed simulator should mimic the dynamic nature of the NIBP patient and create a stable response of the living subject to the cuff during the measurement cycle. Therefore, it is possible to use a simulator to determine the repeatability of these moni-tors.” Some manufacturers of blood pressure monitors are also in favor of this conclusion [26]: „NIBP modules from different manufacturers will usually give different results on the same simulator setting. This is normal and expected primarily due to a number of algorithm differences between manufacturers. THE OEM NIBP MODULE THAT BEST MATCHES UP WITH TARGET VALUES IS NOT NECESSARILY THE BETTER MODULE. Remember, NIBP simulators can-not be used to measure clinical accuracy.”

At all presets, the tested devices had a very good repeatability of the measurements, as evidenced by the overall repeatability (the average of the standard deviations for each preset). However, the total sample
standard deviation in the evaluation of the procedure is not taken as the average of those individual series of measurements at each preset, but as the standard of all measurements across all presets, so the resulting stan-dard deviation of the whole set is significantly higher. Such a method of evaluating sample standard deviation is suitable when conducting clinical trials according to validation protocol (AAMI [5], BHS [6, 7], ESH [8], EN 1060–4 [15] etc.) on a patient population. The authors consider the use of such evaluation procedure with blood pressure simulators as slightly problematic. Cur-rent simulators are not yet able to replace real patients, they still have technical limits and it is not advisable to use them to test the overall accuracy with oscillometric devices. Some problems of using simulators for valida-tion of blood pressure monitors are also discussed in [27]. In spite of it, a development of simulators which might be used for blood pressure monitors testing is still in progress [28].

In addition, the blood pressure monitors that allow simultaneous measurement by an auscultation method on the same arm are at a disadvantage, as testing under (a) Decree No. 210/2000 Coll. [20] is, in our opinion, very demanding. It is necessary to have the consent of the persons for the measurement, and in addition, it is necessary to have persons with different blood pressure values, which can make it very difficult to perform metrological verification itself in clinical practice. The cost of such metrological verification is certainly also questionable. The price of a calibrated blood pressure simulator is not negligible, and the overall range of tests, where several repetitive measurements are performed, is time consuming. Therefore, the price of metrological verification is almost certainly at least comparable to the purchase price of commonly available electronic blood pressure monitors used in healthcare facilities. This fact could lead the healthcare providers rather to a regular replacement of instrumentation than the implementation of metrological verification of older devices (environ-mentally high load), or in the worst case, the absence of regular controls (hazard for humans).

Non-invasive automatic sphygmomanometers cur-rently marketed must now undergo a clinical test accord-ing to one of the procedures described in EN 81060–2 [12]. For this reason, the authors believe that regular checks of the static accuracy of the cuff indication as recommended in the EN 80601–2–30 + A1 [11] stan-dard can be considered as sufficient verification of the accuracy of the blood pressure monitors.

In the course of work on this article, new legislation concerning metrological inspections of measuring instruments came into force in Slovakia. It is the Decree No. 161/2019 Coll. [29] Annex No. 37 which deals with metrological checks of non-invasive blood pressure devices. This Annex no longer describes the exact technical procedure for checking these devices, but paragraph 6.2.4 states the obligation to test the overall accuracy of the system using a blood pressure simulator.

Conclusion

In this paper, the suitability of using a blood pressure simulator to determine the overall error (clinical accuracy) of two automated non-invasive oscillometric blood pressure monitors was examined. The overall error was evaluated using two calibrated blood pressure simulators for two commercial blood pressure monitors. The differences between the two simulators were statistically significant – both for SBP and DBP mea-surements and both for Hartmann Digital HG 160 and HuBDIC HBP–1520 tonometers (p < 0.001). Without precise knowledge of the relationship between the blood pressure monitor and the simulator used, it is not appropriate to use simulators to determine the overall error. On the other hand, the tested devices had a very good repeatability of the measurements at all presets, with both simulators. From this point of view, it is better to use simulators to determine the stability of measure-ment by a given tonometer rather than its clinical accu-racy.

Acknowledgement

This work has been supported by the grant no. SGS20/167/OHK3/3T/13 of the Czech Technical Uni-versity in Prague

Ing. Jan Havlík, Ph.D.

Department of Circuit Theory

Faculty of Electrical Engineering

Czech Technical University in Prague

Technická 2, CZ-166 27 Prague 6

E-mail: xhavlikj@fel.cvut.cz

Phone: +420 224 352 048