Clinical Biochemistry 93 (2021) 90–98
Available online 5 April 2021
0009-9120/© 2021 The Canadian Society of Clinical Chemists. Published by Elsevier Inc. All rights reserved.
A model to establish autoverification in the clinical laboratory
Deniz Ilhan Topcu
a,*, Ozlem Gulbahar
baDepartment of Biochemistry, Faculty of Medicine, Bas¸kent University, Ankara, Turkey
bDepartment of Medical Biochemistry, Faculty of Medicine, Gazi University, Ankara, Turkey
A R T I C L E I N F O Keywords:
Autoverification Total testing process Information technologies
A B S T R A C T
Objectives: Autoverification is the process of evaluating and validating laboratory results using predefined computer-based algorithms without human interaction. By using autoverification, all reports are validated ac- cording to the standard evaluation criteria with predefined rules, and the number of reports per laboratory specialist is reduced. However, creating and validating these rules are the most demanding steps for setting up an autoverification system. In this study, we aimed to develop a model for helping users establish autoverification rules and evaluate their validity and performance.
Design & methods: The proposed model was established by analyzing white papers, previous study results, and national/international guidelines. An autoverification software (myODS) was developed to create rules ac- cording to the model and to evaluate the rules and autoverification rates. The simulation results that were produced by the software were used to demonstrate that the determined framework works as expected. Both autoverification rates and step-based evaluations were performed using actual patient results. Two algorithms defined according to delta check usage (Algorithm A and B) and three review limits were used for the evaluation.
Results: Six hundred seventeen rules were created according to the proposed model. 1,976 simulation results were created for validation. Our results showed that manual review limits are the most critical step in deter- mining the autoverification rate, and delta check evaluation is especially important for evaluating inpatients.
Algorithm B, which includes consecutive delta check evaluation, had higher AV rates.
Conclusions: Systemic rule formation is a critical factor for successful AV. Our proposed model can help labo- ratories establish and evaluate autoverification systems. Rules created according to this model could be used as a starting point for different test groups.
1. Introduction
The clinical laboratory influences clinical decisions and 60%–70%
decisions on hospitalization and discharge of patients; moreover, the use of various treatments is partially based on considerations of laboratory test results [1,2].
One of the most critical steps of clinical laboratories is the test report verification process at the post-analytical phase [3]. Generally, the evaluation is human-centric, employing mental algorithms performed by an expert on one or more analysis results. The purpose of this step is to detect potential errors before test results are released by the labora- tory. Thus, information collected from pre-analytical, analytical, and post-analytical phases of the total testing process (TTP) is manually
assessed [4]. The entire procedure is time-consuming, subjective, and depends on the experience and education of laboratory staff [5].
The workload of clinical laboratories is increasing because of the expansion of test panels, increased number of samples analyzed, ex- pectations of high-quality, and shorter target turnaround times. How- ever, recent advances in information and automation technologies allow clinical laboratories to respond to this escalating workload. In all test processes, developing technology helps clinical laboratories work more effectively [6,7]. An example of this involves the use of autoverification (AV), a tool for report verification in the post-analytic phase of TTP.
AV performs test result verification through algorithms, similar to approaches used by laboratory personnel during manual verification.
These algorithms are based on the evaluation of test results obtained in
Abbreviations: TEa, allowable total error; AV, autoverification; CV, coefficient of variation; RI, reference interval; SA, statistical analysis; HIS, Hospital Information System; LIS, laboratory information system; QC, quality control; TTP, total testing process; VALAB, validation assisted in the laboratory; VDL, verification decision limit.
* Corresponding author.
E-mail address: ditopcu@gmail.com (D.I. Topcu).
Contents lists available at ScienceDirect
Clinical Biochemistry
journal homepage: www.elsevier.com/locate/clinbiochem
https://doi.org/10.1016/j.clinbiochem.2021.03.018
Received 3 January 2021; Received in revised form 23 March 2021; Accepted 29 March 2021
the analytical phase, together with information obtained in pre- and post-analytic phases [4,8]. AV algorithms are usually designed with data filters based on instrument error and quality control (QC) flags; serum indices for hemolysis, icterus, and lipemia; critical values and result range limits (reference ranges or other limits); delta checks; and con- sistency checks, which are used to determine abnormal combinations of test results [9].
AV has been used to various degrees by clinical laboratories for nearly 20 years [10]. The first reported literature reference to AV soft- ware was published in 1992, and Validation Assisted in the Laboratory (VALAB) was the first system developed in this context [11]. By 2007, the American Association for Clinical Chemistry reported that approxi- mately 52% of clinical laboratories in the United States used AV. In 2013, a study conducted in Spain, reported that 64% of surveyed labo- ratories used AV and that the frequency of AV usage increased as the laboratory size increased [10]. Today, laboratories use AV systems for various tests, including routine biochemistry, immunoassay, urinalysis, hematology, coagulation, immunology, and blood gas tests [8,12–20].
AV is developed using rule-based systems. In addition to rule-based approaches utilizing expert systems, other tools employing artificial intelligence, such as Bayesian networks, artificial neural networks, or various statistical approaches, have been proposed [21–23]. In the rule- based approach, the test result is predominantly evaluated using the “if- then” type logic, according to different criteria, and this constitutes a rule. After evaluation, results that fall outside of defined acceptability criteria are set aside for manual evaluation by laboratory staff [4,24]. In addition to the approval of results, these systems may also perform operations, such as reflex test requests, re-testing, and automated addition of comments [20]. Expert systems are computer programs that provide capabilities similar to human experts and are suitable candi- dates for developing decision-based software. These systems use both facts and rules, which are similar to “if-then” statements, to solve complex decision-making problems [6,25].
AV can enhance turnaround time, laboratory error detection rate, operational performance, and clinician satisfaction with services [9].
With AV, laboratory staff is required to manually review only a small fraction of specimens and results that might be problematic, and the use of computers can detect rare events that can evade manual verification [12,19]. Thus, the specialist can devote more time examining prob- lematic results that require special attention [12]. Furthermore, the use of computers enables use of complex algorithms, thereby utilizing more comprehensive criteria for result evaluation [4].
To achieve optimized advantage, criteria used for evaluating test results in AV must include all relevant information retrievable for pre- analytic, analytic, and post-analytic phases [4]. Obtaining this infor- mation can be facilitated by middleware that connects the Hospital In- formation System or Laboratory Information System (LIS) with analytical instruments. Mandatory and optional requirements of AV systems have been described briefly in the CLSI AUTO-10A and AV guidance document of the Department of Examination and Diagnostic Services [4,26].
Despite its advantages, AV has challenges and potential disadvan- tages. Only a few studies have focused on the use of AV in the clinical laboratory [12]. AV does not have a standard set of criteria, but it should be integrated to minimize the risk to patients, and where applicable, conform to international guidelines to ensure process quality [9].
Creating AV rules is a challenge, and it brings with it the burden of rule maintenance and testing in an informatics support system, centered on the laboratory. As useful as laboratory-generated middleware rules are, they are inclined to be unique to each laboratory; hence, no broadly scripted common sets of middleware rules exist [24]. Successfully using AV in the long term comes with a heavy burden of maintaining duly trained and qualified personnel to use these systems. Moreover, vali- dating the AV process is time-consuming, as each component of the system needs to be considered [19].
In this study, we developed a standard model to help clinical
laboratories in rule development, which can serve as a starting point for the AV setup of quantitative tests. We aimed to develop a model that allows for the development of flexible rules and algorithm definitions, conforming to different test groups, instruments, and clinical settings.
Then, the developed model was evaluated using simulated and actual patient results for accuracy and AV rate. We have developed a software that uses expert system architecture for helping generate, test, and validate AV rules structured in line with the proposed model.
2. Materials and methods 2.1. Model development
Evaluation criteria that can be used during report verification were determined to develop a model to cover pre-analytical, analytical, and post-analytical phases in the TTP. Available personal communications (e.g. B. Marquardt 2015,), literature references, and guidelines [5,9,19,26–28] were reviewed to develop a comprehensive list of eval- uation criteria to assemble as part of the model for report verification, covering all three main phases of the TTP. Table 1 outlines specific criteria and the applicable testing phase.
If the test result and other related information meet all evaluation criteria in the final step of the algorithm, the result was classified as
“Verified” and marked in green to enable users to evaluate results. In cases where the test result or other related information did not meet the evaluation criteria, the result was classified as “To Be Reviewed.” “To Be Reviewed” results were divided into three subclasses, namely “Warn- ing,” “Attention,” and “Watch,” according to evaluation criteria and released with predetermined colors that represent the severity level (Table 1). For report evaluation, if all tests on a report were evaluated as
“Verified,” the report was classified as “Verified Report.” Otherwise the report is classified as “Not Verified Report.” Fig. 1 represents a scheme of
Table 1
Evaluation criteria and related testing process with a generated rule, color code, and simulated result count. The Color Code shows rule type as Red for Warning, Orange for Attention, and Yellow for Watch. The Color Code for valid results is shown in green. Rule count demonstrates the number of rules related to the evaluation step. Valid and invalid results specify the number of generated results related to each step for evaluating “Verified” and “Not Verified” conditions, respectively.
Step
No Evaluation
Item Related
Test Phase Color Code Rule
Count Simulated Result Count Valid
Results Invalid Results 1 Calibration
Validity Analytical Red 29 29 29
2 Quality Control Validity
Analytical Red 29 29 29
3 Rerun Results Analytical Orange 29 29 29
4 Result Format Analytical Red 29 87 29
5 Device and
Result Flags Analytical Red 29 203 116
6 Sample Type Pre-
analytical Red 29 29 29
7 Serum Indices Pre-
analytical Red 87 87 87
8 Numeric
Results Analytical Red 29 87 29
9 Analytical
Limits Analytical Orange 29 29 58
10 Critical
Values Post-
analytical Orange 23 23 23
11 Verification Decision Limits
Post-
analytical Yellow 230 230 460 12 Delta Check Post-
analytical Yellow 45 135 90
the algorithm for the developed model. Definitions related to each step are given in the supplemental file. The details of each evaluation step are as follows.
2.1.1. Calibration validity
Calibration validity evaluates calibration stability time (h), last successful calibration time, and current result time. If the defined cali- bration period has expired, all results are marked with an error flag.
2.1.2. Quality control validity
QC evaluation criteria are based on the time recorded from the last acceptable QC and approval status. Evaluation of QC validity does not examine which or how specific rules (e.g. Westgard rules) are applied, but whether the QC challenge is passed.
2.1.3. Rerun results
Rerun result evaluation involves evaluating whether the test result represents the result of a rerun of the same test on the sample. Rerun results are checked at the beginning of the algorithm because rerun Fig. 1. Algorithm of the proposed model: Algorithms A and B were created according to the usage of delta check information, and the AV rates of the two algorithms was compared. In algorithm A, if the result is out of the verification decision limit (VDL), then the result is directed to manual verification. In al- gorithm B, VDL and delta check are evaluated simultaneously; if delta check evaluation is valid for the out of the VDL result, then this result is considered
“Verified”. A result flagged with
“Warning” indicates a pre-analytical or analytical problem in which the user must take corrective action. Results flagged with “Attention” and “Watch”
correspond to analytical and post- analytical problems, respectively, and prompts users to act consistent with the clinical context and other laboratory findings.
operations are often performed to address a problem. Therefore, two options are provided in the developed algorithm: (1) all rerun results are directed to the manual review and (2) the result is verified if the rerun result is within limits, established by ± one coefficient of variation (%
CV).
2.1.4. Result format
The result format obtained from the analyzer is compared with test format definition for both numeric and non-numeric results. In brief, depending on the test type, the result may contain either numeric, al- phanumeric, or both characters. For example, a numerical value is expressed in conventional units for the glucose test, such as 105, whereas results such as “Negative” or “Negative 0.05” can be obtained for some serology tests. Moreover, some devices can use “>” and “<” in the field requiring a numeric value for results outside the analytical measurement range.
2.1.5. Device and result flags
The flag information associated with results from the device is evaluated. In the algorithm, flag information triggers a warning. Flags indicate a need for evaluating a single test result in the sample or all test results analyzed in that sample. For example, a short flag can be used for related tests or all tests in that sample.
2.1.6. Sample type
Transmitted sample type from the analyzer and test definition compared to check if they match. More than one sample type can be defined for a test and multiple sample types are acceptable.
2.1.7. Sample Serum indices
Here, serum indices analyzed by the device are evaluated. For evaluation, serum index values obtained from the sample are compared with allowable serum index limits specified by the user for each test. The proposed model allows different serum index definitions according to the device and in different formats, such as concentration, semi- quantitative, and absorbance value.
2.1.8. Numeric results
In this step, numerical test results are rounded to decimal points according to test definition and checked for non-negative values.
2.1.9. Analytical limits
Analytical limit evaluation assesses whether the test result is within the analytical measurement range for the device. Additionally, dilution ratios can be defined, which are provided to the user for results above the analytical limit.
2.1.10. Critical value
Critical value evaluation includes evaluating whether the test result is outside specified critical value limits. The model proposes a flexible definition of critical value based on the patient’s age, gender, clinical information, and specific conditions, such as urea and creatinine for patients undergoing dialysis and CK for patients with neuromuscular disease.
2.1.11. Verification decision limits
Additional review limits can be used in AV systems to determine results that require verification other than analytical and critical value limits. These review limits are called “Verification Decision Limit” (VDL) in this study. Three VDLs were evaluated to compare the AV rates of limits in the proposed model. For each VDL type, AV rates were calcu- lated. Chosen VDLs were as follows:
1. Reference interval (VDL RI): In this approach, RIs are used as VDLs, according to the age and gender of patients.
2. Reference intervals ± allowable total error (VDL RI-TEa): Ranges obtained by including the error allowable values of the test to lower and upper limits of the RI are used as VDLs.
3. Determination of VDL by statistical analysis (VDL-SA): VDL can be calculated according to the 2nd and 98th percentiles of previous test results for a given time interval. Results of the calculation are clas- sified according to patient sex and age, following the criteria, before calculating VDL [24,27]:
•Distinction between outpatient and inpatient.
•Distinction between adults and children in different age groups.
•Distinction of different biological statuses, such as Tanner stage and pregnancy.
•Distinction of clinical departments: Departments that may have extreme results, such as the hemodialysis unit, intensive care unit, transplantation clinic, and emergency department.
In the study, retrospective patient data was used for the determina- tion of the VDL-SA. VDL-SA calculation R script was given as supple- mentary file.
2.1.12. Delta check
Consistency between two consecutive results of a patient is evaluated according to the defined acceptable amount of change and time window between results. In the developed model, four delta calculations are used, namely absolute change, percentage change, absolute change rate, and percentage change rate. Detailed calculations are given in the sup- plemental file.
Delta check change and time window information were defined using the information given in the literature [29,30]. Reference change values with percent change can also be used in the development model.
2.1.13. Consistency checks
Consistency checks can be difficult to apply manually; computers can be used to ensure that checks have been applied, especially across staff with different levels of experience. A proposal for consistency check evaluation is given as supplemental file. However, evaluation of this step is not included in the context of this study.
2.2. Testing the proposed model
A software was designed to generate (1) rules according to the model, (2) simulation results for validating the software and rules, and (3) assess the performance of AV.
The myODS software was developed using Microsoft Visual C#, CLIPS, and R programming languages [6] and consists of three basic modules; User Interface, Expert System, and Data Process and Analyses (Fig. 2).
2.3. Assessment of AV rules and software
Two methods with simulation and actual patient results were used for the assessment of AV rules and software.
2.3.1. Assessment with the simulation results
Simulation results were used to confirm if the software and AV rules worked as expected. These results were created according to guidelines [4,22], which cover all rules and related conditions. The assessment was continued until all unexpected situations were resolved.
2.3.2. Assessment with actual patient results
To assess the proposed model with actual patient data, 29 biochemistry tests, which are frequently used in clinical biochemistry laboratories, were selected. Selected tests and basic definitions of tests are given in Table 2.
Ethics committee approval for data to be used in the study was ob- tained from the Ethics Committee of Gazi University (meeting no. 04 on
November 4, 2017). Two-year records (2015–2016) of the Central Biochemistry Laboratory, Gazi University Health Research and Appli- cation Hospital, including selected test results, were obtained from the LIS (ENLIL-Java, Eroglu Software, Eskisehir, Turkey). Information, including name, identity number, request number, date of birth, and sample number, was anonymized while preserving associations.
Randomly generated numbers and letter-number combinations were produced while exporting data from LIS. All data were transferred to the myODS database using CSV files. Calibration, QC, and flag information cannot be accessed from the LIS. Thus, these steps were evaluated using simulation results.
All test results for order were accepted as patient report for the context of the study. All clinical departments and both outpatient and inpatient results were included to compare the AV rates of the developed model. Evaluation of step-based, test-based, and report-based AV rates were assessed with patient results.
Descriptive statistics of frequency and percentage were used for AV rates. The chi-squared test was used, and an effect size (Cohen’s w) was calculated for the evaluation of different AV rates. Cohen’s w values with 0.1, 0.3, and 0.5 correspond to small, medium, and high effect sizes. All analyses were performed externally using outputs from the software with R 3.5.3 [31].
3. Results
3.1. Rule generation and verification
A total of 617 evaluating rules were created for selected tests using the proposed model according to laboratories’ test information with minimal user interaction (Table 1). A total of 1,976 simulation results were generated with myODS to validate the software and proposed model (Table 1). Detailed examples of rules are given in the Fig. 2. myODS architecture.
Table 2
Selected definitions of 29 selected biochemistry tests.
Test Name Result Type Specimen
Type Result
Unit Decimal
Count
Albumin Numeric Serum g/dL 1
ALP Numeric Serum U/L 0
ALT Numeric Serum U/L 0
Amylase Numeric Serum U/L 0
AST Numeric Serum U/L 0
BUN Numeric Serum mg/dL 1
CK Numeric Serum U/L 0
CK-MB Numeric Serum U/L 0
Iron Numeric Serum μg/dL 1
Direct Bilirubin Numeric Serum mg/dL 2
Phosphorus Numeric Serum mg/dL 2
GGT Numeric Serum U/L 0
Glucose Numeric Serum mg/dL 0
HDL-Cholesterol Numeric Serum mg/dL 0
Calcium Numeric Serum mg/dL 2
Chloride Numeric Serum mmol/L 0
Creatinine Numeric Serum mg/dL 2
LDH Numeric Serum mg/dL 0
LDL-Cholesterol Numeric Serum mg/dL 0
Lipase Numeric Serum U/L 0
Magnesium Numeric Serum mg/dL 2
Potassium Numeric Serum mmol/L 2
Total Protein Numeric Serum g/dL 1
Sodium Numeric Serum mmol/L 0
Total Bilirubin Numeric Serum mg/dL 2
Total
Cholesterol Numeric Serum mg/dL 0
Triglyceride Numeric Serum mg/dL 0
UIBC Numeric Serum µg/dL 0
Uric Acid Numeric Serum mg/dL 1
supplementary file.
3.2. Assessment with actual patient results
Within the scope of the study, 289,207 reports including 3,188,095 test results were evaluated, of which 62.7% and 37.3% were for in- patients and outpatients, respectively. Of these, 85% belonged to adult patients and 15% to children. The 29 selected routine biochemistry tests, comprising approximately 85% of the test requests, were performed in clinical chemistry analyzers.
3.2.1. Evaluation Step-Based AV rates
The total number of tests and number of tests not verified according to the evaluation step are shown in Table 3. VDL evaluation step for all three limits had the highest “To Be Reviewed” test count. (16.18% – 29.44%). The step for the evaluation of the serum indices has the second-lowest AV rate and 2.26% of the results were held for manual evaluation. When the results were examined, a total of 3,184,698 test results were checked, and 2.1%, 0.15%, and 0.007% of results were manually reviewed because of hemolysis, icterus, and lipemia, respec- tively. When results from delta check evaluation were examined, 691,425 (29.39%) of 2,352,974 tests checked at this stage were found to be within the time window determined for the delta check. Of the results evaluated, 671,063 (97.06%) complied with the delta check rule. The number of tests that did not meet the delta check criteria and hold for manual verification was 20,362 (2.94%). We also observed that when the widest manual review limit (VDL-SA) was used in this step, it has a statistically lower verification rate compared to the serum indices evaluation step (Chi–square value = 365,836, p < 0.001, effect size = 0.24).
3.2.2. Test- and Report-based AV rates
Test-based AV rates using three VDLs are demonstrated in Table 4 for both algorithms A and B. Verification rates obtained with algorithm B are higher than those with algorithm A. If VDLs determined by SA were used, a higher AV rate was obtained.
Report-based AV rates for algorithms A and B were evaluated for all patients, inpatients, and outpatients (Table 5). Our results showed that algorithm B has a statistically higher verification rate for all VDLs and patient groups with small to small-medium effect sizes. Furthermore, when the inpatient and outpatient effect sizes were compared, we observed that the inpatient group had a larger effect size than out- patients for all VDL types. This is an interesting finding, and it could be hypothesized that the most significant verification difference between outpatients and inpatients is the VDL evaluation step. Table 5 also shows with expanding VDL limits, both algorithms A and B have higher AV rates across all report types. These results are in agreement with our expectations.
4. Discussion
As with other TTP steps, the verification of laboratory results in the post-analytical phase has benefited significantly from advances in computational technologies. Standardizing the post-analytical phase by making it as objective as pre-analytical and analytical phases is possible [5]. The standardization provided by AV can increase the productivity of laboratories and improve the quality of test results [26]. However, the effectiveness of each system may be different, because it is not estab- lished how the rules of AV and the limits of these rules are determined [8,16]. The most critical and time-consuming processes in developing AV include creating rules and testing that these rules work as expected [12,32]. However, information on how users can set up AVs is limited [12]. An initial objective of the study was to develop a model for facil- itating AV rule development, which would enable users to select eval- uation steps and define these steps in detail. The efficiency of the model was also evaluated.
As mentioned in the literature review, the most common evaluation criteria used for AV include QC, device flags, serum integrity (e.g., serum indices and clot information), reference ranges, clinical infor- mation, related test results, and rerun information [10,27]. Critical value, delta check, verification limits, and moving average evaluations were reported to be less commonly used [10]. In the context of this study, evaluation steps were chosen to cover all phases of TTP, and national and international guidelines were used for determining rules [4,26]. All information related to each step is given in the supplemen- tary document. In the study, 617 rules were generated for 29 tests (Table 1). The proposed model can be used for creating AV rules sys- tematically according to laboratories’ existing test information. These rules can be used as a starting point for laboratories that are planning to use AVs. Moreover, ground rules can be applied to laboratories with existing LIS, middleware, or other commercial AV systems.
Review limits can be as narrow as the reference range or as wide as the analytical measurement limit of the analyzer. Moreover, it is rec- ommended that clinical decisions, technological possibilities, and other AV strategies that are developed, should be included in the process of determining these values as application standards [24]. Our results revealed that even the widest manual review step (VDL–SA) has 13.9%
lower verification rate compared to serum index step which has the second–lowest verification rate (Chi–square value = 365,836, p <
0.001, effect size = 0.24). Additionally, tests were mostly held for the manual verification in the VDL evaluation step (Table3). Therefore, it can thus be suggested that this step determines the AV rate of the pro- posed model.
In contrast to previous studies, we evaluated three VDLs for both test and report-based comparisons using retrospective data. Our findings revealed that VDL-SA review limit, which was calculated using the 2nd and 98th percentiles of retrospective patient data, has higher AV rates.
These limits are consistent with values present in the literature. Our report-based AV rates are lower than rates reported in various Table 3
Assessment of evaluation steps for algorithm A with verification decision limit selected as reference range ± allowable total error.
Evaluation Item All Results (n) To Be Reviewed (n) To Be Reviewed (%) Most frequent 3 tests caused “To Be Reviewed” flag VDL
Reference interval 3,045,579 896,511 29.4400 Creatinine Urea Uric acid
Reference interval ± TEa 692,605 22.7400 Glucose Protein Albumin
Statistical analysis 492,850 16.1800 Albumin Protein Calcium
Serum Indices 3,184,698 72,066 2.2629 CK-MB LDH D. Bilirubin
Analytical Limits 3,112,624 52,834 1.6974 LDH Lipase CK-MB
Delta Check* 2,352,974 20,362 0.8654 Albumin Calcium Protein
Critical Values 3,059,790 14,211 0.4644 Potassium Magnesium Calcium
Result Format 3,187,881 3,183 0.0998 CK-MB LDL-C Potassium
Rerun Results 3,188,095 214 0.0067 Sodium Chloride Potassium
Numeric Results 3,112,632 8 0.0003 LDL-C Chloride Creatinine
Sample Type 3,184,698 0 n/a n/a n/a n/a
*When reference range ± TEa was used as verification decision limit (VDL), n/a: not applicable, TEa: allowable total error.
publications [19,27,32]. This difference could be because our study was conducted in a tertiary referral hospital, which included inpatients with a high number of pathological results. Moreover, the low report-based AV rate in our study could be because rules in systems studied in the literature have been optimized over time [17,27]. Thus, laboratories can start using AVs with narrow limits, such as reference interval, and then these ranges can be extended to limits, such as laboratories’ own result distribution.
This step also reveals different AV ratios between inpatient and outpatient reports. When report-based AV rates for outpatients and in- patients were compared, the higher rate of AV in outpatient results can be explained by the prevalence of comorbidities in inpatients. This was distinctive in intensive care patients. Consistent with our results, studies have demonstrated that increasing the use of detailed information, such as diagnosis, clinical presentation, and medication, to increase the effectiveness of AV is crucial [19,21].
In contrast to previous studies, we tested two algorithms for VDL and
delta check evaluation to assess their performance. Our results demon- strated that using delta check evaluation is significant for inpatients who had more pathological results compared with outpatients. Algorithm B, which combines VDL and delta check evaluation, has a statistically higher AV rate for inpatient reports (Fig. 1, Table 5). Additionally, the AV rate increase in the inpatient group had a larger effect size than outpatient group. This finding suggests that the use of delta check, as in algorithm B, is important for inpatients and can increase the AV rate and reliability of AV.
The proposed model included the additional result format, sample type, and numeric result evaluation steps, which are generally not a part of AV. If such controls are not available with LIS or middleware, these evaluation criteria could be beneficial. Result format evaluation can be critical when using different analytical instruments that have incom- patible result fields. Our results revealed a very low number of invalid results in these steps because of LIS support. Our proposed model also provides a systematic method to define rules related with all phases of Table 4
Test-based autoverification rates for algorithms A and B according to all verification decision limits.*
Algorithm A Algorithm B
Test All Results (n) VDL – RI (%) VDL – RI ± TEa (%) VDL – SA (%) VDL – RI (%) VDL – RI ± TEa (%) VDL – SA (%)
Albumin 182,843 59.5 63.1 65.2 84.1 86.0 86.6
ALP 148,063 72.6 78.2 85.3 83.6 87.0 91.0
ALT 181,596 76.8 81.3 85.4 84.8 87.4 89.5
Amylase 44,432 76.6 83.3 86.1 83.2 88.4 90.6
AST 178,149 74.5 77.7 78.4 82.9 84.6 85.0
BUN 217,973 65.3 71.2 79.6 83.8 87.0 90.8
CK 43,472 78.0 85.6 88.7 85.1 90.1 92.1
CK-MB 43,970 40.3 47.4 54.9 40.3 47.4 54.9
Iron 10,614 78.6 90.5 95.0 81.4 91.6 95.6
D. Bilirubin 149,946 50.6 56.3 64.4 52.1 57.3 64.8
Phosphorus 122,335 70.4 79.6 78.2 84.5 89.5 88.9
GGT 112,011 66.2 72.2 87.5 66.2 72.2 87.5
Glucose 156,918 49.1 58.6 91.9 73.2 78.6 95.3
HDL-C 9,452 58.7 71.4 91.7 61.9 73.9 92.8
Calcium 191,242 66.9 74.3 72.7 85.9 89.4 89.0
Chloride 146,042 70.9 77.9 85.7 81.6 86.1 91.0
Creatinine 223,346 63.7 69.2 78.1 82.1 85.1 89.4
LDH 96,028 46.8 55.5 72.6 46.8 55.5 72.6
LDL-C 9,337 59.0 71.9 91.5 61.4 73.6 92.2
Lipase 12,900 71.3 80.4 83.4 77.0 84.5 85.9
Magnesium 19,833 72.6 79.1 73.8 78.3 83.4 79.8
Potassium 221,167 77.9 84.1 78.5 86.5 90.1 87.9
Total Protein 130,697 54.8 61.4 66.1 78.7 82.4 84.8
Sodium 224,079 76.2 78.5 85.0 86.4 87.9 91.6
T. Bilirubin 152,442 76.7 85.7 87.4 86.2 91.8 92.4
Total Cholesterol 9,513 49.0 65.5 92.8 50.5 66.5 93.0
Triglyceride 9,692 57.7 74.2 94.2 59.9 75.4 94.5
UIBC 9,660 73.1 94.8 94.0 73.1 94.8 94.0
Uric Acid 130,343 69.4 78.3 83.5 69.4 78.3 83.5
*Algorithms A and B have the same autoverification rates for GGT, LDH, UIBC, and uric acid because delta check was not used for these tests. D. Bilirubin: Direct Bilirubin, HDL-C: HDL-Cholesterol, LDL-C: LDL-Cholesterol, RI: Reference interval, RI-TEa: Reference interval ± allowable total error, SA: Statistical analysis, T.
Bilirubin: Total Bilirubin, UIBC: Unsaturated iron binding capacity, VDL: verification decision limit.
Table 5
Report-based autoverification rates and their differences for all verification decision limits. Report counts are n = 289,207 for all reports, n = 107,874 for outpatient reports, and n = 181,333 for inpatient reports.
VDL Algorithm A (%) Algorithm B (%) Difference (%) Chi-square value a Effect Size b
Reference Interval All Patients 18.0 28.1 10.1 8,317 0.12
Inpatients 10.0 22.8 12.8 10,835 0.17
Outpatients 31.5 37.0 5.6 741 0.06
Reference Interval ± TEa All Patients 25.2 35.5 10.3 7,257 0.11
Inpatients 14.4 27.9 13.5 9,908 0.16
Outpatients 43.4 48.3 4.9 526 0.04
Statistical Analysis All Patients 34.4 44.8 10.4 6,539 0.11
Inpatients 21.2 35.4 14.2 9,010 0.17
Outpatients 56.6 60.6 4.0 358 0.05
ap < 0.001 for all chi-square values.
b Effect size was calculated by using Cohen’s w effect size. TEa: allowable total error, VDL: verification decision limit.
TTP.
Related test evaluation provides a consistency check between different results and can be used to increase AV rates while preserving reliability [27]. We proposed a systemic approach for related test eval- uation, but it was not included in the scope of the study. Development of qualitative test evaluation algorithms is planned for further studies.
The effectiveness of the AV system can be defined as the capability to detect results at a constant rate that need additional inspection while verifying typical results and would not require further action if reviewed by experienced laboratory staff [9]. One of the limitations of this study is that we only evaluated AV rates as an indicator of efficiency. However, as in other AV development strategies, after obtaining a rule set, all laboratories should test if the AV system is working as expected and working in compliance with the laboratory’s current verification policy.
5. Conclusions
Rule creation is the most critical step for ensuring the reliability and efficiency of AV. Each laboratory should determine its own rule set, based on the tests used, patient population, and specialization charac- teristics. The model we proposed was designed for establishing detailed rules, which cover all TTP phases, while providing flexibility for different laboratory needs. Additionally, more rules could be defined for other test groups, such as hormone, hematology, coagulation, and immunology, by using the same algorithm with minimal modification.
Therefore, this comprehensive model can be a good starting point for AV setup.
Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Acknowledgements
The authors would like to acknowledge Prof. Dr. Sedef Yenice and Prof. Dr. Edward Randell for their useful suggestions.
Funding
This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.
Appendix A. Supplementary data
Supplementary data to this article can be found online at https://doi.
org/10.1016/j.clinbiochem.2021.03.018.
References
[1] U.-P. Rohr, C. Binder, T. Dieterle, F. Giusti, C.G.M. Messina, E. Toerien, H. Moch, H.H. Sch¨afer, Y. Wang, The value of in vitro diagnostic testing in medical practice:
A status report, PLOS ONE 11 (3) (2016) e0149856, https://doi.org/10.1371/
journal.pone.0149856.
[2] M. Plebani, Harmonization in laboratory medicine: more than clinical chemistry?
Clin. Chem. Lab. Med. 56 (2018) 1579–1586, https://doi.org/10.1515/cclm-2017- 0865.
[3] O. Martinez-Nieto, A. Lozano-Gait´an, P. Beltr´an-Diaz, I.L. Mojica-Figueroa, O.
L. Morales-Reyes, M.A. Isaza-Ruget, Autoverification of the automated blood cell counter (CBC) in a reference laboratory in Bogota, Colombia, J. Bras. Patol. e Med.
Lab. 51 (6) (2015), https://doi.org/10.5935/1676-2444.20150058.
[4] W. Neeley, Clinical and Laboratory Standards Institute., Autoverification of clinical laboratory test results: approved guideline, Clin. Lab. Stand. Inst. (2006).
[5] G.C. Guidi, G. Poli, A. Bassi, L. Giobelli, P.P. Benetollo, G. Lippi, Development and implementation of an automatic system for verification, validation and delivery of laboratory test results, Clin. Chem. Lab. Med. 47 (2009) 1355–1360, https://doi.
org/10.1515/CCLM.2009.316.
[6] J.L. Sepulveda, D.S. Young, The ideal laboratory information system, Arch. Pathol.
Lab. Med. 137 (2013) 1129–1140, https://doi.org/10.5858/arpa.2012-0362-RA.
[7] G.M. Saadawi, J.H.J. Harrison, Definition of an XML markup language for clinical laboratory procedures and comparison with generic XML markup, Clin. Chem. 52 (2006) 1943–1951, https://doi.org/10.1373/clinchem.2006.071449.
[8] M.S. Feitosa, D.H. Bücker, S.M.E. Santos, L.S. Vasconcellos, Implementation of criteria for automatic release of clinical chemistry test results in a laboratory at an academic public hospital, J. Bras. Patol. Med. Lab. 52 (3) (2016) 149–156, https://
doi.org/10.5935/1676-2444.20160026.
[9] E.W. Randell, G. Short, N. Lee, A. Beresford, M. Spencer, M. Kennell, Z. Moores, D. Parry, Autoverification process improvement by six Sigma approach: clinical chemistry & immunoassay, Clin. Biochem. 55 (2018) 42–48, https://doi.org/
10.1016/j.clinbiochem.2018.03.002.
[10] R. G´omez-Rioja, V. Alvarez, M. Ventura, M.J. Alsina, N. Barba, M. Cort´es, M.
A. Llopis, C. Martínez, M. Ibarz, Current status of verification practices in clinical biochemistry in Spain, Clin. Chem. Lab. Med. 51 (2013) 1739–1746, https://doi.
org/10.1515/cclm-2012-0659.
[11] P.M. Valdigui´e, E. Rogari, J.X. Corberand, B. Boneu, The performance of the knowledge-based system VALAB revisited: an evaluation after five years, Eur. J.
Clin. Chem. Clin. Biochem. 34 (1996) 371–376, https://doi.org/10.1515/
cclm.1996.34.4.371.
[12] J. Li, B. Cheng, L.i. Yang, Y. Zhao, M. Pan, G. Zheng, X. Xu, J. Hu, T. Xiao, Y. Cai, Development and implementation of autoverification rules for ELISA results of HBV serological markers, J. Lab. Autom. 21 (5) (2016) 642–651, https://doi.org/
10.1177/2211068215601612.
[13] X. Zhao, X.F. Wang, J.B. Wang, X.J. Lu, Y.W. Zhao, C.B. Li, B.H. Wang, J. Wei, P. Guo, J.P. Xiao, J.H. Wang, X.L. Yang, Multicenter study of autoverification methods of hematology analysis, J. Biol. Regul. Homeost. Agents 30 (2016) 571–577. https://www.ncbi.nlm.nih.gov/pubmed/27358150.
[14] J. Li, B. Cheng, H. Ouyang, T. Xiao, J. Hu, Y. Cai, Designing and evaluating autoverification rules for thyroid function profiles and sex hormone tests, Ann.
Clin. Biochem. 55 (2) (2018) 254–263, https://doi.org/10.1177/
0004563217712291.
[15] L. Onel¨ov, E. Gustafsson, E. Gr¨onlund, H. Andersson, G. Hellberg, I. J¨arnberg, S. Schurow, L. S¨oderblom, J.P. Antovic, Autoverification of routine coagulation assays in a multi-center laboratory, Scand. J. Clin. Lab. Investig. 76 (6) (2016) 500–502, https://doi.org/10.1080/00365513.2016.1200135.
[16] J. Wu, M. Pan, H. Ouyang, Z. Yang, Q. Zhang, Y. Cai, Establishing and evaluating autoverification rules with intelligent guidelines for arterial blood gas analysis in a Clinical Laboratory, SLAS Technol. 23 (6) (2018) 631–640, https://doi.org/
10.1177/2472630318775311.
[17] A.-E. Sediq, A.G.H. Abdel-Azeez, Designing an autoverification system in Zagazig University Hospitals Laboratories: preliminary evaluation on thyroid function profile, Ann. Saudi Med. 34 (5) (2014) 427–432, https://doi.org/10.5144/0256- 4947.2014.427.
[18] Y. Zhao, L. Yang, G. Zheng, Y. Cai, Building and evaluating the autoverification of coagulation items in the laboratory information system, Clin. Lab. 60 (2014) 143–150.
[19] M.D. Krasowski, S.R. Davis, D. Drees, C. Morris, J. Kulhavy, C. Crone, T. Bebber, I. Clark, D.L. Nelson, S. Teul, D. Voss, D. Aman, J. Fahnle, J.L. Blau,
Autoverification in a core clinical chemistry laboratory at an academic medical center, J. Pathol. Inform. 5 (1) (2014) 13, https://doi.org/10.4103/2153- 3539.129450.
[20] O. Martinez-Nieto, A. Lozano-Gaitn, P. Beltrn-Diaz, I.L. Mojica-Figueroa, O.
L. Morales-Reyes, M.A. Isaza-Ruget, Autoverification of the automated blood cell counter (CBC) in a reference laboratory in Bogota, Colombia, J. Bras. Patol. e Med.
Lab. 51 (2015) 369–375.
[21] R.M. Dorizzi, B. Caruso, S. Meneghelli, P. Rizzotti, The DNSevTM expert system in the auto-verification of tumour markers and hormones results, Accredit. Qual.
Assur. 11 (2006) 303–307, https://doi.org/10.1007/s00769-006-0160-6.
[22] Ferhat Demirci, Pinar Akan, Tuncay Kume, Ali Riza Sisman, Zubeyde Erbayraktar, Suleyman Sevinc, Artificial neural network approach in laboratory test reporting:
learning algorithms, Am. J. Clin. Pathol. 146 (2) (2016) 227–237, https://doi.org/
10.1093/ajcp/aqw104.
[23] Quang A. Le, Greg Strylewicz, Jason N. Doctor, Detecting blood laboratory errors using a Bayesian network: an evaluation on liver enzyme tests, Med. Decis. Mak. 31 (2) (2011) 325–337, https://doi.org/10.1177/0272989X10371682.
[24] Jay B. Jones, A strategic informatics approach to autoverification, Clin. Lab. Med.
33 (1) (2013) 161–181, https://doi.org/10.1016/j.cll.2012.11.004.
[25] G.A. Edwards, Expert systems for clinical pathology reporting, Clin. Biochem. Rev.
29 (Suppl 1) (2008) S105–S109.
[26] Türkiye Cumhuriyeti Sa˘glık Bakanlı˘gı Sa˘glık Hizmetleri Genel Müdürlü˘gü Tetkik Hizmetleri Daire Bas¸kanlı˘gı, Onay destek sistemi kılavuzu, (2018). https://
shgmlabdb.saglik.gov.tr/Eklenti/15045/0/tibbi-laboratuvarda-onay- desteek41916132pdf.pdf (Last access: March 2021).
[27] Mu-Chin Shih, Huey-Mei Chang, Ni Tien, Chiung-Tzu Hsiao, Ching-Tien Peng, Building and validating an autoverification system in the Clinical Chemistry Laboratory, Med. 42 (11) (2011) 668–673, https://doi.org/10.1309/
LM5AM4IIXC4OIETD.
[28] E.W. Randell, S. Yenice, A.A. Khine Wamono, M. Orth, Autoverification of test results in the core Clinical Laboratory, Clin. Biochem. 73 (2019) 11–25, https://
doi.org/10.1016/j.clinbiochem.2019.08.002.
[29] Sang Hyuk Park, So-Young Kim, Woochang Lee, Sail Chun, Won-Ki Min, New decision criteria for selecting delta check methods based on the ratio of the delta difference to the width of the reference range can be generally applicable for each clinical chemistry test item, Ann. Lab. Med. 32 (5) (2012) 345–354, https://doi.
org/10.3343/alm.2012.32.5.345.
[30] CLSI, Use of Delta Checks in the Medical Laboratory, 1st edition CLSI Guideline ep33, Clinical and Laboratory Standards Institute, Wayne, PA, 2016, (2016). CLSI.
Use of delta checks in the medical laboratory; Approved Guideline – First Edition.
CLSI Document EP33. Wayne, PA: Clinical and Laboratory Standards Institute;
2016.
[31] R. Core Team, R: A language and environment for statistical computing, 2017.
[32] N. Torke, L. Boral, T. Nguyen, A. Perri, A. Chakrin, Process improvement and operational efficiency through test result autoverification, Clin. Chem. 51 (2005) 2406–2408, https://doi.org/10.1373/clinchem.2005.054395.
