Understanding Quality Assessment Strategies and Applying Them to Telehealth Pharmacy Practice

in Telehealth: Strategies for Establishing Pharmacy Practice Models in Ambulatory Care Settings
Free access

The framework for quality assurance (QA) has a long history in healthcare, stretching back centuries since the time of Hippocrates.1 Medicine evolved into an evidence-based care culture, which is reflected in modern practice.2 QA in healthcare is one of the top priorities for most providers and health organizations. This is because healthcare organizations recognize that QA programs can help prevent, detect, and potentially correct any problems that may arise in patients, which will translate into higher quality and standards of care.3 Whereas QA is a structured and continuous program, quality assessment is the method for data collection and its subsequent analysis to which its results are compared to predefined standards.4 QA programs have evolved significantly over time and are described below, from surveys evaluating hospital performance to establishing metrics for the digital age, like telehealth.

In the United States, an effort to achieve high-quality standards was developed gradually. Between 1995 and 2000, several quality improvement initiatives were performed.5 Eventually, the Health Care Finance Administration (HCFA) identified quality indicators to routinely assess the excellence of care administered to its Medicare beneficiaries.6 These quality indicators were broken into six domains of healthcare: safety, effectiveness, timeliness, patient-centeredness, efficiency, and equity.7 The last three domains, as identified by the Institute of Medicine (IOM), are patient-centered and revolve around patient satisfaction or, in other words, patient experience. In general, quality indicators for healthcare are a vital tool used to track and evaluate the performance of an individual provider and a healthcare system. The specific indicators developed by HCFA focused on reviewing the impact of an intervention on the health status of a patient (e.g., percentage of patients diagnosed with hypertension with controlled blood pressures). Although this appeared to be the gold standard of assessing quality, relying solely on outcome measures to demonstrate therapeutic effectiveness has many drawbacks. Outcomes are the result of many contributing factors, many of which are out of an individual healthcare provider's control.

Parallel to the efforts of the HCFA, in 1994 the Healthcare Cost and Utilization Project (HCUP) created a set of quality indicators for hospitals, which was derived from discharge data. The Agency for Healthcare Research and Quality (AHRQ) maintains these commonly used indicators, which incorporate structural measures that address a provider's or a system's capacity (e.g., ratio of providers to patients) and process measures suggest what a provider does to improve health (e.g., percentage of patients who receive a preventive service like a screening colonoscopy).8 Although AHRQ Quality Indicators (Qis) are used widely, many organizations have developed and use their own sets of quality indicators (Table 10.1). Most recently, some organizations have been developing quality indicators for telehealth, which will be described later in this chapter.

Table 10.1.

AHRQ Quality Indicator Modules for Adult Patients9

Domain

Definition

Example Indicators

Prevention Quality Indicators (PQIs)

Calculates inpatient admissions that may have been avoided with access to outpatient care.

  • Diabetes short-term complication admission rate

  • Perforated appendix admission rate

  • Diabetes long-term complication admission rate

  • Chronic obstructive pulmonary disease admission rate

  • Hypertension admission rate

  • Congestive heart failure (CHF) admission rate

  • Low birth weight rate

  • Dehydration admission rate

  • Bacterial pneumonia admission rate

  • Urinary tract infection admission rate

  • Angina admission without procedure

  • Uncontrolled diabetes admission rate

  • Adult asthma admission rate

  • Rate of lower-extremity amputation among patients with diabetes

Inpatient Quality Indicators (IQIs): Provider-Level Volume

Focuses on volume and measures the total amount of admissions and procedures performed.

  • Esophageal resection volume

  • Pancreatic resection volume

  • Abdominal aortic aneurysm (AAA) repair volume

  • Coronary artery bypass graft (CABG) volume

  • Percutaneous transluminal coronary angioplasty (PTCA) volume

  • Carotid endarterectomy (CEA) volume

IQIs: Mortality Indicators

Includes indicators that cover procedures where higher mortality is associated with poorer care.

  • Esophageal resection mortality rate

  • Pancreatic resection mortality rate

  • AAA repair mortality rate

  • CABG mortality rate

  • PTCA mortality rate

  • CEA mortality rate

  • Craniotomy mortality rate

  • Hip replacement mortality rate

IQIs: Mortality Indicators with Varying Mortality Rates

Describes procedures where volumes vary throughout the country. These indicators cover procedures where higher mortality rate is associated with poorer care.

  • Acute myocardial infarction (AMI) mortality rate

  • AMI mortality rate, without transfer cases

  • CHF mortality rate

  • Acute stroke morality rate

  • Gastrointestinal hemorrhage mortality rate

  • Hip fracture mortality rate

  • Pneumonia mortality rate

IQIs: Utilization Indicators

Includes indicators that measure high or low rates of specific procedures. High or low rates for these indicators represent inappropriate care delivery.

  • Cesarean section delivery rate

  • Primary cesarean delivery rate

  • Vaginal birth after cesarean (VBAC) rate

  • VBAC rate, uncomplicated

  • Laparoscopic cholecystectomy rate

  • Incidental appendectomy in the elderly rate

  • Bilateral cardiac catherization rate

Patient Safety Indicators (PSIs): Provider-level

Uses hospital inpatient discharge data to understand patient safety. Focuses on postoperative complications, adverse events, challenges with procedures, and trauma related to birth.

  • Postoperative pulmonary embolism or deep vein thrombosis

  • Postoperative respiratory failure

  • Postoperative sepsis

  • Postoperative physiologic and/or metabolic derangements

  • Postoperative abdominopelvic wound dehiscence

  • Postoperative hip fracture

  • Postoperative hemorrhage or hematoma

  • Decubitus ulcer

  • Selected infections due to medical care

  • Iatrogenic pneumothorax

  • Accidental puncture or laceration

  • Foreign body left in during procedure

  • Birth trauma resulting in injury to neonate

  • Obstetric trauma due to vaginal delivery with instrument

  • Obstetric trauma from a vaginal delivery without an instrument

  • Obstetric trauma related to cesarean section delivery

  • Complication of anesthesia

  • Death in low-mortality diagnosis-related groups (DRGs)

  • Death in surgical inpatients with treatable serious complications (a previously known failure to rescue)

  • Transfusion reactions (AB/Rh)

After these quality indicators were developed, public reporting efforts began in 2006 under the Physician Quality Reporting Initiative (PQRI). Under the Tax Relief and Health Care Act (TRHCA) of 2006, PQRI enacted a pay-for-reporting system. This program focused on providing a formal mechanism for healthcare providers to evaluate and improve the quality of the care they provide to Medicare beneficiaries.10 The program provided incentive payments, which consisted of a 1.5% bond on Medicare Part B fee-for-service (FFS) charges for successful reporting on three quality measures. Then, in 2008, the Medicare Improvement for Patients and Providers Act (MIPPA) was enacted, and the incentive payments were increased to 2%.5 Although this was the beginning of formalized quality assurance programs, the Patient Protection and Accountable Care Act which was signed into law in 2010 highlighted the importance of a continuous quality program in healthcare.5 The overarching goal of this law was to improve quality of care, expand access, and lower healthcare costs for Medicare and Medicaid beneficiaries. To achieve this goal, the Patient-Centered Outcomes Research Institute (PCORI) was launched and began researching the effectiveness of various clinical interventions determining which treatments were the most effective for various patient populations. These findings guided evidence-based treatment algorithms to ensure high-quality care is maintained. In 2015, the Medicare Access and CHIP Reauthorization Act (MACRA) was passed. This law transitioned payments to the Merit-based Incentive Payment System (MIPS), which incorporates alternative payment models and merit-based payment systems.11 MIPS combines three of CMS's value-based programs: the physician quality reporting system, the value-based payment modifier, and the Medicare electronic health record (EHR) incentive program.12 Although this value-base model can be utilized across inpatient and outpatient facilities, because of this combination of reporting programs many outpatient providers and organizations have elected to enroll in MIPS payment models. Overall, 280 quality measures have been identified (Table 10.2).11

Table 10.2.

Example of MIPS Quality Measures13

Heart failure (HF): beta-blocker therapy for left ventricular systolic dysfunction (LVSD)

Urinary Incontinence: assessment of presence or absence of urinary incontinence in women aged 65 or older

Urinary incontinence: plan of care for urinary incontinence in women aged 65 years or older

Chronic obstructive pulmonary disease (COPD): long-acting inhaled bronchodilator therapy

Prostate cancer: avoidance of overuse of bone scan for staging low-risk prostate cancer patients

Prostate cancer: combination androgen deprivation therapy for high-risk or very high-risk prostate cancer

Preventive care screening: body mass index (BMI) screening and follow-up plan

TELEHEALTH AND QUALITY ASSURANCE STRATEGIES

Telehealth encounters have surged during the COVID-19 pandemic. In a study by Koonin and colleagues, telehealth visits increased substantially from 2019 through 2020. For example, in the first quarter of 2020, there was a 50% increase in telehealth visits compared to the same period in 2019.14 In another report by the Mayo Clinic, in-person visits declined a dramatic 78% during the COVID-19 pandemic from mid-March to mid-April. However, during that same time, telehealth was widely utilized, and the health system reported an astonishing 10,880% increase in this digital healthcare service.15 Although there has been a large effort to develop quality measures for in-person visits in inpatient and outpatient settings, these assessments for telehealth are lacking. Patients have generally responded positively to telehealth as a means of obtaining healthcare. Studies have demonstrated that patients are generally satisfied with telehealth services and report that saving time is a primary benefit of digital platforms.16,17 However, for all the purported benefits that telehealth has provided for patients and healthcare providers, the need to ensure that patients are deriving the same benefit from a digital health visit that they would receive from an in-person visit remains. Prior to the COVID-19 pandemic, several studies questioned if a telehealth platform is a more advantageous way to measure, track, and report the quality of care given to patients.18 But with the widespread adoption of telehealth as a means of managing acute and chronic health conditions, the question of quality assessment for this visit type remains even more paramount.

Patient Experience

Patient experience is often tied to essential quality indicators. However, measuring the patient experience has been challenging. In 2008, with passage of MIPPA there was also a provision to assess the patient experience as part of the quality assessment. A variety of surveys have been used to measure patient encounters in both hospitals and outpatient clinics, but MIPPA tied these metrics to financial reimbursements, as detailed above.5,19 A variety of surveys have been used in the ambulatory setting such as the Consumer Assessment of Healthcare Providers and Systems Clinician and Group (CG-CAHPS) survey for outpatient surgery and the National Ambulator Medical Care Survey (NAMCS), or the Ambulatory Care Experiences Survey (ACES). Press Ganey's CG-CAHPS is one of the most widely used and validated patient experience surveys. It assesses the patient experience of adults and children in ambulatory or specialized care (Table 10.3). By weaving in the patient perspective with these quality indicators, the approach to quality becomes more holistic and allows care to be patient-centered and embody a team-based approach. Efforts like this provide national benchmarks for patient outcomes in a structured manner that assists healthcare professionals and organizations to provide high-quality evidence-based care to their patients.

Table 10.3.

Select Quality Indicators for Telehealth: NQF Recommendations28

Domain

Subdomain

Measure Concept

Experience

Patient, family, and/or caregiver

Overall improvement in quality of life because services are received at home

Experience

Patient, family, and/or caregiver

Patients able to interpret diagnosis and treatment instructions through telehealth

Experience

Patient, family, and/or caregiver

Decrease in wait times for patients

Financial impact/cost

Financial impact on health system or payer

Visit duration measured versus in-home care

Financial impact/cost

Financial impact on care team

Decrease in no-show rate

Access to care

Access for care team

In-person visit agreed to after a telehealth consultation

Access to care

Access for care team and for patient, family, and/or caregiver

Number of interprofessional visits

Effectiveness

System effectiveness

Amount of time to schedule a visit

Effectiveness

System effectiveness

Amount of time to check in for a visit

Effectiveness

Clinical effectiveness

Relationship of the telehealth modality to the therapeutic need of the patient

Effectiveness

Operational effectiveness

Can telehealth offer similar quality of services across a population of similar patients?

Example of CG-CAHPS Survey Questions20

Each using the scale: 1  Very poor 2  Poor 3  Fair 4  Good 5  Very good

  • Friendliness/courtesy of the care providers

  • Explanations the care provider gave you about your problem or condition

  • Concern the care provider showed for your questions or worries

  • Care provider's efforts to include you in decisions about your treatment

  • Information the care provider gave you about medications (if any)

  • Instructions the care provider gave you about follow-up care (if any)

  • Degree to which the care provider talked with you using words you could understand

  • Amount of time the care provider spent with you

  • Your confidence in this care provider

  • Likelihood of your recommending this care provider to others

Telehealth Pharmacy Practice Patient Experience

Assessing the patient experience with telehealth across different demographics and socioeconomic situations is essential for organizations. In general, patients appear to be very satisfied with the telehealth platform to access their healthcare providers. In a retrospective single-center cohort study in New York City, 38,609 Press Ganey patient satisfaction surveys were evaluated and found that scores were significantly higher (94.9% vs. 92.5%; p <0.001) compared to in-person visits.21 Compared to in-person visits, there are several ways beyond the survey to assess patient experience with telehealth. For example, other key data points can be passively pulled which illuminate a patient's preferences to the telehealth platform. For example, identifying patient visits as either telephone only, video only, or in-person can help determine patient preferences. Organizations can also use custom patient survey questions to glean information on telehealth quality and the patient experience. By asking questions regarding time or money saved, access to technology and preference of using this technology can help organizations understand the patient experience with telehealth. An alternative to developing internal patient survey questions is to use established questionnaires to measure the patient experience. Commonly used validated surveys include the Telehealth Usability Questionnaire (TUQ) or the Telemedicine Satisfaction and Usefulness Questionnaire.22 These questionnaires assess usefulness, ease of use, quality of the telehealth interface, quality of the interaction with telehealth, reliability of the system, and patient experience.23,24

Clinical Quality Audits

With this infrastructure, benchmark reporting, and methodology being set at a national level, it is critical that individual provider groups and organizations understand how to perform a clinical quality audit. The clinical audit is a key part of the quality improvement process, but it is also a measure of effectiveness. In general, it requires measuring the outcomes achieved against established standards, which were reviewed above. It is crucial to understand that the clinical audit goes beyond data collection. Successful clinical audits measure patient outcomes against standards or defined quality indicators, with the expectation of improving that practice. Clinical audits should be repeated frequently to ensure that any changes made to a practice demonstrate improvement.25 The first step to a successful clinical audit is to identify the topic to be evaluated. The ideal topic should be one that meets the following benchmarks:25

  • High volume of work

  • Risky

  • Highly variable

  • Complex

  • Potentially innovative

An overview of how to perform a clinical audit is drawn in Figure 10.1. Following proper identification of a topic, the scope of the project should be developed along with the corresponding methodology. The goal of the audit could be to evaluate a new process (e.g., patient adherence to warfarin using a telehealth platform) and/or improvement of a current process.26 Once the topic is identified and its corresponding methodology is developed, quality indicators should be selected. With these two key pieces of information defined, data collection can begin. This data collection may be performed either prospectively or retrospectively. The data itself can be quantitative or qualitative and include elements such as surveys, interviews, and questionnaires.27 Discrete data points from medical records like lab results or diagnosis codes can be included. A patient's medical record is primarily the main source of information for most clinical audits, but it is important to know that it may be lacking all the required data points needed.25 Following data collection, a comparison of your collected data with the benchmark standards must occur. This is the cornerstone of the clinical audit. Results from this comparison can demonstrate whether a standard is met and ultimately that further improvement is needed. If results are close to the standard used, then the clinical audit can enter a monitoring phase.25,27 Lastly, the cycle ends with change implementation. Following a change, it is important to go back and perform the clinical audit again to ensure that the changes made were positive and align with the standards previously identified.27

FIGURE 10.1.
FIGURE 10.1.

Key Components of the clinical audit process. Adapted from Ashmore S, Ruthven T. Clinical audit: a guide. Nursing Management (Harrow), 2008;15(1):18-22.

Telehealth Clinical Quality Audits

Identifying clinical outcome targets is as important as identifying quality outcomes. Although this is currently ill defined and the overall data are lacking, the rapid adoption of telehealth offers an opportunity to compare outcome measures following telehealth and in-person visits among patients with chronic health conditions like heart disease and diabetes. Although telehealth may offer opportunities to increase patient compliance, pharmacists must follow recommended and evidence-based practices when they provide care through this modality.

To perform clinical audits of telehealth, it is important that key indicators be abstracted in a chart review (Table 10.3). Having components like a telehealth consent and indicators about mode of technology used for a patient visit can make this abstraction easier. Lastly, it is crucial that organizations evaluate medication safety within the telehealth context. Reviewing charts for adverse events and creating voluntary reporting tools to catch medication errors related to telehealth are essential.

When you are determining which metrics to track for a telehealth service, it may be worthwhile to evaluate any metrics that are being tracked for a similar in-person service. Often, the goals of a service may be very similar regardless of whether the service is being delivered in person or by using telehealth. To date, very little has been published on the effectiveness of pharmacy services provided via traditional methods (i.e., face to face) compared with telehealth services.

Medication Safety

Routine clinical quality audits are one piece of the quality assurance puzzle. A key component of quality assurance for any organization is ensuring safe practices for patients. One way to improve practice is through a medication safety review. Medication errors and medication-related adverse events have serious ramifications in patient care. Errors due to healthcare are the fourth leading cause of death in the United States and the leading cause of preventable death.29 Medication errors are also associated with increased length of hospital stays and additional costs. Therefore, to prevent erroneous morbidity and mortality from medication-related adverse events or errors, detection of these events is a paramount step.

On a national level, various regulatory agencies and federal organizations collect data and create standardized recommendations or alerts, such as the Food and Drug Administration Med-Watch program, the Joint Commission on the Accreditation of Healthcare Organizations, and the Institute of Safe Medication Practices. However, individual organizations must also develop tools to analyze errors and identify areas to improve quality and make system changes. In order to monitor safe medication processes, organizations must evaluate both adverse events and medication errors. However, the methods used to evaluate each of these categories varies. Most organizations use chart review to detect adverse events. By utilizing monitoring tools within the electronic health record, incident reporting, or auditing claims data, adverse events can be passively pulled and demonstrate clinical outcomes for the entire patient population in question.30 Medication errors are primarily detected by direct observation and voluntary reporting. Most organizations have employee reporting tools such as the Patient Safety Net (PSN) through which healthcare providers can summarize a patient event and each event undergoes a root cause analysis. Detection methods commonly used to assess medication errors are outlined in Table 10.4. Organizations can use these events to implement changes. Interestingly, the adoption of telemedicine may result in a reduction of medication and medical errors. A study using remote pharmacist review of medication orders for three community hospitals found an increase in interventions resulting in improved patient safety.31 By leveraging this technology, patient safety can be optimized in new ways; however, more data are needed to support patient safety benefits.

Table 10.4.

Commonly Used Methods to Assess Medication Errors and Adverse Events30

Method

Advantages

Disadvantages

Efficacy

Chart abstraction/review

  • Retroactive

  • Uses standardized criteria

  • Time consuming

  • Labor-intensive

Gold standard to detect adverse events

Claims data

  • Uses local data

  • Captures unapparent failures

  • Litigation based and may have legal implications

Adverse events detected

Incident reporting (sentinel events)

  • High-quality data

  • Captures active and unapparent failures

  • Only identifies severe, unexplained events or deaths

  • Can underestimate rates of adverse events

  • May cause feelings of blame, fear of punishment

  • Contributes to reports and alerting

  • Measures adverse events

Voluntary reporting

  • Variety of sources can report events

  • Uses a structured and simplified form

  • Captures active and unapparent failures

  • Promotes a culture of safety

  • Variable quality of reporting

  • Underreporting

  • Individuals may feel the tool contributes to a blame culture

  • No data integration

  • Contributes to reports and alerting

  • Provides path for feedback and opportunities for corrective actions

  • Medication errors are detected

Computer monitoring

  • Multidata source integration

  • Real time

  • Adverse event prevention

  • Poor software

  • Poor triggers

  • Undetermined future risks

  • Inadvertent errors in data inserted

  • Captures prescribing faults

  • Captures prescription errors

  • Captures dispensing errors

Direct care observation

  • Accurate

  • Captures active errors

  • Time consuming

  • Difficult to train

  • Data are centralized around administration errors

Patient monitoring

  • Data from outpatients

  • Wide impacts

  • Tools are not standardized and vary (interviews, questionnaires, focus groups, etc.)

  • Can help direct future developments of patient-centered care

Maintaining Quality and Safety with Pharmacy Telehealth Practice

At the time of this writing, standardized quality assessment and assurance approaches to telehealth pharmacy practice models have largely not been developed. In fact, national conversations about this topic are ongoing. However, thanks to the above-outlined robust quality initiatives that have taken place over the past three decades, we can incorporate existing approaches and modify them to a telehealth pharmacy practice visit. To perform a quality assessment, it is imperative that quality indicators be identified. Nonprofits like the National Quality Forum (NQF) are attempting to develop standardized quality indicators for telehealth that focus on access to care, financial impacts or cost, the telehealth experience, and effectiveness of treatment. However, benchmarks are currently not well defined and need to be further developed.

REFERENCES

  • 1

    De Jonge V, Sint Nicolaas J, van Leerdam M, et al.Overview of the quality assurance movement in health care. Best Pract Res Clin Gastroenterol. 2011;25(3):337-347. doi: 10.1016/j.bpg.2011.05.001.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 2

    Guyatt G, Cook D, Haynes B. Evidence based medicine has come a long way. BMJ. 2004;329(7473):990-991. doi: 10.1136/bmj.329.7473.990.

  • 3

    Institute of Medicine (US) Committee to Design a Strategy for Quality Review and Assurance in Medicare; Lohr KN (ed). Medicare: A Strategy for Quality Assurance: Volume 1. Washington, DC: National Academies Press, 1990. 9, Methods of Quality Assessment and Assurance. https://www.ncbi.nlm.nih.gov/books/NBK235459.

    • Search Google Scholar
    • Export Citation
  • 4

    Mainz J, Hammershøy EM, Worning AM, et al.Quality assessment, quality assurance and quality improvement in health care. Concepts and terminology. Ugeskr Laeger. 1992;154(17):1151-1156.

    • Search Google Scholar
    • Export Citation
  • 5

    Marjoua Y, Bozic K. Brief history of quality movement in US healthcare. Curr Rev Musculoskelet Med. 2012;5(4):265-273. doi: 10.1007/s12178-012-9137-8.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 6

    Jencks SF, Cuerdon T, Burwen DR, et al.Quality of medical care delivered to Medicare beneficiaries: A profile at state and national levels. JAMA. 2000;284:1670-1676. doi: 10.1001/jama.284.13.1670.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 7

    Berwick DM. A user's manual for the IOM's “Quality ChasmReport. Health Aff. 2002;21(3):80-90. doi: 10.1377/hlthaff.21.3.80.

  • 8

    Types of health care quality measures. (n.d.). https://www.ahrq.gov/talkingquality/measures/types.html. Accessed on May 18, 2021.

  • 9

    Farquhar M. AHRQ Quality indicators. In: Hughes RG (ed). Patient Safety and Quality: An Evidence-Based Handbook for Nurses. Rockville, MD: Agency for Healthcare Research and Quality; 2008 Apr. Chapter 45. https://www.ncbi.nlm.nih.gov/books/NBK2664. Accessed May 18, 2021.

    • Search Google Scholar
    • Export Citation
  • 10

    Dowd B, Li CH, Swenson T, et al.Medicare's Physician Quality Reporting System (PQRS): Quality measurement and beneficiary attribution. Medicare Medicaid Res Rev. 2014 Jun 25;4(2). doi: 10.5600/mmrr.004.02.a04.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 11

    Manchikanti L, Hammer M, Benyamin R, et al.Merit-based incentive payment system (MIPS): Harsh choices for interventional pain management physicians. Pain Physician. 2016;19:E15-E32.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 12

    Yon K, Sinclair J, Bentley OS, et al.Impact of quality measures performed through pharmacist collaboration with a primary care clinic. J Am Pharm Assoc. 2003; 2020;60(3 S): S97-S102. https://doi.org/10.1016/j.japh.2020.01.011.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 13

    2020 Annual Call for Quality Measures Fact Sheet. (2020, May 22). Retrieved May 06, 2021. https://www.cms.gov/files/document/2020-mips-call-quality-measure-overview-fact-sheet.pdf

  • 14

    Koonin L, Hoots B, Tsang C, et al.Trends in the use of telehealth during the emergence of the COVID-19 pandemic—United States, January-March 2020. Weekly. 2020;69(43):1595-1599.

    • Search Google Scholar
    • Export Citation
  • 15

    Demaerschalk B, Blegen R, Ommen S. Scalability of telemedicine services in a large integrated multispecialty health care system during COVID-19. Telemed e-Health. 2021:96-98. http://doi.org/10.1089/tmj.2020.0290.

    • Search Google Scholar
    • Export Citation
  • 16

    Polinski JM, Barker T, Gagliano N, et al.Patients’ satisfaction with and preference for telehealth visits. J Gen Intern Med. 2016 Mar;31(3):269-275. doi: 10.1007/s11606-015-3489-x.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 17

    Jacobs J, Ferguson J, Van Campen J, et al.Organizational and external factors associated with video telehealth use in the veterans’ health administration before and during the COVID-19 pandemic. Telemedicine and e-Health. 2021. doi: http://doi.org/10.1089/tmj.2020.0530.

    • Search Google Scholar
    • Export Citation
  • 18

    Halpren-Ruder D, Chang A, Hollander J, et al.Quality assurance in telehealth: Adherence to evidence-based indicators. Telemed J E Health. 2019;25(7):599-603. doi: 10.1089/tmj.2018.0149.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 19

    Mazurenko O, Collum T, Ferdinand A, et al.Predictors of hospital patient satisfaction as measured by HCAHPS. J Healthcare Manag. 2017;62(4):272-283. doi: 10.1097/JHM-D-15-00050.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 20

    An overview of Version 3.0 of the CAHPS Clinician & Group Survey. (2015, June 17). Retrieved May 6, 2021. https://www.ahrq.gov/sites/default/files/wysiwyg/cahps/surveys-guidance/cg/about/cg_3-0_overview.pdf.

  • 21

    Ramaswamy A, Yu M, Drrangsholt S, et al.Patient satisfaction with telemedicine during the COVID-19 pandemic: Retrospective cohort study. J Med Internet Res 2020;22(9):e20786. doi: 10.2196/20786.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 22

    Chuo J, Macy ML, Lorch SA. Strategies for evaluating telehealth. Pediatrics 2020 Nov;146(5):e20201781. doi: 10.1542/peds.2020-1781.

  • 23

    Parmanto B, Lewis Jr A, Graham K, et al.Development of the telehealth usability questionnaire (TUQ). Int J Telerehabil. 2016;8(1);3-10. doi: 10.5195/ijt.2016.6196.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 24

    Bakken S, Grullon-Figueroa L, Izquierdo R, et al.Development, validation, and use of English and Spanish versions of the telemedicine satisfaction and usefulness questionnaire. J Am Med Inform Assoc. 2006l 13(6):660-667. doi: 10.1197/jamia.M2146.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 25

    Esposito P, Dal Canton A. Clinical audit, a valuable tool to improve quality of care: General methodology and applications in nephrology. World J Nephrol. 2014;3(4):249-255. doi: 10.5527/wjn.v3.i4.249.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 26

    Buth S, D’Cruz L. Clinical audit—process and outcome for improved clinical practice. Dent Update. 2012;39:710-714. doi: 10.12968/denu.2012.39.10.710.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 27

    Soimakallio S, Alanen A, Jarvinen H, et al.Clinical audit: Development of the criteria of good practices. Radiat Prot Dosimetry. 2011;147(1-2):30-33. doi: 10.1093/rpd/ncr260.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 28

    Creating a framework to support measure development for telehealth. (2017, August). http://www.qualityforum.org/Publications/2017/08/Creating_a_Framework_to_Support_Measure_Development_for_Telehealth.aspx. Accessed May 18, 2021.

  • 29

    Light D, Lexchin J, Darrow J. Institutional corruption of pharmaceuticals and the myth of safe and effective drugs. J Law, Med Ethics. 2013;14(3):590-610. doi: 10.1111/jlme.12068.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 30

    Montesi G, Lechi A. Prevention of medication errors: Detection and audit. Br J Clin Pharmacol. 2009;67(6):651-655. doi: 10.1111/j.1365-2125.2009.03422.x.

    • Crossref
    • Search Google Scholar
    • Export Citation
  • 31

    Schneider PJ. Evaluating the impact of telepharmacy. Am J Health Syst Pharm. 2013;70:2130-2135. doi: 10.2146/ajhp130138.