Loading...
Loading...

“You’re almost glancing behind a curtain”: HIV care provider perceptions regarding integration and value of pre-visit electronic tablet-based patient-reported outcomes data collection in two North American clinics Creative Commons

Link for citation this article Add this article in bookmark list
Rob Fredericksen, Division of Allergy and Infectious Diseases, Harborview Medical Center, 325 Ninth Avenue, Seattle, WA 98104, Box 359931, USA.
Vincent Hodge, Division of Allergy and Infectious Diseases, Harborview Medical Center, 325 Ninth Avenue, Seattle, WA 98104, Box 359931, USA.
Alexandra Musten, Division of Allergy and Infectious Diseases, Harborview Medical Center, 325 Ninth Avenue, Seattle, WA 98104, Box 359931, USA.
Abigail Kroch, Division of Allergy and Infectious Diseases, Harborview Medical Center, 325 Ninth Avenue, Seattle, WA 98104, Box 359931, USA.
Jean Bacon, Division of Allergy and Infectious Diseases, Harborview Medical Center, 325 Ninth Avenue, Seattle, WA 98104, Box 359931, USA
Brenda Jacobs, Division of Allergy and Infectious Diseases, Harborview Medical Center, 325 Ninth Avenue, Seattle, WA 98104, Box 359931, USA.
Moti Ramgopal, Division of Allergy and Infectious Diseases, Harborview Medical Center, 325 Ninth Avenue, Seattle, WA 98104, Box 359931, USA.
Shivali Suri, Division of Allergy and Infectious Diseases, Harborview Medical Center, 325 Ninth Avenue, Seattle, WA 98104, Box 359931, USA.
Divya Korlipara, Division of Allergy and Infectious Diseases, Harborview Medical Center, 325 Ninth Avenue, Seattle, WA 98104, Box 359931, USA.
Emma Fitzsimmons, Division of Allergy and Infectious Diseases, Harborview Medical Center, 325 Ninth Avenue, Seattle, WA 98104, Box 359931, USA.
Duncan Short, Division of Allergy and Infectious Diseases, Harborview Medical Center, 325 Ninth Avenue, Seattle, WA 98104, Box 359931, USA.
Heidi Crane, Division of Allergy and Infectious Diseases, Harborview Medical Center, 325 Ninth Avenue, Seattle, WA 98104, Box 359931, USA.
William Lober, Division of Allergy and Infectious Diseases, Harborview Medical Center, 325 Ninth Avenue, Seattle, WA 98104, Box 359931, USA.
Justin Mcreynolds, Division of Allergy and Infectious Diseases, Harborview Medical Center, 325 Ninth Avenue, Seattle, WA 98104, Box 359931, USA.
Kevin Gough, Division of Allergy and Infectious Diseases, Harborview Medical Center, 325 Ninth Avenue, Seattle, WA 98104, Box 359931, USA.
Mark Naccarato, Division of Allergy and Infectious Diseases, Harborview Medical Center, 325 Ninth Avenue, Seattle, WA 98104, Box 359931, USA.
Deborah Yoong, Division of Allergy and Infectious Diseases, Harborview Medical Center, 325 Ninth Avenue, Seattle, WA 98104, Box 359931, USA.
Jeff Berry, Division of Allergy and Infectious Diseases, Harborview Medical Center, 325 Ninth Avenue, Seattle, WA 98104, Box 359931, USA.
Darrell Tan Division of Allergy and Infectious Diseases, Harborview Medical Center, 325 Ninth Avenue, Seattle, WA 98104, Box 359931, USA.
Journal of AIDS and HIV Research, Journal Year: 2022, Volume and Issue: Vol.14(1), P. 22 - 31 https://doi.org/10.5897/JAHR2022.0548

Published: Jan. 1, 2022

This article is published under the license License

Loading...
Link for citation this article Related Articles

Abstract

This study was conducted to assess provider perceptions regarding value of tablet-based patient-reported outcome (PRO) assessments in routine care. Patients self-administered a brief PRO assessment of several clinical domains on-site prior to HIV clinic visits. Providers were given succinct summary results. 1:1 interviews were conducted with providers regarding PROs utility and their integration into care, and coded interview transcripts into thematic areas. Providers described how PROs helped prioritize topics for discussion during the visit, and facilitated their identification and ability to address sensitive issues, particularly depression/suicidality, sexual behavior, and intimate partner violence. PROs further facilitated comprehensive identification of other issues and concerns which led to an additional but manageable workflow impact, regarded as a valuable tradeoff. Integration of PROs into workflows met with initial challenges that were easily resolved. Providers found PROs with results delivery prior to patient appointments valuable for routine HIV care and feasible for integration into clinic workflow.

Keywords

Physician-patient relations., Patient-reported outcomes


 INTRODUCTION




 



With longer life expectancy afforded by advances in antiretroviral therapy (ART) in recent decades, people living with  HIV  (PWH)  now  manage  HIV as  a  chronic, treatable condition (Murphy et al., 2001). In HIV care, emphasis has shifted to managing multiple comorbidities and addressing quality of life. Providers  are  now  tasked with assessing many associated symptoms and health behaviors, typically in the context of a time-constrained appointment. Many issues, particularly health behaviors, are missed, as they are not directly observable. Such issues are most easily measured by patient report. However, eliciting full or accurate patient reports come with many challenges. In addition to operating within time constraints, these include (1) social desirability bias particularly when discussing highly sensitive or potentially embarrassing topics; (2) limited communication skills to convey symptoms or feelings; and (3) social and linguistic and/or cultural barriers (Kissinger et al., 1999; Williams et al., 2002; Narayan, 2010; Fredericksen et al., 2012). As a result, issues such as ART adherence, substance use, depression, and sexual risk behavior are often poorly identified and/or assessed by clinicians (Crane et al., 2017).



Routine, standardized collection of patient information using patient-reported measures or outcomes (PROs) offers an evidence-based solution. PROs are reports elicited directly from patients regarding their health status (Snyder et al., 2012), which can include symptoms, health behaviors, and other relevant context (e.g., living and social circumstances). In HIV care, same-day, on-site PRO collection using hand-held computer tablets with results available to providers during clinic visits in real time, has improved provider ability to detect and address depression/suicidal ideation, inadequate ART adherence, substance use, cognitive functioning, and symptoms. Integrating PROs into clinical care of patients with chronic conditions, such as cancer, rheumatoid arthritis, and HIV, has helped improve patient outcomes (Ruland et al., 2010; Cleeland et al., 2011; Basch et al., 2016, 2017), has proven useful to providers (Wolfe et al., 2003; Stover et al., 2015; Fredericksen et al., 2016), has shown to be both acceptable and useful to patients in helping them prioritize what to discuss (Wolfe et al., 2003; Stover et al., 2015; Fredericksen et al., 2016), and has improved both satisfaction with care (Wasson et al., 1999; Taenzer et al., 2000; Chen et al., 2013; Nelson et al., 2015) and patient-provider communication (Wagner et al., 1997; Taenzer et al., 2000; Brown et al., 2001; Detmar et al., 2002; Velikova et al., 2004) from the perspective of both parties. PROs have also positively impacted delivery of care, for example, in reducing emergency department utilization and hospitalization among cancer patients (Basch et al., 2016).  


Studies outside of HIV care, primarily from oncology, have shown providers find PROs useful, in areas such as identifying less-observable or infrequently-discussed symptoms, behaviors, and psychosocial issues, in addressing sensitive topics, and in helping focus the agenda for the visit (Espallargues et al., 2000; Ruland et al., 2010; Berry et al., 2011; Chen et al., 2013; Crane et al., 2017; Kjaer et al., 2018; Fredericksen, 2020). However, studies of provider perception of value in HIV care have been  limited  to  a  single  study  in  academic-based hospital and clinic settings (Fredericksen et al., 2016). In addition, there is a dearth of research regarding PRO implementation with respect to provider satisfaction. Provider perceptions of the utility and acceptability of tablet-based patient-reported outcomes (PRO) assessment and its integration into routine HIV care in two contrasting North American clinics, a hospital-based ambulatory clinic in Toronto, Canada, and a community-based clinic in rural Fort Pierce, FL, US were assessed.


 



 



 METHODS




 


Provider interviews were part of a broader evaluation project, the PROgress study, the goal of which was to understand the impact of integrating self-administered touch screen electronic PROs into clinical HIV care in two North American HIV care outpatient clinics: St. Michaels Hospital (SMH), in Toronto, Canada, and Midway Specialty Care Center (MSCC) in Ft. Pierce, FL, US, between August 2018 and July 2020. A full description of the PROgress study is available at: https://progresshivcare.org/.


PROs were integrated into clinical care in each clinic. PWH self-administered a ~10 min PRO assessment of several clinical domains on-site immediately prior to their routine care visit. Study coordinators delivered paper-based feedback to providers immediately after completion which succinctly summarized the results, which providers reviewed just prior to meeting with the patient. Pager alerts informed providers in real-time when patients indicated frequent suicidal ideation and/or intimate partner violence in the PRO.


After PRO results had been routinely delivered to providers for at least 1 month, we conducted 1:1 semi-structured interviews with providers about integration into workflow and practice, and perceptions of usefulness to practice and patient care.


Sites


Study sites were selected based on (1) their expressed interest in exploring PRO implementation in their practice, (2) having a substantial number of providers to experience the intervention, (3) substantial patient caseloads per provider, and (4) patient demographic, clinical, and geographic diversity. SMH in Toronto, Ontario, is an urban outpatient hospital-based clinic serving ~1800 to 2000 PLWH; 70% are men, with a high proportion of men who have sex with men (MSM). MSCC in rural Ft. Pierce, FL, serves ~1500 patients, with a high rate of uninsured and impoverished patients, 60% of whom are women. More than half of MSCC patients are ethnic minorities.


PRO assessment


PRO measures were selected based on brevity, validity, and potential to inform provider decision-making during the visit, building on lessons learned from PRO integration in the Centers for AIDS Research Network of Integrated Clinical Systems cohort5. Each clinics leadership, which included site providers, were availed of common PRO measures and solicited for discussion of site-specific PRO needs. At both sites, the PRO assessment included: mental health [depression/suicidal ideation (PHQ-9) (Spitzer et al., 1999; Kroenke et al., 2001), anxiety (single item from HIV symptom index) (Justice et al., 2001)]; health behaviors (antiretroviral adherence  (Simoni  et  al.,  2006;  Lu  et  al., 2008), substance use  (ASSIST, AUDIT-C) (Bush et al., 1998, 2002; Bradley et al., 2003; Newcombe et al., 2005), nicotine use (Kiechl et al., 2002; Nance, 2017), sexual risk behavior (Fredericksen et al., 2018); circumstantial factors [housing status (Whitney, 2020), intimate partner violence (Fitzsimmons et al., 2019)], and other forms of screening [nutrition (Canadian Nutrition Screening Tool) (Laporte et al., 2015), attitudes toward medications (2 items from HATQOL) (Holmes and Shea, 1998), sexual orientation, gender identity]. For all measures, questions had multiple choice response options; the HATQOL used a Likert scale. An exception was a question which used a visual analogue scale for measuring self-reported antiretroviral adherence; for this, patients estimated the percentage of medication taken in the past month.


Measures unique to individual sites included a measure at SMH querying citizenship status, as well as a measure querying sex practices under the influence of illicit drugs. MSCC included a review of symptoms index. The number of questions included in the assessments ranged from 65 to 101 at MSCC and from 51 to 100 at SMH depending on skip logic, which was programmed to minimize patient response burden. A small number of questions were presented to the patient on the screen at a time with large touch-sensitive buttons next to each response option. After responding to the final question on the page, a new screen with subsequent questions automatically appears. Patients navigate forward or backward in the assessment using arrow icons at screen base, allowing them to skip a section without answering by using the forward button, or to change their response from an earlier screen. At the bottom of the screen, a progress bar indicates the proportion of the PRO assessment that has been completed. Staff may also view this bar from their own work stations, and remotely monitor patient progress.


Results delivery to providers


At each site, a designated project Research Coordinator delivered the PRO results to providers. The PRO results were shown on a 1-page physical print-out which listed health domains in order of clinical relevance/urgency that had been previously agreed upon by each clinics providers and clinic leadership. Health domains were listed on each side of the print-out.


Interview guide development


Two seasoned qualitative researchers with expertise in evaluating patient perceptions of PRO data collection in HIV care developed a semi-structured interview guide in conjunction with team members with expertise in health evaluation research and clinical HIV care. Areas of inquiry were informed by previous study findings with providers on this topic (Fredericksen et al., 2016), and included ease of integration into clinic flow and practice, perceived usefulness to their practice, impact on workflow and time, impact on patient-provider communication, and impact on quality of care.


Interviews


Providers on-site were interviewed after at least one month of having routinely received PRO results, in order to ensure they had time to integrate the new procedure into their practice. Interviews were up to 60 min in length. Providers were not remunerated for interviews. Qualitative data were collected by digital recorder, and transcribed by an external transcription agency (Verbal Ink). All study activity was approved by respective ethics boards; for MSCC, approval was gained through the University of Washington Institutional Review Board, and for SMH approval was gained through the Review of Ethics Board within that institution.


Analysis


Initial analysis was performed using Dedoose qualitative software, first coding within general pre-established thematic areas based on the interview questions. These codes included: perceived impact of PROs on patient care, integration into clinic flow and practice, impact on workflow and time, and usability of the PRO results form. Within each thematic area, using an open-coding” process, two coders independently identified key concepts, reconvening to reconcile interpretive differences in coding schematics toward the goal of creating a unified coding scheme for data categorization. Within the theme of impact on care”, we created the following codes:  (1) usefulness for prioritizing discussion topics with the patient; (2) usefulness for identifying topics prone to social desirability bias; (3) allows for deeper line of inquiry; (4) helps familiarize with patients; and (5) improves care. Additionally, we created themes within integration into clinic flow and practice”, including (1) initial issues, (2) impact on provider workload, (3) usability of feedback form, and (4) general recommendations. Coders then used this schematic to code remaining data, and used a memo-ing process to summarize each provider quote in order to help reference the range of types of statements at a glance.


 



 



 RESULTS




 


Of the providers interviewed (n=12), five MDs, one PA, two Pharm Ds, one nurse practitioner, one social worker, and two RNs (four at Midway, seven at SMH) were interviewed. Interview  excerpts are organized here first into two sections: (1) impact on care, and (2) integration into workflow.


Impact on care


Providers reported the use of PROs had a positive impact on care, particularly in terms of helping prioritize topics for discussion, reducing social desirability bias, legitimizing deeper lines of inquiry on sensitive topics, helping familiarize providers with patients and the scope of their needs, and improving identification of issues with subsequent increase in referrals. Responses in each of these areas are summarized:


Helps prioritize topics for discussion


Several providers commented on the usefulness of PROs for helping set the agenda for the visit:


“You have the whole picture of what needs to be addressed in that visit, what we are missing[otherwise] you wont be able to ask so many questions in such a short period of time. So to me, its a great tool” (Provider #1, SMH).


“We don't usually have that much time because our patient load is largeit's not possible to address all of those things [in the PRO], but the fact that it's all printed out for you, it's there, and wherever concerns are, it's highlighted so it can be dealt with” (Provider #2, SMH).


One provider noted the usefulness of the PRO in allowing them to skip topics that are not currently an issue for the patient:


“Its a launching pad for discussion. So instead of re-asking all these questions, my usual drill of asking about medication adherence, tolerance, side effects and all that stuffI have them look at the [PRO results summary] right away and talk about adherence, taking their medsbut if they said 100% [adherent], I wouldnt go over thatyoure not spending a lot of times asking about things that are irrelevant, because they already asked them on here” (Provider #3, SMH).


Helped identify issues prone to social desirability bias


This was particularly true of depression and suicidal ideation:


A lot of things are being picked up that, even after [the patient] being in clinic that day, had not gotten picked up beforesomebodys depression score was rated very highso clearly, things are not obviously being recognized with just our conversations” (Pharmacist #1, SMH).


“Before the PRO was happening, rarely, rarely would I engage in questions about mental healthonly like in the most obvious [situations], like you look at someone and they're clearly distraught, maybe tearful and that sort of thing. But as everyone knows, I feel you can mask that stuff very easily. People do it all the time every day. And unless prompted or asked specifically around depression and suicidal ideation, people won't admit to itthere's also a big stigma and shame around mental health issues. These are literally people that I don't think I would have flaggedthe best element of this tool is, if nothing else, it helps us engage in a topic or conversation that would be awkward otherwise” (Provider #4, SMH).


“I know Ive learned more about the patients, 'cause I've been in this clinic a long time and I know a lot of the patients a long time. And to see that some have thought of suicide, I would never think that person has gone there. So it just goes to show patient may be coming for their visit every three or four months, and you really don't know exactly what's going on in their livesso, [the PROs] helped me to know my patients better” (Provider #2, SMH).


It was also valued in the insight the PROs shed onto substance use:


“It offers opportunities to explore something that might have been glossed over a little bit. In some patients we're


seeing a lot of alcohol use and a lot of cannabis. It sometimes helps us, to have another way of asking because sometimes when we ask people directly, you get different answers. We're finding people who are smoking more cannabis than I had realized, because it's easy [in person, for a patient] to say, "Not too much." And even then I push to try to get people to quantify it, but those sorts of questions are probably better asked on the tablet here” (Provider #5, SMH).


“A lot of it will confirm things that I already know. It does help me be more sensitive to issues that I wasn't aware of. Alternatively, if I have a patient who says to me they're clean and I see on the PRO that they're not, that gives me an opportunity to intervene and link them to some care that I wouldn't have offered otherwise” (Provider #1, Midway).


Providers reported that the use of PROs reduced their own social desirability bias, allowing them to feel more comfortable following up with questions that otherwise could be perceived by patients as too personal or asked out of the bluewithout prior context:


“[The PRO] allows us to engage., 'cause I can just say, "Hey, the reason I'm here talking to you right now is that I couldn't help noticing, when you were on that iPad, that you hit this button that said that you had sort of a lot of thoughts about self-harm. I'm just curious to know what's going on with you." That is easy. I can say, "This happened, which led to this question." Right? It's a very easy flow” (Provider #4, SMH).


“Sometimes you dont want to ask. Like you know that their partner has died, and they have many other issues, and you dont want to ask….oh, by the way are you having sex again?Now you know” (Provider #2, Midway).


One provider noted that the PROs likely reduced their own unintended bias, as well, challenging their own presumptions about patients:


“I think it was nice because sometimes I don't think to ask that particular question of that particular demographic, and to have that down in black and white across the board for everybody, it was interesting to see some answers that might have surprised meI had a few patients that noted that affordable housing was difficult for them and it has never really been anything we approached, and the patient was always [adherent to ART and stable], so I never really questioned it. And to be able to say, "Hey, this patient has trouble paying their bills, their electric bill," and to be able to deploy my case management team to help link them to community resources was nice. And I was really shocked when I saw that, 'cause it's a  patient that always shows up, is always so nicely dressed and very put together. And to hear him say, "The money only stretches this far and then we have to struggle the rest of the month," it was an interesting perspective that I probably would have never seen without the [PRO]” (Provider #1, Midway).


Allows for deeper lines of inquiry


As an impartial data collection tool that reduces social desirability bias, some providers reported it allowed for deeper questioning on sensitive topics:


“If there's something that's flagged, then I'll ask them more specifically about it. If the screening question for alcohol use shows up, then I may probe a little harder than I would have. I try to remember to ask about alcohol use routinely anyway, but [having the PRO results available] might just cause me to go a little deeper. If there's a relationship issue, I'll ask more about that as well” (Provider #5, SMH).


“I look at [the PRO results] with them and I discuss it with them. It'sthis third thing is in the room with you. Yesterday, I had this patient who has two partners and so, "How did that go? Are you back with your partner?" It creates more diverse questions by [reviewing the results] with the patient. "Why are you depressed? What's going on here? Is this new?" (Provider #3, Midway)


One provider noted that the use of PROs gives them more to discuss with their relatively stable patients:


“For me, the patient concerns are the most important part of their visit. The T-cell environment, all this stuff, is minor now. The [antiretroviral] medication is what it is. [The PRO] creates another discussion point for taking care of patients, apart from the routine. How many times can you tell a patient for ten years their T-cells and viral load counts are fine? You want to tell them about something else” (Provider #3, Midway).


Another provider found that PROs, in their ability to illustrate symptoms over multiple time points, helped open more evidence-based inquiry about changes over time, in this case, for depression:


“Some [patients] depression scores would vary. Maybe somebody was a 9 (on the PHQ-9) and it was mild and then [at a later visit] was 15 and it was moderate. So I would be, "What's going on with life? How are your depression symptoms? Has anything happened that might make this worse?" That would be the biggest difference, that I noticed” (Provider #1, Midway).


Helps familiarize providers with patients and a fuller scope of their needs


Providers   reported   that   the   PROs   were   useful   for orienting themselves to the needs of patients with whom they were less familiar, such as patients new to care:


“Well, since I'm getting to know them, it gives me an idea of themis this a patient that's engaging in risky sex? Should I have more of a conversation regarding that, as opposed to me just going to the room and addressing the HIV? Maybe we need to sit down and have a conversation about other sexually transmitted infections that they could get. And of course, [the PROs] help us address mental health that most patients don't want to talk about” (Provider #4, Midway).


This was also true of patients with whom providers had assumed they knew well:


“Honestly, I was surprised at how useful I was going to find [the PROs]I figured my patients and I have a pretty good relationshiptheyre gonna tell me all these things, [but I was] hearing things I hadnt heard before. Youre almost glancing behind a curtain” (Provider #1, Midway).


Other members of the care team outside of the main provider found the information provided by PROs helped put their conversations with them into better context. For example, one pharmacist echoed other such staff in noting the PROs usefulness for helping them understand issues pertinent to ART adherence:


“Normally I would ask questions about pharmacy-related issues, like adherence, drug coverage, drug access, thinks like that...through the [PROs], these questions are already answered, and there's other social determinants of health that might if someone scores high on depression or suicidality, these are issues that might be barriers to adherence to therapy, so they might be red flags for me” (Provider #6, SMH).


Provides better care/referrals/outcomes


Providers concurred that PROs improve patient care, most notably by identifying previously hidden needs and allowing them to address them:


“[Patients] now feel more safe that they have someone they could speak to if, in the future, that [partner violence] happenssince we've already had this conversation, they know that there's someone in the hospital they can talk to about it if it repeats” (Provider #4, SMH).


“[In the case of suicidal ideation], I have a discussion with the patient about it, and we unpack it just like we would unpack any other issue the patient brings up, and figure out what needs to be done, and give them a chance to talkwe might refer to a mental health provider, for example” (Provider #7, SMH).


Others noted benefits in terms of patientsrelationship with care:


“It makes patients feel as if theyre more involved in their care and thats important” (Provider #3, Midway).


“I guess maybe the other related impact it has on patient care isit signals to patients that were thinking holistically about their care” (Provider #7, SMH).


Integration into workflow


Integration of PROs into workflow of each clinic was characterized by an initial brief period of adjustment, typically prolonged over a few days.


“It's always frustrating, I think, working in a team dynamic that requires so many moving parts, but we all adapted. At the end, there was minimal complaint. We worked it out” (Provider #1, Midway).


“Overall thumbs up. However, there are certainly things that could have been better I suppose is one way of putting it, or one could also just say as expected theres lots of things that needed to be fixed along the way” (Provider #7, SMH).


Initial issues included the need to purchase an additional Wi-Fi router in order for iPad tablet reception to extend to distal exam rooms, and the need for a dedicated printer exclusively for PRO printing so that delivery of PRO results were not held up by other print jobs. The most concerning pre-implementation issue for providers and staff, however, was the potential impact on clinic flow. 


“At first, it was very cumbersome. I'd have a 10:00 appointment that I wouldn't see until 10:20 because it took that long to finish” (Provider #1, Midway).


“If a patient is delayed because they're doing the [PRO], then it delays every other patient that comes after them. And some people are much faster on the tablet than other people” (Provider #5, SMH).


Differing speeds of completing PROs, relative to arrival time and provider readiness to see the patient, caused pressure:


“Ideally, itd be nice if yes, the patient did [the PRO], the nursing goes in, then pharmacy, plus or minus social worker and then the physician to wrap everything up with all of that. But I think because everybody wants to get in and out I think that gets all chopped up. And so I think thats … the bottle neck is at the very beginning where theyre trying to get them to complete a [PRO] first. I don't know how to fix that unless you get the ball rolling earlier and   have   [PROs]   already  completed  and  everything completely donetheres just thaturgency, of course, to get it in to see the patient” (Provider #8, SMH).


“In an ideal situation all these get done prior to myself as the main provider, a physician that walks into the room. Thats not always possible, and my practice is in the clinic when there is time, Ill wait. But when theres three or four other patients, when its busy, I cant wait. I cant hold everything up and twiddle my thumbs waiting for this thing to be done” (Provider #7, SMH).


Initial negative impact on clinic flow required protocol decision-making in both clinics in order to avoid such delays. This included starting patients on the PRO as soon as possible if they had checked in early, creating a cut-off time for stopping the PRO if patients were not done, and having staff remotely monitor speed of progress:


[Patients] usually have some wait time in the room…while they're waiting, they're able to do [the PROs]. It keeps them busy (Provider #6, SMH).


“Once we streamlined the process, and I kind of tracked the progress [of patients completing the PRO] on the website, it slowed things up a little bit lessif [after] 15 minutes and [the patient] wasnt done, the research coordinator would end the session and print out and deliver the results], so that way we could move onthat was helpful” (Provider #1, Midway).


It also meant as a clinic proactively deciding which patients should not begin the PRO, such as not administering to patients who are late for their appointment, patients with known low literacy, patients who are extremely ill (e.g., nausea) and/or too disabled to self-administer a questionnaire. At times, the decision to administer the PRO was made on a case-by-case basis upon arrival, after having started the PRO.


“I think I pulled [a patient] off [of the PRO} because I thought the time would be better spent with me, and it was probably a suicidal ideation one. Where we already knew from the previous visit it was positive. Or, the patient had a fever that day or the patient complained of chest pain. Im like, OK, no, theres no PRO [for that patient] today. We need to move along” (Provider #2, Midway).


Other logistics required resolution during the integration phase, including whether and how other members of the care team outside of the main provider are alerted to particular results (e.g., mental health issues for the social worker, or adherence issues for the pharmacist). One clinician commented on the latter issue:


“I know the pharmacist will look at the drugs and tick off that he or  she  has seen them. The social worker may be


doing something different. Im not sure what the nurses are doing. Were hoping after this week when weve all thought about it and been forced to think about it, to meet and see how we could use this. Should everybody look at everything?...when I see the pharmacist has looked at the meds [adherence reporting] and theyre happy, Im less concerned about that piece. I dont suspect that everybody looks at everything, but I dont think weve decided whos looking at what and how they deal with the information other than the alerts we sorted out” (Provider #3, SMH).


This issue was easily resolved:


“So what we've decided is, whoever gets that information looks and then if you need to interact and let, like I said, a dietician know or social workerI initial that I've seen it and then if there's anybody needs to be informed as well, I let them know” (Provider #1, SMH)


Both sites noted that having a dedicated staff member to act as a ‘champion’ in ensuring the integration goes smoothly was deemed essential.


“[The study coordinator] is very central to this, a very important part of this whole thing. I bet you those will be the growing pains once someone is not on site to help us, and it's going to be a little trickier, I think. Then we're gonna have to figure out more systems work around that” (Provider #4, SMH).


“I don't think it can run without a dedicated person...you can give it to [Medical Assistants} in time but you need to be trainedonce it gets in, it's seamless” (Provider #3, Midway).


Impact on provider workload


Upon delivery of the PRO, providers reported that the impact on their time and workload was dependent on factors such as the level of familiarity with the patient, the number and depth of concerns identified the providers professional style, and the availability of referral resources in-house. In general, where discussions with patients regarding PROs had an additional time impact on workflow, this was regarded as a manageable impact; the identification of issues and comprehensiveness of care were a valuable and acceptable trade-off.


“PROs add time, and on it add time because you know, we now have to explore an issue that was not otherwise going to have been discussed necessarily during that particular visit” (Provider #6, SMH).


“We talk about more stuff. But at least it is helping with depression. It shortens [having to ask about] that one piece…[but]   it  elongates  the  appointments. I  mean,  it depends. If that PRO is pretty clean [not many symptoms or high risk behaviors] it doesnt. But if were talking about three things because of the PRO, yeah. It elongates it [but] even though its frustrating to me, Im like okay, Ill get useful information” (Provider #2, Midway).


“I really feel prescreening helps and I feel it makes the visit go fastwhen you have already found out something that you might not have focused on” (Provider #4, Midway).


“It ebbs and flowsyou never know in what clinic how many people will have something that highlights something I need to see them aboutso I mean, yeah, it's a slight impact, if any, in terms of workload for me, but it must be said that [depression, suicidal ideation, and partner violence] are 100 percent a priority anyway, so even if I was seeing other people and I was totally crushed in terms of time and that, it's like, whatever, I'll drop it all to see that person immediately. Right? Its obviously a priority” (Provider #4, SMH).


Usability of PRO results form


Since providers interviewed helped design the PRO feedback form to their specifications prior to project launch, satisfaction with the form, and perceived usefulness as a clinical tool was uniformly high. However, providers had additional requests for modifications for ease of use. For SMH, a common request was to group domains together:


“The format needs to be restructured, I think for ease of readability. So I thinkwhat weve proposed makes more senseour plan is to group topics together going forward: the pharmacist is going to make sure he or she looks at the issues around adherence, and medication payment and things, all the time. [Each provider will] go to that one place [on the form]. The social worker will look at the housing and finances piece. The physician will look at the more medical pieces. Its easier if theyre grouped together like that, and its more appealing and your eyes can follow it” (Provider #3, SMH).


Other requests concerned modification of how scoring and interpretation of results are presented:


“I think the areas that I have to pause and read through are the smoking section or the alcohol section, a little bit, because there's no real key as to what each score is. It just says at risk or not at risk. For the smoking, I have to see, yes, former smoker, then how many smokes per day. Are they currently smoking? It's broken down in a way that I have to read over” (Provider #1, Midway).


For the IPV question, one provider considered the pros and cons of raw scores vs. text-based answers on the feedback form (the IPV  item  displays results in the latter format):


“Scales 0 to 5. I always find that very interesting, 0, 1, no harm, medium harm, to grade of these, meaning how trapped do you feel. [But the current response format is] just open ended, "Yes, I feel trapped." What does that mean? But again, that also now gives you the opportunity to go deeper into those questions by just having a yes. There's two sides to that. If you give a grade for it, it's good for data analytics, but this [text interpretation] works well for providers” (Provider #3, Midway).


Provider recommendations: Looking ahead


Providers recommended several actions that would improve the usefulness of PROs in their practice. The most common of these recommendations was to have the PRO results auto-populate the electronic medical record. Providers also expressed a desire for input on an ongoing basis as to how frequently certain PRO measures are administered, with some interest in tailoring the PRO for individual patients or groups of patients. There was interest among individual providers on specific topics: for example, post-traumatic stress disorder, cardiovascular health behaviors, gastro-intestinal symptoms. Providers also suggested administering PROs in additional commonly-spoken languages within each clinic, including French (SMH), Haitian Creole (Midway), and Spanish (both sites). Finally, providers recommended that the clinic designate responsibility to a specific staff member to oversee the administration of PROs including collection and maintenance of tablets, management of patient flow, and ensuring results are delivered to providers.


 



 



 DISCUSSION




 


Providers reported PROs allowed for more comprehensive identification of issues and concerns, and were particularly valued for identifying and addressing sensitive issues that would likely have been missed, particularly those prone to social desirability bias, such as depression/suicidality, substance use, sexual behavior, and intimate partner violence. In addition, PROs also helped prioritize areas to discuss during the visit, highlighting the most pertinent patient concerns and allowing clinicians to bypass areas of inquiry that had been reported as stable. Impact on time and workflow varied between providers; all providers interviewed found the impact on workflow increased, yet was manageable, echoing prior work which found minimal impact on visit length or consultation time (Fredericksen et al. 2020). Gaining additional information, particularly in the areas of suicidal ideation and partner violence, was regarded as a valuable  trade-off.  These  findings  echo similar   studies which found similar impacts on social desirability bias (Detmar et al., 2002; Velikova et al., 2004; Mark et al., 2008a; Fortner et al. 2008a; Fredericksen et al., 2016; Tufano et al. 2016; Sharma et al., 2016) agenda-focusing (Mark et al., 2008b; Johnson et al. 2008b; Fredericksen et al., 2016; Tufano et al., 2016) and overall comprehensiveness (Velikova et al., 2004; Booth et al. 2004).


Initial integration of PROs into each clinic yielded workflow challenges, such as the need to account for timing of patient arrival relative to their providers readiness to see the patient, and patient ability to complete and speed of completion of the PROs. In both clinics, these challenges were met with the creation of or adjustments to protocols, including identification of patients that should not complete PROs (such as those who are severely symptomatic, e.g., nausea), identifying whether an appropriate block of time exists for the patient to complete the PROs prior to seeing the provider, and enhancing real-time communication between varying provider types, such as pharmacists and social workers in addition to the regular provider, in order to ensure all parties had reviewed the results. These initial challenges were characterized as temporary and resolvable.


Providers cautioned that the feasibility of continued use of PROs in their clinics was dependent on the role of a coordinator to oversee and manage their administration, a role which was acknowledged could possibly be integrated into other positions, such as that of medical assistant, in order to ensure minimal disruption to flow. Providers also strongly urged the need for integration of PRO results into the electronic medical record in order to avoid duplication of data entry and to streamline use. Utility of the PRO results form was contingent on ease of interpretation of data and formatting of results into topic-specific blocks, such as medication adherence and satisfaction with medications. These qualifications align with previous studies in which providers also noted that adoption of PROs in practice was dependent on minimal disruption to clinic flow, manageable patient response burden, ease of provider access to data, quantity/ presentation of data, and a focus on directly actionable domains such as depression (Fredericksen et al., 2012; Chen et al., 2013; Fredericksen et al., 2016; Barr et al., 2020). In this project, provider involvement in selection of the domains and measures of interest, as well as provision of input on the format of reported results prior to integration, was likely critical to ensuring buy-in, as has been the case in prior work (Fredericksen et al., 2012). 


As a highly treatable chronic condition, HIV care in recent decades has shifted its focus to maintaining or improving patient quality of life over the life course (Murphy et al., 2001) with a population at increased risk for multiple co-morbidities (American Foundation for AIDS Research 2021). With the median age of this population increasing, the need has intensified for providers  to  quickly  and  effectively address several co-morbidities, concerns, and health behaviors, within the time constraints of a brief visit. The complex needs of PWH in the modern era warrant a broad assessment across multiple health-related domains, with attention to domains such as substance use and depression for which PWH are at higher risk, yet which are prone to social desirability bias. Such domains may be difficult for providers to elicit or for patients to express in this context, even among well-known patients with whom rapport is well-established, as demonstrated by the study. The findings suggest that PRO collection can act as a launchpad, rather than a replacement, for otherwise challenging discussions, and act as an expeditious means of assessing multiple patient needs, allowing for provision of care that is comprehensive, tailored to individuals, and patient-driven. 


Strengths


This is among the first of studies to assess both integration and utility of PROs among health care providers.


Limitations


Providers were interviewed at their place of employment, which may have introduced bias toward reporting favorably toward PROs.


 



 



 CONCLUSION




 


Providers found PROs with results delivery prior to patient appointments both useful and acceptable for routine HIV care. Integration into care was characterized by the need to resolve issues of how PROs fit into clinic flow and was contingent on having a dedicated staff person. The value added by PROs to patient care in terms of addressing topics not otherwise likely to have been identified, particularly depression and suicidal ideation, offset additional burden on clinic flow and provider workload.




 CONFLICT OF INTERESTS




The authors have not declared any conflict of interests.




 ACKNOWLEDGMENTS




 


This study was funded by ViiV Healthcare PO# 3001390801. The authors are thankful to the patients, providers, and research staff from the two PROgress study clinics, St. Michaels Hospital in Toronto, Ontario, Canada, and the Midway  Specialty  Care Center  in  Fort Pierce, Florida, USA. Additionally, they thank the PROgress study steering committee, comprising patients living with HIV, HIV care providers, and HIV care researchers, for their oversight and guidance.



 REFERENCES


 










































































































































































































































































































































American Foundation for AIDS Research (2021). HIV and comorbidities.


View


 

Barr PJ, Berry SA, Gozansky WS, McQuillan DB, Ross C, Carmichael D, Austin AM, Satterlund TD, Schifferdecker KE, Council L, Dannenberg MD, Wampler AT, Nelson EC, Skinner J (2020). No date for the PROM: the association between patient-reported health events and clinical coding in primary care. Journal of Patient Reported Outcomes 4(1):17.
Crossref


 
 

Basch E, Deal AM, Dueck AC, Scher HI, Kris MG, Hudis C, Schrag D (2017). Overall Survival Results of a Trial Assessing Patient-Reported Outcomes for Symptom Monitoring During Routine Cancer Treatment. Journal of American Medicine Association 318(2):197-198.
Crossref


 
 

Basch E, Deal AM, Kris MG, Scher HI, Hudis CA, Sabbatini P, Rogak L, Bennett AV, Dueck AC, Atkinson TM, Chou JF, Dulko D, Sit L, Barz A, Novotny P, Fruscione M, Sloan JA Schrag D (2016). Symptom Monitoring With Patient-Reported Outcomes During Routine Cancer Treatment: A Randomized Controlled Trial. Journal of Clinical Oncology 34(6):557-565.
Crossref


 
 

Berry DL, Blumenstein BA, Halpenny B, Wolpin S, Fann JR, Austin-Seymour M, Bush N, Karras BT, Lober WB, McCorkle R (2011). Enhancing patient-provider communication with the electronic self-report assessment for cancer: a randomized trial. Journal of Clinical Oncology 29(8):1029-1035.
Crossref


 
 

Bradley KA, Bush KR, Epler AJ, Dobie DJ, Davis TM, Sporleder JL, Maynard C, Burman ML, Kivlahan DR (2003). Two brief alcohol-screening tests from the Alcohol Use Disorders Identification Test (AUDIT): validation in a female Veterans Affairs patient population. Archives of Internal Medicine 163(7):821-829.
Crossref


 
 

Brown RF, Butow PN, Dunn SM, Tattersall MH (2001). Promoting patient participation and shortening cancer consultations:a randomised trial. British Journal of Cancer 85(9):1273-1279.
Crossref


 
 

Bush K, Kivlahan DR, McDonell MB, Fihn SD, Bradley KA (1998). The AUDIT alcohol consumption questions (AUDIT-C): an effective brief screening test for problem drinking. Ambulatory Care Quality Improvement Project (ACQUIP). Alcohol Use Disorders Identification Test. Archives of Internal Medicine 158(16):1789-1795.
Crossref


 
 

Chen J, Ou L, Hollis SJ (2013). A systematic review of the impact of routine collection of patient reported outcome measures on patients, providers and health organizations in an oncologic setting. BMC Health Services Research 13:211.
Crossref


 
 

Cleeland CS, Wang XS, Shi Q, Mendoza TR, Wright SL, Berry MD, Malveaux D, Shah PK, Gning I, Hofstetter WL, Putnam JB, Vaporciyan AA (2011). Automated symptom alerts reduce postoperative symptom severity after cancer surgery: a randomized controlled clinical trial. Journal of Clinical Oncology 29(8):994-1000.
Crossref


 
 

Crane HM, Crane PK, Tufano JT, Ralston JD, Wilson IB, Brown TD, Davis TE, Smith LF, Lober WB, McReynolds J, Dhanireddy S, Harrington RD, Rodriguez CV, Nance RM, Delaney JA, Safren SA, Kitahata MM, Fredericksen RJ (2017). HIV provider documentation and actions following patient reports of at-risk behaviors and conditions when identified by a web-based point-of-care assessment. AIDS Behavior 21(11):3111-3121.
Crossref


 
 

Detmar SB, Muller MJ, Schornagel JH, Wever LD, Aaronson NK (2002). Health-related quality-of-life assessments and patient-physician communication: a randomized controlled trial. Journal of American Medicine Association 288(23):3027-3034.
Crossref


 
 

Espallargues M, Valderas JM, Alonso J (2000). Provision of feedback on perceived health status to health care professionals: a systematic review of its impact. Medical Care 38(2):175-186.
Crossref


 
 

Fitzsimmons E, Loo S, Dougherty S, Mathews W, Mugavero M, Mayer K, Potter J, O'Cleirigh C, Crane P, Crane HM, Fredericksen RJ (2019). Development and Content Validation of the IPV-4, a Brief


 
 

Patient-Reported Measure of Intimate Partner Violence for Use in HIV Care. International Society of Quality of Life Research Conference.San Diego, California.


 
 

Fredericksen RJ, Crane P, Tufano J, Ralston J, Schmidt S, Brown TL, Harrington DR, Dhanireddy S, Stone T, Lober W, Kitahata M, Crane HM (2012). Integrating a web-based patient assessment into primary care for HIV-infected adults. Journal of AIDS and HIV Research 4(2):47-55.
Crossref


 
 

Fredericksen RJ, Crane HM, Lober W, Fitzsimmons E, Short D (2020). PROgress evidence review and summary: impact of administering patient-reported measures and outcomes (PROs) within HIV routine care.


 
 

Fredericksen RJ, Mayer KH, Gibbons LE, Edwards TC, Yang FM, Walcott M, Brown S, Dant L, Loo S, Gutierrez C, Paez E, Fitzsimmons E, Wu AW, Mugavero MJ, Mathews WC, Lober WB, Kitahata MM, Patrick DL, Crane PK, Crane HM (2018). Development and Content Validation of a Patient-Reported Sexual Risk Measure for Use in Primary Care. Journal of general internal medicine, 33(10):1661-1668.
Crossref


 
 

Fredericksen RJ, Tufano J, Ralston J, McReynolds J, Stewart M, Lober WB, Mayer KH, Mathews WC, Mugavero MJ, Crane PK, Crane HM (2016). Provider perceptions of the value of same-day, electronic patient-reported measures for use in clinical HIV care. AIDS Care 28(11):1428-1433.
Crossref


 
 

Holmes WC, Shea JA (1998). A new HIV/AIDS-targeted quality of life (HAT-QoL) instrument: development, reliability, and validity. Medical Care 36(2):138-154.
Crossref


 
 

Justice AC, Holmes W, Gifford AL, Rabeneck L, Zackin R, Sinclair G, Weissman S, Neidig J, Marcus C, Chesney M, Cohn SE, Wu AW (2001). Development and validation of a self-completed HIV symptom index. Journal of Clinical Epidemiology 54(Suppl)1:S77-90.
Crossref


 
 

Kiechl S, Werner P, Egger G, Oberhollenzer F, Mayr M, Xu Q, Poewe W, Willeit J (2002). Active and passive smoking, chronic infections, and the risk of carotid atherosclerosis: prospective results from the Bruneck Study. Stroke 33(9):2170-2176.
Crossref


 
 

Kissinger P, Rice J, Farley T, Trim S, Jewitt K, Margavio V, Martin DH (1999). Application of computer-assisted interviews to sexual behavior research. American Journal of Epidemiology 149(10):950-954.
Crossref


 
 

Kjaer A, Rasmussen TA, Hjollund NH, Rodkjaer LO, Storgaard M (2018). Patient-reported outcomes in daily clinical practise in HIV outpatient care. International Journal of Infectious Disease 69:108-114.
Crossref


 
 

Kroenke K, Spitzer RL, Williams JB (2001). The PHQ-9: validity of a brief depression severity measure. Journal of General Internal Medicine 16(9):606-613.
Crossref


 
 

Laporte M, Keller HH, Payette H, Allard JP, Duerksen DR, Bernier P, Jeejeebhoy K, Gramlich L, Davidson B, Vesnaver E, Teterina A (2015). Validity and reliability of the new Canadian Nutrition Screening Tool in the 'real-world' hospital setting. European Journal of Clinical Nutrition 69(5):558-564.
Crossref


 
 

Lu M, Safren SA, Skolnik PR, Rogers WH, Coady W, Hardy H, Wilson IB (2008). Optimal recall period and response task for self-reported HIV medication adherence. AIDS Behavior 12(1):86-94.
Crossref


 
 

Mark TL, Fortner B, Johnson G (2008a). Evaluation of a tablet PC technology to screen and educate oncology patients. Support Care Cancer 16(4): 371-378.
Crossref


 
 

Mark TL, Johnson G, Fortner B, Ryan K (2008b). The benefits and challenges of using computer-assisted symptom assessments in oncology clinics: results of a qualitative assessment. Technology in Cancer Research and Treatment 7(5):401-406.
Crossref


 
 

Murphy EL, Collier AC, Kalish LA, Assmann SF, Para MF, Flanigan TP, Kumar PN, Mintz L, Wallach FR, Nemo GJ, I. Viral Activation Transfusion Study (2001). Highly active antiretroviral therapy decreases mortality and morbidity in patients with advanced HIV disease. Annual Internal Medicine 135(1):17-26.
Crossref


 
 

Nance RD, Fredericksen RJ, Cropsey K, Chander G, Mugavero M, Christopoulos K, Geng E, Mathews W, Hahn A, Mayer K, O'Cleirigh C, Eron J, Saag M, Kitahata MM, Crane HM (2017). E-cigarette use among persons living with HIV. International AIDS Society Meeting. Paris, France.


 
 

Narayan MC (2010). Culture's effects on pain assessment and management. American Journal of Nursing 110(4):38-47.
Crossref


 
 

Nelson EC, Eftimovska E, Lind C, Hager A, Wasson JH, Lindblad S (2015). Patient reported outcome measures in practice. Biomedical Journal 350:g7818.
Crossref


 
 

Newcombe DA, Humeniuk RE, Ali R (2005). Validation of the World Health Organization Alcohol, Smoking and Substance Involvement Screening Test (ASSIST): report of results from the Australian site. Drug and Alcohol Review 24(3):217-226.
Crossref


 
 

Ruland CM, Holte HH, Roislien J, Heaven C, Hamilton GA, Kristiansen J, Sandbaek H, Kvaloy SO, Hasund L, Ellison MC (2010). Effects of a computer-supported interactive tailored patient assessment tool on patient care, symptom distress, and patients' need for symptom management support: a randomized clinical trial. Journal of the American Medicine Informatics Association 17(4):403-410.
Crossref


 
 

Sharma P, Dunn RL, Wei JT, Montie JE, Gilbert SM (2016). Evaluation of point-of-care PRO assessment in clinic settings: integration, parallel-forms reliability, and patient acceptability of electronic QOL measures during clinic visits. Quality of Life Research 25(3):575-583.
Crossref


 
 

Simoni JM, Kurth AE, Pearson CR, Pantalone DW, Merrill JO, Frick PA (2006). Self-report measures of antiretroviral therapy adherence: A review with recommendations for HIV research and clinical management. AIDS Behavior 10(3):227-245.
Crossref


 
 

Snyder CF, Aaronson NK, Choucair AK, Elliott TE, Greenhalgh J, Halyard MY, Hess R, Miller DM, Reeve BB, Santana M (2012). Implementing patient-reported outcomes assessment in clinical practice: a review of the options and considerations. Quality of Life Research 21(8):1305-1314.
Crossref


 
 

Spitzer RL, Kroenke K, Williams JB (1999). Validation and utility of a self-report version of PRIME-MD: the PHQ primary care study. Primary Care Evaluation of Mental Disorders. Patient Health Questionnaire. Journal of American Medicine Association 282(18):1737-1744.
Crossref


 
 

Stover A, Irwin DE, Chen RC, Chera BS, Mayer DK, Muss HB, Rosenstein DL, Shea TC, Wood WA, Lyons JC, Reeve BB (2015). Integrating patient-reported outcome measures into routine cancer care: cancer patients' and clinicians' perceptions of acceptability and value. EGEMS (Wash DC) 3(1):1169.
Crossref


 
 

Taenzer P, Bultz BD, Carlson LE, Speca M, DeGagne T, Olson K, Doll R, Rosberger Z (2000). Impact of computerized quality of life screening on physician behaviour and patient satisfaction in lung cancer outpatients. Psycho-Oncology 9(3):203-213.
Crossref


 
 

Velikova G, Booth L, Smith AB, Brown PM, Lynch P, Brown JM, Selby PJ (2004). Measuring quality of life in routine oncology practice improves communication and patient well-being: a randomized controlled trial. Journal of Clinical Oncology 22(4):714-724.
Crossref


 
 

Wagner AK, BL Ehrenberg, Tran TA, Bungay KM, Cynn DL, Rogers WH (1997). Patient-based health status measurement in clinical practice: a study of its impact on epilepsy patients' care. Quality of Life Research 6(4):329-341.
Crossref


 
 

Wasson JH, Stukel TA, Weiss JE, Hays RD, Jette AM, Nelson EC (1999). A randomized trial of the use of patient self-assessment data to improve community practices. Effective Clinical Practice 2(1):1-10.


 
 

Whitney BM, Fitzsimmons EF, Jackson MK, Brown SM, Nguyen J, Nance RN, Ruderman S, Delaney JAC, Crane HM, Fredericksen RJ (2020). Patient perceptions and understanding of a housing status measure for use in HIV care. 27th Annual Conference of International Society of Quality of Life. online.


 
 

Williams MV, Davis T, Parker RM, Weiss BD (2002). The role of health literacy in patient-physician communication. Family Medicine 34(5):383-389.


 
 

Wolfe F, Pincus T, Thompson AK, Doyle J (2003). The assessment of rheumatoid arthritis and the acceptability of self-report questionnaires in clinical practice. Arthritis and Rheumatology 49(1):59-63.
Crossref