In voicemails, spam folders and text messages across Vermont, an artificial intelligence tool is filing through years of pharmaceutical records, incoherently listing drugs and requesting the wrong refills on sensitive medication.
And customers often have no idea they agreed to it.
“He calls you all the time, and he doesn’t say what the drug is,” said Kathy Callaghan, a Vermont customer of Kinney Drugs, referring to a new AI tool recently put to use by the Vermont and New York pharmacy chain.
Nearly every day, she recalled, her phone rings, and she hears an incomprehensive mutter of a drug from the AI voice. Confused, she approves the refill order, thinking the request is legitimate and she really is out of a prescription.
So far, it hasn’t been, she said.
“I’ve got four bottles of something that I take twice a week sitting here,” Callaghan said.
Earlier this summer, amid a nationwide wave of AI technology entering healthcare systems, Kinney Drugs implemented a new AI assistant for prescription refill services, lauding the tool as a way to improve communication and efficiency. But customers say it’s doing the opposite.
According to nearly a dozen Kinney Drugs customers across its stores in Vermont and New York who spoke to VTDigger, the AI rollout has coincided with a range of problems. Delays have grown between physicians ordering drugs and the pharmacy processing them. The system has been unable to locate longtime customer accounts, has ordered incorrect dosages, and has failed to notify customers when refills were ready for pickup. And these inaccuracies are raising privacy concerns amongst customers.
Their mounting frustrations towards the AI is leading some customers to leave the pharmacy entirely, they said, and forcing others to continue using a system they distrust due to a lack of pharmaceutical options in rural areas.
As AI pops up in healthcare systems across the country, several lawyers told VTDigger that the technology is outpacing regulation, leaving consumer protections thin.
“There’s a delay in laws that are coming onto the books, so we’ve not seen a detailed picture of how companies are going to comply or how states are going to enforce the laws,” said Cody Venzke, a senior staff attorney at the ACLU.
In Vermont, a new data privacy bill, S.71, signed by Gov. Phil Scott on June 16, takes effect in two years. That delay creates a gap in compliance to a bill that privacy advocates say is already too weak to protect consumers.
“Even if it were in effect, I’m not sure that it would necessarily cover the situation,” said Ryan Kriger, a Vermont lawyer and resident who testified on the bill.
‘No value, no benefit’
Kinney Drugs implemented the AI virtual assistant for patient communication and refill services in May. Soon after, its customers began to experience problems, they said.
“My first impression was it was just more complicated and cumbersome than the previous recording and service,” said Matt Penney, who’s been receiving medication at a Kinney Drugs in Vermont for around two years.
Previously, to refill a prescription at Kinney Drugs, a customer would call a prompt-based, touch-tone, phone system that would give them the opportunity to navigate a keypad to reach a pharmacist. Customers who spoke to VTDigger said this old system worked smoothly, and they had no problems.
Not so much with the new system, they said.
“It’s just something that wastes time and delays customer service. As far as I’m concerned, it has absolutely no value, no benefit to the customer,” Vermont Kinney Drugs customer Dave Bellini said.
The AI assistant, which Kinney Drugs named Burt, after its founder, was built by third-party AI pharmaceutical company, Synerio. In the pharmacy’s terms, Kinney Drugs states that “Burt may not be accurate, complete or up-to-date and may be misleading or contain errors and omissions.”
Any customer who provides their telephone number to Kinney Drugs automatically consents to interacting with the tool, though they may later opt out, according to the pharmacy’s terms. While some customers recalled signing onto a new refill system earlier this summer, others said they didn’t realize the company implemented the new tool until they called for a refill and heard the AI voice.
“Burt demands that you talk to him,” said Maria Aveni, a Vermont Kinney Drugs customer who has worked in healthcare as an anesthesiologist. “There’s no option to use a keypad to enter what you want to, which has been the case with other telephone interactions I’ve had. So it’s more of a talk to Burt or don’t get your drugs.”
According to Aveni and other customers who spoke to VTDigger, Kinney Drugs did not communicate about the tool before implementation.
When this month Aveni experienced a delay in receiving medication that her physician had ordered, she felt grateful it was not a lifesaving drug. Nevertheless, she was alarmed. Since then, she said she only goes in person to her pharmacy for refills in order to avoid AI interactions.
But for others, who live in rural areas and have limited access to transportation, going in person to receive refills as a way around the AI is impossible.
A Kinney Drugs representative who did not want to be identified by name, wrote in a statement to VTDigger that the company is “aware that some patients have experienced frustration with Burt, especially during rollout,” and that it is reviewing each concern and attempting to reach each patient.
While customers said they were confused over why Kinney Drugs switched to the AI, the company cited a clear reason: to eliminate repetitive tasks and free up time for its pharmacists.
Its leaders believe the AI has helped in achieving this goal, with one director saying that “it’s definitely doing the trick and handling those phone calls in such a nice manner that it’s reducing the calls to our pharmacy.”
But customers’ experience with the tool has led to an oftentimes opposite effect. As they encounter problems with the AI, many are turning to pharmacists more for help. In Vermont, pharmacists are already overburdened due to a chronic worker shortage.
“It’s slowing down productivity of an already stressed staff, and because that staff has treated me for a long time, and I feel loyalty, it isn’t just a financial concern for me, it’s an inhumane way to treat the staff,” Aveni said.
‘Privacy is just about out the window’
While the AI tool’s reported inaccuracies are the primary frustration for many customers, others are citing privacy concerns for leaving the chain.
“Once I went into the store and actually read the privacy policy, I was shocked, and immediately transferred my prescriptions to UVM Hospital,” Tuan Green, a former Vermont Kinney Drugs customer, said.
The pharmacy’s privacy policy, released the same month as the AI assistant, discloses, for example, that customers’ protected health information may be used in AI tools to improve the pharmacy’s operations.
“These days, privacy is just about out the window,” Stephen McArthur, a Vermont Kinney Drugs customer, said.
Beyond Kinney Drugs, customers and privacy experts have also raised concerns with Synerio, the Texas-based technology company that supplies the AI tool to the pharmacy.
Joseph Near, an associate professor of computer science at the University of Vermont, said that, privacy-wise, the AI tool is worse than the pharmacy’s previous refill system because this additional company, Synerio, now also has access to customers’ health data.
“If I just interact with my healthcare provider, I’m relying on the security that they have. If they now start interacting with many other organizations, I now have to start worrying about all the security issues with each of them,” Bradley Malin, a professor of biomedical informatics at Vanderbilt University Medical Center, said, referring to Synerio.
The Kinney Drugs representative said that the company maintains “multiple layers of security controls,” invests in “information security technologies to help protect sensitive data” and has the protocols in place to respond to potential data breaches.
AI in healthcare settings also raises a host of concerns surrounding the federal health privacy law HIPAA, several lawyers and data privacy experts said, pointing to possible data sharing of protected health information.
However, as long as the pharmacy has a business associate agreement — the contract under HIPAA that establishes that third-party vendors securely handle protected health information — the companies are contractually obliged to handle data securely. The Kinney Drugs representative declined to share the agreement, citing an inability to disclose private vendor contracts, but wrote that the company “maintains the appropriate agreements.”
Synerio did not respond to multiple requests for comment, but also says on its webpages that the tools it creates are built to be compliant with HIPAA.
While Synerio did not respond to VTDigger’s questions about customer protections, Kinney Drugs’ pharmacy management system, McKesson, acknowledged them in public filings, writing that “reliance on third-party AI tools and solutions may expose us to risks that are outside of our control, including compliance gaps.”
‘Consent is not a realistic solution’
As states move to create laws aimed at protecting data privacy, burgeoning AI tools, like the one Kinney Drug’s implemented, could serve as a test case for how effective these laws are.
Zach Tomanelli, a privacy advocate with the Vermont Public Interest Research Group, said that customers’ experience with AI is an example of why the Vermont data privacy bill should have gone further. The law gives residents the right to know what information is being collected about them, correct inaccurate data and opt out of their data being sold. But lawyers and advocates say it includes too many exemptions for companies collecting data.
Near, the UVM computer science professor, said that regulations like the new Vermont law are inherently flawed simply because they are based on consumer consent. Kinney Drugs, for instance, spreads its terms and conditions across 11 documents.
“I think that people absolutely do not understand what they’re consenting to, and so basing our regulation and what we think is okay on the idea of consent is not a realistic solution to these kinds of privacy concerns nowadays,” he said.
Stacey Tovino, a University of Oklahoma professor who teaches HIPAA, said the privacy law lets patients petition to correct inaccurate health information. But the process is so cumbersome that Tovino herself said she would just switch pharmacies to avoid the exposure.
Customers in rural areas of Vermont, however, may not have an accessible option to switch to, and even those who do say they don’t want to, citing the personal connection they’ve built with the small-scale pharmacy chain and the people who work there.
“I don’t want to leave the pharmacy that I can walk to, that I really like,” Aveni said. “But I will if it doesn’t sort itself out.”
