From prototype to production: Deploying clinical AI summaries without a playbook
Clinical teams in post-acute care organizations routinely face the challenge of processing extensive documentation to make timely decisions about patient referrals. The development and deployment of WellSky Summarize, an AI-powered clinical summarization tool for clients of the WellSky CarePort Referral Intake solution, was designed to help streamline workflows and improve efficiency.
Why clinical AI summaries matter
Post-acute providers must evaluate referrals for appropriateness, medical necessity, insurance eligibility, and provider capacity. Referral packets often arrive fragmented, inconsistent, and lacking pertinent details. Admission/intake coordinators and clinical reviewers spend significant time sifting through these documents, which can delay responses, delay patient transitions of care, and/or result in missed opportunities for the post-acute providers. The introduction of AI clinical summaries in the WellSky CarePort Referral Intake solution aims to help reduce the time required to understand a patient’s needs and overall clinical picture, helping providers respond with improved efficiency during the referral process.
Building the foundation: Early steps and prompt engineering
The project began with our clinical team identifying the essential information that post-acute care providers in skilled nursing facilities and home health agencies would be looking for in the referral packets that they received. Input and feedback from clients validated that our set of questions would retrieve the most pertinent information, helping to address real-world needs.
Prompt engineering was a critical step in refining and improving AI results. The AI-generated summaries were compared to our clinician’s summaries after manual review of a referral packet. The initial large set of questions was reduced to 10, making the process more manageable and focused.
Establishing the golden data set: Scoring and collaboration
With refined prompts in hand, the team turned its attention to evaluation, building a golden data set to understand how well the AI performed in realistic scenarios. Rather than relying on anecdotal feedback, the team scored AI responses using clearly defined clinical criteria, creating a consistent way to assess quality and reliability.
Regular assessments to analyze the data informed iterative improvements to some prompts and to ensure inter-rater reliability was maintained.
Lessons learned: Collaboration and adaptability
The project highlighted the importance of clear scoring criteria, the necessity of efficient operational tools, and early involvement of project management. Cross-functional collaboration proved essential, with clinical, engineering, solutions, data analytics, legal, and UX teams working closely together to address challenges and drive the project forward.
Conclusion: Moving forward with our AI Clinical Summarization tool
Deploying clinical AI summaries without a known playbook requires agility, collaboration, and a willingness to adapt. The experience underscores the value of iterative development and the importance of building responsible, user-centered AI solutions. As the AI clinical summarization tool is utilized by our clients who use the WellSky CarePort Referral Intake solution, ongoing feedback, the addition of new prompts, and the expansion into other levels of care will continue to shape its impact on post-acute care workflows.
