Health Record App – A Generative AI Use Case

Over the last year, I started to develop an interest in Machine Learning, specifically in Generative AI. I have been attracted by the potential of it and how it can change different aspects of our current and future life. I read books and took trainings to deep dive into this technology.

In this post, I will deep dive into a Generative AI use case many of you will identify with and a solution I built around it. I am a Software Developer at heart, so now I work on getting the most out of Generative AI by getting my hands dirty and creating solutions to real-world problems.

Health Record Use Case

How do you feel about visiting a doctor? For many of you it may not be a nice process, especially when you need to go to the Emergency Room. You have to remember dates and times (when a specific symptom begun), the frequency of symptoms, the medicine you took. You may or may not remember all that. As humans, you are probably busy and when you or your relatives become sick, you are not at your best to remember details. The problem is that doctors need detailed, accurate information to work on a diagnosis and prescribe medicine or advise you on the steps forward.

Moreover, a conversation with the doctor occurs in the consultation. A lot of details are shared and sometimes, you forget details. Yes, the doctor will give you a report, but small details usually get lost. As a patient, you must own you medical data, so that you can review it, share it, and keep it for historical purposes.

Therefore, there is a real-world problem to solve. You need a solution to store symptoms, the medicine you took, historical events and an overview of your conversation with the doctor. You store this data long term to get insights into your health over time. How do you solve this problem?

A solution based on Amazon Bedrock and AWS HealthScribe

Firstly, you need to address data input, storage and output. Data input must be easy and user friendly. Having the user typing details will create friction, specially when the user has little time. The data output must be clear, concise and straightforward, but keeping enough contextual details so that the doctors have the information they need. You do not want the user to have a list of dates and events to be read one by one. Also, data must be securely stored and processed. You must ensure that your data is private and not publicly available, so that only approved users access it.

Having the above high level needs, I suggest the following solution:

1. Choose AWS HealthScribe to capture, process and store the conversation on the consultation. AWS HealthScribe is a new HIPAA-eligible machine learning (ML) capability that combines speech recognition and generative AI to transcribe patient-clinician conversations and generate easy-to-review clinical notes.
2. Create a simple user interface to capture voice, so the user only needs to start a voice recording to store their medical details, symptoms and medicines.
3. Use a Large Language Model to summarize the health information. To do this, I chose Anthropic Claude 3 Sonnet via Amazon Bedrock. The driver for this is that Amazon Bedrock is a fully managed service that makes high-performing foundation models (FMs) from leading AI startups and Amazon available for your use through a unified API. Moreover, it provides features to easily create Generative AI applications without the burden of managing complex infrastructure.
4. The summarization of the user health will be shown on the simple user interface.

Here is the solutions’s architecture diagram:

The flow of events is as follows:

1. The user authenticates to the Health Record App via Amazon Cognito which handles user authentication and authorization for web and mobile apps.
2. The user inputs health data. It could be a voice recording of the medical consultation or input her symptoms or medicines taken. The Health Record App will create an audio recording in the former case or a text file in the latter. Either files will be uploaded into an Amazon S3 bucket.
3. An AWS Lambda function validates the files and send them to an Amazon S3 bucket that Amazon Bedrock Knowledges Bases uses as source to index data. Only audio recording files go to AWS HealthScribe, which process it and generate a json output file. This AWS Lambda function also start the synchronization in Amazon Bedrock Knowledges Bases.
4. An authenticated user asks for a summary of her health data. This request goes through an Amazon API Gateway to an AWS Lambda function which verifies and validates the query, applies filtering of not allowed words and topics, and finally sends the query in a single API call via the “Retrieve and Generate” action method, which queries the Knowledge Base and generates responses based on the retrieved results.
5. The final response is sent back to the authenticated user.

As you see, the architecture is based on the Retrieval Augmented Generation (RAG) approach. The main drivers for that decision are:

• The data is not stale. The user will add new records over time.
• RAG provides the expected outcomes with lower investment. Fine tunning was considered but it requires additional effort to keep the model in-sync when new health information is added.
• Data can be stored in a separate data store for additional integration and processing.

The solution addresses the functional requirements and the identified quality attributes. However, there is an additional room for improvement on the user experience. Let’s see how to address this.

Making the user experience more human

The user interface is a simple form to input and output data. You can add colours, images and animations to make it easy to use but it will never feel human. The users will feel like they are interacting with a machine. To make the interaction smooth you need something different than a simple chatbot.

To solve this problem, my co-worker Marcelo Cuna suggests to use the Soul Machines AI Assitants. Soul Machines is a leading innovator in humanizing AI experiences. They leverage cutting-edge, patented Biological AI technology to transform highly personalized digital connections with a commitment to making AI accessible and ethical for all. You can see more information about them here

The integration of the Soul Machine AI Assistant is straightforward. Here, you will see a revised version of the solutions architecture diagram:

The Soul Machine AI Assistant is now the user interface. It integrates with the GenAI backend via Amazon Lex, which is an AWS service for building conversational interfaces for applications using voice and text. It provides the deep functionality and flexibility of natural language understanding (NLU) and automatic speech recognition (ASR) so you can build highly engaging user experiences with lifelike, conversational interactions, and create new categories of products. An AWS Lambda moves the transcription of the conversation into the Amazon S3 input bucket. Those are the changes needed.

Now, the user will talk to the AI Assistant to input data, receiving confirmation or asking followup questions. Also, the user will instruct it to speak up the summary of recent health events to the doctor (in my personal experience, I found this useful when you have a sore throat, for example).

The AI Assistant is not a replacement of a doctor. The purpose is to provide a human like interaction to the users, so they feel more comfortable when interacting with the app.

The following video shows a demo of the Health Record App in action (Spanish version):

Conclusion

Everything begins with identifying the problem you are trying to solve. Generative AI promises to change the way we work, how we interact with systems and how to ingest data and create content. However, it requires engineering discipline to understand the problem, identify business requirements and then come with a solution which can be based on Generative AI.

You see how AWS HealthScribe and Amazon Bedrock provide you with managed services that allow you to focus on the root problem, on adding value without the hassle of configuring and managing infrastructure. Also, you have seen Amazon Bedrock’s additional features to easily create Generative AI solutions, like Amazon Bedrock Knowledge Bases.

Finally, you must think on your users and how they will interact with your solutions. Soul Machines AI Assistants are a solution to provide a human like interaction in our Generative AI apps. It integrates with your AWS environment in a straightforward way.