[PharmaSEE] Project Outline & Demo
- 다영 윤
- Feb 25, 2022
- 3 min read
Updated: Feb 27, 2022
PharmaSEE is a team project undertook for the Software Engineering course at Hanyang University, under the supervision of Dr.YounjunWon . We won 3rd place in the SKT-Hanyang University AI IC-PBL contest with this project. The code can be found at this link, in which you will find 4 repositories as explained below.
Server: Python Django code for the backend server, implementing RESTful APIs using the django-rest-framework. Details of the backend developemnt process can be found here.
Ai_Recognition: Training and testing the object detection model using the Tensorflow Object Detection API.
Client: Frontend Javascript code in React Native, that renders native mobile applications for both Android and IOS.
Documentation: Latex code and pdf of documentation on the poject requirements and software use cases. Link to the pdf
Project Outline
PharmaSEE is a mobile application that supports the medication intake of the elderly, using AI methods including object detection and voice recognition. The app allows patients to register the pills that they have to take daily and provides ways for the user to check if the amount of pills and the type of pills they are taking is accurate. This can be a challenging task for the elderly, especially if they are suffering from several chronic diseases.
We hope that our application can provide protection against the dangers of medication misuse. Additionally, the caretakers of the patients can monitor the medication that was taken that day through the mobile app, reducing the burden of ensuring patient safety.
How does PharmaSEE prevent medication misuse?
An object detection deep learning model, which we trained using the Tensorflow Object Detection API, checks the type / amount of pills in the image taken by the patient. The results will be compared with the medication information the has been pre-registered. In the case the user is overtaking or missing out a certain medicine, the app will let them know what is wrong.
For example, in the image below, the user is taking a medicine that a user has already took that day, indicated by an X. A message in text will be provided sayinh that this pill should not be taken at the moment. On the left is the image taken by the user, and on the right is the resulting image that has been processed in the backend server.
How can caretakers remotely monitor patients' medication intake?
Caretakers and patients can follow each other throught the app, allowing the caretaker to view the medication the patient took that day. Also, we used a programmable AI speaker from SKT called Nugu, so the caretakers can simply inquire this information from the speaker as well.
The diagram shows how user prompts such as 'Did mom take her pills today?' is picked up by the speaker and the data is queried from the backend server. The speaker will use the received data to give back a prompt containing information about the medicine taken that day.
Demo Video
English subtitles aren't there yet, but working on it!
Development Process
Listed software requirements and designed the database schemas/tables
Developed REST API for CRUD operations on user data
Set up the AI speaker and developed a proxy server for database access
Deployed the backend server on AWS Lightsail
Object Detetion w/ Deep Learning
Created image data and labeled the images for training
Trained with the tensorflow object detection api
Deploying the tf model using tf model serving
Team Members
Dayoung Yun ✉
Backend development - proxy server for the AI speaker, image transmission, TF model serving
Dohyeong Han ✉
Backend development - user data management, cloud deployment with AWS lightsail
Miju Kang ✉
Frontend development - native application for Android and IOS
KyoungWhan Mheen ✉
Object Detection training in Tensorflow, UI/UX design
Morgan Jeon ✉
Product management
Comments