[PharmaSEE] Backend Server Development

Software in use

• Django 3.0, django-rest-framework

• PostgreSQL

• Postman and Ngrok for API testing

• Nugu Play Builder

• Tensorflow Object Detection API

• TF Model Serving

• AWS Lightsail

• Development Env: Ubuntu 18.04

Bakend Requirements

1. CRUD operations for User, Pills, Reminders (django models)

2. Serve API requets from Nugu speaker

3. Transfer images w/ client and perform object detection. Input and output images managed in database as table DNNImages.

Database Schema

The follower set in the User model is for the caretaker-user monitoring system. Patients can allow certain caretakers to view their information by adding them to their follower_set in the database.

Pill, Reminder Model

pharmasee/models.py

class Pill(TimestampedModel): name = models.CharField(max_length=100) image_dir = models.ImageField(upload_to="pharmasee/pill/%Y/%m/%d") effect = models.CharField(max_length=500) side_effect = models.CharField(max_length=500) def __str__(self): return self.name class Meta: ordering = ["-id"] class Reminder(TimestampedModel): title = models.CharField(max_length=100) user_id = models.ForeignKey( settings.AUTH_USER_MODEL, related_name="reminders", on_delete=models.CASCADE ) pill_id = models.ForeignKey(Pill, on_delete=models.CASCADE) dose = models.CharField(max_length=100) when_to_take = models.TimeField(auto_now=False) taken_time = models.TimeField(auto_now=False) is_taken_today = models.BooleanField(default=False) dose_taken_today = models.IntegerField(default=0) def __str__(self): return self.title class Meta: ordering = ["-id"]

Pill models can be created by users, and some are pre-registered using data from the Korea's open api portal.

Reminders are entities that stores the dose and time to take a certain pill for a user. taken_time and dose_taken_today stores the time and dose of the pill if it has been taken today. (therefore is_taken_today must be true for these fields to be valid) The reminder table is largely queried in the pill_ai app views.

Since we use the django-rest-framework to develop REST apis, we use serializers instead of Forms. Serializers allows user input data to be converted to native Python data types that can be easily rendered to Json or XML. This form of data is used to created instances of the Pill and Reminder models we made. (More on Serializers)

Since the models are relatively simple we use the ModelSerializer class to simplify the code.

pharmasee/serializers.py

class PillSerializer(serializers.ModelSerializer): class Meta: model = Pill fields = '__all__' class ReminderSerializer(serializers.ModelSerializer): class Meta: model = Reminder fields = '__all__'

Similarily, we use ModelViewSet and drf Router utility classes to implement urls and views, so there isn't much to comment on that.

Validating Pills with Object Detection

A big feature of this project is to provide mobile application users accurate medication information, especially for those who find it hard to keep track of different pills they are taking at the same time. For this purpose we want app users to simply take pictures of the pills before intaking, and have the backend server detect what is in the picture, compare it with the infromation in the database, and diagnose if there is nothing for wrong. For instance, are there any pills that the user is missing/overtaking? Are there any that shouldn't be taken together?

For this purpose we first need an object detection model. We used the tensorflow obejct detection api to train the model with the image of data of several pills together. The training process will be specified in an upcoming post.

The trained object detection model shows results like this. The output of the model are sets of {x-coordinate, y-coordinate, width, height, class}. This example is for a test we ran using 5 classes of pills. We plan to increase the number of pills we can classify with one model, as we improve the image data set we have.

Our trained model will be deployed in the backend server machine using tensorflow model serving. With tf model serving we can launch a http server in our local machine whose sole purpose is to wait for http requests and receive images, inferencing with the devployed tf model, and sending back object detection results in a Json-like format. This way we can avoid the pain of having to load the tf model everytime a mobile app client requests for object detection results. This can be very time consuming, making the user experience horrible.

Image transfer between client and server

The client side sends an API request to the server at the url 'ip address/pill_ai/img_upload'. (defined in urls.py files) The image file should be sent under the field name 'image'. (this name should be shared between the backend and frontend developers)

Inside the view, we parse out the 'image' field from the request data and format it into a dict type object, so the Serializer object can map the image file data to the 'input_image' field of the DNNImage model. Since we are handling image data, we must include the MultiPartParser class to indicate that we will be handling files uploaded from the frontend client.

pill_ai/views.py

class DnnImageView(APIView): parser_classes = (MultiPartParser, FormParser) def post(self, request, *args, **kwargs): file = request.FILES.get('image') data = { 'input_image': file } file_ser = DnnImageSerializer(data=data) if file_ser.is_valid(): file_ser.save() data = file_ser.data return Response(data, status=status.HTTP_201_CREATED) else: return Response(status=status.HTTP_400_BAD_REQUEST)

pill_ai/serializers.py

from rest_framework import serializers from .models import DnnImage class DnnImageSerializer(serializers.ModelSerializer): class Meta: model = DnnImage fields = '__all__'

pill_ai/models.py

class DnnImage(TimestampedModel): input_image = models.ImageField(upload_to=file_upload_path_input) output_image = models.ImageField(upload_to=file_upload_path_output, blank=True, null=True) correct = models.BooleanField(default=False) status_mesg = models.TextField(blank=True, null=True) def save(self, *args, **kwargs): self.predict_pills() super(DnnImage ,self).save(*args, **kwargs) def predict_pills(self, *args, **kwargs): output_image, correct, status_mesg = predict(self.input_image) self.output_image = InMemoryUploadedFile( file=output_image, field_name='ImageField', name=self.input_image.name, content_type='image/jpeg', size=sys.getsizeof(output_image), charset=None ) self.correct = correct self.status_mesg = status_mesg

When the save() method is conducted on the DNNImage serializer(file_ser.save() in views.py) , the overridden method in the model definition will be called. This method invokes predict_pills(), which takes the image blob saved in the input_image field, performs object detection, and produces the output_image and status_mesg field.

Take notes of how we used the InMemoryUploadedFile class to create an ImageField entry for the output_image field.

def predict(img): img = Image.open(img) h, w = img.height, img.width img = img.convert('RGB').resize((512, 512)) img = np.array(img) img = np.expand_dims(img, axis=0) data = json.dumps({ "instances": img.tolist() }) headers = {"content/type": "application/json"} response = requests.post('http://localhost:8501/v1/models/pills_model:predict', data=data, headers=headers) status_mesg = "" predictions = response.json()['predictions'][0] opencv_img = cv2.cvtColor(np.squeeze(img), cv2.COLOR_RGB2BGR) status_mesg = draw_label_for_single_image(opencv_img, predictions, status_mesg, 0.8) opencv_img = cv2.cvtColor(opencv_img, cv2.COLOR_BGR2RGB) img = np.array(opencv_img) img = Image.fromarray(img) img = img.resize((w, h)) return (img_to_bytes(img), False, status_mesg) def img_to_bytes(img): output = BytesIO() img.save(output, format='JPEG', quality=100) output.seek(0) return output