Description as a Tweet:

Safety health wearable and app tailored to the elderly, people with dangerous health conditions, and people who suffer from memory loss.

Inspiration:

Many of our grandparents or parents may live alone, and their safety is our first priority. As people get older, they are more susceptible to dangerous health conditions and memory loss. HealthAlert helps them take care of themselves in different environments, like hot or humid climates, and immediately alert close family members if the wearable detects alarming events, like a seizure or fall.

We wanted to try something new for this hackathon, and since there were a lot of Arduino-related workshops and we had recently received an Arduino Nano 33 BLE Sense, we decided to make a hardware hack that uses Bluetooth.

Moreover, we wanted to experiment with Flutter’s notification and background activity capabilities with location-based and time-based notifications. While we used Flutter before, we were still fairly new to the framework and its features.

What it does:

The main goal of HealthAlert is to help those susceptible to medical conditions or memory loss take care of themselves more easily and alert close family members of alarming motions. The device has the following capabilities:
- Environment sensing: The temperature and humidity data captured/processed from the sensors on the Arduino Nano 33 BLE Sense are sent to the mobile app connected through Bluetooth Low Energy (BLE). If the temperature or humidity are at unhealthy levels (too high or too low), notifications are sent to the user to remind them to drink water, wear a coat, limit exercise, etc. If the temperature and humidity levels are healthy, the user is encouraged to exercise.

- Fall detection: If the wearable detects sudden acceleration that lasts for a couple clock cycles (fall) followed by a 10+ second period of no movement, an SMS message is automatically sent to the user-defined phone number (e.g. a close family member) that notifies them about the situation. On the hardware side, a buzzer starts beeping after the fall is detected to allow the user to respond by moving, indicating they can get back up. If the user does not move for 10+ seconds after the fall is detected, a different buzzer alarm sounds as the SMS message is sent.

- Seizure/severe tremor detection: If the wearable detects severe tremor motion that could indicate a seizure, the buzzer alarm goes off and an SMS message is automatically sent to the user-defined phone number.

- Reminders when exiting home: Using geofences, the associated app will send reminders the wearer might want. For example, if you constantly forget your papers or breakfast, this app will remind you to gather your papers or eat breakfast before you leave your home too far.

- Reminders at set times: The app is also able to create notifications at later times with reminders or messages of encouragement.

How we built it:

The wearable itself is an Arduino Nano 33 BLE Sense mounted on a small breadboard with a buzzer and LEDs to indicate connectivity/errors with sensors (green = working, connected; yellow = connecting...; red = problem with initializing BLE or sensors). Straps are used to mount the device on the user's arm.

The Arduino microcontroller was programmed in the Arduino IDE (mainly C++) to continuously retrieve sensor /status data to send to the app and detect falls or seizures.

Fall detection is handled by an algorithm that detects total acceleration from the onboard IMU that goes over a certain threshold multiple clock cycles in a row, followed by a period of no movement (acceleration below another threshold).

Seizure/tremor detection uses machine learning to differentiate between seizure-like severe tremors and normal or noise movement, like waving and scrubbing dishes. Samples simulating seizure-like tremors and normal movement were collected and used to train a model on Google Colab using Tensor Flow Lite to obtain a model header file that can be used for classification in the main Arduino code.

The app was created in Flutter, and the flutter_reactive_ble package was used to communicate with the Arduino through Bluetooth Low Energy. The app is user-friendly and displays current temperature/humidity data and status information (fall/seizure/weather).

In the settings menu, the user can add the phone number to send an SMS to when a fall or seizure is detected. The sms_maintained package is used to automatically send the SMS.

In the reminder and location reminder sections, the user can create notifications with custom messages and push requirements. With this app the user can push notifications based on time and date and location.

Technologies we used:

C/C++/C#
Python
Arduino
Microcontrollers
AI/Machine Learning
Misc

Challenges we ran into:

Although we had worked with app development in previous hackathons, it was our first time creating a complex project with Arduino (retrieving/processing sensor data, using Tensor Flow Lite to create and use ML models, etc.) and our first experience with Bluetooth. As a result, we had a lot of obstacles to overcome.

Bluetooth came with a steep learning curve, and it was difficult to integrate Bluetooth in both the Arduino and the Flutter app. Additionally, we had some difficulty figuring out how to run the Flutter app in the background, even when the app is not open. Eventually, we were able to get started, and after some troubleshooting, we figured out how to have the Flutter app continuously receive data/updates from the Arduino and send out notifications under specific circumstances.
Learning how to train and use a ML model was difficult too, since neither of us had much experience with machine learning. Since Tensor Flow Lite is relatively new, there is limited documentation (and limited Stack Overflow troubleshooting). Additionally, due to the size and processing needs of the model and classification, when the classifier was initially combined with the rest of the program, it would cause the MbedOS on the Arduino to crash. After following a couple of tutorials and experimenting with the code, we were able to successfully classify severe tremors vs. noise movement.

Accomplishments we're proud of:

We are proud of being able to accomplish so much in one project, as well as being able to workaround some of the problems we had.

For example, we initially wanted to use a passive buzzer so that we could easily control different tones from the Arduino. However, our Nano 33 BLE Sense board wouldn’t compile with the tone() function, so we found a workaround by using an active buzzer and “changing the pitch” by adding different milliseconds between each digital signal sent to the buzzer.

On the software side, there were some initial problems with interpreting how Flutter runs code in the background and implementing geo-based packages, but we got around them, and developed a local system of reminders based around geofences and time.

What we've learned:

We learned a lot about Arduinos/microcontrollers and I/O control. We also learned about how BLE works, with peripherals, central devices, and services, as well as integrating BLE in an Arduino project and a Flutter app project.

Additionally, we learned how to implement geofencing and train and use a machine learning model for a microcontroller.

By running into problems like diverging curves on the training/validation loss graph when we first tried to train a ML model, we were able to better understand how to troubleshoot and diagnose model behavior.

What's next:

Improving the model for classifying seizure tremors by using a more accurate training method, increasing the sample size, and talking with health care professionals.
Improving the UI to make it more user-friendly.
Adding more sensors to monitor changes in heart rate, body temperature, etc.
Adding a Wifi chip and a database instead of Bluetooth so that the data can be used/seen by family members at a longer distance (through an app or website), since many elderly people may not have smartphones or may find it difficult to use smartphones.

Built with:

On the hardware side, we used an Arduino Nano 33 BLE Sense, LEDs, an active buzzer, a pushbutton, a breadboard, jumper cables, and resistors.
For the app, we used Flutter and VSCode. We used several Flutter packages, including flutter_reactive_ble, sms_maintained, google_fonts, and several Geo-based packages.
We also used Tensor Flow Lite for integrating ML in the project, and a Tensor Flow tutorial in Google Colab to help train the model.

Prizes we're going for:

Best Documentation
Best Mobile Hack
Best Machine Learning Hack
Best Healthcare Hack
Best Hardware Hack

HealthAlert