Can Slack remind you about your coffee?

A big part of our working culture in Nexocode is participation in different kinds of hackathons. We love to continually challenge ourselves, keep up with industry trends and also have some fun with amusing projects.

Recently we organized our internal hackathon - nexothon. My team decided to test how new technologies, such as facial-recognition, can improve existing processes. We also thought about personalization, a topic that has been a hype in the IT for a while now. Automatically creating a basic user profile can add some extras in day to day situations. Let’s imagine you are waiting in line at the restaurant. Wouldn’t it be great if you could see information about the availability of your favorite dishes, including waiting time? Or if the waiter would be able to propose you ‘your usual’, as soon as you enter, based on your order history - isn’t that cool?

Our task was to find an existing problem in our office and solve it within 24 hours. We asked our colleagues when and where recognizing a person and their look/face can be useful. Among many propositions, one stood out and sounded like a real fun: ‘Lots of folks forget about their coffee cups at the coffee machine pedestal. Can you send them direct notification?’ - Challenge accepted!

Definition of the problem

The situation which should be handled by our system was defined as follows: if the person makes a coffee but forgets to take it within 2 minutes, he/she should be notified directly to avoid disturbing everyone. The typical way of handling this situation was screaming out loud: “Who left the coffee?!”. Notifying the coffee owner directly requires the system to be able to differentiate people by their pictures. There were specific circumstances when to ping the person: coffee was made, but the cup was not taken within 2 minutes; otherwise, no one should be bothered. The additional feature that we included in requirements, and was advised by our mentor, regards privacy: the system will take a photo only when a person allows so by pressing the dedicated button.

Hardware

The heart of our system was Raspberry Pi 4 running Raspbian OS connected to WiFI (to send a message via Slack). Additionally, few peripherals were used. For capturing pictures, we used a standard PC web camera connected to the USB port. There is one button, one distance sensor (ultrasonic sensor), and one LED board. Everything was connected using a breadboard, a few cables, and transistors.

Implementation

We decided to use Python for two main reasons:

• It’s fast for prototyping - (we had only 24 hours)
• There are many general available libraries in the field we tackled

Components

The project was structured into logical modules as presented below:

Camera.py module contains logic for image capturing and processing, including face recognition.

Face recognition is achieved by utilizing computer vision library OpenCV. The library leverages Haar Cascade classifier which is a machine learning based approach where a cascade function is trained from a lot of positive and negative images. It is then used to detect objects in other images. OpenCV comes with a trainer and detector and there are pre-trained classifiers including face classifier.

Our face recognition case consists of 3 phases:

• Data gathering - this phase means running a Camera module in training mode which captures 30 images (of the particular face) and creates a data set assigned to specific id.
• Training phase - We run this automatically right after the data set is created. In this phase photos from the dataset are used by OpenCV trainer which generates trainer.yml output file.
• Face recognition - Using OpenCV recognizer to make the “prediction” e.g. recognise the face on the new photo. Recogniser uses data from trainer.yml.

Second input module is button.py which signals if button is pressed. Distance.py and coffe_detector.py handle input from the ultrasonic sensor to tell if the cup is its range or not. When the object appear on the pedestal picture was taken and used to detect familiar face.

Our system has several outputs used in a different ways. Slack module sends direct messages to the recognised person or on the #random channel if the person is unknown (face is not in database or picture is not ‘good enough’). Messages include pictures taken by the camera when the cup is put on the pedestal. We use Slack API which requires few standards steps to set up the communication:

• Creating the application in App Management Page.
• Adding OAuth scopes (enable application to view information about the user’s identity and send direct messages)
• Using security token when the calling api - it is generated automatically when the app is added

The purpose of the lcd.py module was to communicate with lcd connected to Raspberry (to display the name) and the nexoboard.py sends a message to the internal motivation system called nexoboard.

State machine implementation

The final part of the application was to the action flow with proper usage of all components. There are two factors that control the current state of the system. The first is input from distance sensor which can have two values 0 or 1 depending on an object within range. The second is time, more specifically there is not cup system is in idle state and If the cup is on the pedestal k==1 and 2 minutes elapsed then the system goes to state send message. We used the whiteboard to define the states and the flow.

Our system moves from one state to another and at any given time there is only one active state in the system. As you may know, such a computation model is called State Machine. Similar behavior can be observed in many devices in a modern facilities like elevators or traffic lights. Below you can see our implementation of the state machine:

def state_machine(coffeeDetector):
    global state
    global t1
    global t2
    P = coffeeDetector.consent_button_pressed()
    K = coffeeDetector.coffee_on_tray()
    switcher = {
        1: waiting_for_action,
        2: waiting_for_consent,
        3: waiting_for_coffee,
        4: known_face_and_wait_for_coffee_done,
        5: wait_for_pickup,
        6: unknown_face_and_wait_for_coffee_done,
    }
    machine_step = switcher.get(state)
    machine_step(P, K)
    print("State:", state, machine_step.__name__, "t1:", t1, "t2:", t2)

And here are the entry point of the application (some lines were removed for clearness):

if __name__ == "__main__":
        # setup
        coffeeDetector = CoffeeDetector(consent_button_pin=3, distance_trigger=5, distance_echo=7)
        while keep_alive:
            # in case any of the timer goes below zero
            t1 = 0 if t1 < 0 else t1
            t2 = 0 if t2 < 0 else t2
            state_machine(coffeeDetector)
            if t1 > 0:
                t1 -= timer_decrement
            if t2 > 0:
                t2 -= timer_decrement
            time.sleep(cycle_length)

Running tests

Raspberry PI 4 has enough power to run all components, and use OpenCV modules very smoothly. Recognizing a person takes about a second, and a few seconds after slack message is delivered - your coffee will never be cold again!

When the person wasn’t in the picture database or system couldn’t recognize a face (e.g. the picture was not good enough), then it sends a general notification to the #random channel.

Summary

This time it was about the lonely coffee cup, but maybe next time it will be a targeted advertising system in a beauty store or recommendation advisor in your favorite restaurant.

Surely we will see more and more examples of similar devices. Creating the system that uses AI and face-recognition seems to be more accessible than ever.

Of course, you can imagine many cases along with possibilities to resolve them by extending the system. Our setup was using one camera and proximity sensor, but depending on the given scenario, that could be changed accordingly. For us creating that prototype was a really fun experience.

P.S.

Some time after the hackathon and during further designing the board, we changed the way we control the LED diode. It was previously hardwired to the button (when the button has been pressed the diode was turning on) but right now it is software controlled. The behavior stays the same - when the button is pressed the diode is turned on but we now have the power to control that diode from the software.

After some final changes, hardware debugging we ordered a new boards from the factory. Couple weeks of waiting and here we are - brand new boards from the factory:

Useful Links:

• https://opencv.org/about/
• https://api.slack.com/
• https://www.raspberrypi.org/
• https://www.pyimagesearch.com/2017/09/04/raspbian-stretch-install-opencv-3-python-on-your-raspberry-pi/
• https://www.hackster.io/mjrobot/real-time-face-recognition-an-end-to-end-project-a10826
• https://tutorials-raspberrypi.com/raspberry-pi-ultrasonic-sensor-hc-sr04/