Can Slack remind you about your coffee?

Pawel Wojciak - April 7, 2020

A big part of our working culture in Nexocode is participation in different kinds of hackathons. We love to continually challenge ourselves, keep up with industry trends and also have some fun with amusing projects.

Recently we organized our internal hackathon - nexothon. My team decided to test how new technologies, such as facial-recognition, can improve existing processes. We also thought about personalization, a topic that has been a hype in the IT for a while now. Automatically creating a basic user profile can add some extras in day to day situations. Let’s imagine you are waiting in line at the restaurant. Wouldn’t it be great if you could see information about the availability of your favorite dishes, including waiting time? Or if the waiter would be able to propose you ‘your usual’, as soon as you enter, based on your order history - isn’t that cool?

Our task was to find an existing problem in our office and solve it within 24 hours. We asked our colleagues when and where recognizing a person and their look/face can be useful. Among many propositions, one stood out and sounded like a real fun: ‘Lots of folks forget about their coffee cups at the coffee machine pedestal. Can you send them direct notification?’ - Challenge accepted!

Definition of the problem

The situation which should be handled by our system was defined as follows: if the person makes a coffee but forgets to take it within 2 minutes, he/she should be notified directly to avoid disturbing everyone. The typical way of handling this situation was screaming out loud: “Who left the coffee?!”. Notifying the coffee owner directly requires the system to be able to differentiate people by their pictures. There were specific circumstances when to ping the person: coffee was made, but the cup was not taken within 2 minutes; otherwise, no one should be bothered. The additional feature that we included in requirements, and was advised by our mentor, regards privacy: the system will take a photo only when a person allows so by pressing the dedicated button.

Hardware

The heart of our system was Raspberry Pi 4 running Raspbian OS connected to WiFI (to send a message via Slack). Additionally, few peripherals were used. For capturing pictures, we used a standard PC web camera connected to the USB port. There is one button, one distance sensor (ultrasonic sensor), and one LED board. Everything was connected using a breadboard, a few cables, and transistors.

System setup

Initial test with pizza box simulating coffee machine:D

Implementation

We decided to use Python for two main reasons:

  • It’s fast for prototyping - (we had only 24 hours)
  • There are many general available libraries in the field we tackled

Components

The project was structured into logical modules as presented below:

Project structure

Camera.py module contains logic for image capturing and processing, including face recognition. Face recognition is achieved by utilizing computer vision library OpenCV. The library leverages Haar Cascade classifier which is a machine learning based approach where a cascade function is trained from a lot of positive and negative images. It is then used to detect objects in other images. OpenCV comes with a trainer and detector and there are pre-trained classifiers including face classifier.

Our face recognition case consists of 3 phases:

  • Data gathering - this phase means running a Camera module in training mode which captures 30 images (of the particular face) and creates a data set assigned to specific id.
  • Training phase - We run this automatically right after the data set is created. In this phase photos from the dataset are used by OpenCV trainer which generates trainer.yml output file.
  • Face recognition - Using OpenCV recognizer to make the “prediction” e.g. recognise the face on the new photo. Recogniser uses data from trainer.yml.

Second input module is button.py which signals if button is pressed. Distance.py and coffe_detector.py handle input from the ultrasonic sensor to tell if the cup is its range or not. When the object appear on the pedestal picture was taken and used to detect familiar face.

Our system has several outputs used in a different ways. Slack module sends direct messages to the recognised person or on the #random channel if the person is unknown (face is not in database or picture is not ‘good enough’). Messages include pictures taken by the camera when the cup is put on the pedestal. We use Slack API which requires few standards steps to set up the communication:

  • Creating the application in App Management Page.
  • Adding OAuth scopes (enable application to view information about the user’s identity and send direct messages)
  • Using security token when the calling api - it is generated automatically when the app is added

The purpose of the lcd.py module was to communicate with lcd connected to Raspberry (to display the name) and the nexoboard.py sends a message to the internal motivation system called nexoboard.

State machine implementation

The final part of the application was to the action flow with proper usage of all components. There are two factors that control the current state of the system. The first is input from distance sensor which can have two values 0 or 1 depending on an object within range. The second is time, more specifically there is not cup system is in idle state and If the cup is on the pedestal k==1 and 2 minutes elapsed then the system goes to state send message. We used the whiteboard to define the states and the flow.

Prototyping application flow

Our system moves from one state to another and at any given time there is only one active state in the system. As you may know, such a computation model is called State Machine. Similar behavior can be observed in many devices in a modern facilities like elevators or traffic lights. Below you can see our implementation of the state machine:

And here are the entry point of the application (some lines were removed for clearness):

Running tests

Raspberry PI 4 has enough power to run all components, and use OpenCV modules very smoothly. Recognizing a person takes about a second, and a few seconds after slack message is delivered - your coffee will never be cold again!

Notification received via Slack

When the person wasn’t in the picture database or system couldn’t recognize a face (e.g. the picture was not good enough), then it sends a general notification to the #random channel.

Summary

This time it was about the lonely coffee cup, but maybe next time it will be a targeted advertising system in a beauty store or recommendation advisor in your favorite restaurant. Surely we will see more and more examples of similar devices. Creating the system that uses AI and face-recognition seems to be more accessible than ever. Of course, you can imagine many cases along with possibilities to resolve them by extending the system. Our setup was using one camera and proximity sensor, but depending on the given scenario, that could be changed accordingly. For us creating that prototype was a really fun experience.

P.S.

Some time after the hackathon and during further designing the board, we changed the way we control the LED diode. It was previously hardwired to the button (when the button has been pressed the diode was turning on) but right now it is software controlled. The behavior stays the same - when the button is pressed the diode is turned on but we now have the power to control that diode from the software.

Designing the board

The board prototype

After some final changes, hardware debugging we ordered a new boards from the factory. Couple weeks of waiting and here we are - brand new boards from the factory:

Boards from the factory

Tempted to work
on something
as creative?

That’s all we do.

join nexocode

Find us on

Need help with implementing AI in your business?

Let's talk blue circle

This site uses cookies for analytical purposes.

Accept Privacy Policy

In the interests of your safety and to implement the principle of lawful, reliable and transparent processing of your personal data when using our services, we developed this document called the Privacy Policy. This document regulates the processing and protection of Users’ personal data in connection with their use of the Website and has been prepared by Nexocode.

To ensure the protection of Users' personal data, Nexocode applies appropriate organizational and technical solutions to prevent privacy breaches. Nexocode implements measures to ensure security at the level which ensures compliance with applicable Polish and European laws such as:

  1. Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing Directive 95/46/EC (General Data Protection Regulation) (published in the Official Journal of the European Union L 119, p 1); Act of 10 May 2018 on personal data protection (published in the Journal of Laws of 2018, item 1000);
  2. Act of 18 July 2002 on providing services by electronic means;
  3. Telecommunications Law of 16 July 2004.

The Website is secured by the SSL protocol, which provides secure data transmission on the Internet.

1. Definitions

  1. User – a person that uses the Website, i.e. a natural person with full legal capacity, a legal person, or an organizational unit which is not a legal person to which specific provisions grant legal capacity.
  2. Nexocode – NEXOCODE sp. z o.o. with its registered office in Kraków, ul. Generała Henryka Kamieńskiego 51, 30-644 Kraków, entered into the Register of Entrepreneurs of the National Court Register kept by the District Court for Kraków-Śródmieście in Kraków, 11th Commercial Department of the National Court Register, under the KRS number: 0000686992, NIP: 6762533324.
  3. Website – website run by Nexocode, at the URL: nexocode.com whose content is available to authorized persons.
  4. Cookies – small files saved by the server on the User's computer, which the server can read when when the website is accessed from the computer.
  5. SSL protocol – a special standard for transmitting data on the Internet which unlike ordinary methods of data transmission encrypts data transmission.
  6. System log – the information that the User's computer transmits to the server which may contain various data (e.g. the user’s IP number), allowing to determine the approximate location where the connection came from.
  7. IP address – individual number which is usually assigned to every computer connected to the Internet. The IP number can be permanently associated with the computer (static) or assigned to a given connection (dynamic).
  8. GDPR – Regulation 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of individuals regarding the processing of personal data and onthe free transmission of such data, repealing Directive 95/46 / EC (General Data Protection Regulation).
  9. Personal data – information about an identified or identifiable natural person ("data subject"). An identifiable natural person is a person who can be directly or indirectly identified, in particular on the basis of identifiers such as name, identification number, location data, online identifiers or one or more specific factors determining the physical, physiological, genetic, mental, economic, cultural or social identity of a natural person.
  10. Processing – any operations performed on personal data, such as collecting, recording, storing, developing, modifying, sharing, and deleting, especially when performed in IT systems.

2. Cookies

The Website is secured by the SSL protocol, which provides secure data transmission on the Internet. The Website, in accordance with art. 173 of the Telecommunications Act of 16 July 2004 of the Republic of Poland, uses Cookies, i.e. data, in particular text files, stored on the User's end device.
Cookies are used to:

  1. improve user experience and facilitate navigation on the site;
  2. help to identify returning Users who access the website using the device on which Cookies were saved;
  3. creating statistics which help to understand how the Users use websites, which allows to improve their structure and content;
  4. adjusting the content of the Website pages to specific User’s preferences and optimizing the websites website experience to the each User's individual needs.

Cookies usually contain the name of the website from which they originate, their storage time on the end device and a unique number. On our Website, we use the following types of Cookies:

  • "Session" – cookie files stored on the User's end device until the Uses logs out, leaves the website or turns off the web browser;
  • "Persistent" – cookie files stored on the User's end device for the time specified in the Cookie file parameters or until they are deleted by the User;
  • "Performance" – cookies used specifically for gathering data on how visitors use a website to measure the performance of a website;
  • "Strictly necessary" – essential for browsing the website and using its features, such as accessing secure areas of the site;
  • "Functional" – cookies enabling remembering the settings selected by the User and personalizing the User interface;
  • "First-party" – cookies stored by the Website;
  • "Third-party" – cookies derived from a website other than the Website;
  • "Facebook cookies" – You should read Facebook cookies policy: https://www.facebook.com/policy/cookies
  • "Other Google cookies" – Refer to Google cookie policy: www.google.com/policies/technologies/types/

3. How System Logs work on the Website

User's activity on the Website, including the User’s Personal Data, is recorded in System Logs. The information collected in the Logs is processed primarily for purposes related to the provision of services, i.e. for the purposes of:

  • analytics – to improve the quality of services provided by us as part of the Website and adapt its functionalities to the needs of the Users. The legal basis for processing in this case is the legitimate interest of Nexocode consisting in analyzing Users' activities and their preferences;
  • fraud detection, identification and countering threats to stability and correct operation of the Website.

4. Cookie mechanism on the Website

Our site uses basic cookies that facilitate the use of its resources. Cookies contain useful information and are stored on the User's computer – our server can read them when connecting to this computer again. Most web browsers allow cookies to be stored on the User's end device by default. Each User can change their Cookie settings in the web browser settings menu: Google ChromeOpen the menu (click the three-dot icon in the upper right corner), Settings > Advanced. In the "Privacy and security" section, click the Content Settings button. In the "Cookies and site date" section you can change the following Cookie settings:

  • Deleting cookies,
  • Blocking cookies by default,
  • Default permission for cookies,
  • Saving Cookies and website data by default and clearing them when the browser is closed,
  • Specifying exceptions for Cookies for specific websites or domains

Internet Explorer 6.0 and 7.0
From the browser menu (upper right corner): Tools > Internet Options > Privacy, click the Sites button. Use the slider to set the desired level, confirm the change with the OK button.

Mozilla Firefox
browser menu: Tools > Options > Privacy and security. Activate the “Custom” field. From there, you can check a relevant field to decide whether or not to accept cookies.

Opera
Open the browser’s settings menu: Go to the Advanced section > Site Settings > Cookies and site data. From there, adjust the setting: Allow sites to save and read cookie data

Safari
In the Safari drop-down menu, select Preferences and click the Security icon.From there, select the desired security level in the "Accept cookies" area.

Disabling Cookies in your browser does not deprive you of access to the resources of the Website. Web browsers, by default, allow storing Cookies on the User's end device. Website Users can freely adjust cookie settings. The web browser allows you to delete cookies. It is also possible to automatically block cookies. Detailed information on this subject is provided in the help or documentation of the specific web browser used by the User. The User can decide not to receive Cookies by changing browser settings. However, disabling Cookies necessary for authentication, security or remembering User preferences may impact user experience, or even make the Website unusable.

5. Additional information

External links may be placed on the Website enabling Users to directly reach other website. Also, while using the Website, cookies may also be placed on the User’s device from other entities, in particular from third parties such as Google, in order to enable the use the functionalities of the Website integrated with these third parties. Each of such providers sets out the rules for the use of cookies in their privacy policy, so for security reasons we recommend that you read the privacy policy document before using these pages. We reserve the right to change this privacy policy at any time by publishing an updated version on our Website. After making the change, the privacy policy will be published on the page with a new date. For more information on the conditions of providing services, in particular the rules of using the Website, contracting, as well as the conditions of accessing content and using the Website, please refer to the the Website’s Terms and Conditions.

Nexocode Team