Hospitality Sector

Hotel Image Recognition and Classification Solution

Custom Backend Image Recognition

Developing a model that can categorize hotel images?
Challenge Accepted!

Hotel Image Recognition and Classification Solution

Process & Story

The process of image recognition is always challenging, but it is challenging when dealing with images that are not well-labeled or contain a variety of objects. In the context of hospitality, this means trying to develop a model that can correctly identify and categorize photos of hotels. This was the task at hand for our team as we sought to create an AI solution based on visual image recognition.

Problem

Hospitality platforms deal with hundreds of thousands of images for hotel offers. For each offer, booking engines present several selected images that vary from the bedroom, bathrooms, lobby, pool, restaurant, window view, and nearby attractions. To provide the best user experience, the platform should show an optimal number of photos from each category to present the hotel offer in the best possible way visually. We’ve observed that booking engine sites deal with vast numbers of hotel photos, some with very low image quality or not particularly relevant to travelers, but the problem here is the lack of description for content in the image resulting in the wrong order of the photos.

To show the right image to the user, the platform must first categorize the image. This can be done either manually by visual content specialists or automatically with the help of machine learning and visual image recognition.

It is impossible to keep up with a costly manual process for big hospitality platforms that integrate with various vendors and update their offers daily. Booking engines are in great need of an artificial intelligence tool that can automatically detect and categorize photos for all their hotel offers.

This process of image classification is a complex task as it not only requires identifying what’s in the image but also understanding the context of the image and how it relates to other images in the same category.

Solution

The solution was to create a machine learning model that could sift through the image data and automatically categorize the photos. The first step was to gather a dataset of images representing the different types of hotels and their interiors found in the hospitality industry. This dataset was used to train a machine learning model that could be used for image recognition.

As this is an image recognition problem, we decided to apply ResNet architecture for the machine learning model. The Residual Network (ResNet) is a type of Convolutional Neural Network (CNN) architecture that addresses the “vanishing gradient” issue, allowing for networks with hundreds of convolutional layers to outperform shallower ones. The ResNets architecture and convolutional neural networks are frequently used in deep learning computer vision applications such as object detection, image classification, and image segmentation.

For the deep learning model building and training, we used the fast.ai library, which increases the level of abstraction of PyTorch. It is relatively new but already supports good practices and is always up to date with advancements in deep learning. We fine-tuned the ResNet deep learning model pre-trained on the ImageNet dataset with the hotel images dataset.

Once the model was trained, it was then tested on a new set of images to see how well it could categorize them. The results were promising, and the model was able to identify and categorize a variety of hotel images correctly. However, there were still some challenges that needed to be addressed.

The next step was to create a prototype application that would allow users to upload photos and identify the category of each photo or to do the same through our dedicated API endpoints.

Case Study Schema Hotel Image Recognition and Classification Solution

Tools

Google Cloud Platform

Google Cloud Platform

PyTorch

PyTorch

Python

Python

Have a similar project in mind?

LET'S TALK!

Challenges

Gathering data and building a hotel images training set
We needed to create a dataset for the training process from scratch, and we decided to rely on free and open-source resources. Indoor scene recognition from Kaggle and MIT resources datasets fitted pretty well for our task. They needed some preprocessing, and there were many more categories with unrelated images for hotel recognition. When it comes to training data, it is always easier when you have more than required, as you can exclude unnecessary categories easily. The second batch of hotel images was imported by simply using Google Graphics. They have an open API, and the GitHub repository provides a convenient library.

Limited data set and data augmentation
Image training set data is a never-ending stream. When it comes to image recognition tasks, more data means better accuracy. However, sometimes gathering more data is not possible or practical. In those cases, you can resort to data augmentation, which is the process of modifying existing data to increase its size and diversity. This is precisely what we did with our image data. We randomly rotated, flipped, and scaled the images to make them look different. This not only helped us increase the size of our dataset but also improved the accuracy of our model. We ended up with a large-scale curated dataset of images for hotel recognition.

Preprocessing data for image classification
Most computer vision models are susceptible to image noise and other artifacts. This can be a significant challenge when you are working with a limited amount of data. In our case, we used a preprocessing step to clean up the hotel room images and make them more suitable for image recognition tasks. We manually verified the types and number of available categories for classification.

Transfer learning
When it comes to deep learning models, you usually start with a pre-trained model and then tweak it for your specific task. This is known as transfer learning. It can be a helpful strategy when you don't have enough data or when your data is not suitable for training a model from scratch. Only a few top layers of weights are responsible for choosing the proper class. All layers below recognize elements with a smaller level of abstraction, i.e., gradients, curves, lines, pixels, etc. So we only have to retrain the top layers from scratch and slightly adjust the others. We used a pre-trained deep learning model initially trained on the ImageNet dataset. This saved us a lot of time and resources, resulting in good accuracy. You can read more about our approach to transfer learning here.

Confusing categories from test set
During model training, we identified several shortcomings. Our model struggled with correctly classifying some of the categories like front hotel vs. outdoor, reception vs. lobby, or bar vs. restaurant. Some bars may be confused with restaurants, as well as a lobby and reception. Another challenge was that the model sometimes had trouble distinguishing between a hotel image and an image of a nearby attraction. This is because some hotels are located close to popular tourist destinations, and it cannot be easy to distinguish which picture is from the hotel and which image is from the attraction. To overcome this, we worked on a better category setup and removing unnecessary classes.

Our other projects:

Insurance Sector

Auto Insurance Fraud Claim Detection

Developing an AI model that can detect car insurance fraud based on car accident claims?
Challenge Accepted!

Fraud Detection Predictive Modeling

Fintech Sector

Loan Application Fraud Detection

Developing a model that can identify fraudulent loan applications for a financial institution?
Challenge Accepted!

Fraud Detection Predictive Modeling

Fintech Sector

Predicting Bank Loan Defaults for Profit Maximization

Developing an AI model for loan default prediction and using it to increase bank profits?
Challenge Accepted!

Predictive Modeling

See more projects

Find us on

Need help with implementing AI in your business?

Let's talk blue circle

This site uses cookies for analytical purposes.

Accept Privacy Policy

In the interests of your safety and to implement the principle of lawful, reliable and transparent processing of your personal data when using our services, we developed this document called the Privacy Policy. This document regulates the processing and protection of Users’ personal data in connection with their use of the Website and has been prepared by Nexocode.

To ensure the protection of Users' personal data, Nexocode applies appropriate organizational and technical solutions to prevent privacy breaches. Nexocode implements measures to ensure security at the level which ensures compliance with applicable Polish and European laws such as:

  1. Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing Directive 95/46/EC (General Data Protection Regulation) (published in the Official Journal of the European Union L 119, p 1); Act of 10 May 2018 on personal data protection (published in the Journal of Laws of 2018, item 1000);
  2. Act of 18 July 2002 on providing services by electronic means;
  3. Telecommunications Law of 16 July 2004.

The Website is secured by the SSL protocol, which provides secure data transmission on the Internet.

1. Definitions

  1. User – a person that uses the Website, i.e. a natural person with full legal capacity, a legal person, or an organizational unit which is not a legal person to which specific provisions grant legal capacity.
  2. Nexocode – NEXOCODE sp. z o.o. with its registered office in Kraków, ul. Wadowicka 7, 30-347 Kraków, entered into the Register of Entrepreneurs of the National Court Register kept by the District Court for Kraków-Śródmieście in Kraków, 11th Commercial Department of the National Court Register, under the KRS number: 0000686992, NIP: 6762533324.
  3. Website – website run by Nexocode, at the URL: nexocode.com whose content is available to authorized persons.
  4. Cookies – small files saved by the server on the User's computer, which the server can read when when the website is accessed from the computer.
  5. SSL protocol – a special standard for transmitting data on the Internet which unlike ordinary methods of data transmission encrypts data transmission.
  6. System log – the information that the User's computer transmits to the server which may contain various data (e.g. the user’s IP number), allowing to determine the approximate location where the connection came from.
  7. IP address – individual number which is usually assigned to every computer connected to the Internet. The IP number can be permanently associated with the computer (static) or assigned to a given connection (dynamic).
  8. GDPR – Regulation 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of individuals regarding the processing of personal data and onthe free transmission of such data, repealing Directive 95/46 / EC (General Data Protection Regulation).
  9. Personal data – information about an identified or identifiable natural person ("data subject"). An identifiable natural person is a person who can be directly or indirectly identified, in particular on the basis of identifiers such as name, identification number, location data, online identifiers or one or more specific factors determining the physical, physiological, genetic, mental, economic, cultural or social identity of a natural person.
  10. Processing – any operations performed on personal data, such as collecting, recording, storing, developing, modifying, sharing, and deleting, especially when performed in IT systems.

2. Cookies

The Website is secured by the SSL protocol, which provides secure data transmission on the Internet. The Website, in accordance with art. 173 of the Telecommunications Act of 16 July 2004 of the Republic of Poland, uses Cookies, i.e. data, in particular text files, stored on the User's end device.
Cookies are used to:

  1. improve user experience and facilitate navigation on the site;
  2. help to identify returning Users who access the website using the device on which Cookies were saved;
  3. creating statistics which help to understand how the Users use websites, which allows to improve their structure and content;
  4. adjusting the content of the Website pages to specific User’s preferences and optimizing the websites website experience to the each User's individual needs.

Cookies usually contain the name of the website from which they originate, their storage time on the end device and a unique number. On our Website, we use the following types of Cookies:

  • "Session" – cookie files stored on the User's end device until the Uses logs out, leaves the website or turns off the web browser;
  • "Persistent" – cookie files stored on the User's end device for the time specified in the Cookie file parameters or until they are deleted by the User;
  • "Performance" – cookies used specifically for gathering data on how visitors use a website to measure the performance of a website;
  • "Strictly necessary" – essential for browsing the website and using its features, such as accessing secure areas of the site;
  • "Functional" – cookies enabling remembering the settings selected by the User and personalizing the User interface;
  • "First-party" – cookies stored by the Website;
  • "Third-party" – cookies derived from a website other than the Website;
  • "Facebook cookies" – You should read Facebook cookies policy: https://www.facebook.com/policy/cookies
  • "Other Google cookies" – Refer to Google cookie policy: www.google.com/policies/technologies/types/

3. How System Logs work on the Website

User's activity on the Website, including the User’s Personal Data, is recorded in System Logs. The information collected in the Logs is processed primarily for purposes related to the provision of services, i.e. for the purposes of:

  • analytics – to improve the quality of services provided by us as part of the Website and adapt its functionalities to the needs of the Users. The legal basis for processing in this case is the legitimate interest of Nexocode consisting in analyzing Users' activities and their preferences;
  • fraud detection, identification and countering threats to stability and correct operation of the Website.

4. Cookie mechanism on the Website

Our site uses basic cookies that facilitate the use of its resources. Cookies contain useful information and are stored on the User's computer – our server can read them when connecting to this computer again. Most web browsers allow cookies to be stored on the User's end device by default. Each User can change their Cookie settings in the web browser settings menu: Google ChromeOpen the menu (click the three-dot icon in the upper right corner), Settings > Advanced. In the "Privacy and security" section, click the Content Settings button. In the "Cookies and site date" section you can change the following Cookie settings:

  • Deleting cookies,
  • Blocking cookies by default,
  • Default permission for cookies,
  • Saving Cookies and website data by default and clearing them when the browser is closed,
  • Specifying exceptions for Cookies for specific websites or domains

Internet Explorer 6.0 and 7.0
From the browser menu (upper right corner): Tools > Internet Options > Privacy, click the Sites button. Use the slider to set the desired level, confirm the change with the OK button.

Mozilla Firefox
browser menu: Tools > Options > Privacy and security. Activate the “Custom” field. From there, you can check a relevant field to decide whether or not to accept cookies.

Opera
Open the browser’s settings menu: Go to the Advanced section > Site Settings > Cookies and site data. From there, adjust the setting: Allow sites to save and read cookie data

Safari
In the Safari drop-down menu, select Preferences and click the Security icon.From there, select the desired security level in the "Accept cookies" area.

Disabling Cookies in your browser does not deprive you of access to the resources of the Website. Web browsers, by default, allow storing Cookies on the User's end device. Website Users can freely adjust cookie settings. The web browser allows you to delete cookies. It is also possible to automatically block cookies. Detailed information on this subject is provided in the help or documentation of the specific web browser used by the User. The User can decide not to receive Cookies by changing browser settings. However, disabling Cookies necessary for authentication, security or remembering User preferences may impact user experience, or even make the Website unusable.

5. Additional information

External links may be placed on the Website enabling Users to directly reach other website. Also, while using the Website, cookies may also be placed on the User’s device from other entities, in particular from third parties such as Google, in order to enable the use the functionalities of the Website integrated with these third parties. Each of such providers sets out the rules for the use of cookies in their privacy policy, so for security reasons we recommend that you read the privacy policy document before using these pages. We reserve the right to change this privacy policy at any time by publishing an updated version on our Website. After making the change, the privacy policy will be published on the page with a new date. For more information on the conditions of providing services, in particular the rules of using the Website, contracting, as well as the conditions of accessing content and using the Website, please refer to the the Website’s Terms and Conditions.

Nexocode Team