Facial recognition to detect duplicate Captain accounts

Careem Captains are the backbone of our business, enabling us to simplify and improve the lives of our customers daily. Their dedication ensures a seamless experience for every ride and delivery they make. For this reason, we are committed to supporting their wellbeing and ensuring they have the resources and environment needed to excel. As a marketplace platform, when Captains register with Careem they gain access to a supply of orders, collaborative engagement from restaurant partners in the case of delivery, and 24/7 Captain support.

As is the case with all marketplace platforms, duplicate accounts can be created on occasion. While some duplicates are unintentional, in some cases Captains create multiple accounts with the intention of gaining access to more orders and bonuses. This practice can create imbalances across Careem systems and prevent other Captains from receiving adequate orders as well as affecting the overall ride-hailing and delivery experience for customers. 

To prevent such cases of duplicated accounts, we make sure to inform Captains of the benefits of maintaining a balanced platform for the sake of all stakeholders, when they are onboarded onto the platform. In recent years we have also employed facial recognition technology, powered by machine learning, to help identify duplicate Captain accounts. By detecting suspicious activities proactively, we can identify behaviors that do not align with our values and continue to deliver excellence across all areas of our business, from customers to Captains alike.

Identifying duplicate accounts using facial recognition

Ideally, such cases should be caught during registration, but no control system is perfect. Moreover, merely verifying documents proved insufficient, as photos and documents can be manipulated or tweaked.

To uncover duplicate accounts, we leveraged facial recognition technology to process the Captains’ photos. This technology is widely used for various applications, including security systems, authentication processes, and user-friendly interfaces in consumer electronics. The process starts with detection, where human faces are distinguished from other objects in the image, followed by the identification of these detected faces.

Facial recognition techniques vary from traditional methods to advanced neural networks. When real-time processing is necessary, high inference speed is crucial. In the case of detecting duplicate Captain accounts, we were able to process offline, so latency wasn’t as significant.

What is facial recognition technology?

Facial recognition is the task of identifying or verifying a face in an image or video frame using a pre-existing database of faces. It typically involves two main tasks; face detection, which locates faces within an image or video, and face recognition, which identifies or verifies the person by comparing the detected face to a database of known faces.

Various methods exist for face detection. Classical approaches include finding face parts in the image, extracting facial features, or template matching. The Histogram of Oriented Gradients (HOG) has been popular for a long time due to its high speed and performance. However, modern face detectors typically use Convolutional Neural Networks (CNNs), such as MTCNN, Faceness-Net, Selective Refinement Network, and RetinaFace. HOG remains useful when high speed is necessary, even at the cost of accuracy.

The process of facial recognition technology begins with extracting the embeddings – numerical representations of facial features. These embeddings are then compared to identify or verify the person. If you have a large dataset of faces, you could train models for face recognition, often based on Siamese networks. Many pre-trained models are also available and can be used to extract the embeddings without training.

Embedding comparison is usually done by calculating cosine or Euclidean distance or using Approximate Nearest Neighbors (ANN) algorithms. Cosine and Euclidean distances measure the similarity between two embeddings, while ANN quickly finds the closest matches in a large dataset.

Execution

To acquire and process the images we accessed the Captain photos stored on Amazon S3.

Next, we extracted the embeddings from these images using an open-source model suitable for commercial use. We opted for the face_recognition library due to its popularity, speed, and high performance on diverse datasets, such as the “labeled faces in the wild” dataset. Labeled Faces in the Wild is a database of face photographs designed to study the problem of unconstrained face recognition.

Using a model that works well on this dataset was essential since the Captains’ photos varied in backgrounds, angles, and poses.

Storing the embeddings

Once extracted, the embeddings needed to be stored somewhere. In our early tests with a small data sample, we serialized the embeddings and stored them in a BLOB (Binary Large Object) field in a database. Then, we loaded this data into memory and compared all vector pairs to find matches.

One challenge was the method’s lack of scalability when working with the full dataset of over 2 million images. To manage this greater scale, we transitioned to FAISS-GPU, a library designed for efficient similarity searches. Preliminary research indicated that fraudsters often created multiple accounts, so just searching for the most similar pairs wasn’t sufficient. We discovered that retrieving the top five similar photos covered most scenarios. Building an index for two million embeddings and retrieving the top five most similar vectors for each one took less than an hour.

Analyzing the data

Exploring data thoroughly is crucial when working on machine learning projects since it can deliver a broad range of insights.

Initially, we faced technical issues with image processing. For instance, the library expected images in RGB format, but some were in RGBA format. This problem was solved by converting the images to the correct format. More frequently, the images were of poor quality due to bad lighting, low resolution, noisy backgrounds, or faces occupying only a small section of the image. After experimenting, we primarily addressed these issues using the library’s built-in upsampling function, which enhanced the image quality.

When analyzing the duplicate groups, we employed a connected components algorithm to identify similar photo groups. Visualizing these groups proved interesting and informative. In the plot above, each point is a photo of a Captain. The lines between the points mean that these photos are similar to each other. We identified various types of fraudster clusters. In some instances, multiple accounts shared similar photos. In others, there were chains of accounts with only pairs or triplets of similar photos. While this observation was intriguing, it was not immediately useful compared to the next one.

Unexpectedly, the analysis revealed strange clusters containing hundreds of similar photos. We discovered that these photos were taken in one office, with a portrait on the wall in the background. The model mistakenly identified the portrait as a face, misidentifying such photos as similar. To address this, we detected all faces in every image and used only the largest one for embedding extraction.

Postprocessing and blocking duplicated accounts

Identifying similar pairs of Captains is just one part of our process.

We measure the similarity between each pair of photos using Euclidean distance. To determine if the photos are similar enough to be flagged, we established a threshold on the distance. We sorted the pairs by distance and tested different threshold levels (e.g., 0.05, 0.15, 0.25). We set the final threshold so that nearly all pairs beneath it were genuinely similar, aiming to minimize false positives.

A dedicated team that specializes in reviewing potential fraud cases manually inspected a sample of flagged data. They noted several patterns that should have been excluded. For instance, when a Captain changes limo companies, their old account is closed, and a new one is opened. Even though both accounts share the same information, they are legitimate.

Fraud is typically indicated when similar accounts are both active, and these accounts should be blocked.

The situation becomes less clear when one account is active and the other is closed, and the pair isn’t filtered out by our business rules. Initially, we blocked some of these cases but stopped because each case needed manual verification, and we wished to prevent false blockings.

Outcome of using facial recognition technology

Our fraud detection system using facial recognition technology has proven to be an effective tool in identifying and blocking duplicated accounts that had been fraudulently created. By leveraging advanced machine learning techniques we detected over 2,000 fraudulent accounts, which were blocked after thorough background checks. Through this, we were not only able to protect our business and maintain the integrity of our platform, but also create an environment built on trust for our Captains by eliminating unethical activities that disadvantage those trying to make an honest living. 

The key takeaways from this project include the importance of thorough data analysis, the need for scalable and efficient processing solutions like FAISS-GPU, and the value of a dedicated team for manual review to ensure that, for example, accounts that were duplicated accidentally weren’t blocked.

Moving forward, our registration process has become more stringent, significantly reducing the creation of duplicate accounts. This proactive approach not only prevents duplicate accounts but also builds trust with our Captains and customers. We aim to stay ahead of evolving fraud tactics as we continue to enhance our system, ensuring that our platform remains secure and reliable.

By sharing our journey and insights, we hope to contribute to the broader discussion on fraud detection and inspire others to adopt innovative approaches to combating fraud in their domains.

About the author

Andrei Lukianenko is a Data Scientist working in Careem’s Integrity team. In his free time, he studies foreign languages, reads fantasy books, and writes reviews on modern AI papers.

Check out another article from our Careem Engineering team here. Interested in knowing more about the awesome talent we have at Careem? Click here.