Building Careem’s Integrity Feature Platform (Part 1: The Foundation)

How Careem’s Integrity team serves several hundred fraud-detection features in single-digit milliseconds and ships new ones in hours instead of days

Every day at Careem, an incredibly high volume of transactions occurs across ride-hailing, food delivery, and digital payments. Protecting these touchpoints from fraud, without introducing friction that hurts the user experience is a massive engineering challenge.

The Integrity (Fraud) team at Careem is responsible for providing risk assessments across these various touchpoints in the user journey. Because fraud attacks evolve rapidly, the system relies heavily on Machine Learning models and rules to evaluate behavior.

Crucially, these risk checks happen synchronously. The system must evaluate the risk of a transaction in real-time before allowing the user journey to proceed.

2. Key Concepts & Terminology

To understand the scope of the problem, it is important to define exactly what happens during a risk assessment.

What is a “Transaction”?

In the context of Careem Integrity, a transaction is not just a financial payment. It is any critical touchpoint or action a user takes within the Careem Super App ecosystem.

• Examples of Transactions:
  • A user requests a ride (Ride-Hailing).
  • A user places a food order (Careem Food).
  • A user attempts to add a new credit card (Careem Pay).

What is a “Feature”?

A feature is a quantifiable data point or signal that describes historical or current behavior. Features provide the context needed to assess the risk of a transaction. In our Integrity platform, features generally fall into three subtypes:

• Scalar: A simple, single-value data point (e.g., user_account_age_days).
• Time-Series / Windowed: An aggregation computed over a specific time window (e.g., user_cancellation_count_30_days or device_velocity_24_hours).
• Set / Collection: A group of unique values associated with an entity (e.g., a set of all phone numbers linked to a specific user).

What is a “Rule”?

A rule is the actual logic that consumes one or more features to make a single decision. Rules are authored by Data Scientists and Fraud Analysts.

• Example of a Rule:
  • IF (user_cancellation_count_30_days > X) AND (user_account_age_days < Y) THEN Block_Transaction

Importantly, a single Transaction evaluates one or more rules simultaneously in parallel. The individual results of these rules are then combined to determine the final, unified transaction response (e.g., Allow, Block, or Require 2FA). To execute all these rules efficiently, the Integrity system must seamlessly fetch the union of all required Features.

3. The Core Challenges

Building a system to supply features for synchronous risk checks presents a unique set of competing constraints:

Challenge 1: The Heavy Aggregation Problem

Fraud models and rules rely heavily on historical behavior to establish a baseline. We routinely need to evaluate data spanning multiple months or even years (e.g., “number of trips taken by user in the last 2 years”). Querying years of transactional data at the exact moment a user requests a ride is computationally prohibitive.

Challenge 2: The Latency Constraint (Protecting UX)

During a synchronous transaction (e.g., a ride request), the system must fetch a large volume of data—often 50 to 100+ features—to evaluate all active rules in parallel. Because the Careem client is waiting synchronously for this response, high latency directly degrades the User Experience (UX). The overall end-to-end (E2E) execution latency for a transaction is strictly bounded to fractions of a second. To fit the feature fetching step within this tight budget without becoming the bottleneck, doing complex, on-the-fly database joins is impossible.

Challenge 3: The Freshness Constraint (Data Delay)

Latency (execution speed) is different from Data Delay (freshness). While fetching features must be lightning fast, the underlying data itself must also be sufficiently up-to-date. For the vast majority of features, a data delay of a few hours or even days is perfectly acceptable (e.g., long-term historical aggregations). However, for a small subset of highly critical features (e.g., detecting an ongoing velocity attack), we expect the freshness delay to be at most a few seconds. Balancing this need for near real-time freshness for specific features alongside massive batch aggregations is a significant architectural hurdle.

Challenge 4: The Agility Bottleneck (Time-to-Market)

Fraud is adversarial. When a new fraud vector is identified, the team needs to create new features and update rules immediately. If onboarding a new feature requires long engineering sprints, database migrations, or heavy API code changes, the response is blocked, and the business loses money to fraud during the delay.

4. The Essential Building Blocks of a Feature Platform

Before discussing specific technologies, any successful Feature Platform must possess five distinct conceptual building blocks:

Block A: Feature Registry (The “What”)

A single source of truth that defines what a feature is and what it belongs to. In the registry, we explicitly define core Entities (e.g., User, Captain, Device). Engineers, collaborating with Data Scientists, then define a Feature, link it to a specific Entity, and link it to the underlying Computation logic (which we will describe later). This acts as a centralized catalog, ensuring features are discoverable, reusable, and version-controlled.

Block B: Feature Computation (The “How it’s built”)

The engine responsible for turning raw data (events, transactional databases) into the defined features. This can be broken down into:

• Batch Computation: Expressed as SQL queries running against our Hive data warehouse. These run periodically based on scheduled intervals to process historical data.
• Streaming Computation: Also expressed as SQL queries, but these run continuously on top of Kafka streams to process real-time events.
• Unified Output (updates): Crucially, both Batch and Streaming computations do not write absolute, final states directly to a database. Instead, they emit their computed updates as “updates” to a centralized Kafka topic. The exact nature of a update depends on the feature type (e.g., a new value to replace an old scalar, a new item to append to a set, or a time-windowed increment).

Block C: Orchestration (The “How it’s structured”)

An automated control plane that bridges the Registry and the storage layer. Its primary responsibility is provisioning the Online DB infrastructure needed to support the features. It translates feature configurations (entities, namespaces, data types) into actual infrastructure by dynamically creating the required tables in DynamoDB.

Block D: Persistence Layer (The “How it gets saved”)

The active worker layer responsible for taking the computed feature updates and persisting them. It handles the core composition logic: merging the incoming update from Kafka with the existing value currently present in DynamoDB. How this merge happens is entirely dictated by the feature type (e.g., replacing a Scalar, appending to a Set, or adding/managing time buckets for a Windowed feature).

Block E: Feature Serving (The “How it’s consumed”)

The API or SDK that the Integrity risk models and rules engine call during a transaction. It abstracts away the underlying storage and provides a simple interface: “Give me these 50 features for user X.”

5. How We Built It (Part 1 Architecture)

To unblock our Data Scientists while strictly adhering to the synchronous latency constraints, our architecture fundamentally relies on Pre-computation. By computing the heavy aggregations ahead of time, we ensure the synchronous serving path remains lightweight.

5.1. Solving the Heavy Aggregation & Freshness Problems (Computation & Ingestion)

• Shift-Left Computation (Batch): To solve the heavy aggregation problem, we do not compute long-window aggregations from raw data on-the-fly. Instead, scheduled batch jobs run in Hive to compute incremental daily results (updates). When a risk check occurs, the system efficiently combines the pre-computed updates to yield the final value.
• Micro-Updates (Streaming): To solve the freshness problem for highly critical features, we support streaming jobs running on Kafka. These jobs compute micro-updates at incredibly high frequencies—every 1 second or even smaller. This ensures the Online Store receives near-instantaneous updates for real-time velocity checks. (Note: The deep mechanics of these computation, persistence, and inference optimizations warrant their own dedicated blog post later in this series!)
• The Persistence Layer (Type-Aware Composition): As mentioned, all computations emit updates to Kafka. A dedicated Persistence Layer consumes these topics and executes the composition logic against the Online Store. If it’s a Scalar feature, it simply replaces the old value. If it’s a Set, it appends the new value. If it’s a Windowed feature, it manages the necessary time buckets in the DB. This entirely decouples computation logic from storage mechanics.

5.2. Solving the Latency Constraint (Storage & Serving)

• The Storage Layer (DynamoDB): The Persistence Layer applies all merged updates into DynamoDB, which serves as our high-speed feature storage.
• O(1) Lookups & Single-Digit ms Latency: By the time a transaction occurs, the heavy lifting is already done. The Serving layer simply executes a direct, highly optimized lookup against DynamoDB, fetching the pre-computed features and achieving ultra-fast, single-digit millisecond latency at P95.

5.3. Solving the Agility Bottleneck (The Registry)

• Centralized Registry: We implemented a configuration-driven approach. Currently, Engineers (Devs) define the feature logic in the registry, collaborating closely with DS on the requirements.
• Automated Orchestration: A separate orchestration layer automatically detects these configurations. It handles data type management and dynamically provisions new tables in DynamoDB based on the specified Entity and Namespace.
• Dynamic Onboarding: Because of this automation, new features are rolled out to production without requiring any new code deployments or service restarts, vastly accelerating time-to-market.

To give you an idea of how this looks in practice, Developers define Features using a declarative Python framework. This explicitly maps the Entity, the dual Computation engines (Batch + Streaming), and how the data should be bucketed and materialized online:

# 1. Define the Entity
user_entity = Entity(
    namespace="integrity_namespace",
    name="user",
    keys=[EntityKey(field_name="user_id", data_type=FeatureDataType.INT)]
)

# 2. Define the Computations (Streaming & Batch)
streaming_calc = StreamingCalculation(
    name="user_completed_trips_streaming",
    sql_query="""
        SELECT user_id, COUNT(*) AS trip_count
        FROM ${topic:dummy_events_topic}
        WHERE status = 'COMPLETED'
        GROUP BY user_id
    """,
    entities=[user_entity]
)

batch_calc = BatchCalculation(
    name="user_completed_trips_batch",
    sql_query="""
        SELECT user_id, COUNT(trip_id) AS trip_count
        FROM dummy_db.dummy_fact_table
        WHERE status = 'COMPLETED'
          AND day >= date(cast(${var:startTime} AS TIMESTAMP))
          AND day < date(cast(${var:endTime} AS TIMESTAMP))
        GROUP BY user_id
    """,
    aggregation_period=timedelta(days=1),
    entities=[user_entity]
)

# 3. Define the Feature and its Materialization
user_completed_trips = Feature(
    namespace="integrity_namespace",
    name="user_completed_trips",
    data_type=FeatureDataType.INT,
    feature_type=FeatureType.TIMESERIES,
    entities=[user_entity],
    calculations={
        TimeseriesLabel.REALTIME: FeatureCalculation(
            calculation=streaming_calc,
            field_name="trip_count"
        ),
        TimeseriesLabel.HISTORICAL: FeatureCalculation(
            calculation=batch_calc,
            field_name="trip_count"
        )
    },
    online_materialization=OnlineMaterialization(
        format=BucketedFeatureDetails(
            timeseries={
                TimeseriesLabel.REALTIME: [
                    BucketChain(duration=timedelta(minutes=10), count=6), # 1 hour
                    BucketChain(duration=timedelta(hours=1), count=24)    # 24 hours
                ],
                TimeseriesLabel.HISTORICAL: [
                    BucketChain(duration=timedelta(days=1), count=30)     # 30 days
                ]
            },
            merging=TimeseriesMerging.ADD
        ),
        storage=OnlineStorageDetails(expected_reads_per_second=5000)
    )
)

Once this code is merged, the Orchestrator provisions the DynamoDB table based on the BucketChain materialization configs, and schedules both the Kafka streams and Hive jobs to begin emitting their respective updates.

6. What We Achieved (The Impact)

By decoupling our computation from serving and adopting this orchestration-driven architecture, we realized several massive wins for the Integrity team:

• Lightning-Fast Time-to-Market: The time required to onboard a new feature from a known data source plummeted from 2–3 days down to just 1–2 hours. In the adversarial world of fraud detection, this speed is the difference between stopping an attack and suffering significant financial loss.
• Complex Features Under Budget: We can now support heavy historical aggregations (e.g., “number of trips taken by a user in the last 6 months”) effortlessly. Previously, attempting this would have destroyed our latency; now, because of pre-computation, it happens well within our tight E2E latency budget.
• Independent Scaling: Because the architecture cleanly separates the engines, we can scale and optimize Computation (Hive/Kafka), Persistence (the merger workers), and Serving (the API/DynamoDB) entirely independently based on their unique bottlenecks.

7. What’s Next?

In this post, we laid out the foundational architecture that allows us to rapidly build and deploy features for synchronous risk checks. However, to achieve strict single-digit millisecond latency while merging massive volumes of high-frequency Kafka updates, we had to optimize the Persistence and Serving layers, while ensuring they remained scalable enough to support multiple feature types and different entities.

In Part 2 of this series, we will take a deep dive into the engineering behind Feature Persistence & Serving Optimization, specifically how we use BucketChains and how historical and real-time components are seamlessly merged during serving to form the true heart of our Feature Platform. Stay tuned.

Acknowledgments: Huge thanks to the core engineers who collaborated closely on the foundational design and implementation of this architecture: Akash Saluja, Ivan Mykhailov, Ali Taher, and the rest of the Careem Integrity team.