Licious’s Control Tower: Real-Time Supply-Chain Anomaly Detection with CEP

Team: Kartik Manas Srivastava, Mrinal, Venkateshaprabhu, Shivangi, Soumya, Chetali.

Modern e-commerce and supply chain operations generate a deluge of events — from inventory updates and payment transactions to delivery tracking and user actions. Making real-time sense of these events for anomalies is critical. Licious’s Control Tower is a developer-friendly alerting and automation platform built to tackle this challenge. It uses Apache Flink for Complex Event Processing (CEP) at its core, enabling pattern detection on streaming events with sub-second latency. Surrounding the Flink engine is a suite of microservices (enrichment, workflow, task management, and a BFF layer) that together orchestrate cross-functional alerts and even automated remediation.

In this article, we’ll dive into the Control Tower’s architecture, how it detects anomalies across domains like Inventory, Payments, Mid Mile, Last Mile, and Delivery SLA, and walk through a concrete “Cart Abandonment” use case.

Architecture Overview

Licious’s Control Tower system can be understood in two layers:

(1) a streaming CEP engine built on Apache Flink, which continuously monitors event streams and raises alerts, and (2) a set of Control Tower services that manage configurations, enrich alert data, handle workflows for alerts, and expose a UI/backend interface. Below is a breakdown of the key components:

Streaming CEP Engine

At the heart of Control Tower is Apache Flink, a distributed stream processing engine. Using Flink’s CEP capabilities, the system can detect complex event patterns in an endless stream of events. Each alerting rule is defined as a CEP job that consumes one or more event streams (from Kafka topics) and applies a chain of processors (operators) to identify a pattern or anomaly in real-time. When a pattern match (or anomaly) is detected, the job emits an alert event to downstream sinks.

The CEP jobs leverage several types of operators and stream constructs accessible via platform:

ReadStream: entry point operator to read from an input stream (e.g. a Kafka topic of events). Each stream is defined in the system (with schema) and pulled from the event bus (Kafka).

Filter: filters events by certain criteria (e.g. only events from a specific city or of a certain type). This helps narrow the stream to relevant signals for the rule.

Window: groups events over a time window (e.g. a 5-minute sliding window) for aggregate conditions. For example, a window could detect if an event count exceeds a threshold in that period.

Union/Join/Connect: merges multiple streams or correlates events from different sources. This is useful if an anomaly pattern spans different event types (like correlating payment events with order events).

Custom Timer: a custom CEP operator that schedules future checks or triggers if an expected follow-up event doesn’t occur in time. Timers allow the “absence” of an event to be treated as a pattern. (This is essentially how we detect scenarios like no order placed after adding to cart or rider stationary for X minutes during order delivery.)

Pattern Matching: this functionality in our CEP system utilizes Flink’s CEP library to detect complex patterns in real-time data streams. This feature include identification of specific event sequences, anomalies, or trends within the data.

The Flink jobs run continuously, processing events as they arrive with very low latency. They make use of event-time processing (leveraging timestamps in events) so that patterns are evaluated in the timeline of the actual business events. For instance, if an event carries a timestamp, timers or windows consider that for accuracy even if events arrive slightly out-of-order. Flink’s stateful streaming and timers are key to implementing the CEP rules, enabling real-time anomaly detection, pattern matching, and even fraud detection.

Output Sinks: Once an alert condition is met, the CEP job emits an alert. In Control Tower, each rule’s job defines sinks such as:

• Kafka Sink: Publishes the alert as an event to a Kafka topic on the “Event Bus” (Licious’s Event Platform). The event bus (built on Kafka with a Schema Registry) serves as the central conduit for all events. Downstream services subscribe to these events topics. The event payload typically includes key details of the anomaly (IDs, timestamps, etc.).

• MySQL Sink: Optionally, some alerts are also written to a MySQL database. This can be used for persistent storage or deduplication. For example, the rule can use a primary key (like a session ID or order ID) to ensure the same alert isn’t generated twice for the same entity. It also provides a relational record of alerts that can be queried if needed.

By defining alert logic in CEP jobs, the system achieves true real-time processing — anomalies are detected within seconds of events happening. In production, multiple CEP jobs run in parallel (Flink can run many jobs or one job with multiple rule patterns) to cover a wide range of scenarios concurrently.

Control Tower Services (Alerting & Workflow Management)

While CEP (Flink) does the heavy lifting of detecting anomalies, the surrounding services manage everything around those alert events: configuration, enrichment, workflow orchestration, and user interface. The major services in Licious’s Control Tower include:

• CEP Configuration Service (CT-CEP): This service (part of “CT-CEP” mentioned in above Streaming CEP) allows developers to define and manage alerting rules. It provides APIs (and a UI) to create a new rule, which includes the rule’s JSON definition (the stream sources, operators, conditions, and sinks, as shown later) and associated metadata (alert description, severity, etc.). When a new rule config is created, it’s stored in a configuration database and prepared for deployment. The service can then translate this config into an actual Flink job. Developers can create, update, or delete rules without touching the core code — the Flink jobs are generated dynamically based on configs.

• Rule Deployment & Execution Manager (CT-CEP): Once a rule is defined, developers or system admins trigger its deployment (for example, marking the rule “live” in the UI). The Control Tower will compile the rule into a Flink job and submit it to the Flink cluster (via Flink’s REST API or job manager). This effectively launches the streaming CEP logic in production. The manager also handles stopping jobs when rules are disabled, and can manage versioning (e.g., deploying an updated rule version). This component ensures the Flink CEP layer is dynamic and easily extensible — new patterns can be rolled out quickly by just pushing a new config, rather than redeploying the entire platform.

• Enricher Service: Raw alert events often need additional context before they can be acted upon. The Enricher service is responsible for augmenting alert events with useful information from other systems or databases. When an alert event is triggered on the event bus, the Enricher modify the events to build a full alert payload. For example, an alert might contain just an internal ID or code; the Enricher will look up human-readable details like product name, hub name, or user info. In our Cart Abandonment example, the alert initially has a hubId; the Enricher can query a hub_master table in ClickHouse to retrieve the hubName for that ID. These enrichment queries can be configured as part of the rule definition (as dataRequests in the JSON). The result is a fully populated alert object (with all relevant dimensions) that can be used to inform decision makers or automated tasks.

• Workflow Manager (CT-Workflow): The Control Tower Workflow service acts as the orchestrator for alerts. It subscribes to the alert events from the Kafka event bus. When an alert is received, the Workflow Manager pulls in the necessary configurations for that alert type (e.g., what tasks to create, who to notify) and invokes the Enricher/Alert service (CT-CEP) to get the complete alert details. Then, the workflow service proceeds to initiate the appropriate response. Typically, this means calling the Task Management System (CT-TMS) to create a task or a set of sub-tasks corresponding to the alert. Each alert type can have a predefined remediation workflow — for instance, a “Product Out of Stock” alert might create a task for the Inventory team to replenish stock, whereas a “Payment Gateway Down” alert might create a task for the Developer team to check the payment service. The Workflow Manager ensures each alert triggers the right sequence of actions (tasks, notifications, or even automated scripts) according to customizable workflows. It can also enforce resolution SLAs — for example, if an alert isn’t resolved within a certain timeframe, the Workflow Manager could escalate it (perhaps raising the priority or notifying higher-ups). This timeframe can be configured per alert (the “resolutionThreshold” in the rule config defines how soon it should be addressed).

• Task Orchestrator & TMS: The Task Management System (CT-TMS) is the component that actually records and tracks the tasks and sub-tasks created for alerts. When the Workflow Manager calls CT-TMS, tasks corresponding to the alert are created and assigned to the appropriate team or system. For example, a “Cart Abandonment” alert might automatically create a task in a CRM or marketing system to follow up with the user, whereas an “Inventory Shortfall” alert might create a task for the warehouse manager to initiate stock transfer. The Task Orchestrator logic (within Workflow Manager or TMS) can handle automated workflows — in some cases, it might directly attempt a corrective action. For instance, if an alert indicates a service degradation, an automated workflow might trigger a script to failover to a backup service without waiting for human intervention. Each task or sub-task creation is also published as an event on the event bus, so the entire system remains event-driven and transparent. This means the state of tasks (open, in-progress, resolved) can be listened to by other services or updated on the dashboard in real time.

• Backend-for-Frontend (BFF) & Dashboard Services: To make the Control Tower useful for operations teams and engineers, a frontend UI is provided (the Control Tower Dashboard). The BFF is a layer of services that aggregate data for this UI. Rather than having the UI query many microservices, the BFF provides tailored endpoints that fetch alert data, task data, etc., and format it for front-end consumption. The BFF interacts with the MongoDB data (which has all the alert and task events stored) and with the CT-CEP/Workflow services for any live data or actions. According to the team, the UI services are exposed via CT-UI for simplicity — effectively acting as a gateway to the underlying data. The Control Tower Dashboard then uses these APIs to display all current alerts, their statuses, and details, as well as to allow users to filter, acknowledge, or resolve alerts. (User actions on the dashboard, like marking an alert as resolved, would flow back through the BFF to the TMS or relevant service to update the state).

All these components work in concert. In summary, when an anomaly happens, Flink (CEP) catches it and emits an alert event; the Workflow service picks that up, enriches it, and triggers tasks; tasks are tracked and everything (alerts & task updates) is logged to the event bus and reflected on the dashboard. The design is fully event-driven and real-time. By decoupling detection (Flink) from reaction (Workflow/Tasks) and from visualization (Dashboard), the system is scalable and each part can evolve independently.

Cross-Domain Anomaly Detection in Real Time

One of the biggest advantages of Licious’s Control Tower is its cross-domain coverage. It serves as a unified monitoring and alerting platform across various business functions — from supply chain operations to customer experience. The same system is used to detect and act on anomalies in Inventory, Mid Mile (warehouse and transfers), Last Mile (delivery), Payments, and overall Service SLAs. This cross-functional scope ensures that issues don’t fall through the cracks, and that all teams use a single source of truth for operational alerts.

Business units monitored by Control Tower include Inventory, Hub Operations (Mid Mile), Last Mile, and Payments, among others. Each domain has specific alert types configured.

Let’s look at a few domains and the kind of anomalies the Control Tower CEP rules catch:

• Inventory Alerts: The system watches inventory levels and movements to catch issues like Product Out of Stock situations or Inventory Shortfall at a fulfillment center. For example, if an item’s stock falls below a threshold or an expected inventory update doesn’t arrive (perhaps a procurement order delay), a rule triggers a “Stock Out” alert. This alert would prompt the inventory team to refill or redistribute stock. Such alerts prevent customer orders from failing due to inventory unavailability. (In the dashboard filter above, Product Out Of Stock and Inventory Shortfall are listed as issue types.)

• Mid Mile / Hub Operations Alerts: These cover the internal supply chain between production and distribution hubs. An example is Inward Delay and Outward Delay alerts — if a shipment into a warehouse (inward) or out to another hub/store (outward) is not completed within the expected time window, the system flags it. For instance, suppose a transfer order from the central warehouse to a city hub should have departed by 9:00 AM; if it’s significantly delayed, a “Outbound Delay” alert is generated. This lets the Hub Operations team react (perhaps arranging alternative transport or adjusting downstream schedules). The Control Tower ingests events from WMS (Warehouse Management System) and other supply chain systems (e.g., “sto_outward_delay_events” streams for stock transfer orders) to power these rules. By catching delays early, the team can maintain service levels downstream.

• Last Mile Delivery Alerts: Last mile is critical for customer satisfaction, so the system closely monitors delivery events. A key metric is the Delivery SLA — whether orders are delivered on time. Control Tower implements multi-level alerts for delivery delays. For example, Licious defines thresholds like 0–33% delay (minor lateness), 33–66% delay, and >66% delay (severe lateness). Separate CEP rules track orders that exceed their promised delivery time by these percentages, generating alerts with different priorities. A “0–33% Delay” might create a low-priority alert (just indicating a slight miss), whereas a “>66% Delay” triggers a high-priority alert that could escalate to customer support for recovery actions. The dashboard shows these as Issue Types like “0–33% Delay”, “33–66% Delay”, “>66% Delay” with corresponding priority levels. Another Last Mile example is Rider Stationary alerts — if a delivery executive (rider) hasn’t moved for, say, 10 minutes during an active trip, that could indicate a problem (accident, vehicle issue, etc.). A CEP rule consuming rider_location_events can detect lack of movement and raise an alert so the ops team can call the rider or dispatch help. By monitoring live GPS and order events, the Control Tower gives the Last Mile team instant visibility into problems in the field.

• Payment Alerts: Licious also uses the platform to watch the payment pipeline for anomalies. For instance, a Payment Gateway Failure alert might be defined to trigger if a high rate of payment transactions are failing or if no successful payment event is received within a time window (which could indicate the payment service is down). A CEP rule might consume payment events and order events — if an order is placed but no payment confirmation event arrives in X minutes, it could fire an alert to the Payments/IT team to check the gateway. Likewise, unusual patterns like multiple payment declines for one user in a short time could hint at fraud or technical issues, which the system can flag. These alerts help ensure a smooth customer checkout experience by catching payment issues quickly.

• Service Degradation & SLA Breaches: Beyond the specific domains above, the Control Tower also tracks high-level service metrics. For example, Hub SLA Breach or Service Degradation alerts monitor aggregate performance of a hub or service. If a hub’s order processing falls below a certain success rate, or if the throughput drops below a threshold, an alert is raised. In one screenshot, we see alerts like “Breach 90” and “Breach 85” for a hub — these likely correspond to the hub’s order fulfillment SLA dropping below 90% and then 85%. The system not only catches the drop but also can link related alerts (e.g., earlier breaches that got resolved when metrics recovered) for context. This allows operational leaders to see the health of each node in the supply chain in real time. It’s essentially real-time KPI monitoring implemented as alerts.

Across all these examples, the common thread is real-time pattern detection. The CEP rules are tailored to each domain’s events and normal behavior. Whenever the actual events deviate from the expected pattern (e.g., an event doesn’t happen when it should, or too many events happen out of norm), Control Tower raises a signal. All alerts, whether about a late delivery or a stock shortfall or a payment error, flow into the same dashboard. This unified approach breaks silos — different teams use the same tool and data, fostering faster cross-team collaboration.

Crucially, because the alerts are generated instantaneously, the business can take real-time corrective actions. For example, if a hub outbound delay alert pops up, the mid-mile team might immediately arrange a backup vehicle to ship goods, thus avoiding a cascade of delays. If a payment alert triggers, the tech team might switch traffic to an alternate payment gateway before too many customers are affected. The Control Tower has essentially become the nervous system of Licious’s operations, reflexively detecting pain points and initiating responses.

A snapshot of the Control Tower dashboard showing active alerts across locations and categories. Users can filter by issue type, region, etc. Here we see multiple SLA breach alerts for late orders (with different severity) and an Outward Delay alert at a hub. The dashboard displays how many alerts are open vs. resolved, and selecting an alert shows its details (e.g., expected vs actual times, affected order/hub) on the right.

A snapshot of the Control Tower integrated workflow for degrading Service SLA directly from the control tower dashboard.

Use Case Walkthrough: “Cart Abandonment” Alert

To illustrate how the Control Tower works end-to-end, let’s walk through a concrete rule: Cart Abandonment Alert. This is a classic e-commerce scenario — a user adds items to their cart but does not complete the order. Licious wanted to detect such cases in real time, so that marketing or product teams can intervene (for example, by triggering an in-app reminder, push notification, or logging it for analysis of drop-offs).

Business logic: If a user fires an “Add to Cart” event but does not generate an “Order Placed” event within a specified time (say 15 minute), flag it as a cart abandonment. This pattern involves an absence of an event (order) after another event (add-to-cart), which is a perfect use case for CEP.

Data source: The events from the app/microservices (which include events like ADD_TO_CART, CHECKOUT_CLICKED, ORDER_COMPLETED, etc.) are streamed via Kafka.

Flink CEP configuration: Using Control Tower’s rule DSL, the Cart Abandonment rule is set up with the following JSON structure (simplified for clarity):

{
  "jobName": "cart_abandonment",
  "description": "Cart Abandonment Alert. This is sample configuration.",
  "ruleGroup": { "name": "cep_consumer_growth" },

  "operators": [
    {
      "name": "cart_stream",
      "processor": {
        "type": "ReadStream",
        "streamName": "cart_map"
      }
    },
    {
      "name": "order_stream",
      "processor": {
        "type": "ReadStream",
        "streamName": "order_map"
      }
    },
    {
      "name": "union_streams",
      "processor": {
        "type": "Union",
        "firstOperator": "cart_stream",
        "secondOperator": "order_stream",
        // everything is keyed by the same customer-level key
        "partitionKeys": ["customer_key"]
      }
    },
    {
      "name": "cart_abandonment_pattern",
      "processor": {
        "type": "Pattern",
        "patterns": [
          {
            "patternName": "cart_event",
            "patternType": "begin",
            "condition": {
              "type": "EQUAL",
              "leftParam": { "fieldName": "eventType" },
              "rightParam": { "value": "ADD_TO_CART" }
            }
          },
          {
            "patternName": "order_event",
            "patternType": "notFollowedBy",
            "condition": {
              "type": "EQUAL",
              "leftParam": { "fieldName": "eventType" },
              "rightParam": { "value": "ORDER_CREATED" }
            }
          }
        ],
        "time": { "unit": "SECONDS", "size": 900 }
      }
    }
  ],

  "sinks": [
    {
      "name": "event_bus_sink",
      "processor": {
        "type": "EventBusSink",
        "eventName": "CART_ABANDONMENT",
        "eventType": "cep_alert",
        "scmVersion": 2,
        "clientId": 1,
        "outputFields": [
          "customer_key",
          "session_id"
        ]
      }
    }
  ]
}

Let’s break down what this rule is doing:

Stream ingestion (Source):
cart_stream and order_stream are two ReadStream operators. cart_stream subscribes to the cart_map topic, where every “ADD_TO_CART” event is published. order_stream subscribes to the order_map topic, which carries “ORDER_CREATED” events. The Union operator then merges the two keyed streams and partitions them by customer_key. Because both streams share the same key, all events for one shopper land in the same Flink task. This isolation is important: it prevents one customer’s ADD_TO_CART from being cancelled by another customer’s ORDER_CREATED and guarantees correct per-customer correlation.

CEP pattern logic:
The Pattern operator expresses the classic “no order after add-to-cart” rule, but in event-time instead of explicit timers:

1. Begin pattern (cart_event) – waits for an event where eventType == "ADD_TO_CART".

2. Not-followed-by pattern (order_event) – asserts that no event with eventType == "ORDER_CREATED" arrives within 15 minute for the same customer_key.

If the 15-minute window expires without an order, the pattern fires. In effect, Flink creates an implicit timer when the ADD_TO_CART appears and cancels that timer if an ORDER_CREATED shows up before the deadline; otherwise a cart-abandonment match is emitted.

Sinks (alert output): The sink is the EventBusSink, which publishes the alert to the central Kafka bus. We see it specifying an eventName “CART_ABANDONMENT” and a Kafka topicName “cep.cart_abandonment”. The event will be tagged as a production CEP alert.

With this configuration in place, the developer can deploy the rule. When deployed, CT-CEP translates this JSON into an actual Flink job. The job connects to the Kafka topic and starts processing events.

Suppose a user in Bangalore adds a product to cart at time T. The Flink job sees an ADD_TO_CART event and sets a timer for T+15 minute. If 60 seconds or later an ORDERED event appears for the same session, the timer is deleted (no alert). But if no order event comes, at T+15 minute the timer fires and the job emits an alert event to Kafka. This alert event will have the name CART_ABANDONMENT and carry the user’s ID and cart details.

On the Control Tower Dashboard, this alert might immediately show up as a new issue (perhaps categorized under a “Conversion” or “Consumer” group since it’s customer-facing). The alert would likely be of low priority (as tagged) because cart abandonment is common, but it’s useful to track trends. The Workflow Manager could be configured not to create a manual task for each such alert (since there could be many), but maybe to batch them or log them. In some cases, the response might be automated — e.g., trigger an automated email or in-app message to the user saying “Need help completing your order?”. The flexibility of Control Tower means the action taken on an alert can vary: some alerts create human tasks, others trigger system workflows. In our case, it might integrate with a marketing automation system to handle cart recovery prompts.

From a developer’s perspective, this use case demonstrates how a fairly complex temporal pattern (“event A followed by NOT event B within X minutes”) can be configured through the rule JSON without writing custom Java code. Flink’s CEP under the hood manages the event correlation and timing. The result is a real-time alert that surfaces a valuable insight (potential lost sale) to the business within minutes of it happening.

Real-Time Decisions and Extensibility

The impact of Licious’s Control Tower is significant. It has transformed operations from reactive fire-fighting to proactive monitoring. Thanks to the real-time decision-making enabled by Flink CEP, the company can catch issues in seconds rather than hours. This reduces order failures, improves customer experience (e.g., fewer late deliveries going unnoticed), and even helps in fraud prevention. Engineers and operations managers get a live dashboard of everything going wrong (or likely to go wrong) across the organization, which acts like a command center for the business.

The system is also highly extensible. New event streams can be plugged in (for instance, if Licious starts tracking a new IoT sensor in warehouses, it can be added as a stream and new rules written for its data). New rules can be created for new anomalies without disrupting existing pipelines. The modular rule design (each with its own Flink operators and sinks) means one can build a library of patterns for different needs. In fact, Licious’s use of Flink CEP and a data lake architecture has opened up a range of real-time analytics and alerting use cases across their supply chain and digital operations.

Moreover, cross-functional alerting breaks down data silos. The inventory team, the logistics team, and the customer service team all look at the same Control Tower dashboard, albeit focusing on their respective alerts. This ensures that when an issue in one domain affects another, everyone is aware through a unified platform. For example, a payment gateway issue might eventually cause order delays — since both Payments and Last Mile teams see alerts in one place, they can coordinate more easily.

Finally, the Control Tower’s design balances automation and human action. Some alerts result in fully automated workflows (no human needed), while others require human intervention. By having both, Licious can gradually automate more as confidence grows. The Task Orchestrator can start taking over repetitive remedial steps (like auto-restarting a service, auto-ordering stock replenishment, etc.), leaving humans to handle the exceptions or more complex decisions. This hybrid approach ensures reliability while leveraging real-time data.

In conclusion, Licious’s Control Tower exemplifies a modern real-time alerting system built on streaming technology. It showcases how Apache Flink CEP can be harnessed beyond simple analytics, driving operational intelligence and actions in real time. The developer-friendly rule definitions and the rich ecosystem of services around Flink make it possible to maintain and evolve this system efficiently. For backend and data engineers, this is a great case study of marrying stream processing with microservices to deliver a control tower , a bird’s-eye view and control of the entire business in motion, where insights instantly lead to action.