FLIK Strategy & Roadmap

INT-40-FLIK-Data-Room

2 min read

📄 Initiative Spec: INT-40 FLIK Data Room (Internal Brain)

Status: 🔵 Active (Definition Stage) Owner: azmi Strategic Intent: Establish a Single Source of Truth by consolidating fragmented data into a Data Warehouse (BigQuery) visualized via Metabase.

1. The “Why” (Strategic Context)

  • The Problem: Our data currently lives in siloed MongoDB instances. Running analytical queries on production DBs risks performance, and cross-referencing data (e.g., Shipping vs. Checkout) is manual and error-prone.
  • The Vision: A dedicated Internal Data Room where all production data is piped into BigQuery, cleaned, and made accessible via Metabase for self-service analytics.
  • Value: Instant, reliable metrics without engineering support. “Asking the data” becomes instant.

2. The “What” (Solution Components)

2.1 The Pipeline (ETL)

  • Source: Production MongoDB (Checkout, Shipping, MCA).
  • Destination: Google BigQuery (Data Warehouse).
  • Mechanism: Scheduled ETL jobs (e.g., via Airbyte or custom scripts) running daily/hourly.

2.2 The Semantic Layer (Golden Tables)

  • Raw Data: Direct dumps from Mongo.
  • Transformed Data: Pre-calculated tables for speed and consistency:
    • fact_transactions: Consolidated view of all GMV.
    • dim_merchants: Enriched merchant profiles.
    • fact_loans: MCA performance and repayment status.

2.3 The Visualization Layer (Metabase)

  • Self-Service BI: Connected to BigQuery, allowing non-tech teams to build dashboards.
  • Key Dashboard Modules (Covering Product Data Room):
    • Executive Summary: Headline metrics including Annualized GTV, Active Merchants, and Cumulative Shoppers.
    • Checkout & Growth: GTV Trends, MoM Growth, AOV, and Conversion Funnels (Bolt-style analysis).
    • Merchant Health: Onboarding cohorts (M0-M12 retention), Churn Analysis (Logo vs Dollar), and Multi-product adoption rates.
    • Lending (Investor View): Financing Deployed, Portfolio Outstanding, NPL Rate, and Avg Annualized Yield.
    • Shipping & Ops: Shipment volumes, Commission tracking, and Delivery Success Rates.

3. Implementation Phases

Phase 1: Infrastructure & Ingestion

  1. BigQuery Setup: Provisioning the warehouse.
  2. ETL MVP: Pipeline to replicate core orders and transactions collections to BigQuery (T-1 latency).
  3. Metabase Deploy: Self-hosted Metabase instance connected to BigQuery.

Phase 2: Modeling & Automation

  1. dbt / SQL Modeling: Creating the “Golden Tables” to ensure everyone uses the same definition of “GMV.”
  2. Real-Time Pipelines: Moving critical metrics from T-1 to Near Real-Time (NRT).

4. Key Metrics (ROI)

  • Query Time: Analytical queries take seconds, not minutes.
  • Data Reliability: 100% match between Finance reports and Metabase dashboards.
  • Ad-Hoc Requests: Reduction in “Can you pull this data?” requests to Engineering.

5. Action Items

  • Select ETL tool (Airbyte vs. Custom Script).
  • Provision Google BigQuery project.
  • Deploy Metabase (Docker/K8s).

Graph View