Real-time feature pipelines are the backbone of many production ML systems. In 2026, teams increasingly pair Apache Kafka for event streaming, Feast for feature-store semantics, and BigQuery as the offline store for analytics and batch training. This guide walks data and analytics engineers through a tested, end-to-end pattern for building a cost-efficient, low-latency feature pipeline that supports real-time serving and consistent historical materializations.

Why this stack?

Kafka provides durable, partitioned event streams with strong ecosystem tooling (Connect, Streams, ksqlDB). Feast standardizes feature definitions, enforces entity keys and manages dual storage (offline + online) semantics. BigQuery is a cost-effective offline store for large historical feature materializations and joins with analytical datasets. Combined, the trio gives you:

  • Strong separation between feature definition (Feast) and execution/serving
  • Low-latency online access (Redis / DynamoDB / Bigtable via Feast online store)
  • Scalable offline materialization and analytics in BigQuery
  • Event-driven freshness using Kafka without adding new control planes

High-level architecture

At a glance, the pipeline has four stages:

  1. Event ingestion: production apps and DBs write user/event streams to Kafka topics.
  2. Feature computation: stream processors (Kafka Streams / ksqlDB / Flink) compute real-time features and emit them into Kafka feature topics or push directly to an online store.
  3. Feature storage: Feast coordinates materialization to an offline store (BigQuery) and an online store (Redis, DynamoDB, Bigtable).
  4. Serving and training: online store used for model inference; offline BigQuery tables used for model training and backfills.

Design decisions and trade-offs

Before implementation, align on these choices:

  • Freshness SLA: Do models need sub-second, second, or minute freshness? Kafka + Feast online store supports sub-second writes, but costs and operational complexity rise.
  • Exactly-once vs idempotent: Kafka + Kafka Streams can provide exactly-once processing in many setups. If your online store uses upserts via idempotent keys timestamps, you can tolerate at-least-once semantics with dedupe logic.
  • Storage for online features: Redis is low-latency and common for small feature sets; DynamoDB/Bigtable scale better for high-cardinality keys at higher cost.
  • Backfills & historical join strategy: BigQuery is ideal for large backfills and analytical joins; plan a repeatable materialization job via Feast to sync features into BigQuery.

Step 1 — Define entities and features in Feast

Create a Feast feature repository to centralize definitions. Use Feast feature views for both streaming and batch features.

Example (simplified) feature view YAML for user-level features:

# feature_repo/feature_repo.yaml
project: recommendation
provider: local
online_store:
  type: redis
offline_store:
  type: bigquery
  project: your-gcp-project
  dataset: feast_offline

# feature_repo/feature_views/user_features.py
from feast import Entity, FeatureView, Field, ValueType
from feast.infra.offline_stores.bigquery import BigQuerySource

user_entity = Entity(name="user_id", value_type=ValueType.INT64, description="User id")

user_events = BigQuerySource(
    table_ref="events.user_events",
    event_timestamp_column="event_ts"
)

user_fv = FeatureView(
    name="user_activities",
    entities=["user_id"],
    ttl=None,
    schema=[
        Field(name="avg_session_length", dtype=ValueType.FLOAT),
        Field(name="last_purchase", dtype=ValueType.UNIX_TIMESTAMP)
    ],
    batch_source=user_events,
)

Key points:

  • Keep feature definitions declarative — other teams can reuse them for training and serving.
  • Define an entity for every join key to ensure consistent online/offline joins.
  • Use BigQuery as the canonical batch source for reproducible offline feature materialization.

Step 2 — Ingest events into Kafka

Design your Kafka topics and partitions thoughtfully:

  • Partition by entity key (e.g., user_id) to localize processing and simplify exactly-once semantics when using Kafka Streams.
  • Use compacted topics for the latest-state topics (e.g., user_profile_compacted) and time-partitioned topics for raw event streams.
  • Include event_time, event_id (for dedupe), schema_version and source metadata in the message envelope.

Use Kafka Connect for CDC from Postgres or MySQL when needed; Confluent Cloud or self-hosted Connectors can push direct to Kafka topics. For high-throughput write paths, use asynchronous producers with batching and compression.

Step 3 — Compute streaming features

You have two viable approaches for streaming feature computation:

  1. Compute features in a stream processor and write the result to a Kafka "feature" topic; Feast then materializes to the online store from that topic.
  2. Stream processor computes features and writes directly into the online store via Feast SDK or API.

Approach 1 provides clearer separation and easier replays (materialization jobs can read feature topics back into BigQuery), while approach 2 reduces end-to-end latency.

Example design using Kafka Streams (Java/Python):

  • Consume raw events topic, enrich with lookup tables if necessary (side inputs), compute aggregates (sliding windows, sessionization).
  • Emit keyed upserts to a compacted Kafka topic named features.user_activities.
  • Include processing metadata: computation_ts, input_offsets, schema_version.

Ensure you handle late arrivals with watermarking and a bounded out-of-order tolerance. For example, maintain a 2-minute lateness window for session aggregates and have a strategy for updating features if late events arrive.

Step 4 — Materialize to offline store (BigQuery)

Feast materialization jobs copy features into BigQuery for training and historical joins. Schedule regular batch materializations and be able to run point-in-time joins for reproducible experiments.

Best practices:

  • Use Feast's point-in-time join support to avoid label leakage.
  • Materialize by date partitions to control cost and make incremental updates efficient.
  • For very large backfills, perform a staged write: write to a temporary BigQuery partition/table, validate row counts and timestamps, then atomically swap or copy to the target table.

Performance tip: BigQuery streaming inserts are convenient but expensive at scale. Prefer batch loads (CSV/AVRO) via GCS when doing daily or hourly materializations.

Step 5 — Online serving and inference

Choose an online store that meets your latency and scale needs. Common choices in 2026:

  • Redis (or Redis Enterprise) for sub-millisecond reads and small feature sets.
  • DynamoDB or Bigtable for high-cardinality keys across many entities.
  • Feast supports each of these as online stores and can be run as a service that proxies reads from your model servers.

Serving patterns:

  • Model servers call Feast SDK to fetch feature vectors at inference time. Cache hot lookups in-process when appropriate.
  • For latency-sensitive inference, prefetch and batch requests; ensure your Feast client supports async batched reads.
  • When models are deployed in large fleets, colocate online stores or caches to reduce egress and network latency.

Backfills, replays and model reproducibility

Backfills are the most operationally expensive step. Provide tooling and runbooks that include:

  • How to run a full historical materialization into BigQuery (with job templates and quotas).
  • A validation step comparing summary statistics (row counts, null rates) between the previous and new materializations.
  • Rollback procedures: keep previous partitions for 30–90 days and use a promotion process to switch training pipelines to the new dataset only after validation.

For reproducibility, store the Feast repo version (git SHA), schema versions, and the materialization timestamp alongside training metadata in your model registry.

Observability and SLOs

Key metrics to monitor:

  • End-to-end freshness: time delta between event ingestion and corresponding feature availability in the online store
  • Feature completeness: per-feature null rate and cardinality drift
  • Ingestion lag and consumer lag for Kafka topics (per partition)
  • Materialization job success rate and BigQuery job cost per job
  • Serving latency (P50/P95/P99) for Feast online reads

Set SLOs such as "95% of features available within X seconds" and use synthetic traffic to validate serving paths. Log schema mismatches and count schema_version mismatches; route alerts that exceed thresholds to a runbook for remediation.

Security, compliance and lineage

Implement data governance from the start:

  • Encrypt data in transit and at rest (Kafka TLS, BigQuery CMEK).
  • Restrict Feast repository and materialization jobs via IAM roles; use least privilege for online stores.
  • Track lineage: record which Kafka topic + offsets contributed to a materialization, and store that metadata with BigQuery partitions and model artifacts.

Cost considerations

Major cost drivers:

  • Kafka cluster throughput and retention — compacted topics for state reduce storage costs.
  • BigQuery storage and query costs — reduce scans with partitioning and clustered tables.
  • Online store operations — Redis instance size or DynamoDB read/write capacity.

Optimization tips:

  • Only publish features that are needed by models; avoid duplicating raw event payloads in feature topics.
  • Tune retention: keep raw events for as long as required for backfills, then expire.
  • Batch materializations to reduce BigQuery load instead of frequent small jobs.

Common failure modes and mitigations

  • Late-arriving events causing feature inconsistency: enforce watermarking, maintain correction windows and replay tooling.
  • Schema drift: include schema_version in messages and gate materializations when incompatible changes are detected.
  • Consumer lag spikes: autoscale stream processors and monitor partition skew; repartition producer keys if one partition becomes a hotspot.
  • Backfill cost overruns: stage backfills in test projects and use dry-run validations before full materialization.

Operational checklist

  1. Document entity definitions and feature view YAML in a central Feast repo.
  2. Design Kafka topics and partitioning aligned with entity keys.
  3. Implement streaming feature computations with dedupe and watermarking.
  4. Configure Feast offline store as BigQuery and online store according to latency needs.
  5. Automate materializations with CI/CD and store reproducibility metadata.
  6. Instrument freshness, completeness and latency metrics; set SLOs and alerts.
  7. Run regular disaster recovery and backfill drills to validate procedures.

Example timeline for a first production rollout

For a mid-sized ML team, you can expect:

  • Week 1–2: Define entities and initial feature set in Feast; spin up Kafka topics and basic producers.
  • Week 3–4: Implement streaming processors to compute a small set of core features and write to feature topics.
  • Week 5: Configure Feast online and offline stores; run a first materialization into BigQuery and validate.
  • Week 6: Integrate Feast SDK with model serving; run A/B test with offline and online features.
  • Week 7–8: Harden observability, run failover/backfill drills, and finalize runbooks.

Conclusion

Combining Kafka, Feast and BigQuery gives teams a pragmatic path to build feature pipelines that are both realtime-capable and analytically robust. The architecture separates concerns—streaming compute, serving, and analytic materialization—so you can iterate on features without disrupting production serving. Focus on entity design, idempotent upserts, and observability to keep pipelines reliable and cost-effective as usage grows.

Next steps: start a minimal PoC with one entity and 3–5 features, measure end-to-end freshness and cost, then iterate. With clear SLOs and automated materializations, you’ll have a reproducible, operational feature platform ready for production ML in 2026.