Exactly-Once in Spark Structured Streaming: What That Actually Means

"Exactly-once" in Spark Structured Streaming means each input record is processed once by Spark only if the source is replayable and the sink supports idempotent writes or transactions.Spark’s internal checkpointing ensures each micro-batch is executed once, but sinks (or custom write logic) can still introduce duplicates on retries.Using foreachBatch with custom MERGE/upsert logic can produce duplicates unless you detect and skip already-committed batch IDs.Fix: at start of foreachBatch, check the target (Delta transaction log or tracking table) for the batch_id and skip if committed.Checklist: replayable source, durable checkpointing, idempotent sink/writes (or Delta writeStream), and idempotent foreachBatch logic - all four must hold for true end-to-end exactly-once.