sealed trait WatermarkPropagator extends AnyRef
Interface for propagating watermark. The implementation is not required to be thread-safe, as all methods are expected to be called from the query execution thread. (The guarantee may change on further improvements on Structured Streaming - update implementations if we change the guarantee.)
- Alphabetic
- By Inheritance
- WatermarkPropagator
- AnyRef
- Any
- Hide All
- Show All
- Public
- Protected
Abstract Value Members
- abstract def getInputWatermarkForEviction(batchId: Long, stateOpId: Long): Long
Provide the calculated input watermark for eviction for given stateful operator.
- abstract def getInputWatermarkForLateEvents(batchId: Long, stateOpId: Long): Long
Provide the calculated input watermark for late events for given stateful operator.
- abstract def propagate(batchId: Long, plan: SparkPlan, originWatermark: Long): Unit
Request to propagate watermark among operators based on origin watermark value.
Request to propagate watermark among operators based on origin watermark value. The result should be input watermark per stateful operator, which Spark will request the value by calling getInputWatermarkXXX with operator ID.
It is recommended for implementation to cache the result, as Spark can request the propagation multiple times with the same batch ID and origin watermark value.
- abstract def purge(batchId: Long): Unit
Request to clean up cached result on propagation.
Request to clean up cached result on propagation. Spark will call this method when the given batch ID will be likely to be not re-executed.
Concrete Value Members
- final def !=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def ##: Int
- Definition Classes
- AnyRef → Any
- final def ==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def asInstanceOf[T0]: T0
- Definition Classes
- Any
- def clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.CloneNotSupportedException]) @native()
- final def eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- def equals(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef → Any
- def finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.Throwable])
- final def getClass(): Class[_ <: AnyRef]
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
- def hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
- final def isInstanceOf[T0]: Boolean
- Definition Classes
- Any
- final def ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- final def notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
- final def notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
- final def synchronized[T0](arg0: => T0): T0
- Definition Classes
- AnyRef
- def toString(): String
- Definition Classes
- AnyRef → Any
- final def wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException]) @native()