Skip to content
Snippets Groups Projects
  • Burak Yavuz's avatar
    80bf48f4
    [SPARK-14555] First cut of Python API for Structured Streaming · 80bf48f4
    Burak Yavuz authored
    ## What changes were proposed in this pull request?
    
    This patch provides a first cut of python APIs for structured streaming. This PR provides the new classes:
     - ContinuousQuery
     - Trigger
     - ProcessingTime
    in pyspark under `pyspark.sql.streaming`.
    
    In addition, it contains the new methods added under:
     -  `DataFrameWriter`
         a) `startStream`
         b) `trigger`
         c) `queryName`
    
     -  `DataFrameReader`
         a) `stream`
    
     - `DataFrame`
        a) `isStreaming`
    
    This PR doesn't contain all methods exposed for `ContinuousQuery`, for example:
     - `exception`
     - `sourceStatuses`
     - `sinkStatus`
    
    They may be added in a follow up.
    
    This PR also contains some very minor doc fixes in the Scala side.
    
    ## How was this patch tested?
    
    Python doc tests
    
    TODO:
     - [ ] verify Python docs look good
    
    Author: Burak Yavuz <brkyvz@gmail.com>
    Author: Burak Yavuz <burak@databricks.com>
    
    Closes #12320 from brkyvz/stream-python.
    80bf48f4
    History
    [SPARK-14555] First cut of Python API for Structured Streaming
    Burak Yavuz authored
    ## What changes were proposed in this pull request?
    
    This patch provides a first cut of python APIs for structured streaming. This PR provides the new classes:
     - ContinuousQuery
     - Trigger
     - ProcessingTime
    in pyspark under `pyspark.sql.streaming`.
    
    In addition, it contains the new methods added under:
     -  `DataFrameWriter`
         a) `startStream`
         b) `trigger`
         c) `queryName`
    
     -  `DataFrameReader`
         a) `stream`
    
     - `DataFrame`
        a) `isStreaming`
    
    This PR doesn't contain all methods exposed for `ContinuousQuery`, for example:
     - `exception`
     - `sourceStatuses`
     - `sinkStatus`
    
    They may be added in a follow up.
    
    This PR also contains some very minor doc fixes in the Scala side.
    
    ## How was this patch tested?
    
    Python doc tests
    
    TODO:
     - [ ] verify Python docs look good
    
    Author: Burak Yavuz <brkyvz@gmail.com>
    Author: Burak Yavuz <burak@databricks.com>
    
    Closes #12320 from brkyvz/stream-python.
__init__.py 3.66 KiB