Merge branch 'master' of github.com:python-streamz/streamz into 287

Author: CJ-Wright

.coveragerc

@@ -12,3 +12,4 @@ omit =
 
 exclude_lines =
     if __name__ == '__main__':
+    pragma: no cover

.gitignore

@@ -12,3 +12,4 @@ log
 *.swo
 .cache/
 .ipynb_checkpoints/
+.vscode

.travis.yml

@@ -7,7 +7,7 @@ language: python
 
 matrix:
   include:
-    - python: 3.6
+    - python: 3.7
 
 env:
   - STREAMZ_LAUNCH_KAFKA=true

README.rst

@@ -1,11 +1,11 @@
 Streamz
 =======
 
-|Build Status| |Doc Status| |Version Status|
+|Build Status| |Doc Status| |Version Status| |RAPIDS custreamz gpuCI|
 
 Streamz helps you build pipelines to manage continuous streams of data. It is simple to use in simple cases, but also supports complex pipelines that involve branching, joining, flow control, feedback, back pressure, and so on.
 
-Optionally, Streamz can also work with Pandas dataframes to provide sensible streaming operations on continuous tabular data.
+Optionally, Streamz can also work with both `Pandas <https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.html>`_ and `cuDF <https://docs.rapids.ai/api/cudf/stable/>`_ dataframes, to provide sensible streaming operations on continuous tabular data.
 
 To learn more about how to use Streamz see documentation at `streamz.readthedocs.org <https://streamz.readthedocs.org>`_.
 
@@ -21,3 +21,5 @@ BSD-3 Clause
    :alt: Documentation Status
 .. |Version Status| image:: https://img.shields.io/pypi/v/streamz.svg
    :target: https://pypi.python.org/pypi/streamz/
+.. |RAPIDS custreamz gpuCI| image:: https://img.shields.io/badge/gpuCI-custreamz-green
+   :target: https://github.com/jdye64/cudf/blob/kratos/python/custreamz/custreamz/kafka.py

docs/source/api.rst

@@ -95,6 +95,7 @@ Definitions
 
 .. autofunction:: filenames
 .. autofunction:: from_kafka
+.. autofunction:: from_kafka_batched
 .. autofunction:: from_textfile
 
 .. currentmodule:: streamz.dask

docs/source/core.rst

@@ -156,7 +156,8 @@ Branching and Joining
    zip_latest
 
 You can branch multiple streams off of a single stream.  Elements that go into
-the input will pass through to both output streams.
+the input will pass through to both output streams.  Note: ``graphviz`` and
+``networkx`` need to be installed to visualize the stream graph.
 
 .. code-block:: python
 
@@ -377,3 +378,19 @@ For operations like this Streamz adds virtually no overhead.
 
 Streams provides higher level APIs for situations just like this one.  You may
 want to read further about :doc:`collections <collections>`
+
+
+Metadata
+--------
+
+Metadata can be emitted into the pipeline to accompany the data as a list of dictionaries. Most functions will pass the metadata to the downstream function without making any changes. However, functions that make the pipeline asynchronous require logic that dictates how and when the metadata will be passed downstream. Synchronous functions and asynchronous functions that have a 1:1 ratio of the number of values on the input to the number of values on the output will emit the metadata collection without any modification. However, functions that have multiple input streams or emit collections of data will emit the metadata associated with the emitted data as a collection.
+
+
+Reference Counting and Checkpointing
+------------------------------------
+
+Checkpointing is achieved in Streamz through the use of reference counting. With this method, a checkpoint can be saved when and only when data has progressed through all of the the pipeline without any issues. This prevents data loss and guarantees at-least-once semantics.
+
+Any node that caches or holds data after it returns increments the reference counter associated with the given data by one. When a node is no longer holding the data, it will release it by decrementing the counter by one. When the counter changes to zero, a callback associated with the data is triggered.
+
+References are passed in the metadata as a value of the `ref` keyword. Each metadata object contains only one reference counter object.

docs/source/dataframes.rst

@@ -103,8 +103,8 @@ following:
 
        def on_old(self, state, old):
            total, count = state
-           total = total - new.sum()   # switch + for - here
-           count = count - new.count() # switch + for - here
+           total = total - old.sum()   # switch + for - here
+           count = count - old.count() # switch + for - here
            new_state = (total, count)
            new_value = total / count
            return new_state, new_value

docs/source/index.rst

@@ -5,8 +5,7 @@ Streamz helps you build pipelines to manage continuous streams of data.  It is
 simple to use in simple cases, but also supports complex pipelines that involve
 branching, joining, flow control, feedback, back pressure, and so on.
 
-Optionally, Streamz can also work with Pandas dataframes to provide sensible
-streaming operations on continuous tabular data.
+Optionally, Streamz can also work with both `Pandas <https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.html>`_ and `cuDF <https://docs.rapids.ai/api/cudf/stable/>`_ dataframes, to provide sensible streaming operations on continuous tabular data.
 
 To learn more about how to use streams, visit :doc:`Core documentation <core>`.
 

setup.py

@@ -9,7 +9,7 @@
 
 
 setup(name='streamz',
-      version='0.5.2',
+      version='0.5.3',
       description='Streams',
       url='http://github.com/python-streamz/streamz/',
       maintainer='Matthew Rocklin',

streamz/__init__.py

@@ -8,4 +8,4 @@
 except ImportError:
     pass
 
-__version__ = '0.5.2'
+__version__ = '0.5.3'