Watch timeout leads to exceptions, but should it?

[nolar]

What is the supposed behaviour of a watch stream with a timeout (if explicitly set or if 10s is used by default)?

Should it raise a low-level socket or requests exception? Or should it just silently exit the iterator?

---

Let's take an example script:

import pykube

cfg = pykube.KubeConfig.from_file()
api = pykube.HTTPClient(cfg)  # timeout=10
stream = pykube.Pod.objects(api).watch()

for event in stream:
    print(repr(event))

The default timeout of pukube is 10s. This also applies to watching GET requests.

When this script is executed, the following exception appears after 10s:

$ python _pykube_watch.py 

WatchEvent(type='ADDED', object=<Pod ...-6bf959c74c-ncm4r>)

Traceback (most recent call last):
  File "/Users/svasilyev/.pyenv/versions/kopf/lib/python3.7/site-packages/urllib3/response.py", line 362, in _error_catcher
    yield
  File "/Users/svasilyev/.pyenv/versions/kopf/lib/python3.7/site-packages/urllib3/response.py", line 668, in read_chunked
    self._update_chunk_length()
  File "/Users/svasilyev/.pyenv/versions/kopf/lib/python3.7/site-packages/urllib3/response.py", line 600, in _update_chunk_length
    line = self._fp.fp.readline()
  File "/Users/svasilyev/.pyenv/versions/3.7.3/lib/python3.7/socket.py", line 589, in readinto
    return self._sock.recv_into(b)
  File "/Users/svasilyev/.pyenv/versions/3.7.3/lib/python3.7/ssl.py", line 1052, in recv_into
    return self.read(nbytes, buffer)
  File "/Users/svasilyev/.pyenv/versions/3.7.3/lib/python3.7/ssl.py", line 911, in read
    return self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/svasilyev/.pyenv/versions/kopf/lib/python3.7/site-packages/requests/models.py", line 750, in generate
    for chunk in self.raw.stream(chunk_size, decode_content=True):
  File "/Users/svasilyev/.pyenv/versions/kopf/lib/python3.7/site-packages/urllib3/response.py", line 492, in stream
    for line in self.read_chunked(amt, decode_content=decode_content):
  File "/Users/svasilyev/.pyenv/versions/kopf/lib/python3.7/site-packages/urllib3/response.py", line 696, in read_chunked
    self._original_response.close()
  File "/Users/svasilyev/.pyenv/versions/3.7.3/lib/python3.7/contextlib.py", line 130, in __exit__
    self.gen.throw(type, value, traceback)
  File "/Users/svasilyev/.pyenv/versions/kopf/lib/python3.7/site-packages/urllib3/response.py", line 367, in _error_catcher
    raise ReadTimeoutError(self._pool, None, 'Read timed out.')
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='kube-1....', port=443): Read timed out.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "_pykube_watch.py", line 7, in <module>
    for event in stream:
  File "/Users/svasilyev/.pyenv/versions/kopf/lib/python3.7/site-packages/pykube/query.py", line 178, in object_stream
    for line in r.iter_lines():
  File "/Users/svasilyev/.pyenv/versions/kopf/lib/python3.7/site-packages/requests/models.py", line 794, in iter_lines
    for chunk in self.iter_content(chunk_size=chunk_size, decode_unicode=decode_unicode):
  File "/Users/svasilyev/.pyenv/versions/kopf/lib/python3.7/site-packages/requests/models.py", line 757, in generate
    raise ConnectionError(e)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='kube-1....', port=443): Read timed out.

This is not the expected behaviour. The expected behaviour is same as kubectl get -w — watching forever or until the server disconnects.

---

For reference, the official client does not set the timeouts on the sockets, but instead passes it to the query as a query arg called timeoutSeconds.

Instead, a hidden kwarg _request_timeout is used to restrict the connection timeouts. It is None by default in all APIs.