Executors still queued although job computation finished in dynamic allocation and exhausted resources

[lukleh]

Nomad 0.9.1
Pyspark 2.4.3

example pyspark-shell command running against Nomad cluster:

    --conf spark.nomad.sparkDistribution=local:/usr/lib/spark \
    --conf spark.dynamicAllocation.enabled=true \
    --conf spark.shuffle.service.enabled=true \
    --conf spark.dynamicAllocation.minExecutors=1

In pyspark-shell load the following:

• generate sample data https://gist.github.com/lukleh/13456e9a31def32f1b1eb07d659347e1
• force computation https://gist.github.com/lukleh/dc861417f89b2255bf97be8606f2a430

When NOT setting spark.dynamicAllocation.maxExecutors and cluster exhausts resources, executors are still queued although all calculation finished.