prevent writing to a Span after it finishes

[iparadiso]

EvCacheImpl fires both onComplete and onError for a single EVCacheEvent on several paths. The tracing listener treated each callback as terminal, so the second one re-tagged the span and called a duplicate finish().

Brave converts MutableSpans to zipkin2.Span synchronously inside finish(). A second write outside of this syncronous block mutates a live shared object being serialized to Zipkin for export, which can kill the single reporter thread for the lifetime of the JVM, causing an observation outage if it causes an ArrayOutOfBounds exception with annotations collections.

We added detection for this in Spring to prevent the reporter thread from being terminated and log late writers, which is how we found this use case.

This change wraps the Span as a PendingSpan which only allows it to only be finalized once.