Skip to content

6044 Do not complete requests on receiving notification events#6086

Open
guerrillalg wants to merge 10 commits into
eclipse-ee4j:3.1from
guerrillalg:3.1
Open

6044 Do not complete requests on receiving notification events#6086
guerrillalg wants to merge 10 commits into
eclipse-ee4j:3.1from
guerrillalg:3.1

Conversation

@guerrillalg

Copy link
Copy Markdown

This is a proposal to fix an issue #6044 in CompletedAsyncContextListener. Semantically speaking, this is supposed to be a listener. It is being notified about certain events when complete or error is already happening. It is essentially incorrect to complete the ongoing request at this moment. I propose to only update a flag, which is needed in related code.

Normally, a similar fix should be done in the 2.x as well, however I was not able to reproduce it using a test in that version. We can discuss on this and see how should we deal with it then.

@dmatej

dmatej commented Jun 10, 2026

Copy link
Copy Markdown
Contributor

Hi @guerrillalg , thanks for the contribution, however to merge your contribution we need you to create an account on the Eclipse website, and sign ECA License Agreement.
https://www.eclipse.org/legal/eca/

Another problem is that your change failed to compile on CI. Version 3.1 must work with both Java 11 and 21. I can try locally later, perhaps it is a CI issue.

Btw, catching and ignoring Throwable always smells badly.

The feature change - your change goes against 431b933 as you described in the issue. Now, listener methods will not call this method and set just the completed value. I am not sure if that is desired - what completes the context?:

        @Override
        public void complete() {
            completed.set(true);

            final AsyncContext asyncContext = asyncContextRef.getAndSet(null);
            if (asyncContext != null) {
                asyncContext.complete();
            }
        }

Maybe the listener needs just to check if it did not complete already? Once it reaches completed state, none of the code inside could be executed again.

I am just thinking about it, it doesn't mean you are wrong.

@guerrillalg

Copy link
Copy Markdown
Author

Thanks for looking into the PR, David.
Eclipse agreement is signed.
For the CI, I could build it locally. Then I saw it was read here, but never could not get the details out of it. I will try building using J11. Probably, it is the case. Thanks for your hint. If you can get an error to me, would be helpful.

Now, listener methods will not call this method and set just the completed value. I am not sure if that is desired - what completes the context?

Yes, that's exactly the change I propose. The thing that is going on is that Context is already completing. Jetty is shutting down in our specific case. And during this shutdown context tries to notify all listeners about this. And what I claim to be wrong in the release branch is that a listener was calling back the asyncContext.complete(). So, my proposal is that since it is a listener, it should only listen to events coming in and update its internal state and not to try to complete the context. All tests seem to be fine with this change and semantically it sounds correct as well.

@dmatej dmatej added this to the 3.1.12 milestone Jun 11, 2026
@dmatej dmatej added bug Something isn't working and removed Missing ECA labels Jun 11, 2026
@dmatej dmatej modified the milestones: 3.1.12, 3.1.13 Jun 11, 2026
@dmatej

dmatej commented Jun 11, 2026

Copy link
Copy Markdown
Contributor

Now Java 21 passed, but with 11 we have this - seems you need an older jetty:

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.14.0:testCompile (default-testCompile) on project jersey-6044: Compilation failure: Compilation failure: 
[ERROR] /home/jenkins/agent/workspace/ci-cd-integration_PR-6086/tests/integration/jersey-6044/src/test/java/org/glassfish/jersey/tests/jettyresponseclose/JettyHttpContainerDuringShutdownTest.java:[22,38] cannot access org.eclipse.jetty.ee10.servlet.ServletContextHandler
[ERROR]   bad class file: /home/jenkins/.m2/repository/org/eclipse/jetty/ee10/jetty-ee10-servlet/12.0.27/jetty-ee10-servlet-12.0.27.jar(/org/eclipse/jetty/ee10/servlet/ServletContextHandler.class)
[ERROR]     class file has wrong version 61.0, should be 55.0
[ERROR]     Please remove or make sure it appears in the correct subdirectory of the classpath.

Looking here, Jetty 12 requires Java 17.
https://www.herodevs.com/blog-posts/jetty-end-of-life-dates-jetty-9-10-11-and-12-2026-guide

EDIT: I have committed the switch to Jetty 11 in the new test.

guerrillalg and others added 5 commits June 12, 2026 20:50
Signed-off-by: David Matějček <david.matejcek@omnifish.ee>
- more strict
- more requests
- higher load
- removed swallowing exceptions
- counting and logging results
- higher shutdown timeout after the test
- reliable short timeout in between processing

Signed-off-by: David Matějček <david.matejcek@omnifish.ee>
@dmatej

dmatej commented Jun 12, 2026

Copy link
Copy Markdown
Contributor

Well, I had to check it out locally:

  • The original test swallowed errors and did not prove anything, because it silently swallowed all "connection failed errors" via the empty finally block.
  • My commits added via github gui were wrong way - Jetty 11 doesn't support async requests. Again, test silently swallowed all errors, but some were still visible in the test output.
  • Then I refactored the test, increased the shutdown timeout and number of requests, but the problem is that the test doesn't test anything again.
    • It starts the server
    • Then it starts the client and runs 1000 requests
    • Then it asks server to stop while it is processing requests
    • Then tests that all requests finished - one option is HTTP 204, another is connection refused exception. The problem here is that there are no guarantees that at least one request finished with 204 - and with a very powerful machine even all requests could be processed. I tried to increase the probability so part of requests would be processed and part would not make it, but I cannot reproduce the problem with the test.
  • Note also that some objects created are Closeable.

The test has same results with the current and the old implementation, so I tend to merge the change because it doesn't break anything (and it seems it doesn't fix anything too, it is just nicer).

I made the test more strict, so if there would be anything unexpected, the test should fail. I added also logs, so you can see the sequence.

@dmatej

dmatej commented Jun 13, 2026

Copy link
Copy Markdown
Contributor

Yet one idea - I added some log to the complete() method; while the original code was logging this warning, your code was without that. Basically it seems to me that the listener could be completely removed. I can try that tomorrow ...

INFO: onComplete: org.eclipse.jetty.ee10.servlet.AsyncContextEvent@3d2584c
Jun 13, 2026 2:00:17 AM org.glassfish.jersey.servlet.async.AsyncContextDelegateProviderImpl$ExtensionImpl complete
WARNING: The processing was already completed!
java.lang.RuntimeException: STACK!
        at org.glassfish.jersey.servlet.async.AsyncContextDelegateProviderImpl$ExtensionImpl.complete(AsyncContextDelegateProviderImpl.java:131)
        at org.glassfish.jersey.servlet.async.AsyncContextDelegateProviderImpl$ExtensionImpl$CompletedAsyncContextListener.onComplete(AsyncContextDelegateProviderImpl.java:87)
        at org.eclipse.jetty.ee10.servlet.ServletChannelState.lambda$completed$1(ServletChannelState.java:1143)
        at org.eclipse.jetty.server.handler.ContextHandler$ScopedContext.run(ContextHandler.java:1518)
        at org.eclipse.jetty.server.handler.ContextHandler$ScopedContext.run(ContextHandler.java:1511)
        at org.eclipse.jetty.ee10.servlet.ServletChannelState.runInContext(ServletChannelState.java:1307)
        at org.eclipse.jetty.ee10.servlet.ServletChannelState.completed(ServletChannelState.java:1137)
        at org.eclipse.jetty.util.Callback$3.failed(Callback.java:178)
        at org.eclipse.jetty.ee10.servlet.HttpOutput.onWriteComplete(HttpOutput.java:263)
        at org.eclipse.jetty.ee10.servlet.HttpOutput$WriteCompleteCB.failed(HttpOutput.java:1780)
        at org.eclipse.jetty.server.handler.ContextHandler$ScopedContext.accept(ContextHandler.java:1499)
        at org.eclipse.jetty.server.handler.ContextResponse$1.failed(ContextResponse.java:47)
        at org.eclipse.jetty.util.Callback.failed(Callback.java:639)

# Conflicts:
#	tests/integration/jersey-6044/pom.xml
#	tests/integration/jersey-6044/src/main/java/org/glassfish/jersey/tests/jettyresponseclose/Resource204.java
#	tests/integration/jersey-6044/src/test/java/org/glassfish/jersey/tests/jettyresponseclose/JettyHttpContainerDuringShutdownTest.java
#	tests/integration/pom.xml
@guerrillalg

Copy link
Copy Markdown
Author

Hi @dmatej thanks for your comments. I have finally been able to come back to this issue.

I can easily reproduce the problem as of now.
Green test, no code changes:
before
I comment my fix, test is expectedly yellow:
after
I understand, that the test is time/probability based, which is a not a nice thing.. I have tried to change it to be more deterministic to have a better control, however it depends on some factors. So, it is so far the only way.

Basically it seems to me that the listener could be completely removed.
Well, I guess it was written for some purposes. We are running at this moment with it being commented, no problems seen. Hard to tell if that would affect someone else.

I have also tried to downgrading jetty to version 11 so that a test would be J11 compatible, however it seems that it is only a problem of jetty 12. At least, whenever I do so, test is always working fine, so a test so far is configured to only be executed on a j17.

assertThrows(TimeoutException.class, () -> shutdownServer(100));
for (Future<Response> responseFuture : waitingResponses) {
try (Response response = responseFuture.get(30, TimeUnit.SECONDS)) {
assertThat("Unexpected response code", response.getStatus(), Matchers.anyOf(equalTo(204), equalTo(500)));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

500 should not be acceptable. Can you find some stacktrace what happened?

@dmatej

dmatej commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

The test is incompatible with Jetty 11, you have to use Jetty 12, but that requires Java 17, so you have to annotate the test with @EnabledForJreRange(min = JRE.JAVA_17, disabledReason = "Jetty 12 requires Java 17")

The test failed on CI when executed with JDK 11.

EDIT: Skipping the test is not enough, we have to exclude whole module or at least the compilation.

@dmatej

dmatej commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

@guerrillalg Can you test and review my PR?

  • It removes the listener completely, it doesn't seem it was required.
  • It integrated your test, but just with JDK17+ as Jetty needs it.
  • Replaces unit test based on proxies with a unit test based on mocks.

Especially what I would like to know is

  • if it works for you
  • if you are still able to reproduce those HTTP 500 errors, then please send me the stacktrace, it should be visible somewhere.

dmatej pushed a commit to dmatej/jersey that referenced this pull request Jul 9, 2026
- The original test was created by guerrillakg, and was squashed and further
  updated by David Matejcek.
- Both users signed ECA.

Signed-off-by: David Matějček <david.matejcek@omnifish.ee>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants