System.Net.Sockets.SocketException with --skipnontestassemblies under NUnitConsole 3.20.1

[sebingel]

Originally, I encountered this error when running the tests on TeamCity. However, I can reproduce it 1:1 locally as well.

We are currently upgrading NUnitConsole on our build server from version 3.17.0 to 3.20.1.
In one of our projects, we encountered the following error. I then tested with other NUnitConsole versions. It still works with 3.18.3 and 3.19.2. Only starting from 3.20.1 do I experience issues:

NUnit.Engine.NUnitEngineUnloadException : Multiple exceptions encountered. Retrieve AggregatedExceptions property for more information
----> System.Net.Sockets.SocketException : Eine vorhandene Verbindung wurde vom Remotehost geschlossen
----> System.Net.Sockets.SocketException : Eine vorhandene Verbindung wurde vom Remotehost geschlossen
----> System.Net.Sockets.SocketException : Eine vorhandene Verbindung wurde vom Remotehost geschlossen
----> System.Net.Sockets.SocketException : Eine vorhandene Verbindung wurde vom Remotehost geschlossen
----> System.Net.Sockets.SocketException : Eine vorhandene Verbindung wurde vom Remotehost geschlossen
----> NUnit.Engine.NUnitEngineUnloadException : Agent Process was terminated successfully after error.
----> System.Net.Sockets.SocketException : Eine vorhandene Verbindung wurde vom Remotehost geschlossen
----> System.Net.Sockets.SocketException : Eine vorhandene Verbindung wurde vom Remotehost geschlossen
----> System.Net.Sockets.SocketException : Eine vorhandene Verbindung wurde vom Remotehost geschlossen

For each individual assembly, I also get the following message:

8. Error : C:\VCS\jtl-framework-core\bin\Release\net8.0\JTL.Domain.Base.Unittests.dll
   System.Net.Sockets.SocketException : Eine vorhandene Verbindung wurde vom Remotehost geschlossen
   --SocketException
   Eine vorhandene Verbindung wurde vom Remotehost geschlossen
   bei NUnit.Engine.Communication.Transports.Tcp.SocketReader.GetNextMessage() in D:\a\nunit-console\nunit-console\src\NUnitEngine\nunit.engine.core\Communication\Transports\Tcp\SocketReader.cs:Zeile 42.
   bei NUnit.Engine.Communication.Transports.Tcp.TestAgentTcpProxy.TestRunResult(ITestEventListener listener) in D:\a\nunit-console\nunit-console\src\NUnitEngine\nunit.engine\Communication\Transports\Tcp\TestAgentTcpProxy.cs:Zeile 116.
   bei NUnit.Engine.Runners.ProcessRunner.RunTests(ITestEventListener listener, TestFilter filter) in D:\a\nunit-console\nunit-console\src\NUnitEngine\nunit.engine\Runners\ProcessRunner.cs:Zeile 138.

• The full command line being used:

C:\VCS\NUnit.Console-3.20.1\bin\net462\nunit3-console.exe --noheader --where cat!=IgnoreOnBuildserver --skipnontestassemblies --result=C:\VCS\1.xml JTL.Framework.Core.EventBus.Unittests\bin\Release\net8.0\JTL.Framework.Core.EventBus.Unittests.dll JTL.Framework.Core.EventBus.Unittests\bin\Release\net48\JTL.Framework.Core.EventBus.Unittests.dll bin\Release\net8.0\JTL.WebP.Unittests.dll bin\Release\net8.0\JTL.Shared.Unittests.dll bin\Release\net8.0\JTL.Infrastructure.Unittests.dll bin\Release\net8.0\JTL.Inbound.Mailing.Integrationstests.dll bin\Release\net8.0\JTL.Framework.Core.UnitTests.dll bin\Release\net8.0\JTL.Framework.Core.Dispatcher.Unittests.dll bin\Release\net8.0\JTL.Domain.Base.Unittests.dll bin\Release\net48\JTL.Shared.Unittests.dll bin\Release\net48\JTL.Infrastructure.Unittests.dll bin\Release\net48\JTL.Inbound.Mailing.Integrationstests.dll bin\Release\net48\JTL.Framework.Core.UnitTests.dll bin\Release\net48\JTL.Framework.Core.Dispatcher.Unittests.dll bin\Release\net48\JTL.Domain.Base.Unittests.dll

• Method of installation: zip
• Log files: log.zip
• NUnitConsole 3.20.1
• All versions of the NUnit Framework in use: NUnit (4.3.2), NUnit3TestAdapter (5.0.0)
• Whether the behaviour is any different when the tests are run using the --inprocess flag:
  When i add this flag the error changes to: inprocess_error.txt

I hope this information helps to identify the issue and doesn’t contain too much noise. If any further details are needed, I’m happy to assist.

[CharliePoole]

Some questions...

1. Since inprocess works, I assume you are targeting the .NET Framework with your tests. Correct? What version?
2. Can you try removing the unneeded (by console runner) reference to the NUnit3TestAdapter? What happens?
3. You should narrow this down to the assemblies causing a problem by examining the logs.

Regarding item 3, the zip file of logs contains

• One overall log for the runner itself, named InternalTrace.19472.log.
• 14 agent logs, named like nunit-agent_NNNN.log
• 5 logs of actual tests being run by an agent, like InternalTrace.NNNN.JTL.etc..

The NNNN represents a process id, e.g. process 4360 and it allows you to match the test execution logs to the agent logs. Only five of the agents you started actually ran any tests. You should first figure out what those agents WITHOUT any test execution have in common. For example, if they all target the same runtime, that's a big clue.

[sebingel]

tl;dr:
I left out the --skipnontestassemblies switch and now it works.

Regarding the questions:

1. I wouldn't say it works with "inprocess". It just throws a different exception.
   All our test projects target both net48 and net8.0:
   <TargetFrameworks>net48;net8.0</TargetFrameworks>

2. Removing the reference to NUnit3TestAdapter makes no difference.

3. The agent logs of agents without tests all look identical. Each contains the line:
   Info [ 1] NUnitTestAgent: Running .NET 8.0 agent under .NET 8.0.16
   The agent logs of agents with tests also look identical, but instead have this line:
   Info [ 1] NUnitTestAgent: Running .NET 4.6.2 agent under .NET 4.5

So for now, I’ll just leave out the --skipnontestassemblies switch. It’s redundant anyway and was only active as a precaution, since we only specify assemblies that contain tests.

EDIT:
I accidentally closed the thread with my comment. Fell free to reopen if you need further infos.

[CharliePoole]

Glad you found the cause, which also makes it lots easier on this end. I reopened as we still seem to have a bug here.

When you remove that option, do all 14 tests, including those targeting .NET 8.0, run correctly? The --skipnontestassemblies option triggers some additional inspection of each assembly, which may be where there is some code that fails.

[sebingel]

Yes, every test in every assembly I specified runs successfully when I leave out --skipnontestassemblies, regardless of the target framework.

[andrewimcclement]

I have a similar-looking issue with 3.20.1. I will see if I can pin down my exact issue a little more precisely.

[JTCozart]

I am also having the same issue with 3.20.1.

    <TargetFramework>netcoreapp7.0</TargetFramework>

If I run nunit3-console.exe PROJECT_FOLDER/PROJECT.csproj --where "cat != Integration && cat != LiveEndToEnd" --skipnontestassemblies --result=TestResults/PROJECT.xml I get the agent socket error.

If I omit the skipnontestassemblies like others the tests succeeds.

Downgrading to 3.19.2 is the best workaround I have found so far for my build server.

I use choco to install apps on my build server fresh with each build. 3.20.1 was recently published to choco so it just started failing for me.