chore: build using "latest" container - #681
Open
iboyd-ansys wants to merge 26 commits into
Open
Conversation
github-actions
Bot
added
maintenance
iboyd-ansys
force-pushed
the
chore/build-with-latest
branch
2 times, most recently
from
July 15, 2026 21:19
fd19e3d to
52bec5f
Compare
Codecov Report❌ Patch coverage is ❌ Your patch status has failed because the patch coverage (82.60%) is below the target coverage (90.00%). You can increase the patch coverage or adjust the target coverage. Additional details and impacted files@@ Coverage Diff @@
## main #681 +/- ##
==========================================
+ Coverage 80.08% 81.14% +1.06%
==========================================
Files 50 50
Lines 3479 3574 +95
==========================================
+ Hits 2786 2900 +114
+ Misses 693 674 -19 🚀 New features to boost your workflow:
|
github-actions
Bot
added
the
documentation
iboyd-ansys
force-pushed
the
chore/build-with-latest
branch
7 times, most recently
from
July 17, 2026 10:10
eadc6cf to
2a525a4
Compare
iboyd-ansys
force-pushed
the
chore/build-with-latest
branch
2 times, most recently
from
July 20, 2026 15:07
448eb64 to
8afd201
Compare
docker compose down in _reset_example had no timeout, causing the doc build process to hang indefinitely after a gRPC failure mid-solve. With v27.1 the ENHANCE_YOUR_CALM/too_many_pings disconnect leaves the MAPDL container in a state where graceful shutdown stalls, causing subprocess.run to block forever (observed: 6-hour CI hang). Fix: - docker compose down: pass --timeout 30 so Docker force-kills after 30s - subprocess.run(down): add timeout=60 as outer safety net - subprocess.run(up): add timeout=120 as precaution This ensures the reset callback always returns promptly so sphinx-gallery can record the failure and continue (or retry).
When the SyC gRPC channel fails (e.g. ENHANCE_YOUR_CALM / too_many_pings), _syc_solve catches the exception and the SyC thread exits. However, participant solve threads (Fluent, MAPDL) remain blocked waiting for SyC coupling coordination data that will never arrive. The blocking _join_threads call on partsolve_threads then caused _do_solve to hang indefinitely, preventing the exception from propagating, the example script from exiting, and container cleanup from running. Result was a 6-hour CI hang until the runner timed out. Fix: - Replace _join_threads(partsolve_threads) with _join_threads_or_abort, which polls abort_check() every second and returns early when __solve_exception is set (i.e. SyC solve failed). - Mark participant solve threads as daemon=True so they do not block process exit if they are still alive when Python shuts down. - Add _join_threads_or_abort helper with explanatory docstring. With this fix, once the SyC solve thread fails, _do_solve returns promptly, solve() raises the stored exception, sphinx-gallery catches and records the failure, and _reset_example cleanup runs (docker compose down with --timeout 30, per the companion conf.py fix).
…er lifecycle Instrument the BEFORE and AFTER example callbacks to trace docker compose up/down operations with: - entry/exit markers with example name - timing measurements - try/except handlers to capture TimeoutExpired and other exceptions - specific messages for timeout vs other failures This will help identify if hangs occur in: 1. Launching containers (before examples) 2. Stopping containers (after examples) 3. Within the example execution itself 4. During post-cleanup (10s sleep) Log output will appear in sphinx-gallery build output.
…angs Added two new steps to the docs build job: 1. Start background diagnostics collection: Launches a background process that periodically (every 2 minutes) collects: - docker ps output - System Coupling container logs - Container inspect JSON - Server-side working directory logs (via docker exec) This will continue collecting diagnostics even if the build hangs, so the CI-logs/ artifacts will contain state snapshots from throughout the build execution. 2. Stop background diagnostics collection (always() condition): Kills the background monitoring task and takes a final snapshot at the moment the build step completes or times out. This ensures we can see server-side state and error patterns even if the job is killed mid-execution due to hanging. Combined with the manager.py and conf.py instrumentation, this should give us enough information to identify the hang point.
…iagnostics The default LOG level is ERROR, which means DEBUG and INFO messages from the instrumentation added to ParticipantManager._syc_solve, _do_solve, and _join_threads_or_abort were not appearing in output. Added logging.DEBUG level setup in _reset_example BEFORE phase so that all instrumentation logs are captured and visible during builds. This runs once at the start of example execution, before any threading or gRPC communication occurs.
Fixed Issues: 1. Logging import error: Changed pysystemcoupling.core.util.logging to the correct import path (from ansys.systemcoupling.core.util.logging import LOG) 2. v27.1 gRPC keepalive too aggressive: Added separate PYSYC_GRPC_CHANNEL_OPTIONS_JSON_271 with 300s (5 minute) keepalive interval instead of 90s. v27.1 server has stricter ping-strike limits and rejects 90s keepalive with 'too_many_pings' error. Increased keepalive to 5 minutes to be more conservative. Changes: - Updated conf.py _reset_example to correctly import and set DEBUG logging - Added PYSYC_GRPC_CHANNEL_OPTIONS_JSON_271 env var with 300s keepalive - Updated docs job Build HTML step to use _271 options (for v27.1) - Updated preflight and diagnostic steps to use _271 options - All min_time_between_pings and min_ping_interval aligned to 300s
… collection Improvements to hang diagnostics: 1. grpc_client.py (_read_stdstreams): - Enhanced logging to track chunk reception - Logs begin_streaming() RPC call and return - Tracks chunk count with periodic progress logs (every 50 chunks) - Logs elapsed time and stream termination conditions - Catches and logs exceptions during streaming 2. .github/workflows/ci.yml (background diagnostics): - Expanded working directory collection from server containers - Captures ALL files from /tmp, /workspace, /root/work, /opt/work - Collects directory listings to show file timestamps and activity - Helps determine if server-side solver is actually executing This allows us to definitively identify: - If output stream RPC is receiving data from server - If server working directory is populated (solver is running) - Where exactly the hang occurs in the client-server communication
…sh behavior Enhanced _read_stdstreams() diagnostics: 1. Chunk content analysis: - Detect empty chunks vs whitespace-only vs actual content - Log newline presence in each chunk - Distinguish between control characters and printable text - First 5 chunks and every 100th chunk logged in detail 2. Line flush tracking: - Log EVERY flushed line (first 5 + every 100th) - Show actual line content (first 80 chars) in DEBUG logs - Track lines_flushed count in periodic progress logs This reveals if: - Chunks contain actual content or just whitespace/empty - Lines are being detected (have newlines) and flushed - Output is going to handle_output() function - Why no text appears on console despite chunk reception Critical question: Are chunks coming through but with NO newlines? (Or are they all empty/whitespace?)
…cellation Previously the background diagnostics ran as a separate step's background process. This caused two problems: 1. Step stdout becomes invisible once the step exits, so only the first 'snapshot N...' echo was ever seen in the CI log. 2. The upload/stop steps use 'if: always()' which does NOT fire on manual job cancellation, so diagnostics were never uploaded when the hanging build was killed. Fix: run the background monitor inside the Build HTML step itself. Its output (docker ps + last 80 lines of all SyC/Fluent/MAPDL container logs) goes to the Build HTML step stdout, which is visible in the GitHub Actions UI up to the point of cancellation. ::group:: annotations make each snapshot collapsible. Also: - First snapshot is delayed 60s to allow containers to start - Pattern expanded to include Fluent and MAPDL containers (not just SyC) - Monitor is killed cleanly when the build loop completes - Improved wording in _participant_connect: 'connect() returned' instead of 'SUCCESS' since backend connection may occur later during solve
Previously MAPDL was at v25.2-ubuntu-cicd while System Coupling was upgraded to 27.1 (latest). This version mismatch may cause compatibility issues in the participant connection/initialization. Upgrade to v26.1.0 (stable released version) which is more likely to be compatible with 27.1 SyC while still being conservative (one step older than the development 27.1 release). Fluent remains at v25.2.0 to isolate this change and identify if MAPDL was the source of the connection hang.
…ty issue v26.1.0 MAPDL fails at the client connection phase and cannot reach the MAPDL backend server. v25.2-ubuntu-cicd, while it hangs in the solve phase, at least successfully connects (backend logs show 'Awaiting connections... done' with Fluent and MAPDL connected). Reverting to v25.2 while the v26.1.0 compatibility issue is investigated separately. Non-MAPDL examples continue to run successfully, confirming the infrastructure (gRPC tuning, timeouts, diagnostics) is working correctly. The solve hang is a MAPDL-specific issue to be addressed with the MAPDL team.
Allow pip output to be visible in CI logs so any dependency conflicts or installation errors are visible for debugging.
Enable detailed System Coupling server logging via PYSYC_SERVER_LOGGING_LEVEL environment variable. When set (e.g., to 5 for verbose), the server will write detailed log files: SyC_Log_Controller_<datetime>.txt and SyC_Log_CNode<N>_<datetime>.txt in the working directory. Changes: 1. Modified launch() to check PYSYC_SERVER_LOGGING_LEVEL env var and automatically add "-l <level>" to server launch arguments 2. Added docstring documentation for the environment variable 3. Updated CI workflow to set PYSYC_SERVER_LOGGING_LEVEL=5 for docs build 4. Added step to collect SyC_Log_*.txt files from host and containers after build completes, uploaded with diagnostics This enables debugging of System Coupling server-side issues during CI doc builds by collecting detailed server logs.
Return to the original working baseline configuration while retaining new server-side logging infrastructure: - FLUENT_IMAGE_VERSION: v26.1.0 → v25.2.0 - ansys-fluent-core: 0.40.2 → 0.35.0 v26.1 Fluent container does not launch correctly with current setup. Reverting to known-working baseline (27.1 SyC + 25.2 Fluent + 25.2 MAPDL) to analyze what happens during the solve phase with enhanced server logging.
Add ability to preserve System Coupling containers after exit so logs can be extracted even after timeout. Changes: 1. syc_container.py: Add PYSYC_PRESERVE_CONTAINER env var check. By default, containers are removed with --rm. Set env var to 1/true/yes to keep containers. 2. ci.yml: Set PYSYC_PRESERVE_CONTAINER=1 in Build HTML step so containers persist after 10-minute timeout. 3. ci.yml: Improve log collection to handle both running and stopped containers, with verbose output showing what files are found. 4. ci.yml: Add separate 'Clean up preserved containers' step after log collection to remove stopped containers and free disk space. This ensures System Coupling server logs (SyC_Log_CNode*.txt) can be extracted and analyzed even when the build times out. try Fluent 26.1 again with debug logging Revert "try Fluent 26.1 again with debug logging" This reverts commit 15d9300. try turning off AnsRpcBridge in cht_pipe case delete inadvertently committed files
iboyd-ansys
force-pushed
the
chore/build-with-latest
branch
from
July 21, 2026 13:57
d1770b5 to
920f1c6
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Intermittent issues have been seen when using the 26.1 System Coupling container, but only when running GitHub hosted doc builds.
Aim here is to start building with a newer container ("latest") to see if issues persist with this version. If so, there might be an opportunity still to make server side changes.