Add RStudio 2026.07.1-147 / R 4.6.1 on noble; rootless container builds in /dev/shm - #188
Merged
Conversation
Bumps the RStudio line from 2025.05.1-513 / R 4.5.1 and moves it off jammy
(standard support ends April 2027) onto ubuntu-noble-compile, which is already
built in this repo and used by python-ml-base.
Two template changes were needed for a noble build:
- The libgsl conditional was `!= "jammy" -> libgsl23`, so noble would have taken
the libgsl23 branch; that package does not exist on 24.04, which ships
libgsl27. Inverted to a whitelist, leaving bionic/focal unchanged.
- Posit publishes a single .deb for Ubuntu 22 and 24 under /server/jammy/
(/server/noble/ 404s), so the download path can no longer be derived from the
base release. Added an optional rstudio_deb_release field defaulting to
base_container_version_name, so all existing rows render byte-identically.
Also fixes three pre-existing issues in the RStudio files:
- The task name interpolated {{ item.version_name }}, which no rstudio_versions
entry defines, so it rendered with an empty version.
- RStudio.lua was the only container module in the repo missing
load("singularity"); the wrappers call singularity bare, so it was resolving
from the ambient system PATH rather than the module.
- RStudio.sh tested a directory with -f and had mangled indentation.
Verified: yamllint clean, ansible-lint production profile passes, and
RStudio.def renders byte-identically to HEAD for bionic/focal/jammy rows while
noble gets libgsl27 and the jammy .deb path. No container build was run --
that needs sudo singularity build on the cluster.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RACvroiDb6D5nuP8uCiv1X
group_vars/all still pinned the build binary at singularity 4.1.0, which ships
a setuid starter-suid (-rwsr-xr-x). The --without-suid migration landed in
singularity.yaml and in the version list, but never reached this var, so every
container build was still going through the suid-era binary.
Repoint at 4.4.1, which is built --without-suid: its libexec/singularity/bin
contains only starter, squashfuse_ll and singularity-buildkitd, with no
starter-suid at all.
Not 4.4.2 -- it is registered in roles/compiled/tasks/common/main.yml but is
not installed on /software (only 3.11.3, 4.1.0, 4.1.3 and 4.4.1 are), so
pointing at it would break every build until it is deployed.
Since the binary no longer needs suid, drop `become: true` from all 21
container build tasks and pass --fakeroot instead. --fakeroot is required, not
optional: singularity refuses a def-file build as a non-root user without it
("--remote, --fakeroot, or the proot command are required to build this source
as a non-root user"), and that check fires before any namespace work, so it is
independent of host configuration.
Caveat: --fakeroot needs unprivileged user namespaces on the build host. This
could not be verified end to end -- a test build on an Ubuntu 24.04 workstation
failed at "Failed to create mount namespace" because
kernel.apparmor_restrict_unprivileged_userns=1 there. Production is expected to
permit it; if not, this needs revisiting.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RACvroiDb6D5nuP8uCiv1X
kennedydane
changed the title
The noble build failed at the first apt-get install with "Package
'pandoc-citeproc' has no installation candidate". pandoc-citeproc was dropped
after jammy -- citeproc is built into pandoc itself since 2.11, and noble ships
pandoc 3.1.3. ubuntu-compile.def already guards this package for noble; that
guard was never carried into RStudio.def.
Checking the rest of the package list inside ubuntu-noble-compile.sif rather
than one package at a time turned up a second casualty in the block that had
not run yet: libfreetype6-dev does not exist on noble, which ships
libfreetype-dev. jammy has both, so that one is a rename rather than a removal.
Verified by resolving the rendered noble package list against real apt inside
the base image:
- all 39 base packages: apt-get install --simulate OK
- r-base=4.6.1* and friends from CRAN's noble-cran40: --simulate OK,
candidate 4.6.1-1.2404.1
- bionic/focal/jammy still render byte-identically to HEAD
An earlier availability check via packages.ubuntu.com was unreliable: it serves
a page for obsolete packages, so pandoc-citeproc looked present when apt has no
candidate for it. Checked against apt in the target image from now on.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RACvroiDb6D5nuP8uCiv1X
The noble build reached the CRAN install wave and failed configuring 'fs': "Configuration failed because libuv was not found ... deb: libuv1-dev". fs 2.1.0 links system libuv instead of bundling it, and fs sits under both devtools and tidyverse, so it takes much of the stack down with it. Rather than fix the one reported package, the full explicitly-installed set was checked against Posit's sysreqs API for ubuntu 24.04 -- the 13 CRAN packages plus BiocManager, and the six Bioconductor packages (DESeq2, edgeR, limma, MethylSeekR, methylKit, bsseq). Their combined system requirements were diffed against what the recipe already installs, by comparing simulated apt installs inside ubuntu-noble-compile.sif. Only two candidates came back, and one was a false positive: libfontconfig-dev already Provides libfontconfig1-dev. libuv1-dev is the single genuine gap. Added unconditionally rather than guarded for noble: fs 2.1.0 needs libuv on every release, so rebuilding any existing version today would fail identically. libuv1-dev is present on focal (1.34.2), jammy (1.43.0) and noble (1.48.0), all confirmed in the base images. This is the one intentional render change for pre-existing versions; the render check now asserts it is the ONLY difference vs HEAD rather than dropping the byte-identity assertion. Verified in ubuntu-noble-compile.sif: pkg-config finds libuv 1.48.0, fs 2.1.0 compiles from source and loads. Not covered: bcbioRNASeq and DESeqAnalysis come from r.acidgenomics.com, which Posit's sysreqs index does not cover, so their system deps were not checked. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RACvroiDb6D5nuP8uCiv1X
CI has been failing on this branch, but not because of anything in it: yamllint and ansible-lint both pass, and the failure is the third workflow step, the grep gate for top-level fact injection (ansible-lint has no rule for it). The 8 hits are in two files this branch does not touch. They arrived in 19d84f5 ("Add verkko-fillet 0.1.25"), which reintroduced the pattern that 58ff247 had swept from the tree, and merged while red. Master's own most recent run (29835997086) fails with a byte-identical list, so master is currently red and this branch inherited it. Fixed in place rather than in a separate PR, at the repo owner's request: ansible_env.X -> ansible_facts['env'].X (mashmap, 6) ansible_processor_vcpus -> ansible_facts['processor_vcpus'] (mashmap 1, seqtk 1) This matches the convention 58ff247 established and rsem.yaml already uses. Verified by running the workflow's exact grep locally: no bare references remain. Note, deliberately not changed: the `| default('')` idiom here leaves a trailing colon in LD_LIBRARY_PATH etc. when the variable is unset, which means "current directory". rsem.yaml avoids that with a `{% if ... in ansible_facts['env'] %}` guard. That is pre-existing behaviour and out of scope for this PR. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RACvroiDb6D5nuP8uCiv1X
Posit released 2026.07.1-147 after this branch was opened; the docs page now lists it as the current OSS release. Both debs are still served (2026.07.1-147 and 2026.07.0-139 each return HTTP 200, ~232 MiB), so this is a forward bump rather than a fix for a dead URL. R stays at 4.6.1 -- still the newest in noble-cran40 (4.5.3, 4.6.0, 4.6.1). This renames the artifacts: the tag pair becomes -t RStudio,RStudio2026.07.1-147-R4.6.1 and the image becomes RStudio2026.07.1-147-R4.6.1.sif. Note that a complete RStudio2026.07.0-139-R4.6.1.sif (3.9 GB) already exists on /software from an earlier run and is left in place, along with its .def. It is now orphaned -- nothing in the repo refers to that version. Verified working before the bump: rserver 2026.07.0+139, R 4.6.1, fs 2.1.0, tidyverse 2.0.0, devtools, shiny 1.14.0, DESeq2 1.52.0. Removing it is a manual decision. Separately, that image revealed a silent failure worth carrying forward: bcbioRNASeq is MISSING from it while DESeqAnalysis 0.7.3 installed fine. Both come from r.acidgenomics.com. install.packages() warns rather than errors, so the build reports success either way and the %test block only checks R's version. Not addressed here. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RACvroiDb6D5nuP8uCiv1X
MikeCTZA
approved these changes
Jul 21, 2026
The production build was very slow because the RStudio build had no
SINGULARITY_TMPDIR, so singularity assembled the sandbox under /tmp (visible in
an earlier failure as /tmp/build-temp-885908027/rootfs).
Nine other container tasks already set this. RStudio was the outlier, and the
block was not merely commented out -- it sat at the end of the "Create R
binaries" template loop rather than on the build task, so uncommenting it as-is
would not have affected the build. Moved onto the build command and switched to
the same expression the other tasks use, which honours an operator override:
SINGULARITY_TMPDIR: "{{ lookup('env', 'SINGULARITY_TMPDIR') | default('/dev/shm') }}"
Sizing, measured from the built 2026.07.0-139 image: 3.7 GB compressed, 8.9 GB
unpacked (6.9 GB /usr, mostly texlive; 1.7 GB /library of R packages). The
sandbox plus apt and R build temp wants roughly 15-20 GB of /dev/shm. /dev/shm
is tmpfs, so that is RAM. The lookup above lets a smaller host point elsewhere
without editing the repo.
Nine further build tasks still lack this and remain slow (trycycler, ClinSV,
proteowizard, PopGen_suite, vep, gatk, python-ml, python-ml-base, DIANN). Left
alone as unrelated to this PR.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RACvroiDb6D5nuP8uCiv1X
Adds log_path to ansible.cfg so every run appends to a file instead of only
scrolling past in the terminal. Useful for the long container builds, where the
interesting failure is thousands of lines back.
log_path = ./logs/ansible.log
The directory is tracked via a .gitkeep because ansible does not create it: if
it is missing, ansible warns and silently disables logging rather than failing.
Contents are gitignored (*.log was already covered globally; an explicit
ansible/logs/* rule with negations for .gitkeep and README.md makes the intent
local and obvious).
Verified: the log is created, contains the run, and a second run appends to it
rather than truncating (12 -> 24 lines). git correctly ignores ansible.log while
staging only .gitkeep and README.md.
Three caveats documented in ansible/logs/README.md rather than left to be
discovered:
- Nothing rotates this and container builds are verbose. A logrotate snippet
is included there.
- Task output is captured verbatim. no_log is honoured, but everything else a
module prints ends up in the clear, so the file should be treated as
sensitive.
- The path is relative, so it only resolves as intended when playbooks are run
from ansible/ -- already required for ansible.cfg to be found at all.
ANSIBLE_LOG_PATH overrides it per run, which is handy for giving a long
build its own file.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RACvroiDb6D5nuP8uCiv1X
The SINGULARITY_TMPDIR expression added in a715853 never worked. lookup('env') returns '' rather than undefined when the variable is unset, and `| default(x)` only substitutes for undefined values, so the task was setting SINGULARITY_TMPDIR to the empty string. Verified with singularity 4.4.1, printing SINGULARITY_ROOTFS from %setup: SINGULARITY_TMPDIR="" -> /tmp/build-temp-889771773/rootfs SINGULARITY_TMPDIR unset -> /tmp/build-temp-3153017378/rootfs SINGULARITY_TMPDIR=/dev/shm -> /dev/shm/build-temp-472999930/rootfs So the empty string behaves exactly like unset and builds fall back to /tmp. Adding the boolean argument makes '' take the default: | default('/dev/shm', true) Note this pattern is copied verbatim in nine other container tasks (ubuntu, gnuplot, atfncat, spectronaut, mspipe, bcbio, pggb, beast, snvstory), which have therefore also been building in /tmp rather than /dev/shm. Not changed here: for the larger images that is a real behaviour change, since /dev/shm is RAM and python-ml-base alone is 9.7 GB compressed. Worth a separate, deliberate pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RACvroiDb6D5nuP8uCiv1X
Applies the fd3fb91 fix everywhere else the pattern appears, so container builds actually land in /dev/shm rather than silently falling back to /tmp. | default('/dev/shm') -> | default('/dev/shm', true) 8 occurrences across 7 files: ubuntu (x2), gnuplot, atfncat, spectronaut, mspipe, pggb, snvstory. Two files needed something different: - beast.yaml had the block commented out. Unlike RStudio's, it was correctly positioned on the build task, so it only needed enabling. Switched to the lookup form for consistency. - bcbio.yaml hardcoded "/dev/shm", so it was the one task already building in the right place. Normalised to the lookup form; same default, but an operator can now redirect it with SINGULARITY_TMPDIR. Verified: the expression resolves to /dev/shm when the variable is unset and to the variable's value when set (checked with /scratch). All 11 occurrences are attached to a build task -- confirmed by walking each file and pairing the setting with its enclosing task name, since RStudio's had been sitting on an unrelated template loop. Deliberately not touched: nine build tasks set no SINGULARITY_TMPDIR at all (trycycler, ClinSV, proteowizard, PopGen_suite, vep, gatk, python-ml, python-ml-base, DIANN) and keep building in /tmp. Adding it there is a genuine behaviour change rather than a bug fix -- /dev/shm is RAM, and python-ml-base is 9.1 GB compressed, so it could exhaust /dev/shm on a host where the build currently succeeds. That needs per-image sizing, not a blanket edit. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RACvroiDb6D5nuP8uCiv1X
Completes the sweep: the nine tasks that set no SINGULARITY_TMPDIR at all now
do, so every container build lands in /dev/shm instead of /tmp.
trycycler, ClinSV, proteowizard, PopGen_suite, vep, gatk, python-ml,
python-ml-base, DIANN (x2) -- 10 build commands across 9 files.
Unlike the previous commit this is a behaviour change rather than a bug fix,
since these tasks were never trying to use /dev/shm. Confirmed acceptable:
production nodes have >200 GiB RAM, so the default /dev/shm of half of RAM is
~100 GiB. The largest image involved, python-ml-base, is 9.1 GB compressed --
comfortably within that. Smaller build hosts can still redirect via
SINGULARITY_TMPDIR, which the lookup honours.
Insertion was done programmatically to get the indentation right, since it
varies (vep uses 6 spaces where the rest use 10) and python-ml-base carries an
extra `chdir` inside args that had to stay there. Both were checked by hand in
the diff.
Verified:
- no build task is left without the setting
- 21 build commands, 21 SINGULARITY_TMPDIR settings -- exact parity
- every setting pairs with a Build task, checked by walking each file and
printing the enclosing task name, since RStudio's had previously been
stranded on an unrelated template loop
- yamllint clean, ansible-lint production profile passes
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RACvroiDb6D5nuP8uCiv1X
kennedydane
changed the title
MikeCTZA
approved these changes
Jul 22, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Started as an RStudio version bump. Building it surfaced a chain of related problems in how containers are built, which are fixed here too. 11 commits, 30 files.
1. RStudio 2026.07.1-147 / R 4.6.1 on Ubuntu noble
Bumps the container line from
2025.05.1-513/ R 4.5.1 and moves it off jammy ontoubuntu-noble-compile(24.04), which already exists in this repo. Jammy's standard support ends April 2027.New tag pair:
-t RStudio,RStudio2026.07.1-147-R4.6.1Template changes needed for noble:
!= "jammy" -> libgsl23; noble would have taken that branch, but 24.04 shipslibgsl27. Inverted to a whitelist..debfor Ubuntu 22 and 24 under/server/jammy/;/server/noble/404s. New optionalrstudio_deb_releasefield defaulting tobase_container_version_name.pandoc-citeprocwas dropped after jammy (citeproc is built into pandoc since 2.11).ubuntu-compile.defalready guarded this; the guard was never carried intoRStudio.def.libfreetype6-devdoesn't exist on noble — renamedlibfreetype-dev. Found by checking the whole package list at once rather than one build failure at a time.libuv1-devadded forfs2.1.0, which now links system libuv instead of bundling it.fssits under bothdevtoolsandtidyverse, so it took most of the stack down.Also fixes three pre-existing issues: the task name interpolated
{{ item.version_name }}(undefined for every entry);RStudio.luawas the only container module missingload("singularity");RStudio.shtested a directory with-f.2. Container builds: singularity 4.4.1, rootless
group_vars/allstill pinned 4.1.0 — the suid-era binary. The--without-suidmigration had landed insingularity.yamland the version list but never reached this var.starter-suid-rwsr-xr-xrootNot 4.4.2 — registered in
main.ymlbut not installed, so pointing at it would break every build.become: trueis dropped from all 21 build tasks with--fakerootpassed instead.--fakerootis required, not optional: singularity refuses a def-file build as non-root without it, and that check fires before any namespace work.uidmappackage (newuidmap/newgidmap, setuid root)usernsto the singularity binary —kernel.apparmor_restrict_unprivileged_userns=1blocks it by defaultAll three were hit one at a time during this work. Worth encoding in Ansible rather than leaving as tribal knowledge.
3. Every container build now uses /dev/shm
Builds were landing in
/tmpand were very slow. Two separate bugs:RStudio.yamlhad the setting commented out and attached to the "Create R binaries" template loop rather than the build task — uncommenting it would not have worked.The shared expression never worked anywhere.
lookup('env', 'X')returns''(not undefined) when unset, and| default(x)only substitutes for undefined — so the tasks were settingSINGULARITY_TMPDIR="". Verified by printingSINGULARITY_ROOTFSfrom%setup:SINGULARITY_TMPDIR=""/tmp/build-temp-.../rootfs/tmp/build-temp-.../rootfs/dev/shm/dev/shm/build-temp-.../rootfsFixed with
default('/dev/shm', true). This affected 8 occurrences across 7 files that all looked correct.Then extended to the nine tasks that had no setting at all. That last part is a behaviour change rather than a bug fix; confirmed acceptable as production nodes have >200 GiB RAM (so ~100 GiB
/dev/shm) and the largest image,python-ml-base, is 9.1 GB compressed. Smaller hosts can redirect viaSINGULARITY_TMPDIR.Result: 21 build commands, 21 settings, each verified to sit on a build task.
4. Unrelated fixes carried along
mashmap.yamlandseqtk.yamluse bareansible_env/ansible_processor_vcpus, tripping the fact-injection grep gate. Introduced in19d84f5, which merged while red — master's own run fails identically. Fixed here at the repo owner's request, so merging this turns master green too.log_path = ./logs/ansible.log, appending. Directory tracked via.gitkeepbecause ansible silently disables logging if it's missing. Caveats (no rotation, contents are sensitive, path is CWD-relative) documented inansible/logs/README.md.Verification
RStudio.defrendered against the pre-edit template fromgit show HEAD: bionic, focal and jammy differ only by the intentionallibuv1-devaddition, asserted explicitly rather than by loosening the check.ubuntu-noble-compile.sif: 39 base packages simulate OK;r-base=4.6.1*fromnoble-cran40resolves to4.6.1-1.2404.1.libuv1-dev(libfontconfig1-devwas a false positive —libfontconfig-devProvides it).fs2.1.0 compiles and loads withlibuv1-devpresent.--fakerootbuild verified end to end:%postruns as uid 0.Known issues, not fixed here
bcbioRNASeqsilently fails to install. It's missing from a completed image whileDESeqAnalysisfrom the same acidgenomics repo installed fine.install.packages()warns rather than errors, so%postexits 0 and the build reports success — and%testonly checksR --version. This hole applies to every package in the threeRscriptwaves. A%testasserting key packages load would catch it.RStudio2026.07.0-139-R4.6.1.sif(3.7 GB) is orphaned on/softwareafter the .1 bump, along with its.def. Verified working before the bump; removal is a manual decision.4.4.1/etc/singularity/singularity.confis a regular file, not the symlink to the shared config thatsingularity.yamlintends.mashmap.yaml's| default('')idiom leaves a trailing colon inLD_LIBRARY_PATHwhen unset, which the linker reads as the current directory. Pre-existing;rsem.yamlavoids it with an{% if %}guard.🤖 Generated with Claude Code
https://claude.ai/code/session_01RACvroiDb6D5nuP8uCiv1X