diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index 6e4540d..ef74e9e 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -32,7 +32,7 @@ jobs: ${{ matrix.extra-pytest-warnings }} enable-coverage: ${{ matrix.enable-coverage }} jupyter-platform-dirs: "1" - package-managers: "uv pixi" + package-managers: "uv pixi conda" codecov-flags: "unit" strategy: fail-fast: false diff --git a/doc/howtoguides/requirements.rst b/doc/howtoguides/requirements.rst index b783d5b..4c360f8 100644 --- a/doc/howtoguides/requirements.rst +++ b/doc/howtoguides/requirements.rst @@ -19,7 +19,8 @@ What is stored :term:`package manager` can recreate the environment from this list. * ``manager``: the :term:`package manager` that generated the requirements, with the content of the files it needs to recreate the environment: ``requirements.txt`` for pip-venv, - ``pyproject.toml`` and ``uv.lock`` for uv, ``pixi.toml`` and ``pixi.lock`` for pixi. + ``pyproject.toml`` and ``uv.lock`` for uv, ``environment.yml`` for conda, ``pixi.toml`` + and ``pixi.lock`` for pixi. Use ``--exclude-requirements`` to store nothing. @@ -55,8 +56,8 @@ is not on the ``PATH`` or when a faster implementation should be used .. code-block:: bash - ewoks install demo.json --yes --package-manager-name uv \ - --package-manager-command /path/to/uv + ewoks install demo.json --yes --package-manager-name conda \ + --package-manager-command micromamba Choose the environment ---------------------- @@ -85,27 +86,29 @@ The default name is the identifier of the :term:`workflow`, which is the ``id`` ``graph`` field. A :term:`workflow` without an identifier gets a name derived from its content. -Without ``--env-root`` the :term:`package manager` decides where the environment goes. venv, -uv and pixi create an environment wherever they are told to, so for pip-venv, uv and pixi -ewoks uses ``~/.ewoks/envs``. +Without ``--env-root`` the :term:`package manager` decides where the environment goes: conda +creates it in its own environment directory (``conda config --show envs_dirs``), so +``conda activate `` finds it. venv, uv and pixi create an environment wherever they are +told to, so for those ewoks uses ``~/.ewoks/envs``. An environment that already exists is installed in, which adds the requirements to what is already there. Use ``--clean`` to remove it first. Only a directory that contains a python environment is removed. -The environment of a :term:`workflow` is a directory: for pip-venv it is a virtual -environment, for uv a project with the environment in ``.venv`` and for pixi a workspace with -the environment in ``.pixi/envs/default``. ``ewoks execute --env`` takes that directory, not -the python interpreter inside it. +The environment of a :term:`workflow` is a directory. For pip-venv and conda it is a python +environment, for uv it is a project with the environment in ``.venv`` and for pixi it is a +workspace with the environment in ``.pixi/envs/default``. ``ewoks execute --env`` takes that +directory, not the python interpreter inside it. Limitations ----------- * ``--in-place`` is only supported by pip-venv and uv. The other package managers only install in an environment they created themselves. -* ``--python-version`` is a request: uv can provide any python version, pixi provides the patch - versions built by its channel and pip-venv can only use the version of the python interpreter - that creates the environment. A warning is emitted when the version cannot be provided. +* ``--python-version`` is a request: uv can provide any python version, conda and pixi provide + the patch versions built by their channel and pip-venv can only use the version of the python + interpreter that creates the environment. A warning is emitted when the version cannot be + provided. * uv and pixi resolve the ``distributions`` into a lock file, which requires access to the package index. When that fails, a warning is emitted and the requirements are stored without files: the environment is then recreated from the ``distributions`` list. diff --git a/doc/reference/glossary.rst b/doc/reference/glossary.rst index f2a0375..4c3b11f 100644 --- a/doc/reference/glossary.rst +++ b/doc/reference/glossary.rst @@ -29,7 +29,7 @@ Glossary An execution engine is the underlying software used to execute the :term:`workflow`. :term:`Ewoks` supports multiple execution engines: pypushflow, orange, dask and the ewoks internal excution engine. Package manager - A package manager creates python environments and installs packages in them. :term:`Ewoks` uses package managers to store the environment in which a :term:`workflow` was created and to recreate it: pip with venv, uv and pixi. + A package manager creates python environments and installs packages in them. :term:`Ewoks` uses package managers to store the environment in which a :term:`workflow` was created and to recreate it: pip with venv, uv, conda and pixi. blissdata `Blissdata `_ is an API for accessing data from BLISS in memory. diff --git a/doc/tutorials/install.rst b/doc/tutorials/install.rst index 2d1ad38..de89948 100644 --- a/doc/tutorials/install.rst +++ b/doc/tutorials/install.rst @@ -22,6 +22,7 @@ of the supported ones install/pip_venv install/uv + install/conda install/pixi The producer and the re-producer do not need the same :term:`package manager`: the diff --git a/doc/tutorials/install/conda.rst b/doc/tutorials/install/conda.rst new file mode 100644 index 0000000..ee665bf --- /dev/null +++ b/doc/tutorials/install/conda.rst @@ -0,0 +1,179 @@ +.. _install_conda: + +conda +===== + +End-to-end walk-through with `conda `_. Conda installs python itself, +so the environment of a :term:`workflow` can have another python version than the one that +recreates it. + +Install conda +------------- + +`Miniforge `_ is a conda installation configured +with the conda-forge channel + +.. tabs:: + + .. group-tab:: Linux + + .. code-block:: bash + + export CONDA_ROOT=$HOME/miniforge3 + curl -LsSf -o /tmp/miniforge.sh https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-Linux-x86_64.sh + bash /tmp/miniforge.sh -b -p $CONDA_ROOT + source $CONDA_ROOT/etc/profile.d/conda.sh + + .. group-tab:: macOS + + .. code-block:: bash + + export CONDA_ROOT=$HOME/miniforge3 + curl -LsSf -o /tmp/miniforge.sh https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-MacOSX-arm64.sh + bash /tmp/miniforge.sh -b -p $CONDA_ROOT + source $CONDA_ROOT/etc/profile.d/conda.sh + + .. group-tab:: Windows + + .. code-block:: powershell + + $env:CONDA_ROOT = "$env:USERPROFILE\miniforge3" + irm https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-Windows-x86_64.exe -OutFile miniforge.exe + Start-Process -Wait -FilePath .\miniforge.exe -ArgumentList "/S", "/D=$env:CONDA_ROOT" + & "$env:CONDA_ROOT\shell\condabin\conda-hook.ps1" + +Producer side +------------- + +Create an environment, install :term:`ewoks` in it and store a :term:`workflow` with the +packages of that environment as its ``requirements`` + +.. tabs:: + + .. group-tab:: Linux + + .. code-block:: bash + + conda create --yes --prefix ewoks_producer python pip + conda activate ./ewoks_producer + pip install ewoks + ewoks convert demo demo.json --test + conda deactivate + + .. group-tab:: macOS + + .. code-block:: bash + + conda create --yes --prefix ewoks_producer python pip + conda activate ./ewoks_producer + pip install ewoks + ewoks convert demo demo.json --test + conda deactivate + + .. group-tab:: Windows + + .. code-block:: powershell + + conda create --yes --prefix ewoks_producer python pip + conda activate .\ewoks_producer + pip install ewoks + ewoks convert demo demo.json --test + conda deactivate + +Install any package that provides :term:`tasks ` instead of +``ewoks`` for a real :term:`workflow`. Conda stores the packages it installed +itself and the ones installed with ``pip``: the ``pip`` section of the +``environment.yml`` file in the ``requirements`` is the conda way of expressing +packages of the python package index. + +Re-producer side +---------------- + +Only ``ewoks`` itself is needed to recreate the environment of ``demo.json`` + +.. tabs:: + + .. group-tab:: Linux + + .. code-block:: bash + + conda create --yes --prefix ewoks_reproducer python pip + conda activate ./ewoks_reproducer + pip install ewoks + ewoks install demo.json --yes --env-root ewoks_envs + + .. group-tab:: macOS + + .. code-block:: bash + + conda create --yes --prefix ewoks_reproducer python pip + conda activate ./ewoks_reproducer + pip install ewoks + ewoks install demo.json --yes --env-root ewoks_envs + + .. group-tab:: Windows + + .. code-block:: powershell + + conda create --yes --prefix ewoks_reproducer python pip + conda activate .\ewoks_reproducer + pip install ewoks + ewoks install demo.json --yes --env-root ewoks_envs + +The environment of the :term:`workflow` is a conda environment in ``ewoks_envs/demo``. Without +``--env-root`` it is created in the first environment directory of conda, where +``conda activate demo`` finds it. + +Execute the workflow +-------------------- + +.. tabs:: + + .. group-tab:: Linux + + .. code-block:: bash + + ewoks execute --env ewoks_envs/demo demo.json --outputs=all + + .. group-tab:: macOS + + .. code-block:: bash + + ewoks execute --env ewoks_envs/demo demo.json --outputs=all + + .. group-tab:: Windows + + .. code-block:: powershell + + ewoks execute --env ewoks_envs\demo demo.json --outputs=all + +Clean up +-------- + +.. tabs:: + + .. group-tab:: Linux + + .. code-block:: bash + + conda deactivate + rm -rf ewoks_producer ewoks_reproducer ewoks_envs demo.json + + .. group-tab:: macOS + + .. code-block:: bash + + conda deactivate + rm -rf ewoks_producer ewoks_reproducer ewoks_envs demo.json + + .. group-tab:: Windows + + .. code-block:: powershell + + conda deactivate + Remove-Item -Recurse -Force ewoks_producer, ewoks_reproducer, ewoks_envs, demo.json + +.. note:: + + ``mamba`` and ``micromamba`` are faster implementations of conda. Use + ``--package-manager-command micromamba`` to let ``ewoks install`` invoke one of them. diff --git a/src/ewoks/_requirements/conda.py b/src/ewoks/_requirements/conda.py new file mode 100644 index 0000000..da7b0cb --- /dev/null +++ b/src/ewoks/_requirements/conda.py @@ -0,0 +1,202 @@ +import json +import logging +import os +import platform +import re +import shutil +import sys +from pathlib import Path +from typing import Dict +from typing import List +from typing import Literal +from typing import Mapping +from typing import Optional +from typing import Sequence +from typing import Tuple + +from pydantic import Field + +from .utils import conda_channel +from .utils import requirements_txt +from .utils.base_manager import BaseManager +from .utils.base_manager import BaseManagerInfo +from .utils.base_manager import BaseRequirements +from .utils.environment import Environment +from .utils.files import temporary_files +from .utils.metadata import models + +logger = logging.getLogger(__name__) + +_ENVIRONMENT_FILENAME = "environment.yml" + +_IMPLEMENTATIONS = ("micromamba", "mamba", "conda") + + +class CondaManagerInfo(BaseManagerInfo): + name: Literal["conda"] = Field( + default="conda", + description="Environments described and created by `conda`.", + examples=["conda"], + ) + + +class CondaRequirements(BaseRequirements): + manager: CondaManagerInfo + + +class CondaManager(BaseManager): + """Uses `conda env export` to describe an environment with an + `environment.yml` file and `conda env update` to reproduce it. Unlike the + python-only package managers this includes non-python dependencies. + + The command invokes conda. For example `CondaManager("micromamba")`. The + fastest implementation available is used when no command is provided. + """ + + NAME = "conda" + PRIORITY = 4 + REQUIREMENTS_MODEL = CondaRequirements + COMMAND_EXAMPLE = "conda" + + def __init__(self, *command: str) -> None: + if not command: + command = _detect_implementation() + super().__init__(*command) + + def version(self) -> Optional[str]: + """Returns None when this manager is not available.""" + try: + output = self._check_output("--version") + except RuntimeError: + return None + # For example "conda 25.11.1" or "2.4.0" for mamba + match = re.search(r"(\d+\.\d+(?:\.\d+)*)", output) + if not match: + return None + return match.group(1) + + def is_active(self) -> bool: + """Manager is explicitly active.""" + return "CONDA_PREFIX" in os.environ or _is_conda_environment(Path(sys.prefix)) + + def create_environment( + self, location: Path, python_version: Optional[str] = None + ) -> Environment: + """ + :raises RuntimeError: creation failed + """ + if not python_version: + python_version = platform.python_version() + # 'pip' is needed for the pip section of a conda environment file + self._check_call( + "create", + "--yes", + "--quiet", + "--prefix", + location, + f"python={conda_channel.python_specifier(python_version)}", + "pip", + ) + return self.environment(location) + + def environments_root(self) -> Path: + """First environment directory of conda, in which `conda create --name` + creates environments so they can be activated by name. + """ + try: + # 'conda config --show envs_dirs' is not supported by mamba + info = json.loads(self._check_output("info", "--json")) + # 'envs_dirs' for conda, 'envs directories' for mamba and micromamba + directories = info.get("envs_dirs") or info.get("envs directories") + return Path(directories[0]) + except (RuntimeError, ValueError, LookupError, TypeError) as ex: + logger.warning("Cannot determine where conda creates environments (%s)", ex) + return super().environments_root() + + def _gather_files( + self, distributions: List[models.Distribution], python_version: str + ) -> Dict[str, str]: + if not _is_conda_environment(Path(sys.prefix)): + logger.warning( + "The current python environment is not a conda environment. It will " + "be reproduced from the installed python distributions." + ) + return dict() + output = self._check_output("env", "export", "--prefix", sys.prefix) + return {_ENVIRONMENT_FILENAME: _without_environment_location(output)} + + def _files_from_distributions( + self, distributions: Sequence[models.Distribution], python_version: str + ) -> Dict[str, str]: + requirements = requirements_txt.manifest_requirements(distributions) + return { + _ENVIRONMENT_FILENAME: _environment_yml(requirements, python_version), + } + + def _install_files( + self, files: Mapping[str, str], environment: Environment + ) -> None: + with temporary_files(files) as directory: + self._check_call( + "env", + "update", + "--quiet", + "--prefix", + environment.location, + "--file", + directory / _ENVIRONMENT_FILENAME, + ) + + def _add_ewoks( + self, requirements: BaseRequirements, environment: Environment + ) -> None: + # The environment file pins all versions so they cannot change + pinned = requirements_txt.manifest_requirements(requirements.distributions) + files = { + _ENVIRONMENT_FILENAME: _environment_yml( + [*pinned, "ewoks"], requirements.python.version + ) + } + self._install_files(files, environment) + + +def _environment_yml(requirements: Sequence[str], python_version: str) -> str: + """Content of a conda `environment.yml` file. Python distributions are declared + in the `pip` section, which is how the conda environment file format expresses + requirements that come from the python package index. + """ + dependencies = "" + if python_version: + dependencies += f" - python={conda_channel.python_specifier(python_version)}\n" + dependencies += " - pip\n" + if requirements: + dependencies += " - pip:\n" + dependencies += "".join(f" - {req}\n" for req in requirements) + return f"""channels: + - conda-forge +dependencies: +{dependencies}""" + + +def _detect_implementation() -> Tuple[str, ...]: + """The command is needed before knowing whether conda is available, so this + does not execute anything.""" + for name in _IMPLEMENTATIONS: + if shutil.which(name): + return (name,) + return (_IMPLEMENTATIONS[-1],) + + +def _without_environment_location(environment_yml: str) -> str: + """Remove the name and the prefix of the exported environment: the location + of the environment to create is provided separately. + """ + return "".join( + f"{line}\n" + for line in environment_yml.splitlines() + if not line.startswith(("name:", "prefix:")) + ) + + +def _is_conda_environment(prefix: Path) -> bool: + return (prefix / "conda-meta").is_dir() diff --git a/src/ewoks/_requirements/utils/_supported.py b/src/ewoks/_requirements/utils/_supported.py index ae9eab9..9eb155e 100644 --- a/src/ewoks/_requirements/utils/_supported.py +++ b/src/ewoks/_requirements/utils/_supported.py @@ -2,6 +2,7 @@ from typing import Dict from typing import Type +from ..conda import CondaManager from ..pip_venv import PipVenvManager from ..pixi import PixiManager from ..uv import UvManager @@ -13,6 +14,7 @@ def get_supported_managers() -> Dict[str, Type[BaseManager]]: managers = [ PipVenvManager, UvManager, + CondaManager, PixiManager, ] return {manager_cls.NAME: manager_cls for manager_cls in managers} diff --git a/src/ewoks/_requirements/utils/base_manager.py b/src/ewoks/_requirements/utils/base_manager.py index e91c3a9..82e33eb 100644 --- a/src/ewoks/_requirements/utils/base_manager.py +++ b/src/ewoks/_requirements/utils/base_manager.py @@ -26,7 +26,7 @@ class BaseManagerInfo(models.BaseModel): name: str = Field( description="Package manager that generated the requirements.", - examples=["pip-venv", "uv", "pixi"], + examples=["pip-venv", "uv", "conda", "pixi"], ) version: str = Field( description="Version of the package manager.", examples=["25.0.1"] @@ -36,8 +36,9 @@ class BaseManagerInfo(models.BaseModel): description=( "Content of the files the package manager needs to reproduce the " "environment: 'requirements.txt' for pip-venv, 'pyproject.toml' and " - "'uv.lock' for uv, 'pixi.toml' and 'pixi.lock' for pixi. Empty when " - "the package manager could not generate them." + "'uv.lock' for uv, 'environment.yml' for conda, 'pixi.toml' and " + "'pixi.lock' for pixi. Empty when the package manager could not " + "generate them." ), examples=[{"requirements.txt": "ewoks==7.0.0\nnetworkx==3.4.2\n"}], ) diff --git a/src/ewoks/_requirements/utils/environment.py b/src/ewoks/_requirements/utils/environment.py index b1141ba..6c0d853 100644 --- a/src/ewoks/_requirements/utils/environment.py +++ b/src/ewoks/_requirements/utils/environment.py @@ -102,7 +102,7 @@ def python_executable(prefix: Path) -> Path: def _interpreters() -> Tuple[Path, ...]: """Interpreter locations inside an environment prefix, the first one being where a virtual environment has it. On Windows the other environments (a python - installation and pixi) have it in the prefix itself. + installation, conda and pixi) have it in the prefix itself. """ if sys.platform == "win32": return (Path("Scripts", "python.exe"), Path("python.exe")) diff --git a/src/ewoks/_requirements/utils/metadata/models.py b/src/ewoks/_requirements/utils/metadata/models.py index a29ca8a..72bfd66 100644 --- a/src/ewoks/_requirements/utils/metadata/models.py +++ b/src/ewoks/_requirements/utils/metadata/models.py @@ -82,5 +82,5 @@ class Distribution(BaseModel): installer: Optional[str] = Field( default=None, description="Tool that installed the distribution.", - examples=["pip", "uv"], + examples=["pip", "uv", "conda"], ) diff --git a/src/ewoks/tests/requirements/conftest.py b/src/ewoks/tests/requirements/conftest.py index b19556f..686b156 100644 --- a/src/ewoks/tests/requirements/conftest.py +++ b/src/ewoks/tests/requirements/conftest.py @@ -35,6 +35,9 @@ def _home_variables(home: Path) -> Dict[str, Path]: """Environment variables that move everything a package manager writes outside a python environment to this home directory. """ + condarc = home / ".condarc" + with open(condarc, "w", encoding="utf-8") as fh: + fh.write("channels:\n - conda-forge\n") return { # Home directory on Linux, macOS and Windows "HOME": home, @@ -44,6 +47,12 @@ def _home_variables(home: Path) -> Dict[str, Path]: "XDG_CACHE_HOME": home / ".cache", "XDG_CONFIG_HOME": home / ".config", "XDG_DATA_HOME": home / ".local" / "share", + # Conda downloads packages next to its own installation instead + "CONDA_PKGS_DIRS": home / "conda" / "pkgs", + # Directories in which package managers create named environments + "CONDA_ENVS_DIRS": home / "conda" / "envs", + # Channels are no longer provided by the home directory of the user + "CONDARC": condarc, } diff --git a/src/ewoks/tests/requirements/managers.py b/src/ewoks/tests/requirements/managers.py index b419db1..af07e58 100644 --- a/src/ewoks/tests/requirements/managers.py +++ b/src/ewoks/tests/requirements/managers.py @@ -11,8 +11,10 @@ from typing import Tuple from typing import Type +from ..._requirements.conda import CondaManager from ..._requirements.pip_venv import PipVenvManager from ..._requirements.pixi import PixiManager +from ..._requirements.utils import conda_channel from ..._requirements.utils.base_manager import BaseManager from ..._requirements.utils.metadata import models from ..._requirements.utils.requirements_txt import REQUIREMENTS_FILENAME @@ -82,6 +84,30 @@ def native_files( return dict() +class CondaCase(ManagerCase): + NAME = "conda" + MANAGER_CLS = CondaManager + INSTALLER = "conda" + SLOW = True + CHANNEL_PYTHON = True + + def native_files( + self, distributions: Sequence[models.Distribution], python_version: str + ) -> Dict[str, str]: + pip_dependencies = "".join( + f" - {dist.name}=={dist.version}\n" for dist in distributions + ) + return { + "environment.yml": f"""channels: + - conda-forge +dependencies: + - python={conda_channel.python_specifier(python_version)} + - pip + - pip: +{pip_dependencies}""" + } + + class PixiCase(ManagerCase): NAME = "pixi" MANAGER_CLS = PixiManager @@ -104,6 +130,7 @@ def native_files( MANAGER_CASES: List[ManagerCase] = [ PipVenvCase(), UvCase(), + CondaCase(), PixiCase(), ] diff --git a/src/ewoks/tests/requirements/test_conda.py b/src/ewoks/tests/requirements/test_conda.py new file mode 100644 index 0000000..8040221 --- /dev/null +++ b/src/ewoks/tests/requirements/test_conda.py @@ -0,0 +1,96 @@ +"""Tests specific to the conda package manager.""" + +import sys + +import pytest + +from ..._requirements.conda import CondaManager +from ..._requirements.conda import _environment_yml +from ..._requirements.utils.base_manager import EWOKS_ENVIRONMENTS_ROOT +from ..._requirements.utils.conda_channel import python_specifier +from .utils import PYTHON_VERSION +from .utils import REQUIREMENTS + + +def test_environment_file(): + """Python distributions are declared in the pip section of a conda + environment file.""" + environment_yml = _environment_yml(REQUIREMENTS, PYTHON_VERSION) + + pip_dependencies = "".join(f" - {requirement}\n" for requirement in REQUIREMENTS) + assert ( + environment_yml + == f"""channels: + - conda-forge +dependencies: + - python={python_specifier(PYTHON_VERSION)} + - pip + - pip: +{pip_dependencies}""" + ) + + +def test_environment_file_without_requirements(): + assert ( + _environment_yml([], "") + == """channels: + - conda-forge +dependencies: + - pip +""" + ) + + +def test_default_command(): + """The fastest implementation available is used.""" + manager = CondaManager() + assert manager._cmd_args[0] in ("micromamba", "mamba", "conda") + + +def test_version_not_available(): + assert CondaManager("conda-does-not-exist").version() is None + + +def test_is_active(monkeypatch, tmp_path): + monkeypatch.delenv("CONDA_PREFIX", raising=False) + monkeypatch.setattr(sys, "prefix", str(tmp_path)) + assert not CondaManager().is_active() + + (tmp_path / "conda-meta").mkdir() + assert CondaManager().is_active() + + +def test_is_active_conda_prefix(monkeypatch, tmp_path): + monkeypatch.setattr(sys, "prefix", str(tmp_path)) + monkeypatch.setenv("CONDA_PREFIX", str(tmp_path)) + assert CondaManager().is_active() + + +def test_environments_root(): + """Named environments are created where conda creates them itself.""" + manager = CondaManager() + if manager.version() is None: + pytest.skip("conda is not installed") + + root = manager.environments_root() + + assert root.is_absolute() + assert root != EWOKS_ENVIRONMENTS_ROOT + + +def test_environments_root_not_available(caplog): + """Environments of ewoks when conda cannot provide its environment directory.""" + manager = CondaManager("conda-does-not-exist") + + assert manager.environments_root() == EWOKS_ENVIRONMENTS_ROOT + + assert "Cannot determine where conda creates environments" in caplog.text + + +def test_files_not_available(monkeypatch, tmp_path, caplog): + """Only a conda environment can be exported.""" + monkeypatch.setattr(sys, "prefix", str(tmp_path)) + + assert CondaManager()._gather_files([], PYTHON_VERSION) == {} + + assert "not a conda environment" in caplog.text