ESPRESSIF'S IDF Python Wheels

This project automates the build and upload process of required Python Wheels by ESP-IDF. The wheels for multiple OSes and architectures are being built.

Supported architectures:

• Linux
  • Ubuntu - x86_64
  • ARMv7 - arm32
  • ARM64
• Windows - AMD64
• MacOS
  • x86_64
  • ARM64

Supported Python versions:

• 3.14
• 3.13
• 3.12
• 3.11
• 3.10
• 3.9
• 3.8

Note

This list of supported Python versions is automatically updated by update_python_versions.py script and update-python-versions.yml workflow.

For each release branch of ESP-IDF which is not EOL and ESP-IDF master branch, all the requirements and constraints files are automatically downloaded and wheels are built and uploaded.

Completely Automated

This repository has been completely automated. All the supported versions of ESP-IDF and Python versions are fetch and resolved automatically. The implementation of this logic is in the supported_versions.py script and get-supported-versions.yml workflow.

Also README.md file and pyproject.toml is automatically updated with the script update_python_versions.py and update-python-versions.yml workflow.

Supported Versions Action

This workflow is reusable action and it is possible to be called in other projects - it will generate supported_versions.json file with the following structure, which can be parsed and used in caller workflow to avoid developer interaction of changing the supported versions.

Also it provides the min_idf_major_version and min_idf_minor_version outputs. Map them to environment variables on the job, so they are available to every step regardless of the runner shell (>> $GITHUB_ENV is Bash syntax and is silently discarded on Windows runners, where the default shell is PowerShell):

jobs:
  build-wheels:
    needs: get-supported-versions
    env:
      MIN_IDF_MAJOR_VERSION: ${{ needs.get-supported-versions.outputs.min_idf_major_version }}
      MIN_IDF_MINOR_VERSION: ${{ needs.get-supported-versions.outputs.min_idf_minor_version }}

{
    "supported_idf": [
        "v5.5",
        "v5.4",
        "v5.3",
        "v5.2",
        "v5.1"
    ],
    "oldest_supported_idf": "v5.1",
    "supported_python": [
        "3.13",
        "3.12",
        "3.11",
        "3.10",
        "3.9",
        "3.8"
    ],
    "oldest_supported_python": "3.8"
}

Usage of Manual Wheels Build - DEFINED WHEELS WORKFLOW

If there is a need to manually build and upload wheels the defined-wheels workflow can be used for this. The pip package needs to be specified with marker support (e.g. coredump~=1.2;sys_platform!='win32') and check the architectures which should be wheels built and uploaded for. Multiple wheels can be separated by space.

Then the wheels are built and uploaded for all supported Python versions.

Requirements Lists

These lists are files for requirements that should be added or excluded from the main requirements list which is automatically assembled.

exclude_list.yaml

File for excluded Python packages in the main requirements list.

This YAML file is converted to Requirement from packaging.requirements because pip can handle this format, so the function for converting is designed to be compatible with PEP508 scheme. The opposite logic of exclude_list is handled by the function itself, which means it is supposed to be easy to use for developers, this is also the reason YAML format is used.

For every package_name there are options:

• version
  • supports all logic operators defined by PEP508 for versions (<, >, !=, etc.)
• platform
• python

which could be a string or a list of strings.

exclude_list template:

- package_name: '<name_of_package>'
    version: '<package_version_with_operator>' / ['<package_version_with_operator>', '<package_version_with_operator>']     # optional
    platform: '<platform>' / ['<platform>', '<platform>', '<platform>']                                                     # optional
    python: '<python_version_with_operator>' / ['<python_version>', '<python_version>', '<python_version>']                                                     # optional

The syntax can be converted into a sentence: "From assembled main requirements exclude package_name with version on platform for python version".

example:

- package_name: 'pyserial'
    version: ['>=3.3', '<3.6']
    platform: ['win32', 'linux', 'darwin']
    python: '>=3.9'

This would mean: "From assembled main requirements exclude pyserial with version >=3.3 and <3.6 on platform win32, linux, darwin for python version >=3.9".

From the example above is clear that the platform could be left out (because all main platforms are specified) so the options platform or version or python are optional, one of them or both can be not specified and the key can be erased. When only package_name is given the package will be excluded from main requirements.

PyPI Requires-Python preflight

Before running pip wheel, the build scripts can query the PyPI JSON API so that no release matching the requirement’s version specifier (including ==, ~=, and ranges such as >=x,<y) is installable on the current interpreter according to each candidate release’s Requires-Python metadata. In that case the requirement is skipped (with a log line) instead of invoking pip, which avoids noisy failures such as “No matching distribution found” when pip hides incompatible versions.

This is implemented in _helper_functions.py and wired into:

• build_wheels.py — after exclude_list is applied, for the main graph, include_list.yaml entries, and lines written to dependent_requirements.txt
• build_wheels_from_file.py — each requirement line from dependent_requirements.txt or from the CLI

If the project JSON cannot be fetched (network error, etc.), preflight does not skip; pip runs as usual.

This complements exclude_list.yaml: the YAML still expresses platform, markers, and build/repair policy where PyPI metadata is not enough. Preflight focuses on Python version compatibility declared on PyPI for matching releases.

Source distributions on the Espressif index (PEP 503)

Wheel builds still honor exclude_list.yaml on each CI platform. emit_sdist_requirements.py unions over every supported platform and Python version: if a package has no buildable wheel path for at least one such environment (after exclude merging), it is listed in sdist_requirements.txt and published as an sdist (.tar.gz, .zip, and other formats pip may fetch) on https://dl.espressif.com/pypi. Pip can then fall back to the sdist when no compatible wheel exists for the current environment (for example a dependency excluded only on Windows).

• build_wheels.py assembles the full IDF dependency tree, then emit_sdist_requirements.py computes sdist_requirements.txt (union over all platforms).
• download_sdists.py fetches those sdists from PyPI during upload; upload_wheels.py and create_index_pages.py place them under pypi/<project>/ next to wheels (PEP 503 simple repository layout).
• Per-project index.html pages include data-requires-python on every wheel and sdist link (Simple Repository API, PEP 503 HTML metadata). Values come from PyPI’s Requires-Python field. Pip 8.2+ reads this attribute and ignores incompatible artifacts before download.
• Sdist upload guard: upload_wheels.py uploads sdists only when they appear in sdist_requirements.txt (present in full platforms-dispatch bundle uploads). Ad-hoc defined-wheels uploads without that file skip sdists so accidental pip wheel tarballs do not reach S3.
• After merging index-metadata changes, regenerate all index pages without re-uploading artifacts by running the only-create-and-upload-index workflow against the production bucket.
• verify_s3_sdists.py checks that each line in sdist_requirements.txt has a matching sdist on S3 (--strict in CI fails when the bundle file is missing).

To disable the preflight entirely (e.g. debugging or air‑gapped runs), set the environment variable:

Variable	Effect when set to 1, true, or yes (case-insensitive)
SKIP_PYPI_REQUIRES_PYTHON_CHECK	Skip all PyPI preflight checks; every requirement is passed through to pip wheel.

include_list.yaml

File for additional Python packages to the main requirements list. Built separately to not restrict the main requirements list.

The syntax can be also converted into a sentence: "For assembled main requirements additionally include package_name with version on platform for python version".

native_import_guard.yaml

File for native import checks after wheels are installed in CI (test_wheels_install.py, all test-wheels-install runners). It does not change the requirements list.

The syntax matches exclude_list.yaml / include_list.yaml: “After installing the wheel for package_name (on platform, for python, at version), run imports — or skip — and fail CI if import crashes or errors.”

Top-level options:

• probe_unlisted — if true (default), every other platform wheel is probed with one import <top_level> from the wheel’s top_level.txt (skipped when that file is missing; installs use --no-deps)
• skip_pure_any — if true (default), skip *-none-any.whl
• skip_top_level — names ignored when generating that default import (test, tests, testing)

Each packages: entry:

• package_name — PyPI distribution name (legacy key name is accepted)
• imports — Python statement(s) to run after install (list; multiline allowed)
• skip — if true, do not probe when the filters match
• platform / python / version — optional, same tokens as exclude_list.yaml (omitted = all). Unmatched filters do not skip the probe; the wheel is treated as unlisted (probe_unlisted default import). Use skip: true when a probe should not run.

example:

probe_unlisted: true
skip_pure_any: true
packages:
  - package_name: cffi
    imports:
      - import _cffi_backend

This would mean: load the real C extension for cffi (not import cffi alone). test_wheels_install.py runs these probes after --no-deps install. Unlisted platform wheels — including packages whose YAML row does not match the current platform / python / version — are probed only when top_level.txt has a valid import name (the distribution name is not guessed).

build_requirements.txt

File for the requirements needed for the build process and the build script.

os_dependencies

When there is a need for additional OS dependencies to successfully build the wheels on a specific platform and architecture, the .sh script in the os_dependencies directory can be adjusted.

Universal wheel tag - linking of dynamic libraries

The repair tools are used after build to link and bundle all the needed libraries into the wheel to produce correct universal tag and working wheel. If this is not able to achieve the broken wheel is deleted and not published to Espressif's PyPI.

• auditwheel package to repair Linux's manylinux wheels
• delocate package to repair Mac's dynamically linked libraries
• delvewheel package to repair Windows's DLLs

This logic is done by the repair workflow and the repair_wheels.py script

ARMv7 vs ARMv7 Legacy: same wheel filename, different binaries

Linux ARMv7 and Linux ARMv7 Legacy can both produce a wheel whose filename is identical (same PEP 425 tags) while the ELF contents differ (different glibc/OpenSSL/Rust toolchain lineage). Note: wheels-download-directory-* CI artifacts are the pre-repair build outputs; comparing those can still show identical names until the repair workflow runs. Two bad outcomes follow if that is not handled after repair/merge:

1. Artifact merge / local flatten — downloading multiple wheels-repaired-* artifacts into one directory with merge-multiple: true can make the second file silently overwrite the first on disk before any upload runs.
2. S3 upload — upload_wheels.py publishes to pypi/<package>/<wheel-filename>. Uploading a second wheel with the same key replaces the object; clients then see whichever build ran last. ARMv7 vs Legacy clashes must be caught before upload (merge collision check and distinct manylinux_*_armv7l tags when auditwheel allows), not by refusing re-uploads of an existing key when wheel bytes change between CI runs.

Mitigations in this repo:

• ARMv7 builds use piwheels as the primary index (PIP_INDEX_URL in CI). force_no_binary_linux.txt applies on x86_64/aarch64 Linux only, not in ARMv7 Docker, so pip can reuse upstream linux_armv7l wheels when available. repair_wheels.py retags piwheels linux_armv7l wheels to manylinux_2_36_armv7l / manylinux_2_31_armv7l (via AUDITWHEEL_PLAT) so ARMv7 vs Legacy builds get distinct filenames without auditwheel relinks; it still runs auditwheel on PyPI manylinux_*_armv7l wheels when no linux_armv7l sibling exists.
• Repair sets AUDITWHEEL_PLAT and AUDITWHEEL_ONLY_PLAT per lineage (manylinux_2_36_armv7l vs manylinux_2_31_armv7l) for that remaining manylinux path so repair_wheels.py can emit distinct single-tag filenames when auditwheel applies. If AUDITWHEEL_PLAT is set, ARMv7 “libc detection failed” outcomes are not treated as non-fatal skips for wheels that must be retagged (that would leave identical filenames across lineages).
• The repair workflow merges repaired artifacts using per-artifact subdirectories, then runs check_wheel_collisions.py to fail CI only if the same *.whl basename appears with different contents under both wheels-repaired-linux-armv7 and wheels-repaired-linux-armv7legacy and the wheel is not a pure universal (platform tag any only; those ZIPs can legitimately differ between lineages). Other duplicate basenames across macOS runners or pure-Python wheels are expected and are ignored, before flattening for tests/upload.

macOS Intel (x86_64) and cryptography

cryptography 49.0.0 removed macOS x86_64 wheels and builds official binaries against OpenSSL 4.0.1. A default sdist build on Intel Mac CI links against Homebrew openssl@3, which breaks frozen apps (PyInstaller Symbol not found: _SSL_get0_group_name — see esptool CI example). That is an OpenSSL linkage issue, not a PyInstaller version pin.

macOS Intel wheel builds always run os_dependencies/macos_openssl4_intel.sh (OpenSSL 4, OPENSSL_DIR, OPENSSL_STATIC=1) and pass --no-binary cryptography so cryptography is built from sdist against OpenSSL 4, then repaired with delocate. Find-links does not skip that sdist when it only has an older Intel wheel (or a wheel for another platform): 50.x and later are rebuilt the same way as 49.0.0 whenever PyPI has a newer matching release. Linux, Windows, and macOS ARM continue to use PyPI cryptography wheels as before.

Upstream dropped x86_64 macOS support in 49+; this path is maintained locally until ESP-IDF drops Intel Mac. Scheduled CI runs validate_cryptography_macos_intel_wheel.sh after macOS Intel builds and after delocate repair. For a local isolated rebuild smoke test, use spike_cryptography_macos_intel_openssl4.sh.

macOS: prefer PyPI wheels (avoid CI macosx_* tag stealing)

A local sdist build on GitHub macos-15-* runners often stamps a higher macosx_* tag (macosx_15_0_x86_64, …) than the official cibuildwheel tag (macosx_10_9_x86_64, macosx_11_0_arm64). Pip prefers the highest compatible tag, so the extra-index CI copy wins over PyPI. For psutil 7.2.2 that locally compiled _psutil_osx.abi3.so SIGABRT’d ESP-IDF export.sh / test_pytest_macos.

macOS pip wheel therefore passes --only-binary :all: (Intel cryptography still uses --no-binary cryptography for the OpenSSL 4 rebuild above). repair_wheels.py skips delocate except that Intel cryptography rebuild, and prunes any higher macosx_* sibling that competes for the same distribution, version, CPython (python+ABI family), and machine family (x86_64 / intel / universal2, or arm64 / universal2) so it cannot be uploaded. After install, every platform wheel is import-probed. Leftover higher-tag objects already on the extra-index (for example psutil 7.2.2) must be deleted from the bucket; a later upload only overwrites the same filename.

Activity Diagram

The main file is build-wheels-platforms.yml which is scheduled to run periodically to build Python wheels for any requirement of all ESP-IDF-supported versions.

The diagram was generated with the open-source tool PlantUML (and edited)

Note

Python version dependent wheels explanation

Python dependent wheels are wheels which depend on the CPython’s Application Binary Interface (ABI). These are checked based on the wheel filename format where the abi tag is checked for cp. Such wheels need to be build also for all supported Python versions, not only for the minimum Python version supported by ESP-IDF.

Custom Docker images

Docker files are in its own repository where there are build and published from. https://github.com/espressif/github-esp-dockerfiles

• ARMv7 runner - manylinux_armv7
• ARMv7 runner (older OSes) - manylinux_armv7_legacy
  • For older ARMv7 operating systems
  • For packages requiring glibc 2.31

Warning

piwheels may rely on system-provided shared libraries (i.e. may not bundle .libs/). If a target OS is missing those libraries or has an incompatible version, imports may fail at runtime.