Using Build Containers ====================== We can revisit the :doc:`getting-started` example using containers in podman to manage build dependencies and tools in a repeatable way. We will use `pydantic-core` and a `Universal Base Image (UBI) for Red Hat Enterprise Linux 9 `__ to demonstrate debugging and fixing a build failure. Inputs ------ We start with the same ``requirements.txt`` file: .. literalinclude:: ../example/requirements.txt :caption: requirements.txt and an empty `constraints.txt`. The build container includes Python, rust, and a virtualenv with fromager installed: .. literalinclude:: ../example/Containerfile :caption: Containerfile First try --------- Then we can use the ``bootstrap.sh`` script in the ``docs/example`` directory to build and test the image: .. literalinclude:: ../example/bootstrap.sh :caption: bootstrap.sh The output below is redacted for brevity. Missing sections are replaced with ``...``. .. code-block:: console :emphasize-lines: 61 $ cd docs/example $ ./bootstrap.sh Containerfile ./constraints.txt ./requirements.txt ... podman run -it --rm --security-opt label=disable --volume ./bootstrap-output:/work/bootstrap-output:rw,exec --volume ./bootstrap-ccache:/var/cache/ccache:rw,exec --volume ././constraints.txt:/bootstrap-inputs/constraints.txt --volume ././requirements.txt:/bootstrap-inputs/requirements.txt wheels-builder fromager --constraints-file /bootstrap-inputs/constraints.txt --log-file=bootstrap-output/bootstrap.log --sdists-repo=bootstrap-output/sdists-repo --wheels-repo=bootstrap-output/wheels-repo --work-dir=bootstrap-output/work-dir bootstrap -r /bootstrap-inputs/requirements.txt logging debug information to bootstrap-output/bootstrap.log primary settings file: overrides/settings.yaml per-package settings dir: overrides/settings variant: cpu-fedora patches dir: overrides/patches maximum concurrent jobs: None constraints file: /bootstrap-inputs/constraints.txt wheel server url: network isolation: True loading constraints from /bootstrap-inputs/constraints.txt bootstrapping 'cpu-fedora' variant of [] no previous bootstrap data resolving top-level dependencies before building pydantic-core==2.18.4 resolves to 2.18.4 ... pydantic-core: building wheel for pydantic-core==2.18.4 in /work/bootstrap-output/work-dir/pydantic_core-2.18.4/pydantic_core-2.18.4 writing to /work/bootstrap-output/wheels-repo/build ['/usr/bin/unshare', '--net', '--map-current-user', '/work/bootstrap-output/work-dir/pydantic_core-2.18.4/build-3.11.7/bin/python3', '-m', 'pip', '-vvv', '--disable-pip-version-check', 'wheel', '--no-build-isolation', '--only-binary', ':all:', '--wheel-dir', '/work/bootstrap-output/wheels-repo/build', '--no-deps', '--index-url', 'http://localhost:45837/simple/', '--log', '/work/bootstrap-output/work-dir/pydantic_core-2.18.4/build.log', '/work/bootstrap-output/work-dir/pydantic_core-2.18.4/pydantic_core-2.18.4'] failed with Created temporary directory: /tmp/pip-build-tracker-q7oc2ftp Initialized build tracking at /tmp/pip-build-tracker-q7oc2ftp Created build tracker: /tmp/pip-build-tracker-q7oc2ftp Entered build tracker: /tmp/pip-build-tracker-q7oc2ftp Created temporary directory: /tmp/pip-wheel-_ol69xlg Created temporary directory: /tmp/pip-ephem-wheel-cache-d13st1qz Looking in indexes: Processing /work/bootstrap-output/work-dir/pydantic_core-2.18.4/pydantic_core-2.18.4 Added file:///work/bootstrap-output/work-dir/pydantic_core-2.18.4/pydantic_core-2.18.4 to build tracker '/tmp/pip-build-tracker-q7oc2ftp' Created temporary directory: /tmp/pip-modern-metadata-q7vigowe Preparing metadata (pyproject.toml): started Running command Preparing metadata (pyproject.toml) 📦 Including license file "/work/bootstrap-output/work-dir/pydantic_core-2.18.4/pydantic_core-2.18.4/LICENSE" 🍹 Building a mixed python/rust project 🔗 Found pyo3 bindings 🐍 Found CPython 3.11 at /work/bootstrap-output/work-dir/pydantic_core-2.18.4/build-3.11.7/bin/python3 📡 Using build options features, bindings from pyproject.toml pydantic_core-2.18.4.dist-info Checking for Rust toolchain.... Running `maturin pep517 write-dist-info --metadata-directory /tmp/pip-modern-metadata-q7vigowe --interpreter /work/bootstrap-output/work-dir/pydantic_core-2.18.4/build-3.11.7/bin/python3` Preparing metadata (pyproject.toml): finished with status 'done' Source in /work/bootstrap-output/work-dir/pydantic_core-2.18.4/pydantic_core-2.18.4 has version 2.18.4, which satisfies requirement pydantic_core==2.18.4 from file:///work/bootstrap-output/work-dir/pydantic_core-2.18.4/pydantic_core-2.18.4 Removed pydantic_core==2.18.4 from file:///work/bootstrap-output/work-dir/pydantic_core-2.18.4/pydantic_core-2.18.4 from build tracker '/tmp/pip-build-tracker-q7oc2ftp' Created temporary directory: /tmp/pip-unpack-ye9iobqx Created temporary directory: /tmp/pip-unpack-j0zwnxci Building wheels for collected packages: pydantic_core Created temporary directory: /tmp/pip-wheel-3o238ckb Destination directory: /tmp/pip-wheel-3o238ckb Building wheel for pydantic_core (pyproject.toml): started Running command Building wheel for pydantic_core (pyproject.toml) Running `maturin pep517 build-wheel -i /work/bootstrap-output/work-dir/pydantic_core-2.18.4/build-3.11.7/bin/python3 --compatibility off` 📦 Including license file "/work/bootstrap-output/work-dir/pydantic_core-2.18.4/pydantic_core-2.18.4/LICENSE" 🍹 Building a mixed python/rust project 🔗 Found pyo3 bindings 🐍 Found CPython 3.11 at /work/bootstrap-output/work-dir/pydantic_core-2.18.4/build-3.11.7/bin/python3 📡 Using build options features, bindings from pyproject.toml error: package `pydantic-core v2.18.4 (/work/bootstrap-output/work-dir/pydantic_core-2.18.4/pydantic_core-2.18.4)` cannot be built because it requires rustc 1.76 or newer, while the currently active rustc version is 1.75.0 💥 maturin failed Caused by: Failed to build a native library through cargo Caused by: Cargo build finished with "exit status: 101": `env -u CARGO PYO3_ENVIRONMENT_SIGNATURE="cpython-3.11-64bit" PYO3_PYTHON="/work/bootstrap-output/work-dir/pydantic_core-2.18.4/build-3.11.7/bin/python3" PYTHON_SYS_EXECUTABLE="/work/bootstrap-output/work-dir/pydantic_core-2.18.4/build-3.11.7/bin/python3" "cargo" "rustc" "--features" "pyo3/extension-module" "--message-format" "json-render-diagnostics" "--manifest-path" "/work/bootstrap-output/work-dir/pydantic_core-2.18.4/pydantic_core-2.18.4/Cargo.toml" "--release" "--lib" "--crate-type" "cdylib"` Error: command ['maturin', 'pep517', 'build-wheel', '-i', '/work/bootstrap-output/work-dir/pydantic_core-2.18.4/build-3.11.7/bin/python3', '--compatibility', 'off'] returned non-zero exit status 1 error: subprocess-exited-with-error × Building wheel for pydantic_core (pyproject.toml) did not run successfully. │ exit code: 1 ╰─> See above for output. note: This error originates from a subprocess, and is likely not a problem with pip. full command: /work/bootstrap-output/work-dir/pydantic_core-2.18.4/build-3.11.7/bin/python3 /work/bootstrap-output/work-dir/pydantic_core-2.18.4/build-3.11.7/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py build_wheel /tmp/tmpamjy51ga cwd: /work/bootstrap-output/work-dir/pydantic_core-2.18.4/pydantic_core-2.18.4 Building wheel for pydantic_core (pyproject.toml): finished with status 'error' ERROR: Failed building wheel for pydantic_core Failed to build pydantic_core ERROR: Failed to build one or more wheels Using the pre_built flag ------------------------ The important error embedded in that output is: .. code-block:: console cannot be built because it requires rustc 1.76 or newer, while the currently active rustc version is 1.75.0 So, the version of the Rust compiler we have available in the builder image is too old to use to build this version of ``pydantic-core``. Let's mark ``pydantic-core`` as pre-built and see if any of its installation dependencies present similar issues. To mark the package as pre-built, create a settings file using fromager's canonical form of the package name: .. code-block:: console $ fromager canonicalize pydantic-core pydantic_core .. code-block:: yaml :caption: overrides/settings/pydantic_core.yaml variants: cpu-ubi9: pre_built: true Now when we re-run ``bootstrap.sh``, we see that ``pydantic-core`` will be treated as "pre-built", the wheel is downloaded from pypi.org, and the process completes successfully. .. code-block:: console :emphasize-lines: 14 $ podman run -it --rm --security-opt label=disable --volume ./bootstrap-output:/work/bootstrap-output:rw,exec --volume ./bootstrap-ccache:/var/cache/ccache:rw,exec --volume ././constraints.txt:/bootstrap-inputs/constraints.txt --volume ././requirements.txt:/bootstrap-inputs/requirements.txt wheels-builder fromager --constraints-file /bootstrap-inputs/constraints.txt --log-file=bootstrap-output/bootstrap.log --sdists-repo=bootstrap-output/sdists-repo --wheels-repo=bootstrap-output/wheels-repo --work-dir=bootstrap-output/work-dir bootstrap -r /bootstrap-inputs/requirements.txt logging debug information to bootstrap-output/bootstrap.log primary settings file: overrides/settings.yaml per-package settings dir: overrides/settings variant: cpu-ubi9 patches dir: overrides/patches maximum concurrent jobs: None constraints file: /bootstrap-inputs/constraints.txt wheel server url: network isolation: True loading constraints from /bootstrap-inputs/constraints.txt bootstrapping 'cpu-ubi9' variant of [] no previous bootstrap data treating ['pydantic-core'] as pre-built wheels ... 100%|██████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:07<00:00, 2.35s/pkg] removing prebuilt wheel pydantic_core-2.18.4-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl from download cache writing installation dependencies to /work/bootstrap-output/work-dir/constraints.txt Iteratively debugging the build ------------------------------- Since there are no other build issues, we can start working on making ``pydantic-core`` build. The error caused by the rust version is actually an error in the build settings of ``pydantic-core`` that was `fixed in a later release `__. Until that upstream fix is released, we can have fromager apply a similar patch. Start by removing the settings with the ``pre-built`` flag to have fromager try to build from source. Then add this patch file to change the ``rust-version`` setting: .. literalinclude:: ../example/overrides/patches/pydantic_core-2.18.4/0001-rust-version.patch :caption: overrides/patches/pydantic_core-2.18.4/0001-rust-version.patch :emphasize-lines: 12-13 Then by running ``bootstrap.sh`` again, we can see the patch being applied .. code-block:: console :emphasize-lines: 6 ... pydantic-core: * handling toplevel requirement pydantic-core==2.18.4 [] pydantic-core: new toplevel dependency pydantic-core==2.18.4 resolves to 2.18.4 pydantic-core: preparing source for pydantic-core==2.18.4 from /work/bootstrap-output/sdists-repo/downloads/pydantic_core-2.18.4.tar.gz applying patch file overrides/patches/pydantic_core-2.18.4/0001-rust-version.patch to /work/bootstrap-output/work-dir/pydantic_core-2.18.4/pydantic_core-2.18.4 pydantic-core: updating vendored rust dependencies in /work/bootstrap-output/work-dir/pydantic_core-2.18.4/pydantic_core-2.18.4 pydantic-core: prepared source for pydantic-core==2.18.4 at /work/bootstrap-output/work-dir/pydantic_core-2.18.4/pydantic_core-2.18.4 pydantic-core: getting build system dependencies for pydantic-core==2.18.4 in /work/bootstrap-output/work-dir/pydantic_core-2.18.4/pydantic_core-2.18.4 followed by the wheel being built successfully. .. code-block:: console :emphasize-lines: 13 pydantic-core: getting build backend dependencies for pydantic-core==2.18.4 in /work/bootstrap-output/work-dir/pydantic_core-2.18.4/pydantic_core-2.18.4 pydantic-core: getting build sdist dependencies for pydantic-core==2.18.4 in /work/bootstrap-output/work-dir/pydantic_core-2.18.4/pydantic_core-2.18.4 pydantic-core: adding ('pydantic-core', '2.18.4') to build order pydantic-core: preparing to build wheel for version 2.18.4 created build environment in /work/bootstrap-output/work-dir/pydantic_core-2.18.4/build-3.11.7 installed dependencies into build environment in /work/bootstrap-output/work-dir/pydantic_core-2.18.4/build-3.11.7 pydantic-core: building source distribution in /work/bootstrap-output/work-dir/pydantic_core-2.18.4/pydantic_core-2.18.4 pydantic-core: built source distribution /work/bootstrap-output/sdists-repo/builds/pydantic_core-2.18.4.tar.gz pydantic-core: building wheel for pydantic-core==2.18.4 in /work/bootstrap-output/work-dir/pydantic_core-2.18.4/pydantic_core-2.18.4 writing to /work/bootstrap-output/wheels-repo/build pydantic-core: Requires libraries: libc.so.6, libgcc_s.so.1, libm.so.6 pydantic-core: added extra metadata and build tag (0, ''), wheel renamed from pydantic_core-2.18.4-cp311-cp311-linux_x86_64.whl to pydantic_core-2.18.4-0-cp311-cp311-linux_x86_64.whl pydantic-core: built wheel '/work/bootstrap-output/wheels-repo/build/pydantic_core-2.18.4-0-cp311-cp311-linux_x86_64.whl' in 0:03:53 pydantic-core: built wheel for version 2.18.4: /work/bootstrap-output/wheels-repo/downloads/pydantic_core-2.18.4-0-cp311-cp311-linux_x86_64.whl The :doc:`customization` section explains other techniques for changing the build inputs to ensure packages build properly. The collection of wheels you want to build may have different build-time issues, but you can use this iterative approach to work your way though them until they all build.