Fromager hooks and override plugins

For more complex customization requirements than are supported by the configuration file, create an override plugin.

Plugins are registered using entry points so they can be discovered and loaded at runtime. In pyproject.toml, configure the entry point in the project.entry-points."fromager.project_overrides" namespace to link the canonical distribution name to an importable module.

pyproject.toml snippet

[project.entry-points."fromager.project_overrides"]
flit_core = "package_plugins.flit_core"
pyarrow = "package_plugins.pyarrow"
torch = "package_plugins.torch"
triton = "package_plugins.triton"

The plugins are treated as providing overriding implementations of functions with default implementations, so it is only necessary to implement the functions needed to make it possible to build the package.

Package settings hooks

update_extra_environ(*, ctx: context.WorkContext, req: Requirement, version: Version | None, sdist_root_dir: pathlib.Path, extra_environ: dict[str, str], build_env: build_environment.BuildEnvironment) → None

Update extra_environ in-place

The update_extra_environ can modify the extra environment variables from settings file with dynamic values. The hook must update the extra_environ dict in-place.

The hook is called multiple times during a build. The version argument is None for get_build_backend_dependencies and get_build_sdist_dependencies. For get_install_dependencies_of_sdist, build_sdist, and build_wheel, the version argument contains the resolved version.

Added in version 0.60.

Dependency hooks

get_build_system_dependencies(ctx: context.WorkContext, req: Requirement, sdist_root_dir: pathlib.Path, build_dir: pathlib.Path) → Iterable[str]

Get build system requirements

Defaults to [build-system] requires from pyproject.toml.

The get_build_system_dependencies() function should return the PEP 517 build dependencies for a package.

The arguments are the WorkContext, the Requirement being evaluated, and the Path to the root of the source tree.

The return value is an iterable of requirement specification strings for build system dependencies for the package. The caller is responsible for evaluating the requirements with the current build environment settings to determine if they are actually needed.

get_build_backend_dependencies(ctx: context.WorkContext, req: Requirement, sdist_root_dir: pathlib.Path, build_dir: pathlib.Path, extra_environ: dict[str, str], build_env: build_environment.BuildEnvironment) → Iterable[str]

Get build backend dependencies

Defaults to result of hook call get_requires_for_build_wheel()

The get_build_backend_dependencies() function should return the PEP 517 build dependencies for a package.

The arguments are the WorkContext, the Requirement being evaluated, and the Path to the root of the source tree.

The return value is an iterable of requirement specification strings for build backend dependencies for the package. The caller is responsible for evaluating the requirements with the current build environment settings to determine if they are actually needed.

get_build_sdist_dependencies(ctx: context.WorkContext, req: Requirement, sdist_root_dir: pathlib.Path, build_dir: pathlib.Path, extra_environ: dict[str, str], build_env: build_environment.BuildEnvironment) → Iterable[str]

Get build sdist dependencies

Defaults to result of hook call get_requires_for_build_wheel()

The get_build_sdist_dependencies() function should return the PEP 517 dependencies for building the source distribution for a package.

The return value is an iterable of requirement specification strings for build backend dependencies for the package. The caller is responsible for evaluating the requirements with the current build environment settings to determine if they are actually needed.

Finder hooks

expected_source_archive_name(ctx: context.WorkContext, req: Requirement, dist_version: str) → str | None

The expected_source_archive_name() function is used to re-discover a source archive downloaded by a previous step, especially if the filename does not match the standard naming scheme for an sdist.

The arguments are the Requirement being evaluated and the version to look for.

The return value should be a string with the base filename (no paths) for the archive.

expected_source_directory_name(req: Requirement, dist_version: str) → str

The expected_source_directory_name() function is used to re-discover the location of a source tree prepared by a previous step, especially if the name does not match the standard naming scheme for an sdist.

The arguments are the Requirement being evaluated and the version to look for.

The return value should be a string with the name of the source root directory relative to the ctx.work_dir where it was prepared.

Resolver hooks

resolver_provider(ctx: context.WorkContext, req: Requirement, sdist_server_url: str, include_sdists: bool, include_wheels: bool, req_type: RequirementType | None = None, ignore_platform: bool = False) → PyPIProvider | GenericProvider | GitHubTagProvider | GitLabTagProvider | VersionMapProvider

Lookup resolver provider to resolve package versions

The get_resolver_provider() function allows an override to change the way requirement specifications are converted to fixed versions. The default implementation looks for published versions on a Python package index. Most overrides do not need to implement this hook unless they are building versions of packages not released to https://pypi.org.

For examples, refer to fromager.resolver.PyPIProvider, fromager.resolver.GitHubTagProvider, and fromager.resolver.GitLabTagProvider.

The arguments are the WorkContext, the Requirement being evaluated, a boolean indicating whether source distributions should be included, a boolean indicating whether built wheels should be included, and the URL for the sdist server.

The return value must be an instance of a class that implements the resolvelib.providers.AbstractProvider API.

The expectation is that it acts as an engine for any sort of package resolution whether it is for wheels or sources. The provider can therefore use any value as the “URL” that will help it decide what to download. For example, the GitHubTagProvider returns the actual tag name in case that is different from the version number encoded within that tag name.

The GenericProvider is a convenient base class, or can be instantiated directly if given a version_source callable that returns an iterator of version values as str or Version objects.

from fromager import resolver

VERSIONS = {
    "https://pkg.example/pkg-1.0.tar.gz": "1.0",
    "https://pkg.example/pkg-2.0.tar.gz": "2.0",
}

def _version_source(
        identifier: str,
    ) -> typing.Iterable[tuple[str, Version]]:
    return VERSIONS.items()


def get_resolver_provider(ctx, req, include_sdists, include_wheels, sdist_server_url):
    return resolver.GenericProvider(version_source=_version_source, constraints=ctx.constraints)

GenericProvider, GitHubTagProvider, and GitLabTagProvider take an optional matcher argument. The matcher can be a Callable[[str, str], Version | None] or a regular expression pattern object from re.compile The default match function attempts to convert a tag into a Version and ignores all errors.

def custom_tag_match(identifier: str, item: str) -> Version | None:
    # project-1_2_3 -> 1.2.3
    if item.startswith("project-"):
        return Version(item[8:].replace("_", "."))
    # ignore other tags
    return None

Source hooks

Removed in version 0.80.0: The resolve_source hook and default_resolve_source function were removed. Define a resolver_provider hook to resolve sources.

download_source(ctx: context.WorkContext, req: Requirement, version: Version, download_url: str, sdists_downloads_dir: pathlib.Path) → pathlib.Path

Download the requirement and return the name of the output path.

The download_source() function is responsible for downloading the source from a URL.

The arguments are the WorkContext, the Requirement being evaluated, version of the package being downloaded, the download URL, and the output directory in which the source should be downloaded.

The return value should be a pathlib.Path file path to the downloaded source.

prepare_source(ctx: context.WorkContext, req: Requirement, source_filename: pathlib.Path, version: Version) → tuple[pathlib.Path, bool]

Unpack, modify, and check sdist sources

Calls prepare_new_source() by default.

The prepare_source() function is responsible for setting up a tree of source files in a format that is ready to be built. The default implementation unpacks the source archive and applies patches.

The arguments are the WorkContext, the Requirement being evaluated, the Path to the source archive, and the version.

The return value should be the Path to the root of the source tree, ideally inside the ctx.work_dir directory.

build_sdist(ctx: context.WorkContext, extra_environ: dict, req: Requirement, version: Version, sdist_root_dir: pathlib.Path, build_env: build_environment.BuildEnvironment, build_dir: pathlib.Path) → pathlib.Path

The build_sdist() function is responsible for creating a new source distribution from the prepared source tree and placing it in ctx.sdists_build. The dist name and version of the sdist file must match the Requirement and Version.

The arguments are the WorkContext, the Requirement being evaluated, and the Path to the root of the source tree.

The return value is the Path to the newly created source distribution.

Source helper functions

The following helper functions are available in the fromager.sources module for use in custom source hooks:

fromager.sources.ensure_pkg_info(*, ctx: context.WorkContext, req: Requirement, version: Version, sdist_root_dir: pathlib.Path, build_dir: pathlib.Path | None = None) → bool

Ensure that sdist has a PKG-INFO file.

Returns True if PKG-INFO is present, False if file is missing. The function also updates build_dir if package has a non-standard build directory. Every sdist must have a PKG-INFO file in the first directory. The additional PKG-INFO file in build_dir is required for projects with a non-standard layout and projects using setuptools-scm.

fromager.sources.pep517_build_sdist(ctx: context.WorkContext, extra_environ: dict, req: Requirement, sdist_root_dir: pathlib.Path, version: Version, build_env: build_environment.BuildEnvironment) → pathlib.Path: Use the PEP 517 API to build a source distribution from a modified source tree.

fromager.sources.unpack_source(ctx: context.WorkContext, req: Requirement, version: Version, source_filename: pathlib.Path) → tuple[pathlib.Path, bool]: Extracts a downloaded source archive (.tar.gz or .zip file) into a standardized directory.

Wheel hooks

build_wheel(ctx: context.WorkContext, build_env: build_environment.BuildEnvironment, extra_environ: dict[str, str], req: Requirement, sdist_root_dir: pathlib.Path, version: Version, build_dir: pathlib.Path) → pathlib.Path

The build_wheel() function is responsible for creating a wheel from the prepared source tree and placing it in ctx.wheels_build. The default implementation uses PEP 517 pyproject hook. The dist name and version of the wheel must match the Requirement and Version.

The arguments are the WorkContext, the Path to a virtualenv prepared with the build dependencies, a dict with extra environment variables to pass to the build, the Requirement being evaluated, and the Path to the root of the source tree.

The return value is ignored.

add_extra_metadata_to_wheels(ctx: context.WorkContext, req: Requirement, version: Version, extra_environ: dict[str, str], sdist_root_dir: pathlib.Path, dist_info_dir: pathlib.Path) → dict[str, Any]

Default implementation returns empty dict - no extra metadata.

The add_extra_metadata_to_wheels() function is responsible to return any data the user would like to include in the wheels that fromager builds. This data will be added to the fromager-build-settings file under the .dist-info directory of the wheels. This file already contains the settings used to build that package.

The arguments available are WorkContext, Requirement being evaluated, the resolved Version of that requirement, a dict with extra environment variables, a Path to the root directory of the source distribution and a Path to the .dist-info directory of the wheel.

The return value must be a dict, otherwise it will be ignored.

Additional types

class fromager.build_environment.BuildEnvironment(ctx: context.WorkContext, parent_dir: pathlib.Path)

Wrapper for a virtualenv used for build isolation

property python: Path: Path to Python interpreter in virtual env

Run command in a virtual environment

network_isolation defaults to context setting.

class fromager.context.WorkContext(*, active_settings: packagesettings.Settings | None, patches_dir: pathlib.Path, sdists_repo: pathlib.Path, wheels_repo: pathlib.Path, work_dir: pathlib.Path, constraints_files: tuple[str, ...] = (), cleanup: bool = True, variant: str = 'cpu', network_isolation: bool = False, max_jobs: int | None = None, settings_dir: pathlib.Path | None = None, wheel_server_url: str = '', cooldown: candidate.Cooldown | None = None, max_release_age: datetime.timedelta | None = None)

class fromager.resolver.PyPIProvider(include_sdists: bool = True, include_wheels: bool = True, sdist_server_url: str = 'https://pypi.org/simple/', constraints: Constraints | None = None, req_type: RequirementType | None = None, ignore_platform: bool = False, *, use_resolver_cache: bool = True, override_download_url: str | None = None, cooldown: Cooldown | None = None, supports_upload_time: bool | None = None)

Lookup package and versions from a simple Python index (PyPI)

The override_download_url parameter supports the string template variable: * version (Version object)

class fromager.resolver.GenericProvider(version_source: VersionSource, constraints: Constraints | None = None, req_type: RequirementType | None = None, matcher: MatchFunction | Pattern | None = None, *, use_resolver_cache: bool = False, cooldown: Cooldown | None = None): Lookup package and version by using a callback

class fromager.resolver.GitHubTagProvider(organization: str, repo: str, constraints: Constraints | None = None, matcher: MatchFunction | Pattern | None = None, *, req_type: RequirementType | None = None, use_resolver_cache: bool = True, override_download_url: str | None = None, cooldown: Cooldown | None = None)

Lookup tarball and version from GitHub git tags

Assumes that upstream uses version tags 1.2.3 or v1.2.3.

The override_download_url parameter supports the string template variable: * organization * repo * tagname * version (Version object)

class fromager.resolver.GitLabTagProvider(project_path: str, server_url: str = 'https://gitlab.com', constraints: Constraints | None = None, matcher: MatchFunction | Pattern | None = None, *, req_type: RequirementType | None = None, use_resolver_cache: bool = True, override_download_url: str | None = None, cooldown: Cooldown | None = None)

Lookup tarball and version from GitLab git tags

The override_download_url parameter supports the string template variable: * hostname * project_path * project_name (last component of project_path) * tagname * version (Version object)

fromager.sources.prepare_new_source(ctx: context.WorkContext, req: Requirement, source_root_dir: pathlib.Path, version: Version) → None

Default steps for new sources

patch sources
apply project overrides from settings
vendor Rust dependencies

default_prepare_source() runs this function when the sources are new.