Learnings from the Python 3.14 migration

Personally, I was keen to ensure that Python 3.14 availability on conda-forge was as good as possible on the actual release day. conda-forge’s build infrastructure is a massive benefit, and I wanted to show others how this can result in useful progress for them.

During the work on enabling as many packages as possible to be installable with Python 3.14, I came across several things that could be prepared for the next Python release to improve the overall package availability on release day.

Building Python alphas and betas

The single most important obstacle to making Python builds available is the first successful release candidate (RC) build. This can be improved by working on the RC build as early as possible. However, as the majority of time is spent updating the conda-forge-specific patches to the newest Python version, we can already minimise this beforehand.

For the Python 3.14 series, work began with the PR to python_abi-feedstock to add Python 3.14 support on July 8. This was followed three days later with a PR to python-feedstock to add Python 3.14.0rc1. This PR was merged a month later, which meant that the first Python 3.14 build was available on August 11. Four days later, Python 3.14.0rc2 was merged. On August 20, we started the migration for Python 3.14. This means we started the migration 48 days before the final release.

To prepare for Python 3.15, we already have an open (and green) pull request for Python 3.15.0a2. The python_abi PR for Python 3.15 was already merged on November 16. While there will be several changes until the first release candidate is available, building early means that the incremental changes will be smaller. Thus, building the first release candidate should take less time if we maintain a steady pace of development releases.

While working on the Python 3.15.0a{1,2} build, I also manged to make my first CPython contribution. There is a new function called types_world_is_stopped in CPython that should only be compiled in certain (debug) settings. As conda-forge’s debug build is without assertions (whether this sould be, is another discussion), we did hit an edge case which is not tested by CPython’s CI. My contribution was then to extend the C macro guard to also cover this case. As this has been merged upstream, we can drop this patch in the next alpha release and aren’t accumulated more patches for the next release (for now).

Common Problems of Package Updates

While working on several pull requests that were failing during the Python 3.14 migration, it was easy to identify patterns where packages failed to update to Python 3.14. Many packages already had failing pull requests for Python 3.13, but would have been easily migratable to both Python releases. The problem there was often that for Python 3.13, we had dropped setuptools as a hard dependency in pip. Adding the build backend (setuptools) explicitly to the host environment fixed a lot of builds.

Another change in Python 3.14 was that it was more cautious about how often the reference count is increased for an object. Thus, some unit tests were failing in different packages that checked for refcount == 2, whereas in Python 3.14 these objects would only have a refcount of 1.

In this and many other (package-specific) failures, upstream often already had a fix merged into main, but not for the current release. Thus, to get a Python 3.14 build for the currently released version, one needs to backport this fix. My approach to get this working is by adding an exit 1 to the top of the build script and then running the build. Once the build hits the exit 1 in the script, it (obviously) fails. Then I can cd into the work directory of the current build and add the downloaded source code into a temporary git tree using git init . && git add . && git commit -m "Initial commit" --no-verify --no-gpg-sign. The upstream fix can then be obtained by adding .patch to the URL of the commit or pull request on GitHub. By using git am --reject <patch>, we can add that patch to the git tree and fix any conflicts that occur. We can then use git format-patch --no-signature HEAD^ to generate a clean patch that we can add to the recipe.

Dealing with (half) abandoned feedstock

As not all feedstocks are actively maintained, conda-forge/core can step in if a contribution has waited at least seven days for a review from the maintainer (there should be an explicit ping!). Many Python 3.14 pull requests were immediately successful; however, the maintainers were inactive. Thus, I manually reviewed them and explicitly notified the maintainers that they appear to be fine for merging. For a future release, it would be nice if the autotick bot could also automatically ping maintainers if a pull request has green builds, a CI entry for Python 3.x and has been open without any interaction for Y days. Then, conda-forge/core would only need to review the PRs that are at least 7-10 days old and have passing CI.

The seven-day waiting period on these PRs is a bit annoying if you want to ramp up the Python 3.x builds extremely fast, but given that we merged most dormant PRs 20 days after the final release and got the major packages (with dormant maintainers) in before the final release, we were already quite successful. One thing that would have helped me a lot in pinging dormant releases would have been to see the CI status for each pull request directly on the status page. While writing this post, showing the CI status in the migration table was contributed by someone to conda-forge.org.

Major nodes in the migration path

There is a limited number of central packages in the migration path. While there are some basic ones, such as setuptools, that are used by a huge number of downstream projects that already have Python 3.14 support early on, there are other packages that are not that early with it. Examples for this are pandas, numba and pytorch. These stood out as the packages that were (and still are) at the top of the list of important packages to migrate on the Python 3.14 migration status page. The earlier we hit those packages, the quicker we can also provide feedback to upstream on what works and what doesn’t. Similarly, in some cases, upstream actually uses a conda package in their CI. With us then hitting them in the migration tree, they often can start adding a Python 3.14 job to their CI, as all their dependencies are now available.

Even if they don’t use conda packages in their CI, and we cannot provide them with additional information on failures, hitting them faster ensures that we know that we will need to spend some time on them quite soon. This means that we can, in the meantime, invest in ensuring they are on the latest release and that the recipe meets modern standards.

Preparing for Python 3.15

While Python 3.14 was the release with the best availability curve on conda-forge, we want to ensure it is surpassed by the next release. Thus, we can already start preparing for the Python 3.15 release now. The most obvious one is starting to build Python 3.15 pre-releases as early as possible. This also enables us to submit patches upstream to CPython instead of keeping them local to conda-forge.

In the initial phase of the migration, CI is typically overloaded by the number of pull requests opened. Consequently, the queue often becomes so long that the migration needs to be paused, allowing the remaining work on conda-forge to continue. Thus, in this initial phase, we should ensure that the packages migrated first are all using rattler-build to minimise resources as much as possible.

Before we even start the migration, we can also migrate some packages for the upcoming Python release. Often, packages don’t need to be specific to a Python release. Some packages can be marked as noarch: python if they don’t contain any binary code at all. This is not possible for all non-binary packages, but over the years, we have extended the mechanics of noarch: python to accommodate many cases that could not be converted previously.

If packages contain binary code, they can still be built for a new Python release if they only utilise the limited API. The support for such packages (ABI3) is relatively new in conda-forge and thus seldom utilised. For example, most packages that use Rust code are actually ABI3-compatible and could be marked as such. This would also ease the load on CI in general, as the Python version matrix would be reduced to one entry per architecture.