Instrumented CPython builds with Pixi

EuroPython 2026 Packaging Summit

Lucas Colley

2026-07-13

About Me

Motivation

  • Python libraries with compiled extension modules (like SciPy and PyArrow) want to test against compiler sanitizers
    • e.g. AddressSanitizer (ASan), ThreadSanitizer (TSan), UndefinedBehaviorSanitizer (UBSan)
    • this can help catch bugs, security vulnerabilities, and performance issues
  • However, for full effect, the whole stack of dependencies used at the C-level should be built with the sanitizers, not just the top-level library

Problem

  • How can a library like SciPy easily:
    • install each of its dependenices (CPython, NumPy, OpenBLAS) with each sanitizer?
    • build itself with each sanitizer against these dependencies and run tests?
  • How can this be done in a way which is easy for developers to adopt and understand?
  • How can this scale well to other libraries and sanitizers?

Just make bigger releases?

  • Including these instrumented builds in official releases of each library seems like a bad solution:
    • high release workload (for very small developer audience)
    • not customisable (can’t tweak compilers/ASAN_OPTIONS)
    • having builds tied to released versions does not help development work which spans multiple repos
  • Instead, we want to be able to build these instrumented variants from source, on demand and reproducibly.

Enter Pixi

Pixi

  • pixi is to conda as uv is to pip
    • fast, workspace-based workflow with lock files
  • Pixi handles both Python packages (uses uv under the hood) and conda packages
    • hence can handle dependencies like compilers and CPython
  • Pixi also handles building (potentially non-Python) conda packages from source, and recursive source dependencies.

The solution with Pixi

  • Define Pixi packages for each instrumented variant of each compiled dependency
  • These packages can then be consumed as build/run dependencies of other Pixi packages, and can be used in a project’s Pixi workspace to run tests

UX (1/2)

  • For SciPy, this enables a single pixi run test-asan invocation to trigger all of the following:
    • clone python/cpython and build an instrumented variant from a specific revision
    • clone numpy/numpy and build an instrumented variant from a specific revision, which has a build dependency on the python package just built
    • … (cont.)

UX (2/2)

  • For SciPy, this enables a single pixi run test-asan invocation to trigger all of the following:
    • … (cont.)
    • install these dependencies alongside SciPy’s other dependencies in a locked environment
    • build an instrumented variant of the local scipy source, including build dependencies on the above packages
    • run tests against this instrumented scipy

This one-liner is a lot more accessible and robust than before…

An open PR to SciPy which documented the process before this work:

#. Set ``ASAN_OPTIONS=detect_leaks=0:symbolize=1:strict_init_order=true:allocator_may_return_null=1:use_sigaltstack=0``
#. Compile CPython_ with ``--with-address-sanitizer`` and use it to setup a test environment
#. Build NumPy_ with the ``setup-args`` config setting set to ``-Db_sanitize=address``
    * Eg. ``pip install numpy --no-cache-dir --no-binary numpy -Csetup-args="-Db_sanitize=address" -v``
#. In the project root directory, set the compiler flags as follows:
    * ``CFLAGS="-fsanitize=address -fno-omit-frame-pointer -fsanitize-ignorelist=$(pwd)/tools/asan-ignore.txt"``
    * ``CXXFLAGS="-fsanitize=address -fno-omit-frame-pointer -fsanitize-ignorelist=$(pwd)/tools/asan-ignore.txt"``
#. Build SciPy using ``spin build --clean``

What is maintained where? (1/2)

  • python/cpython/Tools/pixi-packages:
    • recipe defining how to build each instrumented variant from the cpython source
    • Pixi package definitions which can be pointed to via the git URL and revision
  • likewise for numpy/numpy/pixi-packages (the same pattern could follow for other dependencies)

What is maintained where? (2/2)

  • a library like SciPy maintains how it uses these packages in its Pixi workspace:
[feature.asan.dependencies]
python.flags = ["asan"]
python.git = "https://github.com/lucascolley/cpython"
python.subdirectory = "Tools/pixi-packages"
python.rev = "8b5b0c29797cf88d78ef014916a5e5a5d51bbf95"

numpy.flags = ["asan"]
numpy.git = "https://github.com/lucascolley/numpy"
numpy.subdirectory = "pixi-packages/asan"
numpy.rev = "6bc853b2fbb3721332de7e5dc44ea9b282bc260a"

Status

  • SciPy PR demonstrating the full stack working for ASan is under review
  • PyArrow also started experimenting with these packages in this PR for ASan and UBSan

Discussion

  • are people happy with this approach?
  • what are the challenges left to tackle?
  • can we speed up the builds/tests?
  • what is needed to move the CPython/NumPy/SciPy PRs forward?
  • what is needed to enable adoption in PyArrow and other libraries?