Meson-python and conda - am I doing it right?

I am trying to repackage my old code. I’d appreciate if someone could point out things I am doing the wrong way. This is my first project involving meson-python. The code works, I can install my package in n a conda environment and build conda packages. So, what I am really asking for, is code review.

I went with meson-python because I have other packages that include Fortran code. In the past I managed to build them by monkey-patching distutils. Anyway, here is what I came with:

pyproject.toml

[build-system]
build-backend = "mesonpy"
requires = [
    "cython >=0.29",
    "meson-python >=0.13",
    "numpy >=1.20",
]

[project]
name = "pyudunits"
dynamic = ["version"]
authors = [{name = "George Trojan", email = "george.trojan@gmail.com"}]
maintainers = [{name = "George Trojan", email = "george.trojan@gmail.com"}]
readme = "README.rst"
license = {file = "LICENCE"}
classifiers = [
    "Development Status :: 4 - Beta",
    "License :: OSI Approved :: GNU Lesser General Public License v3 (LGPLv3)",
    "Programming Language :: Python",
    "Programming Language :: Python :: 3.8",
    "Programming Language :: Python :: 3.9",
    "Programming Language :: Python :: 3.10",
    "Programming Language :: Python :: 3.11",
    "Topic :: Scientific/Engineering :: Atmospheric Science",
    "Topic :: Software Development :: Libraries :: Python Modules",
]
description="A simplistic units converter based on udunits2."
requires-python = ">=3.8"
# pip would try to find it
#dependencies = [
#  "udunits2"
#]

[project.optional-dependencies]
test = [
  "pytest >=7.0",
]

[project.scripts]
lsu = "pyudunits:lsu"

meson.build

project(
  'pyudunits',
  'c', 'cython',
  version: '2.0b0',
  meson_version: '>= 1.0',
  default_options: [
    'buildtype=debugoptimized',
    'c_std=c11',
  ],
)

py3 = import('python').find_installation(pure: false)

incdir_numpy = run_command(py3,
  [
    '-c',
    'import numpy; print(numpy.get_include())'
  ],
  check: true
).stdout().strip()

# Fails with pkg-config and cmake
# libudunits2 = dependency('libudunits2')
cc = meson.get_compiler('c')
# works when building a conda package, fails with pip
libudunits2 = cc.find_library('udunits2', has_headers: 'udunits2.h', required: false)
if libudunits2.found()
  inc_dirs = include_directories([incdir_numpy])
else
# Environmental variable. Un-mesonic! Is there a better way?
  incdir_conda = run_command(py3,
    [
      '-c',
      'import os; print(os.environ["CONDA_PREFIX"] + "/include")'
    ],
    check: true
  ).stdout().strip()

  inc_dirs = include_directories([incdir_conda, incdir_numpy])
  libudunits2 = cc.find_library('udunits2',
    header_include_directories: inc_dirs,
    has_headers: 'udunits2.h',
    required: true
  )
endif

py3.extension_module('pyudunits',
  'pyudunits.pyx', 'units.c',
  include_directories: inc_dirs,
  dependencies: libudunits2,
  install: true,
)

recipe/meta.yaml

{% set name = "pyudunits" %}
{% set version = "2.0.b0" %}
{% set build_number = 2 %}

package:
  name: {{ name|lower }}
  version: {{ version }}

source:
  path: ..

build:
  skip: true    # [not linux]
  number: {{ build_number }}
  script: "{{ PYTHON }} -m pip install . -v"

requirements:
  build:
    - {{ compiler('c') }}
  host:
    - python {{ python }}
    - meson-python >=0.13
    - numpy {{ numpy }}
    - cython >=0.29
    - udunits2 >=2.2
    - pip
  run:
    - python {{ python }}
    - numpy {{ numpy }}
    - udunits2 >=2.2

test:
  requires:
    - pytest
    - udunits2
    - numpy
  source_files:
    - tests/test_units.py
  commands:
    - pytest tests/ 

about:
  # maybe
  home: https://github.com/yt87/pyudunits
  license: GPL-3.0
  license_family: GPL
  license_file: LICENCE
  summary: Python wrapper for libudunits

extra:
  recipe-maintainers:
    - yt87

If it works already, great! But this is not the right place for code reviews on conda recipes. Conda has a discourse itself, or – if you feel your package is ready for a wider audience – you could submit your recipe to staged-recipes, where that sort of review happens before the package is published to conda-forge.

For code review or questions of meson-python, you’re also very welcome to ask at mesonbuild/meson-python · Discussions · GitHub. Now that it’s here, I’ll comment here though:

  • Overall the pyproject.toml looks perfectly fine to me.
    • I assume you do not want to distribute wheels, because then indeed the udunits2 dependency is going to be a pain, and then you’d also have to use oldest-supported-numpy instead of numpy>=1.20. For local builds and for conda packages, it looks fine as is.
  • The meson.build file looks mostly fine, there are two potential issues:
    • The incdir_numpy thing is a bit painful (I really need to make dependency('numpy') “just work”); if that starts failing on in-tree virtualenvs, use this instead.
    • It looks like you’re having a problem with dependency detection. You need either pkg-config or cmake installed as a build requirement, and if udunits2 does not come with the files needed for pkg-config or cmake to detect udunits2 then the detection will indeed fail (Meson won’t just go build and hope the right header is stumbled upon, like distutils would do). So what you then have to do is craft your own pkg-config file (and file a bug report against udunits2 ideally) - here is an example of how to do this: scipy-docs/blas-lapack.

Thanks. I will use the provided discussions fora in the future.

I do not want to distribute wheels, I find conda / mamba superior, because it manages non-python shared libraries.
With regard to udunits, I am not sure how to create pkg-config file when the library is installed by conda. What would be the paths in the *.pc file?

Conda doesn’t enter into the picture here really. If your installation process creates pkgconfig files and puts them where everything else gets installed as well, conda will bundle them with the package; if not, then not. IOW conda will not create the pkgconfig metadata for you, but that said, the path is usually $PREFIX/lib/pkgconfig/foo.pc on unix and %LIBRARY_PREFIX%\lib\pkgconfig\foo.pc on windows.

Conda-forge may enter the picture. I’d say that it’s a bug in udunits2 to not provide .pc files (and also .cmake files if it uses CMake). A library with no way to detect/consume it isn’t very useful. So a patch for udunits2 would be the way to go (not too difficult), but once you have that you can add it as a patch file in the recipe of conda-forge/udunits2-feedstock and rebuild the package to provide the fix before there is a new upstream release.

Here is how I’d look into an issue like this:

$ # create an environment with the library of interest
$ mamba create -n udunits2 
$ mamba activate udunits2
$ mamba install udunits2 cmake pkg-config
$ # Inspect what gets installed for headers and .pc/.cmake files
$ ls ~/mambaforge/envs/udunits2/lib/pkgconfig/
expat.pc        kdb.pc          libedit.pc      libuv.pc            ncurses++.pc   panelw.pc
form.pc         krb5-gssapi.pc  libkeyutils.pc  libzstd.pc          ncurses.pc     tinfo.pc
formw.pc        krb5.pc         liblzma.pc      menu.pc             ncurses++w.pc  tinfow.pc
gssrpc.pc       libcares.pc     libnghttp2.pc   menuw.pc            ncursesw.pc    zlib.pc
kadm-client.pc  libcrypto.pc    libssh2.pc      mit-krb5-gssapi.pc  openssl.pc
kadm-server.pc  libcurl.pc      libssl.pc       mit-krb5.pc         panel.pc
$ ls ~/mambaforge/envs/udunits2/include/udunit*
/home/rgommers/mambaforge/envs/udunits2/include/udunits2.h
/home/rgommers/mambaforge/envs/udunits2/include/udunits.h
$ ls ~/mambaforge/envs/udunits2/lib/cmake/
c-ares/      expat-2.5.0/ libssh2/     zstd/        
$ ls ~/mambaforge/envs/udunits2/lib/cmake/c-ares/
c-ares-config.cmake          c-ares-targets.cmake
c-ares-config-version.cmake  c-ares-targets-release.cmake

Conclusion: we’ve got .pc and .cmake files for a bunch of libraries, but not udunits2.

What we need now is a new udunits2.pc file. It should look like this:

prefix=/home/rgommers/mambaforge/envs/udunits2
libdir=${prefix}/lib
includedir=${prefix}/include

Name: udunits2
Description: udunits2 is .... 
Version: 2.2.28
Cflags: -I${includedir}

The version and prefix are the two things that need to be dynamically determined at build time, which is why it needs to be part of the udunits2 build. This goes into <prefix>/lib/pkg-config. Conda will take care of updating the absolute path in prefix so that the install is relocatable.

Just for completeness: it is possible to leave dependency('libudunits2') commented out forever. Things will mostly work as expected when udunits2 is correctly installed, and you’re using the default OS shell (Bash on Linux / Z shell on macOS). It has a bit worse behavior because:

  • If udunits2 is installed, you don’t get nice configure output like Dependency udunits2 found: YES 2.2.28
  • If udunits2 is missing, you get an obscure build error about a missing header halfway through your build, rather than an informative message at the start of it.
  • In some corner case situations, <prefix>/include/ may not be on the default search path, and the build will fail even if udunits2 is installed.

What you describe works when I build my package with pip:

python -m pip install --no-build-isolation .

but still fails when I run

conda build .

I managed to patch content of ~/miniforge3/pkgs/udunits2-2.2.28-hc3e0081_0. I added file lib/pkgs/udunits.pc:

prefix=/home/conda/feedstock_root/build_artifacts/udunits2_1643891763601/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pl
exec_prefix=${prefix}
libdir=${prefix}/lib/x86_64-linux-gnu

Name: udunits
Description: Library for handling of units of physical quantities
Version: 2.2.28
Libs:  -L${libdir} -ludunits2
Libs.private: -L${libdir} -ludunits2 -lexpat
CFlags: -I${prefix}/include

I also modified existing files in info to reflect the dynamic prefix. My understanding is that conda uses the pkgs directory as a cache when installing dependencies into the build environment. Of course, the build had to fail the first time.

Now I have to figure out how to convey this to conda-forge maintainers.

Thanks for help at the weekend.

Follow up: Missing pkgconfig file in udunits2 · conda-forge/conda-forge.github.io · Discussion #1957 · GitHub