Viktor Vilhelm Sonesten

Lessons from packaging FMPy to nixpkgs

I recently packaged FMPy to nixpkgs and learned a bunch of stuff. I figure I should enumerate them here, chiefly as a reference to myself. This might expand into some kind of living document down the line.

The do-this-rather-than-that below are specific to the FMPy packaging; they may not be universally applicable.

  1. Use finalAttrs instead of rec
  2. Use replaceVars instead of substituteInPlace when possible
  3. Minimize vendored patches
  4. Use versionCheckHook
  5. Put overridable expressions in passthru
  6. Test your package
  7. Put some thought into meta.broken and meta.platforms

Use finalAttrs instead of rec

What: do stdenv.mkDerivation (finalAttrs: { ... }) instead of stdenv.mkDerivation rec { ... }.

Why: rec adds at-a-glance diffusion to your expressions. Now you need to parse out where the recusion is implicit; finalAttrs makes the recursion stand out.

Example:

stdenv.mkDerivation (finalAttrs: {
  pname = "foobar";
  version = "1.2.3";
  src = fetchurl {
    url = "https://some.website.tld/${finalAttrs.pname}-${finalAttrs.version}.tar.gz";
  };
})

Use replaceVars instead of substituteInPlace when possible

What: do

patches = [
  (replaceVars ./override.patch {
    a = value;
    b = otherValue;
  })
];

instead of

patches = [ ./override.patch ];
postPatch = ''
  substituteInPlace /path/to/some/file \
    --replace-fail "@a@" "${value}" \
    --replace-fail "@b@" "${otherValue}"
'';

Why: cleaner, streamlined, easier to read, scales well.

Note that replaceVars is a specification of the substitution above. See the documentation.

Minimize vendored patches

What: when vendoring patches, minimize them and their requirements. Write patches like:

--- a/src/fmpy/sundials/libraries.py
+++ b/src/fmpy/sundials/libraries.py
@@ -9,6 +9,7 @@ if platform_tuple == 'aarch64_darwin':
     library_dir = library_dir / 'x86_64-darwin'
 else:
     library_dir = library_dir / platform_tuple
+library_dir = Path("@cvode@") / 'lib'

 # load SUNDIALS shared libraries
 sundials_nvecserial     = cdll.LoadLibrary(str(library_dir / f'sundials_nvecserial{sharedLibraryExtension}'))

That is: modify few lines (ideally, just add one), and dont use a precomputed hash (i.e. omit any index [hash before]..[hash after] [file mode]; cf. this patch, which doesn't do that — oops).

Why: this maximizes the possibility that the patch can be applied in future versions. A bad patch to vendor is one that must be amended whenever version is bumped.

Use versionCheckHook

What: use

nativeInstallCheckInputs = [ versionCheckHook ];

Why: the hook will check the build by running the main program with --help and --version and grep for "${finalAttrs.version}". It is a good sanity check, and easy to include.

Put overridable expressions in passthru

What: put expression one might want to override in passthru.

Why: FMPy depends on a custom build of sundials to access a specific solver. This is technical debt, so it is very convenient to make it easy to override this derivation via some fmpy.override { cvode = ...; } so that we can try another solver (or just bump the version).

Don't forget to actually use the passthru expression, lest it is a noop! Instead of someDep, use passthru.someDep.

Test your package

What: test the software you are packaging. If it is scientific software, dump some numerical results to $out/ if possible.

Why: just because a package builds does not mean that it will execute1. Output numbers are not of use for nixpkgs, but they are of use for the meta.maintainers to catch numerical instabilities when code and/or dependencies change. Cf. the simulation tests for FMPy.

Put some thought into meta.broken and meta.platforms

What: fill in all target platforms in meta.platforms and communicate further work via meta.broken.

Why: the software you're packagning probably (maybe) targets more than $(uname --kernel-name --machine); make sure these platforms are declared to reduce future work on the expression. If Nix supports these platforms, then the nixpkgs CI will spin up some environments to build and test execution. For meta.broken: conditionally set it for future work: perhaps some software feature does not build for complex reasons, or stuff just don't work at all for a lower priority platform (e.g. Darwin). Cf. the FMPy expression.

Good software will clearly state which platforms are supported, but you may have to infer from CI scripts or some CMakeLists.txt if that is not the case. Again, cf. FMPy.


Big thanks to everyone that review pull requests over at nixpkgs: you are an incredibly valuable resource!

Footnotes

1 This was especially the case for FMPy because it loads user-supplied shared objects during runtime (that may have been built outside of a Nix environment — I forgot to add a test for that in the PR, though; oops #2).