Using Nix flake inputs with PHP Composer

— Stéphan Kochen, 2024-02-17

For work, I wanted to package and deploy a PHP project using Nix. The project itself is in a private GitHub repository, and it has some dependencies that are also private.

The most straight-forward way to get started packaging PHP projects with Nix is using the buildComposerProject helper.

# default.nix
{ pkgs ? import <nixpkgs> { } }:
let
  version = "1.0.0";
in
pkgs.php.buildComposerProject (finalAttrs: {
  pname = "someproject";
  inherit version;
  src = pkgs.fetchFromGitHub {
    owner = "someone";
    repo = "someproject";
    rev = "v${version}";
    hash = "";
  };
  vendorHash = "";
})

(The easiest way to find the hashes is to just leave them empty, then try a build and look at the failure message.)

I much prefer using flakes, so that's what I used in this project. When you add a flake.nix to a project, access to the source code of the project itself is implicit, as far as the Nix code inside the flake is concerned.

# package.nix
{ php }:
php.buildComposerProject (finalAttrs: {
  pname = "someproject";
  version = "1.0.0";
  src = ./.;
  vendorHash = "";
})

# flake.nix
{
  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixpkgs-unstable";
    systems.url = "github:nix-systems/default";
  };
  outputs = { nixpkgs, ... }:
    let
      forEachSystem = nixpkgs.lib.genAttrs (import systems);
      pkgs = forEachSystem (system: import nixpkgs { inherit system; });
    in
    {
      packages = forEachSystem (system: {
        default = pkgs.${system}.callPackage ./package.nix { };
      });
    };
}

But when you consume the project/flake from somewhere else, Nix still has to do an authenticated fetch. For this, Nix supports providing access tokens for GitHub and GitLab:

# /etc/nix/nix.conf
access-tokens = github.com=...

This allows you to reference your private repository from some other flake, for example where you might keep your NixOS configuration:

# some other flake.nix
{
  inputs = {
    someproject.url = "github:PrivateOrg/someproject";
  };
  # ...
}

The above should work fine for private projects with only dependencies on public PHP packages, but what about dependencies on private PHP packages?

One way to go about this is to set the COMPOSER_AUTH environment variable inside the builder, but you have to be careful to keep credentials out of the derivation, or they'll end up world-readable in the Nix store. Nix provides the impureEnvVars attribute on fixed-output derivations to correctly accomplish this. I've used this in the past, but find managing environment variables in the various contexts Nix can run builds to be tricky. Additionally, applying impureEnvVars to the buildComposerProject family of helpers takes a bunch more Nix code.

Instead, I decided to try using flake inputs to fetch these packages.

Since we already have authenticated fetch set up for GitHub, I considered simply fetching the code from there:

# flake.nix
{
  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixpkgs-unstable";
    systems.url = "github:nix-systems/default";
    private-package = {
      url = "github:PrivateOrg/private-package/v1.0.0";
      flake = false;
    };
  };
  # ...
}

(flake = false means Nix only fetches the URL, and doesn't look for or evaluate a flake.nix inside it.)

Then we can try to use it as a Composer path-repository:

# composer.json
{
  "repositories": [
    {
      "type": "path",
      "url": "$PRIVATE_PACKAGE_PATH",
    }
  ],
  "require": {
    "privateorg/private-package": "*"
  }
}

Did you know you can use environment variables in a path-repository URL? Now you do! How do we provide this variable to our build? That's the neat part: you don't. It doesn't have to be set unless we're updating composer.lock, and we only ever do that inside our development environment. In my case, the development environment uses devenv:

# devenv.nix
{ inputs, ... }: {
  env.PRIVATE_PACKAGE_PATH = inputs.private-package;
}

Later on, I found this to be dangerous. We're asking Composer to record a Nix store path in composer.lock, which will only work when using a flake input (or fixed output derivation) directly. At some point I tried to add a derivation inbetween, and quickly noticed the path is no longer stable between platforms, because my development machine is a Mac, while I deploy on Linux.

A better solution is to have Composer record relative paths. The trick is to use a directory of symlinks, then prepare it for both the build and your development environment. Composer appears to use whatever path you give it as-is for generating composer.lock, which is great for our purpose.

# .gitignore
.flake-inputs

# composer.json
{
  "repositories": [
    {
      "type": "path",
      "url": ".flake-inputs/private-package",
    }
  ],
  "require": {
    "privateorg/private-package": "*"
  }
}

# flake.nix
{
  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixpkgs-unstable";
    systems.url = "github:nix-systems/default";
    devenv.url = "github:cachix/devenv";
    private-package = {
      url = "github:PrivateOrg/private-package/v1.0.0";
      flake = false;
    };
  };

  # For binary cache of devenv builds.
  nixConfig = {
    extra-trusted-public-keys = "devenv.cachix.org-1:w1cLUi8dv3hnoSPGAuibQv+f9TZLr6cv/Hm9XgU50cw=";
    extra-substituters = "https://devenv.cachix.org";
  };

  outputs = { self, nixpkgs, devenv, ... } @ inputs:
    let
      # Subset of inputs that contains our PHP dependencies.
      phpDependencies = {
        inherit (inputs) private-package;
      };

      forEachSystem = nixpkgs.lib.genAttrs (import systems);
      pkgs = forEachSystem (system: import nixpkgs {
        inherit system;
        overlays = [ self.overlays.default ];
      });
    in
    {
      # An overlay offers some more flexibility to the consumer, like
      # overriding nixpkgs to apply updates to system packages. It also makes
      # our package easier to reach from `devenv.nix`.
      overlays.default = final: prev: {
        someproject = final.callPackage ./package.nix {
          # Provide PHP dependencies to the package.
          inherit phpDependencies;
        };
      };
      packages = forEachSystem (system: {
        # So we can still do `nix build`.
        default = pkgs.${system}.someproject;
      });
      # Devenv based development shell.
      devShells = forEachSystem (system: {
        default = devenv.lib.mkShell {
          inherit inputs;
          pkgs = pkgs.${system};
          modules = [ ./devenv.nix ];
        };
      });
    };
}

# package.nix
{ lib, php, phpDependencies, runCommand }:
let
  # Creates a snippet that prepares .flake-inputs.
  mkComposerDeps = dir: ''
    mkdir -p ${dir}/.flake-inputs
  ''
    + lib.concatStrings
        (lib.mapAttrsToList
          (name: input: ''
            ln -sf ${input} ${dir}/.flake-inputs/${name}
          '')
          phpDependencies
        );
in
php.buildComposerProject (finalAttrs: {
  pname = "someproject";
  version = "1.0.0";
  # Add .flake-inputs to our sources.
  src = runCommand "someproject-src" { } ''
    cp -r ${./.} $out
    chmod u+w $out
    ${mkComposerDeps "$out"}
  '';
  vendorHash = "";
  # Don't need .flake-inputs afterwards, because packages were mirrored.
  # By removing the references to the flake inputs, we reduce closure size.
  postInstall = ''
    rm -f $out/share/php/someproject/.flake-inputs
  '';
  passthru = {
    # For devenv.nix
    inherit mkComposerDeps;
  };
})

# devenv.nix
{ pkgs, ... }: {
  # Ensure .flake-inputs is setup when we start a dev shell.
  enterShell = pkgs.someproject.mkComposerDeps "$DEVENV_ROOT";
}

I tried to structure this in a way so that adding a Composer dependency means you only modify composer.json and flake.nix. In general, I try to limit flake.nix to just glue code, as soon as things feel complex.

At this point, composer install will complain:

Root composer.json requires privateorg/private-package *, found privateorg/private-package[dev-main] but it does not match your minimum-stability.

Now we could simply set "minimum-stability": "dev" to work around this, but the underlying problem here is that Composer is unable to determine the version of our package. That's fine for a single dependency, but in my case, I was dealing with package interdependencies, and at that point Composer needs correct version information to resolve the dependency tree. Turns out, Packagist and other Composer repositories normally provide version information out-of-bounds, in a separate packages.json file.

But it's also possible to set the correct version for a package in a path-repository. You just need to set the version property in composer.json of the dependency. Sounds simple, but it really isn't:

  1. If we add it to the source repository of private-package, we need some (automated) process in place to keep it correct.

  2. We can also derive a copy with version set, but that requires tracking the version in multiple places. (It's unfortunately not possible to get at the URL of a flake input.)

In my specific case, our private-package actually already has a build step for assets, so trying to use the source directly was trouble to begin with. Our normal workflow is to have GitHub Actions build assets, zip up the package, then upload it to our custom Composer repository.

So we ended up adding the version attribute to our release packaging of private-package, which is essentially solution 1. Now we just need to fetch the release builds instead of the source code:

# flake.nix
{
  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixpkgs-unstable";
    systems.url = "github:nix-systems/default";
    devenv.url = "github:cachix/devenv";
    private-package = {
      url = "https://composer.example.com/dist/private-package/1.0.0";
      flake = false;
    };
  };
  # ...
}

Our Composer repository is private, though, so we need to configure an authenticated fetch again. The private repository I'm using has HTTPS + Basic auth, which Nix allows you to configure with a netrc file via the netrc-file setting:

# /etc/nix/nix.conf
netrc-file = /etc/nix/netrc

# /etc/nix/netrc
machine composer.example.com
login stephank
password abcdef...

Now Nix is able to fetch the package, but Composer complains again:

The url supplied for the path (.flake-inputs/private-package) repository does not exist

Turns out, we're fetching a ZIP file, whereas previously we had an unpacked source repository. We can actually ask Nix to unpack tarballs and ZIP files for us, but that didn't work for me either:

# flake.nix
# ...
private-package = {
  url = "tarball+https://composer.example.com/dist/private-package/1.0.0";
  flake = false;
};
# ...

error: tarball https://composer.example.com/dist/private-package/1.0.0 contains an unexpected number of top-level files

This appears to be a limitation in Nix with an open bug: https://github.com/NixOS/nix/issues/7083

So, we could add a derivation inbetween to unzip it ourselves, but there's actually a much better way: Composer has an artifact repository type that allows you to hand it a directory of ZIP files:

# composer.json
{
  "repositories": [
    {
      "type": "artifact",
      "url": ".flake-inputs",
    }
  ],
  "require": {
    "privateorg/private-package": "*"
  }
}

This also has the benefit that we no longer need to add a repository entry per dependency. Nice!

Then modify the mkComposerDeps function to create symlinks with a .zip extension:

ln -sf ${input} ${dir}/.flake-inputs/${name}.zip

And that's it! Both Composer and Nix should now be happy.

I do feel like getting here involved a lot of weird details, that maybe I missed some obvious simpler solution and this is all wrong. Certainly, if you have a large number of smaller private dependencies, this may become unwieldy. At that point, the impureEnvVars solution may be the better solution.