= swift2nix: Run Swift inside Nix builds
:sort: 0

:empty:
:nix: https://nixos.org/
:swift2nix: link:swift2nix.tar.gz

While working on a Swift project, I didn't find any tool that would allow Swift
to run inside {nix}[Nix] builds.  Even thought you _can_ run Swift, the real
problem arises when using the package manager.  It has many of the same problems
that other package managers have when trying to integrate with Nix, more on this
below.

I wrote a simple little tool called {swift2nix}[swift2nix] that allows you trick
Swift's package manager into assuming everything is set up.  Here's the example
from swift2nix's README file:

[source,nix]
----
let
  niv-sources = import ./nix/sources.nix;
  pkgs = import niv-sources.nixpkgs { };
  src = pkgs.nix-gitignore.gitignoreSource [ ] ./.;
  swift2nix = pkgs.callPackage niv-sources.swift2nix {
    package-resolved = ./Package.resolved;
  };
in pkgs.stdenv.mkDerivation {
  inherit src;
  name = "swift-test";
  buildInputs = with pkgs; [ swift ];
  phases = [ "unpackPhase" "buildPhase" ];
  buildPhase = ''
    # Setup dependencies path to satisfy SwiftPM
    mkdir .build
    ln -s ${swift2nix.env.dependencies-state-json} .build/dependencies-state.json
    ln -s ${swift2nix.env.checkouts} .build/checkouts

    # Run the tests
    swift test
    touch $out
  '';
}
----

The key parts are lines 15~17: we just fake enough files inside `.build/` that
Swift believes it has already downloaded and checked-out all dependencies, and
just moves on to building them.

I've worked on it just enough to make it usable for myself, so beware of
unimplemented cases.

== Design

What swift2nix does is just provide you with the bare minimum that Swift
requires, and readily get out of the way:

. I explicitly did not want to generated a `Package.nix` file, since
  `Package.resolved` already exists and contains the required information;
. I didn't want to have an "easy" interface right out of the gate, after
  fighting with "*2nix" tools that focus too much on that.

The final actual code was so small (46 lines) that it made me think about
package managers, "*2nix" tools and some problems with many of them.

== Problems with package managers

I'm going to talk about solely language package managers.  Think npm and cargo,
not apt-get.

Package managers want to do too much, or assume too much, or just want to take
control of the entire build of the dependencies.

This is a recurrent problem in package managers, but I don't see it as an
intrinsic one.  There's nothing about a "package manager" that prevents it from
_declaring_ what it expects to encounter and in which format.  The _declaring_
part is important: it should be data, not code, otherwise you're back in the
same problem, just like lockfiles are just data.  Those work in any language,
and tools can cooperate happily.

There's no need for this declarative expectation to be standardized, or be made
compatible across languages.  That would lead to a poor format that no package
manager really likes.  Instead, If every package manager could say out loud what
it wants to see exactly, than more tools like swift2nix could exist, and they
would be more reliable.

This could even work fully offline, and be simply a mapping from the lockfile
(the `Package.resolved` in Swift's case) to the filesystem representation.  For
Swift, the `.build/dependencies-state.json` comes very close, but it is internal
to the package manager.

Even though this pain only exists when trying to use Swift inside Nix, it sheds
light into this common implicit coupling that package managers have.  They
usually have fuzzy boundaries and tight coupling between:

. resolving the dependency tree and using some heuristic to pick a package
  version;
. generating a lockfile with the exact pinned versions;
. downloading the dependencies present on the lockfile into some local cache;
. arranging the dependencies from the cache in a meaningful way for itself
  inside the project;
. work using the dependencies while _assuming_ that step 4 was done.

When you run `npm install` in a repository with no lockfile, it does 1~4.  If
you do the same with `cargo build`, it does 1~5.  That's too much: many of those
assumptions are implicit and internal to the package manager, and if you ever
need to rearrange them, you're on your own.  Even though you can perform some of
those steps, you can't compose or rearrange them.

Instead a much saner approach could be:

. this stays the same;
. this also stays the same;
. be able to generate some JSON/TOML/edn which represents the local expected
  filesystem layout with dependencies (i.e. exposing what the package manager
  expects to find), let's call it `local-registry.json`;
. if a `local-registry.json` was provided, do a build using that.  Otherwise
  generate its own, by downloading the dependencies, arranging them, _etc._

The point is just making what the package manager requires visible to the
outside world via some declarative data.  If this data wasn't provided, it can
move on to doing its own automatic things.

By making the expectation explicit and public, one can plug tools _à la carte_
if desired, but doesn't prevent the default code path of doing things the exact
same way they are now.

== Problems with "*2nix" tools

:node2nix: https://github.com/svanderburg/node2nix

I have to admit: I'm unhappy with most of they.

They conflate "using Nix" with "replicating every command of the package manager
inside Nix".

The avoidance of an "easy" interface that I mentioned above comes from me
fighting with some of the "*2nix" tools much like I have to fight with package
managers: I don't want to offload all build responsibilities to the "*2nix"
tool, I just want to let it download some of the dependencies and get out of the
way.  I want to stick with `npm test` or `cargo build`, and Nix should only
provide the environment.

This is something that {node2nix}[node2nix] does right.  It allows you to build
the Node.js environment to satisfy NPM, and you can keep using NPM for
everything else:

[source,sh]
----
ln -s ${node2nix-package.shell.nodeDependencies}/lib/node_modules ./node_modules
npm test
----

Its natural to want to put as much things into Nix as possible to benefit from
Nix's advantages.  Isn't that how NixOS itself was born?

But a "*2nix" tool should leverage Nix, not be coupled with it.  The above
example lets you run any arbitrary NPM command while profiting from isolation
and reproducibility that Nix provides.  It is even less brittle: any changes to
how NPM runs some things will be future-compatible, since node2nix isn't trying
to replicate what NPM does, or fiddling with NPM's internal.

**A "*2nix" tool should build the environment, preferably from the lockfile
directly and offload everything else to the package manager**.  The rest is just
nice-to-have.

swift2nix itself could provide an "easy" interface, something that allows you to
write:

[source,sh]
----
nix-build -A swift2nix.release
nix-build -A swift2nix.test
----

The implementation of those would be obvious: create a new
`pkgs.stdenv.mkDerivation` and call `swift build -c release` and `swift test`
while using `swift2nix.env` under the hood.

== Conclusion

Package managers should provide exact dependencies via a data representation,
i.e. lockfiles, and expose via another data representation how they expect those
dependencies to appear on the filesystem, i.e. `local-registry.json`.  This
allows package managers to provide an API so that external tools can create
mirrors, offline builds, other registries, isolated builds, _etc._

"*2nix" tools should build simple functions that leverage that
`local-registry.json`{empty}footnote:local-registry[
  This `local-registry.json` file doesn't have to be checked-in the repository
  at all.  It could be always generated on the fly, much like how Swift's
  `dependencies-state.json` is.
] data and offload all the rest back to the package manager itself.  This allows
the "*2nix" to not keep chasing the package manager evolution, always trying to
duplicate its behaviour.
