From 960e4410f76801356ebd42801c914b2910a302a7 Mon Sep 17 00:00:00 2001 From: EuAndreh Date: Mon, 18 Nov 2024 08:21:58 -0300 Subject: v0 migration to mkwb --- src/content/blog/2020/10/05/cargo2nix-demo.tar.gz | Bin 0 -> 174080 bytes src/content/blog/2020/10/05/cargo2nix.adoc | 80 +++++++++ src/content/blog/2020/10/05/cargo2nix.tar.gz | Bin 0 -> 143360 bytes src/content/blog/2020/10/05/swift2nix-demo.tar.gz | Bin 0 -> 174080 bytes src/content/blog/2020/10/05/swift2nix.adoc | 199 ++++++++++++++++++++++ src/content/blog/2020/10/05/swift2nix.tar.gz | Bin 0 -> 143360 bytes 6 files changed, 279 insertions(+) create mode 100644 src/content/blog/2020/10/05/cargo2nix-demo.tar.gz create mode 100644 src/content/blog/2020/10/05/cargo2nix.adoc create mode 100644 src/content/blog/2020/10/05/cargo2nix.tar.gz create mode 100644 src/content/blog/2020/10/05/swift2nix-demo.tar.gz create mode 100644 src/content/blog/2020/10/05/swift2nix.adoc create mode 100644 src/content/blog/2020/10/05/swift2nix.tar.gz (limited to 'src/content/blog/2020/10/05') diff --git a/src/content/blog/2020/10/05/cargo2nix-demo.tar.gz b/src/content/blog/2020/10/05/cargo2nix-demo.tar.gz new file mode 100644 index 0000000..281a91c Binary files /dev/null and b/src/content/blog/2020/10/05/cargo2nix-demo.tar.gz differ diff --git a/src/content/blog/2020/10/05/cargo2nix.adoc b/src/content/blog/2020/10/05/cargo2nix.adoc new file mode 100644 index 0000000..1db3d0c --- /dev/null +++ b/src/content/blog/2020/10/05/cargo2nix.adoc @@ -0,0 +1,80 @@ += cargo2nix: Dramatically simpler Rust in Nix + +date: 2020-10-05 2 + +layout: post + +lang: en + +ref: cargo2nix-dramatically-simpler-rust-in-nix + +--- + +In the same vein of my earlier post on +[swift2nix]({% link _articles/2020-10-05-swift2nix-run-swift-inside-nix-builds.md %}), I +was able to quickly prototype a Rust and Cargo variation of it: +[cargo2nix]. + + +The initial prototype is even smaller than swift2nix: it has only +37 lines of code. + +[cargo2nix]: https://euandre.org/static/attachments/cargo2nix.tar.gz + +Here's how to use it (snippet taken from the repo's README): + +```nix +let + niv-sources = import ./nix/sources.nix; + mozilla-overlay = import niv-sources.nixpkgs-mozilla; + pkgs = import niv-sources.nixpkgs { overlays = [ mozilla-overlay ]; }; + src = pkgs.nix-gitignore.gitignoreSource [ ] ./.; + cargo2nix = pkgs.callPackage niv-sources.cargo2nix { + lockfile = ./Cargo.lock; + }; +in pkgs.stdenv.mkDerivation { + inherit src; + name = "cargo-test"; + buildInputs = [ pkgs.latest.rustChannels.nightly.rust ]; + phases = [ "unpackPhase" "buildPhase" ]; + buildPhase = '' + # Setup dependencies path to satisfy Cargo + mkdir .cargo/ + ln -s ${cargo2nix.env.cargo-config} .cargo/config + ln -s ${cargo2nix.env.vendor} vendor + + # Run the tests + cargo test + touch $out + ''; +} +``` + +That `cargo test` part on line 20 is what I have been fighting with every +"\*2nix" available for Rust out there. I don't want to bash any of them. All I +want is to have full control of what Cargo commands to run, and the "*2nix" tool +should only setup the environment for me. Let me drive Cargo myself, no need to +parameterize how the tool runs it for me, or even replicate its internal +behaviour by calling the Rust compiler directly. + +Sure it doesn't support private registries or Git dependencies, but how much +bigger does it has to be to support them? Also, it doesn't support those **yet**, +there's no reason it can't be extended. I just haven't needed it yet, so I +haven't added. Patches welcome. + +The layout of the `vendor/` directory is more explicit and public then what +swift2nix does: it is whatever the command `cargo vendor` returns. However I +haven't checked if the shape of the `.cargo-checksum.json` is specified, or +internal to Cargo. + +Try out the demo (also taken from the repo's README): + +```shell +pushd "$(mktemp -d)" +wget -O- https://euandre.org/static/attachments/cargo2nix-demo.tar.gz | + tar -xv +cd cargo2nix-demo/ +nix-build +``` + +Report back if you wish. diff --git a/src/content/blog/2020/10/05/cargo2nix.tar.gz b/src/content/blog/2020/10/05/cargo2nix.tar.gz new file mode 100644 index 0000000..8a9985a Binary files /dev/null and b/src/content/blog/2020/10/05/cargo2nix.tar.gz differ diff --git a/src/content/blog/2020/10/05/swift2nix-demo.tar.gz b/src/content/blog/2020/10/05/swift2nix-demo.tar.gz new file mode 100644 index 0000000..f688572 Binary files /dev/null and b/src/content/blog/2020/10/05/swift2nix-demo.tar.gz differ diff --git a/src/content/blog/2020/10/05/swift2nix.adoc b/src/content/blog/2020/10/05/swift2nix.adoc new file mode 100644 index 0000000..84f4d34 --- /dev/null +++ b/src/content/blog/2020/10/05/swift2nix.adoc @@ -0,0 +1,199 @@ += swift2nix: Run Swift inside Nix builds + +date: 2020-10-05 1 + +layout: post + +lang: en + +ref: swift2nix-run-swift-inside-nix-builds + +--- + +While working on a Swift project, I didn't find any tool that would allow Swift +to run inside [Nix][nix] builds. Even thought you *can* run Swift, the real +problem arises when using the package manager. It has many of the same problems +that other package managers have when trying to integrate with Nix, more on this +below. + +I wrote a simple little tool called [swift2nix] that allows you trick +Swift's package manager into assuming everything is set up. Here's the example +from swift2nix's README file: + +``` +let + niv-sources = import ./nix/sources.nix; + pkgs = import niv-sources.nixpkgs { }; + src = pkgs.nix-gitignore.gitignoreSource [ ] ./.; + swift2nix = pkgs.callPackage niv-sources.swift2nix { + package-resolved = ./Package.resolved; + }; +in pkgs.stdenv.mkDerivation { + inherit src; + name = "swift-test"; + buildInputs = with pkgs; [ swift ]; + phases = [ "unpackPhase" "buildPhase" ]; + buildPhase = '' + # Setup dependencies path to satisfy SwiftPM + mkdir .build + ln -s ${swift2nix.env.dependencies-state-json} .build/dependencies-state.json + ln -s ${swift2nix.env.checkouts} .build/checkouts + + # Run the tests + swift test + touch $out + ''; +} +``` + +The key parts are lines 15~17: we just fake enough files inside `.build/` that +Swift believes it has already downloaded and checked-out all dependencies, and +just moves on to building them. + +I've worked on it just enough to make it usable for myself, so beware of +unimplemented cases. + +[nix]: https://nixos.org/ +[swift2nix]: https://euandre.org/static/attachments/swift2nix.tar.gz + +## Design + +What swift2nix does is just provide you with the bare minimum that Swift +requires, and readily get out of the way: + +1. I explicitly did not want to generated a `Package.nix` file, since + `Package.resolved` already exists and contains the required information; +2. I didn't want to have an "easy" interface right out of the gate, after + fighting with "*2nix" tools that focus too much on that. + +The final actual code was so small (46 lines) that it made me +think about package managers, "*2nix" tools and some problems with many of them. + +## Problems with package managers + +I'm going to talk about solely language package managers. Think npm and cargo, +not apt-get. + +Package managers want to do too much, or assume too much, or just want to take +control of the entire build of the dependencies. + +This is a recurrent problem in package managers, but I don't see it as an +intrinsic one. There's nothing about a "package manager" that prevents it from +*declaring* what it expects to encounter and in which format. The *declaring* +part is important: it should be data, not code, otherwise you're back in the +same problem, just like lockfiles are just data. Those work in any language, and +tools can cooperate happily. + +There's no need for this declarative expectation to be standardized, or be made +compatible across languages. That would lead to a poor format that no package +manager really likes. Instead, If every package manager could say out loud what +it wants to see exactly, than more tools like swift2nix could exist, and they +would be more reliable. + +This could even work fully offline, and be simply a mapping from the lockfile +(the `Package.resolved` in Swift's case) to the filesystem representation. For +Swift, the `.build/dependencies-state.json` comes very close, but it is internal +to the package manager. + +Even though this pain only exists when trying to use Swift inside Nix, it sheds +light into this common implicit coupling that package managers have. They +usually have fuzzy boundaries and tight coupling between: + +1. resolving the dependency tree and using some heuristic to pick a package + version; +2. generating a lockfile with the exact pinned versions; +3. downloading the dependencies present on the lockfile into some local cache; +4. arranging the dependencies from the cache in a meaningful way for itself inside + the project; +5. work using the dependencies while *assuming* that step 4 was done. + +When you run `npm install` in a repository with no lockfile, it does 1~4. If you +do the same with `cargo build`, it does 1~5. That's too much: many of those +assumptions are implicit and internal to the package manager, and if you ever +need to rearrange them, you're on your own. Even though you can perform some of +those steps, you can't compose or rearrange them. + +Instead a much saner approach could be: + +1. this stays the same; +2. this also stays the same; +3. be able to generate some JSON/TOML/edn which represents the local expected + filesystem layout with dependencies (i.e. exposing what the package manager + expects to find), let's call it `local-registry.json`; +4. if a `local-registry.json` was provided, do a build using that. Otherwise + generate its own, by downloading the dependencies, arranging them, *etc.* + +The point is just making what the package manager requires visible to the +outside world via some declarative data. If this data wasn't provided, it can +move on to doing its own automatic things. + +By making the expectation explicit and public, one can plug tools *à la carte* +if desired, but doesn't prevent the default code path of doing things the exact +same way they are now. + +## Problems with "*2nix" tools + +I have to admit: I'm unhappy with most of they. + +They conflate "using Nix" with "replicating every command of the package manager +inside Nix". + +The avoidance of an "easy" interface that I mentioned above comes from me +fighting with some of the "\*2nix" tools much like I have to fight with package +managers: I don't want to offload all build responsibilities to the "*2nix" +tool, I just want to let it download some of the dependencies and get out of the +way. I want to stick with `npm test` or `cargo build`, and Nix should only +provide the environment. + +This is something that [node2nix] does right. It allows you to build +the Node.js environment to satisfy NPM, and you can keep using NPM for +everything else: + +```shell +ln -s ${node2nix-package.shell.nodeDependencies}/lib/node_modules ./node_modules +npm test +``` + +Its natural to want to put as much things into Nix as possible to benefit from +Nix's advantages. Isn't that how NixOS itself was born? + +But a "*2nix" tool should leverage Nix, not be coupled with it. The above +example lets you run any arbitrary NPM command while profiting from isolation +and reproducibility that Nix provides. It is even less brittle: any changes to +how NPM runs some things will be future-compatible, since node2nix isn't trying +to replicate what NPM does, or fiddling with NPM's internal. + +**A "*2nix" tool should build the environment, preferably from the lockfile +directly and offload everything else to the package manager**. The rest is just +nice-to-have. + +swift2nix itself could provide an "easy" interface, something that allows you to +write: + +```shell +nix-build -A swift2nix.release +nix-build -A swift2nix.test +``` + +The implementation of those would be obvious: create a new +`pkgs.stdenv.mkDerivation` and call `swift build -c release` and `swift test` +while using `swift2nix.env` under the hood. + +[node2nix]: https://github.com/svanderburg/node2nix + +## Conclusion + +Package managers should provide exact dependencies via a data representation, +i.e. lockfiles, and expose via another data representation how they expect those +dependencies to appear on the filesystem, i.e. `local-registry.json`. This +allows package managers to provide an API so that external tools can create +mirrors, offline builds, other registries, isolated builds, *etc.* + +"\*2nix" tools should build simple functions that leverage that +`local-registry.json`[^local-registry] data and offload all the rest back to the +package manager itself. This allows the "*2nix" to not keep chasing the package +manager evolution, always trying to duplicate its behaviour. + +[^local-registry]: This `local-registry.json` file doesn't have to be checked-in + the repository at all. It could be always generated on the fly, much like + how Swift's `dependencies-state.json` is. diff --git a/src/content/blog/2020/10/05/swift2nix.tar.gz b/src/content/blog/2020/10/05/swift2nix.tar.gz new file mode 100644 index 0000000..bfab3f1 Binary files /dev/null and b/src/content/blog/2020/10/05/swift2nix.tar.gz differ -- cgit v1.2.3