There are noble goals to this project to rewriting Nix to improve performance. The reasons feel valid. However I wonder about certain issues:
- Nix itself has problems of compatibility with Nix! Let me explain...recently NixOS 21.11 released. It still comes packaged with Nix 2.3.16 instead of Nix 2.4 because of incompatibility issues. When the Nix project itself is not able to deliver backwards compatibility I feel uncertain that Tvix might be able to deliver their promise of full compatibility with all the deep architectural changes it plans. Nix does not have a formal specification: Nix is what its C/C++ implementation says. Nix also has deep architectural changes planned going ahead. So Tvix will need to re-architect Nix keeping in mind (and keeping up) with the future changes that are coming. It's going to be tough to build this "alternative" implementation. Users are going to run into weird compatibility issues, not have the patience to resolve them and then just go back to the original Nix implementation.
- A rewrite to a popular project will find it difficult to succeed. There is simply too much momentum in the parent Nix implementation (I'm not talking about the package definitions here) that very few people are likely to adopt this rewrite. This is because most people might not really need the speedup. However, it's possible that the pain around slow evaluation runs so deep that other companies might adopt this too. I don't know enough here.
- The rewrite needs to offer more reason to shift. Right now this rewrite seems to be mainly about performance. There is also a proposal to be able to integrate with GNU Guile. To me this is just avoiding one untyped language (Nix) to replace with another slightly better untyped language (Guile Scheme) (GNU Guix uses Guile also AFAIK). What about introducing typing? https://github.com/tweag/nickel is an interesting project and it would be great to have a system like Nix with optional typing to make writing the derivations more robust. What about rewriting in a safe programming language (e.g. Rust or a GC-ed language)? These things would be likely to excite would-be-contributors and would make your rewrite viable. In other words, the rewrite needs to offer more than just the promise of performance.
Perhaps I have not understood the project goals fully. Maybe the project is all about just being able to leverage the Nix package definitions but have a totally different implementation to instantiate them? But the issue here is that nix-the-language is very intimately tied up with the nix-the-platform and the rewritten implementation will still need to really be extremely compatible to capture all the implicit assumptions in nixpkgs.
Once again, improving nix through a rewrite is a noble goal. It's a tough problem and it would be awesome if the project succeeds!
I have obviously not thought about this as much as you guys and would love to learn how you wish to tackle some of these problems. Maybe some of these issues are not as big as them seem??
I would also like it if the language itself can be decoupled from the build and package manager system, so that it can be embedded in other software that would benefit from a language like nix. I think that might be a worthy goal and could attract people outside of the nix sphere.
Yep, we'd like that, too! Our current thinking around the evaluator design incorporates the idea of evaluation with/without a Nix store, where using the language itself (e.g. to generate JSON structures or whatever) should be perfectly possible without a store.
Dhall was inspired by Nix, and created by a longtime Nixer.
Also as someone who likes simple FP but is not some sophisticated typelevel programmer, Dhall looks really complicated and verbose. To me, the Nickel approach, where authors of libraries in the language (equivalent to things like nixpkgs.lib) can pepper their functions with type annotations, but users can use what looks and feels like a simple configuration language, seems like a better approach for the domain.
I can't imagine getting all of the PHP developers I support comfortable editing the equivalent of shell.nix in Dhall themselves. Nix files I can ask them to edit without taking up too much of their time or focus.
> I would also like it if the language itself can be decoupled from the build and package manager system
I would also like that, but for the exact opposing reason. I love the concepts of nix/guix but i can't stand the dynamic languages with cryptic error messages.
It is a pain but it is manageable with practice. Also the error messages on Nix 2.4 are a lot better and can show the location of errors which is enough 90% of the time.
Nix 2.4 is totally compatible with NixOS. It was held back from default status on the latest NixOS release because it includes a few behavior changes to the CLI that have been jarring for some users. It's to ease the transition for them, not because it's broken or something like that.
You can set up Nix 2.4 on NixOS 21.11 by adding
nix.package = pkgs.nix_2_4;
in your configuration.nix :)
I like the new CLI, too, and it's worth trying if you want to see what all is new in Nix. Here's how to enable flakes on NixOS, if you have Nix 2.4:
Thanks for the information! I've heard about flakes but i failed to understand the usefulness of them... from what i gather it's a very heated topic in the nix community which seems to be splitting the ecosystem. Can you recommend in-depth resources about the pros and cons of both approaches from a critical perspective?
As far as technical pros and cons, it's worth looking at colemickens' PoC demonstrating that in principle, you can use Nix to generate literally identical output with and without flakes: https://github.com/colemickens/nixos-flake-example
> Both approaches
it's also worth noting that prior to flakes, there was no single alternative approach to the main problems flakes aims to solve. There were (and are) a whole bunch of competing and bespoke practices for pinning nixpkgs and other Nix sources. You can take a look at niv and nix-thunk to get a sense of them if you want
It has been released. The 20.11 release contains both a nix_2_3 and a nix_2_4 package. I'm fairly certain the only reason 20.11 defaults to 2.3 is because the development of 2.4 focused a lot on flakes despite the fact that there are still an 'experimental' feature (experimental like Gmail was a beta for 7+ years). If you aren't using Nix already there is no reason you can't just start using 2.4 with flakes enabled.
Some other comments in this thread (i have no idea how true they are) suggested there were serious backwards-incompatibilities preventing nix2.4 from being default in the last NixOS release.
With regards to the language. Is turing completeness really needed? google decided for Bazel that it is not desired to use turing complete language (Starklark). I guess the Nix language is turing complete and therefore for backward compatibility a Turing complete language would be needed.
> Nix does not have a formal specification: Nix is what its C/C++ implementation says.
Yep, and this causes a whole lot of issues. In tandem with evaluator development we are writing an initial language specification which will represent the language as it is used in nixpkgs. Our hope is that we can, in collaboration with the Nix developers, eventually find a fixed point between the two implementations here.
> It still comes packaged with Nix 2.3.16 instead of Nix 2.4 because of incompatibility
Yep, we're very aware - in fact we had some debate on IRC yesterday about whether or not to touch on this in the blog post, but felt that for the majority of people it's a bit too deep into "current Nix politics" (for lack of a better term).
This is why we have explicitly committed to nixpkgs support - many of the experimental features in Nix (some of which are causes for the new incompatibilities) are not actually used in nixpkgs, and they might have a long road to go before getting there. We believe that this gives us enough time to get to a synchronisation point.
> However, it's possible that the pain around slow evaluation runs so deep that other companies might adopt this too. I don't know enough here.
We'll see, on the corporate side of TVL (https://tvl.su) we've had some discussions with a variety of Nix-using companies and there are strong signals that some of our plans would be a big reason for them to switch (or possibly even just partially switch). We want to not worry about this too much this early, to be honest.
> The rewrite needs to offer more reason to shift
Yes, there are more things that will become relevant but felt like they were outside of the scope of this initial announcement. For example:
* Implementation will be in memory-safe languages (evaluator in Rust; we will see about things like the sandbox & builder - Go might fit that reasonably well). This is a big point for companies that are worried about running current Nix (which is fairly easy to segfault) on production machines as root.
* We do intend to have more language-related tooling (including various kinds of static analysis, like dead-code analysis or maybe even going as far as an Erlang-style success typing approach)
* Streaming evaluation: We want to yield derivations while evaluation is still ongoing, essentially turning a whole build process into a sort of graph reduction. This has a huge impact on our ability to implement better Nix-native CI, which is very relevant for TVL's ambitions around better monorepo tooling.
It is also worth noting that a bunch of TVL members are nixpkgs committers, and as we proceed we would like to clean up bits of nixpkgs that use features which only exist "by accident". (An example of such a feature, which fortunately is not used in nixpkgs now, is __findFile: https://cl.tvl.fyi/c/depot/+/1325).
I appreciate that it's kinda difficult to serialise all of our thinking here in a handful of comments and a blog post, so we might not get the full point across. That's okay for now. If you're interested in following along more closely and discussing some bits of this, you're welcome to join our IRC channel :)
That is not a good argument. What matters to companies is: "is this stable enough not to be a concern", and according to GP Nix falls short on that front currently. They will not care how you achieve this --- C, a rigorous process and a world-class army of developers like the kernel is fine; Rust, a sloppier process, and less developers will be fine too. Heck, I'm not sure they'd care if you deliver stable and robust software using hand-written assembly, if not out of concerns for long-term support (and your mental health).
That's not what anyone is doing though. You can run Nix with minor tweaks to e.g. its garbage collector settings[0] and you'll quickly see it crash in various ways. Memory safety is a real concern in programs, not a buzzword.
- Nix itself has problems of compatibility with Nix! Let me explain...recently NixOS 21.11 released. It still comes packaged with Nix 2.3.16 instead of Nix 2.4 because of incompatibility issues. When the Nix project itself is not able to deliver backwards compatibility I feel uncertain that Tvix might be able to deliver their promise of full compatibility with all the deep architectural changes it plans. Nix does not have a formal specification: Nix is what its C/C++ implementation says. Nix also has deep architectural changes planned going ahead. So Tvix will need to re-architect Nix keeping in mind (and keeping up) with the future changes that are coming. It's going to be tough to build this "alternative" implementation. Users are going to run into weird compatibility issues, not have the patience to resolve them and then just go back to the original Nix implementation.
- A rewrite to a popular project will find it difficult to succeed. There is simply too much momentum in the parent Nix implementation (I'm not talking about the package definitions here) that very few people are likely to adopt this rewrite. This is because most people might not really need the speedup. However, it's possible that the pain around slow evaluation runs so deep that other companies might adopt this too. I don't know enough here.
- The rewrite needs to offer more reason to shift. Right now this rewrite seems to be mainly about performance. There is also a proposal to be able to integrate with GNU Guile. To me this is just avoiding one untyped language (Nix) to replace with another slightly better untyped language (Guile Scheme) (GNU Guix uses Guile also AFAIK). What about introducing typing? https://github.com/tweag/nickel is an interesting project and it would be great to have a system like Nix with optional typing to make writing the derivations more robust. What about rewriting in a safe programming language (e.g. Rust or a GC-ed language)? These things would be likely to excite would-be-contributors and would make your rewrite viable. In other words, the rewrite needs to offer more than just the promise of performance.
Perhaps I have not understood the project goals fully. Maybe the project is all about just being able to leverage the Nix package definitions but have a totally different implementation to instantiate them? But the issue here is that nix-the-language is very intimately tied up with the nix-the-platform and the rewritten implementation will still need to really be extremely compatible to capture all the implicit assumptions in nixpkgs.
Once again, improving nix through a rewrite is a noble goal. It's a tough problem and it would be awesome if the project succeeds!
I have obviously not thought about this as much as you guys and would love to learn how you wish to tackle some of these problems. Maybe some of these issues are not as big as them seem??