If you particularly want to replicate the common C situation where your dependencies are in-tree or you just don’t have external dependencies and write everything yourself then you are totally welcome to do that, even with Cargo. Nobody is forcing you to use external dependencies. If your problem is other people writing software with external dependencies, then I guess you have different priorities to them.

My problem is dependencies taking on dependencies taking on hundreds of dependencies, making it more difficult to find a self-contained dependency which solves a problem, compared to C++ (outside of Boost or ffmpeg and such).

That’s fair, and I hope we see crates being more considerate over what deps they pull in (or people making alternatives with minimal dependencies).

Is reproducability achiveable?

If you preserved an immutable tag of the source code and all its dependencies, a copy of the compiler version used and all build flags, then you’ve still got some big holes in your ability to reproduce a binary:

  1. OS version & patches installed
  2. OS configuration
  3. Hardware used (processors can have weird subtle bugs, microcode can affect execution behaviour, etc etc)
  4. Transient issues - the golden copy to be reproduced for some post event investigation might have contained a bit flip leading to impossible to reproduce verification signatures

Etc etc

Or is reproducability just a spectrum and you try to get further along it with some careful attention to detail, rather than an absolute to be achieved?

If the latter are you not cheaper just archiving binaries and tagging them with the source + deps + compiler + arch used to build them? Thats a 5 minute job to setup in your CI process and costs comparatively little to maintain vs wasting expensive human brains chasing down a futile goal.

It's normally taken as binary + transitive dependencies are bitwise identical on multiple builds. Needs a copy of a deterministic compiler along with your source. Whether the build runs the same on different machines is orthogonal to consistently rebuilding it.

Reproducibility is certainly achievable, and an important point for some people.

Look for example at guix, that bootstraps explicitly with what they call "the maxwell equations of software".

Imposing a web-based build system, into the kernel of all places, is a kick to the face to people who care about reproducibility.

This would definitely be done by "vendor"-ing all the dependency sources into the tree, no? Unless that's not the proposal, which I can't imagine it isn't, then I don't see how "web-based" is relevant.

Reproducibility is absolutely achievable. It is difficult with languages that have C system dependencies like C and Rust though. Usually you have to effectively check the compiler and system headers into your repo, which isn't ideal.

For something like Go it's trivial though. If you're using pure Go code then reproducible builds are pretty much as simple as "compile using Go 1.xx".

I don't understand why you think 3. and 4. are issues. Bit flips are very unlikely at compilation scale, and the hardware you use to compile something shouldn't affect the output. In either case you can just compile it twice and on different hardware and compare the result to confirm.

I hope that never ever happens. We don't need a C-like mess where every library has a bespoke build process that doesn't work across OSes.

OP seems to be conflating the concepts of cargo (the rust build system with dependency management functionality) and crates.io (the main and default package repository today).

It is totally possible to use cargo without ever touching crates.io, but if you need any dependencies you'll of course have to provide them through some other means (local file system, git repositories, or a custom package repository).

I could totally see the Linux Kernel setting up their own alternative Rust package mirror that cargo uses: you get the ease of use of cargo while also getting the desired level of curation. It could either be treated as owned packages, only projects written by kernel contributors, or it could be the linux distro packager model, where blessed crates at a point in time are curated, but not necessarily audited beyond some smoke tests.

Cargo does not care where you get your packages from, but it does have a default place. If you don't like it, it is easy to change.

That... is caring where I get packages from. Something like make or cmake doesn't care at all how packages end up on my drive. Almost everyone uses cargo in a way which makes cargo automatically download them from the web.

If you're making that argument, you could easily turn it around: make or cmake do care where your pacakges are; they by default only build things on your local drive.

The point is: all options are available for all systems, so suggesting any workflow doesn't work with any of these is just incorrect.

My point is that building code coming from my local drive is much less problematic than automatically downloading the code from some random website.

And I never said any workflow doesn't work with Cargo. I said I wished it didn't download code (by default) and that it didn't care how code ended up on my drive, that it just worked like a normal build system. That you can use Cargo in that way doesn't help.

> My point is that building code coming from my local drive is much less problematic than automatically downloading the code from some random website.

How does the code get on your hard drive? I'd imagine from downloading it from some random website (arguably _more_ random since it's less likely to be a centralized place like crates.io). Or you could vendor the dependencies so that they're included when you get the source code for the thing you're working on, but as mentioned throughout this discussion, cargo lets you do that too.

> where every library has a bespoke build process doesn't work across OSes

cargo sounds pretty much like a new bespoke build process that doesn't even work across different languages on the same OS

Yes. It turns out this works much better than the situation previously. The bespoke-per language model turns out to have fewer drawbacks than the bespoke-per-library-and-OS model. Note: fewer drawbacks, not none. I would not have predicted this a priori, for what it's worth, and I was very critical of and first one of these I became familiar with (rvm), but it turns out to work well in practice.

I still think cross-language efforts (like bazel and I'm sure there are others, maybe nix?) seem generally better, but I suspect there is some fairly good reason they are less widely used.

The problem with bazel and nix is that they are totalizing build systems. They want everything to bend to their worldview, and ask for an all-in commitment. Cargo etc require somewhat less of a commitment.

Right, but it seems like it would probably make more sense to pick a totalizing system, if you know you'll have a polyglot environment, which seems pretty inevitable for any business (as opposed to just like a side project). But I think there are probably good reasons this isn't the most common way to do things, and I just don't know them.

The reason is that defaults are powerful. Languages tend to have an owner in a position to declare a default for a language. No one is in a position to declare a default for polygot environments.

This also explains why the only places where these polyglot systems seem to be really common is at giant companies, where they have the power to enforce that stuff is interoperable. If your employer tells you to do extra work to make sure that your package can be used by others in the company, you're going to do it, but if you're working on some personal open source project and have the luxury of deciding what to prioritize, most people are generally not going to be spending time on figuring out polyglot build systems and will just use whatever's easiest, which will generally be whatever the language's default is.

That seems like a good way to think of it.

Many businesses do pick totalizing systems -- many medium-size or above corporations use bazel, and a few use nix. There's also some tooling to convert Cargo.toml to bazel and nix build files, though I don't know how well that works.

I think the problem is that bazel and nix are just not that easy to get started with. If you're using them you're likely to have a team (or at least one person) working full-time on them, since bending everything to a singular worldview involves a lot of work.

Good to know!

For other repro curious readers, this seems like a good entry point to follow reproducibility efforts: https://github.com/rust-secure-code/wg/issues/28

How does cargo prevent reproducible builds?

The default behavior of cargo is to download stuff from the internet. This may be the least reproducible thing ever.

I'm honestly astonished that programmers of a language that is deemed to be "safe by default" thought that this behavior was acceptable in any form, not to say the default. If downloading things at build time is somehow necessary, it should be an obscure option behind a flag with a scary name, like --extremely-unsafe-i-know-what-i-am-doing, that prompted the user with a small turing test every time that it is run. Cargo is just bonkers, it doesn't matter at all if it is "convenient" or not. Convenience before basic safety and reproducibility is contrary to the spirit of the language itself.

It's as if bounds checking in the language was deferred to a third party that you need to "trust" in order to believe that you won't have segmentation faults.

It doesn't just download random things. Cargo generates a Cargo.lock file with checksums and will make sure that those checksums match when building later on. It's about as safe as vendoring all dependencies while being far easier to work with (though tools like cargo-vendor do exist, of course).

Edit: for things like the kernel, vendoring dependencies is still probably not a bad idea, of course

What prevents a given URL from disappearing? Does that just break a particular source version of the Linux kernel?

What happens when a given dependency adds new kernel-inappropriate features? Are kernel devs going to act like distro maintainers and decide between forking, maintaining patch sets, etc.?

All crate sources are stored in the crates.io package archive, which never deletes packages.

A dependency veering off in a direction you don't like is one of the risks of using someone else's code instead of writing it yourself. Cargo makes it easy to use forked dependencies, and forking a dependency is almost always less work than if you'd never used it and written the code yourself from the beginning. (And to be clear this is only a problem for future evolution; a crate author cannot remove or modify an already-published version of their crate.)

This is still fairly short sighted. Websites shut down, large websites with big storage demands are especially vulnerable to attrition. Who wants to pay the mounting bill for keeping decades of revisions of historical rust packages online?

I can grab the kernel sources from 1997 and build them today. Will I be able to build rust code from 2022 in 2047, because the 1997 kernel will still build at that date.

"I can grab the kernel sources from 1997 and build them today."

Where would you be grabbing it from? ...From a website? "Websites shut down, large websites with big storage demands are especially vulnerable to attrition. Who wants to pay the mounting bill for keeping decades of revisions of historical Linux kernels online?"

You can easily do this with Rust and Cargo as well.

There is a fairly large difference between archiving your own project's history for as long as you feel like, and archiving the complete history of every significant piece of code ever written in a particular programming language forever.

Who claims that archiving the complete history of every significant piece of code ever written in Rust is necessary? It is easy to archive only the code that your project depends upon. Rust code is no different from C code in this regard.

A couple of things addressing points from different part in this thread:

- Archiving the complete history of all crates in crates.io is perfectly feasible today for an individual. Over time that might change.

- Setting up a mirror is straightforward, should you want to do so: https://github.com/rust-lang/crates.io/blob/master/docs/MIRR...

- crates.io is financed by the Rust Foundation and is at no risk of disappearing, it is a very well funded effort.

- Using cargo with an alternative repo is not difficult, requires some one-time configuration.

- Vendoring your dependencies is supported.

- cargo hits the network to look for semver compatible updated versions of your dependencies on specific moments if you don't have a Cargo.lock file.

- Not updating your dependencies stops you from getting the rug pulled from under you if an unwanted change happens, but it also stops you from getting any desired changes including security vulnerability fixes.

- Even if you vendor all of your dependencies, you still have to audit them the first time and every time you update them. Are you? Most aren't. Code you haven't written yourself can't be assured not to be malicious, and code you've written yourself can still have exploitable mistakes.

So your argument is you think the people who run the crates site don't want to do a good job but the people running kernel.org do? What info are you basing this random-seeming decision on? Do you have any actual data suggesting that the crates site will just disappear like you say?

I'd like to see that data if so -- I have pretty big doubts that your statement has merit without some sort of evidence.

As I said in a parallel comment, there is a fairly large difference between archiving your own project's history for as long as you feel like, and archiving the complete history of every significant piece of code ever written in a particular programming language forever.

Kernel.org's repository is also of major versions, not every minor release and patch. That really wouldn't do for cargo. If it has ever been released, it needs to be kept in storage for as long as the rust ecosystem exists. That's decades, maybe even centuries of passing on the torch and hoping the next guy accepts the responsibility. Hoping you can find a next guy.

So vendor in the dependencies. it's a matter of cloning the dependencies repo and adding a handful of characters to your project's cargo file.

Now the lifetime of the dependency is that of your project. There's even tooling 'cargo-vendor' to help manage this setup.

Alternative of course is implementing it all yourself, which cargo doesn't prevent.

> I can grab the kernel sources from 1997 and build them today.

Can you? Do they still compiler with current compiler? You'll probably need to find a compiler of that time... And also all the interpreter for all the build scripts. Was that using bash or some old Perl? Maybe something more esoteric like m4 or tcl?

The point is that it always had many external dependencies to bootstrap. And adding one is not such a big deal, it just add another thing to archive among the many other things. The crates.io archive is probably not even that big.

I'm not sure why that would be a problem given most of these languages and standards are older than the Linux kernel. The thing about mature technology is specifically that it doesn't have breaking changes every couple of months. This is the way it used to be for a fairly long time.

But even if it has broken, I can just download an old linux distro. They effectively form a cohesive snapshot of the state of the toolchain whenever they were assembled. Slackware 3.1 from 1996 might be appropriate.

> But even if it has broken, I can just download an old linux distro. They effectively form a cohesive snapshot of the state of the toolchain whenever they were assembled. Slackware 3.1 from 1996 might be appropriate.

You will also need era-appropriate hardware to get that software to install.

I'd rather comment than downvote. Who cares about about a kernel build from 1997 (25 years ago)? What was the hardware back then, Pentium 2? Sorry for the snark in advance but: Why make mountains out of molehills? Life is hard enough as it is.

I would imagine the kernel would use cargo vendor, or similar, to lock all dependencies into their chosen source control and quality requirements.

https://doc.rust-lang.org/cargo/commands/cargo-vendor.html

"Never" is a long time, just saying. It'll be impossible to beat the "availability" guarantees of a local mirror (like a thumb drive) of a kernel source tarball.

What happens when a crate version has to be removed due to a critical CVE or court order (IP Law violation, perhaps)? There may come a day where crates.io becomes torn between not breaking Linux source and not hosting actively bad source code.

Note that some of those concerns do apply to vendoring source as well, but the additional download step also removes options that the kernel maintainers have as long as they ship all the source for the kernel in one tarball. Like more control over the timing of inevitable decisions.

> What happens when a crate version has to be removed due to a critical CVE or court order (IP Law violation, perhaps)?

CVE = The Yank flag. Cargo will refuse to add new yanked packages to a lock file, but if a yanked package is already in the lock file, it will still build. The package is not actually deleted. https://doc.rust-lang.org/cargo/commands/cargo-yank.html

Legal = Hard delete. Nobody will go to jail just to avoid breaking your build. Of course, since crates.io and kernel.org are in the same legal jurisdiction, is there any actual difference here?

What happens today when a kernel module has to be removed due to a critical CVE or court order?

That's not just a rhetorical flourish, I'm actually curious what the answer is. As far as I know, (1) it almost never happens and (2) when it does, the change is made in upstream repos and as a practical matter, everyone downloads those changes and their up-to-date local copies lose that code.

Fixing it in the future isn't the point. Breaking previous releases is.

The previous tarballs still work and contain the relevant code. Your build wouldn't rely on hosts complying with court orders in countries you might not live in.

If the code isn't vendored, just referenced with URLs, the old tarballs stop working.

This hypothetical court-order situation is quite far-fetched. If crates.io was ordered to take down some or all versions of a package, an alternative mirror could easily be created elsewhere and you could configure cargo to use it.

But I think the kernel would vendor crate dependencies, partly so that people can build without accessing the network, simply because that's policy in many places.

Does crates.io actively host any code? I thought it was all just readmes and links to github and docs.rs

They do host it. It's registry info is mirrored on github though

To the first question, obviously the sources of dependencies would be brought into the tree. This is easy and there's no reason I'm aware of not to do it for something like the Linux kernel.

To the second set of questions, how is this any different than any other dependency the kernel has? If the answer is "the kernel has no dependencies" then yeah, I'm very sympathetic to the argument that bringing in rust libraries is not a good reason to start having dependencies when none previously existed at all, but is that the case?

You're forgetting about custom build scripts. Thankfully most of the core ones have moved off cloning dependencies for ffi purposes (think cloning an alsa-lib version for ffi), but it used to be super common.

No, it is. Even without `--locked`, the Cargo.lock file is only updated when it no longer fulfills the Cargo.toml because the latter was edited (and then only making the minimal changes necessary), or explicitly using `cargo update`.

I don’t follow - I’m saying the cargo.lock isn’t read unless you specify —locked - I’m not talking about when it gets refreshed?

Yes, it's always read. If the file didn't require updating, a build with and without `--locked` will be identical. If it did require updating, `--locked` will make cargo exit with an error.

Gotcha, makes sense

That's true when running `cargo install` to install an application directly from crates.io, but not when running `cargo build` in an already checked-out repository.

I might be misreading this on an iphone screen but as i follow the logic here: https://github.com/rust-lang/cargo/blob/a77ed9ba87bfeaf3c275...

A cargo build ends up there calling into the resolver’s resolve_ws_with_opts() which would refresh the lockfile.

Not resolve_with_previous() which would use the lock file as-is.

The only reason this sticks in my mind is i ran into an issue building bat after i made some changes, i obviously assumed it was my changes so went through the process of debugging and backing out my changes until finally i was back to a virgin branch and still failing - passing —frozen —locked fixed it.

> It doesn't just download random things.

That's exactly what it does. The developer is not really expected to thoroughly review the codebase of every dependency.

Just like javascript, all sort of supply chain attacks are made possible.

A single malicious library can sneak into large ecosystems easily.

If your project has a Cargo.lock file checked into its repo, then everyone checking that out will download the same code for all dependencies (unless someone manages to compromise the crates.io package archive). That is very far from "the least reproducible thing ever".

Cargo.lock also contains crate hashes. So, if someone compromises crates.io and tampers with a crate, you would notice.

> The default behavior of cargo is to download stuff from the internet. This may be the least reproducible thing ever.

Wait till you find out about java ecosystems

I know investment bank dev teams pulling whatever they need from maven central with no oversight or introspection.

Log4j joins the conversation.

yea i'm glad i wasn't around when that was discovered.

> The default behavior of cargo is to download stuff from the internet.

This is borderline inevitable for most modern development stacks, though .lock files can definitely help, even adding hashes to check against if you care about your dependencies being the same as when you first download/add them to the project and/or inspect the code.

As for worries about the things in those URLs disappearing, in most cases you should be using a proxy repository of some sort, which i've seen leveraged often in enterprise environments - something like JFrog Artifactory or Sonatype Nexus, with repositories either globally, or on a per-project basis.

The problem here is that all of these repositories kind of suck and that the ecosystem around them also does:

  - for example, Nexus routinely fails to remove all of the proxied container images and their blobs that are older than a certain date, bloating disk space usage
  - when proxying npm, Nexus needs additional reverse proxy configuration, since URL encoded slashes aren't typically allowed
  - many popular formats, like Composer (or plenty more niche ones) are only community supported https://help.sonatype.com/repomanager3/nexus-repository-administration/formats (nobody will ever cover *all* of the formats you need, unless you limit yourself to very popular stacks)
  - many of the tech stacks that have .lock files may also include URLs to the registry/repository from which they're acquired, so some patching might be necessary
  - in technologies like Ruby, actually setting up the proxy isn't as easy as running something like "bundle install --registry=..." as it is in npm
  - in other technologies, like Java, you get into the whole SNAPSHOT vs RELEASE issue and even setting up publishing your own packages to something like Nexus can be a bit of work; the lack of proper code libraries for reuse and abundance of code being copy-pasted that i've been being a proof of this in my mind

Of course, i'm mentioning various tech stacks here and i don't doubt that in the long term Rust and other technologies might also address their own individual shortcomings, but my point is that dependency management is just a hard problem in general.

So, for most people the approach that they'll take is to just install stuff from the Internet that other people trust and just hope that the toolchain works as expected, a black box of sorts. I've seen plenty of people just adding packages without auditing 100% of the source code which seems like the inevitable reality when you're just trying to build some software with time/resource constraints.

I'd really like to know where you think C++ dependencies and headers come from.

Downloading C++ dependencies during the build process is equally unacceptable for many situations. Existing C++ build systems and package managers can be configured to do that and those build systems and package managers would be inappropriate for supporting a kernel that values stability and long term support.

So it's a good thing that cargo can be used without downloading dependencies during the build! Just clone the repos of the dependencies (and transitive dependencies), just like you would for a C++ project. Then set up your cargo file to point at the location for your local copy instead of using the default download behavior.

There's even a tool called cargo-vendor that does this for you!

I was just saying anything folks might find objectionable about cargo workflows happens in C and C++ workflows, just in ad hoc ways.

The difference is that no C or C++ package management features are proposed for incorporation in the Linux kernel SDLC.

I wholeheartedly agree. I love Rust, it's the most fun I've had with any programming language (barring perhaps Haskell). But I still run cargo in offline mode with crates.io disabled, pointing cargo to /usr/share/cargo/registry for dependencies (that's where Debian's librust-*-dev packages get installed, the only dependencies I accept in my Rust projects).

So you can change 2 settings to get the behavior you want?

Sounds like they did a decent job anticipating that use case!

To say it explicitly: "I don't have any internet" was a hard constraint on everything from the beginning. Firefox's build requires it. Most distros require it.

Some of it was a bit awkward to actually use this way in the early days, but those harsh edges have been since sanded off.

Absolutely (although see note below)! I don't have any gripes with the tooling from a technical perspective. I do have gripes from a cultural perspective, though.

Addendum: Until quite recently, this was quite cumbersome. It also meant that all cargo invocations (by the same user) would use that override, always. It meant that compiling someone else's project became quite the hassle. Or compiling a project that mostly uses system dependencies, but some crates.io deps. But the situation is improving.

Let's just make "unsafe" the default in Rust, and add a new "safe" keyword to anticipate the use case of overly concerned users like gspr.

Cargo and crates.io have been specifically designed to be reliably reproducible, and had a chance to learn from npm's mistakes to not repeat them.

Cargo binary projects have Cargo.lock by default with checksums of all dependencies used. crates.io doesn't allow changing past releases, and has a policy of not deleting any crates unless legally required (user-accesible "yank" hides, but doesn't delete). Crates.io index is a git repository with full history of all changes to the registry, so you can recreate its state at any point in time (in case you lost your Cargo.lock, you can reliably remake one from the past).

And on top of that there's `cargo vendor` command that makes a local offline copy of everything you use, so you can fully archive a Cargo project and rebuild it any time later.

why do we have to have soooo many package managers? makes me cry inside a bit.

Because not everyone is using Linux distributions, and even those cannot agree in which system packaging they want to have.

Yeah, especially language specific ones. It seems like the world wants to be polyglot. Do we really want to have all the encryption algorithms reimplemented in all the languages?

The answer there is often "use FFI". But if we're all going to use C APIs anyway, then shouldn't our package managers support C?

But C has probably the most awkward build system culture of any language, complicating the job of packaging quite a bit. A lot of language specific package management systems get lift from offloading the ugly C support problems to other layers. See python and "manylinux" for instance.

you are right about C. i mean i do get why we have so many. and i use them so often that its ingrained in me. and how to fix them when i mess up. but sometimes i notice how many libs we are talking about. honestly it is kinda ridiculous. i know how hard it is to manage one repo and keep it safe. maybe im too old. young ones now just download go binaries and run them.