If you don't update your dependencies all the time, you will be vulnerable to old bugs and issues.

The current software engineering paradigm has no meaningful answer to this, no matter what "security experts" tell you. In a sane industry this realization would lead to a change of the paradigm, but people in our industry seem to be only doubling down.

"Use a build tool to do dependency checks!"

And then what?

"Only update libraries if they have vulnerabilities."

So, should this be a manual process where I have to dig through obscure warnings every time I build something?

"No, you can automate it!"

Congrats, you've just outsourced the decision making process about which versions of libraries to use to some 3d party. You've also made your build process reliant on yet another online service.

"But it's not a hard dependency, you can still build if the 3d party service is off."

So the exact software you compile will depend on whether or not you can connect to a 3d party service? Do you understand the actual implications of this?

Etc. People with clever advice don't seem to think it through. We're fighting fragile complexity of too many tools ducktaped together by ducktaping more tools to the whole setup. Again and again and again.

On the contrary, I believe we do. The answer is ongoing maintenance.

The basic problem is that people persist in looking at software as a fundamentally mechanistic artifact you finish and ship in a static environment. Then you move on to the next thing. This is fundamentally incorrect. Software engineering is a process that takes place in an adversarial, human-driven environment. There is fundamentally no automating this away because human decision-making work is required.

Any security expert worth their salt will tell you this. As long as you persist in trying to view software as a fixed artifact in a static environment, you will find there is no meaningful answer. Those who come to terms with the dynamic, human-driven reality will find that there's a well-understood - if inconvenient and expensive - answer.

That’s just one example. Medical machines are expensive as hell and companies don’t want to rebuy or constantly invest in something that works and does one thing well.

Software is never just an artifact. It's a whole system, of which the source code and binaries are some components. If you do not manage those artfacts, the next best option is generally managing the environment.

I probably wouldn't bother to try to port it to modern versions of Qt or whatnot even if I hadn't lost the source code 15y ago.

So yes, it happens, it's possible, it sucks

To give you an example: XLSX parsing. For quite a while, the most performant library ‚xlrd‘ had a big scary unmaintained notice on GitHub, yet people kept recommending and using it, often unnoticed through pandas, because they don’t read docs. I‘d refuse to touch that and instead use openpyxl, and now newer xlrd releases have even completely removed xlsx support.

When you choose software to solve a problem, you're not just choosing software. You're encoding expectations around the problem, the environment, and your needs. If all three of those can be assured to never change then choosing a fixed artifact might be the way forward. Should any of those ever change, then your software will either adapt or become a less suitable solution.

How often can an organization really be that certain?

If you need to make guarantees about your product that are dependent on another piece of software, then you just have your vendor produce a spec and guarantee conformance to the spec, then you audit that the spec fulfills your fixed needs and audit that the implementation conforms to the spec.

Then, if you made a mistake, your liability is limited to your mistakes instead of your mistakes and the mistakes of your dependencies.

If, on the other hand, you do not get any guarantees from your dependencies, and you fail to achieve your guarantees then that is entirely on you as you are the one transforming the absence of guarantees into guarantees.

The issue: ML workloads are very sensitive to the exact version of a framework/library used and these frameworks/libraries are not stable despite what their semantic version says.

So what do you do when you have a python vulnerability? Best case, an update "just works" but in many cases you either need to retrain the model (can be extremely expensive or inconvenient), rewrite it if APIs changed, or hunt down a difference in model output if some downstream dependency broke something. Stability of software in the python world is abysmal.

With a service it's all much easier because you can have pretty robust integration tests and you're good to go. With an ML model it's a lot trickier because even if you have sample payloads and responses there might be other payloads you didn't think about that could cause issues. Or it might rear its ugly head when you finally retrain the model and suddenly model performance is crap.

One recent example that comes to mind is that PyTorch now disables Ampere GPUs' TF32 units by default since there were some hard to find edge cases where the matrix multiply results were completely wrong compared to the expected result. This went mostly unnoticed except in a few cases where users were trying something a bit more sensitive to such issues.

I had a similar issue in one of my own models where there was an error in a normalization layer's implementation in Tensorflow (it was computing the average over the wrong axis iirc). It only became noticeable to me when I came back to retrain the model a few months later with an updated Tensorflow version and found that the results from training were not even comparable.

In every other ecosystem when minor version are updated there are very rarely issues, and there are little issues with compatibility with minor versions of libraries.

In part due to a culture that encouraged monkeypatching. With the result that what behavior you get depends on who last monkeypatched it. So a dependency 3 deep that loads the module you expected to load last suddenly causes the monkeypatch you were depending on to disappear...

If you are willing to run other people's arbitrary (Python) code, then - vulnerability or no, you should consider the system running this code as compromised.

This policy evolved, if memory serves, after a security issue was discovered with the XML library we were using. The fix was not back ported to our version, and the versions in between had multiple breaking changes, so it was a slog to fix it.

It wasn't even that we were having a production issue, because we were still in development and internal testing. Not a single 'real' signature had been generated yet. It was all test assets and test CA certs. But we had enough wisdom to put 2 and 2 together and get 4, so we tried to treat the situation as a dress rehearsal for some later bug that happened after we started playing for keeps. Almost everyone could see this was an untenable situation.

There were consequences of course, but we made a bit of lemonade in the process. Our integration tests were expanded to include a larger percentage of pinning tests for our dependencies. Given the gravity of the situation, we needed those anyway. We just now had a poster child for doing the work.

how did that go ? I often wanted to do it but never got to submit the idea.

It also took some reminders from the leads to add it in instead of letting it slip, but so far it's the least broken process I've gotten to use.

Sorry, this is a total strawman. Automation doesn't need to be integrated with the build in such a way that it creates noise, and it doesn't mean you need to let changes be applied without human intervention.

I'm hooked up with Snyk which scans all of my dependencies and sends me a summary email if it discovers any vulnerabilities. It creates PRs that can patch dependency vulnerabilities, and I can apply them if I approve of the change. If I don't think the change is worth it or the vulnerability is relevant, I can go to the Snyk site and mark a vulnerability to be ignored, or even tell it to ignore it for a month or whatever. I'm sure Snyk isn't unique, either.

Yes, a human should be involved in decisions. No, it doesn't need to be noisy.

I'm on Node.js and working on multiple projects with crazy numbers of dependencies, and the number of "new vulnerabilities" I see is typically a few per month at most. Some months there are no new notifications at all. It simply doesn't change that quickly, and it's not constant noise unless you ignore the notifications, and that's just bad software engineering.

I am currently fighting with our security team because they refuse to allow developers to do this, and instead require my team to file tickets with their team, wait hours for that ticket to be picked up, explain why a thing is a false positive to a person who doesn't understand the codebase they're working on, and hope that person feels that the finding should be ignored.

I do not work at a large company.

This is greatly exacerbated by the fact that Snyk - like all package scanners - primarily flags false positives. When you start digging in to what it flags you'll find that a lot of the vulnerabilities aren't reasonably exploitable, or the impact is questionable, or you can't repro the issue. Those that do legitimately affect the underlying library often don't affect your usage of that library. I'd estimate our false positive rate is >90%. So when we get a finding our options are to go through that costly and disruptive process I mentioned above, or just upgrade the dependency (which carries associated risk of breaking shit in production).

These tools emphasize the quantity of findings they produce, and quietly ignore the quality. As soon as you have any divergence in motivations or incentives between the people running the scanner and the people being blocked by the scanner those low quality findings become incredibly costly.

I'm sorry you're on the receiving end of this problem!

Shill notice: I'm working on an Open Source tool[0] that makes this problem less horrible. My colleague wrote a post about our hypothesis[1] about how we can avoid this false positive trap.

I'd love to chat with anybody feeling this pain (even just as therapy lol).

0: https://github.com/lunasec-io/lunasec

1: https://www.lunasec.io/docs/blog/the-issue-with-vuln-scanner...

The Snyk tools apply to the patch directly to (e.g.) `node_modules` after the latest code has been downloaded.

At this point, I think a lot of problems in software are actually misaligned or misincentivized policy gone awry. When you read about them on forums they look technical, but they're not.

And then there is the "I want the fix, but I don't want the added attack vector of features that comes with it."

Honestly it's just not worth worrying about whether a security hole is exploitable when the decision is whether to accept a minor update patch. Just accept the patch PR and move on.

The times you need to do a major update to get a patch fix it's equally important to just make that upgrade, since that means the ancient version you're using is likely out of maintenance and likely is throwing warnings that you're using a deprecated version.

Yes, it's more work to perform a major upgrade. But that doesn't mean it isn't good engineering to keep your tools up to date.

That is, I think the ask is more of "why haven't we come up with something that is a bit better at mitigating risks, here?"

I believe this is one of them.

To add further insult to injury, colleges send SWE/SDEs to the work force to learn to 'actually code', while companies have no time to train jr developers so they are just thrown in the fire and hope for the best.

1. keeping it running with a healthy userbase and demonstrating high impact does, and

2. if you know how to spin it, then maintenance work can look like "building new stuff".

You've worked for some great orgs, then! My experience is mostly the opposite. One company I worked for did value mentorship, at least somewhat, and it was a factor in promotion decisions, but not mentoring others didn't really stop people from getting promoted.

1) We use very few dependencies. We err on the side of writing our own function, library, or UI component rather than installing a new dependency, even if a "better" open-source version might theoretically be available.

2) We use a monorepo. We have two Go backends sharing the same go.mod, and three Typescript/React frontends sharing the same package.json, all in the same repo. An upgrade for one is an upgrade for all.

3) Every first Friday, I update all dependencies to the latest version. Including major versions. Yes this is a pain sometimes (like when I realized we were on Webpack 4 and the latest was Webpack 5), but if you do it consistently, it's usually not that much trouble. 1-2 hours per month on average.

In terms of security, I think this is a good middle ground. The approach has other benefits than security, too.

This just means you likely have a lower quality version with the same problems but without the benefit of an ecosystem to find them.

We won't write it ourselves if we aren't sure we can do it well, i.e. we're not going to roll our own encryption. Those are the types of dependencies we do use, and upgrade once a month like clockwork.

The point is not "don't use someone else's encryption library," the point is "don't use someone else's LeftPad() function that takes 3 lines to write". That gives us fewer dependencies, which makes it easier to upgrade all dependencies regularly, which is good for security.

While this may solve the "exploit from 3rd party dependencies" this says nothing about the security quality of your functions. Now instead of the vuln being found in the 3rd party package and fixed, the issue remains in your program forever, probably getting exploited by nation state level actors without your knowledge.

Your boring business systems that will live for 50 years need to have boring system level libraries or stuff you maintain. Your fast time to market or non-critical systems are optimized by speed and cost.

If your service depends on a bunch of downstream stuff, your processes need to support upgrade cycles. If you can’t keep up with that, you need to refactor.

Some of this is trivial, and a very good thing - stop using bullshit exercises in fragmentation like isEven.

Some of it is very much not, and will apply pressure to find vendors who are willing to vet their own bundles of commonly used libs.

There will be numerous second order effects of that, including the ghettoization of open source that isn't "QualStrike Approved(tm)".

Capability-based security is one possible answer. The Austral language is trying to make this a first-class language feature:

>The problem is that code is overwhelmingly permissionless. Or, rather: all code has uniform root permissions. The size of today’s software ecosystems has introduced a new category of security vulnerability: the supply chain attack. An attacker adds malware to an innocent library used transitively by millions. It is downloaded and run, with the user’s permissions, on the computers of hundreds of thousands of programmers, and afterwards, on application servers.

>The solution is capability-based security. Code should be permissioned. To access the console, or the filesystem, or the network, libraries should require the capability to do so. Then it is evident, from function signatures, what each library is able to do, and what level of auditing is required.

https://austral.github.io/spec/rationale-capabilities

1. Depending upon context, *any* change to state (network I/O, console I/O, file I/O, environment variables, etc.) is a security vulnerability you can drive a truck through. For example: Foo is a very-locked down package that only appends text to a file the caller owns. It's formally verified to only change the file specified, verified to only append in all possible situtations, verified to not have cross-file-system identity problems, not subject to file renaming race conditions, yada yada yada. Safe? No. "foo('evil_command' '~/.bash.rc')".

2. A useful definition of a "capability" depends upon context. A *dynamic* context. Say Foo is intended to be used for generating log files. There's no way for a third-party library to determine whether the target is actually such a thing. It requires manual tuning which fits the operational context. Perhaps the file must be named "/var/log/${x}.log". Or maybe its "${home}/local/log/${x}.log". Great. Of course, now you've got to verify the context-specific defintions. And verify the tool that does this verification. And the verify the configuration of the tool that does the verification... Safe now? No. Context is *dynamic*. "foo('var/log/evil-hack-attempts.log' '0.0.0.0/0 tried-to-crash-us')", and your other context, the automated blacklisting, locks out the Internet.

3. This gets very complex very quickly. IMHO, configuring Selinux, in a real production environment with real personell, is akin to rolling your own cryptography. Can people learn to do it? Of course. But the same argument applies to writing cryptographic functions. You. Will. Make. Huge. Mistakes.

To be clear, this does not mean that capabilities are useless. It's obviously critical that they exist. However, they ain't magic. You can't "solve security with capabilities." There is a point where throwing more of them at the problem makes things *worse*.

for example we didnt fix how we build applocations and manage dependancies, we invented docker.

we didn't create a secure runtime for the cloud where applications can run, instead we virtualised whole operating systems

"we do not break userland, period" -- Linus Torvalds

not breaking existing stuff is why many abstractions exist, and even why many are put in place early on, to allow for some wiggle room without having to break all the things. people tend to prefer software that continues to work, above all else.

Abstractions that paper over problems that you are unwilling or undisciplined enough to tackle properly are usually bad. I think the grandparent was talking about this kind.

some abstractions carry decades of legacy (e.g. linux file-system hierarchy), and cannot simply be replaced. things that did not connect to the non-existent internet did not have to paper over problems that simply could not exist at that time (VMs, multi-user, multi-tenant hardware). there is plenty of deep software dependencies that are no longer maintained which still rely on these abstractions and cannot be cheaply/reasonably replaced in the field. sure, we have new abstractions since then, but only new and actively maintained software can ever make use of them, and that will only end up in systems that are also actively maintained instead of simply continuing to work with exactly zero effort.

The whole log4j thing happened is because java is very poorly designed from the get go, and one of those points is the insistence of making everything an object, including data. And of course, you often need to output the string representation of data.

From a theory perspective, if you want to rely on language features like strong typing to ensure correctness, then you have to take it all the way to full provability, where you explicitly guarantee bounded input, bounded output, and any side effect (where a network call is a side effect, and if your code doesn't explicitly state that this side effect will happen, it won't run).

Not sure if it's the best, but my current approach is to target major revisions/versions with ABI/API compatibility as viable candidates, and update always, but at a lagged rate, ideally allowing enough time to pass for vulnerabilities to be surfaced in those versions. Log4j is a tough case due to how far back it goes version-wise, but you simply can't win them all.

Another important way to minimize risk is to have a strong ability to provide a corrective action. If replacing the version or the entire dependency of a library you use takes a significant engineering effort, then you are arguably at a higher risk due to that than you are in your decision to never vs. always update.

The only complete solution would be static analysis (read: type systems) that can guarantee everything about some code's contract/behavior relative to the caller. Short of that, it's just going to continue being a bunch of manual half-solutions like API checks, existing type-level checks, manual changelog reading, manual upgrades, testing, etc.

That's my armchair take on it.

Aren't we already doing that, to some degree, by watching for notifications of vulnerabilities, and then manually patching or updating? Sure, with this method, we have the ability to decide to ignore a vulnerability, but I'd guess most people aren't great at assessing risk (and may not fully understand the severity of a vuln, or if their use of the software makes them exploitable), and should probably just update whenever a fix for a vulnerability comes out. So you might as well automate this.

> So the exact software you compile will depend on whether or not you can connect to a 3d party service?

Anyone who builds against public package registries (Maven Central, npm, pypi, crates.io, etc.) already has this problem. Some people will go to the effort of putting their own proxy cache in front of these registries to insulate themselves from downtime, and I expect these sorts of people would do the same for a 3rd-party vulnerability updater service.

Regardless, I wouldn't expect this to be a "pull" model, wherein you wait until the next time you'd build before getting security updates. More likely you would get a notification or even an automatically generated pull request (like GitHub's dependabot does) when a security update is available. Then it's up to you whether you want something else to automatically apply those updates, or you can review them yourself and apply manually.

I think you're just making a mountain out of a molehill. There are several existing options to automate or partially automate this, depending on your trust of the relevant third party. And you can still do it manually, if you so desire.

You could formally spec software and implement it, congrats you're now 1000x slower to market and any change you want to make is another 1000x investment to spec out and prove.

And let's not even mention the ridiculous idea that open source developers be expected to do this. Moreover, this doesn't even handle hardware problems!

Okay, but we could just build all the software internally, reinvent all the wheels. But how is this any better? Now you have even less manpower to fix bugs and you're probably 100x slower to market because you're building all these half baked, bug ridden libraries.

If anything, the best idea is formally verified sandboxing, so that you have strong assurances about certain apps not doing bad stuff, but this doesn't solve all problems either.

It's an unsolvable problem in general, which is why no one has solved it.

I think until Software Engineering catches up in this regard with traditional Engineering disciplines we'll continue to see issues arise from the software realm. As much as the self-titled "Engineers" proclaim it, there's more to engineering than design and production.

I've been asked why I got my P.Eng as it does nothing for me career wise, but I always have the hope that we move towards a profession that does utilize engineers with proper accreditation and responsibilities where appropriate. I work in Industrial Controls and what my job has me doing does have safety concerns. We fired a student at one point working for us because his attitude towards the work was lackluster and lazy. I see it often enough in programs I'm asked to pick up and fix too.

If Software Engineering wants respect, the profession needs to accept that we're going to piss off a lot of Developers & Architects using the title incorrectly. We still need to develop the proper rigor to take Engineering as a discipline seriously, but we also don't need to apply "Engineers" to every job.

You’ve brought up three relevant points: 1. The tools are software, which strength and weakness is being easy to change. 2. The tools rely on abstraction, which strength and weakness is summarizing/hiding complexity. 3. Communication between publishers/consumers of these tools is hard.

Armchair-coaching I now come up with at least three solutions: 1. Re the weakness of #3: ensure when an issue is found at one layer, communication happens to maintainers of all other layers up and down. 2. Re the weakness of #2: collapse the layers – analyze dependencies and don’t go past more than x number of abstractions. 3. Re the weakness of #1: don’t use software.

If you using a library you have to spend time on keeping it up to date (which includes testing and fixing whatever when wrong during an update).

Or even worse you may have a legacy "fat jar" that you can't even manage the transitive dependencies on - they're compiled in, and that's that. Something else has a version conflict? Tough shit, one or the other is going to break, and depending on how bad your build is, you may not even get an explicit choice, it may be thrust upon you by the classloader.

Sometimes you can mitigate it somewhat by encapsulating them in their own service and just firewalling the hell out of it, and modules should help the dependency conflicts somewhat, but the problem is these are basically "UXO in the backyard garden", they are a passive business risk and sooner or later you may get lucky and they go boom even though you did nothing wrong.

Despite your best intentions, sometimes you gotta update dependencies, and if you allow it to fester, then you risk that the time you have to solve the problem is during an urgent security crisis. Generally it is better to update them on your own timetable than to have it forced upon you like that. And the problems and limitations and workarounds only become worse and worse over time if you choose to actively ignore it.

It'd be cool if devs didn't suck ass at writing code but that's the world we live in.

If you don't update your dependencies all the time, you will be vulnerable to old bugs and issues."

>has no meaningful answer to this...a sane industry this realization would lead to a change of the paradigm

This sounds like gene mutation and evolutionary selection pressure. It's sort of a fundamental aspect of the universe no? There may not really be a meaningful "answer" to this in principle just various half-measures to muddle along and adapt to this reality, survival of the fittest in a sense.

Why the scare quotes?

>In a sane industry this realization would lead to a change of the paradigm, but people in our industry seem to be only doubling down.

Maybe because half the time people with extensive training/experience in security start to even say something they are completely disregarded (like here), or are hamstrung by budget because security is viewed as a garbage disposal that profits get shoveled into, or because they are hamstrung by pushback over every little change because people seem to think the main goal of someone doing security is pissing off users.

I'm not making fun of you or trying to be funny - it is simply an observation that I find interesting.

Also nobody that knows what they're doing uses "duct tape" on ducts, so the name is pretty meaningless.

This is already true with the vast majority of software using any an online software repository, e.g. Go, NodeJS, Java, Rust, etc.

With security, you need some online service so you can find the new CVEs every day.

Or you just do it manually if that service is down. It's weird to me that the argument for not having an automated, 3rd-party service is "if it goes down then you'll have to do things manually", when the alternative is "you always have to do it manually".

If you are comfortable trusting a third-party service to tell you when to upgrade, then that is absolutely an improvement over doing security updates manually. This is why I have unattended-upgrades set up on my Debian systems to automatically install updates from Debian Security every day. Sure, it may fail for whatever reason, but I am certainly not going to take the time to (or even remember to) update every day.

But basically if you just subscribe to a few projects' releases you can pretty easily get things pushed to you when it matters.

Also known as doing your job.

I am no expert. I ended up indexing all the open source kruft I use to hold this ship of fools together, then verified that the Log4J pieces were definitely disabled with a bunch of monitoring while I tossed stuff at it. I did mention I am not an expert, right? I am sure there is a more Pro way to do this.

solution: stop eating

Option 2 - which you are advocating - it's every man for himself and we decent into socity of warring tribes of hunter hatherer or subsitence farming at best

let the buyer be aware

But, you're not wrong, some people reach for every library and framework they can, and others focus on building the most minimal and understandable thing that will work. I'm not saying writing your own server in assembly, but, avoid unnecessary junk and magic and libraries.

It's the classic dilemma: in this world of ours, one can either return to monke or progress to crab.

I assume it's one with GC.

well, back to the mainframe it is then

It isn't a language problem, it is a developer practices problem. Developers code like security isn't a thing then are confused when their programs aren't secure.

I'd prefer that the "fix" for this isn't "require that millions of developers around the world all change their behavior, and never screw up in the future".

Languages can still protect against these sorts of bugs. Effect systems can describe as a part of the type system or function signature what kinds of things a function can do. For example, you could have a logging function where you know that the only side-effect is that it can write to stdout or stderr, or a file. Running arbitrary commands would not be allowed, and would cause the program to fail to compile.

As a real-world example, I've used the Slick SQL library in Scala, and the query type has a type parameter for whether the query is an read or a write. It's easy to build a simple abstraction on top of your data layer to never allow a write to go through, for example, if that's something that's useful to prevent bugs. I think one thing I did with that was set things up so callers could not accidentally write to MySQL replicas; only reads could go to them, and writes had to go to the primaries.

Granted, it takes discipline to build and use such a system, especially one that has such fine-grained effects. (I would expect most are more like a binary "does not do I/O" or "does I/O".)

  It isn't a language problem

https://www.adacore.com/about-ada

We had to spend a week back and forth with

"no, we're not vulnerable".

"But we see log4j is right there in the a file in the directory".

"No, it's not vulnerable and it's not being used and we can't reasonably fork the entire dependency just to remove that one sub-dependency from our project's dependency"

"But we see log4j is right there in the a file in the directory".

Also, there was no budget to do any sort of rewrite/refactor/etc anyway.

We've had similar out of a pen test that noted we were using nginx for some of our internal dashboards/tools and the version running (1.18 IIRC) was officially EOLed upstream so a high security issue.

They initially wouldn't listen to the argument that we use stable/LTS Debian/Ubuntu release only, and those hold functionality stable (which is one of the key reasons to use them) and backport security updates where relevant. We pointed them at the package changelogs and they were convinced that there was still a vulnerability not patched but for some reason didn't want to tell us which, when we finally got that out of them it turned out to be one that was introduced in a later version so was never relevant in the first place for the one the stable repositories included.

You would think a penetration testing company would have a clue about something so common…

If it was a credentialed review, you'd have hoped their scanners would plugin to the appropriate debian/ubuntu security database and give you a less false positive prone set of findings (although with debian based stuff you still have the problem of scanners reporting the "unfixed" set sometimes)

"You have a vulnerable version of netty"

"No, we don't, that's a false positive"

"No see, the tool says you need netty 4.x and that's version 1.x, you need to update"

"OK, but the tool is wrong, it's just picking up anything with 'netty' in the name, and that component is a wrapper around netty for some other thing, it only goes up to 1.8"

"You have to update it to 4.x, this is a vulnerability"

"Do you understand this isn't part of the thing you're concerned about"

"I am only concerned about shipping a product without vulnerabilities"

"Well this isn't one, because it's a false positive, here's the CVE, here's what it applies to, here's the link to this package on maven central, it's not part of netty, back off"

"But it's insecure, the tool says so"

We went round and round for about 3 hours until I told the guy (an infosec 'pro') to leave me alone until he'd figured out how to do his job... he came back to me the next morning for another 3 hour slack argument along the exact same lines at which point I told him to leave me alone permanently and communicate through management if he had anything to say. I'm not massively impressed with infosec as a profession after a few similar encounters.

That said, I've seen the inverse problem: engineering staff that either don't understand how their dependencies are managed (not unreasonable for less experienced teams, NPM for example has a tendency for huge dependency trees and this can be a hard problem to manage for a big project) or whom are for a plethora of reasons incentivized to push back on requests for work they don't perceive as important.

The middle ground here requires trust between both parties (InfoSec and Engineering org functions). Sadly, not all security programs are well managed and not all engineering teams have mature practices.

If you have time or capacity, you can get a lot of leverage here by educating your security analysts on how your technology works (e.g. how the build system functions, how your languages of choice share and bundle code). Trust can be built between engineering and infosec teams by educating where possible. Done well such practices can up-level both sides of these discussions and save you stress in the long run.

We took the time to analyze the vulnerability, and multiple competent people concluded: Yup. This is serious. Then we figured out plans, mitigations, and communication paths to update the plans and mitigations together with ops, dev and other folks. And then we shoved all of that as a large document into the larger technical organization as "Holy fuck the sky is falling, start patching, here's how to see when you have to, here's how you do. Go".

And that's also something I'm currently telling our new full-time security engineer. Be a bit mindful in the communication of criticality and vunlerability. Like, with my team of admins and SREs, you can be like "We're all doomed" and someone will be "but we don't deploy that feature, calm down. Would you like some tea? What evil thing did the CVE say to you?". Or they might escalate accordingly, if it looks bad to them, too.

But if you do that to most of our dev-teams? That'll be just an unstructured mess, which will be worse than waiting a moment to structure a proper response to the threat.

Yes it can, and I have worked with 'good' infosec people who were all about improving practices together. That attitude goes a long way - developers (well, the good ones) want to ship secure stuff and want to fold in best practices.

But the profession is unfortunately rife with people who don't know as much as they think they do, and who wade in heavy-handed, laying down the law to developers who might have been round the block once or twice themselves...

I wonder if it wouldn't be better to have a hybrid role that is part of engineering but has specific responsibility in this area, rather than as is often the case, an external party (or a separate team) which is perceived as adversarial.

There is one caveat however: you’ll probably still need dedicated security analysis or security engineers outside the chain of command of your product or engineering org.

There is a degree of necessary opposing pressures and incentives for infosec departments vs product departments: product organizations are rewarded for shipping more. Infosec organizations are rewarded for preventing incidents.

The most common result of this contention are situations people often lament in the comments here—security programs that are either ineffective, heavy handed, or both.

In my mind, a good security program positions itself as an enablement function and should strive to educate and inform over all else. People close to the work will be best positioned to make decisions about effective ways to harden their systems when informed of likely risks, requirements, or other security considerations.

With any luck this minimizes the need to mandate security motivated changes and changes they dynamic when your local security professional comes by with specific concerns. It’ll be less of “the sky is falling” and more of “we saw this specific problem and would like to work with you to fix it”.

What you described is usually the result of some "consulting company" (in our case big ones) that drop stuff with zero actual knowledge.

Every year we "review" these with the board and it is annoying as fuck.

Recently they got a new manager who understands we are in the same boat. They have to provide a report with some findings and I need to have a secure environment. We talked this over and suddenly the board presentation was cool.

As for internal teams, I have a few extraordinary people. They are very young and I had to coach them on how the vulnerability is actually critical or not depending on the context, data, etc.

So not all cybersecurity teams suck :)

In my experience tuning out false positives and other noise is fairly straightforward when you're only looking at CVEs, but things rapidly break down when you're trying to prioritize work with limited resources (only so many security engineers and engineering teams with spare cycles on their roadmap).

I have yet to find a good solution here other than manually whitelisting specific vulns in some kind of tracker and reviewing them whenever the network topology or configuration for an application changes.

Curious to hear your experiences, though I suspect this is a common and difficult to solve problem.

I have tested and deployed several scanning tool (and have contributed to nessus some years ago - their scanning capacities were really mediocre compared to the precision of the results. In out 100+k IPs on 10.x networks the time to scan was horrendous. But once an IP was scanned, the results were fine). A small disclaimer: I use several solutions but have no stakes in any of them (neither company nor personal). I have build relationships with them that helps me to provide substantiated feedback that sometimes event get taken into account :)

There are several issues, none of them has an automated way of fixing.

First there is the criticality of the finding. We found that often (especially on Linux) you will get a "raw" critical issue on a component that was not patched; And why it was not patched? Because the vendor set a low level. It is then that the discussion between the security team and the OS one starts - I have experimentally found that the vendor rating is usually better because they take into account the actual context of the deployment, especially for libraries.

OTOH crowdstrike vulnerability rating (I forgot the name of the component, it is on top of the EDR) is actually two ratings: the CVSS-reated one, and their own where they look at actual exploits and deployment. I have decided that critical+critical = actually critical that must be patched no matter what. This is the only way I found to quantitatively auto-decide on important patches. The goal is of course to patch everything but the reality is that there are so many loose ends in products that fighting for anything below the top two levels does not make sense.

The very sad thing here is that this is perceived as an issue of the OS/middleware and not the shitty products vendors deploy. They use some undocumented or weak solutions and then cry when everything breaks down. I develop myself (amateur developer) and have never had any issues for 30 years when upgrading the OS because I actually care about following the docs.

The second problem is that today products have nested dependencies. For our own products we do an awfully time taking and precise review of the actual impact on the product but this is because people actually care (this is not my team so I ma not taking any credit - I just like their approach). When a customer comes with a question we have a comprehensive response. This works or not, some people when they see "vulnerability" they stop at that and want that word to disappear.

When we are on the receiving side of such products (and log4j is a perfect example), we ask the vendor. If this is open source the answer is usually very good, there are some github/gitlab issues that address the point. Commercial vendors are much worse because they either say "no worries" or take weeks to respond. When a "no worry" comment comes from a company that cannot explain why they limit their passwords to 27 characters, it is worrisome - a bit like when my children say "no worries" (this is the moment when I start to worry).

Scanners are quite slow to respond to non obvious vulnerabilities (the ones you cannot easily tag though OVAL). For log4j for instance I coded a scanner/reflector on the night of Thursday (the information popped up about noon on Thursday western europe time). The scanners followed up after a week, but one could fine good ones open source quickly as well. So my advice would be to closely monitor GitHub for such tools when there is pressure (we switched to another tool for log4j when my team and the OS ones were successively finding better ones). This also led us to investigate a fantastic tool/framework to write scanners, which name escapes me right now but I can comment here when I am back from vacation and get hold of the gal who is looking at that.

The TL;DR version is that

- a few tools provide a way to quantify the risk (that's a huge slit but the best I can do) - at least there is a reasonable chance that you will not miss the worst ones

- GitHub activity when shit hits the fan is extraordinarily useful - the best tools emerge there, before the vendors catch up

- vendors are medium, but is is not as bad as you read

- your CISO should make the effort to communicate that there will be availability issues when the IT/security team decides to switch off a key component (say - email) if this presents a risk for the company. I have the chance to have a CEO who not only understands this but actually challenged me once to why I did not cut earlier. Having that arranged upfront talks a huge stress off the security team.

- and finally the sad reality is that the vast, vast majority of issues we get are from users who click on stuff

I am sure I forgot a lot - let me know if I can help with some details.

Please do feel free to reach out via email or LinkedIn (deets in my profile), always a pleasure talking shop with folks in the industry.

Cheers!

Guru: To not argue with fools.

Man: I disagree.

Guru: Yes, you are right.

I am coming to a point in my 25+ years career where I stopped to argue. I just tell them that they are wrong but I do not care until they put the company on danger.

Unfortunately this leaves me with fat powerpoint presentations which well to be the modern documentation of a company.

I’ve been impressed by automated bug checking tools in the past and I see this as part of the same issue. I don’t see why this would need an FTE to run code against a tool. CI should be enough.

From the parent, it sounded like the their issue wasn’t figuring out if something was a false positive. It was that another “intelligent life form” ran a tool and then wouldn’t accept when another “intelligent life form” assessed that a flag was a false positive.

If the infosec person is just running a tool and reporting results, why are they part of the loop? Make running the tool mandatory for each git push back to the main repo. Then, if/when there is a false positive, allow them to pass if they’ve already been “approved” by some means (like a .infosec_ignore file).

False positives don't waste the security team's budget, and are the only surefire way to justify ongoing expenditures on the scanning tool.

I will concede there probably are some firms out there acting poorly that way. When aren't there? Humans sigh. But by in large automation and the problems inherent are required.

> I will concede there probably are some firms out there acting poorly that way.

This is a hilarious take. Here's mine: The vast majority of these firms are here for liability. They do not provide security, they provide security theater so that if/when something goes wrong, the client can claim to have followed best practices, and that it's not their fault.

This style of theater is RAMPANT in the industry. Literally most of them come in with some version of a shitty automated tool, often with false positives in the 95%+ range, and then you check off checkboxes to make legal happy, and ensure that your insurance (or your customer's insurance) will pay if you get compromised.

This is not limited to small companies, but is instead mostly how large banks, credit firms, and large enterprises work.

The entire damn show is for liability, and half of these "Security professionals" can't do anything other than read the message coming out of their tool. They are utterly incompetent when it comes to applying logic to figure out whether a specific use of a "risky" tool/language feature a genuine problem, vs a god damned fucking regex match from their tool on something completely unrelated.

I went in as a naive young dev, I came out with ZERO respect for this industry, and a healthy dose of skepticism around anything these folks are saying.

Security researchers? Generally fine (although there's a new breed of them that simply submits inane non-vulnerabilities over and over to attempt to get bug bounties from large companies)

Security advice from open source devs? Generally top notch, listen to it.

Security advice from that contractor your company paid? Expect to have 95% false positives, and they'll miss the 2 places you know actually have issues. But it's ok because you'll check all the boxes on their sheet and legal is happy.

---

I'm waiting for insurance companies to wise up and stop covering companies who are breached. Prices are already shooting way up, since it turns out security theater does a very bad job stopping real threats, and that's what this industry is right now. Might was well be the TSA of software, gonna frisk you real good, and then flunk every fucking test.

Due to time and resourcing constraints, and a fixed launch date (don't ask), we had to significantly reduce scope. We are currently launching with probably 30% of the original scope. Once live we'll iterativley add features and grow our product to bring it back to the original vision.

Will we have to do pen testing again in the future? Nope. Even though a lot of our planned post launch features involve integrations with 3rd parties and the exchange of sensitive personal data.

Pen testing is 100% a box ticking exercise in my opinion. And most of the shops that offer this service are set up to provide the service such that large corporates to appease legal.

I really don't like pen testing as a practice and seriously dislike when it's a go live impediment.

could you elaborate on this a bit? what's wrong with it? to me it seems the only logical thing to spend external security budget on. (or internal red team if the company is so bit it can afford it.)

Here's a repo with lots of public reports by various consultancies, you could use that as a starting point: https://github.com/juliocesarfort/public-pentesting-reports

Security vs compliance is real. I love how you just map "compliance = security theater" because that is really the best way to describe it. The TSA bit has me in tears!

Better---just summarize the IPs scanned and report back which services were found running on said IPs. Then in an appendix, list why each service is (or might be) problematic. "Ping? Attackers might be able to figure out your network topology and that is bad because blah blah blah blah."

But 500 pages of this automatic breathless garbage? Utter trash.

In one of my past jobs I was an early-mover on doing a lot of our ops on Linux and the audit tools had no concept of the backport security model whatsoever. If you were running on some kind of LTS distro rather than a current distro, it would see "gosh you're 3 minor versions behind, you have a ton of unpatched vulnerabilities!", but in actuality you didn't, because the fixes got backported into security releases (the "31" in something like "3.1.22~ubuntu31" for example). But the tool was dumb and it just had a table that said "anything under 3.4.14 is vulnerable" even when it wasn't necessarily.

OP's "log4j 1.x is not vulnerable to this the first place" is a similar thing where either someone forgot to put in a value for "min vulnerable version" or the tool doesn't understand the concept of min-version at all. To be fair that can be genuinely tricky in java, best-case you are reading manifest files to try and pick out a version, but some legacy stuff doesn't have manifests, especially very legacy fat-jar stuff. Yes, that stuff didn't go away, it's still out there in places!

And to be clear this was a long-running issue... it wasn't a "oh our tool doesn't understand LTS, I get it" and they left me alone... this was every month they'd be back at me for a new list of utility vulnerabilities even though I repeatedly explained to them that I had set up apt-get to auto-upgrade every night and reboot, so we were running whatever the latest security backports were available, and that just like the last 10 times this was even more false-positives.

And again, this isn't just "we have to run it by you no matter what" and they leave you alone either. What security really wanted was for me to run windows and manually install the updates and get back in line with everyone else, even though that would have been a less secure outcome than my locked-down linux boxes. But this wasn't my day-job at the company so to speak, it was helping out another project with some ops and it needed to be as low-effort as possible (and to be fair their requirements weren't steep, auto-patch-and-reboot was fine by them and never caused any breakage during my tenure there).

The root problem is that these sorts of tools aren't a substitute for an actual security culture, they're often a symptom of a compliance requirement and the whole thing turns into a box-checking exercise. Someone is on their ass about this because of contractual requirements or PCI requirements, and they bought the cheapest thing off the shelf that would check the box, and they are implicitly showing you here that they don't even understand how to interpret the output of that.

I think the problem is that the current audits are inherently worthless. I've never seen another industry that would accept a 95% false positive rate from a tool. But that's on the low end from my experiences (I've done 4 major codebase audits, and worked in the security industry for 5 years)

Tools that routinely spit out 1000 plus vulnerabilities, and 6 months later you've checked off all of them without a single valid vulnerability in the list (but 3 you found yourself during that time that were missed entirely by the shitty regex that is really the entire tool in question).

And that's not helped by the warped incentives. Security has to be, fundamentally, assessed as a holistic thing and even highly skilled technical people can't really do that with modern systems. Now you add in auditors, a profession populated by glorified accountants, working off of gargantuan checklists.

You invite and reward mediocrity. At best.

Anyone good enough to actually understand and able to work with the technology will find a job in the industry, for twice the pay. And the same applies to regulators. They can't retain technically skilled staff, so they are filled with accountants. As a result we get rules and regulations, written by technically incompetent accountants, aimed at technically incompetent accountants.

To fill the gaps we have shitty, false-positive ridden, noisy, outright useless tools that generate reports intended to satisfy accountants. Security has next to nothing to do with that.

And to top it off, we have an entire industry riddled with so-called security engineers who know how to run a tool (with some marginally helpful StackOverflow pointers) and export a report, but can't for their life actually interpret, let alone VERIFY, any of it.

That has as much to do with security engineering as burning ants with a magnifying glass has to do with biology.

If you use Maven, just exclude that transitive dependency from the explicit dependency.

But yeah, dumb warnings are dumb.

  <dependency>
    <groupId>com.example</groupId>
    <artifactId>my-dependency</artifactId>
    <exclusions>
      <exclusion>
        <groupId>org.slf4j</groupId>
        <artifactId>slf4j-log4j12</artifactId>
      </exclusion>
    </exclusions>
  </dependency>

We can't upgrade the entire components that bundled up log4j for various reasons, starting from licensing rules. So we made the decision to strip out the entire JndiLookup class from every project that uses java. Clients do various scans, and rely on dumb version string matching and/or banner grabbing. We have to routinely point them to our VERY detailed and explicit log4j response document, and carefully explain to them that their scanners are relying on insufficient detection methods.

Security teams are quite content with us giving them detailed explanation about the false positive. And then they forget or deliberately choose to ignore the lesson and the next time they run their scans, get the same false positives again.

Even supposedly state-of-the-art security tools, to this day, refuse to actually verify their detections.

Unfortunately it does mean there's no getting around having someone manually deal with false positives.

And the most common use of log4j 1.x (logging to the console or to a file, with a simple configuration) uses none of the vulnerable parts.

This isn't even a new situation. We had the exact same problem some time ago with zlib: it was also embedded everywhere, and just finding all the affected apps was a nightmare. It's the reason most Linux distributions came to deeply dislike bundling libraries, instead of using a dynamically linked shared copy from a separate package. Unfortunately, it seems the lessons from that zlib incident are being forgotten, perhaps because too much time has passed (IIRC, in the order of decades) since then.

Is it, though? Seems like it would be trivial for a distribution maintainer to write a script to go through the dependency trees of every package, and then automatically rebuild any that depend on the vulnerable library. I don't think discovery is the problem.

I suspect this is really a legacy of when disk space was not so cheap; dynamic linking means smaller binaries (usually). And, even today, many distros would probably prefer their users to only download a 2MB library update from them than have to download 500MB worth of updated programs. Bandwidth is still not as cheap as we'd hope.

Edit: just realized you're probably talking about software packages vendoring in their own version of a library, not about a package statically linking with a system-provided copy of the library.

That won't help you rebuild the binary, but at least you can find them.

In addition to making it easier to do security updates in situations like log4j, it'd also make it easy to comply with LGPL license terms.

What you're describing would be more akin to shared linkage just in a single blob. That seems like it'd be a worst-of-both-worlds result.

Afaik, the ELF format allows you to replace sections in a binary. The code (or data) is packed as "Sections" and mapped to memory locations. You use a tool to unpack the ELF sections and replace them. The application accesses those sections of memory and uses whatever ABI the application and library are designed to interact with. As long as you patch a section with the same ABI that an app/library used before, nobody will notice. And the ELF ABI never changes. But of course, not every system uses ELFs.

But all this depends on the compiler/linker/etc and how they generate sections and store them in the binary (for example, is it all compiled into one blob-section, or does it store libraries as separate sections, which might be inefficient?). I somehow doubt Go does things in such a way to allow this kind of patching. They probably generate a giant blob full of random functions and record versions in a header without the ability to extricate a whole library.

With Go, can you look at a compiled binary in an image on the artifact repo and identify what libraries it used and if those libraries have vulnerabilities?

(and it appears that my question was answered in a sibling comment)

Even Linux distributions where rebuilding the world is easy (like Debian) insist on breaking dependencies out into shared libaries, since tracking ad hoc vendored dependencies is just too difficult.

For example, sometimes people play dynamic loading and renaming tricks to allow multiple versions of the same Java library to coexist. That can usually be detected by vulnerability scanners, but not always.

I am also not aware of any Java developers who build 3rd party libraries from source and avoid using maven (or gradle, or the odd one with ivy).

Your "what if..." doesn't describe any workflow that I have ever experienced with Java in a professional (and even hobbyist) context.

https://maven.apache.org/plugins/maven-shade-plugin/

I have exploded war deployments ( https://www.jrebel.com/learn/what-exploded-and-packaged-depl... ) put in a Tomcat image and the scanner is able to identify the libraries that the classes came from and their vulnerabilities.

I've also got fat jars for Spring Boot batch jobs that are single jars with all of the dependencies packaged in them for ease of copying around a single thing on the server... and the dependency checker is able issues there to based on fingerprints of individual class files.

These work on the "you downloaded this from a known package management server". As long as you are using maven to download the dependencies, a vulnerability checker will be able to identify it. One built into maven ( https://owasp.org/www-project-dependency-check/ // https://jeremylong.github.io/DependencyCheck/dependency-chec... ) will be able to catch these as part of the build itself - irrespective of the shading of the final artifact.

If you are building from source, and obscuring the class names - yes, you can foil an external vulnerability checker (and since you're doing it from source, you are also not downloading it and thus avoiding a maven plugin use)... however, I don't know any projects that go through such lengths to compile 3rd party Java code themselves rather than using a jar, and are trying to evade vulnerability (and license) checkers.

That said, I think this missing the full picture- we have the same problem further down the stack, at the OS level, where old machines are running without receiving security updates on a regular basis.

I don't know that there's really much of a good answer to the problem short of "don't leave your children unattended". Unlike real kids and the real world, there's plenty of bad actors looking for unattended services to stab.

And yes, it has dependencies! So does your go file, I'm afraid.

Is that a particular problem? What's the difficulty with recompiling?

When 98% of your users only have the binary and not the source code it means 98% of your users won't even be aware that they need a new version that isn't vulnerable.

Remember the vulnerability disclosure will be something like "Log4Go has a vulnerability!!", not that the binary apps you've downloaded have a vulnerability. So if it affects 200 projects, every one of those projects will also have to publish vulnerability notifications independently.

And frankly 80% of those projects will likely be abandoned or at least neglected after a couple of years, leaving many people with vulnerable binaries they don't even know to check for vulnerabilities.

In other words, they very advantage of Go (easy to distribute binaries directly!) is a security disadvantage. If you instead installed the app using a package manager, the package manager could at least notify you that you have a package with a security issue. But that's nothing new; security and ease-of-use are often at odds.

1. How do I find out which programs on my system are Go programs? I have a lot of programs scattered all over. Someone will have to come up with a method (probably search through $PATH for files, run `strings` on it, grep for "Go", pray that's enough) to identify them.

2. They don't list a website, so I have to google the name of every program and find their website.

3. How do I tell which ones are affected by the vuln? I can hope their websites tell me, but maybe they're no longer maintained, or they haven't updated their READMEs.

4. If they don't provide a patched version (most probably won't), I might have to upgrade, which may break things (probably will), assuming they even have an upgradeable version that isn't affected.

5. If I can't upgrade and have to patch, am I even a software developer and know how to do that?

6. If I luckily happen to be one, what system compiled this app 8 years ago, with what dependencies, with what version of Go, etc? Can I recreate that system and learn their build tools to finally patch & recompile? (I am not a Go developer and it has taken me days to figure out how to get different Go programs to compile, for reasons I don't understand)

7. Once I apply the patch how do I test it? (Log4j had multiple iterations of patches/mitigations, some didn't work)

If I can't find the source code, can't upgrade it, & can't patch it, I am just stuck with a vulnerable version until I can somehow replace the program. For the programs I was able to identify as vulnerable, anyway.

Log4j was "easy" compared to this, because you could just find all the jars on your system and unzip them, find a file, replace it, zip it back up. Log4j was actually a nightmare though, and Go apps will be worse.

Well how are these programs getting onto your system?

Presumably via some package manager, or an image builder?

Don't those manage the update for you?

If you're running code that you don't know who wrote, with no source code, that you can't patch, that nobody is taking responsibility for, then yeah you've got issues. But you had issues already if that's the predicament you're in. Some of this applies to Java too - obfuscation breaks the fixes you described using!

> If you're running code that you don't know who wrote, with no source code, that you can't patch, that nobody is taking responsibility for, then yeah you've got issues

25 million computers around the world still use Windows XP. It was launched 21 years ago, EOL 8 years ago. Very common actually, especially since most software in the world is not open source. People just keep using software that works.

This is the default state of affairs for nearly everyone who uses a computer.

We can either throw our hands up and futilely say "well they shouldn't do that!", or we can try to solve the problem.

This isn't relevant. What's relevant is if any of the specific programs I am responsible for have a vulnerability. If so, I need to update those specific programs. The notion of implementation language, shared library, dependency, these are all abstraction leaks. The atomic unit of responsibility, at the machine level, is program/service.

Alternatively, a more-savvy user has to hear about this Log4Go vulnerability and see if any of their applications use it and might need an update (which might involve contacting the maintainer to inform them that they need to update their software). But the vast majority of users aren't even going to hear about the Log4Go vulnerability, let alone understand what it means or what they need to do to protect themselves.

The safest thing would be for every application to dynamically link to a system-provided version of Log4Go, and OS maintainers would be the ones to send out an update via their OS's auto-update service. This does create other problems, but at least they are problems that technically-unsophisticated users do not need to solve or concern themselves with.

(Of course, there is no mechanism to do this with Go modules, so the entire thing is moot.)

This is why I get annoyed when developers say "I have to vendor library X into my application, because I'm afraid that the system copy will get updated to a version that breaks my app". Well then, maybe you shouldn't use a library whose author can't abide by semver and provide a reasonably-strong promise that minor and patch version updates won't break things. Otherwise you are just outsourcing your problems to your users.

Nor should it. A dependency should be encapsulated by its consumers. A vulnerability that leaks to the consumer application is a bug in the consumer application.

> The safest thing would be for every application to dynamically link to a system-provided version of Log4Go,

Unfortunately this means that any application which uses Log4Go cannot make any definitive assertions about the specific code which is executed by their distributed artifact. This is a problem, and the cost of this problem is far higher than the benefits conveyed by dynamic linking. The future of application distribution is absolutely static linking.

> "I have to vendor library X into my application, because I'm afraid that the system copy will get updated to a version that breaks my app". Well then, maybe you shouldn't use a library whose author can't abide by semver and provide a reasonably-strong promise that minor and patch version updates won't break things. Otherwise you are just outsourcing your problems to your users.

As a software producer, it should not be possible for a dependency to impact my users without my knowledge.

It is infeasible to rely on any specific level of versioning stability from my dependencies. Every dependency I adopt represents a risk which I am taking on.

When they find a critical bug in a Go module or a Rust crate, you usually end up with a litany of packages that had to be updated as well because everyone and their crab relies on that crate/module.

Count in release managers (or whatever that's called at your shop) who pin all dependencies to the most exact minor version before shipping, because they have learned to do it like a mantra without actually understanding all the "whys" of it and just keep religiously applying that hammer everywhere.

People keep complaining about how distributions work and how the traditional package management sucks all the time (of thousands complaining, I estimate that maybe 20 people actually know what they are talking about and thus are qualified to whine), but they somehow keep forgetting that those tools exist for a reason and that they are about to rediscover it the hard way. Chesterton's fence and all that.

It's a fork of log4j 1.2.17 by the original author that fixes a boat load of issues with 1.2.x and is a drop in replacement for log4j. Very useful if you can't upgrade to log4x 2.x.

* We had log4j 1, which was good enough for most use cases and everywhere.

* We had slf4j/logback, which was from the same authors, and broke backward compatibility but gave us a fundamentally better design.

* We had java.util.logging, which while braindead was builtin to Java and hence available everywhere.

*We had commons logging, which was never supposed to escape from tomcat, but got overused everywhere anyway.

All of this was more logging frameworks than was healthy for 1 language, and there are some more smaller ones.

Then apache decides to put new people on log4j, do a backward incompatible v2 design that nevertheless is worse than slf4j. Why?

slf4j itself isn't a logging framework. It's a facade to logging frameworks.

Simple Logging Facade for Java ( https://www.slf4j.org )

It needs a logging framework behind it - log4j, log4j2, logback, commons, JUL.

The question is "why do log4j2?"

Logback went from the log4j1.x path ( https://logback.qos.ch )

Log4j2 has a lot of features that weren't present when the project started ( https://en.wikipedia.org/wiki/Log4j#Apache_Log4j_2 ).

There is a licensing difference between Logback (LGPL) and Log4jx (Apache Commons).

Apache projects in general have this feature creep seeping into every single one of their products that I have used.

It's just hard to trust apache projects now.

Minor nit: log4j doesn't implement the slf4j interface, though. If you do want log4j to do your logging, but use slf4j in your application, you need to include a "bridge" library that adapts slf4j to log4j. And, conversely, if you have dependencies that use log4j, but want things to end up going through your slf4j-supporting backend (say, logback), you need to exclude the "real" log4j jar and include a different library that implements log4j's interface and forwards calls to slf4j.

Because the original Log4j author wanted to do his own thing for a Log4j successor (slf4j/logback), and the Apache Log4j maintainers also had ideas about a next-gen framework and wanted to do that.

That’s Open Source, everyone can do their own thing if they like, and some things will get adopted more than others. Log4j2 got the most adoption because they had the Log4j “brand” and the Apache backing and more company-backed(?) maintainers.

I don’t consider java.util.logging to be braindead. It can in principle be used as a logging facade like slf4j, but not too many do that, and Log4j 1 had come first and kept that momentum.

The real mistake was for Java to not have a standard logging API earlier. (It took six years after Java 1.0.)

- logging methods taking arrays rather than varargs (this would be a trivial change!)

- there being no way to create a SimpleFormatter with a custom format in code other than setting a system property before creating it

- there being no Logger:getLogger override which takes a class, only strings

At some point, i am going to write a little facade which wraps jul and fixes the papercuts, which feels like a very silly thing to be doing!

In any serious application you will want to have a central logging wrapper anyway (I always use one), which automatically takes care of the varargs issue.

The implementation of SimpleFormatter is 30 lines that you can easily copy and adjust as needed, including with your own custom format.

Regarding logger names, I always define an explicit set of logger names for the application, because that makes the logger names independent from implementation-detail subpackages, and any operations-side log filtering based on logger names will remains stable, regardless of how you refactor your implementation into Java packages. Basically, I see the logger names as part of the external interface of the application, and therefore I want to have them decoupled from the internal package structure, so that mere refactorings don't break that interface. You can still easily find all program locations that log to a specific logger, for example by Find Usages on the respective static Logger constant.

I see they did some optimizations to do this work lazily if needed. But when log4j 1 was on its peak, that wasn't done, and using the built in logger was slow enough to have measurable impact .

See https://github.com/openjdk/jdk/blob/6765f902505fbdd02f25b599... at the bottom.

Like what you most likely need is a wrapper for System.out that maybe does some clever buffering and backpressure management, but the rest is sensibly managed outside of the application; whether by logrotate, ELK, or whatever.

Long story short, no modern Java logging framework has any sensible reason to be much longer than a few hundred lines of code.

Even then, of course, this kind of logging should be able to be done in far fewer lines than most Java logging frameworks.

That's exactly what JDK9+ java.lang.System.Logger is. See https://stackoverflow.com/questions/59976828/difference-betw...

And from the motivation section: "The proposed service enables applications to configure the JDK to use the same logging framework as the application: It would only need to provide an implementation of the service that returns platform loggers that wrap the loggers of the preferred logging framework."

[1] https://openjdk.org/jeps/264

I tend to do this, but it's pretty damn easy for me. I have a "2-level-dependency" rule that I generally go by, these days. i.e., Each dependency can have only one dependency, and that dependency can't have any others.

The main way that I deal with it, is to write all my own dependencies, and I don't count toolsets, standard libraries, or platform frameworks (for better, or for ill).

I really, really want dependencies to work. They are a great idea, and I have been promoting modular architecture for decades.

It's just that we are in a "wild west" situation, where we can't trust dependencies.

I'm tired of hearing the phrase "It's a solved problem," with a reference to a dependency.

I don't have the answer. I just have my own rule, which elicits scorn from most modern programmers (I'm painted as a luddite, refusing to "get with the times," or as a "tinfoil-hatter").

WFM. YMMV.

It's a very nice concept, but I kind of have my doubts about maintainability.

Here's the dependency manifest from the app I'm working on now (From my localization file):

    // MARK: -
    // MARK: - DO NOT TRANSLATE BELOW THIS LINE -
    // MARK: -
    "SLUG-VERSION-BMLT"                             =   "BMLTiOSLib: 1.4.2";
    "SLUG-VERSION-KEYCHAINSWIFT"                    =   "KeychainSwift: 20.0.0";
    "SLUG-VERSION-LGVCLEANTIME"                     =   "LGV_Cleantime: 1.3.5";
    "SLUG-VERSION-UICLEANTIME"                      =   "LGV_UICleantime: 1.1.1";
    "SLUG-VERSION-AUTOFILL"                         =   "RVS_AutofillTextField: 1.3.0";
    "SLUG-VERSION-GCD"                              =   "RVS_BasicGCDTimer: 1.5.0";
    "SLUG-VERSION-CHECKBOX"                         =   "RVS_Checkbox: 1.2.0";
    "SLUG-VERSION-OBSERVER"                         =   "RVS_GeneralObserver: 1.1.0";
    "SLUG-VERSION-GST"                              =   "RVS_Generic_Swift_Toolbox: 1.10.1";
    "SLUG-VERSION-MB"                               =   "RVS_MaskButton: 1.2.0";
    "SLUG-VERSION-PP"                               =   "RVS_Persistent_Prefs: 1.3.1";
    "SLUG-VERSION-UKT"                              =   "RVS_UIKit_Toolbox: 1.1.5";
    "SLUG-VERSION-WHITEDRAGON"                      =   "White Dragon SDK: 3.2.2";

All the others, are in my own repos, as top-shelf-quality open-source modules.

[0] https://github.com/evgenyneu/keychain-swift

Sure, I would never write these log4j security-related bugs, because I would never even think to write such obscure functionality into a logging library for my own use.

Regardless: frankly, I think anyone who claims they won't write security bugs... well, that kind of person is probably a bit unrealistic about the flawlessness of own abilities, to put it as politely as possible.

However, it appears that the norm in the industry, is to write lash-up code, as quickly as possible, in the hopes on an MVP that will get brought up by a FAANG.

And I am doubtful that most OSS actually gets all that "other eyeballs vetting" that everyone talks about. Most folks that I know, that use a lot of these dependencies, would be unable to understand what goes on in them, let alone offer fixes.

In fact, I remember a founder, on this very forum, declare that "If you don't get physically sick, looking at your MVP code, then you are spending too much time on code quality."

That posture doesn't exactly inspire confidence. I'm pretty good at what I do, and I tend to "overengineer" my security-related stuff, because I don't really have huge confidence in that. It's not my forte.

But writing good code, is something that I can do.

Like I said, WFM. YMMV.

Sorry if I gave that impression; I wasn't trying to put those words in your mouth.

> And I am doubtful that most OSS actually gets all that "other eyeballs vetting" that everyone talks about.

Agreed, but a reasonably popular OSS project has a much higher chance of getting that vetting (or any vetting at all, really) than a bespoke library that the public never sees. The chances are a little higher for a nice / little-used bespoke library that you end up putting on GitHub or wherever, but it's still pretty low.

> In fact, I remember a founder, on this very forum, declare that "If you don't get physically sick, looking at your MVP code, then you are spending too much time on code quality."

I would never want to work for such a founder. I get the tension between getting a MVP out before you run out of money, but experience shows that prototype code ends up living in production a lot longer than anyone plans for. And I've personally seen those prototypes cause 6 and 7 figures of damage to a company later, when they haven't been properly replaced or at least improved when they should be.

I'm glad what you're doing works for you (you sound like you probably actually are an above-average developer, unlike -- by definition -- most people), but I don't think I could in good conscience recommend your approach as a general practice.

I can live with that, but ... (There's always a "but").

I am not happy at all, with the general industry practice of writing every project to be something that can be understood by inexperienced, undisciplined coders.

Every language and programming methodology has an "advanced" type of thing, requiring people to have experience and/or book-larnin'.

I write Swift at a fairly advanced level. I am not at the level of some heavy-duty advanced Swift people, but I am pretty "idiomatic," in my approach. It is not "rewritten TypeScript," like so much code out there.

My code is very well-documented, and I hold myself to standards of Quality that most folks in the industry consider to be obsessive to the point of insanity. My testing code usually dwarfs my implementation code, and my documentation is, let's say ... complete. You can see what I mean in my latest module[0].

The fact that so many folks have so much scorn for my posture and approach, tells me pretty much all I need to know. My values are not in sync with the prevailing industry values. I guess that I'm an "anomaly." I can live with that, as well.

I won't write junk, so that someone used to junk, can comprehend it. If people aren't willing to learn enough to understand my middle-of-the-road semi-advanced Swift, then I can't help them. Swift is an awesome language. I feel that we are doing ourselves a disservice, if we do not explore it.

I write for myself. I write code and documentation that I want to use (and I use it). I really don't care, whether or not someone else "approves" of it. I am not relying on others to review, maintain, or patch my code.

When I do use other people's code, I vet it fairly carefully. Including a dependency is a really serious matter. I'm handing full control of my execution context to code that someone else wrote. I'd damn well better take that Responsibility seriously.

[0] https://github.com/RiftValleySoftware/RVS_UIKit_Toolbox