On the back of an envelope, exporting a hundred million drones or so per year ought to fix the current Russian budget shortfall. Less would be needed if they can charge high prices, or if the state produces them and so gets the profit instead of just the taxing the manufacturer's profit. It's quite a volume.
Maybe you ought to read the article. Or a few of the articles.
At this stage, according to the numbers presented here, Russia would have to increase taxes on oil to about 115% of the exports (not the profit, the total exports) in order to balance the national budget. These numbers are well attested. They've been growing worse and worse for a long while and as the number 115% shows, they are really very bad now.
When did the Russian economy's problem become bad and how much later did they increase from bad to terrible? Matter of opinion, but if you've been hearing about it for years, that's because it took a long time to get as bad as those 115% indicate.
SHOULD is a requirement. It means that you have to do it unless you know some specific reason that the requirement doesn't apply in your case. "I don't want to" is not a valid excuse, "I don't see a reason to" isn't either.
IIRC this particular rule is a SHOULD because MUAs often send messages without a Message-ID to their submission server, and the submission server adds one if necessary. https://www.rfc-editor.org/rfc/rfc6409.html#section-8.3 The SHOULD lets those messages be valid. Low-entropy devices that can't generate a good random ID are rare these days, but old devices remain in service, so the workaround is IMO justified.
I once had a job where reading standards documents was my bread and butter.
SHOULD is not a requirement. It is a recommendation. For requirements they use SHALL.
My team was writing code that was safety related. Bad bugs could mean lives lost. We happily ignored a lot of SHOULDs and were open about it. We did it not because we had a good reason, but because it was convenient. We never justified it. Before our code could be released, everything was audited by a 3rd party auditor.
Maybe the standards documents you are used to differ from RFCs, but here is the official language:
3. SHOULD This word, or the adjective "RECOMMENDED", mean that there
may exist valid reasons in particular circumstances to ignore a
particular item, but the full implications must be understood and
carefully weighed before choosing a different course.
SHOULD is effectively REQUIRED unless it conflicts with another standards requirement or you have a very specific edge case.
I just don't understand how you get from the text you pasted to "required". Nowhere does it say that anything is effectively required. Words have meaning.
> the full implications must be understood and carefully weighed before choosing a different course.
In this case, the full implication is that your email might be undeliverable. "Should" indicates that the consequences for this fall on the entity that is deviating from the thing they "should" be doing.
But the RFC language clearly anticipates there are situations and good reasons leading to a message that does not include a message-id. Google therefore would be rejecting RFC-compliant emails, and they are the ones who have to justify themselves.
Theoretically, anyway, I expect in practice they'll just ignore the issue or have their own good reason. But they should accept emails with no message-id; there it does strain the imagination to see why lacking an ID would make a message unreadable or undeliverable.
There are indeed such situations. Two situations AFAICR, and neither of them apply to when you connect to someone else's MX.
Gmail rejects the vast majority of compliant messages, I think they've stated in public that they reject >99.9% of messages, and hearsay has it that the percentage for with minor errors like this is even higher.
There are good reasons why a message might be unreadable. For example, message-id is often used by the threading algorithms in MUAs and IMAP servers, and many don't test whether their threading code handles ID-less messages. I use one that deduplicates by ID, what do you think it does when the ID is empty or missing? I don't know, I haven't tested and I'm not going to.
- `MUST: This word, or the terms "REQUIRED" or "SHALL", mean that the definition is an absolute requirement of the specification.`
- `SHOULD: This word, or the adjective "RECOMMENDED", mean that there may exist valid reasons in particular circumstances to ignore a particular item, but the full implications must be understood and carefully weighed before choosing a different course.`
- `MAY: This word, or the adjective "OPTIONAL", mean that an item is truly optional.`
In practical terms:
- MUST: It's always a failure to do this. E.g. you MUST have some form of stored energy in your car for it to propel itself down the highway.
- SHOULD: If you don't do this, it's likely to cause failures unless you really know the situation is one of great exception and have thought about what else this change may affect. E.g. you SHOULD maintain a large distance between yourself and then next vehicle on the highway (an example of an exceptional case might be a standstill backup on the highway).
- MAY: This is something which is actually optional and has negligible impacts to successful operation if you do/don't. E.g. you MAY activate cruise control instead of always manually operating the accelerator.
For your car example as-is it'd probably best be MUST unless there is expectation one might reasonably consider their car exploding a valid scenario. In the real world where the car doesn't actually blow, it'd probably be that you MAY leave your keys in the ignition rather than SHOULD/MUST.
Is that what the spec says? Or is this something that Google decided, by making an optional feature a requirement when interoperating with their systems?
It is something Google decided. SHOULD means the other party should anticipate may not. The party examining a SHOULD and deciding not to do something is obviously not required to consider incompetence of other RFC readers as a reason to global replace SHOULD with SHALL before examining requirements.
You should wear sunscreen to the beach. Its recommended as a good way to prevent sunburn. However, the beach police aren't going to come get you for not wearing it. You just might get a sunburn if you don't plan accordingly with other sun countermeasures.
So if I send an email that lacks a feature that MUST be there, will the email police come get me? At a certain point, looking for an analogy stops making sense I think.
No, its not a "required"... It means someone may have reasons not to use something, and so spec implementors need to allow for circumstances where it is not present.
Those reasons can be anything. Legal, practical, technological, ideaological. You don't know. All you know is not using it is explicitly permitted.
"permitted" is a pretty empty word in the given context. Because dropping such emails is equally "permitted". Sure, there will be no arrests made, but there will be consequences. And those are what this article is about.
If that's your line, then I am equally permitted to send random binary blobs along the way. Not a crime, so totally permitted. They'll just drop the connection.
Buuut I don't think that is at all relevant to the discussion at hand.
I don't even know how you got to "used twice" tbh. Both your own comment AND the post you quoted from only have a single "must".
The only thing that text demands is understanding and carefully weighing the implications. If, having done that, you conclude that you don't want to then there is absolutely nothing in the spec stopping you. Maybe the spec would have been better off putting more stuff in SHALL and less in SHOULD, but as written that is definitely not the case.
Any time any document (standards or otherwise) says something is recommended, then of course you should think it through before going against the recommendation. Going from their verbiage to:
> SHOULD is effectively REQUIRED unless it conflicts with another standards requirement or you have a very specific edge case.
Can you start actually reading what you quote? "The implications must be understood and weighed" does not mean "you are required to have a good reason not to do it". I can know of the Message-ID field, understand what it's used for, and carefully weigh the possible lack of interoperability against the effort required, concluding that I'm too lazy to do it right now. I have then understood and carefully weighed the implications before choosing a different course, but I don't have a good reason to.
Words mean things, especially in standard texts. You can't just carelessly rewrite sentences using not-quite-synonyms like you're doing here.
Oh, and that "must" is not a "MUST". Had the text said "The full implications MUST be understood ..." it would have been a proper requirement by the standard, but this lower-case "must" is just normal part of prose and not the magical "MUST" word which formally imposes a requirement.
This very clearly says that SHOULD is not effectively REQUIRED at all, and is fact nothing more than RECOMMENDED. Really not sure how you misinterpreted this so badly
It's not required but they need to understand the implications. In this case the implication is that Google drops the mail. So clearly they didn't understand the implications.
Google probably did parse these messages as well-formed before inspecting them and deciding to drop them based on the lack of this field. The RFC imposes no mandatory obligation to deliver messages just because they are well-formed.
Assuming we’re talking about RFC 2119, it’s important not to collapse the distinction between SHOULD and MAY, which is there for a reason. MAY elements are legitimately optional, SHOULD elements are there for a reason and are disregarded at one’s own risk.
> SHOULD This word, or the adjective "RECOMMENDED", mean that there may exist valid reasons in particular circumstances to ignore a particular item, but the full implications must be understood and carefully weighed before choosing a different course.
To validly disregard a SHOULD, you need to (a) fully understand the implications, and (b) accept them.
Any time someone disregards a SHOULD and then complains about the result, they are necessarily in the wrong. Either they didn’t fully understand the implications, or they don’t actually accept them.
Agreed. We use RFC language at work for documents, and whenever I see someone using the word SHOULD, I always ask what is the MAY where you don't need to do the thing? If they can't think of one, then make it a MUST. SHOULD always means there are valid reasons to not do it.
Could was yeah nice to have.
Should, was yeah the system shall have it, but it's ok if not.
Shall : ffs we will hunt you and kill you if you don't implement it.
Yes, except there seems to be a move on the best words from SHALL to MUST and from SHOULD to MAY.
IANAL but I recall reading this in e.g. legal language guidance sites.
Pulling exact quotes out, SHOULD means "there may exist valid reasons in particular circumstances to ignore a particular item" while MAY means "an item is truly optional."
I don't think this can be interpreted as simply "should is optional".
Well… yes… no… SHOULD means that you must either do the thing or understand the consequences of not doing it. That's not simply optional, the two options are to ① spend time on writing code and ② spend time on learning the consequences. Either way you need to think hard and spend time. And that's why the definition of SHOULD includes the word "must".
> but the full implications must be understood and carefully weighed before choosing a different course.
In other words, you better have a damn good reason for deciding not to do it.
And (possibly even more to the point), you must decide not to do it, rather than simply throwing code at the wall until most emails make it through unscathed.
I think that is a bit to easy. MAY is described ar optional.
SHOULD - Should really be there. It's not MUST, you can ignore it but do not come crying if your email is not delivered to some of your customers !
you should have though about that before.
3. SHOULD This word, or the adjective "RECOMMENDED", mean that there
may exist valid reasons in particular circumstances to ignore a
particular item, but the full implications must be understood and
carefully weighed before choosing a different course.
If the client SHOULD do something and doesn't, and your server does not know why, you SHOULD disconnect and move on.
If the server has considered fully the implications of not having a Message-ID header, then it MAY continue processing.
In general, you will find most of the Internet specifications are labelled MUST if they are required for the protocol's own state-processing (i.e. as documented), while specifications are labelled SHOULD if they are required for application state-processing in some circumstances (i.e. other users of the protocol).
> If the client SHOULD do something and doesn't, and your server does not know why, you SHOULD disconnect and move on.
That is not a rule.
In this situation the server can reject any message if it wants to, and not doing a SHOULD tests the server's patience, but it's still ultimately in the "server wanted to" category, not the "RFC was violated" category.
> But I'm not playing, because you're right: I don't know why you would be doing that.
This is the comment you originally replied to:
>> If you're implementing a server, "the client SHOULD but didn't" isn't a valid excuse to reject a client either.
>> You can do it anyway, you might even have good reasons for it, but then you sure don't get to point at the RFC and call the client broken.
They're talking about how to categorize actions, just like I am.
So I thought you were playing on that same topic.
But if you weren't on that topic, and given that you quoted only the first one of those sentences, I have a guess.
I think you didn't realize how the second sentence affects the meaning of the first one, and you misunderstood what they were saying as trying to tell servers they can't reject. They were not trying to tell servers they can't reject.
If that's the case, then this whole line of conversation is pointless, because you were rebutting an argument that nobody made.
> It's a description of the behaviour of other Internet hosts, and it's a description of exactly what is happening in the linked article.
Taking a description of behavior and throwing an RFC-style "SHOULD" in front is only going to be correct if you get lucky. "is" and "SHOULD" are different things!
How does Google know whether or not the sender has a valid reason? They cannot know that so for them to reject an email for it means they would reject emails that have valid reasons as well.
How would the sender know the consequences of sending without the header? You shouldn’t assume anything here. As a sender, you should include it unless you’ve already worked out what the recipient is expecting or how it will be handled. Doing this with email is silly because the client is sennding to so many different servers they know nothing about so it’s basically a requirement to include it.
It is not required for the protocol of one SMTP client sending one message to one SMTP server, but it is required for many Internet Mail applications to function properly.
This one for example, is where if you want to send an email to some sites, you are going to need a Message-ID, so you SHOULD add one if you're the originating mail site.
> How does Google know whether or not the sender has a valid reason?
If the Sender has a valid reason, they would have responded to the RFC (Request For Comments) telling implementers what they SHOULD do, rather than do their own thing and hope for the best!
Google knows the meaning of the word SHOULD.
> it means they would reject emails that have valid reasons as well.
No shit! They reject spam for example. And there's more than a few RFC's about that. Here's one about spam that specifically talks about using Message-ID:
> If the server has considered fully the implications
The server "considers" nothing. The considerations are for the human implementers to make when building their software. And they can never presume to know why the software on the other side is working a certain way. Only that the RFC didn't make something mandatory.
The rejection isn't to be compliant with the RFC, it's a choice made by the server implementers.
Either the server must explicitly confirm to servers or the clients must accept everything. Otherwise message delivery is not guaranteed. In the context of an email protocol, this often is a silent failure which causes real-world problems.
I don’t care what the protocol rfc says, the client arbitrarily rejecting an email from the server for some missing unimportant header (for deduction detection?) is silly.
These changes MUST NOT be applied by an SMTP server that
provides an intermediate relay function.
That's Google in this situation.
> Stop hiding behind policy and think for yourself.
Sometimes you should think for yourself, but sometimes, and friend let me tell you this is one of those times, you should take some time to read all of the things that other people have thought about a subject, especially when that subject is as big and old as email.
There is no good reason viva couldn't make a Message-ID, but there's a good reason to believe they can't handle delivery status notifications, and if they can't do that, they are causing bigger problems than just this.
“MAY This word, or the adjective "OPTIONAL", mean that an item is truly optional… An implementation which does not include a particular option MUST be prepared to interoperate with another implementation which does include the option, though perhaps with reduced functionality. In the same vein an implementation which does include a particular option MUST be prepared to interoperate with another implementation which does not include the option (except, of course, for the feature the option provides.)”
Note how it explicitly calls out interoperation with implementations that do or do not implement MAY. As a exception that proves the rule, we can reasonably assume that not interoperating with a system ignoring a SHOULD rule is a correct implementation and it is the fault of whoever is not implementing SHOULD.
Hearsay has it that the reason is spam. Spam messages are said to have massively higher chances of minor RFC violations when they arrive at the destination server.
Most of the time, in my experience, when one encounters a situation like this in Internet tech (i.e. "why is this suggestion treated like a hard requirement?"), this is the answer: "because attackers found a way to exploit the lack of the suggestion's implementation in the wild, so it is now a hard requirement."
The standards, to my observation, tend to lag the CVEs.
Side-note: If someone has built a reverse-database that annotates RFCs with overriding CVEs that have invalidated or rendered harmful part of the spec, I'd love to put that in my toolbox. It'd be nice-to-have in the extreme if it hasn't been created yet.
Because in practice it showed up for a period of time as a common thing in spam-senders. They were trying to maximize throughput and minimize software maintenance costs, so they leave out things that the spec says are optional. But that makes "a commonly-implemented optional thing was left out" into a stronger spam signal.
Is it still a strong spam signal? Hard to say. Sources disagree. But as with laws, heuristics, once added, are often sticky.
CVE classify a lot of things that have nothing to do with security.
Not having a Message-ID can cause problems for loop-detection (especially on busy netnews and mailing lists), and with reliable delivery status notification.
Dealing with these things for clients who can't read the RFC wastes memory and time which can potentially deny legitimate users access to services
> It seems that Gmail is being pedantic for no reason
Well, gmail does not manage usenet groups and mailing lists. Delivery status notifications are considered best effort so it wouldn't make sense to block messages for that case.
Additionally, Gmail adds its own message identifier on every message (g-msgid) because it knows that message ids can not be trusted to be unique.
Finally just calling me ignorant is the cherry on top – please try to keep things civil on here.
> Well, [google] does not manage usenet groups and mailing lists.
They do. Sort of.
Google used to nntp, and manages the largest usenet archive; They still have one of the largest mailing list servers in the world, and they still perform distribution on those lists via SMTP email.
They still have all of the problems associated with it, as do lots of other mail/news/list sites still do that are a fraction of Google's size.
> Delivery status notifications are considered best effort so it wouldn't make sense to block messages for that case.
Sure it does.
You consider them best-effort, but that doesn't follow that I should consider them best-effort. For a simple example: Consider spam.
In any event, if you keep sending me the same message without any evidence you can handle them, I'm not going to accept your messages either, because I don't know what else you aren't doing. That's part of the subtext of "SHOULD".
Most big sites take this policy because it is internal nodes that will generate the delivery notification, but the edge nodes that are tasked with preventing loops. If the edge node adds a Message-ID based on the content, it'll waste CPU and possibly deny service; If the edge node naively adds a Message-ID like an MSA, the origin won't recognise it, and forwarded messages can loop or (if sent to a mailing list) be amplified. There also are other specific documented requirements related to Internet Mail that edge nodes not do this (e.g. RFC2821 § 6.3).
However you seem to be assuming Google is blocking messages "for this case" which is a little presumptuous. Google is presumably trying to save themselves a headache of handling errors for people who aren't prepared to do anything about it, the most common of which is spam. And the use of Message-ID in this application is documented at least as early as RFC2635.
> Additionally, Gmail adds its own message identifier on every message (g-msgid) because it knows that message ids can not be trusted to be unique.
Without knowing what Google does with the g-msgid header, you are making a mistake to assume it is equivalent to the Message-ID header just because it has a similar name. You have no reason to believe this is true.
> Finally just calling me ignorant is the cherry on top – please try to keep things civil on here.
I am sorry you are offended to not know things, but you do not know this thing, and your characterising my actions will make it very difficult for you to learn something new and not be so ignorant in the future.
Think hard exactly about what you want to happen here: Do you want Google (et al) to do something different? Do you want me to agree Google should? Who exactly are you trying to convince of what, and to what end?
I am trying to tell you how to interpret the documentation of the Internet, in this case to be successful sending email. That's it.
I am not likely to try and tell Google what to do in this case because of my own experiences running mail servers over the last 30 years, but listen: I am willing to be convinced. That's all I can do.
If it's something else, I'm sorry I just don't understand.
> If it's something else, I'm sorry I just don't understand.
I'm trying to explain to you that you're speaking very authoritatively about how things were done 30 years ago but things have changed since then, Gmail won't even send a message non-delivery notification to non-DKIM hosts.
So add a message id at the first stop, or hard ban the sender server version until they confirm. A midway point that involves a doom switch is not a good option.
As someone else who does System Engineering, when dealing with ancient protocols, "shall" is extremely difficult barrier to get over since there is always ancient stuff out there and there might be cases not to do it, esp if it's internal communication.
"SHOULD" is basically, if you control both sides of conversation, you can decide if it's required looking at your requirements. If you are talking between systems where you don't control both sides of conversation, you should do all "SHOULD" requirements with fail back in cases where other side won't understand you. If for reason you don't do "SHOULD" requirement, reason should be a blog article that people understand.
For example, "SHOULD" requirement would be "all deployable artifacts SHOULD be packaged in OCI container". There are cases where "SHOULD" doesn't work but those are well documented.
I’m doing some work with an email company at the moment. The company has been in the email space for decades. Holy moly email is just full of stuff like this. There is an insane amount of institutional knowledge about how email actually works - not what the specs say but what email servers need to actually do to process real emails and deal with real email clients and servers.
The rfcs try to keep up, but they’re missing a lot of details and all important context for why recommendations are as they are, and what you actually need to do to actually process real email you see in the wild. (And talk to the long tail of email software).
This conversation makes me think about cornering some of the engineers with a microphone. It’d be great to talk through the specs with them, to capture some of that missing commentary.
In a completely different field, navigating ships at sea, the Collision Regulations which define how people must conduct ships at sea, they use the words "Shall" and "May" to differentiate legal requirements and what may just be best practice. "Should" intuitively means something more like "May" to me
Note "the full implications must be understood and carefully weighed before choosing a different course". Gmail and the other big hosters have full-time spam teams who spend a lot of time weighing implications, so I assume the implications of this was weighed.
I feel like "SHOULD" and "SHOULD NOT" are redundant. You end up having to assume someone else treated them as a "MAY". If you control all the endpoints in a private implementation you can just deviate from the standard & not implement a MUST, it's your private implementation. There's thus no difference in public implementations between "SHOULD" and "MAY", and no difference in internal implementations between any of the words. They are therefore redundant, requirements are either mandatory or optional, there's no middle.
The original email RFC is also completely unaware of how bad spam is. Sure it might mention it but it's not really AWARE of the problem. The truth is, companies like Google, Microsoft and a few others have de-facto adjusted the minimum requirements for an email. Signing, anti-spam-agreements, etc.. are the true standard if you want an email to get from point a to b. (none of which are going to be REQUIRED in the RFC)
Given the amount of time that has passed, the big email companies have had many years to jointly update the RFC according to their experienced reality. They haven't done so and the way you describe how they effectively work together to adjust the minimum requirements (outside the RFC) is de-facto use of market power to the exclusion of other players.
If you read RFC 2119, you'll see that the definition of SHOULD includes the word must. Perhaps it's easier to understand if I rephrase it as: "If you understand the issue and its consequences, SHOULD means optional. If you don't, SHOULD is equivalent to MUST."
Message-ID has maybe half a dozen types of problem. If you know all of them, leaving it out is up to you.
I use matrix. Every chat room I use is unencrypted and all have at least one matrix.org user. I assume it can be encrypted but the usability is such that in practice it's cleartext.
As a counterexample: I use Matrix along with ~30-50 people, on a federated server, and every room is encrypted. After sufficiently stressing to people that they need to save their secure backup key, we've had few problems with encryption usability.
(3) With sufficient thrust, pigs fly just fine. However, this is
not necessarily a good idea. It is hard to be sure where they
are going to land, and it could be dangerous sitting under them
as they fly overhead.
If you pay someone 3x for work/service/product, x is the price and 2x is what you pay to encourage them to make/do the things instead of the people who could/would do it for x.
However, you're not paying 3x. I assume you're not really paying anything notably higher than x, right? So the encouragement is nearly zero.
When I had a job at a power plant near where I live, one of the engineers there pointed at the big chimneys and said (I translate:) "what comes out of those chimneys is cleaner than the air around us". They filter out everything except CO₂ with good efficiency and have big, heavy, expensive machinery and engineers onside 24/7 to monitor that it works.
A 5kg motor, optimised for portability, isn't going to be like that.
A matter of opinion. As I see it the several carcinogenic components of ICE exhausts are problems. Dying of cancer may be a small problem compared to an uninhabitable planet and I can see how one might consider it insignificant in comparison, but in my opinion, cancer is a real problem, including cancer caused by smog.
reply