What do you mean by architecture here? If you mean the roles different nodes take and their topology, I actually doubt you can be decoupled from your architecture in a distributed system because they directly affect your design and capabilities.
You can move away from fully meshed for topologies but how does this choice affect node ups and node downs? Rebalancing can affect how you store data in the cluster.
How many connections should you have between nodes? A single connection makes the ordering guarantee straight-forward. Multiple connections is more performance but requires care if you need ordering and is more efficiently done along side your application.
And what about process placement? On which side of CAP do you want your registries to sit?
If you and your team is capable of "writing your own standard well documented distributed layer" upfront, then you are in a better position than most to take those decisions. But writing a distributed system is hard, so I will gladly start with a well-designed system, especially at the beginning of the project, when it may be unclear which patterns I will need as my application and business grow.
And most times, it will be good enough.
As far as OTP goes, you can plug your own discovery/topology mechanism as well as your own module for handling the connection between nodes. But, as mentioned in my previous reply, some of those issues may be better solved on the side, e.g. a different tcp/udp connection for data transmitting.
I did also mention that for fast time to market it is indeed a good idea to use already available tools. By the way writing a distribution layer is not that difficult. Many companies/startups with scalable back-ends that are not using Erlang/OTP have already done it. The point is scalable software requires knowledge and Erlang/OTP is not going to magically solve it. But it seems many fans are trying to promote it such that it is a magic tool that is going to make a shit software become a hyper scalable one. Just look at the comments below that how people have gone crazy.
Erlang (and by extension Elixir) definitely provides a set of tools which are good for building highly concurrent distributed systems. And your systems are likely to have few errors as well as being resilient, if you know what you are doing.
But indeed, you need to know your tools like every other part of computer science. Storing 9 billion elements in an array, doing linear search and then complaining your linear search is too slow and needs to run faster will be disastrous to your architecture. Likewise, assuming communication is free is equally disastrous.
The problem with building a distribution layer in another language is the effort involved. Most of the companies which pull that off and get stability in addition are usually large, multinational, and has ample amounts of engineering resources to throw at the effort required.
Consider for some reason (either technical or political), this company decides to migrate their web socket servers to Akka/Rust/Go/NodeJS. Integrating these new servers into their core cluster is going to be deadly painful. They rely heavily on Erlang/OTP internal clustering. This is not even considered as a challenge in distributed systems but still really painful to implement because of their design decision. This is what they did wrong. Clustering and message routing part of the application should be technology agnostic.
It isn't that hard. I have code which lets an OCaml program speak the Erlang distribution protocol. It can more or less stand-in for an Erlang node in the distribution layer. Writing the same for another language shouldn't be that hard: the protocol spoken is a simple prefix-protocol which can be parsed with LL(1) from the head plus an atom cache that is easy to maintain.
That said, good Erlang architectures definitely knows when you should fan out and use a common layer for distribution. At $work we have a Kafka cluster for some event handling simply because it is the right tool for the job, and because it makes it easier to interface other programs on top of it.
In my experience, the distribution features are best used as orchestration however. Your distribution layer is Erlang, and then you use other languages for "leaves" in the architecture. If the right library is present in e.g., Go, then by all means use it. Erlang easily speaks protobuf for instance.
That depends what you're going for. Moving web sockets elsewhere are pretty simple regardless of whether you're using Erlang/OTP or not because it's a frontal layer. If you do that all you are doing is creating another layer to hold connections that needs to relay to the rest of the application. The application behind it still has the responsibility to receive and relay everything from those web sockets, determine what message goes to who, etc.
You can move it out just the same as you can elsewhere, you just add an unnecessary layer of complexity and lose a lot of the capabilities already built into the language. It boils down to a simple question of, what would be the purpose behind that decision? What improvement would you get from deciding to move any of that elsewhere? Do you gain speed at the cost of complexity?
Clustering is a non-trivial problem. It can't happen naturally in any language that includes mutable data without explicit oversight and thus most of the needed tooling can't be built into the system and guaranteed to work everywhere. As soon as some part of the system goes from "send message, get response" to "send message, change variable referenced in memory on this machine, get response" you break your ability to naturally cluster.
That leads to a dependency in central relay points for things like setting up web sockets and then having code behind them communicate. The code behind them in whatever language is usually going to be talking to other central relay points like load balancers or pub sub databases rather than directly to other specific servers.
As soon as you have central relay points, they become something else that you have to monitor and scale even if they are very high volume.
There's nothing wrong with this approach and it will certainly work but if you want to avoid that and build a distributed system in another language, you've got to invest a lot in other areas.
> Moving web sockets elsewhere are pretty simple regardless of whether you're using Erlang/OTP or not because it's a frontal layer.
It is supposed to be easy. But they are not using language agnostic messaging. Instead they rely on Erlang distribution protocol. This is what makes it difficult to integrate.
> what would be the purpose behind that decision?
1. To achieve composability. I can replace one web socket server with a C++ implementation to see if I can handle 10 million connections per server. If I fucked it up return back to Erlang implementation otherwise fuck BEAM. I'm moving to native code. The beauty of it is that I can do it one step at a time. Even I can implement web socket layer in multiple languages and experiment all possible options in production and no one even notices it.
2. I don't have to pay for 20 Erlang guys who are mostly senior devs. Get 5 of them to write the core and the rest of the team NodeJS guys.
you're talking about undoing an architecture that lifts the benefits of an entire platform having done the work for you, in order to pick one that is probably easier to migrate to in the future.
You prefer doing all the work mostly from scratch, upfront and straight away, instead of having to maybe adapt a library later if you actually need to migrate.
You can move away from fully meshed for topologies but how does this choice affect node ups and node downs? Rebalancing can affect how you store data in the cluster.
How many connections should you have between nodes? A single connection makes the ordering guarantee straight-forward. Multiple connections is more performance but requires care if you need ordering and is more efficiently done along side your application.
And what about process placement? On which side of CAP do you want your registries to sit?
If you and your team is capable of "writing your own standard well documented distributed layer" upfront, then you are in a better position than most to take those decisions. But writing a distributed system is hard, so I will gladly start with a well-designed system, especially at the beginning of the project, when it may be unclear which patterns I will need as my application and business grow.
And most times, it will be good enough.
As far as OTP goes, you can plug your own discovery/topology mechanism as well as your own module for handling the connection between nodes. But, as mentioned in my previous reply, some of those issues may be better solved on the side, e.g. a different tcp/udp connection for data transmitting.