Going to give the unpopular answer. Queues, Streams and Pub/Sub are poorly understood concepts by most engineers. They don't know when they need them, don't know how to use them properly and choose to use them for the wrong things. I still work with all of the above (SQS/SNS/RabbitMQ/Kafka/Google Pub/Sub).
I work at a company that only hires the best and brightest engineers from the top 3-4 schools in North America and for almost every engineer here this is their first job.
My engineers have done crazy things like:
- Try to queue up tens of thousands of 100mb messages in RabbitMQ instantaneously and wonder why it blows up.
- Send significantly oversized messages in RabbitMQ in general despite all of the warnings saying not to do this
- Start new projects in 2024 on the latest RabbitMQ version and try to use classic queues
- Creating quorum queues without replication policies or doing literally anything to make them HA.
- Expose clusters on the internet with the admin user being guest/guest.
- The most senior architect in the org declared a new architecture pattern, held an organization-wide meeting and demo to extol the new virtues/pattern of ... sticking messages into a queue and then creating a backchannel so that a second consumer could process those queued messages on demand, out of order (and making it no longer a queue). And nobody except me said "why are you putting messages that you need to process out of order into a queue?"...and the 'pattern' caught on!
- Use Kafka as a basic message queue
- Send data from a central datacenter to globally distributed datacenters with a global lock on the object and all operations on it until each target DC confirms it has received the updated object. Insist that this process is asynchronous, because the data was sent with AJAX requests.
As it turns out, people don't really need to do all that great of a job and we still get by. So tools get misused, overused and underused.
In the places where it's being used well, you probably just don't hear about it.
Edit: I forgot to list something significant. There's over 30 microservices in our org to every 1 engineer. Please kill me. I would literally rather Kurt Cobain myself than work at another organization that has thousands of microservices in a gigantic monorepo.
To second this theory with some real-world data. A few startups I worked at a NY scala shop that used tons of Akka (event-driven queuing scala thing). Why? Because a manager at his prior job "saved the company" when "everything was slow" by doing this, so mandated it at the new job?
What were doing that required queueing? Not much, we showed people's 401ks on a website, let them adjust their asset mix, and sent out hundreds of emails per day. As you would expect, people almost never log into their 401k website.
A year or so after working there I realized our servers had been misconfigured all along and basically had 0 concurrency for web-requests (and we hadn't noticed because 2 production servers had always served all the traffic we needed). Eventually we just ripped out the Akka because it was unnecessary and added unnecessary complexity.
In the last month this company raised another funding round with a cash-out option, apparently their value has gone up and they are still doing well!
I think I'm actually somewhat familiar with your story.
There's something about "Java/Scala", "New York startup" and cargo-culted behaviors. I'm sure this happens elsewhere too in other circles, but I've both heard of and read about what you're referring to before.
1. There doesn't seem to be a design review process before people start implementing these things. Devs should make a design document, host a meeting where everyone reads it, and have a healthy debate before implementing stuff. If you have bright people but not-so-bright outcomes, it's because there is no avenue for those who know their shit to speak up and influence things.
2. I will always rather hire a 2-5 YOE dev from a no-name school over a new grad from a top 5 school. The amount that software engineers learn and grow in the first 5 years of their career is immense and possibly more than the rest of their career combined.
Intelligence != experience. We are all aware of the very bright junior engineer that produces huge code volume and creates massive technical debt. It takes experience to go with the boring, simple approach.
No, truly, I feel you. The Waterloo kids are literally cut from a different cloth though I'll take one of them to ten of anyone else. I feel really guilty about grabbing young grads with all that potential and putting them through this...
Grads aren't expected to know anything. It's all the early ex-google hiring that fucked us.
Not sure how big your company is and how large your influence. But it sounds to me like your new hires would benefit from more guidance and oversight. Those mistakes shouldn’t make it production unless nobody gives a damn.
I keep the platform systems running, so they aren't my responsibility, but as their profession used to be mine I think the way they do things is more than a bit mad. Had COVID not upended absolutely everything, I would have moved on years ago. I believe I'll be here until the industry winds change significantly.
I do get to slam my foot down on some things from time to time -- usually for compliance reasons. Like now when people tell me they need credentials for things so they can run their code in dev on their laptops, I tell them no and that they can use mocks for their testing like a professional.
Funnily the junior people expect and have no problem with this -- it's the seasoned folks who chafe and complain.
> I work at a company that only hires the best and brightest engineers from the top 3-4 schools in North America and for almost every engineer here this is their first job.
Are you making this up to make your claim sound more extravagant? Even Citadel is not that picky so unless you work at OpenAI/Anthropic I'm calling nonsense.
There's nothing about the company or the product that requires it but we in fact are that picky when it comes to hiring new grads. Only new grads will work here at the pay we're offering and for all the other reasons mentioned though, so there you go.
To be fair, thousands of microservices in a monorepo sounds better than thousands of microservices, each with its own repository (but sub-repo’ing common libraries so everything jams up in everything else).
When your monorepo grows into the dozens upon dozens of gigabytes in size, the effort to manage it grows exponentially and the tooling available to you to interact with it shrinks exponentially. For example, your options for CI/CD systems and code review tools to use are extremely limited.
Because JS is single threaded, and an average programmer today is too dumb to learn any language beyond JS, you must build everything into "microservices".
Raw intelligence is of limited help when working in an area that requires lots of highly depreciating domain specific knowledge.
Relatively few graduates know their way around the Snowflake API, or the art of making an electron app not perform terribly. Even sending an email on the modern internet can require a lot of intuition and hidden knowledge.
> There's over 30 microservices in our org to every 1 engineer
I wonder if this a factor in making onboarding of new hires difficult?
It's hard to make code changes without understanding the broader scope.
For example, a developer asked to change the payments processor from the Stripe API to the Adyen API needs to know about:
1) credential scoping across microservices and test environments
2) impacts on billing, tax management and customer support
3) the organisation's approach to external dependencies (e.g do they use some kind of internal registry)
Microservices don't inherently make this kind of change more difficult. A well isolated payments microservice might even help. But having hundreds or thousands of microservices spread across several code repos makes it quite difficult to figure out how your change will affect other services.
I acknowledge I've picked an extreme example that wouldn't be asked of a new hire. However, smaller versions of the same challenges will happen in the smaller API upgrades & migrations that make up a big part of what the average CRUD developer does.
My point is with microservices you can, in theory, write all that stuff in a spec, and hand it to some new kid on the block. They then don't need to know about billing or what not, you've taken care of that in the spec.
Not saying it's a good way, but it's one way to end up with an inordinate amount of microservices per dev.
> > There's over 30 microservices in our org to every 1 engineer
> I wonder if this a factor in making onboarding of new hires difficult?
Not so much that but a culture of rapidly reconfiguring the whole organization around new product work every three sprints and no ownership/responsibility for maintaining prior work.
« Use kafka as a basic message queue ». Since i’m guilty of that (i use kafka as the backbone for pretty much any service 2 service communication, under a « job » api) i wonder why you think that’s wrong.
This. And as-mentioned, RabbitMQ is already part of our platform. Using Kafka here is both extremely wasteful and unnecessarily complicates the product.
I work at a company that only hires the best and brightest engineers from the top 3-4 schools in North America and for almost every engineer here this is their first job.
My engineers have done crazy things like:
- Try to queue up tens of thousands of 100mb messages in RabbitMQ instantaneously and wonder why it blows up.
- Send significantly oversized messages in RabbitMQ in general despite all of the warnings saying not to do this
- Start new projects in 2024 on the latest RabbitMQ version and try to use classic queues
- Creating quorum queues without replication policies or doing literally anything to make them HA.
- Expose clusters on the internet with the admin user being guest/guest.
- The most senior architect in the org declared a new architecture pattern, held an organization-wide meeting and demo to extol the new virtues/pattern of ... sticking messages into a queue and then creating a backchannel so that a second consumer could process those queued messages on demand, out of order (and making it no longer a queue). And nobody except me said "why are you putting messages that you need to process out of order into a queue?"...and the 'pattern' caught on!
- Use Kafka as a basic message queue
- Send data from a central datacenter to globally distributed datacenters with a global lock on the object and all operations on it until each target DC confirms it has received the updated object. Insist that this process is asynchronous, because the data was sent with AJAX requests.
As it turns out, people don't really need to do all that great of a job and we still get by. So tools get misused, overused and underused.
In the places where it's being used well, you probably just don't hear about it.
Edit: I forgot to list something significant. There's over 30 microservices in our org to every 1 engineer. Please kill me. I would literally rather Kurt Cobain myself than work at another organization that has thousands of microservices in a gigantic monorepo.