Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

> If everything’s in one log, there’s nothing to coordinate #

On the contrary. Everything becomes coordinated.

The entire "log" becomes a giant ass mutex lock. Good luck scaling it.



There is nothing to coordinate for the application, because, yes, the log coordinates everything. But not globally, on the level of a single event handler execution, or a single key.

That has been proven to scale well - the way we implement that in Restate is classical shared nothing physical partitioning, with indexing on a key granularity.

So nothing like a shared mutex unless you want to access the same key, which otherwise your database synchronizes, if you want any reasonable level of consistency.


I think the author is motte-and-baileying between:

Literally one log - which does indeed reduce your coordination headache, but is susceptible to your "giant ass mutex" comment, and

One log per ... - which brings the coordination problems right back into existence.


I can see where some of that could be written more clearly. To elaborate:

- We mean using one log across different concerns like state a, communication with b, lock c. Often that is in the scope of a single entity (payment, user, session, etc.) and thus the scope for the one log is still small, and it reduces coordination headache for coordinating between the systems. You would have a lot of independent logs still, for separate payments.

- It does _not_ mean that one should share the same log (and partition) for all the entities in your app, like necessarily funneling all users, payments, etc. through the same log. What would be needed if you try and do some multi-key-distributed transaction processing. That goes actually beyond the proposal here, and has some benefits of its own, but have a hard time scaling.


> thus the scope for the one log is still small, and it reduces coordination headache for coordinating between the systems.

This does not follow.

Neither the dev (nor the product seller) is in control of what real-world effects have a causal relationship.

If my actions can be independently processed from your actions, then we don't need to coordinate, and micrologs will perform faster than a monolog.

If my actions cannot be independently processed from your actions, then we do need to coordinate, unless we use a monolog, which kills performance.

Either way, we don't get to choose the level of independence. If the actions are dependent, we either coordinate or or mess up causality.


Exactly what I was thinking. Now what's the best mutex system we've built? An SQL database.


You can use something like DynamoDb with partition per interaction. That would scale great.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: