The open-source infra we use allows us to have a highly-available, scale-able system.
Kafka - buffers data ingest so that we never lose recordings.
Redis - acts as a cache, optimized for write-heavy processing in our app.
Postgres - stores metadata for the data we record.
ClickHouse - stores logs and other time-series data that need to be aggregated.
We're looking to simplify the system and replacing InfluxDB + OpenSearch in the near future with ClickHouse as a more performance and cost-effective solution that has proven simpler to maintain.
What about keeping just ClickHouse and redis? Wouldn't that be enough?
At least for the hobby that don't need to scale to millions? (Representing 99% of the use cases)
Kafka - buffers data ingest so that we never lose recordings. Redis - acts as a cache, optimized for write-heavy processing in our app. Postgres - stores metadata for the data we record. ClickHouse - stores logs and other time-series data that need to be aggregated.
We're looking to simplify the system and replacing InfluxDB + OpenSearch in the near future with ClickHouse as a more performance and cost-effective solution that has proven simpler to maintain.