It took me a minute to grok what this was for, but I think I like it
It doesn't really replace any of the tooling we use to wrangle data at scale (like prefect or dagster or temporal) but as a local library it seems to be excellent, I think what confused me most was the comparison to dbt.
I like the from_* utils and the magic of the Column class operator overloading and how chains can be used as datasets. Love how easy checkpointing is too. Will give it a go
Yes, it's not meant to replace data engineering tools like Prefect or Temporal. Instead, it serves as a transformation engine and ad-hoc analytics for images/video/text data. It's pretty much DBT use case for text and images in S3/GCS, though every analogy has its limits.
It doesn't really replace any of the tooling we use to wrangle data at scale (like prefect or dagster or temporal) but as a local library it seems to be excellent, I think what confused me most was the comparison to dbt.
I like the from_* utils and the magic of the Column class operator overloading and how chains can be used as datasets. Love how easy checkpointing is too. Will give it a go