I've been curious about this project for a while...
If you squint a bit it's sort of like an Airflow that can run on AWS Step Functions.
Step Functions sort of gives you fully serverless orchestration, which feels like a thing that should exist. But the process for authoring them is very cumbersome - they are crying out for a nice language level library i.e. for Python something that creates steps via decorator syntax.
And it looks like Metaflow basically provides that (as well as for other backends).
The main thing holding me back is lack of ecosystem. A big chunk of what I want to run on an orchestrator are things like dbt and dlt jobs, both of which have strong integrations for both Airflow and Dagster. Whereas Metaflow feels like not really on the radar, not widely used.
Possibly I have got the wrong end of the stick a bit because Metaflow also provides an Airflow backend, which I sort of wonder in that case why bother with Metaflow?
Metaflow was started to address the needs of ML/AI projects whereas Airflow and Dagster started in data engineering.
Consequently, a major part of Metaflow focuses on facilitating easy and efficient access to (large scale) compute - including dependency management - and local experimentation, which is out of scope for Airflow and Dagster.
Metaflow has basic support for dbt and companies use it increasingly to power data engineering as AI is eating the world, but if you just need an orchestrator for ETL pipelines, Dagster is a great choice
If you are curious to hear how companies navigate the question of Airflow vs Metaflow, see e.g this recent talk by Flexport https://youtu.be/e92eXfvaxU0
A while ago I saw a promising Clojure project stepwise [0] which sounds pretty close to what you're describing. It not only allows you to define steps in code, but also implements cool stuff like ability to write conditions, error statuses and resources in a much-less verbose EDN instead of JSON. It also supports code reloading and offloading large payloads to S3.
A few years back, the Step Functions team was soliciting input, and the Python thing was something that came up as a suggestion. It's hard, yes, but it should be possible to "Starlark" this and tell users that if you stick to this syntax, you can write Python and compile it down to native StepFunction syntax.
Having said that, they have slightly improved the StepFunctions by adopting JSONata syntax.
I don't think it should need Starlark or a restricted syntax.
You just want some Python code that builds up a representation of the state machine, e.g. via decorating functions the same way that Celery, Dask, Airflow, Dagster et al have done for years.
Then you have some other command to take that representation and generate the actual Step Functions JSON from it (and then deploy it etc).
But the missing piece is that those other tools also explicitly give you a Python execution environment, so the function you're decorating is usually the 'task' function you want to run remotely.
Whereas Step Functions doesn't provide compute itself, it mostly just gives you a way to execute AWS API calls. But the non control flow tasks in my Step Functions end up mostly being Lambda invoke steps to run my Python code.
I'm currently authoring Step Functions via CDK. It is clunky AF.
What it needs is some moderately opinionated layer on top.
Someone at AWS did have a bit of an attempt here: https://aws-step-functions-data-science-sdk.readthedocs.io/e... but I'd really like to see something that goes further and smooths away a lot of the finickety JSON input arg/response wrangling. Also the local testing story (for Step Functions generally) is pretty meh.
If you are ok with executing your SFN steps on AWS Batch, Metaflow should do the job well. It's pretty inhuman to interact with SFN directly.
One feature that's in our roadmap is the ability to define DAG fully programmatically, maybe through configs, so you will be able to have a custom representation -> SFN JSON, just using Metaflow as a compiler
If you squint a bit it's sort of like an Airflow that can run on AWS Step Functions.
Step Functions sort of gives you fully serverless orchestration, which feels like a thing that should exist. But the process for authoring them is very cumbersome - they are crying out for a nice language level library i.e. for Python something that creates steps via decorator syntax.
And it looks like Metaflow basically provides that (as well as for other backends).
The main thing holding me back is lack of ecosystem. A big chunk of what I want to run on an orchestrator are things like dbt and dlt jobs, both of which have strong integrations for both Airflow and Dagster. Whereas Metaflow feels like not really on the radar, not widely used.
Possibly I have got the wrong end of the stick a bit because Metaflow also provides an Airflow backend, which I sort of wonder in that case why bother with Metaflow?