What if you wanted to join a few different sources together, large ones at that, via query and have it filtered out, without reading all of it into memory first of course? Or.. running analytical queries, matches via reconciliation functions, basically doing useful stuff with different data sources available at once.
- Trustfall queries can in principle be compiled into SQL and executed on an analytical database. I haven't built this into Trustfall right now, but a prior project I built that used very similar syntax had this capability so I know it can be done without too much trouble.
- Trustfall's built-in query interpreter is lazily evaluated, and has very strict performance guarantees about what gets loaded into memory when and for how long. I've gone to some rather ridiculous lengths on this, and I look forward to writing it up in blog posts in the future.
- Edges in Trustfall's query graph can be parameterized: they can take an arbitrary number of type-checked arguments of a variety of types, so it should be possible to express just about any kind of join, API endpoint, reconciliation function, window function, etc.
- Whether a data source is large or small doesn't matter to Trustfall. So long as the data provider implements the Adapter interface, it can be plugged in. Whether the underlying data is read into memory all at once or progressively is the adapter implementation's business only -- Trustfall is extremely laissez-faire on this.
- Multiple data sources can be combined together, and queries written over all of them. This is something I'm working on right now, and hope to have more to share on in coming months.
It sounds like you might have a specific use case in mind, perhaps? Feel free to DM me on Twitter if you'd rather not discuss publicly.
