Until you want to be able to sort and filter and do all of the other things that most CRUD applications do.
Once you're there, if you're doing SQL directly, now you're manipulating strings in your code, instead of being able to build some kind of object that then generates a query for you.
Not at all. The only cumbersome thing about SQL, that is common enough and unpleasant to always spell out by hand are basic INSERT and UPDATE commands.
I use a set of 3-4 functions that I reuse pretty much in all my programs for this. It's no ORM, as it doesn't map classes to tables/data in the database. It's just a shortcut to generate and execute INSERT/UPDATE commands on arbitrary tables with arbitrary columns.
Also it has zero bugs, because you can hardly create bugs in something so simple.
Query builders and ORMs often go hand in hand in my experience, as the ORM will expose the ability to perform custom querying capabilities around the specific object you're working with; however, you're correct in that I am more considering query builders than ORMs directly in this instance.
Once you're there, if you're doing SQL directly, now you're manipulating strings in your code, instead of being able to build some kind of object that then generates a query for you.