wiring up an RNG to your CLI has fairly obvious risks, the root of the problem is ~everyone's treating GenAI as if it's AGI - the rest is popcorn fodder.
This is actually a fun way to describe it. I've being saying for a little while now that using AI for things where there's consequences if it fails is a bad idea, but it never occurred to me that this is basically the same concept as some rules in tabletop RPGs.
In D&D 3.5 edition, there was a rule about how you could "take 20" on a d20 roll to get a guaranteed 20 by taking 20 times as long in-game to perform the action, but only if it was a check that didn't have consequences for failure, since it was essentially a shortcut to skip the RNG of rolling until you rolled a 20. Maybe framing it like this might make sense to people a bit more, but if not, I'll at least have more fun making my case.
It seems closer to "roll two or three successive 1s on a D100 and have your LLM hooked directly into your production systems and have your LLM user have DELETE permissions" and probably 1 or 2 other things I'm forgetting.
It pulled an api key from an unrelated file. It wasn’t given delete permission, it found it.
Not picking on you specifically, but in general the comments here have me wondering if AI has stolen our basic reading comprehension, or if we were always this bad.
Anyway, take “LLM user had delete permission” off your list and add “deleting the production db also deletes all the backups” to the list.
A lot of comments in here are obviously describing Americans, as if that's representative of some global human reality, without acknowledging the elephant in the room is a cultural problem.
Americans in particular tend to have a highly entitled and confused "time is money" view governing their existence that enables them to do nothing except when paid by employers, which obviously results in doing absolutely nothing in retirement.
I worked in one of those independent computer stores in the 90s, assembling white box PCs in a dimly lit back room, and systematically removing drivers on early Win95 machines until they'd stop crashing to identify which one was buggy.
PCs were so dynamic at the time, half my paychecks were spent on discounted upgrades before I ever saw the paper. EDO ram? sign me up. 512K of pipelined burst L2 cache? yes please. HX chipset? of course. Dual socket pentium pros? I need a raise.
Similar background re: PC building here, working at a shop that built PCs in the late 90s. I remember seeing boards with these new-fangled USB ports, DIMM memory, Pentium II, the first 3D accelerators, etc. It was a fun time. I got in to the industry right at the end of AT-style boards and power supplies and mostly missed having to deal with that stuff (other than in my personal life, where I still had old stuff).
reply