They don't understand esoteric areas of computer science very well at all.
I had a mistake in which a large back-up file deletion event happened during a robocopy. 600gb of files got 'deleted' (file headers toast etc). Trying to get the LLMs to understand the hunt parameters, what to focus on, what not to focus on - none of them could reasonably come close to doing file content recovery properly. I needed to build a custom solution because the available industry options couldn't do what was required and the LLMs were useless for that (including the latest versions of Claude, Gemini and GPT). They just went around in circles, capped by their apparently weak knowledge of file recovery as a field. That is, creativity was their limitation.
I had a mistake in which a large back-up file deletion event happened during a robocopy. 600gb of files got 'deleted' (file headers toast etc). Trying to get the LLMs to understand the hunt parameters, what to focus on, what not to focus on - none of them could reasonably come close to doing file content recovery properly. I needed to build a custom solution because the available industry options couldn't do what was required and the LLMs were useless for that (including the latest versions of Claude, Gemini and GPT). They just went around in circles, capped by their apparently weak knowledge of file recovery as a field. That is, creativity was their limitation.