This. Most people defer the solving of hard problems to when they write the code. This is wrong, and too late to be effective. In one way, using agents to write code forces the thinking to occur closer to the right level - not at the code level - but in another way, if the thinking isn’t done or done correctly, the agent can’t help.
I can spend all the time I want inside my ivory tower, hatching out plans and architecture, but the moment I start hammering letters in the IDE my watertight plan suddenly looks like Swiss cheese: constraints and edge cases that weren't accounted for during planning, flows that turn out to be unfeasible without a clunky implementation, etc...
That's why Writing code has become my favorite method of planning. The code IS the spec, and English is woefully insufficient when it comes to precision.
This makes Agentic workflows even worse because you'll only your architectural flaws much much later down the process.
I also think this is why AI works okay-ish on tiny new greenfield webapps and absolutely doesn't on large legacy software.
You can't accurately plan every little detail in an existing codebase, because you'll only find out about all the edge cases and side effects when trying to work in it.
So, sure, you can plan what your feature is supposed to do, but your plan of how to do that will change the minute you start working in the codebase.
Yeah, I think this is the fundamental thing I'm trying to get at.
If you think through a problem as you're writing the code for it, you're going to end up the wrong creek because you'll have been furiously head down rowing the entire time, paying attention to whatever local problem you were solving or whatever piece of syntax or library trivia or compiler satisfaction game you were doing instead of the bigger picture.
Obviously, before starting writing, you could sit down and write a software design document that worked out the architecture, the algorithms, the domain model, the concurrency, the data flow, the goals, the steps to achieve it and so on; but the problem with doing that without an agent is then it becomes boring. You've basically laid out a plan ahead of time and now you've just got to execute on the plan, which means (even though you might even fairly often revise the plan as you learn unknown unknowns or iterate on the design) that you've kind of sucked all the fun and discovery out of the code rights process. And it sort of means that you've essentially implemented the whole thing twice.
Meanwhile, with a coding agent, you can spend all the time you like building up that initial software design document, or specification, and then you can have it implement that. Basically, you can spend all the time in your hammock thinking through things and looking ahead, but then have that immediately directly translated into pull requests you can accept or iterate on instead of then having to do an intermediate step that repeats the effort of the hammock time.
Crucially, this specification or design document doesn't have to remain static. As you would discover problems or limitations or unknown unknowns, you can revise it and then keep executing on it, meaning it's a living documentation of your overall architecture and goals as they change. This means that you can really stay thinking about the high level instead of getting sucked into the low level. Coding agents also make it much easier to send something off to vibe out a prototype or explore the code base of a library or existing project in detail to figure out the feasibility of some idea, meaning that the parts that traditionally would have been a lot of effort to verify that what your planning makes sense have a much lower activation energy. so you're more likely to actually try things out in the process of building a spec
I believe programming languages are the better language for planning architecture, the algorithms, the domain model, etc... compared to English.
The way I develop mirrors the process of creating said design document. I start with a high level overview, define what Entities the program should represent, define their attributes, etc... only now I'm using a more specific language than English. By creating a class or a TS interface with some code documentation I can use my IDEs capabilities to discover connections between entities.
I can then give the code to an LLM to produce a technical document for managers or something. It'll be a throwaway document because such documents are rarely used for actual decision making.
> Obviously, before starting writing, you could sit down and write a software design document that worked out the architecture, the algorithms, the domain model, the concurrency, the data flow, the goals, the steps to achieve it and so on;
I do this with code, and the IDE is much better than MS Word or whatevah at detecting my logical inconsistencies.
The problem is that you actually can't really model or describe a lot of the things that I do with my specifications using code without just ending up fully writing the low level code. Most languages don't have a type system that actually lets you describe the logic and desired behavior of various parts of the system and which functions should call which other functions and what your concurrency model is and so on without just writing the specific code that does it; in fact, I think the only languages that would allow you to do something like that would have to be like dependently typed languages or languages adjacent to formal methods. This is literally what the point of pseudocode and architecture graphs and so on are for.
Except what if you don’t really grok those ffmpeg flags and the LLM tells you something wrong - how will you know? Or more common, send you down a re-encode rabbit hole when you just needed a simple clipping off the end?
I also have a 10 year old laptop with no TPM 2.0 module. It was pretty high end for the time too (Dell XPS). I haven't needed it for much in recent years, but it still runs perfectly fine and I'm happy to continue using it if the need arises again. Sounds like I'll have to switch that over to Linux like I have all my other PCs.
Practical question: when getting the AI to teach you something, eg how attention can be focused in LLMs, how do you know it’s teaching you correct theory? Can I use a metric of internal consistency, repeatedly querying it and other models with a summary of my understanding? What do you all do?
Google for non-AI sources. Ask several models to get a wider range of opinions. Apply one’s own reasoning capabilities where applicable. Remain skeptical in the absence of substantive evidence.
Basically, do what you did before LLMs existed, and treat LLM output like you would have a random anonymous blog post you found.
In that case, LLMs must be written off as very knowledgeable crackpots because of their tendency to make things up. That's how we would treat a scientist who's caught making things up.
Conspicuously missing is a direct mention of AI tools. Is MIT, like others, side-stepping the use of AI by students to (help them) complete homework assignments and projects?
A question. If you think AI use by students to "bypass homework" is anything remotely approaching a problem, then I must ask you how you felt/feel about:
- University being cost prohibitive to 90 percent of all humans as financial driven institutions, not performance.
- Before AI, 20 + years of google data indexing/searches fueling academia
- study groups before that allowing group completion (or, cheating, in your view)
- The textbook that costs 500 dollars, or the textbook software from pearson that costs 500, that has the homework answers.
I think it's a silly posit that students using AI is...anything to even think about. I use it at my fortune 500 job every day, and have learned about my field's practical day-to-day from it than any textbook, homework assignment, practical etc.
reply