I’m calling it now, the adoption of AI agents into software development will be one of the most costly mistakes in the field’s history. Agents cannot program, and it’s taking longer and longer to realize that they can’t. They are a highly sophisticated statistical model designed to mimic the distribution of programming. The output is broken, but in a way that’s getting harder and harder to detect. Which is exactly what you’d expect from an increasingly accurate statistical model.
This is a small terminology misconception. The LLM is not doing “training” during inference. It’s still a “machine learning” system.
In terms of learning/retaining information in the short/mid term while the user is using it, as the context grows, it retains that information during the current session. In a lot of systems, sections of that context are then summarised and stored, indexed by a vector, to be retrieved into future contexts that have similar semantics. That’s why some systems seem to be able to “remember” things from previous “conversations”. Your message is vectorised and then that vector used to look up similar past interactions. The model isn’t fine tuning on that, so it’s not “long term” memory, but the model can take it into account for future interactions.
AI companies do then use that (and full conversation histories) to regularly fine tune the models, as well as train new ones. It might not be fresh trained every day but certainly more often than you might think.
They’re a little more reliable than that and are getting significantly more capable at an alarming rate. We absolutely agree that they shouldn’t be trusted and are not very accurate (nor should most humans be trusted or are accurate) but I also think it’s dangerous to underestimate them.
Depends which drink guy at the pub you randomly pick. The attribute that they share with the drink guy at the pub is their reluctance to admit that they don’t know or have no expertise or can’t help you. Clever and experienced people know where their expertise ends and express self doubt when appropriate. LLMs don’t. They can’t. They’re making literally everything they say up. It’s probably right, but they are the script kiddie of conversationalists.
It’s dangerous to underestimate their ability to sound good enough to convince executives to fire humans. It’s dangerous to underestimate the scale of substitution of plausibility over knowledge that will only accelerate with further adoption. It’s dangerous to assume that the interactions that middle and senior management have with staff that do actual work cannot be replicated already with a suitably trained LLM.
Those slashes are doing a lot of work in that sentence!
Humans learn by generalising from examples. Humans learn when you ask them well-designed questions. Humans learn by practising skills repeatedly. Holland park from their mistakes. Humans learn because they are in a constant state of feedback loop. Humans learn by watching other people. Humans learn by experimentation. Humans learn through playing with new things. Humans learn by talking to each other. Humans learn by sitting and thinking things through. Humans learn through thought experiments. Humans learn through seeing and hearing and reading more quickly than doing any of them alone. Humans learn by explaining things to other people, crystallising their experience into verbal solidity. Humans learn through discussion. Humans learn by learning who to trust and how to weight different input by source. Humans learn by learning how to learn more effectively.
LLMs do none of any of those things.
Machine learning is a very, very narrow form of “learning” and you’re conflating the use of a neural network with actual learning, which you then compound by confusing the resulting LLM with the neural network that was used in its creation.
Pulling the wool over people’s eyes about what an LLM is is at least as harmful as underestimating the ability of AI to be so plausible as to disrupt absolutely everything about how money moves around society.