I’m calling it now, the adoption of AI agents into software development will be one of the most costly mistakes in the field’s history. Agents cannot program, and it’s taking longer and longer to realize that they can’t. They are a highly sophisticated statistical model designed to mimic the distribution of programming. The output is broken, but in a way that’s getting harder and harder to detect. Which is exactly what you’d expect from an increasingly accurate statistical model.
They are not the automated from 0 to 100 coders that some people claim them to be. But they are quite capable, definitely much more capable than what anyone could have imagined ten years ago. Given well defined problems they can excel at even relatively complex tasks. I pointed Claude at a latex file of a somewhat complicated nonparametric statistical estimate calculation to look for any mistakes and it was actually able to find some. I then pointed it at a code that replicates the calculations and it was also able to correctly identify some issues with the code. I think this is the way one should use LLMs, not let it loose on coding tasks. In the former way you won’t even be able to burn through your first tier account quota where as in the latter the LLM will likely end up getting in weird loops burning tokens like there is no tomorrow. Also this method of sane usage of LLMs is much more suitable for open local LLMs. I don’t think there is any doubt anymore that LLMs can be very useful tools, not just for doing stuff but learning it too. People should move past the stage of invalid criticisms like “they are just stochastic parrots” and move to more serious matters like environmental impact, greedy fucking CEOs pretending LLMs are replacements for humans, degredation of skills, getting lazy at checking AI code, ethics of capitalizing on collective human knowledge and the unsustainable AI bubble that tech companies are pushing for.
That’s not a criticism per se, it’s a description of how they work.
Sure, and at that level of accuracy it’s also a description of how humans work. I didn’t invent these words myself, I’m just stringing them together based on a stochastic process my brain was trained into.
Like LLMs, some of my speech is semi-random initialization (dada wawa googoo), some of that is mimicry (some of that is mimicry), some of that is reinforcement learning (downvotes incoming), and some of that is the output of a subprocess that uses the same systems prompted at the meta-level and without verbalization (maybe they won’t get the analogy between thinking and LLM scratchpads… how about I use this space to clarify).
Calling an LLM a stochastic parrot has the same social-emotional role as calling a human an animal. Yes, it is correct. But people can infer the connotation.
Humans are animals. LLMs randomly generate text based on the corpus they were trained on and the conversation so far, so stochastic parrot is an accurate description.
LLMs don’t learn. Humans do. LLMs generate text randomly using a massive matrix. Humans don’t; you lied. An LLM is incapable of lying because it has no understanding of truth. It just bullshits convincingly all the time. It’s very very good at it, but it’s all hallucinated for the LLM, true or false.
Expecting your random word generator to tell you truths is insane. The training measure is “sounds right” not “is right”. It passes if it sounds like the other discourse it read. Just like the confident drunk guy at the pub who thinks he knows everything passes of he convinces the other drunk guys at the pub.
Whereas humans learn at school and on the job and the training measure is “your teacher or supervisor approves”. LLMs were not trained on truth or accuracy. Trusting in them and treating them as equivalent to human intelligence, as you and a whole bunch of other folks do, is profoundly unsound, and soon the necessary price rises to pay for the processing costs (let alone the vast, vast, vast, vast, vast debts on the infrastructure) are going to make most slophouses which jettisoned their human talent go out of business. And very, very few people indeed will be sorry at that point.
Meanwhile LLM slop is shitting in github all day long, every day, and shitting on the internet, and it will eat it’s own shit and produce crappier shit.
Your analogies don’t change the truth, and that is that LLMs don’t know the difference between sounds correct and is correct any more than MAGA voters know the difference between sounds good to me and is good for me.
What do you mean LLMs don’t learn? How do you think they became capable of stringing a sentence together?
They don’t learn during a deployment, but neither do humans; humans only learn during sleep. The behaviors a human exhibits while “learning” in the moment are just stochastic parrot behaviors based on their immediate context window, if the human doesn’t sleep in time the event can slip out of their context window and they don’t learn despite having acted as if they do.
You seem to be very naive about human learning in general. What makes the “truth” of school lessons greater than the “truth” of an LLM’s curated dataset it is reinforcement learned on? Have you ever seen actual evidence that mitochondria exist, or are you just stochastically parroting your biology teacher?
I also oppose LLMs in almost all applications (live translation being an example of a good application). But please oppose it with arguments based in reality.
You’re confusing constructing the LLM, which is done with an actual AI (neural network) and a massive corpus of text (stolen from millions of humans in the greatest intellectual theft in history) and running the LLM, which is done with a random number generator and a massive matrix of probable next words.
They don’t learn. They don’t change. They’re as random next time as this time.
False and false. Soooo much pseudoscience.
Wrong again.
If that were true, most people would learn very badly first thing in the morning and get better and better later in the day. I think you’ll find that most school teachers would vehemently disagree with your nonsense conclusions.
Then again, perhaps by “doesn’t sleep in time” you mean stays up all night, then admittedly they might function less well cognitively but (a) we tend not to regularly torture humans that way and (b) you’re massively overstating the role of sleep in the learning process.
No, you seem to be very naive indeed, to extremes, about the intelligence and reliability of LLMs. When I ask them about general things that I know about, I tend to get the right answer about 60%-70% of the time. Why would I believe it when I didn’t know the answer. To trust an LLM to tell you the truth about stuff you aren’t checking when it clearly blags nonsense so frequently when you are is really really stupid.
Most teachers tend to consistently teach the content of the syllabus rather than randomise what they say to classes based on the preceding conversation. They reinforce and update their prior knowledge by also learning from the mark schemes of the tests and exams their students sit.
No. I trust my teachers. I am rational to do so. I don’t trust LLMs. You are irrational to do so.
You are utterly deluded and have bought the hype. You seem unable to distinguish between distinct things and are dismissing a large amount of evidence that your “just as good as a human” is a crap-spewing shit machine, no more honest than donald J trump, and with no less sharting.
Not true. Inference is done by providing the context to the pre-trained neutral network (technically a transformer network not your daddy’s old multilayer perceptron) to generate possible outcomes with logprobs that are then selected based on their likelihood. If it was just frequency-based RNG, they wouldn’t have any semantics in the responses and would sound more like traditional Markov chains (like when you mash a button on predictive text and it spits out correct but meaningless gibberish).
If it were just selecting random words from a matrix of probabilities without the network and attentions, it would also be waaay faster and easier to run on a potato.
The stuff about human learning also isn’t quite right. There are different types of “learning” and different kinds of memory.
Sleep is generally understood physiologically to be required to formulate long term memory (eg. as described in this paper).
The previous commentator was analogising human short and mid-term memory with LLM context windows (also things like vector databases etc.) and long term memory with retraining/merging/fine tuning of LLMs. It’s not totally the same but the analogy is accurate. Brain behaviour is a big influence and inspiration on how machine learning techniques are designed.
Human memory is also notoriously inaccurate and unreliable and tasks done by humans often needs to be double checked and externally verified.
This isn’t to say LLMs are trustworthy or reliable. They are not. More that humans think much more highly of themselves than is really warranted.
I repeat, the LLM is not doing machine learning while users are using it.
We agree here.
And we agree here too, but to trust an LLM to tell you the truth on your question that you don’t know the answer is like trusting some random drunk at the pub, because you don’t know whether the answer is from an LLM hallucination, a random lie/error on reddit or an expert’s contribution to wikipedia.
And to trust an LLM when there’s a trained programmer or professional journalist is stupid. Sure, an LLM might even sometimes write as good or better code than an intern, but again, the LLM is not learning from its mistakes as you correct it. The intern gradually becomes an expert. The LLM does not. Paying interns is an investment in future programmers, who get more expensive the more experienced they are.
The LLM is currently cheaper than the intern, but LLM pricing needs to go up by a factor of about ten to cover running costs let alone pay off the vastly more immense debts of buying all that hardware.
Like I said before, humans sleep every night, with rare exceptions. LLMs do not get retrained every night. The human brain adapts to feedback loops during everyday interactions, not just overnight. It’s a silly analogy and this is a silly point to defend.
There are plenty of textbooks that say that volatile running RAM is like short term memory and hard disks and SSDs are like long term memory, but it would be silly to reverse the analogy as you are doing and claim that sleep is pressing the save button on the day’s learning, or that this makes your word processor the same as your human intelligence because, and this is the point you’ve been trying to argue around and against, they’re doing fundamentally different things, and telling me one was inspired by the other doesn’t change that.
If you truly believe that the LLM is learning like a human or intelligent like a human, you are confusing analogies for reality.
This is a small terminology misconception. The LLM is not doing “training” during inference. It’s still a “machine learning” system.
In terms of learning/retaining information in the short/mid term while the user is using it, as the context grows, it retains that information during the current session. In a lot of systems, sections of that context are then summarised and stored, indexed by a vector, to be retrieved into future contexts that have similar semantics. That’s why some systems seem to be able to “remember” things from previous “conversations”. Your message is vectorised and then that vector used to look up similar past interactions. The model isn’t fine tuning on that, so it’s not “long term” memory, but the model can take it into account for future interactions.
AI companies do then use that (and full conversation histories) to regularly fine tune the models, as well as train new ones. It might not be fresh trained every day but certainly more often than you might think.
They’re a little more reliable than that and are getting significantly more capable at an alarming rate. We absolutely agree that they shouldn’t be trusted and are not very accurate (nor should most humans be trusted or are accurate) but I also think it’s dangerous to underestimate them.