Technology

AI-generated code contains more bugs and errors than human output

Yeah. No shit. I used an LLM's "help" to make fin.

It got me reading and debugging more than 10 times the [bad] code, per day, than I had in the entire prior 10 years of using fish. [And reading the documentation way more too, learning a lot.]

... more bugs and errors than human output

However, it's not necessarily a bad thing, with AI improving efficiency across the initial stages of code generation.

Oh but it's so effortless. HA! Debugging takes a lot more effort. And then still have to just re-write it all yourself any way.

Still, it's a good learning experience.

Dear AI,

Thanks for being so shit.

Taught me a lot.

No shit

Almost as if it was made to simulate human output but without the ability to scrutinize itself.

It's like having a lightning-fast junior developer at your disposal. If you're vague, he'll go on shitty side-quests. If you overspecify he'll get overwhelmed. You need to break down tasks into manageable chunks. You'll need to ask follow-up questions about every corner case.

A real junior developer will have improved a lot in a year. Your AI agent won't have improved.

ChatGPT is great at generating a one line example use of a function. I would never trust its output any further than that.

Microsoft: Let's have it rebuild our most well known product from the ground up!

Yeah no shit

Did they compare it to the code of that outsourced company that provided the lowest bid? My company hasn’t used AI to write code yet. They outcourse/offshore. The code is held together with hopes and dreams. They remove features that exist, only to have to release a hot fix to add it back. I wish I was making that up.

this is expected, isn't it? You shit fart code from your ass, doing it as fast as you can, and then whoever buys out the company has to rewrite it. or they fire everyone to increase the theoretical margins and sell it again immediately

Oh, so my sceptical, uneducated guesses about AI are mostly spot on.

And then it takes human coders way longer to figure out what’s wrong to fix than it would if they just wrote it themselves.

A computer is a machine that makes human errors at the speed of electricity.

No shit. 

I actually believed somebody when they told me it was great at writing code, and asked it to write me the code for a very simple lua mod. It’s made several errors and ended up wasting my time because I had to rewrite it.

Although I don't doubt the results… can we have a source for all the numbers presented in this article?

It feels AI generated itself, there's just a mishmash of data with no link to where that data comes from.

There has to be a source, since the author mentions:

So although the study does highlight some of AI's flaws [...] new data from CodeRabbit has claimed

CodeRabbit is an AI code reviewing business. I have zero trust in anything they say on this topic.

Then we get to see who the author is:

Craig’s specific interests lie in technology that is designed to better our lives, including AI and ML, productivity aids, and smart fitness. He is also passionate about cars

Has anyone actually bothered clicking the link and reading past the headline?

Can you please not share / upvote / get ragebaited by dogshit content like this?

Water makes things wetter than fire does.

I've been coding for a while. I did an honest eager attempt at making a real functioning thing with all code written by AI. A breakout clone using SDL2 with music.

The game should look good, play good, have cool effects, and be balanced. It should have an attractor screen, scoring, a win state and a lose state.

I also required the code to be maintainable. Meaning I should be able to look at every single line and understand it enough to defend its existence.

I did make it work. And honestly Claude did better than expected. The game ran well and was fun.

But: The process was shit.

I spent 2 days and several hundred dollars to babysit the AI, to get something I could have done in 1 day including learning SDL2.

Everything that turned out well, turned out well because I brought years of skill to the table, and could see when Claude was coding itself into a corner and tell it to break up code in modules, collate globals, remove duplication, pull out abstractions, etc. I had to detect all that and instruct on how to fix it. Until I did it was adding and re-adding bugs because it had made so much shittily structured code it was confusing itself.

TLDR; LLM can write maintainable code if given full constant attention by a skilled coder, at 40% of the coder's speed.

No shit, Sherlock (c)

AI-generated code produces 1.7x more issues than human code

As expected

Anyone blindly having AI write their code is an absolute moron.

Anyone with decent experience (5-10 years, maybe 10+?) can absolutely fucking skyrocket their output if they properly set up their environments and treat their agents as junior devs instead of competent programmers. You shouldn't trust generated code any more than you trust someone fresh out of college, but they produce code in seconds instead of weeks.

I have tripled my output while producing more secure code (based on my security audits), safer code (based on code coverage and security audits), and less error-prone code (based on production logs and our unchanged QA process).

Now, the ethical issues and environmental issues, I 100% can get behind. And I have no idea what companies are going to do in 10 years when they have to replace people like me and haven't been hiring or training replacements. But the productivity and quality debates are absolutely ridiculous, as long as a strong dev is behind the wheel and has been trained to use the tools.

I'd never ask a friggin machine to do coding for me, that's MY blast.

That said, I've had good luck asking GPT specific questions about multiple obscure features of Javascript, and of various browsers. It'll often feed me a sample script using a feature it explains ... a lot more helpful than many of the wordy websites like MDN ... saving me shit-tons of time that I'd spend bouncing around a half-dozen 'help' pages.

Not my code though. It contains a shit ton of bugs. When I am able to write some of course.

I find if I ask it about procedures that have any vague steps AI will stumble on it and sometimes put me into loops where it tells me to do A, A fails, so do B, B fails, so it tells me to do A...

And even worse, it doesn't realise it and can't fix the errors.

Hey don't worry, just get a faster CPU with even more cores and maybe a terabyte or three of RAM to hold all the new layers of abstraction and cruft to fix all that!

what's funny is that this was predicted to be that way even before AI-generated code became an option. Hell, I remember doing an assessment back in early 2023 and literally every domain expert i talked with said this thing - it has its use, but purely supplemental and you won't use it on some fundamental because the clean-up will take more time than was preserved. Counterproductive is the word.

...is this supposed to be news?