tfm

cross-posted from: https://lemmy.dbzer0.com/post/49198958

cross-posted from: https://kbin.melroy.org/m/technology@lemmy.world/t/1067613

Coders spent more time prompting and reviewing AI generations than they saved on coding. On the surface, METR's results seem to contradict other benchmarks and experiments that demonstrate increases in coding efficiency when AI tools are used. But those often also measure productivity in terms of total lines of code or the number of discrete tasks/code commits/pull requests completed, all of which can be poor proxies for actual coding efficiency. These factors lead the researchers to conclude that current AI coding tools may be particularly ill-suited to "settings with very high quality standards, or with many implicit requirements (e.g., relating to documentation, testing coverage, or linting/formatting) that take humans substantial time to learn." While those factors may not apply in "many realistic, economically relevant settings" involving simpler code bases, they could limit the impact of AI tools in this study and similar real-world situations.

cross-posted from: https://lemmy.world/post/33160706

cross-posted from: https://piefed.blahaj.zone/post/164305

cross-posted from: https://sopuli.xyz/post/30543305

cross-posted from: https://lemmy.blahaj.zone/post/28957665

(Don’t take this map too seriously, I found it on another social media, not an academic paper).

cross-posted from: https://piefed.social/post/1047817

cross-posted from: https://lemmy.world/post/33017197