The AI Hype Trap – Why Actual Productivity Gains Are Still Elusive

The hours we spend correcting mistakes and getting used to new systems can often overshadow any advantages, reminding us that true innovation takes more than just fancy tools.

I find this really interesting. It's always the same with leaps in reach - sometimes what it promises is simply not that. However AI has made the biggest impact on the way we learn, so I expect we will see that knock on effect in the years to come as well. Whilst it might not be making leaps and bounds in some areas, it's certainly coming.

There's a lot of hype and money to be made. I'm not using these tools at work yet, but I'm old school for coding. They will get better, but mistakes will be made.

Setting-specific factors
We caution readers against overgeneralizing on the basis of our results.
The slowdown we observe does not imply that current AI tools do not often improve developer’s productivity—we find evidence that the high developer familiarity with repositories and the size and maturity of the repositories both contribute to the observed slowdown, and these factors do not apply in many software development settings. For example, our results are consistent with small greenfield
projects or development in unfamiliar codebases seeing substantial speedup from AI assistance.

AI-specific factors
We expect that AI systems that have higher fundamental reliability, lower
latency, and/or are better elicited (e.g. via more inference compute/tokens, more skilled prompting/scaffolding, or explicit fine-tuning on repositories) could speed up developers in our setting (i.e. experienced open-source developers on large repositories).

Agents can make meaningful progress on issues
We have preliminary evidence (forthcoming) that fully autonomous AI agents using Claude 3.7 Sonnet can often correctly implement the core functionality of issues on several repositories that are included in our study, although they fail to fully satisfy all requirements (typically leaving out important documentation, failing linting/styling rules, and leaving out key unit or integration tests). This represents immense progress relative to the state of AI just 1-2 years ago, and if progress continues apace (which is a priori at least plausible, although not guaranteed), we may soon see significant speedup in this setting.

From that same paper, you are doing what they warn against.

Congratulations @revisesociology! You have completed the following achievement on the Hive blockchain And have been rewarded with New badge(s)

You have been a buzzy bee and published a post every day of the week.

_{You can view your badges on your board and compare yourself to others in the Ranking
If you no longer want to receive notifications, reply to this comment with the word STOP}

I agree that AI does have a lot of errors but it is surprising to read that the results were 19% slow.

I started using AI a lot for my coding needs and recently I did a project that took me just 3 weeks. Otherwise it would have taken 3 months to complete. If I had a paid version, I might have been able to complete the project in 3 days maybe.

But yeah true that AI works for some people and it doesn't. It will take a lot more time to mature.

Hey @revisesociology!

Now seniors have wrong impressions due to AI.
Just get it done with AI. It's doable in 10 mins.
Every other app in the world can do it, why can't you with AI?

Expectations are no less than absurd.

Here is a !PIZZA for you 🍕 for calling it out

Another thing that happened is that as we save more time, we fill it with other or superfluous activities. In other words, we come up with more to do to fill our time