Did these developers not have experience with AI?

This is from the article

But Rush and Becker have shied away from making sweeping claims about what the results of their study mean for the future of AI. For one, the study’s sample was small and non-generalizable, including only a specialized group of people to whom these AI tools were brand new.

I’m not sure focusing on one aspect to scope a reasonable and doable study automatically makes it “really low effort”.

You are right, but I believe they should at least have chosen another use case, to make it interesting. I wouldn’t have needed a study to know that an AI performs worse than a developer in a project the developer most likely built them self. The existing project might have some really weird code smells and work arounds that only the developer on the project knows about and understand. There might be relevant context external to the solution. The AI have to be a mind reader in these cases.

But, if you gave the AI and the developer a blank canvas a clear defined task, I just believe it would be a more interesting study.

It kind of sounds like they were just handed a tool they knew nothing about and were asked to perform better with it. A mitter saw is way better and faster than a regular saw, if you know how to use it.