And the tests. Oh god the tests. Personal recent favorite: I asked for some changes to a Dockerfile, which it did ably, and then promptly tested by writing a pytest module that traversed up to the root, read the Dockerfile, and checked that the added lines were present.
Silly AI, doesn't it know that you have to write the test first, see it fail, and only then can you justify making a change to the Dockerfile.
I do wonder if the idea of TDD has influenced AIs to be too prone to testing even when a human would never consider it. I've seen some really silly tests, especially when it starts writing tests for things that are setup to only allow testing. Normally a comment later and it agrees it was pointless, but unless you have something in context to force it, it simply defaults to "test all the things".
Code-generating robots are pretty bad at property-based testing, in my experience. They can do it but they still need a lot of hand-holding. They often regress to writing a mirror implementation as the oracle and trying to enumerate a fixed set of examples they find meaningful.
I'm generally happy with the tests I ask it for, some of which are PBT. It's just the insistence upon memorializing every single change with a test. Maybe encouraging / forcing PBT will dissuade it?
I regret this but at some point I stopped reading generated tests. It feels pointless when our test files are already tens of thousands of lines of — at best — tautological slip which says that the codes does what it does.
I think reading and writing test code is harder than the underlying code itself. You must know both the desired behavior of the code-under-test and whether the test correctly stresses that behavior. And then if you're working on anything more complicated than a single unit test you must also make sure that it doesn't blow up anything adjacent to it (preserve state).
I think enforcing black box testing is the best way to get useful tests out of both humans and robots. They must not know the internals, or it will lead them to do bad things.
