It is still early, but I find that this experiment makes little to no sense and it is barely useful.
The way you test code cannot (and should not) be decoupled by the way in which you architect the code itself.
80%+ of effective testing is not in the testing framework but in the code architecture.
The author doesn't mention how the code is being architected and managed.
For what it is worth, I found that forcing agents on DI/hexagonal architecture and forcing a trivial coverage check is quite useful and produce overall good enough code with relative little effort
I thought it pretty clear that the code was generated by the same agent that received the testing prompt, so there were no constraints on the code structure, and the testing strategy was known at the time the structure was generated.
But what's the initial setup? Was it greenfields every time? And _how_ were the different testing frameworks used? What's the AGENTS.md there? So many things can influence these tests in positive/negative ways that are not included in the write up or results. Seems like a lot of effort without much in the way of actual helpful detail.
> For the first eval, I tried giving agents the zstd RFC (plus errata) and telling them to implement a complete zstd decoder (agents are stuck in a container without internet access). The tests were not given to agents. For something with the surface area of zstd, it's not really reasonable to expect that the tests cover every possible case. For example, even though zstd is a fairly well-tested piece of software, I once found a data corruption bug in zstd. The test suite isn't intended to find extreme corner cases that might be lurking for years and is instead intended to check various cases that can "easily" be derived from the RFC that should work.
So basically, the agents were given the zstd RFC, told to implement a decoder, and also told to use a particular testing strategy (or not, for the control runs).
The setup is described in an earlier post linked in the second paragraph. Agents are given the zstd RFC and told to implement it, greenfield. Testing frameworks were used however the agents decided, which is to say, as mentioned repeatedly throughout the post, badly or not at all. Other factors that could conceivably affect the results can be assumed to be held constant, since the goal is to see what happens when you vary the instructions regarding which testing methodology should be used.
It's intended as a "test to see how effective agents are when guided by someone with no expertise in testing who's maybe heard that you should apply certain techniques or use certain libraries." What do you think would be a more effective test for that purpose?
So not "How well do agents use test/verification techniques?" like the title says then.
It just looks like a lot of effort to go to in a way that doesn't answer the question the title poses.
I'd love to see something that could guide how to set an agent up for success rather than this naive approach which only really answers the questions that most of the audience here probably don't have.
> The way you test code cannot (and should not) be decoupled by the way in which you architect the code itself.
I'm not sure I fully understand. In my view, it's pretty evident that tests should test behavior and not structure. Coupling to implementation details is unavoidable but should be managed, and ideally minimized, to keep the test suite robust and maintainable.
Given the choice between a test suite that monkey patches out some dependency (redis, say) on a per-test basis, and one which substitutes a fake for this dependency in some centralized composition root, which one do you prefer, and why?
You are not wrong but if we go deeper we will find more nuances.
> tests should test behaviour and not structure.
Of what?
The whole idea of software architecture, and underlying of my messages, is to structure the code so that it is easy, but before easy, possible, to work at the right abstraction level.
Which allows to tests the behaviour of components and not their structure.
Trivial example, say your code read from a socket and manage the bytes with some CPU operations and write them to another socket. (You may recognise it is basically what a compressor do)
The way in which you manage read and write to the sockets will make dramatically simpler or much more difficult to write good tests.
If you adopt strategies like sans-io, you will see that the testing is almost trivial.
If all the logic sprawl up from the read syscall in a loop, you will notice how more challenging testing becomes.
---
To answer your question, the way I let LLMs write code is very DI (dependency injection) based.
A class never instantiates another class - all the dependencies are passed to the constructor. Including time. Including whatever DB iteraction.
The reason why I prefer this is that I can test each component at every level of abstraction. I don't have to. But I can.
My current approach is to force coverage higher than say 80% as default and then when a bug is discovered drill down to the components.
The question "for which functions/modules/classes should I write tests?" and the question "how do I structure my functions/modules/classes?" are very closely related questions, which I think is what the other comment was getting at.
That follows largely from what you're saying though, so I think the two of you agree, you're just coming at the same thing from different perspectives. Like you say, you want to avoid/manage coupling tests to implementation details, and typically the best way to do that is to bundle a significant, testable chunk of code into a single module, define a clear public interface to that module, and then test that interface. The internals of that module are then free to change, but the test interface should stay the same.
But the corollary of that is that you can't just design your modules independently of your tests, you need to design them so that they are testable. Which is why the previous comment links testing to code architecture/design.
Regarding dependencies like that, the best situation is where you can either:
(a) design a module so that the dependency is injected with a clear interface (not just "here's an instance of libredis.rs" but "here's a series of callbacks for saving data, those callbacks might save to Redis, but they might be in-memory only, or they might save everything to the blockchain, either way it doesn't matter").
(b) just spin up an instance of Redis and test directly against that — it should be quick enough, and there's plenty of ways to ensure that the tests don't affect each other even while accessing a shared resource.
> I'm not sure I fully understand. In my view, it's pretty evident that tests should test behavior and not structure.
Not OP, but I think I know what OP has in mind. There are some trivial ways in which code can be made easier to test as well as more complex (let's call them architectural) ways.
Examples of trivial conveniences include: making tested functionality accessible from the test (i.e. private class methods in Java present a problem for testing), coding against an interface rather than an implementation (this allows substituting a test implementation for a production implementation that can fish out some bugs), having internal to the code correctness checks that don't necessarily influence program execution (simply having runtime type checks, like it's done in eg. Java, compared to eg. C makes testing easier and more productive). Of course, there's more, these are just very common and easy to understand.
The architectural properties that improve the ability to test a program might be: the so-called "observability" programmed into the product, i.e. the product generates metrics data that's not part of the desired output. The product is split into modules with formally defined interfaces, which allows testing modules in isolation and lowers the number of possible combinations to test.
Improving testability isn't necessarily a good thing because it's likely to negatively impact complexity, size or speed of the program. So, depending on what's more important for any given program, the testing strategy will be different...
It is not clear to me what the author is SPECIFICALLY against.
Only saying "LLM writing" is honestly lazy writing. Specifically what?
I get the glaring cases, I get the idea that if the prose is generated then maybe also the idea, I get the feeling when reading a complete LLM authored piece.
But that doesn't help the piece, because - beside those glaring cases - most writing today is a mix between authors ideas and LLM prose.
What do you mean by "most writing", how are you scoping it? Most HN comments aren't LLM prose. Nor are most HN frontpage submissions. But by reputation, most substack articles or or linkedin posts are.
This is a case where we normalisation of deviance has not yet started biting. And as long as the community manages to make it clear what the norms are and enforce them, we can keep it that way.
Now, if 25% of the frontpage was LLM prose at all times, the site is probably unrecoverably dead. Which is why at least I personally flag anything that I think is ai-written and Pangram concurs. (And write a comment to the effect, or upvote an existing one.)
And it doesn't matter if you say that the ideas were your own, and just the prose was LLM. We can't tell what the idea mix was. But we can tell whether you weren't willing to do your own writing. If you want people to put in the time to read your ideas, human writing is the signalling you need pay for.
Robot vacuums are massively simpler and they fucking break all the time. Wheel motors or their position sensors, belts, plastic gears, contacts that corrode, a circuit board someone decided to not comformally coat and a cat puked on it...
From one perspective I am not against lobbying in general.
The idea is that legislator don't have the full context and don't know the impact of their work - so they welcome industry or matter experts. Which is quite reasonable. I would argue that the job of the legislator is to know what they are doing, but I can see it being hard for specific technical issues that really benefits from first hand experience.
I am not even particularly against the idea of donating money to make sure a particular issue is known to the public, so that the public can make an effective voting decision.
But all of this seems quite too much.
Companies have disproportionate amount of money to donate and much less synchronization issue that citizen.
Yeah, I don't mind walmart making a psa about theft if that's what they want to do with their money. I very much do have a problem with them putting their thumb on the scale and donating millions to one political candidate over another.
You want to hire a lobbyist to show up and educate a senator on soybean production, more power to you, but it shouldn't come with a check.
Lobbying is legalise bribery, plain and simple. You pay money to get politicians to return the favor in terms of regulations and policies that benifit you (even if they are harmful to the general public)
The issue is that lobbying puts the power in the hands of the one that has the most money. Yes, politicians should hear out expert opinions when discussing policy matters because nobody knows everything. However with lobbying, the only people that get a voice are the ones paying the most.
Combine that with these megacorps with infinitely deep pockets, it becomes extremely hard to hear both sides of an issue, and we end up in the opposite of the original issue, where the only context legislators get is the context paid for by monied interests.
It's blatant bribery. I come from a 3rd world country where bribery takes the form of slipping a cop or judge some cash to let you off the hook, and in my mind this is in no way different. Hell, it's maybe even worse because we pretend it isn't blatant corruption and is somehow recognized as okay!
Smith-Waterman Sequence Alignment applied to tool calls. Tool calls are encoded as single characters (Read=R, E=Edit, B=Bash,…), with interesting differences between "successful" and "struggling" sessions.
Another one is "Lag Sequential Analysis", applied to human-agent interactions.
I was only thinking of corpus analysis, but I guess that’s what you get when you give AI a web search tool and keep pushing it to explore more domains to borrow techniques and methods from.
Lately I am finding myself doing more and more of what I called "ambient coding" so that I am not directly using anymore all of those coding harnesses.
There is a lot of positive comments in this comments section that I don't mind being a bit rough.
I think we can do much better.
The workflow of copy to chatgpt and getting feedback is just the first step, and honestly not that useful.
What I would love to see is a tool that makes my writing and thinking clearer.
Does this sentence makes sense? Does the conclusion I am reaching follows from what I am saying? Is this period useful or I am just repeating something I already said? Can I re-arrange my wording to make my point clear? Are my wording actually clear? Or am I not making sense?
Can I re-arrange my essay so that it is simpler to follow?
Revise can answer any of those questions! You just have to ask.
You can also focus your questions by selecting a segment of your document, and then writing a prompt; the agent will see what you've selected and focus its efforts on that. You can even prompt with multiple selections attached at once.
I'm hoping to add more "proactive" AI to this eventually, like automatic comments raising the critiques along the lines of these questions you enumerated. Right now the agent has to be prompted first for it to do any real thinking.
When you translate spec to tests (if those are traditional unit tests or any automated tests that call the rest of the code), that fixes the API of the code, i.e. the code gets designed implicitly in the test generation step. Is this working well in your experience?
I personally found V4-flash an amazing model and really hungry to try 4.1-flash
For software factories, cost is much more a concern that standard development workflow and using anthropic models is just a non starter
reply