How to evaluate a vendor that builds software with autonomous AI agents

Vendors that build software with autonomous AI agents now range from serious engineering operations to a chat window with a sales page in front of it. From the outside they can look similar: both talk about agents, speed and cost. The difference shows up in how the work is specified, gated, isolated, owned and verified. This guide gives buyers a structured way to tell them apart. It is written by such a vendor, so read it as the list of questions we think you should ask us and everyone else, and judge the answers by whether they are written, specific and checkable.

Start with the claim, not the technology

Every vendor in this category makes a throughput claim. The first evaluation question is not whether the claim is impressive but whether it is stated precisely and bounded honestly. A serious vendor will tell you what the claim is based on, what it does not cover, and how you can measure it yourself. A vendor who answers with larger numbers each time you ask is selling a demo.

Our own published claim is that 24 hours of the system's work equals four to eight weeks of a small software company, and that the trial build is the way to measure it on your own code. We also state that the system's measured throughput on AIPOS OÜ's own production systems exceeds the combined work of tens of developers, and we deliberately do not publish those internal figures as marketing numbers, because they are specific to our codebases and not independently checkable from outside. Expect that kind of distinction from any vendor: what is published should be what you can verify.

Question one: who owns the architecture and the specification?

Autonomous agents are good at building against a specification and poor at deciding what the specification should be. The single most important question is therefore: who writes the architecture and the specification, and are they accountable for it?

Acceptable answers name a person, describe what they produce before any code is written, and make that document part of what you receive. Unacceptable answers describe the agents "figuring it out" from a prompt, or assign architecture to whoever is available. In our model AIPOS OÜ designs the architecture and specification in-house for every project and agrees them with the client in writing before any agent touches code; the agents build against them.

Question two: how is quality gated before you see anything?

Speed without a gate produces fast defects. Ask what must be true before a change reaches a delivery to you. The minimum for a serious vendor:

• An automated test suite that every change must pass, written as part of the delivery.
• An independent review of every change by something other than the agent that wrote it, whether a separate review agent or a person, with the authority to reject.
• Documentation updated with the change, not after the project.
• A readable history delivered with the code, so your team can review the changes like any other contribution.

Ask also what happens when a step fails at three in the morning. A system that retries and repairs itself is different from one that waits for a human; the answer tells you whether the "24/7" claim is literal. In our case tests and an independent review agent gate every merge, every outgoing change is scanned, and a self-healing watchdog restarts or repairs stalled cycles.

Question three: how is your code isolated?

Where and how the agents work is a security question before it is a productivity question. The checklist is long enough to deserve its own guide, security when AI agents build your software, but the headline questions are:

• Does each client project run in its own separate, disposable environment, with isolated containers and a locked-down network, or do projects share infrastructure?
• Does the vendor need any connection to your systems at all? The strongest answer is none: existing code is supplied by you as a package and the result comes back the same way.
• Is the control environment separate from the build environment?
• How is the system protected against malicious or accidental instructions hidden in supplied material (code, documents, dependencies), which is untrusted input to an agent?
• Is every delivery scanned and reviewed before it reaches you?
• Which AI model providers are used, under what terms, and how is your code handled by them? This should be in the contract, not in a verbal assurance.

A vendor who cannot answer these in writing has not thought about them, and your code should not be the occasion for them to start.

Question four: what do you own at the end?

Ask for the intellectual-property clause before any other document. You are looking for an assignment of IP in the deliverables to you, the code handed over at agreed milestones with its history, a clear statement that the vendor's platform and methodology stay with the vendor and are not embedded in what you receive, and documentation and tests that let a third party maintain the system. The detailed checklist is in code ownership and IP when outsourcing. Our position: full IP in the deliverables goes to the client; the platform and methodology remain with AIPOS OÜ.

Question five: how is the price set?

Fixed price or hourly tells you how confident the vendor is in its own production method. A vendor with a repeatable system can fix a price on a written scope. A vendor that bills AI-assisted work by the hour has either an unpredictable method or a commercial preference for continuing over finishing. Ask for a written scope description with the price, a change-control mechanism in writing, and acceptance criteria that can be tested. The trade-offs between the models are in fixed price versus time and materials, and the wider comparison with building a team is on instead of hiring developers.

Question six: how do they communicate?

This sounds secondary and is not. Autonomous production works on written specifications; a vendor whose engagement model depends on calls and meetings is either translating conversations into specifications behind the scenes, with the loss that implies, or has not yet operationalised the method. Written-only communication is also what gives you a complete record of every scope, change and acceptance. We work entirely in writing for that reason.

Question seven: can you verify any of this before committing?

The decisive test is empirical. A serious vendor will let you buy a small, real, bounded piece of your own plan at a fixed price and judge the result on a working preview and in the delivered package: the code, the tests, the documentation, the speed, and the experience of the written process. A vendor who offers only demos, case studies or free prototypes on their own sample projects is asking you to evaluate a performance rather than a delivery.

Our trial build is built for this purpose: a genuine part of your development plan, confirmed in writing, delivered within 72 hours as tested and documented code with full IP, for a fixed 9,900 euros that is deducted in full from the main project. The first three clients receive it for 4,900 euros in exchange for a public reference. How to prepare so the trial is a fair test is in how to prepare for a 72-hour trial build.

Reading references and track records

Be careful with both. Logos and testimonials are easy to obtain and hard to verify; invented numbers are easier still. Prefer a vendor who tells you plainly what they will and will not publish and why. Our own track record page describes the system that runs every day and states openly that we publish no client logos, testimonials or invented numbers; founding-client references will appear there as they are agreed in writing. Judge any vendor by whether their published claims are things you could check.

A scoring sheet

For each vendor, score the seven questions on a simple scale: answered in writing and checkable; answered verbally or vaguely; not answered. Then run the empirical test with the best one or two. A vendor who scores well on paper and fails the trial build has told you something important cheaply. A vendor who scores well on paper and passes the trial has earned the main project.

The questions again, in one list:

1. Who writes the architecture and specification, and are they accountable?
2. What gates every change before it reaches us?
3. How is our code isolated, and what are the model providers' terms?
4. What do we own at the end, and when do we receive the code?
5. Is the price fixed on a written scope, with change control in writing?
6. Is the engagement run in writing, with a complete record?
7. Can we buy a small, real piece at a fixed price and judge it ourselves?

Our answers to all seven are on how it works, in the terms and in these guides. The request form is where the empirical test begins.

Frequently asked questions