Is it safe to have AI agents build your software? A security checklist

Having an autonomous system build your software is a security decision, and it should be made the way security decisions are made: by writing down what could go wrong and checking that each risk has a control. This guide gives the threat model for autonomous coding agents working on a client's product and the controls that address it. It is written for CTOs, security leads and the engineering managers who will be asked to sign off. It describes the model we operate under, but the checklist applies to any vendor, including an in-house agent setup, and the right way to use it is to ask for each control in writing.

Two ways a vendor can work, and why it matters

Before the threat model, one distinction decides most of the risk. A vendor that uses coding agents can work in one of two places.

Inside your environment. The agents receive a credential to your code hosting, sometimes to your cloud account or servers, and push changes into your main line. Every control then has to be built around an outside party operating inside your perimeter: scoped tokens, branch protection, your own scanning of what arrives, revocation when the engagement ends. The list is long because the exposure is real.

Inside the vendor's own environment. Nothing of yours is connected. Existing code you want built on is supplied by you as a file package; the work happens in the vendor's isolated infrastructure; the result comes back to you as a tested package that you install yourself. Your systems are never touched, there is nothing to revoke, and your security team reviews a delivery rather than policing live access.

This is the model AIPOS OÜ uses: we never request credentials for, or connect to, a client's code hosting, servers, cloud accounts or development environment, and we never work in a client repository. The sections below are written for that model, with notes on what to demand from a vendor who works the other way.

The threat model in plain terms

An autonomous coding agent reads existing material, reasons about it, writes code, runs tools and produces a result. Each of those steps has a failure mode that a security review must consider:

1. Connection to your systems. The vendor, or whoever controls its environment, can reach things beyond the project: other code, production, internal networks, credentials. The simplest control is to grant no access at all.
2. Cross-client contamination. Code, data or secrets from one client's project leak into another's through shared infrastructure.
3. Instructions hidden in supplied material. Code, documents, configuration and dependencies you supply are untrusted input to an agent. A file or comment can contain text crafted to redirect an agent: exfiltrate data, weaken security, insert a backdoor. This is the agent-specific version of injection, and it is the risk most often overlooked.
4. Unsafe deliveries. A change that introduces a vulnerability, a leaked secret, a malicious dependency or a licence problem reaches you because nothing checked it.
5. Secrets exposure. Credentials end up in logs, prompts, packages or model-provider systems.
6. Model-provider handling. Your code is sent to AI model APIs; what the provider may retain or use is governed by terms you have not seen.
7. Loss of accountability. Nobody can say which change came from where, why it was made, or who approved it.

A vendor who cannot map a control to each of these has not done the work. The sections below give the controls.

Control 1: a separate, disposable environment per project

The foundation is isolation. Each client project should run in its own environment, created for that project and destroyed when it ends, with isolated containers and a locked-down network that permits only the connections the work requires. This addresses cross-client contamination directly: nothing from another project is present, and there is nothing else in the environment to reach.

Ask: Is the environment per project or shared? What is its network allowed to reach? Is it destroyed at the end, and how is that verified? Where does the vendor's own control plane sit relative to it? In our model every project gets a separate, disposable environment inside AIPOS OÜ's infrastructure with isolated containers and a locked-down network, and the control environment is kept apart from the build environment.

Control 2: no connection to your systems

The strongest control on lateral movement is to have no connection at all. Development work does not require a connection to your code hosting, your cloud account, your servers or your identity provider. It requires a copy of the relevant code and material, which you can export as a package, and a way to return the result, which is another package. If a vendor asks for a credential, ask what it is for; if the answer is "to push changes", the vendor has chosen the first model above and the rest of your checklist grows accordingly.

Ask: Do you need any connection to our systems? What exactly, and why? How do we supply existing code instead? Our answer: none. You send a copy as a file package; the result returns to you the same way.

Control 3: treat supplied material as untrusted input

This is the control that distinguishes vendors who understand agents from those who understand demos. Everything a client supplies, including code comments, documentation, configuration, commit messages and third-party dependencies, is input that an agent will read. Some of it may be crafted, by an attacker or by accident, to look like instructions. A serious vendor layers protections against this: separating instructions from content, constraining what agents may do regardless of what they read, monitoring for behaviour that deviates from the specification, and reviewing every output independently of the agent that produced it.

Ask: What stops an instruction hidden in a file from changing what the agent does? How many independent layers are there? What happens if one fails? We run multi-layer protection against supplied material; the independent review and the outgoing scan described next are the last two of those layers.

Control 4: scan and independently review every delivery

Nothing should leave the build environment unchecked. Two gates belong before any package reaches you: an automated scan of the outgoing delivery for secrets, known-vulnerable patterns, suspicious dependencies and policy violations; and a review by something independent of the agent that wrote the change, with the authority to reject it. Together with the test suite, these are what make "tested code" a verifiable statement rather than a slogan.

Ask: What is scanned, and what blocks a delivery? Who or what reviews it, and is it independent of the author? Can a change bypass the gates? In our pipeline every change must pass the test suite and an independent review agent before it is merged, and every delivery is scanned before it is handed over; you receive tested packages only.

Control 5: handle secrets as if they will be seen

Assume anything placed in an agent's environment may end up in a log or a prompt. With development in the vendor's own environment, the question becomes simple: no credential of yours should be in the material you supply. Export code, not configuration with live secrets; supply sample or masked data, not production extracts; describe external systems by their documentation and sandbox details rather than by production keys. On the vendor's side, the outgoing scan should reject any package that contains a secret.

Ask: What do you need from us, and does any of it have to be a live credential? (It should not.) Are deliveries scanned for secrets before they reach us?

Control 6: know the model providers' terms

Autonomous development runs through AI model APIs, and your code is sent to them. The questions are which providers, under which terms, what they retain and for how long, and whether your data is used for training. These answers belong in the contract. A vendor who cannot name the providers or produce the terms is asking you to accept an unknown data processor.

Ask: Which providers? Under which agreements? Retention and training terms? Is this in the contract? In our case the work runs through leading commercial AI models under business terms, on hosting we control, and the providers, terms and data-handling details are set out in the contract.

Control 7: keep accountability in the written record

Every change should be traceable: which scope it belongs to, what specification it implements, what tests it passed, what review approved it. This is partly a security control and partly the basis of incident response: if something is later found wrong, you need to know where it came from. A written-only engagement, a specification that exists before the code, a daily written report, and a delivered git history with descriptive commits are the practical ingredients.

Ask: Can you show, for any change in a delivery, the scope, specification, tests and review behind it? Is the engagement run in writing? Ours is, entirely, and every package carries its complete history.

Controls on your side

Security is shared. When the vendor works in its own environment, your part of the checklist is short:

• Supply copies, not access: export the relevant code as a package and keep every credential out of it.
• Use synthetic or masked data in anything you send; never production extracts.
• Run your own checks on what arrives: your CI, your security scanning, your code review, before anything is merged into your main line.
• Try each milestone on the preview link and compare it with the acceptance criteria before accepting in writing.
• Decide in advance who confirms scopes and accepts deliveries.

If a vendor works inside your environment instead, add: repository-scoped credentials you issue and can revoke, protected main branches, mandatory review of incoming changes, and a plan for removing access when the engagement ends.

How to verify rather than trust

Ask for the controls in writing, then verify them on a trial. A small, real, bounded piece of work at a fixed price lets you observe the whole pipeline: what material was requested, what the daily reports said, what arrived in the package, what the tests and documentation contain, and whether anything surprising appeared. In our model that is the trial build, delivered within 72 hours of written scope confirmation for a fixed 9,900 euros (excl. VAT), deducted in full from the main project; it arrives as a password-protected preview link, a test report and the code as a package you can run yourself or hand to an independent developer for audit. The broader evaluation questions are in how to evaluate an autonomous AI software development vendor, the ownership questions in code ownership and IP when outsourcing, and the engagement itself on how it works and the scale without hiring page. The request form is the first step.

The checklist in one place

Frequently asked questions