Artificial intelligence (AI) is now used in many parts of software development. Developers use large language models, or LLMs, to write new source code, to explain and change existing code, and to help with debugging. This raises an important question. What happens when AI-generated code itself contains security vulnerabilities? Code can look clean and correct and still contain a serious weakness. In other words, code that works is not always code that is safe. This question has been discussed since code generators first appeared, and it was the starting point for our paper.
Early studies on the security of AI-generated code mostly used short, artificial samples, or partly written code that was deliberately vulnerable, and that the model was asked to complete. This kind of setup tends to produce a high proportion of insecure code. For example, Pearce et al. [Pearce 2022] constructed 89 scenarios based on the MITRE CWE Top 25 most dangerous software weaknesses and asked GitHub Copilot to complete them. This produced 1,689 programs, of which about 40% were vulnerable. Khoury et al. [Khoury 2023] used a similar approach with ChatGPT. They asked it to generate 21 small standalone programs in C, C++, Python, and Java, and only 5 were secure to begin with. User studies follow the same pattern at a slightly larger scale. Perry et al. [Perry 2023] had 47 participants complete five short security-related tasks, and participants with access to an AI assistant often wrote less secure code than those without. These studies show that code generators often produce insecure code. However, they rely on toy problems and small task sets, so it is not clear how far their results generalise to the real world. In addition, because the prompts often contained partly vulnerable code, the results were likely inflated. We suspected that the code generated by these models had security problems. However, only by assessing code generated in a real setting can we see the true scope, extent, and scale of the problem. To the best of our knowledge, at the time of our work no study had examined the security of code generated by real developers in real interactions, rather than in controlled experiments or small lab settings. This is the gap we set out to address.
We approached the problem from a practical angle. Instead of using examples written specifically for a research benchmark, we used code from the open-source DevGPT dataset, which reflects more realistic developer use of AI tools. We analysed 2,315 code snippets written in C, C++, and C#. We first screened the files with several static source-code scanners, and we then manually confirmed 56 vulnerabilities in 48 files. After that we asked two further questions. Can modern LLMs detect these security problems, and can they repair them?
The repair side of the problem was especially important to us. Even when AI-generated code contains mistakes, it is still useful to ask whether newer models can find those mistakes and fix them. This is also why a real-world evaluation matters. It is more informative to see how a model performs on code that comes from realistic developer interactions and contains real security issues, rather than idealised test cases.
A program may run correctly in normal use and still fail badly in unusual or malicious cases. Security problems are often hidden in small details, such as input checking, memory handling, or boundary conditions. For this reason it is important to test AI systems carefully, and not only to look at whether their answers seem convincing. One thing that makes this area difficult is that LLMs are often good at sounding confident. They can produce clear explanations and polished code even when the answer is wrong or incomplete.
One small example from our study illustrates this problem. In one file, the whole code fragment was a single line that wrote a message to an error log. There was no real security problem in that line, but the models still reported several possible vulnerability categories. For a non-specialist, this is a useful way to think about the issue. AI can act like a very sensitive alarm. It may catch real smoke, but it may also ring loudly when there is no fire.
One of the most difficult stages of the work was the manual analysis of the scanner output. Running the static analysis tools was relatively straightforward, but interpreting the results was much harder. The scanners produced many warnings, and we had to check each one in context to decide whether it was a real vulnerability, a minor concern, or a false positive. This was especially difficult because many DevGPT snippets were incomplete, were not always compilable, and often lacked the wider project context that normally helps with security assessment. As a result, the manual stage required security knowledge as well as careful code reading, judgement, and repeated validation. In practice this stage became the main bottleneck of the study. It improved the quality of our results, but it also limited how many files we could analyse.
To study detection and repair, we tested models that were current at the time of each experiment. We used GPT-4o in October 2024, and GPT-4.1, GPT-5, and Claude Opus 4.1 in September 2025. We checked whether they could identify the vulnerabilities we had found with the static scanners, and whether they could repair them. Overall detection and repair rates rose from about 50% in 2024 to about 75-80% in 2025. This suggests that LLMs are improving and may become more useful as tools for security-related code review and repair.
Nonetheless, our results show that these systems should not be trusted without question. Even when overall performance is strong, individual mistakes still matter. In security, a single missed vulnerability can be a disaster, and a fix that only partly solves a problem can also be dangerous. The results were promising, but they also reminded us that human review is still essential.
In our view, this work balances two common positions on AI. One position is that AI will soon handle software development on its own; the other is that AI-generated code is too unreliable to be trusted at all. As AI becomes more common in software engineering, we need to understand both what these systems can do and where they still fall short. Security is a good test case, because it requires more than code that runs. It requires care, precision, and an awareness that small mistakes can have large consequences.
Another interesting part of the study was how quickly these systems are improving. Compared with our earlier experiments in October 2024, the newer models available in September 2025 performed better at both detection and repair. A conclusion about AI tools from one year may not hold the next.
AI is no longer a niche research topic or a toy used only by early adopters. It is now discussed widely, and more companies are trying to add it to their development workflows. In many cases this adoption seems to be happening faster than the surrounding engineering discipline. Review processes, security checks, developer training, and organisational policies are still catching up. Our own experiments reflected this. Model performance improved a lot between the first and second rounds of testing, but the need for careful human verification did not disappear. For us this was one of the main lessons behind the paper. AI-assisted development is becoming part of normal software engineering, but we still do not fully understand its long-term risks, limitations, and have yet to establish best practices.
In the end, the story behind this paper is nuanced. AI can produce insecure code, especially when prompts do not emphasise security. But it can also detect and help repair insecure code. We see that AI is becoming capable enough in this area that we need to take it seriously, while still being careful not to trust it too much. Our results suggest that LLMs can make a real contribution to finding and repairing vulnerabilities. They also show that human judgement still matters. AI can help with software security, but it should be used as an assistant, and not as the final authority.
References
Khoury, R., Avila, A. R., Brunelle, J., & Camara, B. M. (2023). How secure is code generated by ChatGPT? In 2023 IEEE International Conference on Systems, Man, and Cybernetics (SMC) (pp. 2445–2451). IEEE.
Pearce, H., Ahmad, B., Tan, B., Dolan-Gavitt, B., & Karri, R. (2022). Asleep at the keyboard? Assessing the security of GitHub Copilot's code contributions. In 2022 IEEE Symposium on Security and Privacy (SP) (pp. 754–768). IEEE.
Perry, N., Srivastava, M., Kumar, D., & Boneh, D. (2023). Do users write more insecure code with AI assistants? In Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security (CCS '23) (pp. 2785–2799). ACM.