GitLab Duo: AI-powered CI/CD pipeline root cause analysis

Higher agent accuracy, lower token cost
Learn how to harden your software factory and let AI agents earn autonomy, without trading away security.
Stay updated with our latest features and improvements.
Published on: June 6, 2024
Discover how we've infused Root Cause Analysis with AI to help remedy broken CI/CD pipelines, including example scenarios and take-away exercises.
Generative AI marks a monumental shift in the software development industry, making it easier to develop, secure, and operate software. Our new blog series, written by our product and engineering teams, gives you an inside look at how we create, test, and deploy the AI features you need integrated throughout the enterprise. Get to know new capabilities within GitLab Duo and how they will help DevSecOps teams deliver better results for customers.
Have you ever encountered a broken CI/CD pipeline and had to halt your DevSecOps workflow, or even delay software deployment, as you try to figure out the root cause? Traditionally, when something goes wrong in the process of creating software, developers have to troubleshoot, dig through log files, and often do a lot of trial and error development. GitLab Duo Root Cause Analysis, part of our suite of AI-powered features, removes the guesswork by determining the root cause for a failed CI/CD pipeline. In this article, you'll learn what Root Cause Analysis is and how to apply the AI-powered GitLab Duo feature to your DevSecOps workflow.
Discover the future of AI-driven software development with our GitLab 17 virtual launch event. Watch today!
GitLab Duo Root Cause Analysis is an AI-powered feature that assists you in determining a root cause and suggesting a fix for a CI/CD job log failure by analyzing the logs.
While Root Cause Analysis is often seen in product incident management, its workflows and debugging practices can be found in any DevSecOps workflow. Ops teams, administrators, and platform engineers are challenged by infrastructure-as-code (IaC) deployment errors, Kubernetes and GitOps problems, and long stack traces while investigating pipeline failures.
GitLab Duo Root Cause Analysis keeps everyone in the same interface and uses AI-powered help to summarize, analyze, and propose fixes so that organizations can release secure software faster.
A pipeline can encounter failures for a variety of reasons, including syntax errors in the code, missing dependencies that the pipeline relies on, test failures during the build process, Kubernetes and IaC deployment timeouts, and numerous other potential issues. When such failures occur, it becomes the responsibility of everyone to meticulously review the logs generated by the pipeline. This job log review process involves scrutinizing the detailed output to identify the specific errors and pinpoint the root cause of the pipeline failure. For example, the following pipeline has multiple job failures that need to be investigated and fixed.
The duration required to fix these failures can vary significantly and is largely influenced by several factors such as:
Manual analysis can be exceedingly challenging and time-consuming, given that log data consists of application logs and system messages with a wide variety of potential sources of failures. A typical pipeline fix can consist of several iterations and context switching. The complexity and the unstructured nature of the logs is a perfect fit for speeding up the task using generative AI. Using AI can reduce the time to identify and fix a pipeline error significantly and also lower the barrier of expertise that would be needed to fix a pipeline such as the above.
Watch GitLab Duo Root Cause Analysis in action:
Root Cause Analysis works by forwarding a portion of the CI/CD job log to the GitLab AI Gateway. GitLab ensures that the portion sent will fit inside the large language model (LLM) token limits alongside a prompt that has been pre-crafted to provide insights into why the job might have failed. The prompt also instructs the LLM to provide an example of how a user might fix a broken job.
Here are two example scenarios where Root Cause Analysis can provide assistance.
A Python application can import package modules with functionality that is not provided in the standard library. The project Challenge - Root Cause Analysis - Python Config implements an application that parses configuration and initializes an SQLite database, which both work well without any dependencies. It uses best practices in CI/CD with a Python environment and caching. The latest feature implementation adds a Redis caching client, and now the CI/CD build is failing for some reason. By using Root Cause Analysis, you can immediately learn that the ModuleNotFoundError text means that the module is actually not installed in the Python environment. GitLab Duo also suggests an example fix: Installing the Redis module through the PIP package manager.
The failing pipeline can be viewed here. The Root Cause Analysis prompt provides a summary of the problem, which seems to be a problem with a missing redis module. Let's try to fix the problem by installing the redis module. You can either call pip install redis in the CI/CD job script section, or use a more sophisticated approach with the requirements.txt file. The latter is useful for a single source of truth for dependencies installed in the development environment and CI/CD pipelines.
After fixing the missing Python dependency, the CI/CD job fails again. Use Root Cause Analysis again to learn that no Redis service is running in the job. Switch to using GitLab Duo Chat and use the prompt How to start a Redis service in CI/CD to learn how to configure the services attribute in the CI/CD job.
Modify the .gitlab-ci.yml with the test job, and specify the redis service.
Running the Redis server allows you to successfully execute the Python application, and print its output into the CI/CD job log.
The solution is provided in the solution/ directory.
Tip: You can also ask