Introduction to devtron

1. What is Devtron

Devtron is a tool integration platform for Kubernetes. Devtron deeply integrates with products across the lifecycle of microservices i.e., CI/CD, security, cost, debugging, and observability via an intuitive web interface. Devtron helps you to deploy, observe, manage & debug the existing Helm apps in all your clusters.

2. Most important features

No Code Software Delivery Workflow for Kubernetes
- Workflow which understands the domain of Kubernetes, testing, CD, SecOps so that you don't have to write scripts
- Reusable and composable components so that workflows are easy to construct and reason through

Multi-cloud Deployment
- Deploy to multiple Kubernetes clusters on multiple cloud/on-prem from one Devtron setup
- Works for all cloud providers and on-premise Kubernetes clusters

Easy DevSecOps Integration
- Multi-level security policy at global, cluster, environment, and application-level for efficient hierarchical policy management
- Behavior-driven security policy
- Define policies and exceptions for Kubernetes resources
- Define policies for events for faster resolution

Application Debugging Dashboard
- One place for all historical Kubernetes events
- Access all manifests securely, such as secret obfuscation
- Application metrics for CPU, RAM, HTTP status code, and latency with a comparison between new and old
- Advanced logging with grep and JSON search
- Intelligent correlation between events, logs for faster triangulation of issue
- Auto issue identification

Cost management
- Resource optimization by HPA.
- Manage idle resources to optimize your cluster.
- Complete insight into the cost of clusters, environments, and applications.

Enterprise-Grade Security and Compliance
- Fine-grained access control; control who can edit the configuration and who can deploy.
- Audit log to know who did what and when
- History of all CI and CD events
- Kubernetes events impacting application
- Relevant cloud events and their impact on applications
- Advanced workflow policies like blackout window, branch environment relationships to secure build and deployment pipelines

Implements GitOps
- GitOps is exposed through API and UI so that you don't have to interact with git CLI
- GitOps backed by Postgres for easy analysis
- Enforce finer access control than Git

Operational Insights
- Deployment metrics to measure the success of the agile process. It captures MTTR, change failure rate, deployment frequency, and deployment size out of the box.
- Audit log to understand the failure causes
- Monitor changes across deployments and reverts easily

3. Challenges before using Devtron

Helm is a command line utility for managing Kubernetes applications. A Helm Chart is a collection of files that describe a related set of Kubernetes resources. With a single chart, you can run an application with bare minimum configs to something very complex, like a full web app inside the Kubernetes cluster. Helm charts help you define, install and upgrade even the most complex Kubernetes application.

Challenges with Helm

No doubt that Helm is a very popular tool to manage deployments on Kubernetes. Helm helps us to install, and upgrade applications using a single command, i.e. helm install stable/mysql. Without Helm, this would be done by creating and applying numerous Kubernetes yaml files. But still, there are certain challenges with Helm which needs to be addressed.
No actual state of deployments

Helm introduces an abstraction layer between Kubernetes and the users. It does not observe the actual state of the resources and results in obscure status of the deployments. To understand this point, let's take an example. Assume you deploy a Helm chart using helm and it is deployed successfully. Now check status of the deployment. This will show deployed . Everything is okay till now. Now delete one of its required resources. After doing this, pods won't run anymore. Now again if querying helm status , it will still shows the status as deployed and it means everything is okay but it's not actually. Helm is not able to identify the live status of resources deployed. Once the application is deployed, helm's work is done.
Troubleshooting and debugging

The biggest challenge for Helm is complexity. The whole system is based on templating helm charts which makes it very difficult to create and debug complex applications that may consist of multiple Kubernetes resources. The more the Helm charts are, the more complex the entire system is. Imagine how much time it would take for a team to spot and resolve a bug in a Helm chart template that has been used numerous times across many Kubernetes resources in a complex application.
No UI for Helm applications

Yes, for some of you, this might not be a challenge with Helm because as a SRE/DevOps developer we all love terminals. But in some cases especially if we have 100s of charts deployed, it becomes really difficult to manage all these charts and Kubernetes resources corresponding to it. Managing these using GUI is much easier than CLI. Therefore, managing and monitoring Helm charts using Helm CLI becomes a tedious task and also very confusing if you are new in the Kubernetes space. At each step of managing helm packages, you have to trigger the helm commands which sometimes becomes a real mess. Just imagine, how convenient it would be to have a UI to manage all your helm charts deployed with CLI as well as for deploying new charts through UI and getting the live status of all those applications deployed.

4. Devtron solutions

Devtron is a lightweight, open-source, easy-to-use web-based tool for developers as well as infra teams that can solve all the challenges described above and many more use cases apart from these.

Shows live status of deployment

Devtron does everything that you cannot do using Helm CLI. As we discussed above that Helm CLI doesn't show the actual state of the deployment. Even if the pods are not running, Helm CLI considers the deployment healthy. But this doesn't happen in the case of Devtron. It keeps on monitoring the state of all the resources in real-time. If everything is running fine then it shows the status as Healthy otherwise it shows degraded or Missing that means some of the required resources are failing or missing. Moreover you can expand the status to check the reasons behind the status Degraded or Missing which can help you to troubleshoot the deployment.
Troubleshooting and debugging

Devtron provides you with resources grouping of all the workloads deployed by the charts which makes it lot more easier for development and infra teams to troubleshoot & debug. As you can see in the image below, for all the workloads you get to look into its manifest, events, and logs as well as you can log in into the terminal too and all these from the UI itself. This makes the life much more easier even if you have 1000s of charts deployed.
Provides UI to manage Helm applications

As we discussed above, Helm does not provide a user interface to manage the Helm applications but this is not in the case of Devtron. All the operations associated with the Helm applications that we do by firing Helm commands can be performed all those tasks in a more convenient manner with the Devtron dashboard only. We can add helm repository, can list all the charts present in the repository and also can deploy the chart on a single click. To get all the charts, go-to the charts store and you can see all the charts present as shown in the image below. Any existing helm chart repository can be added from Global Configurations > Chart Repositories > Add Repository. For more details please have a look at the documentation

You can also easily deploy any chart if you want with just few clicks. To deploy any chart, Click on Chart > Click on Deploy. It will open the chart's values.yaml file as you can see in the image below. Also you can see the README of that chart side-by-side which helps you to see the parameters and its descriptions as you can see in the image below. After making necessary changes, click on Deploy Chart as seen in image below.

5. Overall Architecture

6. Installation

Before Install Devtron first install HELM

Before using Devtron, first, all we have to install HELM in our system. Below I mention everything about how to install HEML in our system in every possible way.
Install From The Helm Project
- From the Binary Releases

Every release of Helm provides binary releases for a variety of OSes. These binary versions can be manually downloaded and installed.

Download your desired version
Unpack it (tar -zxvf helm-v3.0.0-linux-amd64.tar.gz)
Find the helm binary in the unpacked directory, and move it to its desired destination (mv linux-amd64/helm /usr/local/bin/helm)

Install From Script

Helm now has an installer script that will automatically grab the latest version of Helm and install it locally. You can fetch that script, and then execute it locally. It's well documented so that you can read through it and understand what it is doing before you run it.

$ curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3
$ chmod 700 get_helm.sh
$ ./get_helm.sh

Install Through Package Managers
From Homebrew (macOS)

Members of the Helm community have contributed a Helm formula build to Homebrew. This formula is generally up to date.

brew install helm

From Chocolatey (Windows)

Members of the Helm community have contributed a Helm package build to Chocolatey. This package is generally up to date.

choco install kubernetes-helm

From Scoop (Windows)

Members of the Helm community have contributed a Helm package build to Scoop. This package is generally up to date.

scoop install helm

From Apt (Debian/Ubuntu)

Members of the Helm community have contributed a Helm package for Apt. This package is generally up to date.

curl https://baltocdn.com/helm/signing.asc | gpg --dearmor | sudo tee /usr/share/keyrings/helm.gpg > /dev/null
sudo apt-get install apt-transport-https --yes
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/helm.gpg] https://baltocdn.com/helm/stable/debian/ all main" | sudo tee /etc/apt/sources.list.d/helm-stable-debian.list
sudo apt-get update
sudo apt-get install helm

From dnf/yum (fedora)

Since Fedora 35, helm is available on the official repository. You can install helm with invoking:

sudo dnf install helm

From Snap

The Snapcrafters community maintains the Snap version of the Helm package:

sudo snap install helm --classic

From pkg (FreeBSD)

Members of the FreeBSD community have contributed a Helm package build to the FreeBSD Ports Collection. This package is generally up to date.

pkg install helm

From Source (Linux, macOS)

Building Helm from source is slightly more work, but is the best way to go if you want to test the latest (pre-release) Helm version.

$ git clone https://github.com/helm/helm.git
$ cd helm
$ make

Install Devtron

Here I mention how to install Devtron in our system in three possible way,

Install Devtron with CI/CD Integration, Install using AWS EBS CSI Driver and finally, Install Devtron with CI/CD along with GitOps (Argo CD).
- Install Devtron with CI/CD Integration

Run the following command to install the latest version of Devtron along with the CI/CD module:

helm repo add devtron https://helm.devtron.ai 

helm install devtron devtron/devtron-operator \
--create-namespace --namespace devtroncd \
--set installer.modules={cicd}

If you want to configure Blob Storage during the installation, follow these commands depending on which cloud service provider you are using,

MiniO storage :

Run the following command to install Devtron along with MinIO for storing logs and cache.

helm repo add devtron https://helm.devtron.ai 

helm install devtron devtron/devtron-operator \
--create-namespace --namespace devtroncd \
--set installer.modules={cicd} \
--set minio.enabled=true

AWS s3 bucket :

Run the following command to install Devtron along with AWS S3 buckets for storing build logs and cache:

Install using S3 IAM policy.

helm repo add devtron https://helm.devtron.ai

helm install devtron devtron/devtron-operator \
--create-namespace --namespace devtroncd \
--set installer.modules={cicd} \
--set configs.BLOB_STORAGE_PROVIDER=S3 \
--set configs.DEFAULT_CACHE_BUCKET=demo-s3-bucket \
--set configs.DEFAULT_CACHE_BUCKET_REGION=us-east-1 \
--set configs.DEFAULT_BUILD_LOGS_BUCKET=demo-s3-bucket \
--set configs.DEFAULT_CD_LOGS_BUCKET_REGION=us-east-1

Install using access-key and secret-key for AWS S3 authentication:

helm repo add devtron https://helm.devtron.ai

helm install devtron devtron/devtron-operator \
--create-namespace --namespace devtroncd \
--set installer.modules={cicd} \
--set configs.BLOB_STORAGE_PROVIDER=S3 \
--set configs.DEFAULT_CACHE_BUCKET=demo-s3-bucket \
--set configs.DEFAULT_CACHE_BUCKET_REGION=us-east-1 \
--set configs.DEFAULT_BUILD_LOGS_BUCKET=demo-s3-bucket \
--set configs.DEFAULT_CD_LOGS_BUCKET_REGION=us-east-1 \
--set secrets.BLOB_STORAGE_S3_ACCESS_KEY=<access-key> \
--set secrets.BLOB_STORAGE_S3_SECRET_KEY=<secret-key>

Install using S3 compatible storages:

helm repo add devtron https://helm.devtron.ai

helm install devtron devtron/devtron-operator \
--create-namespace --namespace devtroncd \
--set installer.modules={cicd} \
--set configs.BLOB_STORAGE_PROVIDER=S3 \
--set configs.DEFAULT_CACHE_BUCKET=demo-s3-bucket \
--set configs.DEFAULT_CACHE_BUCKET_REGION=us-east-1 \
--set configs.DEFAULT_BUILD_LOGS_BUCKET=demo-s3-bucket \
--set configs.DEFAULT_CD_LOGS_BUCKET_REGION=us-east-1 \
--set secrets.BLOB_STORAGE_S3_ACCESS_KEY=<access-key> \
--set secrets.BLOB_STORAGE_S3_SECRET_KEY=<secret-key> \
--set configs.BLOB_STORAGE_S3_ENDPOINT=<endpoint>

Azure Blob_Storage

Run the following command to install Devtron along with Azure Blob Storage for storing build logs and cache:

helm repo add devtron https://helm.devtron.ai

helm install devtron devtron/devtron-operator \
--create-namespace --namespace devtroncd \
--set installer.modules={cicd} \
--set secrets.AZURE_ACCOUNT_KEY=xxxxxxxxxx \
--set configs.BLOB_STORAGE_PROVIDER=AZURE \
--set configs.AZURE_ACCOUNT_NAME=test-account \
--set configs.AZURE_BLOB_CONTAINER_CI_LOG=ci-log-container \
--set configs.AZURE_BLOB_CONTAINER_CI_CACHE=ci-cache-container

Google Cloud Storage

Run the following command to install Devtron along with Google Cloud Storage for storing build logs and cache:

helm repo add devtron https://helm.devtron.ai

helm install devtron devtron/devtron-operator \
--create-namespace --namespace devtroncd \
--set installer.modules={cicd} \
--set configs.BLOB_STORAGE_PROVIDER=GCP \
--set secrets.BLOB_STORAGE_GCP_CREDENTIALS_JSON=eyJ0eXBlIjogInNlcnZpY2VfYWNjb3VudCIsInByb2plY3RfaWQiOiAiPHlvdXItcHJvamVjdC1pZD4iLCJwcml2YXRlX2tleV9pZCI6ICI8eW91ci1wcml2YXRlLWtleS1pZD4iLCJwcml2YXRlX2tleSI6ICI8eW91ci1wcml2YXRlLWtleT4iLCJjbGllbnRfZW1haWwiOiAiPHlvdXItY2xpZW50LWVtYWlsPiIsImNsaWVudF9pZCI6ICI8eW91ci1jbGllbnQtaWQ+IiwiYXV0aF91cmkiOiAiaHR0cHM6Ly9hY2NvdW50cy5nb29nbGUuY29tL28vb2F1dGgyL2F1dGgiLCJ0b2tlbl91cmkiOiAiaHR0cHM6Ly9vYXV0aDIuZ29vZ2xlYXBpcy5jb20vdG9rZW4iLCJhdXRoX3Byb3ZpZGVyX3g1MDlfY2VydF91cmwiOiAiaHR0cHM6Ly93d3cuZ29vZ2xlYXBpcy5jb20vb2F1dGgyL3YxL2NlcnRzIiwiY2xpZW50X3g1MDlfY2VydF91cmwiOiAiPHlvdXItY2xpZW50LWNlcnQtdXJsPiJ9Cg== \
--set configs.DEFAULT_CACHE_BUCKET=cache-bucket \
--set configs.DEFAULT_BUILD_LOGS_BUCKET=log-bucket

Install using AWS EBS CSI Driver

If you are using EKS version 1.23 or greater, you must install aws-ebs-csi-driver.

Run the following command to install aws ebs csi driver using Helm:

helm repo add aws-ebs-csi-driver \
https://kubernetes-sigs.github.io/aws-ebs-csi-driver \
helm repo update \
helm upgrade --install aws-ebs-csi-driver \
--namespace kube-system aws-ebs-csi-driver/aws-ebs-csi-driver

Check the Status of Devtron Installation

kubectl -n devtroncd get installers installer-devtron \
-o jsonpath='{.status.sync.status}'

Check the installer logs

kubectl logs -f -l app=inception -n devtroncd

Devtron dashboard

kubectl get svc -n devtroncd devtron-service \
-o jsonpath='{.status.loadBalancer.ingress}'

Devtron Admin credentials

kubectl -n devtroncd get secret devtron-secret \
-o jsonpath='{.data.ADMIN_PASSWORD}' | base64 -d

Install Devtron with CI/CD along with GitOps (Argo CD)

Run the following command to install the latest version of Devtron with CI/CD along with GitOps (Argo CD) module:

helm repo add devtron https://helm.devtron.ai

helm install devtron devtron/devtron-operator \
--create-namespace --namespace devtroncd \
--set installer.modules={cicd} \
--set argo-cd.enabled=true

If you want to configure Blob Storage during the installation, follow this commands depending on which cloud service provider you are using,

MiniO Storage

Run the following command to install Devtron along with MinIO for storing logs and cache.

helm repo add devtron https://helm.devtron.ai 

helm install devtron devtron/devtron-operator \
--create-namespace --namespace devtroncd \
--set installer.modules={cicd} \
--set minio.enabled=true \
--set argo-cd.enabled=true

AWS s3 bucket

Run the following command to install Devtron along with MinIO for storing logs and cache.

Install using S3 IAM policy.

helm repo add devtron https://helm.devtron.ai

helm install devtron devtron/devtron-operator \
--create-namespace --namespace devtroncd \
--set installer.modules={cicd} \
--set configs.BLOB_STORAGE_PROVIDER=S3 \
--set configs.DEFAULT_CACHE_BUCKET=demo-s3-bucket \
--set configs.DEFAULT_CACHE_BUCKET_REGION=us-east-1 \
--set configs.DEFAULT_BUILD_LOGS_BUCKET=demo-s3-bucket \
--set configs.DEFAULT_CD_LOGS_BUCKET_REGION=us-east-1 \
--set argo-cd.enabled=true

Install using access-key and secret-key for AWS S3 authentication:

helm repo add devtron https://helm.devtron.ai

helm install devtron devtron/devtron-operator \
--create-namespace --namespace devtroncd \
--set installer.modules={cicd} \
--set configs.BLOB_STORAGE_PROVIDER=S3 \
--set configs.DEFAULT_CACHE_BUCKET=demo-s3-bucket \
--set configs.DEFAULT_CACHE_BUCKET_REGION=us-east-1 \
--set configs.DEFAULT_BUILD_LOGS_BUCKET=demo-s3-bucket \
--set configs.DEFAULT_CD_LOGS_BUCKET_REGION=us-east-1 \
--set secrets.BLOB_STORAGE_S3_ACCESS_KEY=<access-key> \
--set secrets.BLOB_STORAGE_S3_SECRET_KEY=<secret-key> \
--set argo-cd.enabled=true

Install using S3 compatible storages:

helm repo add devtron https://helm.devtron.ai

helm install devtron devtron/devtron-operator \
--create-namespace --namespace devtroncd \
--set installer.modules={cicd} \
--set configs.BLOB_STORAGE_PROVIDER=S3 \
--set configs.DEFAULT_CACHE_BUCKET=demo-s3-bucket \
--set configs.DEFAULT_CACHE_BUCKET_REGION=us-east-1 \
--set configs.DEFAULT_BUILD_LOGS_BUCKET=demo-s3-bucket \
--set configs.DEFAULT_CD_LOGS_BUCKET_REGION=us-east-1 \
--set secrets.BLOB_STORAGE_S3_ACCESS_KEY=<access-key> \
--set secrets.BLOB_STORAGE_S3_SECRET_KEY=<secret-key> \
--set configs.BLOB_STORAGE_S3_ENDPOINT=<endpoint> \
--set argo-cd.enabled=true

Azure Blob storage

Run the following command to install Devtron along with Azure Blob Storage for storing build logs and cache:

helm repo add devtron https://helm.devtron.ai

helm install devtron devtron/devtron-operator \
--create-namespace --namespace devtroncd \
--set installer.modules={cicd} \
--set secrets.AZURE_ACCOUNT_KEY=xxxxxxxxxx \
--set configs.BLOB_STORAGE_PROVIDER=AZURE \
--set configs.AZURE_ACCOUNT_NAME=test-account \
--set configs.AZURE_BLOB_CONTAINER_CI_LOG=ci-log-container \
--set configs.AZURE_BLOB_CONTAINER_CI_CACHE=ci-cache-container \
--set argo-cd.enabled=true

Google Cloud Storage

Run the following command to install Devtron along with Google Cloud Storage for storing build logs and cache:

helm repo add devtron https://helm.devtron.ai

helm install devtron devtron/devtron-operator \
--create-namespace --namespace devtroncd \
--set installer.modules={cicd} \
--set configs.BLOB_STORAGE_PROVIDER=GCP \
--set secrets.BLOB_STORAGE_GCP_CREDENTIALS_JSON=eyJ0eXBlIjogInNlcnZpY2VfYWNjb3VudCIsInByb2plY3RfaWQiOiAiPHlvdXItcHJvamVjdC1pZD4iLCJwcml2YXRlX2tleV9pZCI6ICI8eW91ci1wcml2YXRlLWtleS1pZD4iLCJwcml2YXRlX2tleSI6ICI8eW91ci1wcml2YXRlLWtleT4iLCJjbGllbnRfZW1haWwiOiAiPHlvdXItY2xpZW50LWVtYWlsPiIsImNsaWVudF9pZCI6ICI8eW91ci1jbGllbnQtaWQ+IiwiYXV0aF91cmkiOiAiaHR0cHM6Ly9hY2NvdW50cy5nb29nbGUuY29tL28vb2F1dGgyL2F1dGgiLCJ0b2tlbl91cmkiOiAiaHR0cHM6Ly9vYXV0aDIuZ29vZ2xlYXBpcy5jb20vdG9rZW4iLCJhdXRoX3Byb3ZpZGVyX3g1MDlfY2VydF91cmwiOiAiaHR0cHM6Ly93d3cuZ29vZ2xlYXBpcy5jb20vb2F1dGgyL3YxL2NlcnRzIiwiY2xpZW50X3g1MDlfY2VydF91cmwiOiAiPHlvdXItY2xpZW50LWNlcnQtdXJsPiJ9Cg== \
--set configs.DEFAULT_CACHE_BUCKET=cache-bucket \
--set configs.DEFAULT_BUILD_LOGS_BUCKET=log-bucket \
--set argo-cd.enabled=true

Check the Status of Devtron Installation

kubectl -n devtroncd get installers installer-devtron \
-o jsonpath='{.status.sync.status}'

Check the installer logs

kubectl logs -f -l app=inception -n devtroncd

Devtron dashboard

kubectl get svc -n devtroncd devtron-service \
-o jsonpath='{.status.loadBalancer.ingress}'

Devtron Admin credentials

kubectl -n devtroncd get secret devtron-secret \
-o jsonpath='{.data.ADMIN_PASSWORD}' | base64 -d

Install Devtron on Minikube, Microk8s, K3s, Kind

Minikube Kind Cluster

To install devtron on Minikube/kind cluster, run the following command:

helm repo add devtron https://helm.devtron.ai

helm install devtron devtron/devtron-operator \
--create-namespace --namespace devtroncd \
--set components.devtron.service.type=NodePort

To install devtron on k3s cluster, run the following command:

kubectl apply -f https://raw.githubusercontent.com/rancher/local-path-provisioner/master/deploy/local-path-storage.yaml

helm repo add devtron https://helm.devtron.ai

helm install devtron devtron/devtron-operator \
--create-namespace --namespace devtroncd \
--set components.devtron.service.type=NodePort

Devtron dashboard

To access Devtron dashboard when using Minikube as cluster, run the following command:

minikube service devtron-service --namespace devtroncd

To access Devtron dashboard when using Kind/k3s as cluster, run the following command to port forward the devtron service to port 8000:

kubectl -ndevtroncd port-forward service/devtron-service 8000:80

Devtron Admin credentials

kubectl -n devtroncd get secret devtron-secret \
-o jsonpath='{.data.ADMIN_PASSWORD}' | base64 -d

Drawbacks to do not configure the Blob Storage

You will not be able to access the build and deployment logs after an hour.
Build time for commit hash takes longer as cache is not available.
Artifact reports cannot be generated in pre/post build and deployment stages.

7. Integration

Devtron CI/CD with GitOps integration is used to automate the builds and deployments and enables the software development teams to focus on meeting the business requirements, code quality, and security.

Devtron leverages Kubernetes auto-scaling and centralized caching to give you unlimited cost-efficient CI workers.
Supports pre-CI and post-CI integrations for code quality monitoring.
Seamlessly integrates with Clair for image vulnerability scanning.
Supports different deployment strategies: Blue/Green, Rolling, Canary, and Recreate.
Implements GitOps to manage the state of Kubernetes applications.
Integrates with ArgoCD for continuous deployment.
Checks logs, events, and manifests or exec inside containers for debugging.
Provides deployment metrics like; deployment frequency, lead time, change failure rate, and mean-time recovery.
Seamlessly integrates with Grafana for continuous application metrics like CPU and memory usage, status code, throughput, and latency on the dashboard.