Why the Industry Abandoned Traditional Deployments for GitOps (And How to Master Argo CD)

A few years ago, if you asked any engineering team how they deployed to Kubernetes, the answer was almost always the same: a CI pipeline (Jenkins, GitLab CI, GitHub Actions) equipped with a cluster admin kubeconfig, running bash scripts with kubectl apply -f or helm upgrade --install.

It seemed natural at the time. We had spent a decade building push-based deployment pipelines for virtual machines and bare-metal servers, so when Kubernetes emerged, we simply pointed our existing CI pipelines at the Kubernetes API.

Then reality hit. As organizations scaled from a single development cluster to dozens of multi-region, production-grade Kubernetes environments, the traditional push model didn't just bend — it broke completely.

Having architected cloud platforms and enterprise Kubernetes fleets across Austria and Europe, I lived through this transition firsthand. In this article, I want to explore why the industry decisively abandoned traditional push deployments a few years ago, what fundamental problems forced this revolution, and the concrete, battle-tested Argo CD patterns platform teams use today to run production fleets with absolute reliability.

1. The "Old Way": How We Used to Deploy (and Where It Hurt)

In the traditional push model, the CI/CD pipeline acts as an external orchestrator trying to reach into your cluster. The typical lifecycle looked like this:

1. A developer merges code into the main branch.
2. The CI runner compiles the code, executes unit tests, builds a Docker image, and pushes it to a container registry.
3. In the final pipeline stage, the CI runner loads a stored KUBECONFIG secret containing high-privilege credentials.
4. The runner connects across firewalls and the internet to the Kubernetes API server, executes helm upgrade or kubectl apply, and terminates.

When you have 3 microservices on 1 cluster, this feels fine. But once you grow to 40 microservices running across dev, staging, pre-prod, and multi-region production AKS clusters, this architecture creates an operational minefield.

2. The 5 Breaking Points That Forced the Shift to GitOps

Enterprises didn't switch to GitOps because it was a trendy buzzword. They switched because push-based pipelines created five catastrophic bottlenecks that compromised security, reliability, and sanity:

1. The "Cluster-Admin" Security Nightmare

To let CI runners deploy workloads, engineers had to generate service account tokens or certificate keys with broad permissions and paste them into CI environment variables across dozens of repositories. If a rogue npm package, vulnerable Docker image, or misconfigured pull request ran on a CI runner, an attacker had instant cluster-admin access to your production Kubernetes API.

2. Blind Pipelines and the "Snowflake Cluster"

Push pipelines are fire-and-forget. Once kubectl apply exits with code 0, the CI runner disconnects. It has zero awareness of what happens next.

When a production incident occurred at 2:00 AM, engineers would inevitably run kubectl edit deployment or patch an environment variable directly on the live cluster to restore service. Because nobody had time to re-run a 20-minute CI pipeline in the middle of an outage, that hotfix lived only in cluster memory. Three weeks later, a scheduled CI release would deploy and silently overwrite the live hotfix, causing the exact same outage all over again.

3. Multi-Cluster Scaling Paralysis

Managing 20 clusters with push pipelines meant writing complex bash loops inside YAML files. If the network dropped while the CI script was deploying to cluster #7 of 20, your deployment failed mid-way. Half the fleet was running version 1.4, the other half was on version 1.5, and fixing the split-brain required manual intervention.

4. The Disaster Recovery Illusion

Every engineering manager claimed, "Our infrastructure is completely in Git." But when an actual data center outage wiped out a cluster, rebuilding it from push pipelines often took 2 to 3 days. Why? Because hundreds of manually applied ConfigMaps, CRD versions, ingress annotations, and Helm release values had never actually been committed cleanly to Git.

5. Compliance & Audit Friction

Under compliance frameworks (SOC2, ISO 27001, PCI-DSS), you must prove who changed what in production and when. With traditional push pipelines and occasional direct kubectl access, gathering audit trails was an excruciating process of digging through Kubernetes API audit logs and CI run histories.

3. The Core Paradigm Shift: From Push to In-Cluster Pull

The fundamental breakthrough of GitOps was simple: Kubernetes is already designed as a declarative reconciliation engine. Its entire architecture revolves around continuously keeping the live cluster aligned with a desired state.

Instead of using external CI pipelines to push deployments into our clusters with broad credentials, we let an in-cluster controller like Argo CD continuously monitor Git and pull the desired state directly into Kubernetes.

4. The Two-Repository Strategy: Code vs. Fleet Config

When transitioning to GitOps, the single most common mistake teams make is keeping Kubernetes manifests inside the application source code repository.

When CI builds an image, it attempts to commit the new image tag back to the same repo, which triggers a new CI build, causing infinite commit loops and polluting the Git history.

In enterprise architectures, we strictly decouple application code from fleet configuration:

1. Application Repository: Owned by developers. Contains application source code, Dockerfiles, and unit tests. CI tests PRs and pushes immutable, tagged images to Azure Container Registry (ACR).
2. Fleet Configuration Repository: Owned jointly by DevOps and platform teams. Contains environment overlays (Kustomize/Helm) and Argo CD Application manifests.

gitops-fleet/
├── bootstrap/                    # Cluster root bootstrap
│   ├── root-app.yaml             # App-of-Apps entrypoint
│   └── applicationsets/          # Multi-cluster fleet templates
├── platform/                     # Shared platform services
│   ├── cert-manager/
│   ├── cilium-network-policies/
│   ├── external-secrets/
│   └── ingress-nginx/
└── tenants/                      # Business application workloads
    └── payment-service/
        ├── base/
        │   ├── deployment.yaml
        │   ├── service.yaml
        │   └── kustomization.yaml
        └── overlays/
            ├── dev/
            ├── staging/
            └── prod/
                ├── kustomization.yaml
                └── patches/

5. Declarative ApplicationSets: The Git Directory Generator

In production, writing individual static Application manifests for every microservice and environment creates massive boilerplate. If you have 20 microservices running across dev, staging, and prod, you would have to maintain 60 separate YAML files.

Instead, we use Argo CD ApplicationSets with the Git Generator. The Git directory generator continuously scans your Git repository for folder paths. Whenever an engineer commits a new tenant directory or environment overlay, Argo CD automatically discovers it and instantiates the child Application with zero manual intervention.

Here is a battle-tested ApplicationSet using the Git Directory Generator to dynamically discover and deploy all microservices across all environments:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: tenant-applications-fleet
  namespace: argocd
spec:
  # Git Generator: Automatically discovers all tenant overlay directories in Git
  generators:
    - git:
        repoURL: 'https://github.com/enterprise-org/gitops-fleet.git'
        revision: main
        directories:
          - path: 'tenants/*/overlays/*'
  template:
    metadata:
      # {{path[1]}} is the service name (e.g. payment-service), {{path.basename}} is the env (e.g. prod)
      name: '{{path[1]}}-{{path.basename}}'
      namespace: argocd
      finalizers:
        # Cascades resource deletion if the directory is removed from Git
        - resources-finalizer.argocd.argoproj.io
    spec:
      project: default
      source:
        repoURL: 'https://github.com/enterprise-org/gitops-fleet.git'
        targetRevision: main
        path: '{{path}}'
      destination:
        server: 'https://kubernetes.default.svc'
        namespace: '{{path[1]}}-{{path.basename}}'
      syncPolicy:
        automated:
          prune: true          # Automatically delete resources removed from Git
          selfHeal: true       # Revert manual kubectl edits within seconds
          allowEmpty: false    # Prevents deleting everything if Git path is empty
        syncOptions:
          - CreateNamespace=true
          - ApplyOutOfSyncOnly=true  # Only applies resources with actual diffs
          - PruneLast=true           # Ensures new pods are healthy BEFORE pruning
        retry:
          limit: 5
          backoff:
            duration: 5s
            factor: 2
            maxDuration: 3m

6. Solving Ordering & Webhook Drift with Sync Waves

A classic issue when bootstrapping clusters from scratch is dependency ordering. If your Deployment starts before its Namespace, CustomResourceDefinition (CRD), or SecretStore is initialized, the rollout enters a crash loop.

Argo CD solves this natively with Sync Waves using the argocd.argoproj.io/sync-wave annotation:

• Wave -2: Namespaces and PriorityClasses
• Wave -1: CustomResourceDefinitions (CRDs) and SecretStores
• Wave 0: ConfigMaps, Secrets, RBAC Roles, and ServiceAccounts
• Wave 1: Pre-migration Database Jobs (using Sync hooks)
• Wave 2: Core Deployments and StatefulSets
• Wave 3: Ingress routes and HorizontalPodAutoscalers

7. Secrets Management: Why External Secrets Operator Won

You can never commit plaintext secrets to Git. Early in the GitOps movement, many teams adopted Bitnami Sealed Secrets. While Sealed Secrets worked for small projects, it created major operational friction in enterprise environments:

• If you ever had to rebuild a cluster after a disaster and lost the original cluster master private key, all your SealedSecret manifests in Git became unrecoverable.
• Developers had to use a specialized CLI (kubeseal) every single time they needed to rotate a database credential.

In enterprise architectures, External Secrets Operator (ESO) has become the gold standard. Secrets remain centrally managed, audited, and rotated in Azure Key Vault (or AWS Secrets Manager / HashiCorp Vault).

In Git, we only commit a declarative ExternalSecret manifest that references the vault key. ESO authenticates via Azure Workload Identity (zero static cloud credentials stored in the cluster) and synchronizes the real secret into a native Kubernetes Secret inside the cluster at runtime.

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: payment-db-credentials
  namespace: payments
spec:
  refreshInterval: 1h             # Automatically sync rotated secrets from Key Vault every hour
  secretStoreRef:
    name: azure-keyvault-store
    kind: ClusterSecretStore
  target:
    name: db-credentials         # Name of the native K8s secret generated in the cluster
    creationPolicy: Owner
  data:
    - secretKey: DB_PASSWORD      # Key inside the Kubernetes secret
      remoteRef:
        key: prod-payment-db-pass # Secret name in Azure Key Vault

8. Scaling to Fleets with ApplicationSets

When you manage a fleet of 10, 30, or 100 Kubernetes clusters across multiple geographic regions, maintaining individual Application files for every microservice on every cluster is completely unsustainable.

The ApplicationSet Controller allows you to define one template and use generators to stamp out applications automatically across clusters based on cluster labels (e.g. env: prod, region: westeurope).

Any time you spin up a new AKS cluster tagged with env: production using Terraform, Argo CD immediately discovers the cluster and deploys the entire monitoring, security, and networking baseline without touching a single manifest by hand.

Final Thoughts & What's Next

The shift from traditional push deployments to GitOps wasn't just a change in tooling — it was an architectural maturity milestone for the cloud-native ecosystem. By making Git the single source of truth and letting Argo CD continuously enforce desired state, engineering teams eliminate snowflake clusters, gain automated compliance audits, and sleep peacefully during releases.

If you're currently planning a cloud-native migration, setting up multi-cluster AKS infrastructure, or looking to streamline your Kubernetes platform with Argo CD, feel free to reach out. I'm always open to discussing real-world platform architecture.