Unified Cloud Policy Management for Enterprises in a Multi-Cloud Environment

In the current digital landscape, enterprises are increasingly adopting multi-cloud strategies to capitalize on diverse cloud provider strengths and mitigate vendor lock-in risks. However, this approach introduces new challenges, particularly in policy management and compliance across heterogeneous cloud environments. This blog post explores the need for unified cloud policy management and suggests Cloud Custodian (c7n) as a possible solution for enterprises navigating complex multi-cloud setups[1]. Consistent policy management can also have a positive impact on cloud cost and therefore contribute to the FinOps domain. Examples can be found at the end of the first and second chapter.

In this blogpost it will be discussed, why a unified cloud policy management is necessary. After establishing the need, c7n will be discussed as a possible tool and distinguished strengths are pointed out. Followed by a rough description for an enterprise level implementation based on an actual use case.

The Need for Unified Cloud Policy Management

As organizations expand their cloud footprint across multiple providers, additional governance becomes necessary. Some policies, like companywide requirements or security policies, need to be enforced top down. Other policies might be specific to certain departments or cross functional units. Lastly, certain applications or team specific infrastructure also requires very specific policies. There can be any number of levels specified, but for simplicity of this article we will think of three levels outlined in Figure 1:

Policy application levels

To cover tenant wide policies, one could think of AWS Service Control Policies (SCPs) or Azure Policies on root level. But with multiple cloud providers in play, relying solely on provider-specific mechanisms like SCPs or Azure Policies is insufficient, since they require separate management and do not allow for domain specific (eg.: compute services, iam, ... across cloud providers) policies. Enterprises with high cloud spend need a unified approach that can enforce policies consistently across different cloud platforms without being tied to any single provider's proprietary tools.

Different cloud services and domains often require tailored policy approaches. A unified solution should allow for domain-specific policy management, enabling organizations to craft and enforce policies that cater to the unique requirements of each cloud service they use.

Furthermore, it's crucial to have advanced governance features like policy management, compliance monitoring, and automatic remediation capabilities at the individual resource level. Without automating those topics, they are not maintainable in big organizations.

When we think of individual applications, one might think of Infrastructure as Code (IaC) as the solution for granular policy management, since tools like AWS CloudFormation, Azure Resource Manager, Terraform or Pulumi are invaluable for deploying and managing cloud resources. But they do not provide the broader view across an organization, needed for efficient policy and compliance management.

Managing policies across multiple cloud environments can quickly become complex and error prone. A centralized system for policy management and maintenance is essential for maintaining consistency, reducing administrative overhead, and ensuring that all cloud resources adhere to organizational standards.

To address these issues on the third level, highly automated governance tools are a necessary to apply a granular approach that helps maintaining security and compliance standards across diverse cloud environments.

Automated policy management can also be used to upgrade and automate FinOps practices. Two possible scenarios are time-based deactivation and tagging enforcement policies. Time-based deactivation can significantly reduce cloud costs without manual intervention, potentially cutting compute costs by up to 76.2% for development environments, by reducing uptime to 8 hours on workdays.

Automated tagging enforcement ensures consistent resource tagging, enabling precise cost allocation and detailed spending analysis. By automating these policies, organizations can maintain continuous compliance with FinOps best practices, reduce human error, and allow FinOps teams to focus on strategic cost optimization.

Why Use Cloud Custodian (c7n)?

As a Cloud Native Computing Foundation (CNCF) Incubating project, Cloud Custodian benefits from the support and scrutiny of a vibrant open-source community. This status ensures ongoing development, security updates, and alignment with cloud-native best practices.

Cloud Custodian integrates seamlessly with major cloud providers and Kubernetes, making it an ideal choice for multi-cloud environments. This broad compatibility allows enterprises to implement consistent policies across their entire cloud infrastructure.

Cloud Custodian enables a 'policy as code' approach, allowing organizations to define, version, and manage their policies centrally. This distribution can be applied at different levels: centrally, as described in the setup section below, and decentralized, within CI/CD pipelines for individual projects or teams.

Cloud Custodian can validate IaC compliance within GitOps pipelines before deployment. This pre-deployment check ensures that new resources or changes align with organizational policies from the outset. Policies can be distributed via custom GitOps resources, integrating smoothly with tools like GitHub Actions.

Unlike some other policy management tools such as Open Policy Agent (OPA), Cloud Custodian offers automatic remediation capabilities. This feature allows organizations to not only detect policy violations but also automatically correct them, reducing manual intervention and improving overall security posture.

When compared to commercial solutions like Wiz, Cloud Custodian offers a non-paid, open-source alternative that can be more lightweight and customizable. Commercial solutions usually offer a managed solution with rich feature support. While c7n may require more setup and maintenance, it provides greater flexibility and control over policy management processes.

With regards to the example FinOps Policies for time-based deactivation[2] and tagging enforcement[3] mentioned in the end of the last chapter sample Cloud Custodian implementations can be found in the links in the footnotes.

Setting Up Cloud Custodian in an Enterprise Environment

To leverage Cloud Custodian effectively in an enterprise setting, policies can be distributed and enforced either centrally or decentral. We will focus on the central perspective mostly and give a short outlook on the decentral components in the end.

Enterprises can establish a central repository for defining and managing all cloud governance policies. This should be set up at a cross-provider Cloud Foundation level, enabling an integration of all used cloud providers into the Cloud Custodian setup and allowing deployment of policies and related resources across the organization. This centralized approach ensures consistency and simplifies management.

GitOps principles should be used to automate policy deployment and guarantee maintainability. In order to manage multi-account organizations, Cloud Custodian offers its own multi-account orchestration tool called c7n-org which can be deployed from a central account for each cloud provider. Centralizing the administration into one account per cloud provider also simplifies IAM.

By implement authentication solutions like OIDC (OpenID Connect) into their CI/CD pipelines security into the central account can be ensured. Roles can be set up to switch from the central management account into linked accounts used by the application teams. To enforce the principle of least privilege, Cloud Custodian deployment roles can be administered centrally and restricted to necessary actions. This approach enhances security by ensuring that each policy has a minimal permissions set.

Figure 2 shows a sample architecture for deploying c7n-org from a central GitHub Actions pipeline onto AWS and Azure:

Sample Architecture with two public cloud providers

Another advantage of the centralized Cloud Custodian set up is the possibility of centralized monitoring. It can be integrated with different monitoring and alerting solutions. One prominent solution from the CNCF landscape is Prometheus. By specifying the corresponding metrics endpoints in the Cloud Custodian policies metrics can be pulled directly and consistently to ensure a comprehensive monitoring across all managed resources. Cloud Custodian provides its own unified execution metrics across cloud providers.

As mentioned before, policies can also be enforced decentrally. One approach could be to distribute centrally administered Cloud Custodian policies, via GitOps pipeline integrations like GitHub Actions. This way, policy verification can be shifted left without requiring access to individual cloud accounts. While those approaches differ technically, they can very well be combined into a comprehensive suite of cloud governance policies.

Cloud Custodian (c7n) can be a powerful solution for enterprises seeking unified cloud policy management in multi-cloud environments. Its open-source nature, broad compatibility, and flexible deployment options make it an attractive choice for organizations of all sizes.

By implementing Cloud Custodian with a centralized, GitOps-driven approach, enterprises can achieve consistent policy enforcement, improved compliance, and automated remediation across their diverse cloud environments. This unified approach not only enhances security and compliance but also streamlines operations and reduces the complexity of managing multi-cloud setups.

As cloud adoption continues to grow, tools like Cloud Custodian will play an increasingly crucial role in helping organizations maintain control, security, and efficiency across their cloud infrastructure.

---

Helpful Links:

[1] https://cloudcustodian.io/

[2] https://cloudcustodian.io/docs/usecases/ec2offhours.html

[3] https://cloudcustodian.io/docs/usecases/tagcompliance.html