What is a secret

In this blog, we will show you how HashiCorp Vault can help you manage and eliminate secrets sprawl in Azure and your broader organization in general. Before we dive into defining what secret sprawl is, however, it’s good to understand what we define as a secret. A secret is usually one of two things:

  • Anything that authenticates a person or program to a protected system (usually a string of UTF-8 characters).
  • An encryption key or key pair which provides a means to encrypt/decrypt data.

Some examples of secrets include:

  • Username & password
  • Keypad pin sequence
  • TLS Certificates
  • API Tokens
  • Symmetric encryption keys

Some would argue that data contained in protected systems or stored encrypted in a data store are also secrets, but access and encryption methods are upstream from this data and are simpler to store and manage.

What is secrets sprawl?

Using this definition of a secret, secrets sprawl is the surface area on which an organization’s secrets are stored and the volume of these secrets stored on this surface area. It is still common for secrets to be stored in plain text in a variety of different places such as in files on people’s computers, in version control systems like Azure DevOps or Github, in configuration management platforms like Chef, Puppet or Ansible or embedded in CI/CD pipelines like CircleCI or Jenkins. It also isn’t uncommon to find secrets on the occasional sticky note or piece of paper taped onto the bottom of a keyboard. If you read news headlines regularly, you will sometimes hear about cases of plain text secret storage at all manner or organizations (e.g., Facebook Stored Hundreds of Millions of User Passwords in Plain Text for Years — Krebs on Security).

As operators and developers get more sophisticated, they have turned to better secret storage techniques like encrypted drives, using Gnu Privacy Guard (GPG) keys or Cloud Key Management Service (KMS) keys to encrypt/decrypt files. These methods are more secure, but they still often lead to secrets sprawl if not managed properly.

As secrets sprawl out across your organization, it’s difficult to answer questions like:

  • Where are all my secrets?
  • Who is accessing them and when?
  • Are they still there? When did they get there? Have they changed? When were they removed?
  • How do I rotate all of these secrets to match security policies?

How HashiCorp Vault manages secrets

HashiCorp Vault is a highly scalable, highly available, environment agnostic way to generate, manage, and store secrets. It encrypts data using the Advanced Encryption Standard (AES) using 256 bits in Galois/Counter Mode (GCM). This means it is both highly secure and highly performant.

Once data is encrypted it is stored on a variety of storage backends such as HashiCorp Consul (most recommended), Azure Storage Container and a number of other databases and datastores. Check the Storage Backends – Configuration document for in-depth information on specific backends and high availability support.

Here are some of the features of Vault which enable a stronger workflow for controlling access to sensitive data and secrets.

Authentication methods

Vault offers flexible authentication methods which allow users to access Vault using trusted sources of identity. Users leverage tools like LDAP, GitHub Tokens or Username & Password.

For machines and applications, Vault supports flexible methods like Azure Active Directory/MSI, Kubernetes JWT, TLS Certificates and Vault’s AppRole method. Regardless of authentication method chosen, users or applications will be granted a Vault Token.

Policies

Vault Tokens are scoped using policies. Policies authorize a user or application to access specific paths within Vault. Paths are created on a Radix tree for highly performant lookups of secrets and mapping of policies. Authentication methods, secret engines and administrative functions are mapped to paths making it simple to define authorization to all of Vault through one mechanism. The default action for a policy is to deny access until paths are whitelisted. There are wildcard functions making it easier to allow access to certain path structures.

Let’s look at an example policy granting access to both secret engine path (kv) and management endpoints.

# List, create, update, and delete key/value secrets
path "kv/*"
{
  capabilities = ["create", "read", "update", "delete", "list", "sudo"]
}
 
# Manage secret engines broadly across Vault
path "sys/mounts/*"
{
  capabilities = ["create", "read", "update", "delete", "list", "sudo"]
}
 
# List existing secret engines
path "sys/mounts"
{
  capabilities = ["read"]
}

Leveraging Vault policies helps organizations trust that users and applications only have access to credentials that they are approved for.

Secret Engines

Vault offers a wide array of Secrets Engines that go far beyond just basic K/V management. Vault Secrets Engines can manage dynamic secrets on certain technologies like Azure Service Principles and Databases and Datastores. These secrets are both time and access bound, which often eliminates the need to rotate secrets. Dynamic secrets help reduce the blast damage of any leaked secrets or compromised systems because every authenticated entity will have a unique set of credentials.

Vault can also be used to generate and deploy PKI Certificates and Signed SSH Certificates.

Lastly, Vault provides Encryption as a Service via the Transit Engine. This alleviates the need to integrate complex encryption libraries into application code. Payloads, for example a JSON file, can be sent to Vault and are then encrypted to be stored at rest. When retrieved, Vault will decrypted them and display it to applications on the fly. Because keys are versioned, it’s easy to roll re-encryption of data as necessary for regulatory purposes.

Vault and Azure-specific integrations

Azure users can leverage all of these Vault features to automate their secrets management and retrieval through Azure specific integrations. First and foremost Vault can be automatically unsealed using KMS keys from Azure Key Vault. Next, MSI credentials can be used to authenticate systems and applications preventing the need to distribute initial access credentials. Lastly, Vault can dynamically generate Azure Service Principals and role assignments. This allows users and applications off-cloud an easy method for generating flexible time and permission bound access into Azure APIs.

More information on HashiCorp Vault and Azure integrations can be found on the Hashicorp/Azure Integrations page.

How to get started

If you would like a quick way of testing out Vault in Azure, this GitHub repo contains all the code to create a Vault environment in Azure including all instructions on how to obtain Terraform, run it, connect to your Azure instance and run the Vault commands. This is a great way to learn the concepts covered here with a low barrier to entry.

For more a more extensive, curated approach to learning Vault, HashiCorp offers an extensive curriculum in the HashiCorp Learn site. Here you can learn everything from basic system setup and secrets management to advanced topics like distributed architectures and entity management.

Additional Resources

To see demonstrations of some of the integrations mentioned earlier, watch the webinar that inspired this blog post: Understanding and Eliminating Secret Sprawl Using HashiCorp Vault.

 

Questions or feedback? Let us know in the comments below. 

Follow Jake @JakeLundberg