Deploying GCP Cloud Function to rotate the service account key stored in Secret Manager

Terraform on Google Cloud V1.4 — Cloud Functions

In DevOps, secrets are like keys that unlock important resources and sensitive data. If these secrets are not securely managed, it can lead to unauthorized access, data breaches, and financial losses.

That’s why DevOps teams need to prioritize protecting sensitive information, such as API keys, passwords, and cryptographic keys.

In the last blog post of my blog series, Terraform on Google Cloud, I talked about leveraging GCP Secret Manager to store the service account key that can be accessed securely by GKE, CI workflows and applications.

In this blog I will talk about rotating the service account key stored in Secret Manager using Cloud Functions (Gen 1) and how to deploy the Cloud Function with Terraform.

What are Cloud Functions?

Cloud Functions is an event-driven serverless compute platform. Cloud Functions allows you to write your code without worrying about provisioning resources or scaling to handle changing requirements.

Cloud Functions allow developers to write and deploy lightweight code snippets or functions that are executed in response to events or triggers, such as HTTP requests, Pub/Sub messages, or changes in storage buckets.

Why Cloud Functions?

  • Serverless
  • Event-driven
  • Scalable
  • Cheap
  • Easily integrated with other GCP services

When to use Cloud Functions

  • When you need to respond to events or triggers, such as HTTP requests, database changes, or file uploads
  • When you need to process small amounts of data, and perform quick functions such as rotating the secrets on a schedule
  • When you need cost-optimized stateless operations

When to avoid Cloud Functions

  • When you need stateful operations
  • When your tasks require longer execution times – use Cloud Run instead
  • When your application requires complex networking configurations

Best Practices for using Cloud Functions

  • Keep it small
  • Use environment variables
  • Implement logging and error handling
  • Optimize function performance
  • Monitor and manage costs

Deploying Cloud Function – below steps will be required for this

Write Python code to rotate the service account key, and define dependencies

Before we start, let’s create a directory functions in the root path and key-rotation directory inside functions – functions/key-rotation. Create main.py and requirements.txt under it.

main.py

import os
import json
import base64
from google.cloud import secretmanager_v1 as secretmanager
from google.cloud import iam_admin_v1
from google.oauth2 import service_account

def rotate_service_account_key(event, context):
    """Cloud Function to rotate service account keys stored in Secret Manager"""
    
    # Get configuration from environment variables
    project_id = os.environ.get('PROJECT_ID')
    secret_id = os.environ.get('SECRET_ID', 'artifact-registry-sa-key')
    service_account_email = os.environ.get('SERVICE_ACCOUNT_EMAIL')
    
    if not all([project_id, service_account_email]):
        print("Missing required environment variables")
        return "Error: Missing configuration"
    
    try:
        # Create clients
        secret_client = secretmanager.SecretManagerServiceClient()
        iam_client = iam_admin_v1.IAMClient()
        
        print(f"Starting key rotation for service account: {service_account_email}")
        
        # Create new service account key
        service_account_path = f"projects/{project_id}/serviceAccounts/{service_account_email}"
        
        new_key_request = iam_admin_v1.CreateServiceAccountKeyRequest(
            name=service_account_path,
            private_key_type=iam_admin_v1.ServiceAccountPrivateKeyType.TYPE_GOOGLE_CREDENTIALS_FILE
        )
        
        new_key = iam_client.create_service_account_key(request=new_key_request)
        new_key_data = base64.b64decode(new_key.private_key_data).decode('utf-8')
        
        print("New service account key created successfully")
        
        # Store new key in Secret Manager
        secret_name = f"projects/{project_id}/secrets/{secret_id}"
        
        # Add new secret version
        secret_client.add_secret_version(
            request={
                "parent": secret_name,
                "payload": {
                    "data": new_key_data.encode("UTF-8")
                }
            }
        )
        
        print("New key stored in Secret Manager")
        
        # Get all versions and disable old ones (keep latest 2 versions)
        versions = secret_client.list_secret_versions(
            request={"parent": secret_name}
        )
        
        enabled_versions = []
        for version in versions:
            if version.state == secretmanager.SecretVersion.State.ENABLED:
                enabled_versions.append(version)
        
        # Sort by create time (newest first)
        enabled_versions.sort(key=lambda x: x.create_time, reverse=True)
        
        # Disable old versions (keep latest 2)
        for version in enabled_versions[2:]:
            secret_client.disable_secret_version(
                request={"name": version.name}
            )
            print(f"Disabled old secret version: {version.name}")
        
        # Delete old service account keys (keep only the new one)
        keys = iam_client.list_service_account_keys(
            request={"name": service_account_path}
        )
        
        for key in keys.keys:
            # Skip the key we just created
            if key.name != new_key.name and key.key_type == iam_admin_v1.ServiceAccountKey.KeyType.USER_MANAGED:
                try:
                    iam_client.delete_service_account_key(
                        request={"name": key.name}
                    )
                    print(f"Deleted old service account key: {key.name}")
                except Exception as e:
                    print(f"Failed to delete key {key.name}: {e}")
        
        print("Service account key rotation completed successfully")
        return "Service account key rotation completed."
        
    except Exception as e:
        print(f"Error during key rotation: {e}")
        return f"Error: {e}"

requirements.txt

# Client library for interacting with Google Cloud Secret Manager
google-cloud-secret-manager==2.16.4

# Library for authenticating requests to Google APIs
google-auth==2.23.4

# IAM client for managing service account keys
google-cloud-iam==2.12.0

Create a file key-rotation.tf on root dir and paste below code in that one

Zip the code files

data "archive_file" "key_rotation" {
  type        = "zip"
  output_path = "${path.module}/functions/key-rotation.zip"

  dynamic "source" {
    for_each = [
      "${path.module}/functions/key-rotation/main.py",
      "${path.module}/functions/key-rotation/requirements.txt"
    ]

    content {
      content  = file(source.value)
      filename = basename(source.value)
    }
  }
}

Create a storage bucket and storage object with zipped code file

resource "google_storage_bucket" "cf_bucket" {
  project                     = var.project_id
  name                        = "${var.project_id}-cf-archive"
  uniform_bucket_level_access = true
  location                    = var.region
  public_access_prevention    = "enforced"

  versioning {
    enabled = true
  }

  labels = {
    environment = var.environment
    purpose     = "cloud-functions"
    managed-by  = "terraform"
  }
}

# Bucket object to hold the code for Cloud Function
resource "google_storage_bucket_object" "cf_bucket_object" {
  name   = "key-rotation-${data.archive_file.key_rotation.output_md5}.zip"
  bucket = google_storage_bucket.cf_bucket.name
  source = data.archive_file.key_rotation.output_path
}

Create a Pub/Sub topic for the Cloud Function

# Pub/Sub topic to trigger Cloud Function
resource "google_pubsub_topic" "secret_rotation" {
  name                       = "secret-rotation-topic"
  message_retention_duration = "86400s"  # 24 hours
  project                    = var.project_id

  labels = {
    environment = var.environment
    purpose     = "secret-rotation"
    managed-by  = "terraform"
  }
}

Create service account with access to Secret Manager and service account key creation

# Service account to run the Cloud Function
resource "google_service_account" "key_rotation_cf_sa" {
  project      = var.project_id
  account_id   = "key-rotation-cf-sa"
  display_name = "Key Rotation Cloud Function Service Account"
  description  = "Service account for the key rotation Cloud Function"
}

# Grant necessary permissions to the service account
resource "google_project_iam_member" "key_rotation_permissions" {
  for_each = toset([
    "roles/secretmanager.admin",
    "roles/iam.serviceAccountKeyAdmin",
    "roles/logging.logWriter",
    "roles/monitoring.metricWriter"
  ])

  project = var.project_id
  role    = each.value
  member  = "serviceAccount:${google_service_account.key_rotation_cf_sa.email}"
}

# Allow the Cloud Function to be invoked by Pub/Sub
resource "google_cloudfunctions_function_iam_member" "invoker" {
  project        = var.project_id
  region         = var.region
  cloud_function = google_cloudfunctions_function.key_rotation_cf.name
  role           = "roles/cloudfunctions.invoker"
  member         = "serviceAccount:service-${data.google_project.current.number}@gcp-sa-pubsub.iam.gserviceaccount.com"
}

# Get current project information
data "google_project" "current" {
  project_id = var.project_id
}

Important Note: The original blog had IAM role assignment errors. The correct way to assign IAM roles to Cloud Functions is through google_project_iam_member for project-level permissions, not google_cloudfunctions_function_iam_member for function-specific roles like secretmanager.admin and iam.serviceAccountKeyAdmin.

Create the Cloud Function with Pub/Sub trigger

# Cloud Function (Gen 1)
resource "google_cloudfunctions_function" "key_rotation_cf" {
  name        = "key-rotation-function"
  project     = var.project_id
  region      = var.region
  runtime     = "python39"
  entry_point = "rotate_service_account_key"

  available_memory_mb   = 512
  timeout               = 540
  max_instances         = 10
  
  source_archive_bucket = google_storage_bucket.cf_bucket.name
  source_archive_object = google_storage_bucket_object.cf_bucket_object.name
  
  service_account_email = google_service_account.key_rotation_cf_sa.email

  event_trigger {
    event_type = "google.pubsub.topic.publish"
    resource   = google_pubsub_topic.secret_rotation.name
    failure_policy {
      retry = true
    }
  }

  environment_variables = {
    PROJECT_ID            = var.project_id
    SECRET_ID             = "artifact-registry-sa-key"
    SERVICE_ACCOUNT_EMAIL = google_service_account.artifact_registry_sa.email
  }

  labels = {
    environment = var.environment
    purpose     = "secret-rotation"
    managed-by  = "terraform"
  }

  depends_on = [
    google_project_iam_member.key_rotation_permissions,
    google_storage_bucket_object.cf_bucket_object
  ]
}

Create a Cloud Scheduler with Pub/Sub as target

# Create a scheduler (cron job)
resource "google_cloud_scheduler_job" "secret_rotation" {
  name        = "secret-rotation-job"
  project     = var.project_id
  region      = var.region
  description = "Triggers Cloud Function to rotate service account keys"
  schedule    = "0 2 1 * *"  # At 02:00 on day-of-month 1 (1st of every month)
  time_zone   = "UTC"

  pubsub_target {
    topic_name = google_pubsub_topic.secret_rotation.id
    data       = base64encode(jsonencode({
      message = "trigger service account key rotation",
      timestamp = timestamp()
    }))
  }

  retry_config {
    retry_count = 3
  }

  depends_on = [
    google_pubsub_topic.secret_rotation,
    google_cloudfunctions_function.key_rotation_cf
  ]
}

Add required variables

Add these variables to your variables.tf:

variable "key_rotation_schedule" {
  type        = string
  description = "Cron schedule for key rotation (default: 1st of every month at 2 AM UTC)"
  default     = "0 2 1 * *"
}

variable "cf_timeout" {
  type        = number
  description = "Cloud Function timeout in seconds"
  default     = 540
  validation {
    condition     = var.cf_timeout >= 60 && var.cf_timeout <= 540
    error_message = "Cloud Function timeout must be between 60 and 540 seconds."
  }
}

Add outputs

Add these outputs to your outputs.tf:

# Cloud Function outputs
output "key_rotation_function_name" {
  value       = google_cloudfunctions_function.key_rotation_cf.name
  description = "Name of the key rotation Cloud Function"
}

output "key_rotation_function_url" {
  value       = google_cloudfunctions_function.key_rotation_cf.https_trigger_url
  description = "HTTPS trigger URL for the Cloud Function (if applicable)"
}

output "secret_rotation_topic" {
  value       = google_pubsub_topic.secret_rotation.name
  description = "Name of the Pub/Sub topic for secret rotation"
}

output "scheduler_job_name" {
  value       = google_cloud_scheduler_job.secret_rotation.name
  description = "Name of the Cloud Scheduler job"
}

output "next_rotation_schedule" {
  value       = var.key_rotation_schedule
  description = "Cron schedule for the next key rotation"
}

That’s it. Now it will rotate the key on the 1st of every month at 2 AM UTC.

Deploy the Cloud Function

Just go ahead with deploying everything by running Terraform commands or a CI workflow – I talked about both approaches in my previous blogs.

Local deployment

# Format and validate
terraform fmt -recursive
terraform validate

# Plan and apply
terraform plan -out=tfplan
terraform apply tfplan

Test your Cloud Function

After deployment, you can test the function manually:

# Trigger the function manually via Pub/Sub
gcloud pubsub topics publish secret-rotation-topic --message="manual test"

# Check function logs
gcloud functions logs read key-rotation-function --region=us-central1

# Check the scheduler job
gcloud scheduler jobs list --location=us-central1

# Describe the scheduler job
gcloud scheduler jobs describe secret-rotation-job --location=us-central1

Monitor your function

# View function details
gcloud functions describe key-rotation-function --region=us-central1

# Check recent executions
gcloud functions logs read key-rotation-function --region=us-central1 --limit=50

# Test the scheduler
gcloud scheduler jobs run secret-rotation-job --location=us-central1

Troubleshooting Common Issues

IAM Permission Errors

If you encounter the errors mentioned in the feedback:

Error 400: Role roles/secretmanager.admin is not supported for this resource.
Error 400: Role roles/iam.serviceAccountKeyAdmin is not supported for this resource.

The issue is that these roles should be assigned at the project level using google_project_iam_member, not at the function level using google_cloudfunctions_function_iam_member. The corrected code above addresses this issue.

Function Timeout Issues

If your function times out, increase the timeout value:

resource "google_cloudfunctions_function" "key_rotation_cf" {
  # ... other configuration
  timeout = 540  # Maximum allowed for Gen 1 functions
}

Secret Manager Access Issues

Ensure your service account has the correct permissions:

# Check current IAM policy
gcloud projects get-iam-policy YOUR_PROJECT_ID

# Verify service account permissions
gcloud iam service-accounts get-iam-policy key-rotation-cf-sa@YOUR_PROJECT_ID.iam.gserviceaccount.com

Cloud Functions Generation 1 vs Generation 2

In this blog, we used Cloud Functions Generation 1, which is the original Cloud Functions runtime. However, Google Cloud has evolved and now offers Cloud Functions Generation 2, also known as Cloud Run Functions.

Key Differences:

Cloud Functions Gen 1:

  • Based on the original Cloud Functions runtime
  • Limited to 540 seconds maximum execution time
  • Fewer configuration options
  • Event-driven with specific trigger types

Cloud Functions Gen 2 (Cloud Run Functions):

  • Built on Cloud Run infrastructure
  • Support for up to 60 minutes execution time
  • More configuration flexibility
  • Better performance and scaling
  • Support for more trigger types and HTTP/2

What’s Next?

In the next blog post, I will show you how to migrate this same key rotation function to Cloud Functions Generation 2 (Cloud Run Functions) and take advantage of the improved performance, longer execution times, and better scaling capabilities.

We’ll also explore:

  • Advanced scheduling options
  • Better error handling and retry mechanisms
  • Integration with Cloud Monitoring and Alerting
  • Cost optimization strategies

What we accomplished

In this blog, we successfully:

✅ Deployed Cloud Functions Gen 1 for automated key rotation
✅ Fixed common IAM permission issues with proper role assignments
✅ Set up automated scheduling with Cloud Scheduler
✅ Implemented proper error handling and logging
✅ Created a complete key rotation workflow that runs monthly
✅ Integrated with Secret Manager from our previous setup

Thank you for reading. See you in the next one where we’ll explore Cloud Functions Gen 2!

Connect with me:

Tags: #GoogleCloud #CloudFunctions #Terraform #SecretManager #DevOps #Infrastructure #Automation #Security

Akhilesh Mishra

Akhilesh Mishra

I am Akhilesh Mishra, a self-taught Devops engineer with 11+ years working on private and public cloud (GCP & AWS)technologies.

I also mentor DevOps aspirants in their journey to devops by providing guided learning and Mentorship.

Topmate: https://topmate.io/akhilesh_mishra/