How to List All Projects in Google Cloud Organization Using Python and gcloud

GCP Project: Listing all projects under a Google Cloud organization with Python

When working on Google Cloud Platform, we often need to list all GCP projects under an organization. This is a common task for cloud administrators, and there are two main approaches to do this: using the gcloud command-line tool or Python with the Resource Manager API.

In this comprehensive guide, I’ll walk you through both methods step-by-step, focusing on how to efficiently retrieve a complete list of all projects across your entire GCP organization.

But before we start, let me talk about GCP Org Hierarchy, which is important to understand before we write the code.

GCP Resource Hierarchy Structure

1. Organization (Blue) – The top-level node and root of the hierarchy
   • Represents an entire company or organization
   • Allows centralized visibility and control
2. Folders (Orange) – Grouping mechanism for organizing resources
   • Can be nested (like Team A folder inside Development)
   • Often represent departments, teams, or environments
   • Create logical boundaries for applying policies
3. Projects (Green) – Containers for Google Cloud resources
   • All resources must belong to a project
   • Projects can exist directly under an organization or within folders
   • Provide isolation boundaries for billing and access control
4. Resources (Red) – The actual cloud services
   • Virtual machines, databases, storage buckets, etc.
   • Always belong to a project

Key Points About the Hierarchy

• Inheritance: IAM policies and organization policies flow down the hierarchy
• Flexibility: Projects can be directly under the organization or nested in folders
• Isolation: Projects provide isolation for resources and billing
• Organization: The diagram shows common patterns like having separate folders for Development, Production, and Infrastructure

Using gcloud to List All Projects in an Organization

The quickest way to list all projects in your GCP organization is using the gcloud command-line tool. Let’s start with this approach.

# List all projects in the organization
gcloud projects list --filter="parent.id:[ORGANIZATION_ID]" --format="table(projectId,name,projectNumber)"

Replace [ORGANIZATION_ID] with your actual organization ID (without the “organizations/” prefix).

This command will list all projects that are directly under your organization, but it won’t show projects nested inside folders. To get a complete list including all projects in folders, we need a different approach.

Why Python is Better for Listing All Projects in a GCP Organization

While the gcloud command is simple, it has limitations when dealing with complex organizational structures. Here’s why Python with the Resource Manager API is better:

1. Complete hierarchy traversal: Python can recursively go through all folders and subfolders
2. Customizable output: You can format and filter the results exactly as needed
3. Automation-friendly: Easy to integrate with other systems and scheduled tasks
4. Complex filtering: Apply advanced filters based on any project property

Using Python to List All Projects in a GCP Organization

Now, let’s dive into the Python approach, which will give us a complete view of all projects across the entire organization hierarchy.

Step 1: Create a Service Account with the Right Permissions

First, we need to create a service account with appropriate permissions to view the organization structure:

# Set the project in cloud shell
gcloud config set project someproject

# Create a service account
gcloud iam service-accounts create rm-svc \
    --description="Service account for listing GCP projects" \
    --display-name="resource-manager-list"

# Assign organization-level permission to the service account
gcloud organizations add-iam-policy-binding \ 
ORGANIZATION_ID --member="serviceAccount:rm-svc@someproject.iam.gserviceaccount.com" \
 --role="roles/resourcemanager.folderViewer"

# Create a key for the service account
gcloud iam service-accounts keys create rm-svc.json --iam-account  \
rm-svc@someproject.iam.gserviceaccount.com

The key file will look something like this (with actual values instead of placeholders):

{
  "type": "service_account",
  "project_id": "someproject",
  "private_key_id": "...",
  "private_key": "...",
  "client_email": "rm-svc@someproject.iam.gserviceaccount.com",
  "client_id": "...",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://oauth2.googleapis.com/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "..."
}

Note: There are many better ways to run the code than creating keys, such as using short-lived credentials, but I want to keep it simple for demo purposes.

Step 2: Authenticate as the Service Account

Next, we’ll impersonate the service account to run our Python code:

gcloud auth activate-service-account --key-file=rm-svc.json

Step 3: Install the Required Python Library

We need to install the Python client for the Resource Manager:

# Assuming you already have Python and pip installed
pip install google-cloud-resource-manager 

Step 4: Write the Python Script to List All GCP Projects

Create a file named list_all_gcp_projects.py with the following code:

from google.cloud import resourcemanager_v3

def get_folders(
    parent_id = "organizations/ORGANIZATION_ID",
    folders = None):
    """
    Recursively retrieves all folders in the organization.
    
    This function walks through the entire folder hierarchy starting
    from the organization and collects all folder IDs, including nested folders.
    
    Args:
        parent_id: The starting point (organization ID or folder ID)
        folders: Accumulator list for recursion
        
    Returns:
        List containing all folder IDs in the organization
    """
    if folders is None:
        folders = []

    # Create a client for accessing folders
    client = resourcemanager_v3.FoldersClient()
    
    # Prepare the request to list folders under the parent
    request = resourcemanager_v3.ListFoldersRequest(
        parent=parent_id,
    )

    # Get all folders under the parent
    page_result = client.list_folders(request=request)
    
    # For each folder found, add it to our list and then search its subfolders
    for folder in page_result:
        folders.append(folder.name)
        # Recursive call to find subfolders
        get_folders(parent_id=folder.name, folders=folders)
        
    return folders

def search_projects(folder_id):
    """
    Finds all projects under a specific folder.
    
    Args:
        folder_id: The folder ID to search under
        
    Returns:
        List of project objects found in the folder
    """
    # Create a client for accessing projects
    client = resourcemanager_v3.ProjectsClient()

    # Create a query to find projects under the specified folder
    query = f"parent:{folder_id}"
    request = resourcemanager_v3.SearchProjectsRequest(query=query)
    
    # Get all projects under the folder
    page_result = client.search_projects(request=request)
    
    # Collect all projects into a list
    search_result = []
    for project in page_result:
        search_result.append(project)
        
    return search_result

def search_org_projects(org_id):
    """
    Finds all projects directly under the organization (not in folders).
    
    Args:
        org_id: The organization ID to search under
        
    Returns:
        List of project objects found directly under the organization
    """
    # Create a client for accessing projects
    client = resourcemanager_v3.ProjectsClient()

    # Create a query to find projects directly under the organization
    query = f"parent:{org_id}"
    request = resourcemanager_v3.SearchProjectsRequest(query=query)
    
    # Get all projects under the organization
    page_result = client.search_projects(request=request)
    
    # Collect all projects into a list
    search_result = []
    for project in page_result:
        search_result.append(project)
        
    return search_result

def list_all_projects():
    """
    Lists all projects in the entire organization.
    
    This function:
    1. Gets all projects directly under the organization
    2. Gets all folders in the organization
    3. Gets all projects in each folder
    4. Combines the results and filters for active projects
    
    Returns:
        List of all active project IDs in the organization
    """
    active_projects = []
    organization_id = "organizations/ORGANIZATION_ID"
    
    # Step 1: Get projects directly under the organization
    print(f"Searching for projects directly under the organization...")
    org_projects = search_org_projects(organization_id)
    for project in org_projects:
        if str(project.state) == "State.ACTIVE":
            active_projects.append(project.project_id)
    
    print(f"Found {len(active_projects)} projects directly under the organization")
    
    # Step 2 & 3: Get all folders and the projects within them
    print(f"Searching for folders in the organization...")
    folders = get_folders(parent_id=organization_id, folders=None)
    print(f"Found {len(folders)} folders to check")
    
    # Step 4: Find projects in each folder
    for i, folder in enumerate(folders):
        print(f"Searching folder {i+1}/{len(folders)}: {folder}")
        folder_projects = search_projects(folder)
        
        for project in folder_projects:
            if str(project.state) == "State.ACTIVE":
                active_projects.append(project.project_id)
    
    return active_projects

if __name__ == "__main__":
    print("Starting search for all GCP projects in the organization...")
    projects = list_all_projects()
    print(f"\nSummary: Found {len(projects)} active projects in the organization")
    print("\nProject list:")
    for project in projects:
        print(f"- {project}")

Important: Replace ORGANIZATION_ID with your actual Google Cloud Organization ID in two places in the script.

Step 5: Run the Script

Now we can run our Python script to list all GCP projects:

python3 list_all_gcp_projects.py

How the Python Script Works: Code Breakdown

Let’s understand how the script works in detail:

1. The Folder Traversal Function

The get_folders() function is the foundation of our solution. It uses recursion to walk through the entire folder hierarchy:

def get_folders(parent_id = "organizations/ORGANIZATION_ID", folders = None):

• Starting point: It begins with the organization and recursively explores all folders
• Recursion: For each folder found, it calls itself to find sub-folders
• Accumulation: It builds a list of all folder IDs during the process

This recursive approach ensures we discover all folders regardless of how deeply nested they are in the organization.

2. The Project Search Functions

We have two project search functions:

def search_projects(folder_id):
    # Find projects in a folder
    
def search_org_projects(org_id):
    # Find projects directly under the organization

These functions:

• Create a Resource Manager client
• Build a query to find projects under a specific parent
• Execute the query and collect the results

The key difference is that one searches folders, while the other searches directly under the organization.

3. The Main List Function

The list_all_projects() function brings everything together:

def list_all_projects():

This function:

1. Finds projects directly under the organization
2. Gets all folders in the organization
3. Searches each folder for projects
4. Filters out any non-active projects
5. Returns a complete list of active project IDs

4. The Output

The script provides detailed progress information:

• Shows how many projects are directly under the organization
• Reports the total number of folders found
• Shows progress as it searches each folder
• Provides a final summary and complete project list

Enhancing the Script for More Information

The basic script gives us a list of project IDs, but what if we want more details? Let’s enhance it to include more information:

def list_projects_with_details():
    """
    Lists all projects with additional details.
    
    Returns:
        Dictionary of project details by project ID
    """
    project_details = {}
    organization_id = "organizations/ORGANIZATION_ID"
    
    # Get projects directly under the organization
    org_projects = search_org_projects(organization_id)
    for project in org_projects:
        if str(project.state) == "State.ACTIVE":
            project_details[project.project_id] = {
                "name": project.display_name,
                "number": project.project_number,
                "parent": "organization",
                "create_time": project.create_time.isoformat(),
                "labels": dict(project.labels)
            }
    
    # Get all folders
    folders = get_folders(parent_id=organization_id, folders=None)
    folder_names = {}
    
    # Get folder names for better reporting
    client = resourcemanager_v3.FoldersClient()
    for folder_path in folders:
        folder_id = folder_path.split('/')[-1]
        request = resourcemanager_v3.GetFolderRequest(name=folder_path)
        folder = client.get_folder(request=request)
        folder_names[folder_path] = folder.display_name
    
    # Get projects in each folder
    for folder in folders:
        folder_projects = search_projects(folder)
        for project in folder_projects:
            if str(project.state) == "State.ACTIVE":
                project_details[project.project_id] = {
                    "name": project.display_name,
                    "number": project.project_number,
                    "parent": f"folder:{folder_names.get(folder, folder)}",
                    "create_time": project.create_time.isoformat(),
                    "labels": dict(project.labels)
                }
    
    return project_details

Exporting Results to CSV for Reporting

Let’s add a function to export our results to a CSV file for easy reporting:

import csv
from datetime import datetime

def export_to_csv(project_details):
    """
    Export project details to a CSV file
    
    Args:
        project_details: Dictionary of project details
    """
    timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
    filename = f"gcp_projects_{timestamp}.csv"
    
    with open(filename, 'w', newline='') as csvfile:
        fieldnames = ["project_id", "name", "number", "parent", "create_time", "labels"]
        writer = csv.DictWriter(csvfile, fieldnames=fieldnames)
        
        writer.writeheader()
        for project_id, details in project_details.items():
            row = {"project_id": project_id}
            row.update(details)
            # Convert labels dictionary to string
            row["labels"] = ", ".join([f"{k}={v}" for k, v in details["labels"].items()])
            writer.writerow(row)
    
    print(f"Exported project data to {filename}")

# Using it in the main function
if __name__ == "__main__":
    print("Getting detailed information for all GCP projects...")
    project_details = list_projects_with_details()
    print(f"Found {len(project_details)} active projects")
    
    # Export to CSV
    export_to_csv(project_details)
    
    # Print summary
    print("\nProject summary:")
    for project_id, details in project_details.items():
        print(f"- {project_id} ({details['name']}): {details['parent']}")

Common Issues and Troubleshooting

When listing projects across a GCP organization, you might encounter these common issues:

1. Permission Denied Errors

If you see Permission Denied errors, check:

• The service account has the correct roles (resourcemanager.folderViewer and resourcemanager.projectViewer)
• You’ve activated the service account correctly
• The organization policy allows the service account to access the resources

2. Missing Projects

If projects are missing from your results:

• Ensure the script recursively searches all folders (our implementation does this)
• Check if you’re filtering out projects that aren’t in the “ACTIVE” state
• Verify that the service account has access to all folders

3. Rate Limiting

For very large organizations:

• Add delays between API calls using time.sleep()
• Implement exponential backoff for retries
• Consider paginating results for better performance

Practical Applications for GCP Project Listing

Here are some real-world use cases for listing all projects in your GCP organization:

1. Resource Inventory: Create a complete inventory of all your GCP resources
2. Cost Management: Identify orphaned or unused projects to reduce costs
3. Security Audits: Ensure all projects comply with security policies
4. Compliance Reporting: Generate reports for compliance requirements
5. Project Governance: Enforce naming conventions and organizational structure

Conclusion: Choosing the Right Approach

We’ve explored two methods for listing all projects in a Google Cloud organization:

1. Using gcloud: Simple but limited to direct children of the organization
2. Using Python: More complex but provides a complete view of all projects

For most real-world scenarios, the Python approach is superior because it:

• Handles complex organizational structures with nested folders
• Provides detailed information about each project
• Can be automated and integrated with other systems
• Allows for custom filtering and processing of results

With the Python script provided, you can now efficiently list all projects across your entire GCP organization, regardless of how complex your hierarchy is.

We can use the Resource Manager API to do so much more than this – from managing project hierarchies to implementing custom governance policies. The possibilities are vast!