Python and Cloud Computing: Working with AWS, GCP, and Azure
Table of Contents
Cloud computing has transformed the way we build, deploy, and scale applications. It provides on-demand access to a wide range of computing resources, including servers, storage, and databases, without the need to invest in and maintain physical infrastructure. Python is a popular language for cloud computing due to its versatility, scalability, and ease of use. In this blog post, we will explore how to work with Python in three of the most popular cloud computing platforms: AWS, GCP, and Azure.
1. AWS
Amazon Web Services (AWS) is the leading cloud computing platform, offering a vast array of services for building and deploying applications in the cloud. Python is a first-class citizen on AWS, with support for a wide range of AWS services, including:
1.1 AWS Lambda
AWS Lambda is a serverless compute service that allows you to run your code in response to events, such as changes to data in an Amazon S3 bucket, a new file uploaded to an Amazon S3 bucket, or an Amazon API Gateway HTTP request. Lambda functions can be written in Python, and AWS Lambda supports the latest version of Python, as well as popular Python web frameworks such as Flask and Django.
To create a new Python Lambda function, you can use the AWS Management Console or the AWS Command Line Interface (CLI). For example, the following command creates a new Lambda function called my-function using the Python 3.8 runtime:
aws lambda create-function --function-name my-function --runtime python3.8 --handler index.lambda_handler --zip-file fileb://my-function.zip --role arn:aws:iam::123456789012:role/lambda-role
1.2 Amazon S3
Amazon S3 (Simple Storage Service) is a highly-scalable object storage service that allows you to store and retrieve any amount of data from anywhere on the web. Python provides a number of libraries for working with Amazon S3, including the popular boto3 library.
To use boto3 with S3, you need to install it using pip:
pip install boto3
You can then use the following code to list all the objects in an S3 bucket:
import boto3
s3 = boto3.resource('s3')
bucket = s3.Bucket('my-bucket')
for obj in bucket.objects.all():
print(obj.key)
1.3 Amazon EC2
Amazon Elastic Compute Cloud (EC2) is a web service that provides scalable compute capacity in the cloud. EC2 allows you to launch virtual machines (instances) in a variety of configurations, including different operating systems, instance types, and security groups. Python can be used to automate the deployment and management of EC2 instances using the boto3 library.
For example, the following Python code uses boto3 to launch a new EC2 instance:
import boto3
ec2 = boto3.client('ec2')
response = ec2.run_instances(
ImageId='ami-0c55b159cbfafe1f0',
InstanceType='t2.micro',
KeyName='my-key-pair',
MinCount=1,
MaxCount=1
)
instance_id = response['Instances'][0]['InstanceId']
2. GCP
Google Cloud Platform (GCP) is a suite of cloud computing services offered by Google. It provides a wide range of services for building, deploying, and scaling applications in the cloud. Python is a first-class citizen on GCP, with support for many GCP services, including:
2.1 Google Cloud Functions
Google Cloud Functions is a serverless compute service that allows you to run your code in response to events, such as changes to data in a Google Cloud Storage bucket or a new file uploaded to a Google Cloud Storage bucket. Cloud Functions supports Python 3.7 and 3.8, and it provides built-in support for popular Python web frameworks such as Flask and Django.
To create a new Python Cloud Function, you can use the GCP Console or the Cloud SDK. For example, the following command creates a new Cloud Function called my-function using the Python 3.8 runtime:
gcloud functions deploy my-function --runtime python38 --trigger-http --entry-point my_function
2.2 Google Cloud Storage
Google Cloud Storage is a highly-scalable object storage service that allows you to store and retrieve any amount of data from anywhere on the web. Python provides a number of libraries for working with Google Cloud Storage, including the google-cloud-storage library.
To use google-cloud-storage, you need to install it using pip:
pip install google-cloud-storage
You can then use the following code to list all the objects in a Cloud Storage bucket:
from google.cloud import storage
client = storage.Client()
bucket = client.bucket('my-bucket')
for blob in bucket.list_blobs():
print(blob.name)
2.3 Google Compute Engine
Google Compute Engine is a web service that provides scalable compute capacity in the cloud. Compute Engine allows you to launch virtual machines (instances) in a variety of configurations, including different operating systems, instance types, and security groups. Python can be used to automate the deployment and management of Compute Engine instances using the google-cloud-compute library.
For example, the following Python code uses google-cloud-compute to launch a new Compute Engine instance:
from google.cloud import compute_v1
compute = compute_v1.InstancesClient()
image_response = compute.images().getFromFamily(
project='debian-cloud', family='debian-10'
).execute()
image_self_link = image_response['selfLink']
machine_type = 'zones/us-central1-a/machineTypes/n1-standard-1'
config = {
'name': 'my-instance',
'machineType': machine_type,
'disks': [{
'boot': True,
'autoDelete': True,
'initializeParams': {
'sourceImage': image_self_link
}
}]
}
instance = compute.instances().insert(
project='my-project', zone='us-central1-a', body=config
).execute()
print(instance)
3. Azure
Microsoft Azure is a cloud computing platform that provides a wide range of services for building, deploying, and scaling applications in the cloud. Python is a first-class citizen on Azure, with support for many Azure services, including:
3.1 Azure Functions
Azure Functions is a serverless compute service that allows you to run your code in response to events, such as changes to data in an Azure Blob storage container or an Azure Event Grid event. Functions supports Python 3.6 and 3.7, and it provides built-in support for popular Python web frameworks such as Flask and Django.
To create a new Python Function, you can use the Azure Portal or the Azure CLI. For example, the following command creates a new Function called my-function using the Python 3.7 runtime:
az functionapp create --resource-group my-resource-group --name my-function --storage-account my-storage-account --consumption-plan-location eastus --runtime python --functions-version 3 --os-type linux
3.2 Azure Blob Storage
Azure Blob Storage is a highly-scalable object storage service that allows you to store and retrieve any amount of unstructured data from anywhere on the web. Python provides a number of libraries for working with Azure Blob Storage, including the azure-storage-blob library
To use azure-storage-blob, you need to install it using pip:
pip install azure-storage-blob
You can then use the following code to list all the blobs in a Blob Storage container:
from azure.storage.blob import BlobServiceClient
blob_service_client = BlobServiceClient.from_connection_string(connection_string)
container_client = blob_service_client.get_container_client(container_name)
blobs = container_client.list_blobs()
for blob in blobs:
print(blob.name)
3.3 Azure Virtual Machines
Azure Virtual Machines is a web service that provides scalable compute capacity in the cloud. Virtual Machines allows you to launch virtual machines (instances) in a variety of configurations, including different operating systems, instance types, and security groups. Python can be used to automate the deployment and management of Virtual Machines using the azure-mgmt-compute library.
For example, the following Python code uses azure-mgmt-compute to launch a new Virtual Machine:
from azure.common.credentials import ServicePrincipalCredentials
from azure.mgmt.compute import ComputeManagementClient
from azure.mgmt.network import NetworkManagementClient
from azure.mgmt.resource import ResourceManagementClient
from azure.mgmt.storage import StorageManagementClient
subscription_id = 'your-subscription-id'
resource_group_name = 'your-resource-group-name'
location = 'westus2'
vm_name = 'your-vm-name'
admin_username = 'your-admin-username'
admin_password = 'your-admin-password'
vm_size = 'Standard_B1s'
storage_account_name = 'your-storage-account-name'
storage_container_name = 'your-storage-container-name'
os_disk_name = 'your-os-disk-name'
os_type = 'Linux'
credentials = ServicePrincipalCredentials(
client_id='your-client-id',
secret='your-client-secret',
tenant='your-tenant-id'
)
resource_client = ResourceManagementClient(credentials, subscription_id)
storage_client = StorageManagementClient(credentials, subscription_id)
network_client = NetworkManagementClient(credentials, subscription_id)
compute_client = ComputeManagementClient(credentials, subscription_id)
resource_client.resource_groups.create_or_update(
resource_group_name, {'location': location}
)
storage_account = storage_client.storage_accounts.create(
resource_group_name,
storage_account_name,
{
'sku': {'name': 'Standard_LRS'},
'kind': 'Storage',
'location': location
}
)
container = storage_client.blob_containers.create(
resource_group_name,
storage_account_name,
storage_container_name,
{'public_access': 'blob'}
)
os_disk = {
'name': os_disk_name,
'vhd': {
'uri': 'https://{}.blob.core.windows.net/{}/{}.vhd'.format(
storage_account_name, storage_container_name, os_disk_name
)
},
'caching': 'ReadWrite',
'create_option': 'FromImage'
}
nic = network_client.network_interfaces.create_or_update(
resource_group_name,
'{}-nic'.format(vm_name),
{
'location': location,
'ip_configurations': [{
'name': '{}-ipconfig'.format(vm_name),
'subnet': {'id': '/subscriptions/{}/resourceGroups/{}/providers/Microsoft.Network/virtualNetworks/{}/subnets/default'.format(subscription_id, resource_group_name, resource_group_name)},
'public_ip_address': {
'id': '/subscriptions/{}/resourceGroups/{}/providers/Microsoft.Network/publicIPAddresses/{}-ip'.format(subscription_id, resource_group_name, vm_name)
}
}]
}
)
vm_parameters = {
'location': location,
'os_profile': {
'computer_name': vm_name,
'admin_username': admin_username,
'admin_password': admin_password
},
'hardware_profile': {
'vm_size': vm_size
},
'storage_profile': {
'os_disk': os_disk,
'image_reference': {
'publisher': 'Canonical',
'offer': 'UbuntuServer',
'sku': '18.04-LTS',
'version': 'latest'
},
'data_disks': []
},
'network_profile': {
'network_interfaces': [{
'id': nic.id,
}]
}
}
async_vm_creation = compute_client.virtual_machines.create_or_update(
resource_group_name, vm_name, vm_parameters)
async_vm_creation.wait()
This code uses the `azure-mgmt-compute` library to create a new resource group, storage account, network interface, and virtual machine. It also sets up the OS disk and specifies the image to use for the virtual machine.
3.4 Google Cloud Platform
Google Cloud Platform (GCP) is a cloud computing platform that provides infrastructure services, platform services, and application services. GCP allows you to build, test, and deploy applications on Google’s infrastructure using a variety of tools and services.
3.4.1 GCP Compute Engine
GCP Compute Engine is a web service that provides scalable compute capacity in the cloud. Compute Engine allows you to launch virtual machines (instances) in a variety of configurations, including different operating systems, instance types, and security groups. Python can be used to automate the deployment and management of Compute Engine instances using the `google-cloud-compute` library. For example, the following Python code uses `google-cloud-compute` to launch a new Compute Engine instance:
```python
from google.oauth2 import service_account
from google.cloud import compute_v1
credentials = service_account.Credentials.from_service_account_file(
'path/to/service_account.json')
project_id = 'your-project-id'
zone = 'us-central1-a'
name = 'your-instance-name'
image_project = 'debian-cloud'
image_family = 'debian-10'
machine_type = 'n1-standard-1'
disk_size_gb = 10
compute = compute_v1.InstancesClient(credentials=credentials)
instance_config = {
'name': name,
'machine_type': 'zones/{}/machineTypes/{}'.format(zone, machine_type),
'disks': [{
'boot': True,
'auto_delete': True,
'initialize_params': {
'source_image_family': image_family,
'source_image_project': image_project,
'disk_size_gb': disk_size_gb
}
}]
}
instance = compute.insert(project=project_id, zone=zone, body=instance_config).execute()
print('Instance created: {}'.format(instance['selfLink']))
This code uses the google-cloud-compute library to create a new Compute Engine instance in the specified project and zone. It also sets up the boot disk and specifies the image to use for the instance.
3.4.2 GCP Cloud Storage
GCP Cloud Storage is a web service that provides scalable object storage in the cloud. Cloud Storage allows you to store and retrieve objects (files) of any size in a variety of configurations, including different storage classes and access controls. Python can be used to automate the management of Cloud Storage buckets using the google-cloud-storage library.
For example, the following Python code uses google-cloud-storage to list all the objects in a Cloud Storage bucket:
from google.oauth2 import service_account
from google.cloud import storage
credentials = service_account.Credentials.from_service_account_file(
'path/to/service_account.json')
bucket_name = 'your-bucket-name'
storage_client = storage.Client(credentials=credentials)
bucket = storage_client.bucket(bucket
This code uses the google-cloud-storage library to authenticate the client using the service account key file, and then retrieves a handle to the specified bucket. It then lists all the objects in the bucket.
3.5 Microsoft Azure
Microsoft Azure is a cloud computing platform that provides infrastructure services, platform services, and application services. Azure allows you to build, test, and deploy applications on Microsoft’s infrastructure using a variety of tools and services.
3.5.1 Azure Virtual Machines
Azure Virtual Machines is a web service that provides scalable compute capacity in the cloud. Virtual Machines allows you to launch virtual machines in a variety of configurations, including different operating systems, instance types, and security groups. Python can be used to automate the deployment and management of Virtual Machines using the azure-mgmt-compute library.
For example, the following Python code uses azure-mgmt-compute to launch a new Virtual Machine:
from azure.common.credentials import ServicePrincipalCredentials
from azure.mgmt.compute import ComputeManagementClient
from azure.mgmt.compute.models import DiskCreateOption
subscription_id = 'your-subscription-id'
resource_group_name = 'your-resource-group-name'
location = 'your-location'
vm_name = 'your-vm-name'
vm_size = 'your-vm-size'
admin_username = 'your-admin-username'
admin_password = 'your-admin-password'
credentials = ServicePrincipalCredentials(
client_id='your-client-id',
secret='your-client-secret',
tenant='your-tenant-id'
)
compute_client = ComputeManagementClient(credentials, subscription_id)
async_vm_creation = compute_client.virtual_machines.create_or_update(
resource_group_name,
vm_name,
{
'location': location,
'os_profile': {
'computer_name': vm_name,
'admin_username': admin_username,
'admin_password': admin_password
},
'hardware_profile': {
'vm_size': vm_size
},
'storage_profile': {
'os_disk': {
'create_option': DiskCreateOption.from_image,
'name': vm_name,
'caching': 'ReadWrite',
'managed_disk': {
'storage_account_type': 'Standard_LRS'
}
}
},
'network_profile': {
'network_interfaces': [{
'id': '/subscriptions/{0}/resourceGroups/{1}/providers/Microsoft.Network/networkInterfaces/{2}'.format(
subscription_id,
resource_group_name,
nic_name
),
'properties': {
'primary': True
}
}]
}
}
)
async_vm_creation.wait()
This code uses the azure-mgmt-compute library to create a new resource group, network interface, and virtual machine. It also sets up the OS disk and specifies the image to use for the virtual machine.
3.4.2 Azure Blob Storage
Azure Blob Storage is a web service that provides scalable object storage in the cloud. Blob Storage allows you to store and retrieve objects (files) of any size in a variety of configurations, including different storage tiers and access controls. Python can be used to automate the management of Blob Storage using the azure-storage-blob library.
For example, the following Python code uses azure-storage-blob to list all the blobs in a Blob Storage container:
from azure.storage.blob import BlobServiceClient
account_url = 'your-account-url'
account_key = 'your-account-key'
container_name = 'your-container-name'
blob_service_client = BlobServiceClient(
account_url=account_url,
credential=account_key
)
container_client = blob_service_client.get_container_client(container_name)
blobs = container_client.list_blobs()
for blob in blobs:
print(blob.name)
This code uses the azure-storage-blob library to authenticate the client using the account URL and key, and then retrieves a handle to the specified container. It then lists all the blobs in the container.
4. Conclusion
Python is a powerful language for cloud computing, and it can be used to automate and manage cloud resources across multiple cloud providers. In this blog post, we looked at how Python can be used to work with AWS, GCP, and Azure, including how to authenticate, interact with, and manage resources in each cloud platform.
By using Python, you can write scripts that automate the provisioning, configuration, and management of cloud resources, making it easier and more efficient to deploy and manage your applications in the cloud. With Python’s rich ecosystem of libraries and tools, you can build sophisticated cloud applications and services that scale to meet your needs.
