Google Cloud Storage

You can add your existing Google Cloud bucket as a datasource in PixelBin. This allows you to use PixelBin's real-time file delivery and transformation features on all existing files stored in the drive.

For PixelBin to be able to access your Google Cloud Storage, you will need to create a new Google Service Account and assign it the Storage Object Viewer role for the bucket that you wish to integrate with PixelBin. This tutorial will help you do that. Learn more about Google Cloud Storage's Identity and Access Management.

caution

If the original file has been uploaded to GCS with gzip compression, then content-length header won't be set for this file when fetching it via Pixelbin.

In such cases, GCS doesn't store the original file's metadata, which is required for setting headers like content-length. Setting incorrect value for content-length can break the response.

This is only applicable to images and files larger than 10MB.

Setting up Google cloud

info

In case you find the following instructions invalid/outdated, you can refer to Google's official, albeit more verbose instructions on these pages:

1. Create a service account

The service account is a separate virtual identity that will be used by PixelBin to access the files in your drive's folder.

In the Cloud Console, go to the Service Accounts page. Go to the Service Accounts page.
Click Select a project, choose your project, and click Open.
Click Create a Service Account.
Enter a service account name (e.g. for-pixelbin-access), and an optional description.
Click Save.

2. Create access keys for the Service Account

We need to create access keys that will be used by PixelBin to authenticate itself to the Google Cloud Storage APIs.

In the Cloud Console, go to the Service Accounts page. Go to the Service Accounts page.
Click Select a project, choose a project, and click Open.
Click on the service account that we created in Step 1.
On the page for the service account, click on the Add key button in the Keys section.
Select JSON as the key type and click Create. This will download a JSON file.
You will need to paste the contents of this file to the PixelBin dashboard.

info

Make sure to store this file securely, as it cannot be downloaded again.

3. Assign the Bucket Reader role to the Service Account

In the Cloud Console, navigate to the bucket that you wish to integrate with PixelBin.
Click on the Permissions tab.
Click on Add members button.
Enter the email id of the service account we created in Step 1, and select the Cloud Storage -> Storage Object Viewer role in the Select a role option.
Click Save.

Configure Datasource

We have now created a virtual identity for PixelBin and granted it the Reader role for your bucket.

After entering basic details for the datasource, choose Google Cloud Storage from the set of datasource types.
Enter the name of the bucket you want to integrate with PixelBin.
Copy-paste the contents of the keyFile.json you downloaded from your created service account.
Click Save.

note

We do not start copying files from your storage as soon as you add the datasource. Instead, we will fetch the particular file when you request it through the PixelBin URL endpoint. The files accessed from this origin will not appear in your Storage library.

Configure Cache Warmup

When cache wamup is enabled for a GCS datasource, all new images uploaded to your bucket will be cached on the CDN even before the first request is made. This can help in reducing the latency for the first request made to the image. For enabling warmup, following things are required:

GCP Pub/Sub notifications.

The Pub/Sub notifications is how PixelBin will be notified of new files uploaded to your bucket. For setting up Pub/Sub notifications, follow these steps:

note

To get the permissions that you need to configure and view Pub/Sub notifications for a bucket, ask your administrator to grant you the following roles:

Storage Admin (roles/storage.admin) role on the bucket for which you want to configure Pub/Sub notifications
Pub/Sub Admin (roles/pubsub.admin) role on the project in which you want to receive Pub/Sub notifications

These predefined roles contain the permissions required to configure and view Pub/Sub notifications.

Create a GCS Notification

Install the gcloud command-line tool. Install the gcloud command-line tool.
Run the following command to create a notification for your bucket:

gcloud storage buckets notifications create gs://BUCKET_NAME --topic=TOPIC_NAME -e OBJECT_FINALIZE

BUCKET_NAME is the name of the bucket which you've integrated as a datasource in PixelBin.
TOPIC_NAME is the name of the Pub/Sub topic to send notifications to. If you specify a topic that doesn't exist in your project, the command creates one for you.
OBJECT_FINALIZE is the event type that triggers the notification. This event type triggers the notification when a new object is uploaded to the bucket.

note

For more information on creating GCS notifications, refer to the official Google Cloud documentation.

Create a Pub/Sub Subscription

In the Cloud Console, go to the Pub/Sub page. Go to the Pub/Sub page.
Click Create Subscription.
Enter a subscription name (e.g. pixelbin-cache-warmup-subscription).
Select the topic you created in the previous step.
Set the Delivery type to Pull.
Set the Acknowledgement Deadline to 600 seconds.
Use the subscription filter to filter out prefixes or files you don't want to warmup.
Click Create.

Enable Cache Warmup in PixelBin

Go to the zone you want to enable cache warmup for. Only the zones linked to the GCS datasource will have the cache warmup option.
Go to the Cache Warmup section.
Enable the Cache Warmup toggle.
Enter the Pub/Sub subscription name you created in the previous step.
Enter the list of transformations you want to apply to the images during the cache warmup process.
Click Save.

note

Only the images uploaded after enabling cache warmup will be cached on the CDN. The images uploaded before enabling cache warmup will not be cached.