Why do I see `503 Backend Error` when I try to get a Cloud Storage service account?

I try to deploy a storage triggered Cloud Function, and while this process normally worked just fine with many projects in the same organizations and with similar setups, since yesterday I have trouble to do so.

I can observe the following pattern in all GCP projects across multiple organizations:

• Category A If at least one storage-triggered function has been deployed to a project before yesterday, I can just deploy again the same function or a new function to this project, no problem.

  • These projects have a Cloud Storage service account, as shown in GCP Console under Storage > Settings, ie. [email protected]
  • I can retrieve this account from Cloud Storage API Projects.serviceAccount: get

• Category B If no storage-triggered functions have been deployed to a project before yesterday, I cannot deploy a storage-triggered function, no matter if I deploy functions through Cloud Build, or if I create a simple bucket function through the GCP console, even not when I just use the sample code suggested by the UI as is and link to a default bucket in the project.

  • These projects don't have a Cloud Storage service account, as shown in GCP Console under Storage > Settings
  • Neither can I retrieve this account from Cloud Storage API Projects.serviceAccount: get, I'll see a 503 Backend Error

I can't find any other significant differences in the setups between the two categories, in particular IAM and ACL, enabled APIs, and I've made sure that the function is always created in the same project as the bucket that is the trigger resource.

Error when using gcloud functions deploy

$ are retracted names

gcloud functions deploy $FUNCTION_NAME --region=us-central1 --source=https://source.developers.google.com/projects/$PROJECT_ID/repos/$REPO_NAME/moveable-aliases/$BRANCH_NAME/paths/$FUNCTION_NAME/src --trigger-resource=$API_PROJECT_ID-input --trigger-event=google.storage.object.finalize --timeout=540 --runtime=python37 --entry-point=main --memory=128MB --project=$API_PROJECT_ID --retry

will show this error for the project that fall into category B.

ERROR: (gcloud.functions.deploy) OperationError: code=13, message=Failed to configure trigger GCS Bucket: $API_PROJECT_ID-input

Doing the same in projects of category A succeeds fine as expected.

Error when using Google Cloud Console

On the page Create function,

1. I select Cloud Storage as Trigger
2. Event Type: `Finalize/Create*
3. As Bucket I select a default bucket, e.g. $API_PROJECT_ID.appspot.com
4. Everything else as default, including the suggested sample code
5. SAVE

The deployment will fail for all projects of category B, but succeeds fine for projects in category A.

The deployment status message in the UI shows the following error:

Deployment failure:
Failed to configure trigger GCS Bucket: $API_PROJECT_ID

Error: Cloud Storage service account is not created

I have then started to compare the setups and noticed that in the GCP console, under Cloud Storage > Settings, only the project of category A have a Cloud Storage service account. Projects of category B don't have a Cloud Storage service account. So I tried to retrieve the account from the API:

https://cloud.google.com/storage/docs/json_api/v1/projects/serviceAccount/get

As you can guess, I'll get 200 OK with the corresponding Cloud Storage service account name only for projects in category A, but for all projects in category B I will see this error response with code 503:

{
 "error": {
  "errors": [
   {
    "domain": "global",
    "reason": "backendError",
    "message": "Backend Error"
   }
  ],
  "code": 503,
  "message": "Backend Error"
 }
}

I have tried a few times for many different projects since yesterday, nothing changed.

Further info and attempts to resolve

I have followed the instructions provided on: https://cloud.google.com/storage/docs/getting-service-account using the API Explorer on this page: https://cloud.google.com/storage/docs/json_api/v1/projects/serviceAccount/get

I'm not 100% certain the issues of the 503 error, the lack of service account and the deployment failures (code 13) are related. But IIRC, the storage trigger of GCF is actually based on Pub/Sub and that one will require the service account, as stated in the help text in GCP console under Storage > Settings:

Each project has an associated Cloud Storage service account. This is used to perform certain background actions: receiving PubSub notifications and encrypting/decrypting KMS encrypted objects.

I have also read the advice regarding this service account here: https://cloud.google.com/storage/docs/projects#service-accounts

I have also tried the following workarounds:

• compared IAM between category A and B projects using gsutil iam and GCP console
• compared ACL between category A and B projects using gsutil acl
• deployed manually through the API other test buckets with differing predefinedAcl
• deleted the bucket $API_PROJECT_ID-input and deployed again with no predefinedAcl
• used default buckets of projects that have been automatically created, e.g. $API_PROJECT_ID.appspot.com
• updated the deployment configurations of Cloud Deployment Manager for projects in category B
• tried to disable and enable several APIs, but wouldn't do that for most APIs to avoid causing other problems
• enabled Cloud Build API for projects in category B (which interestingly, only then has brought the inherited IAM permissions of a different Cloud Build service account into the IAM of an $API_PROJECT_ID-input bucket)
• added Storage Object Creator role to the default service account ([email protected])