How to read, write and list folders and files in google bucket?

Chan

I would like to read/write files in Google Cloud Storage bucket with Python.

Suppose I have a folder in gs://my_project/data.

  • How to list the folders and files in the above folder?

  • How to read and write files?

syltruong

There are several ways to perform these operations. The most common one is to use the native Google Cloud Storage API for Python.

In particular, step 0 to use this API is to set up authentication to GCP, which consists in setting up a service account, downloading its json credentials and set an environment variable pointing to it:

export GOOGLE_APPLICATION_CREDENTIALS="[PATH-TO-JSON-CREDS]"


1. Listing files and folders

In GCS, there is no notion of a "directory"/"folder". There are only buckets and blobs/objects. Nevertheless, the / in blob names are usable to emulate a folder-like hierarchy.

To list blobs from gs://my_project/data:

from google.cloud import storage

client = storage.Client()
bucket = client.bucket('my_project')

blobs = list(bucket.list_blobs(prefix='data/'))

2. Reading and writing files

To read from the first blob listed in gs://my_project/data.

target_blob = blobs[0]

# read as string
read_output = target_blob.download_as_string()

To write to new blob, I have found no other way than to write to a local file and upload from file.

target_blob = bucket.blob('new_blob.txt')

local_tmp_path = 'tmp.txt'

# write string
with open(local_tmp_path, 'w') as f:
   f.write('Hello World')

with open(local_tmp_path, 'r') as f:
   target_blob.upload_from_file(f)

Collected from the Internet

Please contact [email protected] to delete if infringement.

edited at
0

Comments

0 comments
Login to comment

Related