Concat Avro files in Google Cloud Storage

You can use the gsutil compose command. For example:

gsutil compose gs://bucket/obj1 [gs://bucket/obj2 ...] gs://bucket/composite

Note: For extremely large files and/or very low per-machine bandwidth, you may want to split the file and upload it from multiple machines, and later compose these parts of the file manually.

On my case I tested it with the following values: foo.txt contains a word Hello and bar.txt contains a word World. Running this command:

gsutil compose gs://bucket/foo.txt gs://bucket/bar.txt gs://bucket/baz.txt

baz.txt would return:

Hello
World

Note: GCS does not support inter-bucket composing.

Just in case if you're encountering an exception error with regards to integrity checks, run gsutil help crcmod to get an instructions on how to fix it.