Add multiple S3 lifecycle rules to S3 bucket with boto3

Idris.AH

I want to add multiple lifecycle rules to a S3 bucket using lambda and boto3. However, with boto3, it only seems to allow you to add 1 lifecycle rule, which also overwrites the already existing ones.

For example, doing the following simply overwrites any pre-existing rules that I have as well as the newly written ones, and only leaves the last one in the list:

bucket_name = "test-bucket"
folder_paths = ["test_folder","test_folder1", "test_folder2"]
expiration = 1

for folder_path in folder_paths:
    client = boto3.client('s3')
    response = client.put_bucket_lifecycle_configuration(
        Bucket=bucket_name,
        LifecycleConfiguration={
            'Rules': [
                {
                    'Expiration': {
                        'Days': expiration
                    },
                    'ID': folder_path,
                    'Filter': {
                        'Prefix': folder_path
                    },
                    'Status': 'Enabled'
                }
            ]
        }
    )

Of course using the AWS console it is possible to add multiple separate S3 lifecycle configurations on a bucket.

Similarly to put_bucket_lifecycle_configuration, I have also tried put_bucket_lifecycle, which gives me the same result.

Is there any way to use boto3 to add multiple S3 lifecycle configurations on a bucket? Am I missing something obvious?

Any help is appreciated and thanks in advance!

Ervin Szilagyi

Of course using the AWS console it is possible to add multiple separate S3 lifecycle configurations on a bucket.

Every bucket has 1 lifecycle configuration, which can have up to 1000 rules. Your console may show something similar:

enter image description here

These are not different lifecycle configurations, they are different rules part of the same lifecycle configuration.

In the input for the put_bucket_lifecycle_configuration we can see that we can pass a list of rules, which can contain 1 ore many more (up to 1000) rules.

for folder_path in folder_paths:
    client = boto3.client('s3')
    response = client.put_bucket_lifecycle_configuration(
        Bucket=bucket_name,
        LifecycleConfiguration={
            'Rules': [
                {
                    'Expiration': {
                        'Days': expiration
                    },
                    'ID': id_rule_1,
                    'Filter': {
                        'Prefix': folder_path
                    },
                    'Status': 'Enabled'
                },
                {
                    'Expiration': {
                        'Days': expiration2
                    },
                    'ID': id_rule_2,
                    'Filter': {
                        'Prefix': folder_path2
                    },
                    'Status': 'Enabled'
                },
             ...
            ]
        }
    )

As the docs says, put_bucket_lifecycle_configuration "creates a new lifecycle configuration for the bucket or replaces an existing lifecycle configuration." If you want to update the lifecycle configuration, you have to use get_bucket_lifecycle_configuration to retrieve the existing rules, modify them and then use put_bucket_lifecycle_configuration to overwrite the existing configuration.

Collected from the Internet

Please contact [email protected] to delete if infringement.

edited at
0

Comments

0 comments
Login to comment

Related

Does s3 lifecycle rules overwrite Deny Delete Bucket or DeleteObject policy is s3 bucket?

Unable to add lifecycle policy to s3 bucket using serverless

Can't add multiple bucket policies to S3 bucket

how to add the trigger s3 bucket to lambda function dynamically(python boto3 API)

How to add tags to an S3 Bucket without deleting the existing tags using boto3?

Download multiple files from S3 bucket using boto3

AWS CDK update/add lifecycle to existing S3 bucket using custom source

Boto3 S3, sort bucket by last modified

Boto3 - S3 Bucket Policy Update

boto3 aws check if s3 bucket is encrypted

boto3 s3 bucket tagging

boto3 : Create a S3 bucket with ACLs disabled

Multiple subdomains for Amazon S3 Bucket?

AWS S3 Bucket with Multiple Regions

Terraform multiple s3 bucket creation

AWS: Boto3 configuring bucket lifecycle - Malformed XML

AWS S3 : Do Lifecycle rules accept regex?

Any limitation on number of s3 lifecycle rules?

Can I add multiple tags to a S3 bucket and not an object using put_bucket_tagging?

AWS CloudFront how to follow S3 bucket redirection rules?

Boto S3 Lifecycle expiration

S3 File Copy:Same bucket: using python boto3 from one folder to the bucket

How to modify the multiple object's ACL in S3 bucket?

AWS S3 lifecycle rule on multiple folders

Multiple Expire lifecycle policy configs in S3

Boto get s3 bucket location

Can we add tag to s3 bucket folder like s3 file

aws lambda - How to add s3 trigger using cloudformation if the s3 bucket is created manually

Amazon S3 - Pipe a string to upload multiple string on S3 Bucket in nodejs

TOP Ranking

HotTag

Archive