Tensorflow - S3 object does not exist

King Dedede

How do I set up direct private bucket access for Tensorflow?

After running
from tensorflow.python.lib.io import file_io and running print file_io.stat('s3://my/private/bucket/file.json') I end up with an error -
NotFoundError: Object s3://my/private/bucket/file.json does not exist

However, the same line on a public object works without an error:
print file_io.stat('s3://ryft-public-sample-data/wikipedia-20150518.bin')

There appears to be an article on support here: https://github.com/tensorflow/examples/blob/master/community/en/docs/deploy/s3.md
However, I end up with the same error after exporting the variables shown.

I have awscli set up with all credentials, and boto3 can view and download the file in question. I am wondering how I can get Tensorflow to have S3 access directly when the bucket is private.

marcin

I had the same problem when trying to access files in private S3 bucket from Sagemaker notebook. The mistake I made was to try using credentials I obtained from boto3, which seem not to be valid outside.

The solution was not to specify credentials (in such case it uses the role attached to the machine), but instead just specify the region name (for some reason it didn't read it from ~/.aws/config file) as follows:

import boto3
import os

session = boto3.Session()
os.environ['AWS_REGION']=session.region_name

NOTE: when debugging this error useful was to look at CloudWatch logs, as the logs of S3 client were printed only there and not in the Jupyter notebook. In there I have first have seen, that:

  1. when I did specify credentials from boto3 the error was: The AWS Access Key Id you provided does not exist in our records.
  2. When accessing without AWS_REGION env variable set I had The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint. which apparently is common when you don't specify bucket (see 301 Moved Permanently after S3 uploading)

Collected from the Internet

Please contact [email protected] to delete if infringement.

edited at
0

Comments

0 comments
Login to comment

Related

Property does not exist on type 'object'

CloudFront + S3 Website: "The specified key does not exist" when an implicit index document should be displayed

AWS s3 api error: specified bucket does not exist

SonataMediaBundle - S3 AWS: 'The configured bucket "my-bucket" does not exist

how to fix property does not exist on type 'Object' in ionic 3?

AWS CLI S3 rm command does not produce error if file does not exist

Property 'map' does not exist on type 'Object' - Ionic 3

Update an object or create it if does not exist

Why am I getting different errors when trying to read s3 key that does not exist

Glacier and S3 using lifecycle rules, specified key does not exist

AWS S3 CopyObjectAsync fails with key does not exist, but get/put succeeds

Write-S3Object : The file indicated by the FilePath property does not exist

FIXED-IONIC 3: property 'questions' does not exist on type 'Object'

Does AWS S3 object support POSIX Hardlinks?

Property '' does not exist on type 'Object'

AWS Tools: Copy-S3Object script fails in 2.x with Error "Bucket does not exist"

JSTL: Does Object Exist

S3 head object but signature does not match

Snapshot does not exist error on restoring from s3 repository

Firebase image object does not exist?

"... matching query does not exist." error, but object clearly does exist

object outer does not exist

FailedPreconditionError: Resource localhost/_AnonymousVar404/N10tensorflow3VarE does not exist

'specified bucket does not exist ' error - S3 bucket with Policy disallow upload from particular IP

Does AWS S3 GetObject read the partial of the Object being uploaded to s3 at the same time

Property does not exist on type 'object'

Populating an array of object with 0 if object does not exist

How does an S3 object invoke a method?

Terraform init Error: Failed to get existing workspaces: S3 bucket does not exist

TOP Ranking

HotTag

Archive