In some cases, you may need to transfer your objects in one of your Amazon S3 buckets to a different AWS account. AWS CLI provides high-level commands on S3 to move objects between two buckets. By also using Amazon S3 bucket policies, you can perform this even if the destination bucket is in another AWS account. Today, I will talk about the steps you need to do along with a few explanation on bucket and object level permissions.
As a start, let’s define a terminology for the rest of the post to make things clear.
origin bucket: An existing S3 bucket which contains objects that will be moved to
destination bucket: An existing S3 bucket which the objects in
origin bucketwill be moved to.
origin account: AWS account which
origin bucketbelongs to.
destination account: AWS account which
destination bucketbelongs to.
Granting Permissions to Origin Account in Destination Bucket
We will use
aws s3 sync command to copy all the objects in
origin bucket to
destination bucket and we wil do this as an IAM user in
destination account which has full access to
S3 sync command will read objects from
origin bucket and copy them by creating as new objects in
destination bucket. Therefore, in some way we need to grant necessary permissions to
destination account to be able to access
origin bucket and have complete read access. To do this, we will use S3 bucket policies which will grant resource level permissions to
1) Sign in AWS Console in
origin account and navigate to
origin bucket. 2) Click
Permissions tab and then,
Bucket Policy button. 3) In
Bucket Policy Editor paste and edit the policy below and save.
In this JSON, replace
<destination-account-id> with AWS Account ID of
destination account and
<origin-bucket-name> with the name of the
origin bucket. As you can understand easily, this policy grants read privileges (
destination account at root level.
destination account create an IAM user which has read access to
origin bucket and full access to
destination bucket. If you use an IAM user with administrator privileges you are fine. You should also set up AWS CLI and programmatic access in your workstation (notebook, desktop, etc). The bucket policy we created in the previous step grants permissions to
destination account in root level. To be able to make AWS CLI actions on
origin bucket we need an IAM user which is granted with permissions by the root account of
Copying S3 Objects Using AWS CLI
As you may already know, AWS CLI is the command line interface tool to access AWS resources from command line. We will use
s3 sync feature which copies one bucket to another.
destination-bucket-name with the names of the buckets below.
This will copy all objects from
origin bucket to
destination bucket along with their metadata. However, if your objects are large and AWS CLI uses multi-part uploads, their metadatas would not be copied.
As you see we did not define any access control list during the command. We can use
--acl parameter for this purpose and provide canned ACLs to apply to all objects.
In most cases, if your objects are small, these would be enough. But, if you need more, please check s3 sync documentation.
What about making the sync operation in reverse way? Accessing destination bucket from origin account…
Actually, we finished the task here. However, what if we perform the operation as one of
origin account’s IAM users and defined bucket policy permissions in
destination bucket? In this case, after sync operation, owner of the new objects would remain as
destination account would only be able to read or delete them, although the objects are stored in the
destination bucket which they own.
It is an interesting issue and AWS differentiates bucket and object ownerships. In this case,
destination account grants permission to
origin account to place their objects; however,
destination bucket does not own the object unless
origin account grants ownerships. To me, it is like
destination account rents
destination bucket to
origin account having the right to evict the objects any time but not change them in any way while these objects are in this bucket.
destination account also owner of the objects stored in
destination bucket we could use
--acl bucket-owner-full-control parameter. However,
origin account would also preserve full control on objects as they are doing the copy operation.
This issue is about bucket and object ownerships and you may come accross during AWS Certification exams. Therefore, I find it useful to mention here. But, if you only to do job, never mind and use the first approach and you are good to go.
In this post, we discussed how you can transfer S3 bucket objects from one account to another using high-level S3 commands. S3 sync is a useful tool and with a good understanding of bucket policies you can transfer your objects easily.
I should note that S3 sync is a high level operation. If you need more features you can always use AWS SDKs. Besides, event triggers, notifications need to be migrated in a different way, because they are not supported by S3 sync command.
In case you need help, contact me and we can find the best solution for you together.
Thanks for reading!