In some cases, you may need to transfer your objects in one of your Amazon S3 buckets to a different AWS account. AWS CLI provides high-level commands on S3 to move objects between two buckets. By also using Amazon S3 bucket policies, you can perform this even if the destination bucket is in another AWS account. Today, I will talk about the steps you need to do along with a few explanation on bucket and object level permissions.
Terminology
As a start, let’s define a terminology for the rest of the post to make things clear.
origin bucket: An existing S3 bucket which contains objects that will be moved todestination bucket.destination bucket: An existing S3 bucket which the objects inorigin bucketwill be moved to.origin account: AWS account whichorigin bucketbelongs to.destination account: AWS account whichdestination bucketbelongs to.
Granting Permissions to Origin Account in Destination Bucket
We will use aws s3 sync command to copy all the objects in origin bucket to destination bucket and we wil do this as an IAM user in destination account which has full access to destination bucket.
S3 sync command will read objects from origin bucket and copy them by creating as new objects in destination bucket. Therefore, in some way we need to grant necessary permissions to destination account to be able to access origin bucket and have complete read access. To do this, we will use S3 bucket policies which will grant resource level permissions to destination account.
1) Sign in AWS Console in origin account and navigate to origin bucket. 2) Click Permissions tab and then, Bucket Policy button. 3) In Bucket Policy Editor paste and edit the policy below and save.
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "GrantCopy",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::<destination-account-id>:root"
},
"Action": [
"s3:Get*",
"s3:List*"
],
"Resource": [
"arn:aws:s3:::<origin-bucket-name>",
"arn:aws:s3:::<origin-bucket-name>/*"
]
}
]
}In this JSON, replace <destination-account-id> with AWS Account ID of destination account and <origin-bucket-name> with the name of the origin bucket. As you can understand easily, this policy grants read privileges (s3.Get* and s3.List*) to destination account at root level.
4) In destination account create an IAM user which has read access to origin bucket and full access to destination bucket. If you use an IAM user with administrator privileges you are fine. You should also set up AWS CLI and programmatic access in your workstation (notebook, desktop, etc). The bucket policy we created in the previous step grants permissions to destination account in root level. To be able to make AWS CLI actions on origin bucket we need an IAM user which is granted with permissions by the root account of destination account.
Copying S3 Objects Using AWS CLI
As you may already know, AWS CLI is the command line interface tool to access AWS resources from command line. We will use s3 sync feature which copies one bucket to another.
Now, replace origin-bucket-name and destination-bucket-name with the names of the buckets below.
aws s3 sync s3://origin-bucket-name s3://destination-bucket-name This will copy all objects from origin bucket to destination bucket along with their metadata. However, if your objects are large and AWS CLI uses multi-part uploads, their metadatas would not be copied.
As you see we did not define any access control list during the command. We can use --acl parameter for this purpose and provide canned ACLs to apply to all objects.
aws s3 sync s3://origin-bucket-name s3://destination-bucket-name --acl public-readIn most cases, if your objects are small, these would be enough. But, if you need more, please check s3 sync documentation.
What about making the sync operation in reverse way? Accessing destination bucket from origin account…
Actually, we finished the task here. However, what if we perform the operation as one of origin account’s IAM users and defined bucket policy permissions in destination bucket? In this case, after sync operation, owner of the new objects would remain as origin account. destination account would only be able to read or delete them, although the objects are stored in the destination bucket which they own.
It is an interesting issue and AWS differentiates bucket and object ownerships. In this case, destination account grants permission to origin account to place their objects; however, destination bucket does not own the object unless origin account grants ownerships. To me, it is like destination account rents destination bucket to origin account having the right to evict the objects any time but not change them in any way while these objects are in this bucket.
To make destination account also owner of the objects stored in destination bucket we could use --acl bucket-owner-full-control parameter. However, origin account would also preserve full control on objects as they are doing the copy operation.
This issue is about bucket and object ownerships and you may come accross during AWS Certification exams. Therefore, I find it useful to mention here. But, if you only to do job, never mind and use the first approach and you are good to go.
Conclusion
In this post, we discussed how you can transfer S3 bucket objects from one account to another using high-level S3 commands. S3 sync is a useful tool and with a good understanding of bucket policies you can transfer your objects easily.
I should note that S3 sync is a high level operation. If you need more features you can always use AWS SDKs. Besides, event triggers, notifications need to be migrated in a different way, because they are not supported by S3 sync command.
In case you need help, contact me and we can find the best solution for you together.
Thanks for reading!
