Shikisoft Blog

Amazon S3 Storage Classes & Lifecycle Management: Optimizing Your Cloud Object Storage

2025-04-30T00:00:00+00:00

As a popular object storage service, Amazon S3 allows you to store data in buckets. However, in most cases, some data becomes irrelevant and less frequently accessed over time. If you leave things as they are, the data will continue to take up space, leading to extra storage costs, not to mention performance inefficiencies.

The good news is that Amazon S3 provides various storage classes for different access patterns that align with your storage performance and cost requirements. Besides, by configuring S3 lifecycle policies, you can transition objects to other storage classes or permanently delete them after a specific period.

This post will guide you through AWS S3 storage classes and introduce lifecycle policies to help you develop a cost-optimized and effective data management process.

S3 Standard & Express One Zone for Frequently Accessed Objects

AWS offers the S3 Standard storage class for frequently accessed objects and the S3 Express One Zone for the most frequently accessed objects requiring high-performance storage. As its name implies, the S3 Express One Zone stores objects in only one AWS Availability Zone, making it a lower-cost option than the S3 Standard.

However, in case of any problem with that AZ, your data may be lost or damaged, since there is no data replication across multiple AZs. Meanwhile, with the S3 Standard storage class, the objects are stored across several Availability Zones. So, the S3 Standard storage class offers more resiliency than S3 Express One Zone. Therefore, we advise you to choose S3 Express One Zone only for your reproducible infrequently accessed data, so as not to take any risk.

What are Your Options for Infrequently Accessed Objects?

Objects you don’t need frequently but still require millisecond access can be transferred to the S3 Standard-IA (Infrequent Access) or S3 One-Zone IA storage class. These classes offer the same durability and low latency as the S3 Standard and S3 Express One Zone storage classes, respectively.

S3 Standard-IA and S3 One-Zone IA both offer lower cost than their frequently accessed storage classes. However, there is an extra retrieval fee when the object is accessed. So, you should only use them if the object is infrequently accessed, a few times a month.

There is a minimum duration for S3 Standard-IA and S3 One Zone-IA storage classes, which is 30 days. If you delete the object or move it to another class within this period, you still pay the 30-day full price in that class in addition to the costs of any class for the remaining days.

Besides, the minimum object size is 128 KB in S3 Standard-IA and S3 One Zone-IA. So, if the object is smaller than 128 KB, you still pay the 128 KB cost. Hence, these classes may not be feasible for small objects.

S3 Glacier Storage Classes: ‘Freeze’ Your Rarely Used Objects in the Glacier

Until now, we have discussed storage options for frequently or infrequently accessed objects. But what about the rarely accessed data you must keep for archival purposes, regulatory requirements, etc.? Amazon S3 Glacier storage classes are built for this purpose. You can choose among three archival storage classes offering different retrieval times and storage costs: S3 Glacier Instant Retrieval, S3 Glacier Flexible Retrieval, and S3 Glacier Deep Archive. Let’s analyze each S3 Glacier storage class.

S3 Glacier Instant Retrieval

Suppose you have rarely accessed data but still need immediate access. The S3 Glacier Instant Retrieval storage class offers millisecond retrieval time, the same as S3 Standard or S3 Standard-IA, with much lower storage costs. It can be suitable for cases like image hosting, storing health records that aren’t used generally but need immediate access during a doctor’s appointment, or archived photos or videos that sometimes require instant access.

So, if you have data you access only once per quarter, the S3 Glacier Instant Retrieval storage class can save you a lot on storage costs. S3 Glacier Instant Retrieval has a minimum storage duration of 90 days.

S3 Glacier Flexible Retrieval

This storage class is ideal for archiving data you access semi-annually. As its name implies, the S3 Glacier Flexible Retrieval storage class offers flexible retrieval times from minutes to hours, unlike the S3 Glacier Instant Retrieval. However, it has the same minimum storage duration as S3 Glacier Instant Retrieval, which is 90 days.

When you restore the object, you can choose from three retrieval tiers.

The expedited retrieval option generally restores an object within 1 to 5 minutes.
Meanwhile, the standard retrieval takes 3 to 5 hours.
The third option is bulk retrieval, which is free, but you must wait for 5 to 12 hours to access your objects.

So, choosing the one that fits your specific use case is your call.

S3 Glacier Flexible Retrieval is usually recommended for backup or disaster recovery because it allows you to archive large sets of data and retrieve them at a lower cost. However, once the objects are archived, you cannot access them in real time. You must submit a restore request to access stored objects and wait until it completes.

S3 Glacier Deep Archive

Finally, let’s discuss the most cost-effective storage class, and my favorite: S3 Glacier Deep Archive. As the name suggests, this storage option is designed to store rarely accessed data deep down the ‘glacier’. It is especially recommended for meeting regulatory requirements or long-term archiving, as it allows storing large amounts of data for many years.

Like S3 Glacier Instant Retrieval, real-time access is unavailable for the S3 Glacier Deep Archive. So, you must create a restore request to access your objects. After that, you can choose one of the two retrieval tiers for your use case.

The standard retrieval option generally retrieves your objects within 12 hours.
Whereas bulk retrieval can take up to 48 hours.

Also, the S3 Glacier Deep Archive differs from other S3 Glacier storage classes with its minimum storage duration of 180 days.

As you will read in my use case below, you can also use Amazon S3 Glacier Deep Archive for backups and disaster recovery if your business recovery time objective (RTO) allows you to wait up to 72 hours to restore from your backups.

S3 Intelligent Tiering: What if you don’t know how often your objects are accessed?

You may have objects whose access patterns you can’t predict. In this case, you can choose the S3 Intelligent-Tiering storage class for automatic cost savings. Once you use this class, AWS will automatically monitor your objects’ access patterns and transfer them to the most effective access tier for you. The objects will be moved between three access tiers based on the access frequency: Frequent Access, Infrequent Access, and Archive Instant Access.

When first transitioning to the S3 Intelligent Tiering storage class, the objects are stored in the Frequent Access tier by default.
Then, if an object has no access for 30 consecutive days, it will be moved to the Infrequent Access tier.
Lastly, the objects not accessed for 90 days will be moved to the Archive Instant Access tier. These tiers offer the same low latency level as the S3 Standard while helping you save on storage costs.

Amazon S3 also offers two optional archiving tiers, Archive Access and Deep Archive Access, which you can activate under the S3 Intelligent Tiering storage class. They correspond to S3 Glacier Flexible Retrieval and S3 Glacier Deep Archive storage classes, respectively.

Automating S3 Storage Management with Lifecycle Policies

If your object’s access pattern changes over time, you can automatically move it to lower-cost classes and eventually delete it when it is no longer relevant to keep. By configuring S3 lifecycle policies, you can transition objects to different storage classes or permanently delete them after a specific period.

An S3 lifecycle policy consists of rules that will apply to your objects throughout their lifetime. You can set up a transition action to automatically move an object to a different storage class after a specified period. Also, you can define a delete action to delete an object permanently.

Its primary difference from S3 Intelligent Tiering is that it doesn’t take objects’ access patterns into account, and you can customize the duration between the transition actions as long as it aligns with the minimum storage duration. For example,

You can move an object automatically to Standard-IA 45 days after creation. You can’t move an object ot Standard-IA or One Zone-IA before 30 days in S3 Standard.
However, Standard-IA has a 30-day minimum storage duration. Hence, you can define another rule to move it to S3 Glacier storage classes after an additional 30 days (total: 75 days).
If you don’t have to keep the object, you can just delete it without moving to S3 Glacier storage classes. You can align the rule according to your use case.

My Use Case: Backup Storage for Final Cut Pro X Libraries

Let me give you another example. As you may already know, I have online courses on Udemy. While producing these courses, I use Final Cut Pro X to edit my videos. So, I occasionally take backups of my Final Cut libraries and upload them to S3.

These backup files are large and take up GBs of storage space. I only access these files if there is a problem on my local disk drives for disaster recovery. Keeping them in the S3 Standard class would cost me a lot, considering the new versions I continue to upload. So, I needed a cost-effective solution for my backup storage.

As a solution, I defined an S3 lifecycle rule to move the object to S3 Glacier Deep Archive storage class on day 0, as I don’t need to access it unless there is a disaster in my office. In that unlikely scenario, I can wait up to 72 hours to restore the backup library.

Besides, to keep the storage costs down even more, another rule automatically deletes any previous versions from the S3 Glacier Deep Archive after 6 months. It is because if I didn’t need a previous version in 6 months, I almost certainly won’t need it later. However, I always keep the latest version in the S3 Glacier Deep Archive. This way, I decreased my backup storage costs on S3 by 10 times.

Conclusion

Amazon S3 provides different storage classes to manage your data storage efficiently and cost-effectively. S3 Lifecycle Management is also an automated way to move your objects between different storage classes or permanently expire them. In this post, we analyzed the main features of each S3 storage class to help you learn how to use S3 lifecycle policies for your unique cases. I hope you can benefit from it.

Thanks for reading, and see you in our other posts.

References

Which Amazon EC2 Instance Type is Right for You?

2025-03-25T00:00:00+00:00

Amazon EC2 instances allow you to create your own virtual servers on the cloud, freeing you from the burden of maintaining your own physical architecture. EC2 provides various instance type choices to host your applications. However, having more choices may sometimes confuse you, and you may feel overwhelmed when choosing the optimal one. So, let me help you by simplifying EC2 instance types in this post.

What is an EC2 instance type?

When creating an EC2 instance, you choose an instance type that determines the hardware characteristics of the physical host server on which it will be placed. The instance type also specifies its size and how much you will pay per minute when it is running.

You may see weird characters in instance types, but each has a purpose. The instance type naming follows a specific pattern. First comes the instance family part, which is followed by the instance size after a dot.

The instance family begins with the series. The series is the most critical part of the instance type because it specifies the main characteristics.
The series is followed by the generation number of the series. Over time, AWS upgrades physical host servers and releases new generations.
Next comes the options part of the generation, which includes a few optional letters or words. These options may represent the CPU architecture of the instance family, such as g for Graviton, i for Intel, or a for AMD. An instance type may also include other options, such as n for Network and EBS optimized or d for those providing instance-store storage. These are the current options:
- a – AMD processors
- g – AWS Graviton processors
- i – Intel processors
- b – Block storage optimization
- d – Instance store volumes
- e – Extra storage or memory
- flex – Flex instance
- n – Network and EBS optimized
- q – Qualcomm inference accelerators
- z – High CPU frequency

The size part after the dot (.) is simple: nano, micro, small, medium, large, xlarge, 2xlarge, etc. It goes on as powers of 2. It determines how much resources will be allocated to the instance. Therefore, together with the instance family, the size finalizes the instance’s cost per minute.

Amazon EC2 provides various instance types with distinct advantages and use cases. The instance families are also grouped into categories. Now, let’s discuss them one by one. I also added some keywords in parenthesis to help you remember the purpose of each instance series.

General Purpose Instance Types

This category offers instance types with balanced memory, compute, and networking resources. They are feasible if your application uses these resources equally, such as web servers.

M Instances (Medium)

The M series instances are in this category, which provides consistent performance and is ideal and cost-effective if you know the expected load on your application most of the time.

T Instances (Threshold)

The other option is the T-series, which are burstable instances. These instances normally operate at a baseline performance (threshold), such as 20%, and accumulate CPU credits when the load is below the baseline. They use these credits to burst to 100% CPU capacity if the load becomes high later. Hence, they are cost-effective when you don’t know your apps’ load characteristics yet or if your app’s load is mostly low. But don’t expect consistent CPU performance as in M instances because they always operate at 100% CPU capacity.

Currently, t4g.micro, t3.micro, and t2.micro instances provide AWS free-tier benefits with 750 hours/month cumulatively. So, I mostly use these instance types during hands-on examples in my courses.

Mac Instances (Mac)

Mac instances are also in this category. They provide on-demand macOS instances on the cloud. So, if you are developing apps for the Apple ecosystem, such as iPhone, Mac, or Apple TV, you can use these instances to develop, build, and test your apps.

Compute-optimized Instance Types

The second category is the compute-optimized instances. They offer high-performance processors for compute-intensive applications. The C-series instances belong to this category.

C Instances (Compute or CPU)

The instances in this series have more virtual CPU cores per GB of memory. They can be used for media transcoding, high-performance web servers, high-performance computing (HPC), scientific computing, batch processing, machine learning inference, and anything else that requires high-performance CPUs.

Memory-optimized Instance Types

Next, you have memory-optimized instance types. The instance types in this category provide fast performance for memory-intensive applications that process large data sets in memory.

There are multiple instance series in this category.

R Instances (RAM)

The R-series instances are designed for standard memory-intensive applications, such as open-source relational databases like MySQL, in-memory caches like Redis, or real-time big data analytics.

X Instances (Extra RAM)

The X-series provides more memory capacity than its R-series counterparts in the same sizes with similar use cases that require more memory.

U Instances (Ultra Large)

U-series instances are designed for large enterprise databases, such as SAP HANA. They are huge and have very high memory and processor capacity.

Z Instances (Z - A sign for High-frequency)

The Z-series is new and offers both high-frequency compute and high memory capacity. It also provides large instance store capacity to increase performance.

Accelerated Computing Instance Types

In accelerated computing, you offload some tasks to specialized hardware accelerators, often GPUs and other co-processors, to speed up processing with parallelization. For example, suppose your app performs graphics processing, data pattern matching, or scientific floating point number calculation. In that case, using a hardware accelerator is more efficient than relying on a standard CPU. So, the instances in this family offer hardware accelerators or co-processors in addition to virtual CPUs.

They are ideal for deep learning and generative AI applications like

code generation,
speech recognition,
video and image generation,
high-performance computing (HPC) applications such as
- computational fluid dynamics,
- weather forecasting,
- computational finance,
graphics processing like
- real-time rendering,
- cinematic-quality graphics,
- game streaming, etc.

Yes, all these require specialized hardware.

G Instances (GPU)

The first members of this category are G-series instances. They are GPU-based instances with Nvidia GPUs for standard machine learning and high-performance needs.

P Instances (Power)

The P-series instances also have Nvidia GPUs. They provide more GPUs and GPU memory than G-based instances, so they are for applications that demand more GPU capacity.

Trn Instances (Trainium)

The Trn-series instances contain AWS’s Trainium chips as hardware accelerators, which are designed for high-performance machine learning training and inference workloads.

Inf Instances (Inferentia)

Another instance series with purpose-built chips is the Inf series. These instances contain AWS’s Inferentia chips specially designed for deploying deep learning and generative AI inference workloads.

DL Instances (Deep Learning)

The DL series is dedicated to deep learning and contains Gaudi accelerators from Habana Labs or Qualcomm AI 100 accelerators.

F Instances (FPGAs)

The F-series instances have field-programmable gate arrays (FPGAs), which are suitable for genetics research, financial analytics, real-time video processing, etc.

VT Instances (Video Transcoding)

The VT series instances are designed for video transcoding needs such as broadcasting live events and video conferencing.

Storage-optimized Instance Types

The next category is storage-optimized instances. These are feasible if your workloads perform high, sequential read/write operations to large data sets in local storage or perform tens of thousands of low-latency, random input-output operations per second (IOPS).

These two access patterns require different types of storage: solid-state drives (SSDs) and hard disk drives (HDDs).

I Instances (IOPS)

The I-series instances have solid-state drive (SSD) disks with high input-output (IO) performance because they are designed for workloads that require real-time, random, and low-latency access.

These applications frequently access random parts of storage, but they return relatively smaller data chunks than others. They are ideal for workloads such as relational and No-SQL databases, real-time databases, and real-time analytics engines.

D Instances (Dense)

On the other hand, the D series instances offer high performance for sequential, not random, input-output operations and high disk throughput. They have hard disk drives (HDDs) instead of SSDs, providing better performance and lower costs for those needs.

You can use them for distributed file systems such as MapReduce File Systems, massively parallel data warehouses such as Redshift, big data analytics workloads such as Spark or Hadoop, or data processing apps such as Kafka or Elasticsearch.

H Instances (Hard Disk - HDD)

Alternatively, you can opt for H-series instances if you need instances with HDD disks that balance CPU and memory. They have less memory per CPU than D-series instances but are cheaper.

HPC Instance Types (HPC)

Lastly, there is the HPC-optimized category. These instances are designed to provide the best price-performance ratio for high-performance computing needs. This category has only a single instance series with the same name.

HPC Instances

The HPC-series instances are in this category. They don’t have hardware accelerators or co-processors; they only have high-performance CPUs. They are cost-effective and feasible for HPC applications with moderate needs, such as weather forecasting or molecular dynamics.

Conclusion

Amazon EC2 provides various instance types for different use cases and budgets. For example, you can use compute-optimized, accelerated computing, or HPC instances for high-performance computing. Each provides a different price per performance. So, it all depends on your application’s type and how much you are willing to spend. Solutions architecting is about choosing the optimal solution that aligns with your budget and needs. I hope this post will be helpful as a summary of EC2 instance types.

By the way, the contents of this blog and the screenshots are from my incoming course on AWS Certified Cloud Practitioner certification. It will help you learn AWS basics while earning a crucial certificate. Follow me on LinkedIn and X.com (formerly Twitter) to know when it is ready.

Thanks for reading, and see you in our next post!

References

Amazon EC2 Instance types

AWS Certified Solutions Architect – Professional: Renewing My Certification Again in 2024

2025-02-28T00:00:00+00:00

Three months ago, on the last day of November last year, I took and passed the famous AWS Certified Solutions Architect – Professional exam for the third time. Due to recent health problems, I couldn’t study as much as I had hoped. Still, I passed with a decent score of 837/1000 and extended it to 2027. I also renewed my first AWS certificate, AWS Certified Solutions Architect – Associate with this exam and guaranteed to be AWS certified for over 10 years.

The exam was different each time I took it. Thus, as of the end of 2024, I want to share my experiences for the third time. I hope it also helps you in your AWS certification journeys.

How did I prepare for the AWS Certified Solutions Architect – Professional exam?

My exam preparations primarily consist of these steps:

Viewing the exam blueprint.
Reading AWS whitepapers if necessary.
Reading each service’s FAQ and, optionally, the primary documentation if it is essential.
Viewing recent re:Invent sessions about services from YouTube when needed.
Taking notes and trying the services included hands-on.

I haven’t taken online certification courses because I am also planning to build one and didn’t want to learn from other instructors and develop my knowledge. Of course, this was the hard way. However, this way, my future courses will be genuine and help you from another perspective.

I already renewed my AWS Certified DevOps Engineer – Professional certification two months before taking the AWS Certified Solutions Architect – Professional exam. Hence, I didn’t have to study DevOps topics for this exam. I just reviewed some of my notes to recall their use cases as a solutions architect.

Reading the documentation and trying the service help greatly. You understand the service well. Watching AWS re:Invent videos is also very valuable because you need to understand AWS services from a solutions architect’s perspective.

About the AWS Certified Solutions Architect – Professional exam content

The AWS Certified Solutions Architect – Professional exam is the same as the AWS Certified DevOps Engineer exam in style but is very different in terms of content. It is not as detailed as the DevOps exam. However, its questions are more scenario-based, mostly with answers asking to select 2-3 choices together, and you must decide what is best for the situation asked as an AWS solutions architect. Mostly, you don’t have to know the exact configuration for a service’s resources in detail. It is more like you must know where to use the service.

As far as I remember, here are the key topics asked in the exam. I divided them into categories to make them easier to identify. However, most questions actually fall into multiple categories.

Organizations

As in all AWS Certified Solutions Architect – Professional exams, AWS Organizations take a central place. However, it is asked more with other domains because it is an organizational structure.
You must know Organizations in general and how to share resources between accounts with the AWS Resource Access Manager.
Applying tag policies across your organization also appeared.
Trusted advisor and GuardDuty usage in a multi-account organizational setting is also crucial.
Know AWS Control Tower in general. Although it was not asked for in detail, like in the DevOps exam, it is still crucial.

Migration

Migrating an on-premises server and data with minimal effort with AWS Systems Manager, AWS Backup, and VM Import/Export
Connecting to an on-premises database to collect database and table information to perform migration analysis. You should know the AWS Database Migration Service and Schema Conversion Tool (SCM) features.
Performing a migration analysis with or without installing anything on the servers: AWS Migration Evaluator, AWS Application Migration Service agent-based and agentless discoveries. Reading the migration whitepaper may be helpful.
Transferring data to AWS with AWS DataSync, AWS Transfer Family, or AWS Storage Gateway.
Migration of OLAP DBs and Oracle DBs to Redshift.
Knowing to use AWS Outposts in migrations to AWS.

Security & Compliance

Knowing how to use IAM permission boundaries and Security Control Policies (SCPs) to limit the permissions of an IAM identity is crucial. You should also know their differences.
AWS Config: How to be notified with SNS when a user changes something in an AWS account without preventing her from doing it.
Differences between AWS CloudHSM, Amazon KMS, and Amazon S3 encryption
Using Security Hub as a central place for security in your AWS organization.
Using AWS Web Application Firewall (WAF) with ALBs and API Gateway APIs
Scanning Amazon ECR images with Amazon Inspector automatically
Know how Identity Center SAML authentication works and how it is configured.
Learn what Audit Manager is used for. It appeared in a few questions.

Serverless

API Gateway was an essential topic in the exam. You should know AWS solutions for:
- APIs accessed from multiple regions
- Private APIs accessed only from VPCs
- APIs triggering AWS Lambda functions
Hosting static sites on Amazon S3 with Amazon CloudFront
Using Amazon RDS access points in Lambda functions ( It may also be considered in the DB category. )
Step Functions were not asked as detailed as in the DevOps Engineer exam. However, they appeared in answers to a few questions. Learn how to orchestrate API calls and Lambda function executions with Step Functions.

Networking

Networking with AWS Transit Gateway and VPN Gateway and connecting to multiple VPCs using a dedicated DirectConnect connection.
Using AWS Firewall Manager to automatically add AWS WAF rules across accounts and new accounts. Using AWS Systems Manager Parameter Store to store parameters.
Routing policies on Amazon Route 53 had a few questions. You may be asked why a specific failover configuration doesn’t work or design a failover solution for a multi-region architecture.

Databases

Solutions for creating leaderboards: The differences between Amazon MemoryDB and Amazon ElastiCache for single-digit latency with the ability to preserve data for later analysis. You may also be asked to do it with Amazon DynamoDB.
Caching Amazon DynamoDB tables with DAX, using DynamoDB Global Tables in multi-region architectures, and DynamoDB auto-scaling were asked.
Some questions involved Aurora and Aurora Global Databases. Amazon RDS was also asked, but less than Aurora. Know how to use Aurora readers.
Knowing Amazon Redshift, in general, also helps. Amazon Neptune was among the options in some questions but not directly mentioned.
Using in-memory DBs for high-performance computing.
Amazon DocumentDB appeared in a question. Amazon Timestream was among the options in some questions.

Storage

You should know Amazon S3 access points and their use cases.
Amazon EFS and Amazon EBS differences. Configuring multi-access to EFS filesystems from other EC2 instances.
Knowing the platforms in Amazon FSx family services would help. For example, which one supports NFS and SMB? The questions were primarily about Windows File Server, but Lustre also appeared.

DevOps Tools

Architecting a CI/CD pipeline using AWS CodePipeline, AWS CodeBuild, and AWS CodeDeploy services with minimal effort.
Using the Service Catalog to launch verified AWS CloudFormation templates without affecting the current user’s abilities.
Container solutions with AWS AppRunner, Amazon ECS, Amazon EKS, and AWS Fargate. Know what they are used for. It was not as detailed as the DevOps exam.
Elastic Beanstalk appeared only once as an answer to launching EC2 instances behind an ALB and performing Blue/Green deployments with minimal effort. The DB was separated naturally.
Know Amazon X-Ray and tracing in general. There were a few questions about Amazon CloudWatch logs and metrics, but it was not emphasized as in the DevOps exam.

Data Analytics

Using Kinesis Data Firehose and Data Streams to stream CloudTrail and CloudWatch logs to S3 and other target locations.
Know the use cases of AWS Glue, Athena, and QuickSight.
Learn Amazon EMR in general. It appeared in some questions and answers.
Know Amazon OpenSearch in general.

Application Integration & Messaging

Know how to use Amazon EventBridge to route events from an API to multiple consumers.
Know the differences between Amazon SQS standard and FIFO queues, SNS standard and FIFO topics, and Amazon MQ.
Know Amazon SES in general. It appeared in some questions and answers but not in a detailed way.

Cost Optimization

Know the differences between Saving Plans, on-demand, reserved, and spot instances.
Learn how to merge EC2 saving plans, compute plans for Fargate and Lambda, Lambda (reserved concurrency or on-demand), and database reserved instances to create a cost-effective solution for a given scenario.
Learn S3 storage classes, such as moving to S3 standard infrequent access or S3 Glacier instant retrieval, to achieve a cost-effective solution in a given scenario.
Know cost explorer and cost allocation tags and how to ensure they are included in resources with AWS Config without affecting developers.
Know AWS Compute Optimizer in general.

End-user Computing

Learn use cases of Amazon Workspaces.
Amazon AppStream also appeared in some answers but was not asked directly.

Machine Learning & IoT

SageMaker appeared in a few questions.
Only IoT Core was in the exam from the AWS IoT services.

My Exam Experience with Pearson Vue

I took the exam from my home office like the previous one with online proctoring from Pearson Vue. As always, I checked in, received a link to take pictures of my environment, and connected to a Vue proctor who checked my desktop and elbows (You shouldn’t wear watches.) before starting the exam.

The exam experience was smooth. I encountered no issues. Although I didn’t see the exam result immediately, I received it a few hours later.

Conclusion

After every exam I take, I write my experiences in a blog post. I passed the AWS Certified Solutions Architect Professional for the third time three months ago but had a chance to write about it now. I hope it guides you in the right direction while preparing for it.

I am also developing courses for AWS certifications. While there are many courses from other successful instructors, my courses will help you by leveraging my years of experience and my teaching style. I will start with the foundational certifications but eventually launch professional ones.

So, stay tuned. You can also follow me on LinkedIn and X.com (formerly Twitter) to be notified when my AWS certification courses are ready.

References

My AWS Certified Solutions Architect - Professional certificate on Credly!

IaaS vs. PaaS vs. SaaS: Cloud Computing Service Types Explained

2025-01-21T00:00:00+00:00

There are three primary types of cloud computing services, depending on how much management the cloud user or the cloud provider performs. As the level of management increases, so do the responsibilities. These are Infrastructure as a Service (IaaS), Platform as a Service (PaaS), and Software as a Service (SaaS).

Generally speaking, ‘X as a Service’ means X is offered to you as a service by a third party. So, in the cloud domain, this can be an infrastructure, a platform, or software provided by the cloud provider, such as Amazon Web Services (AWS). You can leave managing it to the cloud provider and focus on your business.

In this blog post, I will explain cloud service types and their differences so you can clearly understand your responsibilities.

Infrastructure as a Service (IaaS)

The first cloud service type is Infrastructure as a Service (IaaS). In this business model, the cloud provider provides essential IT infrastructure, such as compute, storage, and networking, with virtualization.

You request and configure the necessary resources to run your application, which is often a cloud server with an operating system, some storage, and a network connection. But of course, these can be different resources. In the end, you are billed only for the resources you use. So, you pay as you go.

It is your job to maintain the operating system and runtime through security patches, etc. Besides, you code, deploy, and support the software application in the long run. You are also responsible for choosing the correct configuration and protecting the data.

In this model, you are more flexible because you have more control over your cloud resources. The advantage is that you leave infrastructure management to the cloud provider.

What about examples?

Amazon EC2 is a good example of Infrastructure as a Service (IaaS) on AWS. It provides virtual servers as EC2 instances with many configuration options, including compute sizes, storage, networking, etc.
Another example is Amazon Lightsail, which is a simplified version of EC2 instances with packed features and more comprehensible pricing.

In both services, AWS provides a virtual server with the operating system and other components, which you maintain.

Platform as a Service (PaaS)

The other primary cloud service type, Platform as a Service (PaaS), involves the cloud provider taking over some of your management tasks and responsibilities. In PaaS, the cloud provider also manages the operating system and runtimes, as well as the infrastructure and virtualization.

When a new vulnerability is exposed, the cloud provider patches the operating system for you. So, you don’t have to worry about its maintenance, and AWS has tens of personnel who readily do it for you.

Your job is coding and deploying your software applications and maintaining them. It is also your responsibility to configure the platform correctly and secure and protect your data with those configurations.

What about some examples?

The first is AWS Lambda, which allows you to deploy code in supported programming languages and frameworks, such as Node.js, Python, Go, or Ruby, and execute it whenever needed. It also handles all resource provisioning and operating system maintenance.
The other example is AWS Elastic Beanstalk. You provide a software package in a supported platform, such as Ruby on Rails, Node.js Express, Python Django, or Java Tomcat, and let Elastic Beanstalk provision and manage EC2 instances for you.

Software as a Service (SaaS)

The last cloud service type we will cover is Software as a Service (SaaS). In this business model, the cloud provider performs most of the management tasks, enabling you to focus on your own business.

The service provider handles all infrastructure management, application deployments, data storage, and maintenance. You don’t need to worry about application development and maintenance or anything related to the lower details. You only use the software through an Internet browser, desktop, or mobile app. You pay for your usage as a subscription or on a pay-as-you-go basis.

But don’t you have any responsibilities then?

Yes, you have. Although the service provider manages the data storage, you are still responsible for correctly configuring the service to protect and back up your data.

What about an example?

AWS WorkMail is a fully managed business email and calendar service, like Google Workspace. It is an excellent example of Software as a Service (SaaS) on AWS.

When using AWS WorkMail, you sign up for a monthly fee and use the service via your browser or an email app like Microsoft Outlook or iOS Mail to check your emails. You don’t care how the app is deployed or where the data is stored.

What about AWS services?

Although I have given examples of these cloud computing service types, many AWS services don’t fall into a single category. For example, one can argue that AWS Lambda is an Infrastructure as a Service (IaaS) because sometimes you configure its VPC.

However, if you take VPC settings in AWS Lambda as a platform configuration, it can still be considered in the Platform as a Service (PaaS) category. Besides, not all AWS Lambda functions are required to run in a VPC. Still, I understand and respect if your view is different.

Conclusion

Cloud computing services have three primary types: Infrastructure as a Service (IaaS), Platform as a Service (PaaS), or Software as a Service (SaaS).

You are more flexible but have more responsibilities in Infrastructure as a Service (IaaS). In contrast, the cloud provider takes most of the management, deployment, and maintenance jobs from you in Software as a Service (SaaS). Regarding responsibilities, Platform as a Service (PaaS) stays in the middle.

AWS doesn’t care whether a service fits into one of these categories. It only designs services to address your needs. In addition, AWS doesn’t ask you whether an AWS service is Infrastructure as a Service (IaaS), Platform as a Service (PaaS), or Software as a Service (SaaS) in certification exams.

What matters is you understand these concepts. In this blog post, I aimed to help you by simplifying them.

Thanks for reading, and I look forward to seeing you in another blog post! If you’d like to hear more about AWS and cloud computing in general, follow me on LinkedIn and X (formerly Twitter).

Cheers!

Amazon SNS vs. Amazon SQS: A simple comparison

2024-12-09T00:00:00+00:00

When it comes to building a reliable and scalable cloud infrastructure, efficient communication between your cloud components is essential. There are many AWS services to help you build a reliable communication architecture for your solution. So, in this post, we will discuss and compare two old and popular AWS messaging services, Amazon Simple Notification Service (SNS) and Amazon Simple Queue Service (SQS). These services allow you to easily exchange messages in a distributed system.

Amazon SNS and Amazon SQS are fully managed and serverless AWS services that offer automated communication. While they may sound similar, they each have unique features that set them apart. In the following sections, we will delve into these features, providing a comprehensive understanding of each service. Let’s start with Amazon SNS.

The delivery method is the first and most notable difference between Amazon SNS and Amazon SQS. SNS uses an instant ‘push’ mechanism. This means messages are automatically delivered to the subscribed consumers, and you don’t need to check for or ‘poll’ updates. It sends copies of the same message to multiple consumers via topics.

Amazon SNS supports both application-to-application (A2A) communication (with AWS Lambda, AWS SQS, etc.) and application-to-person (A2P) communication (with email, SMS, etc.). Using Amazon SNS, you can build apps that receive push notifications or notify humans to react to system events.

For example, you can subscribe an AWS Lambda function to an SNS topic, which can do custom processing when receiving a message. You can also subscribe an email address to the same topic to be notified about this event. SNS also allows you to send SMS or mobile push notifications via platform-specific services, such as Apple Push Notification Serves for messages to iOS and macOS apps.

On Amazon SNS, messaging revolves around three main components: the SNS topic, the publisher, and the subscriber. The SNS topic serves as a logical access point where you can group multiple endpoints such as HTTPS, an email address, AWS Lambda, or an Amazon SQS queue. When you name an SNS topic, the service understands which endpoints you want to send messages to. Consumers who wish to receive notifications must subscribe to relevant topics. Multiple consumers can subscribe to a single topic, and multiple publishers can send messages. When a publisher sends a message to a topic, all subscribed clients are immediately notified. However, the publisher and the subscriber must have the necessary IAM permissions.

Amazon SNS offers two different topic types: standard topics and FIFO (first in, first out) topics. When using standard topics, messages can be targeted to many endpoints, such as an email address, text messaging (SMS), AWS Lambda, etc. However, messages are not guaranteed to be delivered in the order received by Amazon SNS. Standard SNS topics perform at least once delivery. So, your endpoints must be ready for duplicate messages.

With FIFO topics, you can guarantee strict message ordering in the order received by your topic. But you can only subscribe SQS queues to a FIFO SNS topic and not other endpoints. The messages sent to a FIFO topic are delivered to subscribed SQS queues exactly once.

Let’s leave comparing SNS topic types to another blog post. You can easily set your SNS topic type while creating a topic on your AWS console:

Amazon SQS uses a pull mechanism!

We mentioned the main features of Amazon SNS above. Now, let’s have a look at Amazon SQS.

Unlike Amazon SNS, Amazon SQS uses a ‘polling’ model, which means messages are not automatically pushed. Instead, they are stored in a queue until the consumer pulls and processes messages from the queue.

Amazon SQS is especially suitable for processing large numbers of messages since it allows for parallel processing with multiple queues. It is generally used to decouple distributed systems, microservices, etc. This enables asynchronous communication between different components. So, using SQS queues can boost your user experience as multiple workloads can be independently processed simultaneously.

For example, I used SQS queues in the past to queue image-processing jobs and indexing jobs for Elasticsearch (Amazon OpenSearch now). In each use case, I implemented AWS Lambda functions to process the images or trigger the indexing operation on the Elasticsearch domain. It was a Rails application, and I replaced Sidekiq workers on EC2 instances with this solution. Not only I decoupled my app and job servers, but I also migrated the background jobs to a cost-effective, reliable and scalable serverless architecture.

The diagram below displays how messages are delivered to consumers through Amazon SQS:

Besides, Amazon SQS has the advantage of persisting of your messages. Messages can stay in the queue for up to 14 days, providing better reliability than Amazon SNS. Also, unlike Amazon SNS, only one consumer can process messages at a time, and it only supports A2A communication. You can easily customize the settings of an Amazon SQS queue, such as message retention period, maximum message size, etc.

Variants of SQS Queue Types

Like Amazon SNS, you can use standard and FIFO queues in Amazon SQS. When you use standard queues, messages will be delivered in no guaranteed order. If you need message ordering, you can create FIFO (first in, first out) queues. They enable the messages to be delivered in the same order they are sent to the queue.

You can use delay queues to postpone the delivery of your messages for a specific time. Besides, Amazon SQS offers dead letter queues to route the messages that can’t be processed due to a failure to be reprocessed later. These features increases reliability of your applications.

Which solution would be best for your need?

We analyzed each service. But how can you decide which one suits your use case best? Let’s focus on the main factors you should take into consideration:

1) First and foremost, Amazon SNS and Amazon SQS serve different purposes and operate in distinct ways.

Amazon SNS excels at sending multiple copies of messages to numerous subscribers, making it ideal for notifications, updates, reminders, and alerts. For instance, you can use Amazon SNS to inform subscribers about marketing campaigns or notifying your personnel on CloudWatch alarms.

On the other hand, Amazon SQS is designed to decouple distributed systems, making it the preferred choice for applications that require asynchronous communication between various components.

2) On Amazon SQS, only one consumer can process a message. Also, Amazon SQS allows for batch processing; sending ten messages in one batch is possible. However, messages cannot be sent in batches with Amazon SNS; one message is forwarded to multiple consumers.

3) If you want to be sure your message is delivered to your consumer, you should choose Amazon SQS. Exact delivery is ensured with Amazon SQS in most cases. However, with Amazon SNS, there is no certainty that your message will reach a consumer if the consumer is unavailable for some reason. It can be deleted without consumer receiving the message. So, if reliability is your priority, then you should go with Amazon SQS, which provides message persistency and features like dead-letter queues.

4) If latency is an issue, consider using Amazon SNS. Although Amazon SQS offers better reliability and persistence, its performance speed is slightly lower than Amazon SNS. The reason is that messages need to be polled in Amazon SQS, which takes extra time, while they are immediately pushed with Amazon SNS.

Despite their differences, Amazon SNS and Amazon SQS can complement each other effectively. When used in a fanout pattern, they can significantly enhance performance. This approach leverages SQS’s asynchronous message processing capability, leading to a more efficient system.

In the fanout pattern, a message published to an SNS topic is sent to many SQS queues simultaneously. To fan out messages to SQS queues, you must first create a standard SNS topic. Then, subscribe the SQS queues to that SNS topic. After that, the message will be delivered to all SQS queues subscribed to the topic. This enables parallel message processing.

Let’s clarify this more with a typical example. Let’s say you use Amazon SNS and Amazon SQS together, and suppose you have an application that allows users to upload photos. When a user uploads an image, a message is published to an SNS topic, which will then trigger multiple SQS queues for various tasks. This means that each process, such as image recognition or thumbnail creation, can be performed in parallel, saving time and improving user experience. Your users won’t have to wait for each process to be completed because they will all be handled in the background. This can be a more efficient and user-friendly solution in such cases.

Conclusion

In this post, we discussed the key features of AWS’s two popular services: Amazon SNS and Amazon SQS. We described their different use cases and functions.

To sum up, Amazon SNS is a push notification service usually used to send quick notifications to multiple subscribers. On the other hand, Amazon SQS is a reliable queue-type messaging solution with a pull mechanism. Moreover, in some cases, combining them can be more efficient.

I hope this post helps you understand better the main characteristics of Amazon SNS and Amazon SQS. Thank you for reading! I look forward to seeing you in our other posts.

References

AWS Certified DevOps Engineer – Professional for the Third Time!

2024-10-01T00:00:00+00:00

Recently, I recertified my AWS Certified DevOps Engineer – Professional (DOP-C02) certification by sitting for the exam for the third time. So, I extended my DevOps Pro certification to 9 years and the AWS Certified Developer – Associate and AWS Certified SysOps Administrator – Associate certifications to 10 years. It was a long marathon.

You need to recertify your AWS certifications in every three years. Although it was my third exam for this certification, it was still hard, and I learned new things during preparation. So, let me give you some insights about the exam coverage, how I prepared, and how online proctoring went with Pearson Vue.

The exam content

I don’t know you, but I don’t enjoy taking exams. However, AWS updates the exam content with new services and best practices every few years, so preparation keeps you updated. I learn new things each time and can see how the AWS platform evolves along the way. Besides, passing the exam boosts your confidence and verifies your knowledge.

So, how was it different this time?

Well, in my first exam in 2018, the focus was on CloudFormation, ElasticBeanstalk, AWS OpsWorks, and EC2 AutoScaling Groups for performing Blue/Green deployments.

This changed in 2021 with the addition of AWS developer tools. AWS CloudFormation was still essential, but AWS CodePipeline, AWS CodeDeploy, AWS CodeBuild, and AWS CodeCommit had crucial shares. There were questions about ElasticBeanstalk and OpsWorks in a reduced capacity. Serverless application deployments with AWS Lambda, API Gateway, and Step Functions were introduced. Some new services, such as Systems Manager, Service Catalog, AWS Config, and Inspector, were also in place.

The main difference in 2024 was the exam’s focus on multi-account AWS architectures with Organizations. For most of the services, you should know how to configure it for your organization in a multi-account setting. This is because, nowadays, using multiple accounts for different purposes is one of the primary AWS best practices.

Some of the topics covered in the exam

Here are some of the topics I encountered during the exam. Of course, the list is not full. Knowing the services well should be your goal.

You should know how to configure multi-account AWS architectures through AWS Organizations, AWS Control Tower, and other services.
You should know how AWS CodeBuild works and how to perform specific tasks, such as image signing with AWS Signer.
Understanding the capabilities of AWS CodeDeploy and the deployment lifecycles for EC2, Lambda, and ECS deployments is crucial.
Unlike 2021, AWS CodeBuild and CodeDeploy were often asked with AWS CodePipeline. Using CodePipeline with CloudFormation, manual approval actions, and usage of action variables were significant. I am proud to say my AWS CodePipeline Step by Step course covers a considerable amount, although it is not tailored for the exam.
Of course, AWS CloudFormation is still crucial. However, this time, I didn’t see questions about advanced topics like resource policies or stack policies. I often saw it in conjunction with other services like CodePipeline, Service Catalog, etc.
Using AWS CDK and integrating it into a pipeline was important.
I saw AWS CodeArtifact a few times. You should understand how it works and how to use it in a multi-account setting.
For monitoring and alerting, you should know CloudWatch metrics and logs, CloudTrail, and EventBridge in detail.
Amazon ECS and ECR were critical. However, there were also questions about Amazon EKS, too. The questions focused on integrating EKS with other AWS services or within an AWS Organization.
Using and mounting EFS volumes are essential.
Step Functions, API Gateway, and AWS Lambda and their deployment types are crucial.
AWS Config and Systems Manager are still crucial. I was expecting more about SSM and its excellent features. AWS may focus on it more in the upcoming exams.
The exam included security services like KMS, GuardDuty, AWS WAF, Inspector, Macie, and Network Firewall.
Many other questions appeared regarding disaster recovery and backup scenarios involving Auroradatabases and DynamoDB tables.
Route 53 is still essential as it is the DNS service that helps you in multi-account, multi-region scenarios.
The questions also included Amazon Athena, Glue, QuickSight, OpenSearch, SQS (DLQs), SNS, and Kinesis Data Firehose.
AWS deprecated OpsWorks and CodeCommit. I only saw CodeCommit in a CodePipeline question, but just as a source action. It wasn’t important for the question. So, if you know CodePipeline well, you can easily discard it.
The most significant change is that I didn’t see a single question about ElasticBeanstalk. It was a lovely service when it was launched, but other AWS services offer better DevOps features, so I was expecting that.

I was amazed at how AWS changed the goal of each question. Knowing how to use an AWS service with others in a multi-account organization is more crucial than ever. The exam focuses on more hands-on practice and less on memorizing.

The questions were often long, and there were many questions where you chose three from six options. As always, even if you don’t know an option exactly, knowing that other options are impossible can help you solve the questions through elimination.

Online Proctoring with Pearson Vue

I work from my home office, so taking courses from the comfort of my office is a blessing. That is why I chose online proctoring for the exam. Here is the process.

Before 30 minutes, I started the exam check-in process, which ran standard tests like open applications and internet speed on my computer through the OnVue app.
It sent me a link as a text message, asking me to take a photo of myself, my ID, and my desk from several angles.
Then, the proctor greeted me and asked me to show my wrists and desk using the webcam.
She started the exam. I never left the camera view.
As a non-native English speaker, I took an extra 30 minutes of ESL. However, I finished the exam 20 minutes early. So, I only used 10 minutes from it.

Overall, I had a positive experience. However, the text size was a little smaller and not easy on my eyes, which caused me to be fatigued in the middle of the exam.

The Result and Exam Score

Surprisingly, the exam page didn’t show whether I passed after I finished. This was my eighth AWS exam, consisting of 3 Associate and 5 Professional levels. It showed the exam result with the ‘Passed’ word in the past. This was the first time I didn’t know the exam result immediately after finishing. I saw the result approximately nine hours later in my AWS certification account. After searching the Internet, I learned this is the standard process nowadays. However, knowing that I passed the exam immediately was a relief in the past after the exam fatigue. I really missed it.

I got 883/1000 by using only AWS documentation and practicing myself, as I will discuss my preparation method below. Due to exam fatigue, I missed two easy questions regarding CloudWatch Logs—S3 integration, which would have taken me over 900. Still, I am happy that I passed it with a decent score, where only 750 points would be sufficient.

How did I prepare?

I know you want a quick way to prepare for the exam. I’m sorry, but you won’t get an easy answer from me. I didn’t use any online courses or practice exams.

Knowing the services well and becoming an expert on the topics were more important for me. This is why I didn’t watch any online courses, which often require you to memorize topics. Besides, I am a course creator, so I wanted to understand AWS’s focus on best practices in the exam. I am considering a course focusing on those.

I don’t find the existing practice exams relevant because they often contain confusing answers. They want you to memorize exam questions.

So, how did I prepare?

I read many AWS whitepapers while taking notes. Here are some:
I tried to learn each service covered in the exam blueprint by reading their documentation and making hands-on examples with them.
I watched many AWS Re:Invent videos from 2022 and 2023.
I summarized what I learned in Word documents to review later.

Of course, as the creator of courses about crucial services like CodePipeline, CloudFormation, and CDK, I already have vast knowledge of these topics.

I can’t say enough how my over ten years of AWS experience helped me during the exam. Sometimes, you just know something doesn’t add up through experience.

Conclusion

AWS Certified DevOps Engineer – Professional is one of the most valuable certifications in the cloud domain. I am glad to see how much hands-on practice is required to pass it. Passing this certification for the third time in the last six years was quite an experience for me. Each time I learned new things, the questions became more complex. However, if you know your domain well, you can pass the exam regardless.

Now, the AWS Certified Solutions Architect – Professional recertification for the third time is on my radar. After that, I am planning to build more content for both exams with new courses to help you. So, follow me on LinkedIn and X (Twitter) to stay tuned.

References

My AWS Certified DevOps Engineer - Professional certificate on Credly!

Using AWS CodeArtifact with AWS CodeBuild: An Angular Build Example

2024-08-28T00:00:00+00:00

AWS CodeArtifact enables you to store your custom packages or fetch packages from public package registries and use them in your software development process. If you use AWS CodeBuild to build your code, you can make CodeBuild retrieve the packages required for your build or test commands from your CodeArtifact repository on AWS instead of public Internet registries.

In this post, I will introduce you to AWS CodeArtifact and provide an example of using it with AWS CodeBuild to build an Angular application.

What is AWS CodeArtifact?

AWS CodeArtifact is a serverless package management service provided by AWS. It helps you store, publish, and share software packages using popular package management tools like npm, pip, Maven, RubyGems, etc. You can store your custom packages or packages from language-native public registries in your CodeArtifact repositories.

Why use AWS CodeArtifact?

If you need a central package management solution, maintaining and scaling your own system is costly, time-consuming, and challenging. Instead, you can use AWS CodeArtifact to let AWS manage it for you.

AWS CodeArtifact is serverless. It is hosted on Amazon S3 and DynamoDB tables. Not only does CodeArtifact free you from the server management burden, but it also redundantly stores your repository data in multiple AZs and automatically scales your repositories based on demand theoretically without limits. Hence, AWS CodeArtifact provides durability and high availability out of the box.
It is cost-effective. You only pay for the storage used, the number of requests made, and data transferred out of an AWS region. Besides, the AWS free tier provides 2GB of storage and 100,000 requests per month for free.
It is secure. The data is encrypted at rest with KMS keys, which can be customer-managed or AWS-managed. The data in transit is encrypted with TLS encryption. You use IAM roles and resource policies on your CodeArtifact repositories to control who can access them.
It is integrated with CloudTrail, so you can log who accessed and when for each action taken on your CodeArtifact repositories.
You can create a single CodeArtifact domain within your AWS organization and share it with multiple AWS accounts as a central location for your packages.

Then, what are CodeArtifact domains and repositories? So, let’s continue with AWS CodeArtifact’s primary concepts.

AWS CodeArtifact – Primary concepts

Asset

The individual files representing packages are called assets in CodeArtifact. It is what you download and store when you fetch a package from a package management service. For example, ‘package.tgz’ files in npm packages are CodeArtifact assets.

Package

A package represents a bundle of software with its dependencies. When you install a package, you actually install a package version, which consists of a version number, such as ‘1.5.2’, a metadata and a set of assets.

Repository

A CodeArtifact repository is the primary place to store and fetch packages on CodeArtifact. It is a set of package versions from the same or different package registries. For example, you can install packages from npm or PyPi in the same CodeArtifact repository.

If a repository has packages accessed from a second repository, it becomes an upstream repository. For example, if your CodeArtifact repository has package versions from npm, CodeArtifact creates a special repository to download packages from npm. This npm repository becomes an upstream repository, and your repository becomes a downstream.

Domain

Finally, domains are top-level CodeArtifact constructs that group and manage multiple repositories. You define external and internal repositories in CodeArtifact domains and access them through your domains.

Domains also allow you to use organization policies to share your repositories between your AWS accounts. For example, you can create a CodeArtifact domain for your production environment in an AWS account in your AWS organization and use it as the central domain for your software packages in other AWS accounts. So your teams can access them easily.

All package versions are stored in CodeArtifact domains. For example, if two repositories in your domain reference the same package version, only one copy of the package asset is stored in the domain. It doesn’t matter how many repositories reference it. This avoids unnecessary duplicates in your domain.

What is AWS CodeBuild?

AWS CodeBuild is a fully managed and serverless continuous integration service. With CodeBuild, you can build and test your code from your source Git repository hosted on GitHub, GitHub Enterprise, Bitbucket, GitLab, or GitLab self-managed. It also supports AWS CodeCommit, but AWS deprecated it in July 2024.

I won’t dive into the details of AWS CodeBuild in this post. If you want to learn AWS CodeBuild, you’re welcome to join my AWS CodePipeline Step by Step course. Now, let’s continue with using AWS CodeArtifact with AWS CodeBuild.

How do you use AWS CodeArtifact repositories in a CodeBuild project?

There are a few common steps for using AWS CodeArtifact:

1) You create a CodeAritfact domain and a repository.

2) You assign necessary IAM permissions to the IAM identity, which will store packages to or download packages from your CodeArtifact repository.

3) You log in to your CodeArtifact repository using AWS CLI. The documentation refers to it as CodeArtifact CLI, but you install it with AWS CLI, and its usage is similar to that of other commands. So, I will refer to it as AWS CLI.

4) You publish your package or install a package using language-native tools such as npm publish or npm install, respectively.

Step 1 - Preparing your CodeArifact domain and repository

To begin with, you create a CodeArtifact domain that will host all your CodeArtifact repositories and package versions.

In the example below, I create a CodeArtifact domain named my-demo-domain.

Then, you create a new repository in your CodeArtifact domain and choose its upstream repositories from the popular central package registries.

Your repository must be created in a CodeArtifact domain. If you don’t yet have a domain, you can create it while creating your repository on the CodeArtifact Console. However, you can create a repository in an existing domain by clicking the ‘Create repository’ button, as shown below.

In the example below, I create a repository named my-demo-repository in the my-demo-domain created before. Although our sample project only needs npm-store (JavaScript) to install the required Node packages, I also added PyPi (Python), RubyGems (Ruby), and Maven Central Repository (Java) registries as upstream repositories to my repository as an example.

These are repositories with external connections, and CodeArtifact uses them to store package versions from the Internet. Because they are defined as upstream for your repository, it can also access their packages. However, you can store your own packages in your repository directly.

On the review page, CodeArtifact shows the upstream and downstream repositories with a nice diagram.

In this diagram, the maven-central-store, rubygems-store, npm-store, and pypi-store repositories were added as upstream repositories. my-demo-repository is a downstream repository that automatically has access to the package versions in them.

After the creation, your CodeArtifact domain will have five repositories as below.

At the beginning, your CodeArtifact repository doesn’t have any packages.

Angular applications use Node packages. So, during the build, we expect CodeBuild to fetch some Node packages from npm and store them in the npm-store repository. However, the npm-store repository on CodeArtifact is empty beforehand.

So, let’s see what happens after integrating it with AWS CodeBuild running a build.

Step 2 – Creating your CodeBuild project

This part is almost the same as using AWS CodeBuild without CodeArtifact. Your CodeBuild project’s source can be any of the source repositories supported. It can also be a part of a CodePipeline build action, which you can learn in my AWS CodePipeline Step by Step course on Udemy.

In the example for this blog post, I use a buildspec file to build an Angular application. I won’t embed the CodeArtifact domain and repository names or AWS account ID in the buildspec file. Instead, I recommend using environment variables. So, in the example below, I create these environment variables in my CodeBuild project:

CODEARTIFACT_DOMAIN: The name of your CodeArtifact domain. In my example, it is ‘my-demo-domain’.
CODEARTIFACT_REPO: The name of your CodeArtifact repository, which is ‘my-demo-repository’ in my example.
CODEARTIFACT_OWNER: The AWS account ID of the owner of the repository. It will be my account ID, but it can also be from a different AWS account whose repository is configured for cross-account access.

The environment variables on CodeBuild look like the image below. You can learn to use environment variables in CodeBuild from my previous post about the subject.

Step 3 – Granting your CodeBuild project necessary permissions

Your CodeBuild project won’t have permission to access CodeArtifact after the creation. So, you should provide permission to log in to CodeArtifact and access your CodeArtifact domain and repositories in its IAM service role.

You can create a customer-managed IAM policy with the permissions below.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "codeartifact:GetAuthorizationToken",
                "codeartifact:GetRepositoryEndpoint",
                "codeartifact:ReadFromRepository"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": "sts:GetServiceBearerToken",
            "Resource": "*",
            "Condition": {
                "StringEquals": {
                    "sts:AWSServiceName": "codeartifact.amazonaws.com"
                }
            }
        }
    ]
}

Then, you should assign it to your CodeBuild project’s IAM role.

In CodeBuild, you provide the commands to build or test your code in your buildspecs. A buildspec consists of these optional phases in the order they are executed: install, pre_build, build, and post_build.

Usually, the pre_build phase is where you log in to central repositories, such as DockerHub or Amazon ECR. However, in the install phase before pre_build, you may also install global packages in your CodeBuild environment, such as Angular CLI. So, the place of your CodeArtifact login command depends on your buildspec file’s structure.

In my sample buildspec below, I install Angular CLI in the install phase and the project dependencies in the pre_build phase. Then, I build the Angular project in the build phase.

version: 0.2
phases:
  install:
    runtime-versions:
      nodejs: 20
    commands: 
      # Install the Angular CLI
      - npm install -g @angular/cli@17

  pre_build:
    commands:
      # Install the project dependencies
      - npm install

  build:
    commands:
      # Build the Angular application
      - ng build -c production
      
artifacts:
  base-directory: dist/my-angular-project
  files:
    - '**/*'

If I place the CodeArtifact in the pre_build phase, the npm install command for Angular CLI in the install phase will still fetch the package from the central npm registry. But I also want CodeBuild to fetch it from CodeArtifact. So, either I would move the Angular CLI installation to the pre_build phase or place the CodeArtifact login before it.

I chose the second option. You can see the final buildspec file below.

version: 0.2
phases:
  install:
    runtime-versions:
      nodejs: 20
    commands:
      # Log in to CodeArtifact
      - echo 'Logging in to CodeArtifact...'
      - aws codeartifact login --tool npm --repository ${CODEARTIFACT_REPO} --domain ${CODEARTIFACT_DOMAIN} --domain-owner ${CODEARTIFACT_OWNER}
      
      # Install the Angular CLI
      - npm install -g @angular/cli@17

  pre_build:
    commands:
      # Install the project dependencies
      - npm install

  build:
    commands:
      # Build the Angular application
      - ng build -c production
      
artifacts:
  base-directory: dist/my-angular-project
  files:
    - '**/*'

Because the build execution logs in to CodeArtifact before all npm install commands, all required Node packages will be stored in the CodeArtifact domain first. Then, they will be installed from CodeArtifact into the CodeBuild environment.

If you joined my AWS CodePipeline Step by Step course, you are already familiar with the ‘aws ecr get-login’ command. The ‘codeartifact login’ command works similarly. It returns an authentication token to be used by the Node package manager (npm) in the HTTP Authorization header while logging in to CodeArtifact.

The Result of the Build

After executing the build project, you will have these in your logs and CodeArtifact repositories.

The CodeBuild logs

After running the build, you will see a build log for the successful login similar to the one below. Please note the build log after the CodeArtifact login command.

The authorization token returned by the ‘codeartifact login’ command provided 12 hours of access, which was much more than needed for my CodeBuild project execution.

The CodeArtifact repositories after the build

After the build, you will see your CodeArtifact repository with new package versions.

Actually, these are from the upstream npm-store repository in your CodeArtifact domain.

CodeArtifact automatically fetches all dependencies from your package.json file in your Node or Angular project. But if your project continues to use the same package versions, CodeArtifact will not fetch those unless you delete them. If your project references a new version, CodeArtifact will fetch it in the subsequent build execution.

Would you like to learn to use AWS CodeBuild with AWS CodePipeline?

If you want to learn to use AWS CodeBuild in your CI/CD pipelines, my AWS CodePipeline Step by Step course on Udemy may be helpful.

In this course, you will learn to use AWS CodePipeline to build CI/CD pipelines on AWS step by step to build your applications with CodeBuild and deploy them to S3 or EC2. You will also learn to create Docker images with CodeBuild and deploy Docker containers from them to Amazon ECS with CodePipeline. We cover many CodeBuild features in detail in my Udemy bestseller course.

The links in this post also provides a special discount for you. You can view all my courses on our Courses page.

Conclusion

AWS CodeArtifact provides a feasible and efficient central package management solution for your software development processes. In this post, I provided an example of using AWS CodeArtifact with your CodeBuild projects.

If you found it helpful, please also share it on your LinkedIn profile or X (Twitter) account to help me reach others like you.

Follow me on LinkedIn and X (Twitter) to hear about my future online courses or blog posts.

Thanks for reading!

References

A Quick Overview of IAM Permission Policies: AWS-Managed, Customer-Managed, and Inline Policies

2024-07-25T00:00:00+00:00

In AWS IAM, permission policies play a crucial role in securely controlling access to your AWS environment. You can attach three types of permission policies to your identities: AWS-managed, customer-managed, and inline policies. In most cases, AWS recommends using managed policies over inline policies, especially customer-managed policies created from AWS-managed policies. However, there may also be cases for inline policies. In this post, we will quickly explore IAM’s permission policies to help you decide which is best for you.

By the way, we will only talk about AWS IAM’s identity-based permission policies in this post. The resource-based trust policies used in IAM roles, in which you specify the principals you trust to assume a role, will be out of the scope of this post. So, when we use the ‘policy’ term below, it will only refer to an IAM permission policy: AWS-managed, customer-managed, or inline. Then, let’s begin.

So, when you attach a policy to an identity (a user, group of users, and a role), that policy determines whether to allow or deny access to your AWS resources. Hence, with security threats in mind, choosing the right policy type for your needs is essential, as well as granting the least privilege. But you wouldn’t want to increase the management overhead while doing this, right?

Each IAM policy type has its use cases. So, let’s analyze each one to see what features it offers.

What do AWS-managed policies offer?

The AWS-managed policy is a standalone policy that exists independently from an identity. That means each AWS-managed policy has its own Amazon Resource Name (ARN). The main advantage of AWS-managed policies is that AWS undertakes the whole process of creating and managing AWS-managed policies. So, you don’t need to bother yourself with writing and maintaining them.

Also, AWS updates the policies when required to add new features. Then, all the identities to whom an AWS-managed policy is attached automatically get the updated policy permissions.

Attaching an AWS-managed policy to an identity is straightforward.

Firstly, from the IAM console, you go to the profile page of the identity (user, user group, or role).
For IAM roles and groups, click the ‘Add permissions’ section on the ‘Permissions’ tab and choose the ‘Attach policies’ option from the dropdown.
For IAM users, you click the ‘Add permissions’ menu and choose ‘Add permissions’ again. Then, you select the ‘Attach policies directly’ option.
After that, you can see all AWS-managed policies available to choose from. You can filter them using the search box. The interface may be slightly different, but you can see an example of AWS-managed policies filtered for ‘cloudwatch’ below.

You can choose the ones that fit your requirements and attach them to your IAM identity.

Currently, there are over a thousand AWS-managed policies. Please remember that AWS-managed policies can be used for more than one identity in one account or across multiple accounts.

Do AWS-managed policies work in all cases?

Besides the advantages, using ready-made AWS-managed policies may sometimes be unfavorable because you cannot change their permissions yourself. Your requirements may differ, or you may need to grant specific permissions to your identities other than those described in the AWS-managed policies.

Besides, AWS creates policies for general use cases and scenarios. Therefore, they could cover more permissions than the ones you intend for your identities. You may also need to limit your permissions for specific resources in your AWS account.

In those cases and many more not covered here, using an AWS-managed policy would be against AWS security best practices for granting the least privilege, which means giving the minimum permissions necessary to your identities. As a solution, you can create customer-managed policies by setting permissions aligning with your needs.

Tailored for your needs: Customer-managed IAM policies

Like AWS-managed policies, customer-managed IAM policies are standalone policies that can be attached to multiple identities. You can easily create and manage policies yourself.

To do this, you go to the IAM console in your AWS account.
On the left-hand side, select the ‘Policies’ section under ‘Access Management’ and then click the ‘Create policy’ button to create one.
On the Create policy page, you will see two policy editor options: one is a ‘Visual’ editor, and the other is ‘JSON.’ JSON editor is the preferred option in most cases. However, you can create a policy from scratch with the Visual editor if you don’t know the JSON syntax.
In the JSON editor, you can either type a new policy or paste an existing policy and then customize it. Usually, AWS recommends copying an existing AWS-managed policy and modifying it according to your requirements.

A typical JSON policy looks like this:

{
	"Version": "2012-10-17",
	"Statement": [
		{
			"Sid": "Statement1",
			"Effect": "Allow",
			"Action": [],
			"Resource": [],
      		"Condition": {

      		}
		}
	]
}

Your JSON policy can contain multiple statements defined under the ‘Statement’ array, which specifies a set of permissions to be allowed or denied. A statement consists of these sections:

Sid: A unique statement ID.
Effect: ‘Allow’ or ‘Deny’
Action: The actions that will be allowed or denied.
Resource: The resources that the policy will cover.
Condition: Optional section for setting conditions to filter the resources to create a more granular policy. For example, you can add a ‘StringEquals’ condition to filter the resources based on a tag value.

You can attach your customer-managed policies like you would an AWS-managed policy. When the managed policies are listed, you can use the search field to find your policy or list only customer-managed policies using the type filter.

Like AWS-managed policies, when you update permissions in a policy, the changes apply to all identities to whom the policy is attached. So, it brings reusability advantages over inline policies, as you will see below.

Having said that, customer-managed policies have many advantages over other types.

Compared to AWS-managed policies, you can make any update or change you want without waiting for AWS to do so.
Besides, unlike AWS-managed policies, you can customize your customer-managed policies to cover only specific resources.
Most importantly, customer-managed policies bring versioning and rollback features. When you update a customer-managed policy, IAM saves it as a new version by keeping up to five versions. So, if you encounter a problem after an update, you can quickly roll back to a previous version of your customer-managed policy.

What about inline IAM policies?

You also have a third policy option to use: inline policies. In most cases, inline policies are not favored much as they offer limited features compared to managed policies. Inline policies are directly embedded into a user, user group, or role, and becomes an inherent part of the identity. Therefore, when the identity is deleted, its inline policies are also destroyed.

So, there is a stricter relationship between the inline policy and the identity compared to managed policies. This usually makes using inline policies inefficient. When you want to change permissions in an inline policy, you must do this from the identity profile. If you have more than one identity containing the same inline policy, repeating the same process for each one becomes tiresome and prone to errors.

On the other hand, this direct, one-to-one relationship between the identity and the policy can be favorable in certain situations. Suppose you need to ensure that a policy’s permissions are not granted to a wrong identity in any way. In that case, you can use an inline policy to prevent inadvertently granting permissions to identities other than that you assign for.

Each IAM policy is a JSON file with a character limit, and the maximum character limit for an inline policy is smaller than that for managed policies. If you reach the maximum character limit in an inline policy, you will need to divide it into multiple inline policies or convert it to a customer-managed policy.

Let’s briefly talk about creating an inline policy as well.

To create an inline policy, first go to your user’s, user group’s, or role’s page on the IAM Console and switch to the ‘Permissions’ tab.
This time, you should select ‘Create inline policy’ from the ‘Add permissions’ menu.
Then, you will see Visual or JSON policy editors again, as in the customer-managed policies. You can add your permissions, which should apply to the relevant identity there.

3 key differences to note between IAM policy types

As discussed above, specific use cases exist for managed and inline policies. Before deciding which one to choose, we recommend you evaluate them individually according to your use case. But to make it easier for you, let’s recap the primary differences that would affect your decision.

Firstly, an inline policy can only apply to a specific identity. Meanwhile, managed policies, either AWS-managed or customer-managed, can be attached to multiple identities. So, if you consider assigning a policy to various identities, using managed policies would be the feasible option. Also, you can convert an inline policy to a managed policy later if you change your mind.
Another significant difference is about policy updates. AWS already updates policies as needed for AWS-managed policies. Also, you can easily update a customer-managed policy by adding or removing permissions yourself. After these updates, relevant changes will apply to all identities to which the policies are attached. On the other hand, if you define the same inline policy in multiple identities, you must edit the policy in each identity one by one to add or remove permissions, which would be an efficiency problem.
Customer-managed policies bring versioning and rollback capabilities to your policies. When customer-managed policies, you can quickly roll back to a previous version, which can be the sole reason to choose them over others.
Managed policies allow more characters compared to inline policies. So, at some point, you may have to divide your inline policies into multiple or convert them to customer-managed policies to provide more permissions.

Conclusion

AWS-managed policies, customer-managed policies, and inline policies are three IAM policy types suitable for different use cases. Whichever you choose, granting the least privilege access to your resources while managing them efficiently is crucial for securing your AWS environment.

In this post, we provided an overview of AWS policy types. We hope this will be helpful when assigning permissions to your IAM identities in your projects.

Thanks for reading!

References

5 Things to Note About AWS Free Tier

2024-05-13T00:00:00+00:00

When I first used AWS back in 2013, I remember missing some free-tier benefits in my first year. I was developing an app with Ruby on Rails and read some fancy blog posts about how another VPS provider performs better than AWS, and I didn’t use EC2 or RDS at the start. I had to manage my servers and MariaDB databases myself. Even though that Linux administration experience helped me become a DevOps engineer today and better understand AWS’s benefits, my life would have been easier then, even with fewer costs. So, the AWS free tier is a great offer, especially when starting to learn the cloud and AWS.

However, if you are new to AWS, understanding the exact coverage of the AWS free tier can be confusing. Besides, you may be concerned about encountering a surprise bill at the beginning of the following month. So, in this post, let’s discuss five of the most crucial things you need to know about the AWS free tier and how to avoid unexpected costs when benefiting from it.

1 - Be sure to understand what is offered under AWS free tier!

First, you should note that most AWS free-tier services have specific usage limits or expirations. AWS offers three types of free-tier offers: always free, 12 months free, and short-term trials.

Always free offers have no expiration, as their name suggests. However, they mostly have usage limits limit allocated for each month. For instance, although you see AWS Lambda with an ‘Always free’ offer on the AWS Free Tier page, at the time of this post, you can only get up to 1 million free requests and 3.2 million seconds of free compute time per month. So, if you exceed this limit, you pay for what you use for the exceeding part.
You can use some services 12 months free right after you sign up for AWS. After one year, this offer expires, and you will start paying standard fees for your usage. The ‘12 months free’ offer for the eligible AWS resources come with certain usage limits. For instance, Amazon S3 is one of the most popular AWS services that provides free usage for 12 months with a limited 50 GB of standard storage, 20,000 get requests, and 2,000 put requests per month.
Short-term trials enable you to try out an AWS product for a given time with a specific limit. For example, Amazon Redshift is a popular AWS data-warehousing service offering a free trial. You can use Amazon Redshift free for two months, valid for only 750 ‘DC2.Large’ node hours per month, and you will be charged after that. Whereas, you can use AWS Security Hub unlimited for 30 days.

Although they are not the goal of this post and their number is small, AWS offers some services ‘always free’ and unlimited. For example, in the Individual Tier, AWS CodeWhisperer provides unlimited code suggestions for your Java, Python, or other supported code. You can also use the AWS Well-Architected Tool for free to review your AWS architecture according to best practices.

Of course, the free tier examples given in this post are subject to change. To find the AWS service that best matches your needs, I recommend you navigate through the AWS free tier page. On this page, please carefully analyze which services are free and under which conditions. So that you avoid any surprises later and plan your budget accordingly.

2 - Configure your settings carefully to stay in the free tier!

When you launch an AWS resource with an AWS free tier offer, you must configure your settings carefully in your AWS account to ensure the services you select are all free tier eligible. For example, in the AWS EC2 instances free trial offer, only the t2.micro, t3.micro, and t4g.micro instance classes are free for up to 750 hours. So, if you accidentally select an instance type different than these, the free tier won’t cover your usage and you will still be charged for it.

Also, not very likely, but AWS’s default settings may sometimes be misleading. For example, while launching an EC2 instance from the EC2 Console, the t2.xlarge instance type may be selected by default. If you proceed without noticing, you must pay for it. So, please check the default settings before launching an AWS resource and ensure all selected configuration fall under the AWS free tier.

3 - Use your free limits wisely!

AWS free tier is calculated and renewed monthly across all regions, not per region. That means when you choose a free tier service, you will be granted free limits to use in all regions, not in each region. Besides, your unused limits do not roll over to the next month. So, I recommend you use your free limits as efficiently as possible.

Also, please be sure to turn off or delete the services you don’t use to avoid unexpected charges. If you forget, these services will still run in the background, and when they expire or exceed the free threshold, you will be charged. It would be frustrating to pay for something you did not use.

4 - Review your free tier usage once in a while!

Checking your AWS Billing Console regularly will help you get an overview of your AWS costs. In your AWS account, you can see your usage activity over time in the AWS Billing Dashboard. This includes active services, cost trends, monthly forecasts, etc.

To find a more detailed breakdown of your services with more granular cost and usage data, you can view AWS Cost Explorer. It provides a graph view, showing your total monthly costs, which can be grouped according to service, region, resource, etc. This allows you to take action on your costs if you see any overused service.

AWS Pricing Calculator is another free tool to check if your planned usage is under the free tier thresholds. Let’s say you want to use AWS Lambda. When you create an estimate and enter the details, such as the number of requests you need and memory requirements in the AWS Pricing Calculator, your estimated calculation will be shown to you. Therefore, you can see whether or how much you will need to pay for your AWS Lambda usage.

5 - Get notified if your usage exceeds AWS Free Tier!

Most AWS users automatically opt-in to receive emails when they exceed 85% of the free tier limits. So, if you exceed 85% of your AWS free tier limits, by default, AWS will send alerts to the email address you used to create your AWS account. However, please ensure that the option to receive free tier usage alerts is enabled from the Billing Preferences page on the AWS Management Console. If you use AWS Organizations and own a management account with consolidated billing, you must also opt-in to receive these alerts from the ‘Billing Preferences’ section in the management account.

These alerts are sent for each free tier service active in the current month. So, I recommend you not ignore or mark them as spam since they are crucial reminders of your usage.

Besides, you can set custom alerts through AWS Budgets to let you know when you exceed or are forecasted to exceed free tier usage limits. By setting a ‘zero spend budget’ template, you can track your usage up to 100%. This will help you stay within your budget and manage your free tier usage.

AWS also offers Cost Anomaly Detection as a free tool to alert you on abnormal costs in your AWS account. When creating a subscription for the AWS Cost Anomaly Detection, you must set a threshold above which an anomaly alert is sent. You can configure other details like alerting frequency or alert recipients on the same page.

Conclusion

AWS free tier can help you gain hands-on experience on AWS. However, if you are unsure what it covers, you may encounter surprising charges at the end. In this post, I tried to share with you how you can utilize AWS free tier services as ‘free’ as possible and how you can track your free AWS usage. I hope it will be helpful for you.

In our online courses, we try to use resources covered by the AWS free tier as much as possible to avoid unexpected costs. So, you can do most hands-on examples for free by just opening a new AWS account. Of course, if the topic requires larger resources beyond the limits offered, we sometimes use them for a brief time. But these cases are rare, and you are warned about their costs during the lectures.

Thanks for reading, and see you in my next post!

References

Importing AWS Resources to CDK Apps with Python

2024-04-24T00:00:00+00:00

In my AWS CDK with Python Step by Step course, I teach you to define your constructs from the ground up. You learn to specify your AWS resources through CDK constructs using Python’s object-oriented methods.

However, what if you need to use or reference an existing resource from your AWS account, such as a VPC? Can you import a resource to your AWS CDK app?

So, in this blog post, I will discuss how to import an existing resource as a CDK construct. But you cannot achieve this in environment-agnostic stacks. Hence, we will start with specifying targetted CDK environments for your CDK stacks.

How does resource importing work on AWS CDK?

Before implementing, let’s first understand how CDK imports an existing resource from your AWS account.

In a CDK app, you specify all resources as L1, L2, or L3 constructs in your CDK stacks. Then, when you use the cdk synth or cdk deploy commands, the CDK Toolkit synthesizes a CloudFormation template from your code for each CDK stack. So, the templates are usable for all the target CDK environments you deploy.

However, the process is somewhat different if a resource needs to be imported to your CDK stack.

1) You again declare the existing AWS resource you want to import as a CDK construct using a special from method provided by its L2-level CDK construct. You need to give the resource ID, ARN, or another field to search for the resource. After that, you can reference it in other constructs, like a regular L2-level construct of its type.

2) When you execute the cdk synth or cdk deploy commands, CDK uses the AWS SDK for your programming language to search for the resource in your target CDK environment and synthesizes the template with that resource’s referenced values if the search is successful.

3) CDK deploys the stack to your target CDK environment.

You cannot modify an AWS resource imported to your CDK environment as a CDK construct. You can only use it in read-only mode. You reference it as a CDK construct while configuring other constructs to benefit from CDK’s object-oriented methods.

This also won’t work in an environment-agnostic CDK stack because the CDK Toolkit must know the target CDK environment before looking for the resource. It gets them from your CDK app. Therefore, let’s continue with discussing CDK environments and how you specify the target CDK environments for your stacks.

What is a CDK environment?

A CDK environment is where you deploy your CDK stack. CDK stacks correspond to CloudFormation stacks during deployment. So, they are regional resources. Hence, a CDK environment is your deployment target consisting of an AWS account and a region.

When you initialize a CDK app, it sets up an environment-agnostic stack by default. In other words, it initalizes a CDK stack with no CDK environment setting. For environment-agnostic stacks, the CDK Toolkit deploys the synthesized CloudFormation templates to the CDK environment of your current AWS CLI profile.

By the way, a CDK environment doesn’t exist only in theory. You must physically bootstrap your CDK environment before executing any CDK deployments to prepare the environment beforehand. You do this using the cdk bootstrap command, which deploys a CloudFormation stack in the target AWS account and region for the necessary resources used by the CDK Toolkit. The bootstrap command uses a template provided by the CDK team. So, you don’t need to do anything other than execute the command to bootstrap your CDK environment.

However, bootstrapping an environment is not sufficient to import an existing resource to your CDK app. You must also set the CDK environment of your stack in your app code to inform the CDK Toolkit about the target environment for the search. Importing doesn’t work in environment-agnostic stacks.

There are a few differences between environment-agnostic stacks and stacks with specified environments, including this resource importing feature. We can discuss them in more detail later in another blog post. But for now, let’s continue with setting your stack’s CDK environment.

How do you set your CDK stack’s target environment?

When you initialize a CDK app, you will see a comment in the stack definition in your app.py file, which explains the initialization of an environment-agnostic stack by default. As also described in this comment, there are two ways to set the target environment of your CDK stack.

Method 1: Specifying AWS account ID and region code

The first option is hardcoding your AWS account ID and region, as shown below.

app.py file

...

app = cdk.App()
SampleStack(app, "SampleStack",
           env=cdk.Environment(account='444433332222', 
                                region='eu-central-1')
)

Here, 444433332222 is the hypothetical number for my AWS account ID, and eu-central-1 is the code of the AWS Frankfurt region. Please replace them with your own AWS account number and region.

Of course, a CDK environment must be bootstrapped in this AWS account for this region. You must also have permission to create CloudFormation stacks and resources specified in the target environment.

This hardcoded setting always tries to deploy your CDK stack to the same AWS account and region, regardless of your AWS CLI configuration. In this way, you lose the reusability of your CDK code for other AWS accounts and regions. However, by hardcoding its environment as a target, you ensure that the imported resource exists.

However, there is another method that enables you to deploy your app in other environments, too, without making your stack environment-agnostic. I never liked hardcoding values in my code, so let’s also discuss the second method as an alternative.

Method 2: Using the current AWS CLI profile settings

You can use the environment variables set for your AWS CLI profile while configuring the environment of your CDK stack. So, for the same stack, the example becomes like below.

app.py file

...

app = cdk.App()
SampleStack(app, "SampleStack",
           env=cdk.Environment(account=os.getenv('CDK_DEFAULT_ACCOUNT'), 
                                region=os.getenv('CDK_DEFAULT_REGION')),
)

Here, you reference the CDK_DEFAULT_ACCOUNT and CDK_DEFAULT_REGION environment variables using Python’s os library. These are implied environment variables that the CDK Toolkit sets in reference to the values of your current AWS CLI profile. Before performing the lookup method to search for the AWS resource, the CDK Toolkit will replace them with the values from your AWS CLI profile configuration. So, the search will be performed on the target environment.

OK, you also set the environment of your CDK stack. Now is the time to discuss importing an existing resource in your CDK app.

The from_xxx() methods provided by CDK constructs

L2-level CDK constructs are the first-class citizen CDK constructs that provide many object-oriented features, making up most of the CDK’s advantages over JSON or YAML CloudFormation templates. The from_xxx() methods are among those (xxx changes according to the resource type), which enable you to import existing AWS resources as L2-level CDK constructs from your target environment by searching for their specific features.

You can’t use an imported CDK construct to modify the underlying resource. However, you can use its construct object like regular objects of that resource type to configure other resources.

Now, let’s provide an example by importing an existing VPC to our CDK app to launch an EC2 instance in it.

Example: Importing a VPC to your CDK stack

Suppose you have a preconfigured VPC in your AWS account. Instead of creating a new one, you want to create your CDK app’s resources in this VPC. You may have other resources managed by other CDK apps and may not want to configure a separate VPC for each app. Whatever your reason, you can use the ec2.Vpc construct’s from_lookup() method to import your VPC as below.

CDK constructs correspond to one or more AWS resources. You define them in your CDK stacks. Below is an example of importing an existing VPC in the init method of a sample CDK stack.

sample_app.sample_stack.py

...

class SampleStack(Stack):

    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        my_vpc = ec2.Vpc.from_lookup(self, 'MyVpc', vpc_id='vpc-0f1a1122223333456')

...

Here, we provided the ID of the VPC we want to import to the vpc_id attribute. So, the CDK Toolkit will perform the lookup by searching for a VPC with this VPC ID in the target CDK environment of the stack and proceed with the synthesis afterward.

Instead of the VPC ID, you can filter the VPCs according to a tag value via the tags attribute or use the vpc_name to filter according to its name.

VPC construct also provides the from_vpc_attributes method if you need to search for a VPC by providing more attributes, such as isolated subnet IDs, availability zones, etc. But if you use an attribute in this method, the values you provide, such as isolated subnet IDs, must match the existing values in length and order. Using the from_lookup method is recommended instead.

Anyway. The returned my_vpc construct is an object of the aws_ec2.Vpc construct class. So, you can use it while configuring an EC2 instance.

sample_app.sample_stack.py

...

class SampleStack(Stack):

    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        my_vpc = ec2.Vpc.from_lookup(self, 'MyVpc', vpc_id='vpc-0f8a1264829501695')

        my_server = ec2.Instance(self, 'WebServer',
                                    vpc=my_vpc,
                                    instance_type=ec2.InstanceType.of(instance_class=ec2.InstanceClass.T3, instance_size=ec2.InstanceSize.MICRO),
                                    machine_image=ec2.MachineImage.latest_amazon_linux2023()            
                                )

        ...

Please note how we assign the my_vpc variable representing the imported VPC to the vpc_id attribute. Its usage is the same as that of a VPC construct you created from scratch in the same app.

Many L2-level CDK constructs provide the from_xxx class methods similar to importing an existing resource, where xxx changes according to the resource type. For example, you can import an existing AWS CodeCommit repository using the from_repository_arn or from_repository_name class methods of the aws_codecommit.Repository construct by providing the repository ARN or repository name, respectively. But unlike aws_ec2.Vpc, the aws_codecommit.Repository construct doesn’t provide the from_lookup method. So, it would be best if you referred to the CDK Construct Library reference for the import method of your L2-level construct class. But they often begin with from_.

Learn AWS CDK with Python!

Would you like to learn how to initialize new CDK apps and define your L1, L2 constructs, or CDK patterns (L3)? By joining my online AWS CDK course on Udemy, you can learn AWS CDK with Python as the programming language, from the most basic features to advanced concepts like CDK aspects and testing constructs.

With this course, I helped many people like you to understand AWS CDK concepts with practical hands-on examples. Besides, as a reader of this blog, you can join my AWS CDK with Python Step by Step course with a special discount using the links provided in this post.

So, enroll today and start learning AWS CDK step by step!

Conclusion

Sometimes, you need to reference an existing AWS resource in your CDK apps. In this post, I provided some insights into importing your resources as CDK constructs using the class methods provided by L2-level CDK constructs. I hope you find it helpful while using AWS CDK.

Thanks for reading!

Shikisoft Blog

Amazon S3 Storage Classes & Lifecycle Management: Optimizing Your Cloud Object Storage

S3 Standard & Express One Zone for Frequently Accessed Objects

What are Your Options for Infrequently Accessed Objects?

S3 Glacier Storage Classes: ‘Freeze’ Your Rarely Used Objects in the Glacier

S3 Glacier Instant Retrieval

S3 Glacier Flexible Retrieval

S3 Glacier Deep Archive

S3 Intelligent Tiering: What if you don’t know how often your objects are accessed?

Automating S3 Storage Management with Lifecycle Policies

My Use Case: Backup Storage for Final Cut Pro X Libraries

Conclusion

References

Which Amazon EC2 Instance Type is Right for You?

What is an EC2 instance type?

General Purpose Instance Types

M Instances (Medium)

T Instances (Threshold)

Mac Instances (Mac)

Compute-optimized Instance Types

C Instances (Compute or CPU)

Memory-optimized Instance Types

R Instances (RAM)

X Instances (Extra RAM)

U Instances (Ultra Large)

Z Instances (Z - A sign for High-frequency)

Accelerated Computing Instance Types

G Instances (GPU)

P Instances (Power)

Trn Instances (Trainium)

Inf Instances (Inferentia)

DL Instances (Deep Learning)

F Instances (FPGAs)

VT Instances (Video Transcoding)

Storage-optimized Instance Types

I Instances (IOPS)

D Instances (Dense)

H Instances (Hard Disk - HDD)

HPC Instance Types (HPC)

HPC Instances

Conclusion

References

AWS Certified Solutions Architect – Professional: Renewing My Certification Again in 2024

How did I prepare for the AWS Certified Solutions Architect – Professional exam?

About the AWS Certified Solutions Architect – Professional exam content

Organizations

Migration

Security & Compliance

Serverless

Networking

Databases

Storage

DevOps Tools

Data Analytics

Application Integration & Messaging

Cost Optimization

End-user Computing

Machine Learning & IoT

My Exam Experience with Pearson Vue

Conclusion

References

IaaS vs. PaaS vs. SaaS: Cloud Computing Service Types Explained

Infrastructure as a Service (IaaS)

What about examples?

Platform as a Service (PaaS)

What about some examples?

Software as a Service (SaaS)

What about an example?

What about AWS services?

Conclusion

Amazon SNS vs. Amazon SQS: A simple comparison

Amazon SNS is push-based!

What is an SNS topic?

There are two types of SNS topics: Standard and FIFO.

Amazon SQS uses a pull mechanism!

Variants of SQS Queue Types

Which solution would be best for your need?

Combination of SNS and SQS: The Fanout Pattern

Conclusion

References