Amazon S3 Glacier
glacierS3 Glacier storage classes for low-cost long-term archival storage.
Show deep diveHide deep dive
- Three Glacier-tier S3 storage classes: S3 Glacier Instant Retrieval (millisecond retrieval, min storage 90 days, min object 128 KB, cheaper than Standard-IA if accessed quarterly), S3 Glacier Flexible Retrieval (formerly S3 Glacier
- retrieval options Expedited 1-5 min, Standard 3-5 hr, Bulk 5-12 hr
- min 90 days
- min object 40 KB), S3 Glacier Deep Archive (cheapest
- Standard 12 hr, Bulk 48 hr
- min 180 days
- min object 40 KB)
- all offer 11 9's durability across 3+ AZs (Glacier Flexible Retrieval also offers single-AZ variant)
- legacy Vault API (Glacier-only) still exists but S3 API access preferred
Amazon S3 Object Lock
s3-object-lockS3 feature that prevents object deletion or overwrite for a fixed period or indefinitely (WORM).
Show deep diveHide deep dive
- Two retention modes: Governance mode (users with s3:BypassGovernanceRetention can override) and Compliance mode (no one, including root user, can override or shorten retention)
- Legal Hold places indefinite hold independent of retention period
- must be enabled at bucket creation (cannot be added later via console, requires AWS Support for existing buckets)
- requires versioning enabled
- commonly used for regulatory compliance (SEC 17a-4, FINRA, CFTC)
- does not affect object permissions, only deletion/overwrite
Amazon SageMaker
sagemakerFully managed platform for building, training, and deploying machine learning models.
Show deep diveHide deep dive
- Components: Studio (web IDE), Notebook Instances (legacy Jupyter), Training jobs, Endpoints, Batch Transform, Pipelines (CI/CD for ML), Model Registry, Feature Store, Model Monitor, Clarify, Debugger, Ground Truth, Data Wrangler, Autopilot, Canvas, JumpStart, Neo, HyperParameter Tuning
- training data formats: CSV, Protobuf RecordIO, JSON, Libsvm, JPEG, PNG
- input modes: File (download to volume first) and Pipe (stream from S3, requires RecordIO, faster)
- inference options: real-time endpoints (persistent HTTPS), serverless inference, asynchronous inference (large payloads, long-running), Batch Transform (offline scoring)
- Managed Spot Training up to 90% savings
- built-in algorithms include XGBoost, Linear Learner, K-Means, DeepAR, BlazingText, Image Classification, Object Detection, Random Cut Forest
SageMaker Batch Transform
sagemaker-batch-transformSageMaker feature for offline inference on entire datasets without a persistent endpoint.
Show deep diveHide deep dive
- Designed for batch scoring of large datasets in S3
- input/output formats: CSV, JSON, RecordIO
- supports input/output filtering and joining of input with predictions
- auto-scales compute for the duration of the job then terminates (no idle costs)
- suitable for use cases not requiring sub-second latency (nightly scoring, ETL pipelines)
- billed per instance-hour for transformation duration
- differs from real-time endpoints which run continuously
SageMaker Built-in Algorithms
sagemaker-built-in-algorithmsPre-built ML algorithms packaged in SageMaker containers ready for training and deployment.
Show deep diveHide deep dive
- Includes: XGBoost (gradient boosted trees), Linear Learner (regression/binary/multiclass), K-Means (clustering), K-Nearest Neighbors, Factorization Machines, PCA (dimensionality reduction), Random Cut Forest (anomaly detection), DeepAR Forecasting (time series), BlazingText (Word2Vec, text classification), Image Classification, Object Detection, Semantic Segmentation, Object2Vec, Sequence-to-Sequence, IP Insights, LDA and NTM (topic modeling), Neural Topic Model
- most expect RecordIO-Protobuf or CSV input
- supported on training instances and managed spot training
- documentation specifies hyperparameters and instance types per algorithm
SageMaker Canvas
sagemaker-canvasNo-code visual interface for building ML models from tabular, image, and text data.
Show deep diveHide deep dive
- Targets business analysts without ML expertise
- supports classification, regression, time-series forecasting, image and text classification
- uses SageMaker Autopilot under the hood for AutoML
- data sources: S3, Redshift, Snowflake, RDS, local files, Salesforce, Databricks, plus Athena
- users can share trained models to SageMaker Studio for further refinement
- generative AI integration with Bedrock for text generation/summarization
- billed per workspace session and training hours
SageMaker Clarify
sagemaker-clarifySageMaker feature that detects bias in data/models and explains predictions.
Show deep diveHide deep dive
- Pre-training bias metrics: Class Imbalance (CI), Difference in Proportions of Labels (DPL), Kullback-Leibler Divergence (KL), Jensen-Shannon Divergence (JS)
- post-training bias metrics evaluate model predictions
- explainability via SHAP (Shapley Additive exPlanations) values for global and per-instance feature attribution
- integrates with Model Monitor for ongoing bias and explainability monitoring
- supports tabular, NLP, and computer vision data
- outputs reports to S3
SageMaker Data Wrangler
sagemaker-data-wranglerVisual data preparation tool in SageMaker Studio for ML workflows.
Show deep diveHide deep dive
- 300+ built-in transformations (handle missing values, encode categoricals, normalize, parse dates, custom PySpark/Python/SQL)
- imports from S3, Athena, Redshift, Snowflake, Databricks, EMR, Feature Store
- data quality and insights report generates bias/correlation/anomaly analysis
- exports prepared data as SageMaker Pipeline, Python script, Feature Store ingestion job, or training job
- runs on managed ml.m5.4xlarge by default
- billed per hour of instance use
SageMaker Debugger
sagemaker-debuggerSageMaker feature that monitors training jobs in real-time to detect problems and optimize models.
Show deep diveHide deep dive
- Captures tensors (weights, gradients, biases, activations) during training and saves to S3
- built-in rules detect common problems like vanishing/exploding gradients, overfitting, underfitting, dead ReLU, loss not decreasing, low GPU utilization
- custom rules supported
- profiling captures CPU, GPU, memory, I/O metrics to identify training bottlenecks
- supports TensorFlow, PyTorch, MXNet, XGBoost
- results visualized in SageMaker Studio
- helps debug training failures and reduce training time/cost
SageMaker DeepAR
sagemaker-deeparSageMaker built-in algorithm for forecasting one-dimensional time series using recurrent neural networks.
Show deep diveHide deep dive
- Supervised algorithm trained on a collection of related time series
- produces probabilistic forecasts (point and quantile predictions)
- supports related (covariate) and categorical features
- common uses: demand forecasting, financial planning, capacity planning
- input format: JSON Lines (start, target, optional cat and dynamic_feat)
- hyperparameters include context_length, prediction_length, num_layers, num_cells, learning_rate, epochs
- supports CPU and GPU instances
SageMaker Elastic Inference
sagemaker-elastic-inferenceService that attaches low-cost GPU-powered acceleration to EC2/SageMaker/ECS instances for ML inference.
Show deep diveHide deep dive
- Reduces inference cost by up to 75% by attaching fractional GPU only when needed instead of a full GPU instance
- supports TensorFlow, PyTorch, MXNet, ONNX (via MXNet/PyTorch)
- accelerator types: eia2.medium/large/xlarge with different memory/throughput
- AWS announced end of new customer access in 2023, customers should migrate to Inferentia-powered Inf1/Inf2 instances
SageMaker Endpoints
sagemaker-endpointsSageMaker real-time inference endpoints; persistent HTTPS endpoints for low-latency predictions.
Show deep diveHide deep dive
- Endpoint types: real-time (always-on, billed per instance-hour, sub-second latency), serverless inference (auto-scales to zero, pay per request, cold starts possible), asynchronous inference (large payloads up to 1 GB, long inference time up to 1 hour, requests queued)
- multi-model endpoints host thousands of models on a single endpoint, models loaded on demand from S3
- multi-container endpoints run multiple containers behind one endpoint
- auto-scaling supported via Application Auto Scaling on invocations per instance or custom metrics
- production variants enable A/B testing and canary deployments
- shadow testing routes copies of traffic without affecting users
SageMaker Feature Store
sagemaker-feature-storeCentralized repository for storing, sharing, and managing ML features across training and inference.
Show deep diveHide deep dive
- Two stores: Online (low-latency real-time inference, single-digit ms reads from a key-value store) and Offline (historical training data stored in S3 as Parquet, queryable via Athena)
- ingestion via streaming or batch (SageMaker Processing/Glue)
- supports feature groups with record identifier and event time
- integrates with Data Wrangler, Pipelines, Studio
- enables time-travel queries for point-in-time-correct training data
- avoids training/serving skew
- pricing per online write/read and offline storage
SageMaker Ground Truth
sagemaker-ground-truthData labeling service that produces high-quality training datasets using human annotators.
Show deep diveHide deep dive
- Workforce options: Amazon Mechanical Turk (public), private workforce (your employees), vendor workforce (AWS Marketplace vendors)
- supports image classification, object detection, semantic segmentation, text classification, named entity recognition, video object tracking, 3D point cloud
- uses active learning to auto-label easy examples reducing human effort and cost
- Ground Truth Plus offers fully managed labeling service (AWS provides workforce and project management)
- outputs to S3 in augmented manifest format directly consumable by SageMaker training
SageMaker Hyperparameter Tuning
sagemaker-hyperparameter-tuningSageMaker Automatic Model Tuning; finds the best model version by optimizing hyperparameters.
Show deep diveHide deep dive
- Search strategies: Bayesian (default, learns from prior trials), Random (independent samples), Grid (exhaustive), Hyperband (early-stopping for deep learning)
- supports continuous, integer, and categorical hyperparameters
- objective metric must be emitted by the training algorithm to CloudWatch Logs
- warm start initializes a new tuning job from prior jobs to reduce cost
- parallel and sequential trial execution
- integrates with built-in algorithms and custom training scripts
- billed per training job hour
SageMaker JumpStart
sagemaker-jumpstartSageMaker hub of pre-trained foundation models and solution templates for one-click deployment.
Show deep diveHide deep dive
- Includes hundreds of pre-trained models for text, vision, tabular tasks from Hugging Face, PyTorch Hub, TensorFlow Hub, plus proprietary models
- foundation models include Llama, Falcon, Stable Diffusion, FLAN-T5, BloomZ for direct deployment or fine-tuning
- solution templates provide end-to-end ML workflows (fraud detection, demand forecasting, churn prediction)
- fine-tune models with your data via Studio UI
- deploy to real-time or async endpoints
- supports private model hubs for organization-curated models
SageMaker Model Cards
sagemaker-model-cardsSageMaker feature for documenting ML model information, intended use, and performance.
Show deep diveHide deep dive
- Centralizes model documentation: intended use, model owner, training datasets, evaluation metrics, ethical considerations, caveats
- integrates with Model Registry to attach cards to registered model versions
- supports approval workflows and lineage tracking
- helps satisfy governance and regulatory requirements (e.g., model risk management)
- cards stored in JSON and viewable in Studio
SageMaker Model Monitor
sagemaker-model-monitorSageMaker feature that monitors deployed models for data and prediction quality degradation.
Show deep diveHide deep dive
- Four monitor types: Data Quality (statistical drift in input features vs. baseline), Model Quality (prediction accuracy vs. ground truth), Bias Drift (using Clarify metrics), Feature Attribution Drift (SHAP-based)
- baselines computed from training data
- monitoring schedules run on captured inference data (Data Capture must be enabled on endpoint)
- alerts via CloudWatch metrics and EventBridge
- supports built-in statistical analysis and custom containers
- output to S3
SageMaker Model Registry
sagemaker-model-registrySageMaker centralized catalog for versioning, approving, and deploying ML models.
Show deep diveHide deep dive
- Model package groups contain versioned model packages
- approval states: Approved, Rejected, PendingManualApproval
- supports cross-account sharing
- integrates with Pipelines for automated registration
- CloudFormation and IaC support
- tracks model lineage including training data, training job, hyperparameters, metrics
- can trigger downstream deployment via EventBridge on approval
- supports custom metadata
SageMaker Neo
sagemaker-neoSageMaker feature that optimizes ML models for deployment on edge devices and cloud instances.
Show deep diveHide deep dive
- Compiles models trained with TensorFlow, PyTorch, MXNet, ONNX, XGBoost, scikit-learn for target hardware: Intel/AMD/ARM CPUs, NVIDIA GPUs, Inferentia, edge devices (Raspberry Pi, Jetson, smartphones)
- typically 2x performance improvement with no accuracy loss by applying hardware-specific optimizations
- compiled models can be deployed via SageMaker endpoints or downloaded for edge deployment
- integrates with IoT Greengrass for edge model deployment via SageMaker Edge Manager (deprecated in 2024)
SageMaker Pipelines
sagemaker-pipelinesSageMaker native CI/CD service for orchestrating ML workflows from data prep to deployment.
Show deep diveHide deep dive
- Define pipelines as Python DAGs using SageMaker Python SDK
- step types: ProcessingStep, TrainingStep, TransformStep, TuningStep, ModelStep, ConditionStep, CallbackStep, LambdaStep, EMRStep
- supports parameterization for reuse
- integrates with Model Registry, Feature Store, Clarify, Model Monitor
- lineage tracking shows data and model provenance
- supports caching to skip redundant steps
- execution via Studio UI, SDK, EventBridge schedules
- integrates with CodeCommit/CodePipeline for full CI/CD
SageMaker Processing
sagemaker-processingSageMaker feature for running data preprocessing, postprocessing, and model evaluation jobs.
Show deep diveHide deep dive
- Runs preprocessing/postprocessing scripts on managed infrastructure with full Spark or scikit-learn containers, or custom containers
- built-in processors: SKLearnProcessor, PySparkProcessor, XGBoostProcessor
- auto-scales instances for the job duration
- input/output via S3
- commonly used for ETL, feature engineering, model evaluation, bias and explainability analysis
- integrates with Pipelines as ProcessingStep
- billed per instance-second
SageMaker Training
sagemaker-trainingSageMaker managed training service for running ML training jobs at any scale.
Show deep diveHide deep dive
- Supports built-in algorithms, deep learning frameworks (TensorFlow, PyTorch, MXNet, Hugging Face), and custom containers
- instance types: ml.m/c/r (CPU), ml.p/g (GPU), ml.trn1 (Trainium), ml.inf (Inferentia)
- distributed training via SageMaker Distributed Data Parallel (SMDDP) and Model Parallel (SMP) libraries, or framework-native (Horovod, PyTorch DDP)
- managed Spot Training saves up to 90% with automatic checkpointing
- supports warm pools to reduce instance startup time
- output models saved to S3
- integrates with HPO, Experiments, Debugger
- supports incremental training and transfer learning
Savings Plans
savings-plansFlexible pricing model offering compute discounts for 1- or 3-year commitments to consistent usage.
Show deep diveHide deep dive
- Three types: Compute Savings Plans (most flexible, applies to EC2 across any region/family/size/OS/tenancy plus Fargate and Lambda, up to 66% off), EC2 Instance Savings Plans (commit to a specific instance family in a region, up to 72% off, regardless of AZ/size/OS/tenancy), SageMaker Savings Plans (up to 64% off SageMaker instance usage)
- commitment in $/hour
- payment options: All Upfront, Partial Upfront, No Upfront (lowest discount)
- compared to RIs: SPs apply automatically to matching usage, don't require reservation of specific configurations
- can't be sold or modified
- Cost Explorer recommends Savings Plans
AWS Secrets Manager
secrets-managerService for managing and rotating secrets like database credentials and API keys.
Show deep diveHide deep dive
- Secrets encrypted with KMS
- native automatic rotation for RDS, Redshift, DocumentDB via AWS-provided Lambda templates
- custom rotation via your own Lambda function for other secrets (OAuth tokens, third-party APIs)
- rotation schedule configurable
- cross-region replication for DR
- resource-based policies control cross-account access
- retrieve via GetSecretValue API or client-side caching libraries (Java/Python/.NET)
- $0.40 per secret per month + $0.05 per 10,000 API calls
- differs from Parameter Store (which is cheaper but lacks built-in rotation)
Security Groups
security-groupsStateful virtual firewall at the instance/ENI level controlling inbound and outbound traffic.
Show deep diveHide deep dive
- Attached to ENIs (not instances directly)
- stateful (return traffic automatically allowed)
- support only allow rules, no deny
- default SG allows all outbound and inbound from same SG
- newly created SG allows all outbound and no inbound
- can reference other SGs as source/destination by ID for dynamic membership
- default limit: 60 inbound + 60 outbound rules per SG, 5 SGs per ENI
- bound to a single VPC
- evaluates all rules to determine if traffic is allowed (unlike NACLs which evaluate in order)
- when allowing communication between two instances, use private IP addresses, not public/Elastic IPs
AWS Security Hub
security-hubService that aggregates security findings from multiple AWS services into a single dashboard.
Show deep diveHide deep dive
- Aggregates from GuardDuty, Inspector, Macie, IAM Access Analyzer, Firewall Manager, Systems Manager Patch Manager, plus 60+ partner integrations
- runs automated compliance checks against standards: AWS Foundational Security Best Practices (FSBP), CIS AWS Foundations Benchmark, PCI DSS, NIST 800-53
- findings in standard AWS Security Finding Format (ASFF)
- cross-region aggregation
- delegated administrator in Organizations
- integrates with EventBridge for automation
- pricing per finding ingested + check evaluated
AWS Security Token Service (STS)
stsSecurity Token Service that issues temporary, limited-privilege credentials for AWS principals.
Show deep diveHide deep dive
- Global service with endpoints in every region (use regional endpoints to reduce latency and for FIPS compliance)
- key APIs: AssumeRole (cross-account or escalation), AssumeRoleWithSAML (federated SAML), AssumeRoleWithWebIdentity (Cognito, Facebook, Google, Amazon, custom OIDC), GetSessionToken (MFA-enforced sessions), GetFederationToken (federated user, IAM user calling), DecodeAuthorizationMessage (error details)
- session duration default 1 hour, up to 12 hours for roles (configurable per role)
- credentials consist of AccessKeyId, SecretAccessKey, SessionToken
- supports external ID for cross-account third-party access
- session policies further restrict permissions at AssumeRole time
AWS Serverless Application Model (SAM)
samAWS Serverless Application Model; framework for building serverless applications using simplified syntax.
Show deep diveHide deep dive
- Extension of CloudFormation with transform AWS::Serverless-2016-10-31
- resource types: AWS::Serverless::Function (Lambda + role + event sources), AWS::Serverless::Api (API Gateway), AWS::Serverless::HttpApi, AWS::Serverless::StateMachine (Step Functions), AWS::Serverless::Application, AWS::Serverless::LayerVersion, AWS::Serverless::SimpleTable
- SAM CLI commands: sam init (scaffold), sam build (package code+deps), sam local invoke/start-api (local testing), sam package, sam deploy (one-click deploy with guided prompts), sam sync (faster iterations)
- under the hood transforms to native CloudFormation
- can mix SAM and CFN resources in same template
AWS Service Catalog
service-catalogService for managing catalogs of approved IT services for users to self-provision.
Show deep diveHide deep dive
- Products are CloudFormation templates (or Terraform via Service Catalog Engine)
- portfolios group products and are shared with users/groups/roles
- constraints control how products are deployed (launch, notification, template, stack set, tag update)
- supports shared portfolios across accounts/OUs via AWS Organizations
- integrates with Service Catalog AppRegistry for application metadata
- AWS Marketplace integration
- commonly used for self-service IT in enterprises
- free service, pay for provisioned resources
Service Control Policies (SCP)
scpService Control Policies; Organizations policies that set maximum permissions for accounts and OUs.
Show deep diveHide deep dive
- Attached to root, OUs, or individual accounts
- do NOT grant permissions, only set the maximum boundary
- for an action to succeed, both the IAM policy AND the SCP (and any permissions boundary) must allow it
- do not affect the management (master) account
- default FullAWSAccess SCP allows everything
- common SCP patterns: deny specific regions, deny disabling logging (CloudTrail/Config), deny root user actions, restrict instance types, require encryption
- require All Features mode in Organizations (cannot use with Consolidated Billing only)
Service Quotas
service-quotasCentral service for viewing and managing AWS service quotas (formerly limits).
Show deep diveHide deep dive
- Successor to the AWS Trusted Advisor service limits feature
- supports both adjustable and non-adjustable quotas
- request quota increases via console or API (RequestServiceQuotaIncrease)
- some quotas auto-approve based on account history
- cross-region template (Service Quotas requests applied uniformly via Quota Request Template + Organizations)
- CloudWatch alarms on quota utilization (e.g., 80% threshold)
- applies to most services including EC2, VPC, Lambda, S3, DynamoDB, ELB
AWS Shield
shieldDDoS protection service for AWS resources.
Show deep diveHide deep dive
- Two tiers: Shield Standard (free, automatic for all AWS customers, protects against common network/transport layer DDoS, integrated with CloudFront/Route 53/Global Accelerator) and Shield Advanced ($3,000/month, 1-year commitment)
- Shield Advanced adds: protection for ELB/EC2/EIP/Route 53/CloudFront/Global Accelerator, 24/7 Shield Response Team (SRT) access, cost protection for scaling during attacks, AWS WAF included at no extra cost, attack diagnostics and reports, integrates with Firewall Manager for centralized policies
Amazon Simple Notification Service (SNS)
snsFully managed publish/subscribe messaging service for pub-sub and notifications.
Show deep diveHide deep dive
- Topic types: Standard (unlimited throughput, best-effort ordering, at-least-once delivery) and FIFO (300 messages/sec or 10 MB/sec, exact ordering, exactly-once via dedup, can only deliver to SQS FIFO)
- subscribers: SQS, Lambda, Firehose, HTTP/S, Email, SMS, mobile push (APNS, FCM, ADM, Baidu)
- message filtering via subscription filter policies (JSON, max 5 attributes, conditionals: exact, anything-but, prefix, numeric range, exists, AND/OR)
- message size up to 256 KB (use Extended Client Library for larger via S3)
- cross-region/cross-account topic subscriptions supported
- SSE with KMS
- DLQ via SQS for failed deliveries
- message retries with delivery policy
Amazon Simple Queue Service (SQS)
sqsFully managed message queue service that decouples microservices and serverless apps.
Show deep diveHide deep dive
- Queue types: Standard (nearly unlimited throughput, at-least-once delivery, best-effort ordering) and FIFO (up to 3000 msg/sec with batching/300 without, exactly-once, strict ordering)
- message size up to 256 KB (use Extended Client Library to store larger in S3)
- message retention 1 minute to 14 days (default 4 days)
- visibility timeout 0 sec to 12 hours (default 30 sec) hides messages from other consumers during processing
- long polling (WaitTimeSeconds 1-20s) reduces empty responses and cost vs short polling (default 0)
- delay queues hide new messages 0-15 min
- DLQ moves failed messages after maxReceiveCount
- SSE via KMS
- FIFO uses message group ID for ordering within group and parallel processing across groups
Amazon Simple Storage Service (S3)
s3Object storage service offering industry-leading scalability, durability, and availability.
Show deep diveHide deep dive
- Buckets are global namespace, regional in residency
- 11 9's durability across 3+ AZs (One Zone classes: single AZ)
- object size 0 bytes to 5 TB (single PUT max 5 GB, multipart upload required above)
- multipart upload recommended for objects > 100 MB
- storage classes: Standard, Standard-IA (30-day min, 128 KB min), One Zone-IA (single AZ, 30-day min), Intelligent-Tiering (auto-moves between tiers), Glacier Instant Retrieval (90-day, 128 KB), Glacier Flexible Retrieval (90-day, 40 KB, 1-5 min/3-5 hr/5-12 hr), Glacier Deep Archive (180-day, 12 hr/48 hr), Express One Zone (single AZ, ultra-low latency)
- encryption: SSE-S3 (default, AES-256), SSE-KMS, SSE-C (customer-provided), DSSE-KMS (dual-layer), client-side
- versioning suspendable but not removable once enabled
- MFA Delete adds extra auth for delete
- replication: CRR and SRR (replicates only objects created after rule)
- strong read-after-write consistency for all operations
- access via bucket policies, ACLs (legacy), IAM, access points
- presigned URLs for time-limited access
AWS Single Sign-On (SSO)
ssoAWS Single Sign-On (renamed to IAM Identity Center); workforce identity service for SSO.
Show deep diveHide deep dive
- Same as IAM Identity Center (rebranded in 2022)
- identity sources: built-in directory, Active Directory (AWS Directory Service or AD Connector), external IdP via SAML 2.0
- permission sets defined as IAM policies and assigned to users/groups for specific AWS accounts
- auto-provisions short-lived IAM roles in target accounts
- integrates with AWS Organizations
- supports SCIM for automated user provisioning from IdPs like Okta, Azure AD
- SSO access to AWS console, CLI v2 (aws sso login), and SAML/OIDC business apps
AWS Snowball
snowballPetabyte-scale data transport device for offline data migration to AWS (legacy series).
Show deep diveHide deep dive
- Original Snowball (50 TB or 80 TB usable, now mostly succeeded by Snowball Edge)
- shipped to your location, you load data, return to AWS for ingestion to S3
- encrypts data with KMS, includes E-Ink shipping label and tamper-evident enclosure
- for ongoing or multi-petabyte migrations consider Snowmobile (deprecated) or Snowcone
- current Snow Family is primarily Snowball Edge (Storage Optimized, Compute Optimized) and Snowcone
AWS Snowball Edge
snowball-edgeSnowball device with onboard compute and storage for edge processing or data transfer.
Show deep diveHide deep dive
- Two configurations: Storage Optimized (210 TB NVMe or 80 TB HDD usable) and Compute Optimized (42 TB or 28 TB with GPU)
- runs EC2 instances and Lambda functions locally
- supports S3-compatible object storage, NFS, and block storage
- clusterable (3-16 devices for HA)
- ideal for disconnected environments (ships, factories, military) and data migration
- encrypted at rest
- managed via AWS OpsHub GUI or CLI
AWS Snowcone
snowconeSmallest member of the Snow Family for portable edge computing and data transfer.
Show deep diveHide deep dive
- Two models: HDD Snowcone (8 TB HDD) and SSD Snowcone (14 TB NVMe)
- weighs ~2.1 kg, ruggedized, dust/water-resistant
- runs EC2 compute and supports DataSync agent for transfers
- can be battery-powered or used in vehicles, drones, IoT scenarios
- suitable for small-scale data collection in disconnected/limited-connectivity environments
- ships device with data via DataSync or by physical return to AWS
Stable Diffusion
stable-diffusionStability AI's Stable Diffusion text-to-image foundation models available via Amazon Bedrock.
Show deep diveHide deep dive
- Generates images from text prompts
- available models on Bedrock include Stable Diffusion XL (SDXL) and Stable Image variants
- supports image-to-image (img2img), inpainting, outpainting via API parameters
- output formats PNG/JPEG
- common parameters: prompt, negative_prompt, cfg_scale, seed, steps, style_preset
- billed per generated image
- usable through Bedrock InvokeModel API, integrates with Bedrock Agents and Guardrails
AWS Step Functions
step-functionsServerless orchestration service that coordinates workflows of AWS services using state machines.
Show deep diveHide deep dive
- Two workflow types: Standard (long-running up to 1 year, exactly-once, at-least-once on retry, billed per state transition) and Express (high-throughput up to 100K/sec, at-most-once or at-least-once, max 5 minutes, billed per invocation + duration)
- state types: Task (do work via Lambda, ECS, SNS, SQS, DynamoDB, etc.), Choice (branching), Parallel, Map (iterate), Wait, Pass, Succeed, Fail
- defined in Amazon States Language (ASL, JSON-based) or via Workflow Studio drag-and-drop
- input/output processing filters: InputPath, Parameters, ResultSelector, ResultPath, OutputPath
- supports error handling with Retry and Catch
- nested workflows via Map and Parallel
- integrates with 200+ AWS services via SDK integration
AWS Storage Gateway
storage-gatewayHybrid cloud storage service connecting on-premises environments to AWS storage.
Show deep diveHide deep dive
- Four gateway types: S3 File Gateway (NFS/SMB to S3 with local caching), FSx File Gateway (SMB cache for FSx for Windows), Volume Gateway (iSCSI block storage, cached mode keeps hot data local with S3 backing, or stored mode keeps all data local with async S3 backup), Tape Gateway (iSCSI VTL to S3/Glacier/Deep Archive)
- deployed as VM (VMware, Hyper-V, KVM, EC2) or hardware appliance
- supports Active Directory integration for SMB
- encryption in-transit with TLS
- data stored encrypted in S3 with SSE
- differs from DataSync which is for one-time/scheduled bulk transfer
AWS Systems Manager
systems-managerUnified management service providing operational data and automation for AWS and on-premises resources.
Show deep diveHide deep dive
- Capabilities organized into categories: Operations Management (Explorer, OpsCenter, Incident Manager), Application Management (Application Manager, AppConfig, Parameter Store), Change Management (Change Manager, Automation, Maintenance Windows, Change Calendar), Node Management (Fleet Manager, Compliance, Inventory, Session Manager, Run Command, State Manager, Patch Manager, Distributor), Shared Resources (Documents)
- SSM Agent required on managed nodes (pre-installed on most AMIs)
- Session Manager provides shell access without SSH/bastion
- Patch Manager automates OS patching with baselines and maintenance windows
- State Manager enforces desired configuration
- Run Command executes commands across fleet
- integrates with CloudWatch, Config, IAM, KMS
Systems Manager Parameter Store
parameter-storeSystems Manager feature for hierarchical storage of configuration data and secrets.
Show deep diveHide deep dive
- Parameter types: String, StringList, SecureString (encrypted with KMS)
- tiers: Standard (10,000 parameters per region, up to 4 KB per parameter, free) and Advanced ($0.05/parameter/month, up to 8 KB, supports parameter policies and higher throughput)
- parameter policies: Expiration, ExpirationNotification, NoChangeNotification (Advanced only)
- hierarchical organization with / paths
- versioning automatic
- integrates with CloudFormation, Lambda, ECS, CodeBuild via SSM references
- cheaper alternative to Secrets Manager for non-rotating values like AMI IDs and config
- no built-in secret rotation
