AWS Big Data

1. Introduction
2. AWS Data Lake
3. AWS Data Pipeline (aka Big Data Pipelines or Data Streams)

Introduction

• aws.amazon.com/big-data
• blogs.aws.amazon.com/bigdata
• Querying Amazon Kinesis Streams Directly with SQL and Spark Streaming
• Using Spark SQL for ETL
• whizlabs.com: AWS Kinesis vs Kafka Apache

AWS Data Lake

• Building a Data Lake on AWS AWS provides a highly scalable, flexible, secure, and cost-effective solution for your organization to build a Data Lake – a data repository for both structured and unstructured data that is designed to be easily accessible for on-demand data analytics enabling you to answer questions as they arise.
• (2026) Amazon Aurora PostgreSQL now supports direct querying of Apache Iceberg and Parquet data in your data lake 🌟 - Amazon Aurora PostgreSQL extends its query engine to directly query Apache Iceberg and Parquet formats natively stored in S3 data lakes.

AWS Data Pipeline (aka Big Data Pipelines or Data Streams)

• AWS Data Pipeline
• AWS Data Pipeline Documentation
• medium: No-Code Data Collect API on AWS A No-Code Data Collections mechanism for Big Data Pipelines on AWS.
• AWS Big Data Blog: Category - AWS Data Pipeline
• (2026) AWS Glue 6.0 now available with 30% lower price and full Apache Iceberg v3 support 🌟 - AWS Glue 6.0 launches with a significant 30% cost reduction and native, deep integration with Apache Iceberg v3 for data lakes.

Amazon DynamoDB

• (2026) Amazon DynamoDB now supports real-time vector search at any scale 🌟 - Amazon DynamoDB introduces native real-time vector search capabilities for hyperscale AI applications.