Event-driven Cloud-native Data Lake for enterprise use
Author: Javad Zeynal | July 3, 2025
Abstract
In modern enterprise environments, data is growing rapidly — in both volume and complexity. Organizations increasingly seek ways to centralize their data, automate processes, and ensure that both storage and access comply with strict security and governance requirements.
In modern enterprise environments, data is growing rapidly — in both volume and complexity. Organizations increasingly seek ways to centralize their data, automate processes, and ensure that both storage and access comply with strict security and governance requirements.
Project experience
For one of my recent projects, I had the opportunity to design and implement a cloud-native, serverless data lake architecture on AWS. The goal was to provide a scalable and flexible platform where structured and unstructured data could be securely ingested, managed, and made available for downstream processing — while ensuring full automation and compliance.
At the core of the system, I implemented a RESTful API layer using Amazon API Gateway. This served as the main entry point for interacting with the data lake, enabling consistent and validated operations. Incoming API requests were processed by AWS Lambda functions, which handled data extraction, validation, and storage of metadata in Amazon DynamoDB.
The database layer was carefully designed following AWS best practices to ensure both scalability and performance. Using DynamoDB Streams, changes to the database were propagated in real time to a second layer of Lambda functions. These functions orchestrated cloud infrastructure updates based on the current state of the system — ensuring that infrastructure and data evolved in sync.
One of the key architectural decisions in this project was to separate the storage layer from the access management layer. While data was stored in Amazon S3, access was managed through S3 Access Points, providing fine-grained control. Access Points were created in the storage account, while IAM roles and users were provisioned in a separate access management account. This design provided strong governance and supported compliance with the client's cybersecurity policies.
To support a robust and maintainable platform, additional capabilities were implemented, including S3 lifecycle rules, event notifications, and...
Want to learn more?
If you are considering building a modern data platform or want to modernize your existing data architecture, I’d be happy to discuss how this kind of approach could help your organization. Let's talk!
For one of my recent projects, I had the opportunity to design and implement a cloud-native, serverless data lake architecture on AWS. The goal was to provide a scalable and flexible platform where structured and unstructured data could be securely ingested, managed, and made available for downstream processing — while ensuring full automation and compliance.
At the core of the system, I implemented a RESTful API layer using Amazon API Gateway. This served as the main entry point for interacting with the data lake, enabling consistent and validated operations. Incoming API requests were processed by AWS Lambda functions, which handled data extraction, validation, and storage of metadata in Amazon DynamoDB.
The database layer was carefully designed following AWS best practices to ensure both scalability and performance. Using DynamoDB Streams, changes to the database were propagated in real time to a second layer of Lambda functions. These functions orchestrated cloud infrastructure updates based on the current state of the system — ensuring that infrastructure and data evolved in sync.
One of the key architectural decisions in this project was to separate the storage layer from the access management layer. While data was stored in Amazon S3, access was managed through S3 Access Points, providing fine-grained control. Access Points were created in the storage account, while IAM roles and users were provisioned in a separate access management account. This design provided strong governance and supported compliance with the client's cybersecurity policies.
To support a robust and maintainable platform, additional capabilities were implemented, including S3 lifecycle rules, event notifications, and...
Want to learn more?
If you are considering building a modern data platform or want to modernize your existing data architecture, I’d be happy to discuss how this kind of approach could help your organization. Let's talk!
