127 points by cloud_architect 1 year ago ago | flag | hide | 17 comments
john_tech 4 minutes ago | prev | next
This is a great article, it's amazing to see a scalable, serverless real-time data processing architecture.
coding_fan 4 minutes ago | prev | next
Could you share more on how you achieved this? I'm looking for a similar solution.
Sure, we used AWS Lambda for computing, DynamoDB for data storage and Kinesis for real-time data.
streaming_expert 4 minutes ago | prev | next
I also recommend using a data stream to update caches and for data warehousing.
That is a good suggestion. We used AWS Data Streams for that. It helped us to achieve low-latency.
cloud_warrior 4 minutes ago | prev | next
I have implemented similar architecture on GCP using Cloud Functions, Cloud Firestore and Pub/Sub.
bigdata_projects 4 minutes ago | prev | next
Very cool! I'm curious, how did you handle data security and encryption?
We used AWS KMS for encrypting the sensitive data at rest and in transit.
aws_enthusiast 4 minutes ago | prev | next
Nice, this is a great use case for serverless. Have you experienced any cold start issues?
Yes, we faced cold start issues initially. However, increasing the memory size of the Lambda functions helped us reduce that.
new_to_all 4 minutes ago | prev | next
Thanks for the valuable information, I'm new to real-time data processing. Greatly appreciate it!
data_scientist 4 minutes ago | prev | next
Impressive! It's amazing to see the power of serverless in modern data processing.
cost_analyst 4 minutes ago | prev | next
How did you handle the cost aspect of such a scalable system?
We focused on optimizing our Lambda function size and usage, this helped us to avoid any unexpected bills.
newbie 4 minutes ago | prev | next
Great to know, I'm about to start a similar project and your insights will surely help!
helpful_hand 4 minutes ago | prev | next
There are some useful tools like AWS Cost Explorer to monitor and optimize your spending on the cloud services.
cloud_operations 4 minutes ago | prev | next
Also, using reserved Concurrency for Lambda functions gave us more control over the system.