It's a interesting challenge from Cloud Guru Platform

I was very much excited when I have come to know about this challenge.. I thought I will complete that in my busy schedule to get confident as well to judge myself on those technologies..

What is the challenge and task ?

The task is basically to develop one ETL program in AWS (using different services) with Python and put the end result in visualization dashboard.. This type of idea always attracts me and that is the reason I could not ignore to participate and playing around all services..

My approaches to design this framework

I have used few services for security I have gone for IAM role, for processing like ETL I have chosen Lambda(asked to use), after transformation to load the data I have used RDS(PostgreSQL). Then I have visualized with Quicksight.. For sending any notification after successful run SNS has been used and Cloudwatch schedule has been used to schedule the job(to trigger lambda function) once in a day automated way

I have maintained one config file in json format and used it from S3 bucket to make the code as much as possible independent on the parameter.. For few things I have declared as environment variable in lambda function itself..

My learning from this challenge

The things I have learned little bit that is cloudformation.. I had plan to learn but never got a chance , Thanks Cloud Guru to give me this chance.. For whole things(IAM role,policy,Lambda func, Lambda layer, Cloudwatch Schedule,SNS topic/subscription) to deploy at once I was able to prepare cloudformation template (yaml file) and able to deploy using cloudformation stack.. This is great thing I have got after completing the framework.. Wish to learn more after that on cloudformation because it is really interesting..

One more thing was I neither used quicksight nor I have worked in visualization, But this challenge helped me to gain knowledge on visualization as well

one more thing here I have used auto refresh on visualization dashboard, have used quicksight scheduled refresh features

Repository & output

Source code - My Git Hub Repo

Quicksight Dashboard -

cloudFormation Design

!!!!! That's It, It was really nice journey.. Would like to get feedback.. Oh!! will wait for result from cloud guru and hopefully will be able to get some 'swag'/'prizes'.. Finger Crossed :D