What is serverless-data-pipeline?

Serverless Data Pipeline is a public demo project that shows how an image can flow through a small chain of cloud services without any servers running in the background. You upload an image in a browser, and it moves through Amazon S3 for storage, AWS Lambda for processing, and DynamoDB for recording information about it, all in real time. The flow works like this: the browser first asks a Lambda function for a special upload link, which checks the file's type and size before handing it over. The browser then uploads the image directly to S3 using that link, without the file ever passing through the Lambda function itself. Once S3 receives the new file, it triggers another Lambda function that reads metadata from the image and writes a record into DynamoDB. A separate endpoint lets the browser check the status of an upload and get a temporary link to view it. The whole setup avoids things like a private network (VPC) or a NAT gateway, and since it is built on Lambda and other managed services, it scales down to nothing and costs nothing when no one is using it. The README explains several reasons this demo is safe to leave publicly accessible: the browser never receives real AWS credentials, only a short-lived, narrowly scoped upload permission, uploaded files and their database records automatically delete themselves after 24 hours, the API is rate limited to prevent abuse from running up a bill, and file types are checked both in the browser and again on the server side so the check cannot be skipped by a user editing the request. Deploying it requires AWS credentials stored as GitHub Actions secrets. Pushing to the main branch triggers GitHub Actions to install each Lambda function's dependencies and run Terraform to create the infrastructure. After the first successful deployment, you retrieve the API's web address from Terraform's output and paste it into a placeholder in the included demo web page, which can then be hosted on any static site host, such as Netlify's drag and drop upload, with no build step required. The project is built with Terraform for infrastructure, AWS Lambda running Node.js 20, S3, DynamoDB, an HTTP API Gateway, and GitHub Actions for deployment.