[{"content":" # Hi, I am Harish Engineer I build scalable and high-performance systems. Twitter LinkedIn GitHub Mail ","date":"15 February 2025","externalUrl":null,"permalink":"/","section":"","summary":"","title":"","type":"page"},{"content":"","date":"15 February 2025","externalUrl":null,"permalink":"/categories/","section":"Categories","summary":"","title":"Categories","type":"categories"},{"content":"","date":"15 February 2025","externalUrl":null,"permalink":"/categories/go/","section":"Categories","summary":"","title":"Go","type":"categories"},{"content":"","date":"15 February 2025","externalUrl":null,"permalink":"/projects/","section":"Projects","summary":"","title":"Projects","type":"projects"},{"content":"","date":"15 February 2025","externalUrl":null,"permalink":"/categories/redis/","section":"Categories","summary":"","title":"Redis","type":"categories"},{"content":"Redis built from scratch in Go.\nCore Features # In-memory KEY VALUE store RESP Protocol: Redis Serialization Protocol Core Commands: GET, SET, ECHO, and PING. Passive Expiry: Redis\u0026rsquo; passive expiry mechanism, ensuring automatic removal of expired keys upon access. RDB Persistence: Redis Database Files persistence for data persistence. Handle Concurrent Clients Commands # GET get the VALUE using the KEY SET save the KEY VALUE pair data PING to check the health of Redis server SAVE save the in-memory data to .rdb file In Progress # Redis Replication Redis Stream Redis Transaction Redis Pub/Sub Source Code # harisheoran/redis-go Redis built from scratch in Go. Go 3 0 Workflows # High Level Workflow # RDB Persistence Workflow # Working Demos # RDB Persistence # Master-Slave Replication # ","date":"15 February 2025","externalUrl":null,"permalink":"/projects/redis-go/","section":"Projects","summary":"","title":"Redis-Go: Redis built from scratch in Go","type":"projects"},{"content":" Multi-threaded Proxy server # A multi-threaded forward proxy server in Go using low-level socket connections to handle concurrent client requests efficiently.\nCaching: Build in-memory LRU cache to store and serve frequently accessed response Scalability: Integrated LRU cache with mutexes for thread safety and semaphore-based concurrency control (buffered channels) to manage request limits Technical Stack: Go, HTTP, TCP/IP, Concurrency, Channels, Mutex. 🚀 Features # HTTP proxy Support Concurrent request handling with semaphore limiting Caching Strategy In-memory LRU Cache Handling request on TCP level Source Code # harisheoran/multi-threaded-proxy-server Muliti threaded proxy server with caching support. Go 0 0 Download # Download the v1.0.1\n📺 Watch full demo # 📂 Project Structure # ├── app.go # contains server classes ├── go.mod # go module file ├── helper.go # helper functions ├── internal │ └── cache │ └── lru.go # LRU cache implementation ├── main.go ├── proxy.go # proxy implementation 🛠️ Setup \u0026amp; Run # Clone the repository # git clone \u0026lt;repository-url\u0026gt;\ncd multithreaded-proxy-web-server\nInstall Go dependencies # go mod tidy\nBuild the binary # go build -o bin/proxy .\nExecute the binary # ./bin/proxy -port=9000\nnow the proxy server is running on port 9000\nTest Locally using Postman # First, setup proxy setting to port 9000 in postman application, and you are all good to send your requests.\nTo be implemented (open for contribution) # HTTPS support Cache Invalidation policy ➜ multithreaded-proxy-web-server git:(main) ","date":"5 February 2025","externalUrl":null,"permalink":"/projects/multithreaded-proxy-web-server/","section":"Projects","summary":"","title":"Multi-threaded Proxy server","type":"projects"},{"content":"","date":"5 February 2025","externalUrl":null,"permalink":"/tags/","section":"Tags","summary":"","title":"Tags","type":"tags"},{"content":"","date":"5 February 2025","externalUrl":null,"permalink":"/tags/tcp/","section":"Tags","summary":"","title":"Tcp","type":"tags"},{"content":"Introducing Chat System Enterprise, a cutting-edge, on-premises chat application designed specifically for companies that prioritize security, privacy, and control. It provides organizations with the ability to host their own private communication platform, ensuring sensitive data remains within their infrastructure. Tailored for corporate environments, it empowers teams to collaborate effectively while meeting strict compliance and security requirements.\nSource Code # harisheoran/my-chat-system Scalable group Chat application Go 0 0 Architectue # 📺 Watch full demo # Product Description # Chat System Enterprise is a high-performance communication platform built to handle the demands of modern businesses. Engineered with scalability and reliability in mind, it features:\nReal-time Messaging: Seamless group chat functionality.\nScalability and Performance: Built with GoLang, Kafka, and Redis to handle high user loads efficiently.\nCustomization and Control: Fully hostable on on-premises servers, providing complete data sovereignty.\nTechincal Stack # Backend: GoLang for high-performance and efficient server-side processing.\nMessaging and Data Handling: Kafka and Redis for real-time and scalable messaging infrastructure.\nDeployment: Designed for on-premises hosting to provide data control and compliance.\n","date":"1 January 2025","externalUrl":null,"permalink":"/projects/my-chat-system/","section":"Projects","summary":"","title":"Chat System: On-Premises Chat Application for Enterprises","type":"projects"},{"content":"","date":"1 January 2025","externalUrl":null,"permalink":"/tags/kafka/","section":"Tags","summary":"","title":"Kafka","type":"tags"},{"content":"","date":"1 January 2025","externalUrl":null,"permalink":"/tags/redis/","section":"Tags","summary":"","title":"Redis","type":"tags"},{"content":"","date":"29 October 2024","externalUrl":null,"permalink":"/categories/aws/","section":"Categories","summary":"","title":"Aws","type":"categories"},{"content":"","date":"29 October 2024","externalUrl":null,"permalink":"/tags/eks/","section":"Tags","summary":"","title":"Eks","type":"tags"},{"content":"","date":"29 October 2024","externalUrl":null,"permalink":"/categories/k8s/","section":"Categories","summary":"","title":"K8s","type":"categories"},{"content":"Today, we are going to deploy Scale Mesh API on Kubernetes with production standards.\nWe are going to deploy Scale Mesh, a vercel alternative to deploy your web apps with ease.\nWhat we are going to learn during this deployment? Pods Deployments Replica Set Service Namespace Secrets Ingress Ingress Controller Custom Resources Custome Resource Defination Custom Controller Operators Helm Charts Service Account OIDC 📺 Watch on Youtube # About API # Application Structure # This is a 2 tier application,\napi-server is the main API. postgres db is the primary database. Prerequiste # Install AWS CLI eksctl Deployment Architecture # Create a Kubernetes Cluster # Create the Kubernetes cluster on your favourite cloud or even bare metal, I am choosing the AWS as I got some free credits :)\nWe are using eksctl cli tool to create the cluste, but you can use another tools like Terraform.\nI am creating the cluster in a existing VPC, so need to provide the vpc and subnet details\nCreate an eks.yaml and copy the content of this snippet eksctl create cluster -f eks.yaml Wait for a while\u0026hellip; a liitle more while\u0026hellip;\nCreate a tag on subnet of your VPC, so that the AWS Load Balancer controller can auto discover the subnets. Tag the public subnet- kubernetes.io/role/elb: 1 Tag the private subnet- kubernetes.io/role/internal-elb: 1 Start Deploying # Setup # Clone the Scale Mesh Respository. Go the k8s directory, it contains all the YAML files. cd ./scale-mesh/k8s Export these 2 variables in your shell,\nChoose your region export REGION=ap-south-1 Choose the name of the cluster export CLUSTER_NAME=scale-mesh-test-cluster Deployement Visualization. # \u0026hellip;.\nCreate a namespace # Namespace is used to group resources within a cluster.\nIt is not a network or security boundry. used to origanise the resources. kubectl apply -f namespace.yaml Deploy the Database as Stateful Set # Why Stateful Set? # Pods are ephermeral in nature, means they can destroyed and created at any time, so stateless in nature also, but database data is critical to our application, we can\u0026rsquo;t just lose our users data, so we need a persistant disk to save the data to.\nStateful Set A Stateful Set is a type of worload resource which is used to manage the deployement and scaling of pods where maintaining a stable and unique identity for each pod is necessary and where application needs persistance volumes such as database.\nPersistant Volume Claim(PVC)\nPVC is a request for storage, it consumes storage resources defined by Persistant Volume.\nDynamically Provisioning: K8s automatically provision the a PV that matches the request, This only works if a storage class is available and configured in the cluster. Binding: Once a PVC is bound to a PV, it remains bound until released or deleted explicitly. PersistentVolume (PV)\nA PersistentVolume (PV) is a storage resource in Kubernetes that provides an abstraction for how storage is provisioned and used.\nInstall the CSI driver on Cluster # The CSI specification provides a standard that enables connectivity between storage systems and container orchestration (CO) platforms.\nThe Container Storage Interface (CSI) driver in Kubernetes is necessary for enabling dynamic provisioning and management of storage resources from various storage providers in a standardized and portable way.\nKubernetes natively supports a limited set of storage backends. The CSI standard allows external storage providers (like Amazon EBS, Google Persistent Disk, Ceph, NFS, and others) to be integrated easily without modifying the core Kubernetes codebase.\nInstall Container Storage interface in Worker node, so that it can support storage to our container, as our postgres db pod need storage for volumes.\neksctl create addon --name aws-ebs-csi-driver --cluster $CLUSTER_NAME --region ap-south-1 Deploy the Postgres DB using Helm # Helm is a package manager for Kubernetes, we can install the package from Helm repositories, we can manager multiple releases of a single package(support rollback release also).\nAdd the Postgres Helm repo We are using Bitnami Helm repo for the Database. helm repo add bitnami https://charts.bitnami.com/bitnami Install the Postgres DB helm install maindb bitnami/postgresql --version 15.5.37 --create-namespace --values=values.yaml IMAGE HERE\nCreate the IAM policy # Create and attach an IAM policy to eks Node Role with permissions to: EC3 Volumes permission to create and delete volumes Create and describe EC2 tags So that out worker node have permission to access the volumes and create some tags.\nCheck the Postgres DB status # Deploy the API # Create the env secrets for the API. SESSION_SECRET DATABASE URI kubectl apply -f api-server-secrets.yaml kubectl apply -f api-server.yaml Pods Pods are the running specification of a container or multiple containers.\nDeployment Deployment is a workload resources which deploy the pods using replica set controller which ensure that the desired state of the pod is the actual state of pod in the cluster.\nService Service is a method for exposing a network application that is running as one or more Pods in your cluster.\nTypes:\nNodePort ClusterIP Loadbalancer Create an IAM OIDC provider. # In an Amazon EKS (Elastic Kubernetes Service) environment, the IAM OIDC (OpenID Connect) provider is crucial for enabling Kubernetes workloads to authenticate and securely interact with AWS services. why it\u0026rsquo;s needed and what it does:\nFine-Grained Permissions for Pods: In a Kubernetes cluster, workloads running in Pods may need to access AWS services like S3, DynamoDB, Secrets Manager, or others. Using the OIDC provider, you can associate IAM roles with specific Kubernetes service accounts. How IAM role and K8s service account work together in EKS?\nA Kubernetes service account is created and annotated with an IAM role that provides permissions for AWS services. IAM Role for Service Account (IRSA) integration allows the service account to assume the IAM role and use it to interact with AWS resources securely. This way, workloads running in your EKS cluster can access AWS services using the IAM role, while still using the service account for Kubernetes API interactions.\nCreating IAM Roles for Pods: You create IAM roles with specific policies for AWS services your Pods need to access. Each role includes a trust relationship that allows the OIDC provider to authenticate and authorize access. The OIDC provider acts as a bridge between the EKS cluster and AWS IAM, facilitating this authentication process.\neksctl utils associate-iam-oidc-provider \\ --region $REGION \\ --cluster $CLUSTER_NAME \\ --approve Kubernetes Service Accounts\nA Kubernetes Service Account is an identity that is used by Pods to interact with the Kubernetes API. It provides a way for applications running in a Kubernetes cluster to authenticate to the cluster and make API calls (for example, to list resources, create or delete objects, etc.).\nService accounts enable you to apply Role-Based Access Control (RBAC) policies Pod level authentication Secure Access Set up Ingress # Ingress exposes the HTTP, HTTPS routes from outside the cluster to the services within the cluster.\nTraffic routing is controlled by rules defined in Ingress resources.\nTypes of routing\nHost Based (DNS) Path Based (/) Tag the public subnet- kubernetes.io/role/elb: 1\nTag the private subnet- kubernetes.io/role/internal-elb: 1\nRead more on how to set up Ingress Controller here.\nInstall the IAM policy curl -o iam-policy.json https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/v2.8.2/docs/install/iam_policy.json Create the IAM policy aws iam create-policy \\ --policy-name AWSLoadBalancerControllerIAMPolicy \\ --policy-document file://iam-policy.json Create the service account and associate the IAM role eksctl create iamserviceaccount \\ --cluster=$CLUSTER_NAME \\ --namespace=kube-system \\ --name=aws-load-balancer-controller \\ --attach-policy-arn=arn:aws:iam::\u0026lt;AWS_ACCOUNT_ID\u0026gt;:policy/AWSLoadBalancerControllerIAMPolicy \\ --override-existing-serviceaccounts \\ --region $REGION \\ --approve Add the Helm repo helm repo add eks https://aws.github.io/eks-charts Install helm install aws-load-balancer-controller eks/aws-load-balancer-controller -n kube-system --set clusterName=$CLUSTER_NAME --set serviceAccount.create=false --set serviceAccount.name=aws-load-balancer-controller --set region=$REGION --set vpcId=\u0026lt;vpc id of cluster\u0026gt; Wait for the installation of controller kubectl get deploy -n kube-system -w Create the ingress resource kubectl apply -f ingress.yaml Check the ingress resource to view the loadbalancer DNS kubectl get ingress -n api-ns View logs of controller if alb is not created kubectl logs -n kube-system -l app.kubernetes.io/name=aws-load-balancer-controller ERROR # We got an error, as we need to run the ECS task from the EKS cluster, we didn\u0026rsquo;t attach any role \u0026amp; permission to it.\nhandlers.go:89: Unable to Run the ECS TASK operation error ECS: RunTask, https response error StatusCode: 400, RequestID: 0d384199-4c27-4c57-b7f5-0bd0704fcae5, AccessDeniedException: SOLUTION: Give permission to the API pod to access the ECS task.\n{ \u0026#34;Version\u0026#34;: \u0026#34;2012-10-17\u0026#34;, \u0026#34;Statement\u0026#34;: [ { \u0026#34;Sid\u0026#34;: \u0026#34;VisualEditor0\u0026#34;, \u0026#34;Effect\u0026#34;: \u0026#34;Allow\u0026#34;, \u0026#34;Action\u0026#34;: [ \u0026#34;ecs:SubmitTaskStateChange\u0026#34;, \u0026#34;ecs:UpdateCluster\u0026#34;, \u0026#34;ecs:UpdateClusterSettings\u0026#34;, \u0026#34;ecs:StartTask\u0026#34;, \u0026#34;ecs:DescribeTaskSets\u0026#34;, \u0026#34;ecs:DescribeTaskDefinition\u0026#34;, \u0026#34;ecs:DescribeClusters\u0026#34;, \u0026#34;ecs:RunTask\u0026#34;, \u0026#34;ecs:ListTasks\u0026#34;, \u0026#34;ecs:ListTaskDefinitionFamilies\u0026#34;, \u0026#34;ecs:StopTask\u0026#34;, \u0026#34;ecs:UpdateServicePrimaryTaskSet\u0026#34;, \u0026#34;ecs:DescribeTasks\u0026#34;, \u0026#34;ecs:ListTaskDefinitions\u0026#34;, \u0026#34;ecs:UpdateTaskSet\u0026#34;, \u0026#34;ecs:CreateTaskSet\u0026#34;, \u0026#34;ecs:ListClusters\u0026#34; ], \u0026#34;Resource\u0026#34;: \u0026#34;*\u0026#34; }, { \u0026#34;Sid\u0026#34;: \u0026#34;VisualEditor1\u0026#34;, \u0026#34;Effect\u0026#34;: \u0026#34;Allow\u0026#34;, \u0026#34;Action\u0026#34;: \u0026#34;iam:PassRole\u0026#34;, \u0026#34;Resource\u0026#34;: [ \u0026#34;arn:aws:iam::637423604544:role/ecsTaskExecutionRole\u0026#34;, \u0026#34;arn:aws:iam::637423604544:role/scale-mesh-s3\u0026#34; ] } ] } Don\u0026rsquo;t forget to give the permission of IAM Pass role, as another service(EKS) is giving the role to ECS, so need permssion of assuming an IAM role\nCreate a new service account to attach to the API pod. kubectl create serviceaccount api-svc-account -n api-ns Attach the IAM policy to the service account. eksctl create iamserviceaccount \\ --name api-svc-account \\ --namespace api-ns \\ --cluster $CLUSTER_NAME \\ --attach-policy-arn=arn:aws:iam::637423604544:policy/scale-mesh-ecs-task-for-eks \\ --override-existing-serviceaccounts \\ --region $REGION \\ --approve Continous Delivery # Setup ArgoCD\nSecrets Management # Currently, our api secrets are only encrypted using base64 which is not secures.\n","date":"29 October 2024","externalUrl":null,"permalink":"/projects/scale_mesh_deployment/","section":"Projects","summary":"","title":"Kubernetes Production Deployment of Scale Mesh","type":"projects"},{"content":"","date":"29 September 2024","externalUrl":null,"permalink":"/tags/docker/","section":"Tags","summary":"","title":"Docker","type":"tags"},{"content":"Scale Mesh: A Vercel alternative.\n🧑‍💻You can deploy your web applications with just a click.\nFeatures:\nIt dynamically builds and hosts user web apps straight from GitHub repos. View the logs of your deployments. Build \u0026amp; deploy multiple web apps simultaneously from GitHub repos. Architecture # Source Code # Harish Sheoran / Scale Mesh 0 0 📺 Demo # There are two Architectures of Scale Mesh\nBuild User\u0026rsquo;s app using AWS ECS. Build User\u0026rsquo;s app using Kubernetes Job. ⚙️ Components: # Build Server Reverse Proxy API Main API Server Log Collection Pipeline 🏗️ Build Server # It builds the code and pushes the artifacts to the S3 bucket.\n🤔 What it do? # It is a custom Docker container which takes the GitHub repository URL, clones it and build the application artifacts and push to the S3 bucket.\n🤔 How it works? # It is a docker container built using multi-stage dockerfile, which builds the Go binary in the first stage and then runs the binary in 2nd stage, this binary contains the code to build the user\u0026rsquo;s web app code and push the build artifacts to the S3 bucket.\nThis Docker image is hosted on private AWS ECR.\n🔧 Workflow\n1. Kubernetes Version # A k8s Job triggered which uses our image from ECR, and create \u0026amp; run the Docker container.\n2. Initial version # It uses AWS ECS to run the container as a ECS Task.\nWhy run the container as a task? # Task is a short lived process which do the job and destroy the container, so efficient uses of resources and saving the cost. 👉 So, the user can build \u0026amp; deploy multiple apps simultaneously, and the load will not affect the primary api-server.\n🙋API Server # The main backend API is the one via which the user interacts with the platform.\n🌊Working Flow ➡️User Authentication. ➡️Save the user\u0026rsquo;s web app project info(GitHub repo URL, name) to deploy. ➡️Create single/multiple deployments from the project. ➡️Create the project and Deploy the web app.\n🧑‍💻Logging Pipeline # The build-Server container pushes the logs to Redis using the pub/sub feature and a web socket server is subscribing to the Redis channel to view the logs.\nDeployment on Kubernetes Cluster # Scale Mesh is deployed on Kubernetes Cluster.\nCurrent Issues: # User can start multiple build job task, so it should be handled in Message Queue. No logs database, so need Clickhouse DB. For sending logs in redis pub/sub, we can use Kafka. ","date":"29 September 2024","externalUrl":null,"permalink":"/projects/scale-mesh/","section":"Projects","summary":"","title":"Scale Mesh: A better Vercel alternative","type":"projects"},{"content":"","date":"29 July 2024","externalUrl":null,"permalink":"/tags/cli/","section":"Tags","summary":"","title":"CLI","type":"tags"},{"content":"Repoguard is CLI tool built in Go language to monitor access of GitHub Repositories.\nSource Code # harisheoran/repoguard Monitor access of GitHub Repository access. Go 0 0 How to Install # Go to the Release section Download for your OS wget \u0026lt;file link from release section for your OS\u0026gt; tar -xzvf \u0026lt;downloaded tar file\u0026gt; mv repoguard /usr/local/bin How to use Repoguard? # Configure the CLI repoguard configure and provide it the GitHub Token\nusers command to get a list of collaborators.\nUse flag --u for providing the username of GitHub. Use flag --r for providing the GitHub repository name. revoke to revoke access of a user.\nUse flag --o for providing the owner of GitHub(the name of admin github account) Use flag --r for providing the repository name of GitHub account of that owner. Use flag --u for providing the username whom access you want to revoke. Features # Implemented # Get a list of users who have access to the GitHub Repository. Revoke access of user. Configure repoguard for your GitHub account. Demo # ","date":"29 July 2024","externalUrl":null,"permalink":"/projects/repoguard/","section":"Projects","summary":"","title":"Repoguard: Secure Your Repos, Manage Access Seamlessly.","type":"projects"},{"content":" AWS Cloud Cost Optimization: Monitoring and Eliminating Stale Resources # Managing cloud costs is crucial for any organization leveraging AWS services. Imagine you have EC2 instances with attached EBS volumes containing important data. You take regular snapshots for backups, but when you delete the EC2 instances and their volumes, you might forget about those snapshots. This oversight can lead to unnecessary costs, especially in large organizations where tracking such stale resources is challenging. In this blog, we\u0026rsquo;ll explore an efficient AWS Lambda function designed to identify and eliminate these forgotten snapshots, helping you optimize your cloud expenditure.\nSource Code # harisheoran/aws_cost_optimization AWS Cost Optimization Lambda to monitor the stale Snapshots. HCL 0 1 Demo # AWS Lambda # AWS Lambda service provides serverless computing on which we can run our code as a Lambda function which triggers based on an event.\nWhy Lambda? # AWS lambda is serverless which means it only runs when it is triggered and the rest of the time it sits idle, not utilizing any compute resource and it is auto-scalable also.\nYou can write lambda function in any language you choose, I am using Go language.\nBuild it # Let\u0026rsquo;s take the above scenario in which at the end we have stale snapshots which are not associated with volumes, basically costing us money. Solution # First, get the Snapshot\u0026rsquo;s volume ID and snapshot ID. Check that the volume exists or not using that volume ID if volume does not exist, delete that snapshot using snapshot ID. Code it # AWS provide AWS SDK for Go to interact with AWS resources.\nInstall AWS SDK v2 Configure the ADK to send request Install AWS Go lambda package We mainly use AWS EC2 service, so refer to EC2 service package for documentation.\nGet the Snapshot using ownerID (owner ID of the caller\u0026rsquo;s AWS account, so as to get only our snapshots) // get owner ID of caller AWS account stsClient := sts.NewFromConfig(configs) callerId, err := stsClient.GetCallerIdentity(context.TODO(), \u0026amp;sts.GetCallerIdentityInput{}) if err != nil { fmt.Println(\u0026#34;ERROR\u0026#34;, err) } ownerID := *callerId.Account // get snapshots Volume ID snapshot, err := client.DescribeSnapshots(context.TODO(), \u0026amp;ec2.DescribeSnapshotsInput{ OwnerIds: []string{ownerID}, }) if err != nil { fmt.Println(\u0026#34;ERROR\u0026#34;, err) } get the Snapshot\u0026rsquo;s volume ID and snapshot ID and send a request using that volume ID and catch the error type to check the stale snapshot and delete the snapshot using snapshot ID. // check for stale snapshots: check for every snapshot\u0026#39;s Volume ID, and check if exists for _, snap := range snapshot.Snapshots { snapID := *snap.SnapshotId volID := *snap.VolumeId _, err := client.DescribeVolumes(context.TODO(), \u0026amp;ec2.DescribeVolumesInput{ VolumeIds: []string{volID}, }) if err != nil { var volErr smithy.APIError if errors.As(err, \u0026amp;volErr) { switch volErr.ErrorCode() { case \u0026#34;InvalidVolume.NotFound\u0026#34;: fmt.Println(\u0026#34;Deleted the %s stale snapshot\u0026#34;, snapID) deleteStaleSnapshots(snapID, client) default: fmt.Printf(\u0026#34;ERROR\u0026#34;, volErr) } } else { fmt.Printf(\u0026#34;Error in checking volume\u0026#34;) } } else { fmt.Println(\u0026#34;No stale snapshots available.\u0026#34;) } } func deleteStaleSnapshots(snapID string, client *ec2.Client) { output, err := client.DeleteSnapshot(context.TODO(), \u0026amp;ec2.DeleteSnapshotInput{ SnapshotId: \u0026amp;snapID, }) if err != nil { fmt.Println(\u0026#34;ERROR\u0026#34;, err) } fmt.Println(*output) } Start the lambda from the main function. func main() { lambda.Start(awsCostOptimize) } Create a Lambda function # Go to Lambda service in the AWS console(you also can use Terraform, which I have already created, check the source code under iac directory) Create a binary of the code, lambda only accepts the name bootstrap CGO_ENABLED=0 go build -o bootstrap main.go\nCreate a zip file to upload code to lambda. zip lambda-handler.zip bootstrap\nUpload the zip file Try to run it from the test section After running, it failed, why? Let\u0026rsquo;s check the logs.\nThis is an error -\nERROR operation error EC2: DescribeSnapshots, https response error StatusCode: 403, RequestID: 63f1db48-7100-4025-88e4-b914523d871a, api error UnauthorizedOperation: You are not authorized to perform this operation. User: arn:aws:sts::637423604544:assumed-role/cost_optimization-role-el6aymo6/cost_optimization is not authorized to perform: ec2:DescribeSnapshots because no identity-based policy allows the ec2:DescribeSnapshots action By looking at the error, we can see that we are not authorized to perform this action.\nSolution\nBy default AWS doesn\u0026rsquo;t allow any service to talk to another AWS service which in our case Lambda service is trying to talk to the AWS EC2 service, this is for security.\nIAM Roles\nWe have to assign IAM roles to Lambda which has permission to perform our desired action on EC2 service.\nGo to configuration \u0026gt; permission and click on role_name link Create an Inline policy and allow these 3 permissions on the EC2 Service Test the function again and it ran successfully.\nTry to create some ec2 instances and attach some snapshots and then delete those ec2 instances and you will be left with stale snapshots, now try to run the lambda function, you can do it manually or schedule a cron job.\nThis Cost optimization technique can be applied to other scenario also.\n","date":"12 July 2024","externalUrl":null,"permalink":"/projects/cloud_cost_optimization/","section":"Projects","summary":"","title":"AWS Cost Optimization","type":"projects"},{"content":"","date":"12 July 2024","externalUrl":null,"permalink":"/tags/lambda/","section":"Tags","summary":"","title":"Lambda","type":"tags"},{"content":"","date":"29 March 2024","externalUrl":null,"permalink":"/tags/aws-certificate-manager/","section":"Tags","summary":"","title":"AWS Certificate Manager","type":"tags"},{"content":"","date":"29 March 2024","externalUrl":null,"permalink":"/tags/aws-codebuild/","section":"Tags","summary":"","title":"AWS CodeBuild","type":"tags"},{"content":"","date":"29 March 2024","externalUrl":null,"permalink":"/tags/aws-codedeploy/","section":"Tags","summary":"","title":"AWS CodeDeploy","type":"tags"},{"content":"","date":"29 March 2024","externalUrl":null,"permalink":"/tags/aws-codepipeline/","section":"Tags","summary":"","title":"AWS CodePipeline","type":"tags"},{"content":"","date":"29 March 2024","externalUrl":null,"permalink":"/tags/aws-systems-manager/","section":"Tags","summary":"","title":"AWS Systems Manager","type":"tags"},{"content":"","date":"29 March 2024","externalUrl":null,"permalink":"/tags/cloudfront/","section":"Tags","summary":"","title":"Cloudfront","type":"tags"},{"content":"","date":"29 March 2024","externalUrl":null,"permalink":"/tags/dynamo-db/","section":"Tags","summary":"","title":"Dynamo DB","type":"tags"},{"content":"End-to-End Production-Grade Deployment: Advanced DevOps Practices on AWS\nAbout Project\nDeveloped a professional resume website hosted on AWS, utilizing various AWS services for robustness and security. Leveraged AWS CloudFront CDN for efficient content delivery and AWS S3 bucket for hosting static website content, ensuring scalability and reliability. Implemented HTTPS protocol for enhanced security, supported by SSL/TLS certificate managed by AWS Certificate Manager. Managed DNS with AWS Route 53 for seamless domain management and resolution. Engineered a view tracking API using Go language, integrated with AWS DynamoDB for efficient data storage and retrieval. Containerized the View Tracking API using Docker for streamlined deployment and scalability. Orchestrated deployment pipeline on AWS utilizing AWS CodePipeline, CodeBuild, and CodeDeploy for automated building, testing, and deployment of the API Docker image. Employed various AWS services including IAM Roles and IAM Users for secure access management and authorization. Source Code # harisheoran/AWS-Cloud-Resume AWS Full Stack Project CSS 5 3 Full Demo # View full Playlist.\nBuild Website # Build a simple static Resume Website. I used this Bootstrap Template.\nHere is my resume website code.\nWebsite Hosting # Host the content on S3 Bucket\nCreate a S3 Bucket as same name as you Domain name. Remove check from Block Public Access . Upload all the website files. Enable Static Website hosting. Go to Properties \u0026gt; Static website hosting\nEnable it \u0026amp; fill Index Document Go to the URL it has created, you will get 403 error WHY? What happens?\nWe need to allow to access S3 by defining Bucket Policy { \u0026#34;Version\u0026#34;: \u0026#34;2012-10-17\u0026#34;, \u0026#34;Statement\u0026#34;: [ { \u0026#34;Sid\u0026#34;: \u0026#34;AddPerm\u0026#34;, \u0026#34;Effect\u0026#34;: \u0026#34;Allow\u0026#34;, \u0026#34;Principal\u0026#34;: \u0026#34;*\u0026#34;, \u0026#34;Action\u0026#34;: \u0026#34;s3:GetObject\u0026#34;, \u0026#34;Resource\u0026#34;: \u0026#34;arn:aws:s3:::YOUR_BUCKET_NAME/*\u0026#34; } ] } Now visit the Static website URL of S3 bucket, website is accessible now Issue : We unchecked the Block Public Access, which makes the S3 bucket\u0026rsquo;s content accessible to everyone even to hackers which is an security issue.\nWhy do we need Cloudfront? # Now our website is puclicly accessible, why do we need the Cloudfront.\nCloudFront is a Content Delivery Network(CDN) service also provided by AWS, it enables users to deliver content globally with low latency, high data transfer speed, and securely delivered. leveraging AWS edge locations around the world to cache and deliver content from the nearest location to the user, reducing latency and improving performance. It also offers features such as SSL/TLS encryption, access controls, and real-time metrics.\nIssue : Our Website is not secure, it is using http protocol First, check the Block Public Access on S3 bucket. Create a SSL certificate using AWS Certificate Manager, create this certificate in North Verginia region. Create a CNAME record for this certificate in Hosted Zone. Must Read : Understanding DNS and Amazon\u0026rsquo;s Route 53 Cloud Service\nCreate Cloudfront Distribution Follow this blog After creating the CDN, add the Bucket policy (CDN will provide you after creating it) to S3 bucket, so that you can not view content of S3 bucket without bypassing the CDN. Create CNAME record with SSL certificate while creating the CDN, so it can use HTTPS protocol. Visit the CDN url, our resume website is live now, but it is not using our Domain name.\nRoute 53 # To manage our DNS records.\nCreate a Hosted Zone.\nAdd the Nameservers of Hosted Zone to the Domain Provider Nameservers. Create a A Record in hosted zone for Alias to redirect the request from domain name to CloudFront Distribution.\nCI pipeline for frontend Code. # We have to upload modified files to S3 bucket and remove the previous ones.\nUsing Github Actions. Do not use AWS credentials publicly, use them as secret variables. name: Push to S3 on: [push] jobs: deploy: runs-on: ubuntu-latest steps: - name: Checkout uses: actions/checkout@v4 - name: Setup AWS CLI uses: aws-actions/configure-aws-credentials@v4 with: aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }} aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }} aws-region: ${{ secrets.AWS_REGION }} - name: Sync files to S3 bucket run: | aws s3 sync ./website s3://harisheoran.xyz --delete View Tracking API # API is build using Go, Gin Framework and AWS SDK v2. It uses AWS Dynamo DB to store the views.\nWorking : When we send a GET request on \u0026ldquo;/\u0026rdquo; endpoint, it fetch the current views and add one more to it and save it to the database.\nCRUD operations in Dynamo DB with GO\nContainerize the API # Using Go lang as base image, download the dependencies and build a executable file and run it.\nFROM golang:1.22 WORKDIR /app COPY go.* ./ RUN go mod download COPY . . RUN mkdir build RUN CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -o build/main . EXPOSE 4000 ENTRYPOINT [ \u0026#34;./build/main\u0026#34; ] My learning \u0026amp; mistakes during building docker image\nLocalhost is the localhost of docker itself not the host machine. Multistage Docker build issue - Can\u0026rsquo;t access the shell of container. Docker Volume helps a lot In development to sync local directory with container Named Volume (Anonymous Volumes) help in not syncing the container\u0026rsquo;s specific directory Bash Script is a must skill to have - $(pwd)\ndocker container run -v $(pwd):/app -v name:/app/main -d -p 4000:4000 --env-file ./.env view_api_img Checking the logs of the container. Setting the environment for container. CI pipline using AWS CodeBuild # Purpose: To build the API image and push to Docker Hub Registry.\nCreate an IAM Role.\nCreate a Group for the project and attach the policy for AWS CODEBUILD full access.\nCreae a user and add to the group and attach Policies.\nList IAM Role (To attach it in Code build service role) To pass a role to a service (here we are giving role to our CodeBuild service). Pipeline: Checkout the code from Github Repository and use Dockerfile to build the image and push to Docker Hub using Docker Credentials.\nConnect to the Github account.\nNeed a Service Role -\nWHY? : So that CodeBuild can perform action on your behalf. How it works: First for which service you need this role and give it permission(Attach policies) for what it is going to do like in our case it need to access AWS Systems Manager Parameter store. Buildspec file: Define how your CI is going to do.\nSteps:\nLogin to Docker Hub, build the image and push the image\nCredentials: Store them in AWS Systems Manager Parameter store.\nLogs are very important, so use the Cloudwatch logs.\nStore this file in root directory.\n# buildspec.yml version: 0.2 env: parameter-store: DOCKER_USERNAME: docker_username DOCKER_PASSWORD: docker_password DOCKER_URL: docker_url phases: build: commands: - cd tracker_api/ - echo $DOCKER_PASSWORD | docker login -u $DOCKER_USERNAME --password-stdin $DOCKER_URL - docker build -t \u0026#34;$DOCKER_USERNAME/view_api_img:latest\u0026#34; . - docker push \u0026#34;$DOCKER_USERNAME/view_api_img:latest\u0026#34; post_build: commands: - echo \u0026#34;Successfully pushed to Docker hub Registry\u0026#34; artifacts: files: - \u0026#39;**/*\u0026#39; name: my-artifact CD pipeline using AWS CodeDeploy # Purpose: To deploy and run the latest code of application on target compute platform.\nIs it looking for change in the codebase? Is it invoked by AWS Pipeline?\nCreate Target Compute \u0026gt; EC2 instance.\nFirst, go to CodeDeploy \u0026gt; Applications \u0026gt; Create application and create an application, choose the EC2 as compute platform.\nCreate Deployment Group.\nInstall the CodeDeploy agent on EC2.\nWhy do we need CodeDeploy agent? ; It automates code deployments to any instance, including Amazon EC2 instances and instances running on EC2.\nAttach Tags to the EC2 instance. WHY? : Create all the resources of one project under one tag, helps in filtering the AWS resources. Here, we need tag for the code deploy service to id the target EC2 instance, we can assign single or multiple ec2 instances using tags.\nNow, we need a permissions -\nAs, our target EC2 instance going to talk to Code Depoly and vice versa. So, create two service roles, for Code Deploy and EC2. Add another IAM policy to our group to view Tags Keys and values.\nFinally create the Deployment.\nUpload our .env file on EC2 using SSH . It stores the AWS Credentials as API is using Dynamo DB.\nDepolyment Group\nCreate a Deployment Group. Create a appspec.yml\nHere, we define how we want to deploy and run the application, need two scripts to start and stop the container. # appspec.yml version: 0.0 os: linux hooks: ApplicationStop: - location: scripts/stop_container.sh timeout: 300 runas: root AfterInstall: - location: scripts/start_container.sh timeout: 300 runas: root Running the API # Run Migration First from container shell to create the AWS Dynamo DB Table. docker exec -it \u0026lt;docker container id\u0026gt; bash go run migrations/migrations.go ERRORS # AWS CodeBuild CodeDeploy agent was not able to receive the lifecycle event. Check the CodeDeploy agent logs on your host and make sure the agent is running and can connect to the CodeDeploy server.\nSOLUTION:\nCheck Logs at /var/log/aws/codedeploy-agent Restart codedeploy agent service Check IAM role policies for Code Deploy and EC2 AWS CodePipeline Unable to create role\nfor primary source and source version arn:aws:s3:::codepipeline-ap-south-1-972545925755/view_tracking_pipeli/SourceArti/Od8R23 SOLUTION:\nhttps://docs.aws.amazon.com/codepipeline/latest/userguide/troubleshooting.html#codebuild-role-connections\nhttps://stackoverflow.com/questions/64300151/aws-codebuild-failed-client-error-authorization-failed-for-primary-source-and-s\nAWS CodeDeploy during running the CodePipeline Same Issue with Code Deploy - Unable to access the artifact with Amazon S3 object key \u0026lsquo;view_tracking_pipeli/BuildArtif/mOIyXyD\u0026rsquo; located in the Amazon S3 artifact bucket \u0026lsquo;codepipeline-ap-south-1-972545925755\u0026rsquo;. The provided role does not have sufficient permissions.\nSOLUTION:\nUploading Build Artifact Error - https://medium.com/@shanikae/insufficient-permissions-unable-to-access-the-artifact-with-amazon-s3-247f27e6cdc3 Cache Invalidation issue of CloudFront Cache Invaildation Issue: Latest changes are unavailable to the users because the contents are cached at the CloudFront edge locations. The default caching policy keeps the contents cached for up to 24 hours (TTL).\nSOLUTION:\nhttps://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/Invalidation.html#invalidating-objects-api Request get blocked from resume website to the API\nThe page at \u0026lsquo;https://harisheoran.xyz/' was loaded over HTTPS, but requested an insecure resource \u0026lsquo;http://13.232.228.220:4000/\u0026rsquo;. This request has been blocked; the content must be served over HTTPS.\nSOLUTION:\nSo, we have to set up a proxy which forward the request to port 4000 and attach a TLS certificate so that it uses HTTPS protocol.\nSet up Reverse Proxy using Nginx - https://phoenixnap.com/kb/nginx-reverse-proxy Installed the TLS cert using Certbot Create A record in Domain Management with EC2 IP address. QUESTIONS\nWhy do we need the Deployment Groups? Why do we need to upload Artifacts to S3 bucket during AWS CodePipeline Blogs refrences # AWS Refrence for hosting simple static Website using S3\nDNS \u0026amp; Route 53\nHosting Website\nHosting Website with CDN OAI\nWhat is CloudFront\nDynamo DB with GO\n","date":"29 March 2024","externalUrl":null,"permalink":"/projects/aws_resume/","section":"Projects","summary":"","title":"Enterprise-Grade Website Deployment: Advanced DevOps Practices on AWS","type":"projects"},{"content":"","date":"29 March 2024","externalUrl":null,"permalink":"/tags/go/","section":"Tags","summary":"","title":"Go","type":"tags"},{"content":"","date":"29 March 2024","externalUrl":null,"permalink":"/tags/route-53/","section":"Tags","summary":"","title":"Route 53","type":"tags"},{"content":"","date":"29 March 2024","externalUrl":null,"permalink":"/tags/s3/","section":"Tags","summary":"","title":"S3","type":"tags"},{"content":"Creating Infrastructue on AWS using Terraform.\nThis project is part of AWS Resume Project. In this part we are going to create Infrastructure on AWS using Terraform.\nWorkflow of the project - Source Code # harisheoran/AWS-Cloud-Resume AWS Full Stack Project CSS 5 3 How to run the Project? # Clone the repository.\nCreate an S3 bucket and Dynamo DB for remote backend with lock mechanism. Read this blog\nSetup Hashicorp Vault.\nCreate EC2 instance, go to the vault directory and create the infrastructure using Terraform. terraform init \u0026amp;\u0026amp; terraform apply Upload both scripts on the ec2 server scp -i \u0026lt;aws key pair\u0026gt; install_vault.sh vault.sh ubuntu@\u0026lt;ip address\u0026gt;:/home/ubuntu/\nChange the permissions of both scripts chmod 700 install_vault.sh vault.sh\nExecute the scripts in order ./install_vault.sh ./vault.sh Create main resources, go to iac directory and apply the terraform\nCreate terraform.tfvars and provide the values bucket_domain=\u0026#34;\u0026#34; # Domain name domain=\u0026#34;\u0026#34; # Domain name role_id = \u0026#34;\u0026#34; # Role ID of hashicorp Vault secret_id = \u0026#34;\u0026#34; # Secret ID of Hashicorp Vault main_region = \u0026#34;\u0026#34; # AWS region, you can choose whichever you like cert_region = \u0026#34;\u0026#34; # AWS region for creating SSL/TLS certificate, must be in North Vergina s3_remote_backend_name = \u0026#34;\u0026#34; # S3 bucket name in which you want to upload terraform state file dynamo_db_remote_backend = \u0026#34;\u0026#34; # Dynamo DB table name for remote backend remote_backend_region = \u0026#34;\u0026#34; # Remote backend region env = \u0026#34;\u0026#34; # prod, dev, test Initialize the terraform terraform init\nApply the terraform terraform apply\nset is_zone to true for first time. Now copy output Nameservers and paste them in your Administrative DNS server(From which you bought the domain name). now run again terraform apply and set is_zone to false to create rest of the resources. Visit your Domain name.\nInfrastructure Requirements # S3 bucket (for hosting website files)\nUpload files Enable static website hosting Attach Bucket Policy Create a SSL/TLS certificate for website\nUse AWS Certificate Manager (ACM) Create this in North Verginia region Create Route 53 hosted zone and create CNAME record of certificate.\nCreate Cloudfront CDN and use S3 bucket as OAI\nAttach Cert to it Create A record for cdn url, alias to main Domain. Terraform # Terraform is a tool by Hashicorp for managing you infrastructure using code. Define your infrasturctue in HCL.\nBenefits # Easy to manage the infra with code. Works with major Cloud Providers. Support Hybrid Cloud. Working # Initialize the terraform terraform init Plan the infrastructure terraform plan Apply the infrastructure terraform apply Lets understand the Terraform practically by creating infra on AWS.\nFirst, we need to initialize the terraform and provide it the cloud service on which we want to create infra. terraform { required_providers { aws = { source = \u0026#34;hashicorp/aws\u0026#34; version = \u0026#34;~\u0026gt; 5.0\u0026#34; } } } # Configure the AWS Provider provider \u0026#34;aws\u0026#34; { region = \u0026#34;us-east-1\u0026#34; } terraform block is to configure the terraform with cloud provider.\nprovider block is to configure the cloud with the region.\nNow initialise the terraform terraform init Understand by creating a S3 bucket. # resource \u0026#34;aws_s3_bucket\u0026#34; \u0026#34;my-main-s3\u0026#34; { bucket = \u0026#34;harisheoran.xyz\u0026#34; tags = { Name = \u0026#34;main\u0026#34; Environment = \u0026#34;prod\u0026#34; } } resource block is to create resource on the cloud.\naws_s3_object is Terraform resource type, which represents a cloud\u0026rsquo;s resource or service or component.\nmy-main-s3 is the logical name of the resource. This allows you to refer to this specific instance of the resource within your Terraform configuration.\n{ } this block contains the configuration of the resource.\nExecute terraform plan It is a dry run, it show the logs of the infrastructure which is going to create.\nFinally, create the infrastructure. terraform apply Terraform State # Terraform State is the brain of terraform, using the state it knows about all the resources it created. Terraform create a state file after creating the resources, it records the logs of resources it has created.\nTerraform State working # Terraform create state file when you execute terraform apply for the first time, it saves the info of all the resources you created. When you want to create more resources or update the existing one, you write the configuration and apply the changes, then it checks the state file and checks for existing resources, and compare it with the what it have to create or update and find the difference and update or create accordingly.\nTerraform State benfits # Let say we create an EC2 instance using Terraform and after that we want to add a tag to it.\nif state don\u0026rsquo;t exist, Terraform has no idea that it already created an EC2 instance, so it\u0026rsquo;ll create a new one. if state exist, it\u0026rsquo;ll check first that EC2 instance is already created and it\u0026rsquo;ll add tag to it. Terraform State Drawbacks # Terrform State Solutions # Terraform Modules? # Modules are like resuable functions of terraform to create infra. If we have to create big infrastructure, then it would be mess to create all resources in one file, it\u0026rsquo;ll be hard to understand and maintain the code by team. So, we can create different modules for infra, and these modules don\u0026rsquo;t have to exist in our repo, they can be hosted anywhere.\nHow to create Modules? # Create a new directory for each module mkdir modules mkdir modules/s3_bucket Create the terraform files inside the module normally as we did before. Define which module to use in root main.tf module \u0026#34;s3_bucket\u0026#34; { source = \u0026#34;./modules/s3_module\u0026#34; } Variables # Variables are like as the name suggest.\nInput Variables # To Provide variables to terraform files\nvariable \u0026#34;bucket_name\u0026#34; { type = string description = \u0026#34;Bucket name same as Domain name.\u0026#34; default = \u0026#34;harisheoran\u0026#34; } Output Variables # To get values after resource is successfully created or modified\noutput \u0026#34;website_address\u0026#34; { value = aws_s3_bucket.main_s3_bucket.website_endpoint } How to pass values to input variables? # Many ways to do this, but standard way is using .tfvars file Define input variable and provide values of those variable in tfvars and use it.\nterraform apply -var-file=”prod.tfvars” Multi Region # Create SSL/TLS certificate using ACM in North Vergina Region.\nAttach an alias to each Provide block and attach refrence to resources while creating them. # Configure the AWS Provider Region 1 provider \u0026#34;aws\u0026#34; { region = \u0026#34;ap-south-1\u0026#34; alias = \u0026#34;indian_region\u0026#34; } provider \u0026#34;aws\u0026#34; { region = \u0026#34;us-east-1\u0026#34; alias = \u0026#34;north_v_region\u0026#34; } module \u0026#34;s3\u0026#34; { source = \u0026#34;./modules/s3\u0026#34; bucket_name = var.bucket_name providers = { aws = aws.indian_region } } module \u0026#34;acm\u0026#34; { source = \u0026#34;./modules/acm\u0026#34; bucket_name = var.bucket_name providers = { aws = aws.north_v_region } } Secret Manager: Hashicorp Vault # Setup on Vault\nVault has role similiar to IAM Role.\nSo, create a App Role so that terraform can access the vault, first create the policy and attach the policy to role. Now, get the secret key and id (similiar to AWD secret key value)\nConditionals in Terraform # Why do we need to use conditionals? As I bought domain from another provider than AWS, so I have to create first Hosted zone and add Nameservers in my Hostinger Domain management, so that route 53 has permission to manage the DNS records for me.\ncount = var.is_zone ? 0 : 1 first_run is a boolean variable If var.is_zone is true, set count to 0 (don\u0026rsquo;t create any instances). If var.is_zone is false, set count to 1 (create one resource). Depends # One module dependencies on another, like CDN requires S3 bucket.\ndepends_on = [ module.route53_hosted_zone, module.route53_hosted_zone, module.s3, module.cert] Errors # Since you have a module call which is using the count meta-argument, that means you will have to either reference a specific index of the module (because it becomes a list). acm_cert_arn = module.cert[0].acm_cert_arn Terraform Workspaces? # Blogs refrence\nCreating and validating ACM certificates with Terraform Remote Backend for state\n","date":"15 March 2024","externalUrl":null,"permalink":"/projects/aws_resume_iac/","section":"Projects","summary":"","title":"Infrastructure as Code: Terraform-Powered Production-Grade deployment on AWS","type":"projects"},{"content":"","date":"15 March 2024","externalUrl":null,"permalink":"/categories/terraform/","section":"Categories","summary":"","title":"Terraform","type":"categories"},{"content":"","date":"19 January 2024","externalUrl":null,"permalink":"/tags/argocd/","section":"Tags","summary":"","title":"Argocd","type":"tags"},{"content":"","date":"19 January 2024","externalUrl":null,"permalink":"/categories/devops/","section":"Categories","summary":"","title":"DevOps","type":"categories"},{"content":"In this project, we\u0026rsquo;ll dive into the practical world of code deployment, exploring the seamless integration of development and operations that powers modern software delivery. It focus on Kubernetes deployment, from transforming code into containerized brilliance to orchestrating the containers.\nHere are project requirements.\nProject Requirements Source Code:\nSimple web application (static HTML or basic web server using a language of your choice). Version Control:\nSet up a Git repository. Commit your initial code. Continuous Integration (CI):\nChoose a CI tool (Jenkins, GitLab CI, or GitHub Actions). Configure the CI pipeline to trigger on each commit. Include steps to: Build your web application. Run automated tests (if applicable). Create a build artifact. Containerization:\nUse Docker to containerize your application. Push Docker image to a container registry (Docker Hub, Google Container Registry, etc.). Continuous Deployment (CD):\nSet up a simple CD pipeline. Deploy Docker container to a Kubernetes cluster. Choose deployment target: local cluster (Minikube/kind), cloud service (EKS, GKE, AKS), or self-managed cluster. Monitoring (Optional):\nIntegrate basic monitoring (e.g., Prometheus, logging). Follow these steps, explaining your choices and demonstrating the flow. This assignment will showcase your skills in setting up a CI/CD pipeline and deploying applications to Kubernetes. If you have any questions along the way, feel free to ask!\nProject Workflow # Source Code # harisheoran/WebServer-Docker-CI-CD-k8s Full DevOps Project, includes containerization of GO web server, CI using Github Actions, CD using ArgoCD and deployed on Kubernetes. JavaScript 3 0 Project Demo # Kubernetes Basics # K8s Architecture # Difference between Container, Pods, Deployment # What is the difference between Container v/s Pod v/s Deployment?\nRead here 👇 pic.twitter.com/nHSuyAO0Hm\n\u0026mdash; Harish Sheoran (@harisheoran) January 23, 2024 Let\u0026rsquo;s create a Pod K8s Service # Assume there is no service in k8s, we created 3 pods using deployment, kubeproxy assign Dynamic IP address to each pod, Let\u0026rsquo;s say if one pod goes down, Replica set controller is actively looking for desired state and actual state and it automatically creates the pod, but this time it have a new IP address, so now user is now unable to connect to that pod as this pod have new IP address.\nService comes into picture here and instead of assigning IP address to each pod, it assign IP to service and it access Pods via Service.\nNow we are using Service and all request are coming to its IP address, wait, what if one of the pod goes down, and Replica Set Controller creates the pod again, now how service identifies the newly created pod.\nService Discovery Mechanism This Mechanism assign labels to each pod and identifies them using selectors and even if they goes down, Replica Set Controller auto heal with same label again.\nExposing to external world using Service\nCluster IP NodePort Load Balancer First build a simple Web server # You can choose the language of your choice for this task, I am using the Go language.\nCreate a main.go file. touch main.go go mod init webserver package main import ( \u0026#34;fmt\u0026#34; \u0026#34;net/http\u0026#34; ) func main() { fs := http.FileServer(http.Dir(\u0026#34;./static\u0026#34;)) http.Handle(\u0026#34;/\u0026#34;, fs) port := 3000 fmt.Printf(\u0026#34;Server is listening on 3000\u0026#34;) err := http.ListenAndServe(fmt.Sprintf(\u0026#34;:%d\u0026#34;, port), nil) if err != nil { fmt.Printf(\u0026#34;Error: \u0026#34;, err) } } This server is just serving a index.html and main.js file.\nCreate a static directory for html and js files. Here is index.html Here is main.js mkdir static touch index.html main.js Run the website go run main.go Go to 127.0.0.1:3000 Containerize your application using Docker # We use docker to containerize our application, let\u0026rsquo;s write the Dockerfile.\nWe are going to use the Multistage Docker build or Distroless Images.\nMust read about Multi-Stage Docker Build of GO web server\n# Build stage FROM golang:latest as build WORKDIR /app COPY go.mod ./ RUN go mod download # Copy the necessary files of the application COPY main.go . COPY static ./static # build the binary of app named \u0026#34;main\u0026#34; RUN CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -o main . # 2nd stage, run the binary of application FROM scratch # Copy \u0026#34;main\u0026#34; binary and static folder into current dir COPY --from=build /app/main . COPY --from=build /app/static ./static EXPOSE 3000 # execute the binary CMD [\u0026#34;./main\u0026#34;] Build the Docker Image docker build -t memesoftheday-image . Build \u0026amp; run the container and check the application is running or not docker container run -d -p 4000:3000 --name memes-container memesoftheday-image Version Control # Use Git to control the version of source code and push it to GitHub.\nContinuous Integration (CI) # If you are a beginner to CI, so understand the problem first\nThe Image of the application is present on your machine and let\u0026rsquo;s say we are deploying our application on AWS or Azure - we have to manually build and run the image every time we make changes to verify it. and have to manually push the image to AWS or Azure. We don\u0026rsquo;t like manual things, we are Engineers, Let\u0026rsquo;s Automate it using the CI pipeline.\nWhat\u0026rsquo;s the solution? # We build a Ci pipeline which uses our Dockerfile from the repository to build the image and push it to a registry.\nso, you don\u0026rsquo;t have to worry about building the image each time you make changes. and it uses CI server computing to build the image. There are so many CI tools like Jenkins, Github Actions and many more. I think GitHub Actions is the right choice for our requirements.\nUnderstand the workflow of the CI pipeline # Check out the Github repository. Build the docker image using Dockerfile. Login to Docker Hub. Push the image to Docker Hub. Read about using Github Actions\nManage Secrtets\nGithub Actions Officials(https://docs.github.com/en/actions/publishing-packages/publishing-docker-images)\nCreate a .github/workflows directory.\nmkdir -p .github/workflows \u0026amp;\u0026amp; cd .github/workflows It triggers the build-on code push in main branch.\nname: Create and build docker image to Docker Hub on: push: branches: [\u0026#34;main\u0026#34;] env: REGISTRY: docker.io IMAGE_NAME: \u0026#34;memesoftheday-image\u0026#34; jobs: push_to_registry: name: Push Image to Docker Hub runs-on: ubuntu-latest steps: - name: checkout the repository uses: actions/checkout@v4 - name: Login to Docker Hub uses: docker/login-action@f4ef78c080cd8ba55a85445d5b36e214a81df20a with: username: ${{secrets.DOCKERHUB_USERNAME}} password: ${{secrets.DOCKERHUB_PASSWORD}} - name: Extract metadata (tags, labels) for Docker id: meta uses: docker/metadata-action@9ec57ed1fcdbf14dcef7dfbe97b2010124a938b7 with: images: ${{secrets.DOCKERHUB_USERNAME}}/memesoftheday-image - name: Build \u0026amp; push the image uses: docker/build-push-action@3b5e8027fcad23fda98b2e3ac259d8d67585f671 with: context: . file: ./Dockerfile push: true tags: ${{ steps.meta.outputs.tags }} labels: ${{ steps.meta.outputs.labels }} Continuous Deployment (CD) # We want to reflect the changes we made in our code into the applications which we deploy on Kubernetes, which comes into the picture of the CD.\nWe are going to use the ArgoCD, so install the ArgoCD as an operator on our k8s cluster.\nCreate Kubernetes Cluster We use Minikuebe as a local k8s cluster Install Minikube\nStart Minikube cluster\nminikube start --driver=docker Install Kubectl\nInstall ArgoCD operator\nCreate a new Argo CD cluster with the default configuration using this manifest file.\nCreate a new directory for our manifest file\nmkdir k8s touch argocd.yml apiVersion: argoproj.io/v1alpha1 kind: ArgoCD metadata: name: example-argocd labels: example: basic spec: {} ArgoCD is created as a service, you can verify using\nkubectl get svc Now if you see that it is created as ClusterIP, which can access only within cluster, so, to access ArgoCD in our browser we have to change its type to NodePort .\nkubectl edit svc example-argocd-server change the type from Cluster IP to NodePort Check the service again, and check its port - 31080 kubectl get svc example-argocd-server NodePort 10.105.171.247 \u0026lt;none\u0026gt; 80:31080/TCP,443:32101/TCP 21h So, to access the argocd on the browser using node port, we need the IP of the cluster minikube ip Visit the URL - http://minikubeip:port In my case, it is 192.168.49.2:31080\nUsername is admin Password is store as secret kubectl get secrets you\u0026rsquo;ll get example-argocd-cluster, password is stored in this secret.\nkubectl edit secrets example-argocd-cluster Copy the password Minikube uses a simple encryption algorithm base64 echo dkxpeEEzeWVnYk53NEZrU1VtdW4xUlBhOHBoTWNyN0k= | base64 -d Login into argoCd Create Deployment and service for your application # Create Deployment.yml in k8s directory # This file describes a Kubernetes Deployment, which manages a set of pods. apiVersion: apps/v1 # Specifies the Kubernetes API version used for this resource. kind: Deployment # Indicates that this is a Deployment resource, used to manage application deployments. metadata: name: memesoftheday-deployment # Assigns a name to the Deployment, in this case, \u0026#34;memesoftheday-deployment.\u0026#34; labels: app: memesoftheday # Labels are used to organize and select resources. This label is named \u0026#34;app\u0026#34; and set to \u0026#34;memesoftheday.\u0026#34; spec: replicas: 3 # Specifies that the desired number of replicas (pods) is 3. selector: matchLabels: app: memesoftheday # Defines how the Deployment identifies which pods to manage based on labels. template: metadata: labels: app: memesoftheday # Labels assigned to the pods created by this Deployment. spec: containers: - name: memesostheday-container # Specifies the name of the container within the pod. image: harisheoran/memesoftheday-image:main # Specifies the Docker image to be used for the container, pulled from \u0026#34;harisheoran/memesoftheday-image\u0026#34; with the \u0026#34;main\u0026#34; tag. ports: - containerPort: 3000 # Specifies that the container within the pod will listen on port 3000. imagePullPolicy: If Create service.yml in the same directory # This file describes a Kubernetes Service, which exposes pods to the network. apiVersion: v1 # Specifies the Kubernetes API version used for this resource. kind: Service # Indicates that this is a Service resource, used to expose pods. metadata: name: memesoftheday-service # Assigns a name to the Service, in this case, \u0026#34;memesoftheday-service.\u0026#34; spec: type: NodePort # Specifies that the Service should be of type NodePort, making it accessible externally on each node. selector: app: memesoftheday # Specifies the labels used to select pods that this service will route traffic to. In this case, pods with the label \u0026#34;app: memesoftheday.\u0026#34; ports: - port: 80 # Specifies that the Service should be accessible on port 80 externally. targetPort: 3000 # Specifies that incoming traffic on port 80 should be forwarded to the pods on port 3000. nodePort: 30007 # Specifies a static port on each node (accessible externally) where the Service will be available, set to 30007. and create a new app and fill the details accordingly to your repo in the below format. Wait for it, will create the pods of our application defined in our deployment. We successfully deployed the application pods. # You can check the pods using the command\nkubectl get pods -o wide Check the port of pods kubectl get svc visit your minikube ip with nodeport, in my case, it is - http://192.168.49.2:30007/\nMonitoring # To monitor our k8s cluster, we use Prometheus and Grafana.\nInstall Prometheus using Helm charts helm repo add Prometheus-community https://prometheus-community.github.io/helm-charts helm repo update helm install prometheus prometheus-community/prometheus Now, it will create a prometheus pod and service but it is available as Cluster IP, to access the Prometheus, we need to change it service to NodePort.\nkubectl expose service prometheus-server --type=NodePort --target-port=9090 --name=prometheus-server-ext Check it port using the command kubectl get svc Visit this port in browser with minikube ip\nTo view these logs in graph form, we use Grafana\nInstall Grafana using Helm Chart helm repo add grafana https://grafana.github.io/helm-charts helm repo update helm install grafana grafana/grafana Expose Grafana service as NodePort to access in browser. kubectl expose service grafana --type=NodePort --target-port=3000 --name=grafana-ext Check the grafana Nodeport using command and visit the port with minikube ip kubectl get svc Click on to create your first Data source. Add Data source to Prometheus Enter the Prometheus IP address Go back to Home and click on Create DashBoard \u0026gt; Import DashBoard \u0026gt; Enter 3662 ID to import the template of Dashboard. Our Grafana Dashboard That\u0026rsquo;s it, our project is done, if you have any questions about this project or if stuck at any point, feel free to ping me. # ","date":"19 January 2024","externalUrl":null,"permalink":"/projects/opsdock/","section":"Projects","summary":"","title":"End to End DevOps Deployment: From CICD to orchestration with logging \u0026 monitoring","type":"projects"},{"content":"","date":"19 January 2024","externalUrl":null,"permalink":"/tags/github-actions/","section":"Tags","summary":"","title":"Github Actions","type":"tags"},{"content":"","date":"19 January 2024","externalUrl":null,"permalink":"/tags/grafana/","section":"Tags","summary":"","title":"Grafana","type":"tags"},{"content":"","date":"19 January 2024","externalUrl":null,"permalink":"/tags/prometheus/","section":"Tags","summary":"","title":"Prometheus","type":"tags"},{"content":"Deploying a three tier web application for production.\nWeb application code belongs to this source\nDeployment Architecture # Let\u0026rsquo;s start Deploying in Production # We are going to deploy 3 Tier Web application on AWS in production.\n(I used a spare domain)\nAlert!! For Production with security\nTech Stack\nFrontend: React Backend: Node JS Database MongoDB We are deploying this web app in a containerized environment with Production strategies.\nContainerization # Write Backend Dockerfile # We are using Node js in our backend.\nWarning! Do not pass any secret variables at build time, we\u0026rsquo;ll provide them at runtime.\nFROM node:20.16.0-bookworm-slim ENV NODE_ENV=production WORKDIR /app COPY --chown=node:node package.json . RUN npm install --only=production COPY --chown=node:node . . EXPOSE 8000 USER node CMD [ \u0026#34;node\u0026#34;, \u0026#34;index.js\u0026#34; ] Use Node JS image with a specific tag in prod, don\u0026rsquo;t use the latest tag, as if the latest tag is new and if a new bug is found in the latest image, it might break your prod.\nWe are not using an alpine image, as it contains musl implementation of the C library, if your app\u0026rsquo;s operations need C libraries then it might behave differently, use booksworm image with slim, it is lightweight.\nuse ENV NODE_ENV=production, it\u0026rsquo;ll make install dependencies for prod\nSet a working directory, which will contain our code.\nFirst, copy only the package JSON file, Why?\nDocker uses a layering mechanism to build and when we do rebuild the image, it checks the changes in each layer if no changes, it uses the last build layer and decreases the build time and dependencies don\u0026rsquo;t change that much, so we put it first. --chown=node:node copy using only a specific user, it won\u0026rsquo;t give root permission, helps in security as root user has all privileges.\nInstall production dependencies only, we don\u0026rsquo;t dev dependencies in prod.\nNow, copy all the code to the current directory.\nAssign a new user, by default, it uses the root user which has all privileges so it can cause security issues.\nStart the server, don\u0026rsquo;t use npm run in prod.\nWrite Frontend Dockerfile # In frontend we are using react, so we can build files which we can then serve in web server.\nDon\u0026rsquo;t use npm run dev in production, it\u0026rsquo;s bad practice, instead build the static files and serve.\nWe are using Multi stage docker Build, In the First stage, we\u0026rsquo;ll build our react app and in 2nd stage, we use those build files and server them in a web server, and in 2nd stage we don\u0026rsquo;t need NodeJS, we copy only build files, it\u0026rsquo;ll reduce our image size significantly.\nRead more on Multi Stage Build on my blog.\nFROM node:20.16.0-bookworm-slim AS build WORKDIR /app COPY --chown=node:node package.json . RUN npm install --only=production ENV PATH /app/node_modules/.bin:$PATH COPY --chown=node:node . . RUN npm run build FROM nginx:1.27.0-alpine3.19 AS prod COPY --from=build /app/dist /usr/share/nginx/html RUN rm /etc/nginx/conf.d/default.conf #COPY ./nginx.conf /etc/nginx/conf.d/default.conf EXPOSE 3000 ENTRYPOINT [\u0026#34;nginx\u0026#34;, \u0026#34;-g\u0026#34;, \u0026#34;daemon off;\u0026#34;] RUN npm run build after Installing prod dependencies, build the react app, in my case, I am using vite.\nIn 2nd stage, use the nginx web server alpine image, here this is lightweight and we don\u0026rsquo;t need c libraries, so we can use alpine images.\nCopy all the build files from 1st stage into 2nd stage.\nRemove the default nginx config file, we\u0026rsquo;ll provide our own at runtime.\nStart the nginx server.\nPut unnecessary files that we don\u0026rsquo;t need to run in a container like .git, .gitignore, and local builds in the .dockerignore file.\nCreating and Starting the Containers. # Secrets # We are going to pass secrets at runtime, so it won\u0026rsquo;t be available to the end user.\nWays to Pass secrets at runtime.\nenv_file to provide the secret file. Volumes: We can use bind mounts volume to mount(kind of sync) to the container directory. Networks # Our user needs to access only Fronted, not the database directly, so we are going to use one more security layer by putting our containers in different networks.\nWe create 2 networks,\nFrontend Network: In this Network, we\u0026rsquo;ll run the frontend and backend container. Backend Network: In this Network, we\u0026rsquo;ll run the backend and database container. This creates a layer of separation between our frontend and database, and the backend can talk to both the database and frontend, it enhances security.\nDatabase Docker Compose # Mongo Db container requires 2 secrets, Database username and password, store them in a .env file, (Don\u0026rsquo;t push it to GitHub, put it in gitignore)\ndatabase: image: mongo ports: - \u0026#34;27017:27017\u0026#34; env_file: - ./env/.env.db volumes: - my-db:/data/db healthcheck: test: echo \u0026#39;db.runCommand(\u0026#34;ping\u0026#34;).ok\u0026#39; | mongosh localhost:27017/test --quiet interval: 3s retries: 5 start_period: 30s networks: - backend deploy: restart_policy: condition: on-failure First, Image, ports, env_file. Volume to store the database data on the host machine, this creates a volume on the Host machine. Why? Containers are ephemeral in nature means they can be destroyed and created, so any data in that container will be lost. Using Volumes, we can save that data in the host machine and if the container is destroyed, start the new container with the same volume.\nWarning! Using a database in a container with volumes is not the way for prod, what if the host machine goes down?\nPut a simple health check by pinging the db. Put this in the Backend Network. Restart policy in case the container crashes. Backend Container # backend: image: harisheoran/backend_img:master ports: - \u0026#34;8000:8000\u0026#34; env_file: - ./env/.env depends_on: - database networks: - backend - frontend deploy: restart_policy: condition: on-failure Define image, env file, ports, and networks. depends_onThis creates an order of starting the containers, so before starting the backend container it\u0026rsquo;ll check if the database container is created and running, this prevents failure as if we start the backend with checking db, it won\u0026rsquo;t be connected to a DB, so prevents a failure. Frontend Container # frontend: image: harisheoran/frontend_img:master ports: - \u0026#34;3000:3000\u0026#34; networks: - frontend deploy: restart_policy: condition: on-failure volumes: - ./nginx/nginx.conf:/etc/nginx/conf.d/nginx.conf:ro Define image, ports, network\nMounting secrets using Bind Mount Volume (It basically syncs the Host directories/files with the Containers, it does not create a volume on the Host like the database Volume.)\nrodefines that files are with readonly permissions, which helps in security, if somebody gets access to our container, then they are unable to write this file.\nDefine Volume \u0026amp; Networks # volumes: my-db: networks: frontend: driver: bridge backend: driver: bridge Define Volume for Database. Define Networks, Docker use bridge Networks internally. Branch Protection # Use the GitHub branch protection rule to protect the prod branch, in our case,\nDon\u0026rsquo;t push to prod branch directly without raising PR. Can\u0026rsquo;t merge before test completions. Here my prod branch is master branch.\nCI Pipelines # We are using GitHub Actions for creating pipelines. Pipeline Secrets Management # Use GitHub Action secrets to store pipeline secrets. 1. Code scan Pipeline # We do not want to break our production with base code, so we can test the code before merging to main, so far we don\u0026rsquo;t have Tests for our code, so we scan our code for any vulnerabilities with codeql\nview the pipeline\n2. Main Prod CI pipeline # We want to build the code once merged to the main branch push the image to the registry and scan the image.\nSteps\nDefine the triggering event, in our case, we want to run the pipeline on code merged/push to the main branch Define on which OS, we are going to run our jobs. Define required Permissions. Check out the repo on the machine. Setup buildx to build the image, this new buildx helps in caching the image also by layer, Our layering strategy will pay off here by reducing our build time. Login to the Image registry, we are using Docker Hub. (Save the Login credentials as secret in Github Actions Secrets) Extract Metadata and labels to provide tags to the image, in our case, we are using a branch name as a tag, we can also use tag or custom tags. Build and Push the image using the Dockerfile, here we are also using GitHub API for caching for faster builds. Scan the Image using trivy. Tiggers the CD Pipeline. View Pipelines YAML or All Pipelines files\n3. CD Pipeline # Docker doesn\u0026rsquo;t have a dedicated tool like Argo to implement GitFlow\nSteps\nSSH into the AWS EC2 using an SSH action\nGenerate SSH Keys on your local machine. Copy content of public key to AWS EC2 authorized_keys file. Save content of private key in GitHub actions secret. Save other secrets - Host username, password, ssh port. Ran the docker commands to update the containers.\nPull the latest Image Run the compose file delete old containers and images. Create Infrastructure using Terraform # Crete infra on AWS, see the code here\nSet Up Nginx as Reverse Proxy with SSL certificate # Setup Nginx as a revere proxy server { server_name sparrowbit.co; location / { proxy_pass http://127.0.0.1:3000; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; } } Set up DNS records in your Domain provider Dashboard, use A record to save the IPv4 address of the ec2 instance. Install SSL certificate using Certbot, you can do this using AWS itself, {read my blog](https://harisheoran.github.io/projects/aws_resume/) to do in AWS. Main Challenges # Passing application secrets in React Container at Runtime\nREASON: Application static files are being served in the Nginx server, and It does not have a Node js environment, so can\u0026rsquo;t load the env secrets. Solution: Don\u0026rsquo;t store any sensitive info in the frontend site, store env in the backend. GitHub Actions Caching to reduce build time.\nSolution: Use buildx to speed up the build process. Use GitHub API cache to use a cache of the previous build process. Yet to Implement # Server config management using Ansible. Testing of Prod Infra. Orchestration system for scaling. Secure VPC + NACL + Nat Gateways + Private Subnets. Branching Strategy Releasing Pipelines. Image Scan Pipeline. ","date":"16 January 2024","externalUrl":null,"permalink":"/projects/three_tier_mern_deployment/","section":"Projects","summary":"","title":"3 Tier MERN web app Deployment for Production","type":"projects"},{"content":"","date":"16 January 2024","externalUrl":null,"permalink":"/tags/cicd/","section":"Tags","summary":"","title":"CICD","type":"tags"},{"content":"","date":"20 November 2023","externalUrl":null,"permalink":"/tags/ansible/","section":"Tags","summary":"","title":"Ansible","type":"tags"},{"content":"Micro Blog website deployed on AWS\nSource Code # harisheoran/programmer-server Micro Blog website for developers deployed on AWS EC2 in containerized environement. EJS 0 0 Workflow # Demo # Tech Stack and tools # Application # Node JS Express JS Mongo DB Bootstrap 5 DevOps # Docker Docker Compose VCS (Git \u0026amp; Github) Terraform Ansible Nginx AWS EC2 Jenkins Next Goals # Contianer Orchestration using k8s Website Setup # Prerequisite # Install \u0026amp; Configure AWS CLI\nInstall Docker and start its service\nInstall Terraform \u0026amp; Ansible\nCreate two env files\n.env NODE_ENV=prod MONGO_USER= MONGO_PASSWORD= SESSION_SECRET= .env2 MONGO_INITDB_ROOT_USERNAME= MONGO_INITDB_ROOT_PASSWORD= Inventory file Development Setup # How to run the project locally # Install npm packages npm install Build the Docker image and run the container # sudo docker-compose -f docker-compose.yml -f docker-compose.dev.yml up -d You can access the application on localhost:3000 and hit the /blogserver .\nProduction Setup # Infrastructure as Code # Run the Terrafrom File to launch 2 ec2 instances # cd terraform terraform init terraform apply It\u0026rsquo;ll output 2 IP addresses, copy them(You\u0026rsquo;ll need to configure our ec2 instances).\nConfiguration Management # For Pipeline # Using Ansible\nFirst, we configure the Pipeline server cd ansible Paste the IP address in inventory.ini\nansible-playbook -i ./inventory.ini ./pipeline.yml It\u0026rsquo;ll install Jenkins on EC2 and start serving on port 8080.\nNow install basic plugins and some additional plugins like\nGithub Integration For main server # Now, configure the main ec2 instance, execute the following command\nansible-playbook -i ./inventory.ini ./run_app.yml Web application will start serving at public ip address of ec2 instance.\nCI Setup # After creating EC2 instance, copy the IP address of ec2 instance to inventory.ini file (in ansible directory)\n[pipeline] \u0026lt;paste ip address here\u0026gt; ansible_ssh_private_key_file=/home/harisheoran/my-key.pem ansible_user=ubuntu Set your path of ansible_ssh_private_key (your aws key pair value file)\n","date":"20 November 2023","externalUrl":null,"permalink":"/projects/containerized_nodejs_mongodb_app_deployment/","section":"Projects","summary":"","title":"Containerized Web App Deployment on AWS","type":"projects"},{"content":"","date":"20 November 2023","externalUrl":null,"permalink":"/tags/docker-compose/","section":"Tags","summary":"","title":"Docker-Compose","type":"tags"},{"content":"","date":"20 November 2023","externalUrl":null,"permalink":"/tags/ec2/","section":"Tags","summary":"","title":"EC2","type":"tags"},{"content":"","date":"20 November 2023","externalUrl":null,"permalink":"/tags/jenkins/","section":"Tags","summary":"","title":"Jenkins","type":"tags"},{"content":"","date":"20 November 2023","externalUrl":null,"permalink":"/tags/nginx/","section":"Tags","summary":"","title":"Nginx","type":"tags"},{"content":"","date":"20 November 2023","externalUrl":null,"permalink":"/tags/terraform/","section":"Tags","summary":"","title":"Terraform","type":"tags"},{"content":"","date":"15 November 2023","externalUrl":null,"permalink":"/categories/blog/","section":"Categories","summary":"","title":"Blog","type":"categories"},{"content":" Containerization using Multi-Stage Docker Build # Multi-Stage Docker Build or Distroless Image is a concept of creating image in multiple stages.\nWe build a docker image of a simple go web server using the concept of multi-stage docker build.\nfunc main() { http.HandleFunc(\u0026#34;/\u0026#34;, func(w http.ResponseWriter, r *http.Request) { fmt.Fprintf(w, \u0026#34;Go Web Server\u0026#34;) }) port := 3000 fmt.Printf(\u0026#34;Server is listening on 3000\u0026#34;) err := http.ListenAndServe(fmt.Sprintf(\u0026#34;:%d\u0026#34;, port), nil) if err != nil { fmt.Printf(\u0026#34;Error: \u0026#34;, err) } } The Evolution: From 815MB to 6.71MB 🚀 # A simple Go web server docker image is 815 MB which is quite large.\nEnter the magic of multi-stage Docker builds. By breaking down the build process into distinct stages, we\u0026rsquo;ve successfully trimmed down our image to a sleek 6.71MB. That\u0026rsquo;s a jaw-dropping 99.17% reduction in size, making our deployment faster, more scalable, and resource-efficient. How do we build the image? We divide this image into 2 stages,\nBuild Stage Production Stage Build Stage # In this stage, we take golang as Base image and install all the dependencies it needs and create a single binary and we don\u0026rsquo;t need to define our entrypoint here.\nFROM golang:1.21 as build WORKDIR /app COPY go.mod ./ RUN go mod download COPY *.go ./ RUN CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -o main . Production Stage # In this stage we take scratch as Base image, it is a bare minimum image to run our binary.\nCopy the binary build from build stage and execute it as an entrypoint.\nWe don\u0026rsquo;t need golang image in our production, we only need the executable binary.\nFROM scratch COPY --from=build /app/main . EXPOSE 3000 CMD [ \u0026#34;./main\u0026#34; ] The Command That Did the Magic ✨ # Build the executable binary.\nRUN CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -o main . Breaking down the command:\nCGO_ENABLED=0: Disabling CGO for a statically linked binary. GOOS=linux: Targeting a Linux operating system for compatibility. go build -a -installsuffix cgo -o main .: Building a binary optimized for minimal size. Benefits Beyond Size: Security Boost 🛡️ # Reducing image size isn\u0026rsquo;t just about efficiency – it\u0026rsquo;s also about security. Our final image contains only the essentials needed to run the Go application, minimizing attack surfaces and potential vulnerabilities. With fewer dependencies and a smaller footprint, our web server is now more resilient and secure in a production environment.\nSource Code # harisheoran/Multi-stage-docker-build-GO-web-server Multi Stage Docker Build or Distroless Image of a simple GO web server. Go 0 0 ","date":"15 November 2023","externalUrl":null,"permalink":"/projects/multi_stage_docker/","section":"Projects","summary":"","title":"Multi-Stage Docker Build","type":"projects"},{"content":"","date":"10 November 2023","externalUrl":"https://harisheoran.hashnode.dev/docker-and-nodejs-a-love-story-with-containers-and-code-no-drama-just-dockerama","permalink":"/projects/nodejs_docker/","section":"Projects","summary":"","title":"Containerization of NodeJS app","type":"projects"},{"content":"","date":"28 September 2023","externalUrl":null,"permalink":"/categories/android/","section":"Categories","summary":"","title":"Android","type":"categories"},{"content":"","date":"28 September 2023","externalUrl":null,"permalink":"/tags/arduino/","section":"Tags","summary":"","title":"Arduino","type":"tags"},{"content":"🗳️ Electra Touch: Polypurpose Polling System\nWe are thrilled to share a glimpse of our current project – a versatile polling system designed for a variety of applications, featuring cutting-edge technology integration. Here\u0026rsquo;s a concise overview:\nProgress so far: # Components: # Arduino ESP8266 Fingerprint Sensor (R307) Purpose: # Our primary objective is to create a comprehensive prototype for election scenarios.\nKey Features: # 1️⃣ User Registration: # Users are required to register themselves. Their registration data is securely uploaded to Firebase Database. 2️⃣ Android App: # An Android app has been developed. Only individuals with admin access can access and utilize the app. 3️⃣ Election Process: # During the election, users must verify their registration status. If registered, they can cast their votes for their preferred party. 4️⃣ Vote Tracking: # Vote counts are meticulously recorded in Firebase Database. 5️⃣ Election Results: # The final election results are dynamically displayed on the Android app. Source Code # Android App # harisheoran/Electra-Touch Kotlin 0 0 Team Members # Maaz Khan Ridhi Bansal Richa Mathur me ","date":"28 September 2023","externalUrl":null,"permalink":"/projects/electratouch/","section":"Projects","summary":"","title":"Electra Touch","type":"projects"},{"content":"","date":"28 September 2023","externalUrl":null,"permalink":"/tags/esp8266/","section":"Tags","summary":"","title":"ESP8266","type":"tags"},{"content":"","date":"28 September 2023","externalUrl":null,"permalink":"/categories/iot/","section":"Categories","summary":"","title":"Iot","type":"categories"},{"content":"","date":"28 September 2023","externalUrl":null,"permalink":"/tags/jetpack-compose/","section":"Tags","summary":"","title":"Jetpack Compose","type":"tags"},{"content":"","date":"28 September 2023","externalUrl":null,"permalink":"/tags/kotlin/","section":"Tags","summary":"","title":"Kotlin","type":"tags"},{"content":"","date":"28 September 2023","externalUrl":null,"permalink":"/tags/mvvm/","section":"Tags","summary":"","title":"MVVM","type":"tags"},{"content":"","date":"2 September 2023","externalUrl":null,"permalink":"/categories/ai/","section":"Categories","summary":"","title":"Ai","type":"categories"},{"content":"Talk to your pdfs using GPT.\nTech Stack # Python Open AI Langchain Source Code # harisheoran/PDF-Pulse Ask your PDF📄 anything. Python 0 1 Demo # ","date":"2 September 2023","externalUrl":null,"permalink":"/projects/gptfordoc/","section":"Projects","summary":"","title":"GPT for PDFs","type":"projects"},{"content":"I’m Harish, a software engineer, I build scalable and high-performance systems.\nWork Experience # Company Link Role Dates Location Sarvam Software Engineer April 2025 - Present Bengaluru, Karnataka, India, On-site Rava AI Software Engineer May 2024 - April 2025 Fully Remote, San Francisco, CA Technologies I Work With Resume PDF # View Resume PDF ","date":"12 July 2023","externalUrl":null,"permalink":"/resume/","section":"","summary":"","title":"About","type":"page"},{"content":" Email Me Feel free to reach out for collaborations, inquiries, or just to say hi!\nSend an Email Connect with Me Twitter LinkedIn GitHub YouTube ","date":"12 July 2023","externalUrl":null,"permalink":"/contact/","section":"","summary":"","title":"Contact","type":"page"},{"content":" App to browse characters, episodes and search details from Rick \u0026amp; Morty Tv show. Download app View source code on Github # harisheoran/rick-morty-collection Explore the characters of the Rick and Morty Tv show. Kotlin 1 0 Gallery # Demo # Read about its MVVM architecture here. # Tech Stack \u0026amp; Libraries # Kotlin Language MVVM Architecture Retrofit Picasso Epoxy RecyclerView Paging 3 Moshi API # The Rick and Morty API MVVM Architecture # architecture png Start from Network Layer\n1. Network Layer # 4 main components are\nService Interface It define how the app should talk to server interface RickAndMortyService { @GET(value = \u0026#34;character/\u0026#34;) suspend fun getCharactersPage(@Query(value = \u0026#34;page\u0026#34;) pageIndex: Int): Response\u0026lt;GetCharactersPageResponse\u0026gt; } Retrofit service Creates an implementation of the API endpoint defined by our interface and pass that service to our API client. object NetworkLayer { val moshi = Moshi.Builder() .add(KotlinJsonAdapterFactory()) .build() private val retrofit: Retrofit = Retrofit.Builder() .client(getLoggingHttpClient()) .baseUrl(\u0026#34;https://rickandmortyapi.com/api/\u0026#34;) .addConverterFactory(MoshiConverterFactory.create(moshi)) .build() private val rickAndMortyService: RickAndMortyService by lazy { retrofit.create(RickAndMortyService::class.java) } val apiClient = ApiClient(rickAndMortyService) } API Client\nThrough which our app will talk to server. It has the methods to talk to API and return the response inside our Network Error handling Network Error handling Class SimpleResponse\nA simple kind of wrapper class\nIt have 3 constructor as a parameter - Status, reponse from API client and exception\nIt has some properties to check weather the response is successfull or failed\n2. Repository Layer # We have 3 repository for fetching Character list( for home screen), Single Charater and Episodes.\nclass CharacterListRepository { suspend fun getCharactersPage(pageIndex: Int): GetCharactersPageResponse? { val request = NetworkLayer.apiClient.getCharactersPages(pageIndex) if (request.failed || !request.isSucceed) { return null } return request.body } } Sending request to api using Api client val request = NetworkLayer.apiClient.getCharactersPages(pageIndex) then checking the request and return accordingly.\n","date":"29 June 2023","externalUrl":null,"permalink":"/projects/rickmorty/","section":"Projects","summary":"","title":"Rick \u0026 Morty Collection App","type":"projects"},{"content":"","date":"15 April 2023","externalUrl":null,"permalink":"/tags/cloud-firestore/","section":"Tags","summary":"","title":"Cloud Firestore","type":"tags"},{"content":" App to get access of study materials for college students. # Tech Stacks # Kotlin Language Android Jetpack Components Firebase Cloud Firestore Lottie Android View on Website # Download pre-alpha version # View on Github # harisheoran/collegehub App to get access of study materials for college students. Kotlin 0 0 ","date":"15 April 2023","externalUrl":null,"permalink":"/projects/collegehub/","section":"Projects","summary":"","title":"College Hub App","type":"projects"},{"content":"","date":"15 April 2023","externalUrl":null,"permalink":"/tags/firebase/","section":"Tags","summary":"","title":"Firebase","type":"tags"},{"content":"","date":"29 October 2022","externalUrl":null,"permalink":"/tags/bootstrap/","section":"Tags","summary":"","title":"Bootstrap","type":"tags"},{"content":"","date":"29 October 2022","externalUrl":null,"permalink":"/tags/cloudflare/","section":"Tags","summary":"","title":"Cloudflare","type":"tags"},{"content":"","date":"29 October 2022","externalUrl":null,"permalink":"/tags/express-js/","section":"Tags","summary":"","title":"Express JS","type":"tags"},{"content":"","date":"29 October 2022","externalUrl":null,"permalink":"/tags/mongodb/","section":"Tags","summary":"","title":"MongoDB","type":"tags"},{"content":"","date":"29 October 2022","externalUrl":null,"permalink":"/tags/nodejs/","section":"Tags","summary":"","title":"NodeJs","type":"tags"},{"content":"","date":"29 October 2022","externalUrl":null,"permalink":"/categories/web/","section":"Categories","summary":"","title":"Web","type":"categories"},{"content":" Programmer Server # A place for programmers to discuss, share thier ideas and make connections.\nView on Github\nMemes of the day # Browse top memes of the day\nView on Website\nWeb Mini Projects # Weather Application Random Color Generator Pokemon Generator Weight Convertor View on Website\n","date":"29 October 2022","externalUrl":null,"permalink":"/projects/webprojects/","section":"Projects","summary":"","title":"Web Projects","type":"projects"},{"content":"","date":"29 May 2022","externalUrl":null,"permalink":"/categories/linux/","section":"Categories","summary":"","title":"Linux","type":"categories"},{"content":"","date":"29 May 2022","externalUrl":"https://harisheoran.hashnode.dev/processes-in-linux","permalink":"/projects/linux03/","section":"Projects","summary":"","title":"Processes in Linux","type":"projects"},{"content":"","date":"25 May 2022","externalUrl":"https://harisheoran.hashnode.dev/linux-system-administrator","permalink":"/projects/linux02/","section":"Projects","summary":"","title":"Linux System Administrator","type":"projects"},{"content":"","date":"14 May 2022","externalUrl":"https://harisheoran.hashnode.dev/the-first-step-toward-becoming-a-linux-wizard","permalink":"/projects/linux01/","section":"Projects","summary":"","title":"The first step toward becoming a Linux Wizard","type":"projects"},{"content":" # Building backend services in Go # Connecting to the Database # Need Database driver to translate the commands frm Go to SQL commands.\nCreate the Database connection pool, this is an important difference to understand. Go manages these connections as needed, automatically opening and closing connections to the database via the driver.\nUnderstanding the Database # Table have row and column\nColumn specifies the type of data like Name, age, email\nRow specfies a single record in the table.\nPrimary Key: Unique Identifier\nForiegn Key: A column that links one table to another by referencing a primary key.\nDesigning Database interaction # Create another directory to store database related code like models and related function, we\u0026rsquo;ll use this directory as a library.\nThe pkg directory is being used to hold ancillary non- application-specific code, which could potentially be reused. A database model which could be used by other applications in the future (like a command line interface application) fits the bill here pkg/models/models.go contains the database model\npkg/models/postgres/.go contain methods to execute commands on database like INSERT, DELETE, etc.\n","externalUrl":null,"permalink":"/notes/backend/","section":"Notes","summary":"","title":"","type":"notes"},{"content":"","externalUrl":null,"permalink":"/authors/","section":"Authors","summary":"","title":"Authors","type":"authors"},{"content":"","externalUrl":null,"permalink":"/blogs/","section":"Blogs","summary":"","title":"Blogs","type":"blogs"},{"content":"","externalUrl":null,"permalink":"/notes/","section":"Notes","summary":"","title":"Notes","type":"notes"},{"content":"","externalUrl":null,"permalink":"/series/","section":"Series","summary":"","title":"Series","type":"series"}]