

Project 2: Automated MLOps CI/CD Pipeline
Overview:
Developed a comprehensive, end-to-end MLOps pipeline with a zero-cost focus, designed to handle large-scale data and ensure reproducibility. The project integrates key MLOps tools like Apache Spark for data processing, MLflow for experiment tracking, DVC for data versioning, and GitHub Actions for CI/CD automation, all demonstrated within a local development and AWS Free Tier environment.
Project Description
This project focuses on building a modern and robust MLOps pipeline, targeting critical skills in scalability, reproducibility, and automation. Beginning with an extensive Exploratory Data Analysis (EDA), the pipeline implements scalable data engineering with Apache Spark. It features MLflow for tracking experiments and models, and DVC for versioning data and artifacts, ensuring a reproducible workflow.
A core achievement was establishing a complete CI/CD system using Docker and GitHub Actions, which automates the testing and deployment process. The project also explores container orchestration with Minikube and real-time data concepts with Apache Kafka. A key mandate was to achieve these advanced MLOps goals with a zero-cost strategy, leveraging local implementations and adhering strictly to the AWS Free Tier for cloud components. This project provides a holistic view of building and managing a production-ready ML system.
Problem Statement
While many organizations are adopting machine learning, a significant gap exists between a working model and a scalable, production-ready system. The challenges of ensuring reproducibility, managing different environments, and automating a model's entire lifecycle—from data preprocessing to deployment and monitoring—are critical hurdles. There is a pressing need for end-to-end, automated MLOps pipelines that can handle large datasets, maintain a consistent and tested environment, and enable reliable, continuous deployment.

Methodologies & Project Workflow
Level 0: Project Setup & Foundational Exploratory Data Analysis (EDA)
Objective: To set up the project environment, understand the dataset, and perform initial data exploration.
Key Methodologies & Steps:
Project Setup: Created a project structure with a
README.mdandMakefile, outlining project goals and commands for a reproducible setup.Virtual Environment Management: Used
venvto create an isolated Python environment andrequirements.txtto manage dependencies.Foundational EDA: Performed an initial analysis of the dataset using
pandasandmatplotlib, checking for missing values, and generating basic statistical summaries and visualizations.
Wrapped-up Writing: This level established a strong project foundation by ensuring a reproducible development environment and a clear understanding of the data's initial state through foundational EDA.
Level 1: Scalable Data Engineering & Versioning with Spark & DVC
Objective: To transition from local data processing to a scalable solution and ensure data is versioned for reproducibility.
Key Methodologies & Steps:
Data Engineering with Apache Spark: Implemented a scalable data processing pipeline using PySpark on a local Spark standalone cluster. This ensured that the project could handle large-scale datasets, a key requirement for any robust MLOps system.
Data Versioning with DVC: Utilized Data Version Control (DVC) to track the dataset and the processed data files. This ensured that the exact versions of data used for each experiment were saved, making the entire pipeline reproducible.
Wrapped-up Writing: This level established a strong foundation by demonstrating proficiency in data engineering with Apache Spark and leveraging DVC for data versioning, addressing the critical MLOps challenges of scalability and reproducibility.
Level 2 (Revised): Configurable & Tracked Model Training
Objective: To develop and train a machine learning model while ensuring that all experiments, models, and data artifacts are meticulously tracked for reproducibility and comparison.
Key Methodologies & Steps:
Configurable Training: Made the training process configurable using a
config.yamlfile, allowing for easy adjustment of hyperparameters without code changes.Model Training: Trained a machine learning model using the data prepared in Level 1.
Experiment Tracking with MLflow: Integrated MLflow to manage the entire ML lifecycle. This included logging model parameters, metrics, and artifacts for each training run. The MLflow tracking server provided a centralized hub to compare different experiments and choose the best-performing model.
Wrapped-up Writing: This level showcased the ability to manage the ML lifecycle professionally by making the training process configurable and integrating MLflow for robust experiment tracking.
Level 3: Containerized CI & Quality Gates
Objective: To automate the quality assurance process with continuous integration and ensure a consistent environment using containerization.
Key Methodologies & Steps:
Containerization with Docker: Packaged the ML application and its dependencies into a Docker image. This containerized approach provided a consistent and isolated environment for development, testing, and deployment.
Continuous Integration (CI) with GitHub Actions: Implemented a CI workflow using GitHub Actions to automatically test the entire pipeline whenever code was pushed to the repository. The workflow included linting, unit tests, and integration tests to ensure code quality and functionality.
Quality Gates: Configured the CI pipeline to enforce quality gates, such as a test failure preventing the code from being merged, thereby ensuring high code quality.
Wrapped-up Writing: This level demonstrated an end-to-end understanding of MLOps automation by building a robust CI pipeline with GitHub Actions and Docker, enabling consistent and reliable code validation.
Level 4: Automated Deployment of P1 API (CD with SAM/OIDC)
Objective: To automate the deployment of a previously developed ML API to a production environment using a secure and repeatable process.
Key Methodologies & Steps:
Continuous Deployment (CD) to AWS: The CI pipeline was extended to include a CD workflow that automatically deployed the updated Project 1 API to AWS Lambda and API Gateway.
Secure Deployment: Utilized OpenID Connect (OIDC) and an IAM Role to enable a secure, password-less authentication and deployment process from GitHub Actions to AWS.
Infrastructure as Code (IaC): Leveraged AWS Serverless Application Model (SAM) to define and deploy the infrastructure, ensuring the entire stack is repeatable and version-controlled.
Wrapped-up Writing: This level demonstrated a comprehensive approach to modern MLOps by implementing a secure and automated CD pipeline using GitHub Actions, OIDC, and AWS SAM, ensuring reliable and repeatable production deployments.
Level 5: Orchestration & Real-Time Concepts (Minikube & Kafka POC)
Objective: To explore and apply more advanced MLOps concepts, including container orchestration and real-time data streaming, in a practical, hands-on environment.
Key Methodologies & Steps:
Container Orchestration with Minikube: Gained practical experience with container orchestration by deploying a simple application to a local Minikube cluster. This provided foundational knowledge of Kubernetes concepts like pods, services, and deployments without the cost of a full-scale cloud cluster.
Real-Time Data Streaming with Apache Kafka: Explored the basics of real-time data pipelines by setting up a local Apache Kafka broker. This involved creating producers and consumers to simulate a real-time data flow, a crucial component for modern, event-driven ML systems.
Wrapped-up Writing: This level highlighted the ability to go beyond the basics, demonstrating a foundational understanding of container orchestration with Minikube and real-time data handling with Apache Kafka, showcasing readiness for more complex, enterprise-level MLOps challenges.
Level 6: Advanced MLOps - Monitoring, Optimization & Production Considerations
Objective: To consider and plan for the full production lifecycle of an ML model, including monitoring, optimization, and future enhancements.
Key Methodologies & Steps:
Production Readiness Review: Considered aspects like monitoring model performance drift, logging, alerting, and cost optimization.
Optimization: Explored potential optimizations for the inference pipeline, such as using optimized libraries or hardware.
Future Enhancements: Proposed a plan for future improvements, including moving to cloud-native services like AWS Step Functions and exploring a full Kubernetes deployment on a managed service like EKS.
Wrapped-up Writing: This final level demonstrated a forward-looking and holistic understanding of the MLOps lifecycle, moving beyond initial deployment to address critical long-term production concerns, monitoring, and strategic planning.
Key Results & Model Performance
After a rigorous experimentation process comparing multiple models on the full 6.3 million row dataset, the XGBoost model was selected as the champion for this use case. The final evaluation on the held-out test set demonstrated a strong ability to identify fraudulent transactions while maintaining efficiency.
Model | F1-Score | Training Time (CPU) | Justification for Selection |
|---|---|---|---|
XGBoost (Champion) | 0.888 | ~20 seconds | Selected. Achieved a significantly higher F1-score, which is the critical business metric for balancing precision and recall in fraud detection. The extremely fast training time also makes it more practical for rapid iteration and retraining. |
TensorFlow MLP | 0.837 | ~4.5 minutes | Not Selected. While achieving a near-perfect ROC AUC, its lower F1-score and higher computational cost made it the less optimal choice for this production scenario. |
Key Features & Technical Details
Scalable Data Processing: A configurable Apache Spark pipeline processes a large 6.3 million-row dataset, performing feature engineering and transformations to handle data at scale.
Comprehensive Version Control: Implemented a unified versioning strategy using Git for code, DVC for large data artifacts (with an AWS S3 remote), and MLflow for tracking experiments and model versions.
Multi-Model Experimentation: Developed a flexible training script that supports a "bake-off" between an XGBoost and a TensorFlow neural network, with all experiments meticulously tracked in MLflow.
Automated CI/CD Pipeline: Built a robust GitHub Actions pipeline that automatically performs code quality checks (flake8), runs unit tests (pytest), builds Docker images, and securely deploys the inference API to AWS Lambda via AWS SAM and a secure OpenID Connect (OIDC) connection.
Dual-Target Deployment: The same inference API is deployed to two modern environments: a serverless stack on AWS and a local container orchestration cluster managed by Minikube.
Real-Time Architecture POC: A Proof-of-Concept using Apache Kafka was developed to simulate a real-time stream of transactions, showcasing an understanding of event-driven architectures.
Multi-Faceted Monitoring Stack: A comprehensive monitoring solution was implemented, including Evidently AI for data and model performance drift, and a live stack with Prometheus and Grafana for real-time visualization of application metrics like API latency and request rates.
Technical Stack
Data Engineering: Python, Apache Spark, PySpark, Pandas
MLOps & Reproducibility: MLflow, DVC (Data Version Control)
Containerization & Orchestration: Docker, Docker Compose, Minikube
CI/CD & Cloud: GitHub Actions, AWS SAM (Serverless Application Model), OpenID Connect (OIDC), AWS Lambda, Amazon API Gateway, Amazon ECR
Real-Time Data: Apache Kafka
Project Management: Git, GitHub, Makefile,
venv
Challenges & Solutions
Throughout the project, several significant challenges were encountered and overcome, leading to a deeper understanding of MLOps and cloud deployment practices.
Challenge: Ensuring Reproducibility of the Entire ML Lifecycle
The Problem: Building an ML pipeline often leads to reproducibility issues. Differences in library versions, data states, and model artifacts can cause a pipeline to fail or produce different results when run on a different machine or environment, or even by a different person.
The Solution: A full reproducibility stack was implemented from the ground up. The entire application, including dependencies, was containerized with Docker to ensure a consistent environment. DVC was used to version both the raw and processed data, while MLflow tracked every experiment's parameters, metrics, and model artifacts. This comprehensive approach guaranteed that the entire workflow could be reliably reproduced and audited.
Challenge: Automating a Secure and Reliable Cloud Deployment
The Problem: Automating Continuous Deployment (CD) to AWS requires a secure method for GitHub Actions to authenticate and deploy resources without using long-lived credentials like IAM keys, which pose a significant security risk.
The Solution: A secure, password-less authentication flow was implemented using OpenID Connect (OIDC). A federated trust relationship was established between GitHub and AWS IAM, allowing the GitHub Actions workflow to assume a specific IAM Role with deployment permissions. This enabled a seamless and secure automated CD pipeline to AWS using AWS SAM, ensuring repeatable and version-controlled infrastructure deployment.
Challenge: Orchestrating Complex Multi-Container Environments Locally
The Problem: Developing a project with multiple services, such as a local Spark cluster, an MLflow server, and a database, can be complex to set up and manage manually. This makes it difficult for a development team to have a consistent and easily shareable environment.
The Solution: The entire local development environment was orchestrated using Docker Compose. This allowed all services to be defined in a single
docker-compose.ymlfile and started with a single command, providing a consistent, reproducible, and easy-to-manage multi-container environment for development and testing.
Challenge: Establishing a Robust Monitoring Stack for Production
The Problem: Production machine learning models can experience performance degradation (model drift) or data quality issues over time. Monitoring these issues is crucial for maintaining model reliability and business value.
The Solution: A multi-faceted monitoring solution was implemented. Evidently AI was used to generate automated reports on data and model performance drift. For application performance monitoring (APM), a live stack was set up using Prometheus for metrics collection and Grafana for real-time visualization of key metrics like API latency and request rates. This provided a comprehensive view of both the model's health and the application's performance.




Future Enhancements & Improvement
Full Cloud-Native Deployment: Transition all locally managed MLOps components (Spark, MLflow, Minikube, Kafka) to managed cloud services (e.g., AWS EMR for Spark, AWS MSK for Kafka, AWS EKS for Kubernetes orchestration), demonstrating large-scale production readiness.
Real-Time Model Monitoring & Alerting: Extend the local Evidently AI and Prometheus/Grafana setup to a cloud environment, implementing real-time data and model drift monitoring with proactive alerting mechanisms to ensure continuous model reliability.
Automated Model Retraining: Implement automated triggers for model retraining (e.g., based on detected data drift or performance degradation) to ensure the model continuously adapts to new patterns and maintains optimal performance.
Advanced CI/CD Strategies: Enhance the CI/CD pipeline to support multi-stage deployments (Dev/Staging/Production), incorporating advanced deployment patterns like blue/green or canary releases for zero-downtime updates.
Refined Security & Governance: Implement fine-grained IAM permissions based on the Principle of Least Privilege for all AWS resources, and explore advanced secrets management for sensitive credentials within the pipeline.
Inference Optimization: Apply advanced techniques such as model quantization (e.g., ONNX, TF Lite) or compilation (e.g., TensorRT) to optimize the deployed model's inference performance and further reduce latency.
More Works
(GQ® — 02)
©2024
More Works
(GQ® — 02)
©2024

Project 1: Real-Time Fraud Detection API
2025 - Completed
API Development
XGBoost
FastAPI
AWS Cloud
Serverless Deployment
SHAP

Project 1: Real-Time Fraud Detection API
2025 - Completed
API Development
XGBoost
FastAPI
AWS Cloud
Serverless Deployment
SHAP

Project 3: QA Bot with Multi-turn RAG for e-Commerce
2025 - Completed
RAG
Multilingual
FastAPI
Gemini API
Hugging Face
Pinecone

Project 3: QA Bot with Multi-turn RAG for e-Commerce
2025 - Completed
RAG
Multilingual
FastAPI
Gemini API
Hugging Face
Pinecone

Mini 1: OCR Text Extractor with GCP Vision API & Streamlit
2025 - Completed
GCP Vision API
Streamlit
OCR
Data Extraction
Cloud

Mini 1: OCR Text Extractor with GCP Vision API & Streamlit
2025 - Completed
GCP Vision API
Streamlit
OCR
Data Extraction
Cloud

Mini 2: Serverless ETL Pipeline on GCP for CSV Automated Processing to BigQuery
2025 - Completed
Data Engineering
ETL
Serverless
GCP
BigQuery
Cloud Functions

Mini 2: Serverless ETL Pipeline on GCP for CSV Automated Processing to BigQuery
2025 - Completed
Data Engineering
ETL
Serverless
GCP
BigQuery
Cloud Functions

Mini 3: Sentiment Analysis API
2025 - Completed
NLP
Sentiment Analysis
Hugging Face
Transformer
RoBERTa
Streamlit

Mini 3: Sentiment Analysis API
2025 - Completed
NLP
Sentiment Analysis
Hugging Face
Transformer
RoBERTa
Streamlit


Project 2: Automated MLOps CI/CD Pipeline
Overview:
Developed a comprehensive, end-to-end MLOps pipeline with a zero-cost focus, designed to handle large-scale data and ensure reproducibility. The project integrates key MLOps tools like Apache Spark for data processing, MLflow for experiment tracking, DVC for data versioning, and GitHub Actions for CI/CD automation, all demonstrated within a local development and AWS Free Tier environment.
Project Description
This project focuses on building a modern and robust MLOps pipeline, targeting critical skills in scalability, reproducibility, and automation. Beginning with an extensive Exploratory Data Analysis (EDA), the pipeline implements scalable data engineering with Apache Spark. It features MLflow for tracking experiments and models, and DVC for versioning data and artifacts, ensuring a reproducible workflow.
A core achievement was establishing a complete CI/CD system using Docker and GitHub Actions, which automates the testing and deployment process. The project also explores container orchestration with Minikube and real-time data concepts with Apache Kafka. A key mandate was to achieve these advanced MLOps goals with a zero-cost strategy, leveraging local implementations and adhering strictly to the AWS Free Tier for cloud components. This project provides a holistic view of building and managing a production-ready ML system.
Problem Statement
While many organizations are adopting machine learning, a significant gap exists between a working model and a scalable, production-ready system. The challenges of ensuring reproducibility, managing different environments, and automating a model's entire lifecycle—from data preprocessing to deployment and monitoring—are critical hurdles. There is a pressing need for end-to-end, automated MLOps pipelines that can handle large datasets, maintain a consistent and tested environment, and enable reliable, continuous deployment.

Methodologies & Project Workflow
Level 0: Project Setup & Foundational Exploratory Data Analysis (EDA)
Objective: To set up the project environment, understand the dataset, and perform initial data exploration.
Key Methodologies & Steps:
Project Setup: Created a project structure with a
README.mdandMakefile, outlining project goals and commands for a reproducible setup.Virtual Environment Management: Used
venvto create an isolated Python environment andrequirements.txtto manage dependencies.Foundational EDA: Performed an initial analysis of the dataset using
pandasandmatplotlib, checking for missing values, and generating basic statistical summaries and visualizations.
Wrapped-up Writing: This level established a strong project foundation by ensuring a reproducible development environment and a clear understanding of the data's initial state through foundational EDA.
Level 1: Scalable Data Engineering & Versioning with Spark & DVC
Objective: To transition from local data processing to a scalable solution and ensure data is versioned for reproducibility.
Key Methodologies & Steps:
Data Engineering with Apache Spark: Implemented a scalable data processing pipeline using PySpark on a local Spark standalone cluster. This ensured that the project could handle large-scale datasets, a key requirement for any robust MLOps system.
Data Versioning with DVC: Utilized Data Version Control (DVC) to track the dataset and the processed data files. This ensured that the exact versions of data used for each experiment were saved, making the entire pipeline reproducible.
Wrapped-up Writing: This level established a strong foundation by demonstrating proficiency in data engineering with Apache Spark and leveraging DVC for data versioning, addressing the critical MLOps challenges of scalability and reproducibility.
Level 2 (Revised): Configurable & Tracked Model Training
Objective: To develop and train a machine learning model while ensuring that all experiments, models, and data artifacts are meticulously tracked for reproducibility and comparison.
Key Methodologies & Steps:
Configurable Training: Made the training process configurable using a
config.yamlfile, allowing for easy adjustment of hyperparameters without code changes.Model Training: Trained a machine learning model using the data prepared in Level 1.
Experiment Tracking with MLflow: Integrated MLflow to manage the entire ML lifecycle. This included logging model parameters, metrics, and artifacts for each training run. The MLflow tracking server provided a centralized hub to compare different experiments and choose the best-performing model.
Wrapped-up Writing: This level showcased the ability to manage the ML lifecycle professionally by making the training process configurable and integrating MLflow for robust experiment tracking.
Level 3: Containerized CI & Quality Gates
Objective: To automate the quality assurance process with continuous integration and ensure a consistent environment using containerization.
Key Methodologies & Steps:
Containerization with Docker: Packaged the ML application and its dependencies into a Docker image. This containerized approach provided a consistent and isolated environment for development, testing, and deployment.
Continuous Integration (CI) with GitHub Actions: Implemented a CI workflow using GitHub Actions to automatically test the entire pipeline whenever code was pushed to the repository. The workflow included linting, unit tests, and integration tests to ensure code quality and functionality.
Quality Gates: Configured the CI pipeline to enforce quality gates, such as a test failure preventing the code from being merged, thereby ensuring high code quality.
Wrapped-up Writing: This level demonstrated an end-to-end understanding of MLOps automation by building a robust CI pipeline with GitHub Actions and Docker, enabling consistent and reliable code validation.
Level 4: Automated Deployment of P1 API (CD with SAM/OIDC)
Objective: To automate the deployment of a previously developed ML API to a production environment using a secure and repeatable process.
Key Methodologies & Steps:
Continuous Deployment (CD) to AWS: The CI pipeline was extended to include a CD workflow that automatically deployed the updated Project 1 API to AWS Lambda and API Gateway.
Secure Deployment: Utilized OpenID Connect (OIDC) and an IAM Role to enable a secure, password-less authentication and deployment process from GitHub Actions to AWS.
Infrastructure as Code (IaC): Leveraged AWS Serverless Application Model (SAM) to define and deploy the infrastructure, ensuring the entire stack is repeatable and version-controlled.
Wrapped-up Writing: This level demonstrated a comprehensive approach to modern MLOps by implementing a secure and automated CD pipeline using GitHub Actions, OIDC, and AWS SAM, ensuring reliable and repeatable production deployments.
Level 5: Orchestration & Real-Time Concepts (Minikube & Kafka POC)
Objective: To explore and apply more advanced MLOps concepts, including container orchestration and real-time data streaming, in a practical, hands-on environment.
Key Methodologies & Steps:
Container Orchestration with Minikube: Gained practical experience with container orchestration by deploying a simple application to a local Minikube cluster. This provided foundational knowledge of Kubernetes concepts like pods, services, and deployments without the cost of a full-scale cloud cluster.
Real-Time Data Streaming with Apache Kafka: Explored the basics of real-time data pipelines by setting up a local Apache Kafka broker. This involved creating producers and consumers to simulate a real-time data flow, a crucial component for modern, event-driven ML systems.
Wrapped-up Writing: This level highlighted the ability to go beyond the basics, demonstrating a foundational understanding of container orchestration with Minikube and real-time data handling with Apache Kafka, showcasing readiness for more complex, enterprise-level MLOps challenges.
Level 6: Advanced MLOps - Monitoring, Optimization & Production Considerations
Objective: To consider and plan for the full production lifecycle of an ML model, including monitoring, optimization, and future enhancements.
Key Methodologies & Steps:
Production Readiness Review: Considered aspects like monitoring model performance drift, logging, alerting, and cost optimization.
Optimization: Explored potential optimizations for the inference pipeline, such as using optimized libraries or hardware.
Future Enhancements: Proposed a plan for future improvements, including moving to cloud-native services like AWS Step Functions and exploring a full Kubernetes deployment on a managed service like EKS.
Wrapped-up Writing: This final level demonstrated a forward-looking and holistic understanding of the MLOps lifecycle, moving beyond initial deployment to address critical long-term production concerns, monitoring, and strategic planning.
Key Results & Model Performance
After a rigorous experimentation process comparing multiple models on the full 6.3 million row dataset, the XGBoost model was selected as the champion for this use case. The final evaluation on the held-out test set demonstrated a strong ability to identify fraudulent transactions while maintaining efficiency.
Model | F1-Score | Training Time (CPU) | Justification for Selection |
|---|---|---|---|
XGBoost (Champion) | 0.888 | ~20 seconds | Selected. Achieved a significantly higher F1-score, which is the critical business metric for balancing precision and recall in fraud detection. The extremely fast training time also makes it more practical for rapid iteration and retraining. |
TensorFlow MLP | 0.837 | ~4.5 minutes | Not Selected. While achieving a near-perfect ROC AUC, its lower F1-score and higher computational cost made it the less optimal choice for this production scenario. |
Key Features & Technical Details
Scalable Data Processing: A configurable Apache Spark pipeline processes a large 6.3 million-row dataset, performing feature engineering and transformations to handle data at scale.
Comprehensive Version Control: Implemented a unified versioning strategy using Git for code, DVC for large data artifacts (with an AWS S3 remote), and MLflow for tracking experiments and model versions.
Multi-Model Experimentation: Developed a flexible training script that supports a "bake-off" between an XGBoost and a TensorFlow neural network, with all experiments meticulously tracked in MLflow.
Automated CI/CD Pipeline: Built a robust GitHub Actions pipeline that automatically performs code quality checks (flake8), runs unit tests (pytest), builds Docker images, and securely deploys the inference API to AWS Lambda via AWS SAM and a secure OpenID Connect (OIDC) connection.
Dual-Target Deployment: The same inference API is deployed to two modern environments: a serverless stack on AWS and a local container orchestration cluster managed by Minikube.
Real-Time Architecture POC: A Proof-of-Concept using Apache Kafka was developed to simulate a real-time stream of transactions, showcasing an understanding of event-driven architectures.
Multi-Faceted Monitoring Stack: A comprehensive monitoring solution was implemented, including Evidently AI for data and model performance drift, and a live stack with Prometheus and Grafana for real-time visualization of application metrics like API latency and request rates.
Technical Stack
Data Engineering: Python, Apache Spark, PySpark, Pandas
MLOps & Reproducibility: MLflow, DVC (Data Version Control)
Containerization & Orchestration: Docker, Docker Compose, Minikube
CI/CD & Cloud: GitHub Actions, AWS SAM (Serverless Application Model), OpenID Connect (OIDC), AWS Lambda, Amazon API Gateway, Amazon ECR
Real-Time Data: Apache Kafka
Project Management: Git, GitHub, Makefile,
venv
Challenges & Solutions
Throughout the project, several significant challenges were encountered and overcome, leading to a deeper understanding of MLOps and cloud deployment practices.
Challenge: Ensuring Reproducibility of the Entire ML Lifecycle
The Problem: Building an ML pipeline often leads to reproducibility issues. Differences in library versions, data states, and model artifacts can cause a pipeline to fail or produce different results when run on a different machine or environment, or even by a different person.
The Solution: A full reproducibility stack was implemented from the ground up. The entire application, including dependencies, was containerized with Docker to ensure a consistent environment. DVC was used to version both the raw and processed data, while MLflow tracked every experiment's parameters, metrics, and model artifacts. This comprehensive approach guaranteed that the entire workflow could be reliably reproduced and audited.
Challenge: Automating a Secure and Reliable Cloud Deployment
The Problem: Automating Continuous Deployment (CD) to AWS requires a secure method for GitHub Actions to authenticate and deploy resources without using long-lived credentials like IAM keys, which pose a significant security risk.
The Solution: A secure, password-less authentication flow was implemented using OpenID Connect (OIDC). A federated trust relationship was established between GitHub and AWS IAM, allowing the GitHub Actions workflow to assume a specific IAM Role with deployment permissions. This enabled a seamless and secure automated CD pipeline to AWS using AWS SAM, ensuring repeatable and version-controlled infrastructure deployment.
Challenge: Orchestrating Complex Multi-Container Environments Locally
The Problem: Developing a project with multiple services, such as a local Spark cluster, an MLflow server, and a database, can be complex to set up and manage manually. This makes it difficult for a development team to have a consistent and easily shareable environment.
The Solution: The entire local development environment was orchestrated using Docker Compose. This allowed all services to be defined in a single
docker-compose.ymlfile and started with a single command, providing a consistent, reproducible, and easy-to-manage multi-container environment for development and testing.
Challenge: Establishing a Robust Monitoring Stack for Production
The Problem: Production machine learning models can experience performance degradation (model drift) or data quality issues over time. Monitoring these issues is crucial for maintaining model reliability and business value.
The Solution: A multi-faceted monitoring solution was implemented. Evidently AI was used to generate automated reports on data and model performance drift. For application performance monitoring (APM), a live stack was set up using Prometheus for metrics collection and Grafana for real-time visualization of key metrics like API latency and request rates. This provided a comprehensive view of both the model's health and the application's performance.




Future Enhancements & Improvement
Full Cloud-Native Deployment: Transition all locally managed MLOps components (Spark, MLflow, Minikube, Kafka) to managed cloud services (e.g., AWS EMR for Spark, AWS MSK for Kafka, AWS EKS for Kubernetes orchestration), demonstrating large-scale production readiness.
Real-Time Model Monitoring & Alerting: Extend the local Evidently AI and Prometheus/Grafana setup to a cloud environment, implementing real-time data and model drift monitoring with proactive alerting mechanisms to ensure continuous model reliability.
Automated Model Retraining: Implement automated triggers for model retraining (e.g., based on detected data drift or performance degradation) to ensure the model continuously adapts to new patterns and maintains optimal performance.
Advanced CI/CD Strategies: Enhance the CI/CD pipeline to support multi-stage deployments (Dev/Staging/Production), incorporating advanced deployment patterns like blue/green or canary releases for zero-downtime updates.
Refined Security & Governance: Implement fine-grained IAM permissions based on the Principle of Least Privilege for all AWS resources, and explore advanced secrets management for sensitive credentials within the pipeline.
Inference Optimization: Apply advanced techniques such as model quantization (e.g., ONNX, TF Lite) or compilation (e.g., TensorRT) to optimize the deployed model's inference performance and further reduce latency.
More Works
(GQ® — 02)
©2024

Project 1: Real-Time Fraud Detection API
2025 - Completed
API Development
XGBoost
FastAPI
AWS Cloud
Serverless Deployment
SHAP

Project 3: QA Bot with Multi-turn RAG for e-Commerce
2025 - Completed
RAG
Multilingual
FastAPI
Gemini API
Hugging Face
Pinecone

Mini 1: OCR Text Extractor with GCP Vision API & Streamlit
2025 - Completed
GCP Vision API
Streamlit
OCR
Data Extraction
Cloud

Mini 2: Serverless ETL Pipeline on GCP for CSV Automated Processing to BigQuery
2025 - Completed
Data Engineering
ETL
Serverless
GCP
BigQuery
Cloud Functions

Mini 3: Sentiment Analysis API
2025 - Completed
NLP
Sentiment Analysis
Hugging Face
Transformer
RoBERTa
Streamlit


Project 2: Automated MLOps CI/CD Pipeline
Overview:
Developed a comprehensive, end-to-end MLOps pipeline with a zero-cost focus, designed to handle large-scale data and ensure reproducibility. The project integrates key MLOps tools like Apache Spark for data processing, MLflow for experiment tracking, DVC for data versioning, and GitHub Actions for CI/CD automation, all demonstrated within a local development and AWS Free Tier environment.
Project Description
This project focuses on building a modern and robust MLOps pipeline, targeting critical skills in scalability, reproducibility, and automation. Beginning with an extensive Exploratory Data Analysis (EDA), the pipeline implements scalable data engineering with Apache Spark. It features MLflow for tracking experiments and models, and DVC for versioning data and artifacts, ensuring a reproducible workflow.
A core achievement was establishing a complete CI/CD system using Docker and GitHub Actions, which automates the testing and deployment process. The project also explores container orchestration with Minikube and real-time data concepts with Apache Kafka. A key mandate was to achieve these advanced MLOps goals with a zero-cost strategy, leveraging local implementations and adhering strictly to the AWS Free Tier for cloud components. This project provides a holistic view of building and managing a production-ready ML system.
Problem Statement
While many organizations are adopting machine learning, a significant gap exists between a working model and a scalable, production-ready system. The challenges of ensuring reproducibility, managing different environments, and automating a model's entire lifecycle—from data preprocessing to deployment and monitoring—are critical hurdles. There is a pressing need for end-to-end, automated MLOps pipelines that can handle large datasets, maintain a consistent and tested environment, and enable reliable, continuous deployment.

Methodologies & Project Workflow
Level 0: Project Setup & Foundational Exploratory Data Analysis (EDA)
Objective: To set up the project environment, understand the dataset, and perform initial data exploration.
Key Methodologies & Steps:
Project Setup: Created a project structure with a
README.mdandMakefile, outlining project goals and commands for a reproducible setup.Virtual Environment Management: Used
venvto create an isolated Python environment andrequirements.txtto manage dependencies.Foundational EDA: Performed an initial analysis of the dataset using
pandasandmatplotlib, checking for missing values, and generating basic statistical summaries and visualizations.
Wrapped-up Writing: This level established a strong project foundation by ensuring a reproducible development environment and a clear understanding of the data's initial state through foundational EDA.
Level 1: Scalable Data Engineering & Versioning with Spark & DVC
Objective: To transition from local data processing to a scalable solution and ensure data is versioned for reproducibility.
Key Methodologies & Steps:
Data Engineering with Apache Spark: Implemented a scalable data processing pipeline using PySpark on a local Spark standalone cluster. This ensured that the project could handle large-scale datasets, a key requirement for any robust MLOps system.
Data Versioning with DVC: Utilized Data Version Control (DVC) to track the dataset and the processed data files. This ensured that the exact versions of data used for each experiment were saved, making the entire pipeline reproducible.
Wrapped-up Writing: This level established a strong foundation by demonstrating proficiency in data engineering with Apache Spark and leveraging DVC for data versioning, addressing the critical MLOps challenges of scalability and reproducibility.
Level 2 (Revised): Configurable & Tracked Model Training
Objective: To develop and train a machine learning model while ensuring that all experiments, models, and data artifacts are meticulously tracked for reproducibility and comparison.
Key Methodologies & Steps:
Configurable Training: Made the training process configurable using a
config.yamlfile, allowing for easy adjustment of hyperparameters without code changes.Model Training: Trained a machine learning model using the data prepared in Level 1.
Experiment Tracking with MLflow: Integrated MLflow to manage the entire ML lifecycle. This included logging model parameters, metrics, and artifacts for each training run. The MLflow tracking server provided a centralized hub to compare different experiments and choose the best-performing model.
Wrapped-up Writing: This level showcased the ability to manage the ML lifecycle professionally by making the training process configurable and integrating MLflow for robust experiment tracking.
Level 3: Containerized CI & Quality Gates
Objective: To automate the quality assurance process with continuous integration and ensure a consistent environment using containerization.
Key Methodologies & Steps:
Containerization with Docker: Packaged the ML application and its dependencies into a Docker image. This containerized approach provided a consistent and isolated environment for development, testing, and deployment.
Continuous Integration (CI) with GitHub Actions: Implemented a CI workflow using GitHub Actions to automatically test the entire pipeline whenever code was pushed to the repository. The workflow included linting, unit tests, and integration tests to ensure code quality and functionality.
Quality Gates: Configured the CI pipeline to enforce quality gates, such as a test failure preventing the code from being merged, thereby ensuring high code quality.
Wrapped-up Writing: This level demonstrated an end-to-end understanding of MLOps automation by building a robust CI pipeline with GitHub Actions and Docker, enabling consistent and reliable code validation.
Level 4: Automated Deployment of P1 API (CD with SAM/OIDC)
Objective: To automate the deployment of a previously developed ML API to a production environment using a secure and repeatable process.
Key Methodologies & Steps:
Continuous Deployment (CD) to AWS: The CI pipeline was extended to include a CD workflow that automatically deployed the updated Project 1 API to AWS Lambda and API Gateway.
Secure Deployment: Utilized OpenID Connect (OIDC) and an IAM Role to enable a secure, password-less authentication and deployment process from GitHub Actions to AWS.
Infrastructure as Code (IaC): Leveraged AWS Serverless Application Model (SAM) to define and deploy the infrastructure, ensuring the entire stack is repeatable and version-controlled.
Wrapped-up Writing: This level demonstrated a comprehensive approach to modern MLOps by implementing a secure and automated CD pipeline using GitHub Actions, OIDC, and AWS SAM, ensuring reliable and repeatable production deployments.
Level 5: Orchestration & Real-Time Concepts (Minikube & Kafka POC)
Objective: To explore and apply more advanced MLOps concepts, including container orchestration and real-time data streaming, in a practical, hands-on environment.
Key Methodologies & Steps:
Container Orchestration with Minikube: Gained practical experience with container orchestration by deploying a simple application to a local Minikube cluster. This provided foundational knowledge of Kubernetes concepts like pods, services, and deployments without the cost of a full-scale cloud cluster.
Real-Time Data Streaming with Apache Kafka: Explored the basics of real-time data pipelines by setting up a local Apache Kafka broker. This involved creating producers and consumers to simulate a real-time data flow, a crucial component for modern, event-driven ML systems.
Wrapped-up Writing: This level highlighted the ability to go beyond the basics, demonstrating a foundational understanding of container orchestration with Minikube and real-time data handling with Apache Kafka, showcasing readiness for more complex, enterprise-level MLOps challenges.
Level 6: Advanced MLOps - Monitoring, Optimization & Production Considerations
Objective: To consider and plan for the full production lifecycle of an ML model, including monitoring, optimization, and future enhancements.
Key Methodologies & Steps:
Production Readiness Review: Considered aspects like monitoring model performance drift, logging, alerting, and cost optimization.
Optimization: Explored potential optimizations for the inference pipeline, such as using optimized libraries or hardware.
Future Enhancements: Proposed a plan for future improvements, including moving to cloud-native services like AWS Step Functions and exploring a full Kubernetes deployment on a managed service like EKS.
Wrapped-up Writing: This final level demonstrated a forward-looking and holistic understanding of the MLOps lifecycle, moving beyond initial deployment to address critical long-term production concerns, monitoring, and strategic planning.
Key Results & Model Performance
After a rigorous experimentation process comparing multiple models on the full 6.3 million row dataset, the XGBoost model was selected as the champion for this use case. The final evaluation on the held-out test set demonstrated a strong ability to identify fraudulent transactions while maintaining efficiency.
Model | F1-Score | Training Time (CPU) | Justification for Selection |
|---|---|---|---|
XGBoost (Champion) | 0.888 | ~20 seconds | Selected. Achieved a significantly higher F1-score, which is the critical business metric for balancing precision and recall in fraud detection. The extremely fast training time also makes it more practical for rapid iteration and retraining. |
TensorFlow MLP | 0.837 | ~4.5 minutes | Not Selected. While achieving a near-perfect ROC AUC, its lower F1-score and higher computational cost made it the less optimal choice for this production scenario. |
Key Features & Technical Details
Scalable Data Processing: A configurable Apache Spark pipeline processes a large 6.3 million-row dataset, performing feature engineering and transformations to handle data at scale.
Comprehensive Version Control: Implemented a unified versioning strategy using Git for code, DVC for large data artifacts (with an AWS S3 remote), and MLflow for tracking experiments and model versions.
Multi-Model Experimentation: Developed a flexible training script that supports a "bake-off" between an XGBoost and a TensorFlow neural network, with all experiments meticulously tracked in MLflow.
Automated CI/CD Pipeline: Built a robust GitHub Actions pipeline that automatically performs code quality checks (flake8), runs unit tests (pytest), builds Docker images, and securely deploys the inference API to AWS Lambda via AWS SAM and a secure OpenID Connect (OIDC) connection.
Dual-Target Deployment: The same inference API is deployed to two modern environments: a serverless stack on AWS and a local container orchestration cluster managed by Minikube.
Real-Time Architecture POC: A Proof-of-Concept using Apache Kafka was developed to simulate a real-time stream of transactions, showcasing an understanding of event-driven architectures.
Multi-Faceted Monitoring Stack: A comprehensive monitoring solution was implemented, including Evidently AI for data and model performance drift, and a live stack with Prometheus and Grafana for real-time visualization of application metrics like API latency and request rates.
Technical Stack
Data Engineering: Python, Apache Spark, PySpark, Pandas
MLOps & Reproducibility: MLflow, DVC (Data Version Control)
Containerization & Orchestration: Docker, Docker Compose, Minikube
CI/CD & Cloud: GitHub Actions, AWS SAM (Serverless Application Model), OpenID Connect (OIDC), AWS Lambda, Amazon API Gateway, Amazon ECR
Real-Time Data: Apache Kafka
Project Management: Git, GitHub, Makefile,
venv
Challenges & Solutions
Throughout the project, several significant challenges were encountered and overcome, leading to a deeper understanding of MLOps and cloud deployment practices.
Challenge: Ensuring Reproducibility of the Entire ML Lifecycle
The Problem: Building an ML pipeline often leads to reproducibility issues. Differences in library versions, data states, and model artifacts can cause a pipeline to fail or produce different results when run on a different machine or environment, or even by a different person.
The Solution: A full reproducibility stack was implemented from the ground up. The entire application, including dependencies, was containerized with Docker to ensure a consistent environment. DVC was used to version both the raw and processed data, while MLflow tracked every experiment's parameters, metrics, and model artifacts. This comprehensive approach guaranteed that the entire workflow could be reliably reproduced and audited.
Challenge: Automating a Secure and Reliable Cloud Deployment
The Problem: Automating Continuous Deployment (CD) to AWS requires a secure method for GitHub Actions to authenticate and deploy resources without using long-lived credentials like IAM keys, which pose a significant security risk.
The Solution: A secure, password-less authentication flow was implemented using OpenID Connect (OIDC). A federated trust relationship was established between GitHub and AWS IAM, allowing the GitHub Actions workflow to assume a specific IAM Role with deployment permissions. This enabled a seamless and secure automated CD pipeline to AWS using AWS SAM, ensuring repeatable and version-controlled infrastructure deployment.
Challenge: Orchestrating Complex Multi-Container Environments Locally
The Problem: Developing a project with multiple services, such as a local Spark cluster, an MLflow server, and a database, can be complex to set up and manage manually. This makes it difficult for a development team to have a consistent and easily shareable environment.
The Solution: The entire local development environment was orchestrated using Docker Compose. This allowed all services to be defined in a single
docker-compose.ymlfile and started with a single command, providing a consistent, reproducible, and easy-to-manage multi-container environment for development and testing.
Challenge: Establishing a Robust Monitoring Stack for Production
The Problem: Production machine learning models can experience performance degradation (model drift) or data quality issues over time. Monitoring these issues is crucial for maintaining model reliability and business value.
The Solution: A multi-faceted monitoring solution was implemented. Evidently AI was used to generate automated reports on data and model performance drift. For application performance monitoring (APM), a live stack was set up using Prometheus for metrics collection and Grafana for real-time visualization of key metrics like API latency and request rates. This provided a comprehensive view of both the model's health and the application's performance.




Future Enhancements & Improvement
Full Cloud-Native Deployment: Transition all locally managed MLOps components (Spark, MLflow, Minikube, Kafka) to managed cloud services (e.g., AWS EMR for Spark, AWS MSK for Kafka, AWS EKS for Kubernetes orchestration), demonstrating large-scale production readiness.
Real-Time Model Monitoring & Alerting: Extend the local Evidently AI and Prometheus/Grafana setup to a cloud environment, implementing real-time data and model drift monitoring with proactive alerting mechanisms to ensure continuous model reliability.
Automated Model Retraining: Implement automated triggers for model retraining (e.g., based on detected data drift or performance degradation) to ensure the model continuously adapts to new patterns and maintains optimal performance.
Advanced CI/CD Strategies: Enhance the CI/CD pipeline to support multi-stage deployments (Dev/Staging/Production), incorporating advanced deployment patterns like blue/green or canary releases for zero-downtime updates.
Refined Security & Governance: Implement fine-grained IAM permissions based on the Principle of Least Privilege for all AWS resources, and explore advanced secrets management for sensitive credentials within the pipeline.
Inference Optimization: Apply advanced techniques such as model quantization (e.g., ONNX, TF Lite) or compilation (e.g., TensorRT) to optimize the deployed model's inference performance and further reduce latency.
More Works
©2024

Project 1: Real-Time Fraud Detection API
2025 - Completed
API Development
XGBoost
FastAPI
AWS Cloud
Serverless Deployment
SHAP

Project 3: QA Bot with Multi-turn RAG for e-Commerce
2025 - Completed
RAG
Multilingual
FastAPI
Gemini API
Hugging Face
Pinecone

Mini 1: OCR Text Extractor with GCP Vision API & Streamlit
2025 - Completed
GCP Vision API
Streamlit
OCR
Data Extraction
Cloud

Mini 2: Serverless ETL Pipeline on GCP for CSV Automated Processing to BigQuery
2025 - Completed
Data Engineering
ETL
Serverless
GCP
BigQuery
Cloud Functions

Mini 3: Sentiment Analysis API
2025 - Completed
NLP
Sentiment Analysis
Hugging Face
Transformer
RoBERTa
Streamlit