How we achieved 61% model accuracy with AWS Glue solutions

Challenge: Lack of real-time data streaming and operational efficiency

Our client faced a significant challenge in fine-tuning their machine learning models hosted on AWS SageMaker that processes load and shipment data. These ML models’ inability to seamlessly connect with real-time data stored in Azure SQL Managed Instance (SQL-MI) hindered overall operational efficiency. Overcoming this challenge was pivotal in elevating the client’s machine-learning capabilities, significantly enhancing their performance and effectiveness.

Solution: Real-time data accessibility and enhanced operational efficiency by leveraging AWS Glue

Our data architects implemented a comprehensive solution leveraging AWS Glue to improve data accessibility and optimize model performance.

Tech Stack

Here’s what we implemented:

Data migration:
We developed custom Glue Jobs to efficiently extract and transfer data from Azure SQL-MI to designated AWS S3 buckets. Additionally, automated workflows ensured seamless data transfer, eliminating manual intervention and potential delays.

Data integration:
Our team leverages Glue crawlers to automatically discover and catalog the migrated data within AWS S3. Further, custom Glue jobs were employed to perform data cleansing and transformation tasks, ensuring data consistency for ML models.

Automated data pipelines:
We built automated data pipelines utilizing Glue to ensure smooth data flow from AWS S3 to AWS SageMaker. This real-time data accessibility empowered ML models to learn and adapt quickly, leading to improved decision-making.

Serverless model refresh:
Our team implemented AWS Lambda functions to automatically initiate the refresh of ML models when changes occurred in AWS S3 objects. This ensured that the model seamlessly incorporated up-to-the-minute insights, resulting in optimal performance.

Benefits: 61% improved model accuracy and 82% cost savings

Our implementation leveraging AWS Glue yielded a multitude of benefits for the client:

• Enhanced operational efficiency through automated data pipelines and serverless model refresh.
• Significantly reduced model refresh time, ensuring agile and responsive ML models.
• Increase in model accuracy through real-time data accessibility, thereby improving decision-making on load and shipment optimization.
• Optimized AWS costs by eliminating the necessity for dedicated infrastructure, contributing to substantial cost savings.

Ready to elevate your ML model capabilities?

Our collaboration with MODE Global stands as a testament to the transformative power of AWS Glue solutions in addressing critical challenges within the logistics industry. By overcoming data accessibility hurdles and enhancing operational efficiency, we not only achieved substantial cost savings but also propelled MODE Global towards a future of enhanced accuracy and competitiveness.

Embark on a journey towards efficiency and precision in logistics. Connect with us today to explore how our expertise can redefine your capabilities and position you as an industry leader.