AWS Native Multi Stage Data Pipeline Implementation
Executive Summary
Being a powerful player in the wellness & fitness niche, Naring Health handles multiple diet-related services under one roof. On the one hand, they evaluate DNA, microbiome, and metabolic reaction to particular meals or dietary practices to define a sufficient eating program to prevent or treat disease. On the other hand, they integrate epigenetic profiling and genetic information to identify disease mechanisms. Accordingly, such complex and precise procedures require efficient management to ensure the accuracy of the delivered outcomes, so the Matoffo team designed a powerful cloud solution that perfectly meets customer needs.
About the Customer
Naring Health is a US-based company that combines both Precision Nutrition and Multi Omics Diagnostic technologies in order to manage diseases efficiently. As a trusted healthcare provider, Naring Health possesses a proven track record for creating thriving healthcare service organizations, designing state-of-the-art technology solutions, driving visible results in clinical settings, and innovating on how their clients comprehend the connection between diet and illness. Simply stated, Naring Health proves that diet is not an adverse punishment but rather a tasty & healthy approach to nutrition that helps you cope with chronic health problems in an easy and pleasant manner.
Customer Challenge
There is no doubt that processing a large amount of data can be truly challenging, especially when it comes to valuable information that comes from medical devices. Therefore, Naring Health faced an urgent need to manage their data within a safe cloud environment quickly. In addition to time efficiency, the customer put an emphasis on value for money as well.
Bearing in mind key Naring Health’s challenges, we decided to create a pipeline in order to help Data Science and R&D departments with demonstrating analyte quantitation and disease state modeling lift from liquid chromatography-mass spectrometry data. Notably, this data pipeline should provide end-to-end processing of 500 raw mass spectrometry files within only 6 hours. Besides, we were supposed to enable data processing containers to work simultaneously in order to deliver the first results faster.
Why AWS
Naring Health requested to develop a complex yet functional solution that can boost overall business efficiency by handling multiple data processing activities at the same time. When it comes to time- and cost-efficiency, there is no better solution than AWS. After a thorough research and discovery phase, we reached a consensus on the cloud-native tools we should implement.
Thanks to AWS, Naring Health can obtain large-scale computing capacity more quickly and cheaply. What is more, the reasonable price of this solution will not affect its quality, while the company will undeniably benefit from a vast scope of AWS tools, such as security configuration controls or efficient data management.
Why Matoffo
As a proven cloud computing expert, the Matoffo team definitely possesses the necessary skills and knowledge to solve Naring Health’s issue. Furthermore, working with multiple clients from diversified industry verticals, we gained unique experience that helps us reliably cope with tasks of any type and complexity. As a result, we assisted Naring Health throughout the whole development process, from the first interview till the final deployment of the turnkey solution, in order to provide our client with a robust cloud-based solution that brings meaningful changes to society.
Matoffo Solution
First and foremost, we created a highly available and scalable architecture. As we moved through the development process, the main workflow was built on AWS ECS using AWS Step Functions to make the higher level of control and parallel executions. Concerning the solution’s current setup, it also includes a few AWS Lambda functions that manage parameters and trigger executions.
AWS Fargate was selected over AWS Lambda to run ETL code and deal with time-consuming processing, which could go out of AWS Lambda run time limits. In addition to all of the above, we implemented specific mechanisms to avoid a rerun of processed files, as well as data catalog and data pipeline processing reports. For infrastructure deployment, we implemented Terraform while Serverless was used for Lambda functions deployment.
To complete this solution, we used the following tech stack: AWS Lambda, Amazon ECS, AWS Fargate, AWS Step Functions, Amazon S3, Amazon DynamoDB, Terraform, and Serverless. The development team included three DevOps engineers responsible for different processes, including Amazon setup, high-quality codes for Lambda, and technical realization. In general, it took us about two months to deliver a ready-to-use AWS solution.
Business Value
There is no doubt that the solution is useless when it does not provide any value to the customer’s business. With this crucial warning in mind, Matoffo created an elegant yet functional AWS-powered solution that helps Naring Health with the following aspects:
– Automated Data Processing
While it typically takes a lot of time to process tons of data manually, our solution eliminates the need to spend valuable time on such activities.
– Raw Data Turned into Meaningful Insights
A robust tech stack supplemented by developers’ talented hands enabled us to transform a considerable set of raw data into relevant medical insights.
– Broader Opportunities
With our unique cloud-based project, data scientists gain more time to expand and promote their AI knowledge.
– Cost- and Time-Efficiency
Given the paramount importance of Naring Health’s pain points, Matoffo designed an AWS environment that saves both the client’s time and budget.
Client's Feedback
Despite the festive period, the Matoffo team delivered the solution on time while also meeting all customer requirements concerning price and functionality. What is more, we believe that a great project not only solves particular problems but also provides business with new opportunities, so we undoubtedly succeeded with this aspect. As a result of our joint work, both parties to the contract are satisfied with the outcome, so we would be glad to continue cooperation with Naring Health in the future.
OUR FOCUS IS
CLOUD SOLUTIONS
Our primary goal is to deliver value to our clients by resolving technical challenges and helping them achieve their objectives. We utilize cloud solutions as a powerful toolset to make this happen. With Matoffo expertise, companies can significantly reduce the time from idea to market and rapidly scale their digital business. Additionally, Matoffo enables seamless adoption of rapidly growing tech capabilities, transforming businesses to stay competitive in the market.