OncoGlue – Frontline ETL for Pipeline Result Data Warehouse
Use Case:
In this repo we focus on the parquet output from nemo's tidywigits and tidydragen tools, and how to organise the data in a way that is easy to track and query in the Cloud Datalake setup fashion.
There are multiple modules for operating nemo in this repo. The operation stages are separate but interconnected, like episodes. Depending on your use case, they are organised so that you gradually work through the modules (or episodes) to achieve the desired result, or get the idea on how to tailor them to your operation needs. The episodes assume you have minimal or no Cloud Data Engineering experience, and gradually introduce the concepts and techniques as you follow the modules.
Try in the following order. You can stop at any point of the exploration.
- Episode 1: 01-nemo-operator (I am just curious how nemo works.)
- Episode 2: 02-nemo-query-examples (I want to query it over nemo output parquet files.)
- Episode 3: 03-nemo-batch-operator (I have a couple of datasets to process.)
- Episode 4: 04-nemo-schema-translator (I want to keep track of schema and translate to other formats like Trino/Presto/Athena query engine, dbt, YAML, CSV formats.)
- Episode 5: 05-nemo-schema-migrator (Run the nemo migration in bulk.)
Although the modules are written to be Cloud provider-agnostic, there are subtle references to AWS S3 buckets. You can map these to your own Cloud environment's storage counterpart.
Automation:
The aforementioned episodes are intentionally ad-hoc in nature, for exploration and educational purposes. We do have further development in progress for a more advanced approach, with an AWS-specific infrastructure setup to automate nemo operation in a routine workflow-run setting. This involves leveraging AWS Cloud mechanisms such as:
- EventBridge event subscription for Oncoanalyser pipeline completion events,
- Sending the workflow output tidying task request into an AWS SQS queue, and
- Invoking nemo deployed as an AWS Lambda function.
We will update this section once the initial automation development is ready. Please reach out to us for progress in the meantime, if any.
Create a Python virtual environment (any method) and install the dev toolchain requirements.
See README_DEV.md for more comprehensive setup details.
conda activate oncoglue
make install
make check