Data Engineering

Airflow is an open-source platform to programmatically author, schedule, and monitor workflows.

A DAG Workflow is a process model that organizes tasks in a directed acyclic graph structure.

Dagster is an open-source data orchestrator for building and monitoring data pipelines.

Data Engineering involves designing and building systems for collecting, storing, and analyzing data.

A data pipeline is a series of processes that move and transform data from one system to another.

Delta Lake is an open-source storage layer that brings reliability and performance to data lakes.

Flyte is an open-source platform for building and managing data workflows and machine learning pipelines.

LakeFS is an open-source data versioning tool for managing data lakes with Git-like capabilities.

Tecton is a platform for managing and operationalizing machine learning features at scale.