Skip to content
/ flyte Public
forked from flyteorg/flyte

Scalable and flexible workflow orchestration platform that seamlessly unifies data, ML and analytics stacks.

License

Notifications You must be signed in to change notification settings

jrobinAV/flyte

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Flyte and LF AI & Data Logo

Flyte

🏗️ 🚀 📈

Build. Deploy. Scale.

Flyte is an open-source orchestrator that facilitates building production-grade data and ML pipelines. It is built for scalability and reproducibility, leveraging Kubernetes as its underlying platform. With Flyte, user teams can construct pipelines using the Python SDK, and seamlessly deploy them on both cloud and on-premises environments, enabling distributed processing and efficient resource utilization.

Current Release Sandbox Status Test Status License OpenSSF Best Practices Flyte Helm Chart Flyte Twitter Flyte Slack


Getting started with Flyte


Features

We ship new features, bug fixes and performance improvements regularly. Read our release notes to stay updated.

🚀 Strongly typed interfaces: Validate your data at every step of the workflow by defining data guardrails using Flyte types.
🌐 Any language: Write code in any language using raw containers, or choose Python, Java, Scala or JavaScript SDKs to develop your Flyte workflows.
📊 Map tasks: Achieve parallel code execution with minimal configuration using map tasks.
🌟 Dynamic workflows: Build flexible and adaptable workflows that can change and evolve as needed, making it easier to respond to changing requirements.
🌳 Branching: Selectively execute branches of your workflow based on static or dynamic data produced by other tasks or input data.
📂 FlyteFile & FlyteDirectory: Transfer files and directories between local and cloud storage.
🗃️ Structured dataset: Convert dataframes between types and enforce column-level type checking using the abstract 2D representation provided by Structured Dataset.
🛡️ Recover from failures: Recover only the failed tasks.
🔁 Rerun a single task: Rerun workflows at the most granular level without modifying the previous state of a data/ML workflow.
🚦 Versioned workflows: Reproduce results and roll back to a previous workflow version any time.
🔍 Cache outputs: Cache task outputs by passing cache=True to the task decorator.
🚩 Intra-task checkpointing: Checkpoint progress within a task execution.
🌎 Multi-tenancy: Multiple users can share the same platform while maintaining their own distinct data and configurations.
Timeout: Define a timeout period, after which the task is marked as failure.
🔒 Immutability: Immutable executions help ensure reproducibility by preventing any changes to the state of an execution.
🧬 Data lineage: Track the movement and transformation of data throughout the lifecycle of your data and ML workflows.
📈 Data visualization: Visualize data, monitor models and view training history through plots.
🏭 Dev to prod: As simple as changing your domain from development or staging to production.
💸 Spot or preemptible instances: Schedule your workflows on spot instances by setting interruptible to True in the task decorator.
☁️ Cloud-native deployment: Deploy Flyte on AWS, GCP, Azure and other cloud services.
📅 Scheduling: Schedule your data and ML workflows to run at a specific time.
📢 Notifications: Stay informed about changes to your workflow's state by configuring notifications through Slack, PagerDuty or email.
⌛️ Timeline view: Evaluate the duration of each of your Flyte tasks and identify potential bottlenecks.
💨 GPU acceleration: Enable and control your tasks’ GPU demands by requesting resources in the task decorator.
🐳 Dependency isolation via containers: Maintain separate sets of dependencies for your tasks so no dependency conflicts arise.
🔀 Parallelism: Flyte tasks are inherently parallel to optimize resource consumption and improve performance.
💾 Allocate resources dynamically at the task level.
⏯️ Wait for external inputs before proceeding with the execution.

Quick start

If you want to try out Flyte, the easiest way to get started is by using the Flyte Hosted Sandbox, available at https://sandbox.union.ai/. It allows you to experiment with Flyte's capabilities without installing anything on your local machine.

However, if you prefer to install Flyte locally and run a workflow, our getting started guide is the best place to start. It provides step-by-step instructions on how to install Flyte locally and run your first workflow.

If you're unsure what a Flyte task and workflow are, be sure to check out our guides on both!

Deploy Flyte to production

The deployment guide provides useful information on how to self-host and manage Flyte.

Adopters

Join the likes of LinkedIn, Spotify, Freenome, Pachama, Gojek, and Woven Planet in adopting Flyte. For a full list of adopters and information on how to add your organization or project, please visit our ADOPTERS page.

Roadmap

Stay up to date with the latest project roadmap by checking out our live roadmap on GitHub.

Resources

📆 Weekly office hours: Live informal sessions with the Flyte team held every week. Book a 30-minute slot and get your questions answered.
👥 Biweekly community sync: A biweekly event where the Flyte team provides updates on the project and community members can share their progress and ask questions.
💬 Slack: Join the Flyte community on Slack to chat with other users, ask questions, and get help.
⚠️ GitHub: Check out the Flyte project on GitHub to file issues, contribute code, and stay up to date on the latest development.
📹 Youtube: Tune into panel discussions, customer success stories, community updates and feature deep dives.
📄 Blog: Here, you can find tutorials and feature deep dives to help you learn more about Flyte.
💡 RFCs: RFCs are used for proposing new ideas and features to improve Flyte. You can refer to them to stay updated on the latest developments and contribute to the growth of the platform.

How to contribute

There are many ways to get involved in Flyte, including:

We ❤️ our contributors

953358370901257597118271592984394310830562888811578108056158892184082372896568151852437936015277247631688870927777173452166474898565628986239450880580350323567779831223062603653394572570781228521086981538806453936213261742139142716548702118151754698929928166891937524160650517098894349093160909764361047153313394496745846792724868813302333125543414728265026554345877981179967195110820382072088200209243648304358781916389936511035227846541316881177744713605291042572538760119702581973368315508713994552110974449699333181059180421934393965996099861221940598349643165094906774758183378073226241168692913854113062123054334265487366561716500430375389975434806901302711670020642248654333860700576540257714896664777167782230138253138103843962286288302231071921218666947358951336529176984748488594201737393449803919853373103761951251051153481371700632614101430853310201242112692566122863313070236881763917309187268346585092599579670316716684317784075181406337416211245063235151789133317241193451861521126732473595086045365068105571814365977800661030084735036857538281015793222695370910587696225038146886843729944195813880715877000120668349100569684173517646399428256953021836330140230153451399253644905067987175488237515359319823956220965018921752004011377022223806652473994910430635319111751041527932781860964865843214593596231091927689726061478720441052299714054801004789826048414806211335881100597998483070069161722433367672539117366388192149921891175392747594611796986304786244025051659415869111421145677374901991034518429532638140089789272376163346030335921922904936594527875586965284497907255769683682749213393610438373332725875803281027207165266271052654053467642291774140143026771971263003151068153661568889937967114232404149968689343819010430512195344216461847646765941174730201354781518414436899736989112714477269703328455404056828107115357320474710217128419026265392850217804851253032755936685311318345091078931324225130002224465646729471486202804703533182199429103263313130875210976393625318519037413772114784116739793833655626670894866754748863181308139027713919823880001770392617851051119318734341672417330872419419101450450788414709014796400253353644124210778098651937811132451016594730508330361604708107592392948148147854100517263167280758916837146386887615811751844696692852310113228575146424054105082160113356122339164210247627390781496808752486650526115494996105856558378320753165133823807434118945041823250369170839638848382966451733751510090384230295799570725486791506754953085803627753472562559728279238894122166981983649055879913779418982821856758026092524

License

Flyte is available under the Apache License 2.0. Use it wisely.

About

Scalable and flexible workflow orchestration platform that seamlessly unifies data, ML and analytics stacks.

Resources

License

Stars

Watchers

Forks

Packages

No packages published

Languages

  • Python 38.0%
  • Shell 22.9%
  • Go 19.8%
  • Mustache 5.2%
  • Smarty 4.9%
  • Makefile 4.7%
  • Other 4.5%