Skip to content

🔌 FlowyML Plugin System

FlowyML features a powerful native plugin system that allows you to integrate with ANY ML tool — MLflow, Kubernetes, AWS S3, and more — without external framework dependencies.

🧩 One Framework, Any Infrastructure

Install only what you need. Write infrastructure-agnostic code. Deploy anywhere — from your laptop to Vertex AI.

🚀 Quick Start

Via CLI

1
2
3
4
5
6
7
8
# Initialize your stack
flowyml stack init --tracker mlflow --store gcs --orchestrator vertex_ai

# Install configured plugins
flowyml stack install

# Verify everything is ready
flowyml stack validate

Via Python

1
2
3
4
5
6
# Your code stays infrastructure-agnostic
from flowyml.plugins import start_run, log_metrics, save_model

start_run("training")
log_metrics({"accuracy": 0.95})
save_model(model, "classifier")  # Goes wherever config says

Via YAML (flowyml.yaml)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
plugins:
  experiment_tracker:
    type: mlflow
    tracking_uri: http://localhost:5000
  artifact_store:
    type: gcs
    bucket: my-ml-artifacts
  orchestrator:
    type: vertex_ai
    project: my-gcp-project
    region: us-central1

⚖️ Which Plugin Do I Need?

I want to... Plugin Type Recommended Plugin
Track experiments & metrics Experiment Tracker mlflow or wandb
Store artifacts in the cloud Artifact Store gcs (GCP) or s3 (AWS)
Run pipelines on cloud Orchestrator vertex_ai (GCP) or sagemaker (AWS)
Push Docker images Container Registry gcr (GCP) or ecr (AWS)
Register models Model Registry vertex_model_registry or sagemaker_model_registry
Deploy models as endpoints Model Deployer vertex_endpoint or sagemaker_endpoint

🎯 Key Benefits

📦 No Framework Overhead

Install only what you need. Each plugin brings only its direct dependencies (e.g., mlflow, boto3).

🔧 Three Ways to Configure

Use CLI commands, Python code, or YAML config files — whatever fits your workflow.

🔌 Auto-Discovery

Publish plugins as PyPI packages with entry points — FlowyML discovers and registers them automatically.

📦 Available Plugins

🔬 Experiment Trackers

Plugin Description Packages
mlflow MLflow tracking & model registry mlflow
wandb Weights & Biases tracking wandb
neptune Neptune.ai tracking neptune
tensorboard TensorBoard visualization tensorboard

💾 Artifact Stores

Plugin Description Packages
gcs Google Cloud Storage ✅ google-cloud-storage, gcsfs
s3 AWS S3 ✅ boto3, s3fs
azure_blob Azure Blob Storage azure-storage-blob, adlfs

🐳 Container Registries

Plugin Description Packages
gcr Google Artifact Registry ✅ google-cloud-artifact-registry
ecr AWS ECR ✅ boto3
acr Azure Container Registry azure-containerregistry

☁️ Orchestrators

Plugin Description Packages
vertex_ai Google Vertex AI Pipelines ✅ google-cloud-aiplatform
sagemaker AWS SageMaker Pipelines ✅ sagemaker
kubernetes Kubernetes kubernetes
airflow Apache Airflow apache-airflow

🏷️ Model Registries & Deployers

Plugin Description
vertex_model_registry Vertex AI Model Registry ✅
sagemaker_model_registry SageMaker Model Registry ✅
vertex_endpoint Vertex AI Endpoints ✅
sagemaker_endpoint SageMaker Endpoints ✅

🏗️ Architecture

The plugin system is built on three core components:

1️⃣ Plugin Registry

The central hub that manages all available plugins:

1
2
3
4
5
6
7
from flowyml.plugins import list_plugins, get_plugin

# List all available plugins
plugins = list_plugins()

# Get a specific plugin instance
tracker = get_plugin("mlflow", tracking_uri="http://localhost:5000")

2️⃣ Base Plugin Classes

Consistent interfaces for each plugin type:

1
2
3
4
5
6
from flowyml.plugins.base import (
    ExperimentTracker,
    ArtifactStorePlugin,
    OrchestratorPlugin,
    ContainerRegistryPlugin,
)

3️⃣ Stack Configuration

YAML-based infrastructure definitions with environment variable support:

1
2
3
4
5
plugins:
  artifact_store:
    type: s3
    bucket: ${AWS_BUCKET}
    region: ${AWS_REGION}

🎯 Entry Point Discovery

Plugins can register themselves automatically via Python entry points:

1
2
3
# In your plugin's pyproject.toml
[project.entry-points."flowyml.plugins"]
my_tracker = "my_package.plugins:MyCustomTracker"

FlowyML will auto-discover and register your component!

📦 Unified Plugin Management

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
# List available plugins
flowyml plugin list

# Install a plugin
flowyml plugin install mlflow

# Show plugin info
flowyml plugin info mlflow

# Search for plugins
flowyml plugin search kubernetes

📚 Deep Dive Guides

⚙️ Stack Configuration

Configure infrastructure in YAML. Multi-stack for dev/staging/prod.

→ Stack Configuration Guide

📦 Native Plugins

Complete guide to every built-in plugin with setup instructions.

→ Native Plugins Guide

🔧 Custom Plugins

Build your own plugins with the extensible base classes.

→ Creating Plugins Guide

🎯 Practical Examples

Copy-paste recipes for K8s, GCP, AWS, and hybrid stacks.

→ Practical Examples

🔀 Type-Based Routing

Auto-route Models, Datasets, and Metrics to the right stores.

→ Type Routing Guide

🏭 Production Tutorial

End-to-end: Docker, resources, stacks, and remote execution.

→ Production Pipeline Tutorial

🆘 Need Help?