Skip to content

🏗️ Stack Architecture Guide

Overview

flowyml's stack system provides a flexible, modular architecture for running pipelines across different infrastructure environments. Stacks are composable collections of components that define where and how your pipelines execute.

Core Concepts

Stack Components

A Stack is composed of several components:

┌─────────────────────────────────────┐
│           flowyml Stack             │
├─────────────────────────────────────┤
│ ▸ Orchestrator (optional)           │
│   - Vertex AI, Kubeflow, Airflow   │
│                                     │
│ ▸ Executor                          │
│   - Local, Remote, Kubernetes      │
│                                     │
│ ▸ Artifact Store                    │
│   - Local FS, GCS, S3, Azure       │
│                                     │
│ ▸ Metadata Store                    │
│   - SQLite, PostgreSQL, MySQL      │
│                                     │
│ ▸ Container Registry (optional)     │
│   - GCR, ECR, Docker Hub           │
└─────────────────────────────────────┘

Component Types

1. Orchestrator

Manages pipeline workflow execution and scheduling.

  • Vertex AI: Google Cloud's managed ML platform
  • Kubeflow: Kubernetes-native ML workflows
  • Airflow: Workflow scheduling and monitoring
  • None: Direct execution (development)

2. Executor

Runs individual pipeline steps.

  • LocalExecutor: Runs steps in current process
  • RemoteExecutor: Submits to remote compute
  • KubernetesExecutor: Runs in K8s pods
  • VertexAIExecutor: Runs on Vertex AI

3. Artifact Store

Stores pipeline artifacts and outputs.

  • LocalArtifactStore: Local filesystem
  • GCSArtifactStore: Google Cloud Storage
  • S3ArtifactStore: Amazon S3
  • AzureBlobStore: Azure Blob Storage

4. Metadata Store

Tracks pipeline runs, metrics, and lineage.

  • SQLiteMetadataStore: Local SQLite database
  • PostgreSQLMetadataStore: PostgreSQL database
  • CloudSQLMetadataStore: Google Cloud SQL

5. Container Registry

Manages Docker images for containerized execution.

  • GCRContainerRegistry: Google Container Registry
  • ECRContainerRegistry: AWS Elastic Container Registry
  • DockerHubRegistry: Docker Hub

6. Model Registry (optional)

Versions, stages, and resolves trained models. Exposed on a live stack as stack.model_registry.

  • mlflow_registry: MLflow Model Registry
  • azureml_registry: Azure ML model registry
  • vertex_model_registry: Vertex AI Model Registry (GCP)
  • sagemaker_model_registry: SageMaker Model Registry (AWS)
  • Built-in SQL registry when none is configured

7. Model Deployer (optional)

Serves a registered model to an inference endpoint. Exposed on a live stack as stack.model_deployer and used by the deployment layer.

  • local_docker: docker run on the local machine
  • kubernetes / openshift: Deployment + Service + Ingress/Route
  • vertex_endpoint / sagemaker_endpoint / gcp_cloud_run: managed cloud endpoints

Serving Components: Model Registry & Model Deployer

The model registry and model deployer are first-class stack components, just like the orchestrator or artifact store. This is what lets you write train → register → promote → deploy → serve code that is identical across clouds — only the stack changes.

Flavors are registered and discovered

Every component is registered under a short flavor name in the ComponentRegistry. Built-in serving flavors are registered automatically the first time the registry is created; third-party flavors register themselves via the @register_component decorator or a flowyml.stack_components entry point.

from flowyml.stacks.plugins import get_component_registry

registry = get_component_registry()
registry.list_model_registries()   # ['mlflow_registry', 'azureml_registry', ...]
registry.list_model_deployers()    # ['local_docker', 'kubernetes', 'openshift', ...]

# Resolve a flavor to its class
cls = registry.get_model_deployer("openshift")

Register your own flavor:

from flowyml.stacks.plugins import register_component
from flowyml.deployment.base import BaseDeployer

@register_component(name="my_deployer")
class MyDeployer(BaseDeployer):
    ...

FlowyML routes the class into the right bucket automatically: subclasses of BaseDeployer / ModelDeployerPlugin become model deployers, and subclasses of ModelRegistryPlugin become model registries.

Hydrating from flowyml.yaml

When a stack config declares model_registry / model_deployer, StackConfig.to_stack() resolves each type: against the ComponentRegistry, forwards the remaining keys as constructor kwargs, and attaches the instances to the live stack:

stacks:
  openshift-prod:
    orchestrator: { type: azure_ml, ... }
    artifact_store: { type: azure_blob, ... }
    model_registry:
      type: azureml_registry
      subscription_id: ${AZURE_SUBSCRIPTION_ID}
      resource_group: ml-rg
      workspace_name: ml-ws
    model_deployer:
      type: openshift
      namespace: ml-prod
      registry_uri: registry.apps.example.com/ml

from flowyml.plugins.config import PluginConfig
from flowyml.plugins.stack_config import StackManager

manager = StackManager(PluginConfig("flowyml.yaml"))
stack = manager.get_stack("openshift-prod").to_stack()

stack.model_registry   # AzureMLModelRegistry instance
stack.model_deployer   # OpenShiftDeployer instance

The DeploymentService picks up stack.model_deployer automatically, so flowyml deploy and promote_if_better "just work" against whatever the active stack declares.

Governed, company-level stacks

The same model_deployer / model_registry live under spec.deployment in an enterprise StackDefinition, where a PolicyEngine can allow-list which flavors a team is permitted to use. See Enterprise Stacks.

Stack Types

Local Stack

For development and testing:

from flowyml.stacks import LocalStack

stack = LocalStack(
    name="local",
    artifact_path=".flowyml/artifacts",
    metadata_path=".flowyml/metadata.db"
)

Use Cases: - Local development - Unit testing - Prototyping - Small datasets

GCP Stack

For production on Google Cloud Platform:

from flowyml.stacks.gcp import GCPStack

stack = GCPStack(
    name="production",
    project_id="my-gcp-project",
    region="us-central1",
    bucket_name="my-artifacts",
    registry_uri="gcr.io/my-project"
)

Use Cases: - Production ML training - Large-scale data processing - Team collaboration - CI/CD pipelines

AWS Stack (Coming Soon)

from flowyml.stacks.aws import AWSStack

stack = AWSStack(
    name="aws-prod",
    region="us-east-1",
    s3_bucket="my-artifacts",
    ecr_registry="123456789.dkr.ecr.us-east-1.amazonaws.com"
)

Kubernetes Stack (Coming Soon)

from flowyml.stacks.k8s import KubernetesStack

stack = KubernetesStack(
    name="k8s-cluster",
    namespace="flowyml",
    storage_class="standard"
)

Automatic Component Usage

When you use a stack, all components are automatically used by the pipeline. This is the core concept of stacks - they encapsulate all infrastructure configuration.

How Stacks Work

  1. Create or select a stack with all components configured
  2. Attach stack to pipeline (via constructor or active stack)
  3. Pipeline automatically uses:
  4. Stack's orchestrator (e.g., VertexAIOrchestrator)
  5. Stack's executor (e.g., VertexAIExecutor)
  6. Stack's artifact store (e.g., GCSArtifactStore)
  7. Stack's metadata store (e.g., CloudSQLMetadataStore)
  8. Stack's container registry (e.g., GCRContainerRegistry)

Example: GCP Stack with Automatic Orchestrator

from flowyml import Pipeline
from flowyml.stacks.gcp import GCPStack

# Create GCP stack - includes VertexAIOrchestrator automatically
gcp_stack = GCPStack(
    name="production",
    project_id="my-gcp-project",
    region="us-central1",
    bucket_name="my-artifacts",
    service_account="my-sa@my-project.iam.gserviceaccount.com"
)

# Create pipeline with stack
pipeline = Pipeline("training_pipeline", stack=gcp_stack)
pipeline.add_step(train_model)
pipeline.add_step(evaluate_model)

# Run pipeline - automatically uses Vertex AI orchestrator!
# No need to specify orchestrator explicitly
result = pipeline.run()

Stack Priority in Pipeline.run()

When you call pipeline.run(), the orchestrator is determined in this order:

  1. Explicit orchestrator parameter (if provided) - highest priority
  2. Stack's orchestrator (if stack is set/active) - recommended
  3. Default LocalOrchestrator - fallback
# Option 1: Use stack's orchestrator (recommended)
pipeline = Pipeline("my_pipeline", stack=gcp_stack)
pipeline.run()  # Uses VertexAIOrchestrator from stack

# Option 2: Override orchestrator for this run only
pipeline.run(orchestrator=custom_orchestrator)  # Override stack's orchestrator

# Option 3: Use active stack
from flowyml.stacks.registry import set_active_stack
set_active_stack("gcp-production")
pipeline = Pipeline("my_pipeline")
pipeline.run()  # Uses orchestrator from active stack

Resource Configuration

Define compute resources for your pipelines:

from flowyml.stacks.components import ResourceConfig

# CPU-intensive workload
cpu_config = ResourceConfig(
    cpu="8",
    memory="32Gi",
    disk_size="100Gi"
)

# GPU workload
gpu_config = ResourceConfig(
    cpu="16",
    memory="64Gi",
    gpu="nvidia-tesla-v100",
    gpu_count=4,
    machine_type="n1-highmem-16"
)

# Memory-intensive workload
memory_config = ResourceConfig(
    cpu="32",
    memory="256Gi",
    machine_type="n1-megamem-96"
)

Docker Configuration

Containerize your pipelines:

from flowyml.stacks.components import DockerConfig

# Pre-built image
docker_config = DockerConfig(
    image="gcr.io/my-project/ml-pipeline:v1.0"
)

# Build from Dockerfile
docker_config = DockerConfig(
    dockerfile="./Dockerfile",
    build_context=".",
    build_args={"PYTHON_VERSION": "3.11"}
)

# Dynamic requirements
docker_config = DockerConfig(
    base_image="python:3.11-slim",
    requirements=[
        "tensorflow>=2.12.0",
        "pandas>=2.0.0"
    ],
    env_vars={
        "PYTHONUNBUFFERED": "1",
        "TF_CPP_MIN_LOG_LEVEL": "2"
    }
)

Stack Registry

Manage multiple stacks and switch seamlessly:

from flowyml.stacks.registry import StackRegistry

# Create registry
registry = StackRegistry()

# Register stacks
registry.register_stack(local_stack)
registry.register_stack(gcp_stack)
registry.register_stack(aws_stack)

# List available stacks
print(registry.list_stacks())
# ['local', 'gcp-prod', 'aws-prod']

# Switch stacks
registry.set_active_stack("local")      # Development
registry.set_active_stack("gcp-prod")   # Production

# Get active stack
active = registry.get_active_stack()

Stack Hydration from YAML ⚡NEW

The StackConfig.to_stack() method lets you hydrate a YAML-defined stack configuration into a live, fully-wired Stack object. This bridges the gap between declarative YAML configuration and runtime stack assembly.

How It Works

  1. Define stacks in flowyml.yaml with component configs
  2. StackManager parses the YAML into StackConfig objects
  3. StackConfig.to_stack() resolves each component via ComponentRegistry
  4. Returns a fully-wired Stack ready for pipeline execution

YAML Configuration

# flowyml.yaml
stacks:
  local:
    orchestrator: { type: local }
    artifact_store: { type: local, path: "./artifacts" }

  gcp-prod:
    orchestrator: { type: vertex_ai, project: my-gcp-project }
    artifact_store: { type: gcs, bucket: ml-artifacts }
    model_registry: { type: vertex_model_registry }
    model_deployer: { type: vertex_endpoints }
    experiment_tracker: { type: mlflow }
    artifact_routing:
      Model:   { store: gcs, register: true, deploy: true }
      Dataset: { store: gcs, path: "{run_id}/data/{step_name}" }
      Metrics: { log_to_tracker: true }

  aws-staging:
    orchestrator: { type: sagemaker, region: us-east-1 }
    artifact_store: { type: s3, bucket: staging-ml }
    model_registry: { type: sagemaker_model_registry }

active_stack: local

Hydrating a Stack

from flowyml.plugins.config import PluginConfig
from flowyml.plugins.stack_config import StackManager

# Load from YAML
config = PluginConfig("flowyml.yaml")
manager = StackManager(config)

# Hydrate local stack
local_stack = manager.get_stack("local").to_stack()
assert local_stack.orchestrator  # LocalOrchestrator
assert local_stack.artifact_store  # LocalArtifactStore

# Hydrate GCP stack
gcp_stack = manager.get_stack("gcp-prod").to_stack()
assert gcp_stack.orchestrator  # VertexAIOrchestrator

Artifact Routing

The hydrated stack carries artifact routing rules:

live = manager.get_stack("gcp-prod").to_stack()

# Routing config is attached to the live stack
print(live._artifact_routing.rules)  # {"Model": ArtifactRoutingRule(...), ...}
print(live._model_registry_config)   # {"type": "vertex_model_registry"}

Stack Switching

Use the context manager for temporary stack switching:

from flowyml.plugins.config import PluginConfig
from flowyml.plugins.stack_config import StackManager

config = PluginConfig("flowyml.yaml")
manager = StackManager(config)

with manager.use_stack("gcp-prod"):
    pipeline.run()  # Uses GCP stack
# Reverts to previous active stack

# Or set via environment variable
# FLOWYML_STACK=gcp-prod flowyml run pipeline.py

Using Stacks with Pipelines

Method 1: Direct Assignment

from flowyml import Pipeline
from flowyml.stacks.gcp import GCPStack

stack = GCPStack(...)
pipeline = Pipeline("my_pipeline", stack=stack)

Method 2: Global Registry

from flowyml import Pipeline
from flowyml.stacks.registry import set_active_stack

# Set active stack globally
set_active_stack("production")

# All new pipelines use active stack
pipeline = Pipeline("my_pipeline")

Method 3: Per-Run Override

pipeline = Pipeline("my_pipeline")

# Run locally
pipeline.run(stack=local_stack)

# Run on GCP
pipeline.run(stack=gcp_stack)

Best Practices

1. Environment-Based Configuration

import os
from flowyml.stacks import LocalStack
from flowyml.stacks.gcp import GCPStack

env = os.getenv("ENVIRONMENT", "local")

if env == "production":
    stack = GCPStack(...)
elif env == "staging":
    stack = GCPStack(..., bucket_name="staging-artifacts")
else:
    stack = LocalStack()

2. Configuration Files

# flowyml.yaml
stacks:
  local:
    type: local
    artifact_path: .flowyml/artifacts

  production:
    type: gcp
    project_id: ${GCP_PROJECT_ID}
    region: us-central1
    bucket_name: ${GCP_BUCKET}

3. Validation

# Always validate before production
stack.validate()

# Check configuration
print(stack.to_dict())

4. Cost Optimization

# Use preemptible instances for fault-tolerant workloads
ResourceConfig(
    machine_type="n1-standard-4",
    preemptible=True  # 80% cost savings
)

# Right-size resources
ResourceConfig(
    cpu="2",  # Start small
    memory="8Gi",
    autoscaling=True  # Scale as needed
)

Advanced Patterns

Multi-Cloud Setup

# GCP for training
train_stack = GCPStack(name="gcp-train", ...)

# AWS for inference
inference_stack = AWSStack(name="aws-inference", ...)

# Different pipelines, different clouds
training_pipeline.run(stack=train_stack)
inference_pipeline.run(stack=inference_stack)

Hybrid Execution

# Some steps local, some on cloud
@step(stack=local_stack)
def preprocess():
    ...

@step(stack=gcp_stack, resources=gpu_config)
def train():
    ...

@step(stack=local_stack)
def evaluate():
    ...

Troubleshooting

Common Issues

  1. Authentication Errors 

    gcloud auth login
gcloud auth application-default login

  2. Permission Denied

  3. Check service account roles

  4. Verify IAM permissions

  5. Resource Quota

  6. Request quota increase

  7. Use smaller machine types

  8. Image Not Found

  9. Push image to registry
  10. Check image URI format

Next Steps