Skip to content

Metadata Stores API 🗄️

Metadata stores are the lineage backbone of FlowyML. They record every pipeline run, the status of each step, input/output artifact references, and timing information. FlowyML ships with a lightweight SQLite backend for local development and supports PostgreSQL and cloud-managed alternatives for production deployments. All backends implement the MetadataStore interface documented below.

Metadata Stores track pipeline runs, step status, and artifact lineage.

Base Metadata Store

Bases: ABC

Base class for metadata storage backends.

Functions

list_assets(limit: int | None = None, **filters) -> list[dict] abstractmethod

List assets with optional filters.

Source code in flowyml/storage/metadata.py 
39
40
41
42
@abstractmethod
def list_assets(self, limit: int | None = None, **filters) -> list[dict]:
    """List assets with optional filters."""
    pass

list_pipelines() -> list[str] abstractmethod

List all unique pipeline names.

Source code in flowyml/storage/metadata.py 
24
25
26
27
@abstractmethod
def list_pipelines(self) -> list[str]:
    """List all unique pipeline names."""
    pass

list_runs(limit: int | None = None) -> list[dict] abstractmethod

List all runs.

Source code in flowyml/storage/metadata.py 
19
20
21
22
@abstractmethod
def list_runs(self, limit: int | None = None) -> list[dict]:
    """List all runs."""
    pass

load_artifact(artifact_id: str) -> dict | None abstractmethod

Load artifact metadata.

Source code in flowyml/storage/metadata.py 
34
35
36
37
@abstractmethod
def load_artifact(self, artifact_id: str) -> dict | None:
    """Load artifact metadata."""
    pass

load_run(run_id: str) -> dict | None abstractmethod

Load run metadata.

Source code in flowyml/storage/metadata.py 
14
15
16
17
@abstractmethod
def load_run(self, run_id: str) -> dict | None:
    """Load run metadata."""
    pass

query(**filters) -> list[dict] abstractmethod

Query runs with filters.

Source code in flowyml/storage/metadata.py 
44
45
46
47
@abstractmethod
def query(self, **filters) -> list[dict]:
    """Query runs with filters."""
    pass

save_artifact(artifact_id: str, metadata: dict) -> None abstractmethod

Save artifact metadata.

Source code in flowyml/storage/metadata.py 
29
30
31
32
@abstractmethod
def save_artifact(self, artifact_id: str, metadata: dict) -> None:
    """Save artifact metadata."""
    pass

save_run(run_id: str, metadata: dict) -> None abstractmethod

Save run metadata.

Source code in flowyml/storage/metadata.py 
 9
10
11
12
@abstractmethod
def save_run(self, run_id: str, metadata: dict) -> None:
    """Save run metadata."""
    pass

SQLite Metadata Store

See Also