Kestra Governance Assets

Governance Assets represents one of the most significant enterprise features added to Kestra, addressing a critical need for data governance in modern data orchestration platforms. As enterprises scale their data operations, understanding what resources workflows create, modify, and consume becomes essential for compliance, auditing, and operational visibility.

This project required end-to-end product leadership: from initial concept and user research, through design and DSL specification, to engineering coordination, QA, and client presentations. I owned every aspect of the product development lifecycle, ensuring the feature met both technical requirements and enterprise customer needs.

Enterprise customers managing complex data pipelines needed visibility into:

• •Data Lineage: Understanding how data flows through workflows and what downstream resources depend on upstream outputs
• •Resource Tracking: Knowing which workflows create, modify, or consume specific data assets, tables, files, or services
• •Metadata Management: Attaching rich metadata to resources for documentation, compliance, and discovery
• •Impact Analysis: Identifying which workflows would be affected by changes to upstream data sources

The solution needed to integrate seamlessly with Kestra's declarative workflow DSL while providing powerful governance capabilities that enterprise customers expect from modern data platforms.

As the product owner, I led this feature from initial concept through delivery:

One of the most critical aspects was designing a DSL that felt natural within Kestra workflows while providing the expressiveness needed for complex governance scenarios:

id: data_pipeline_assets
namespace: kestra.company.data

tasks:
  - id: create_staging_layer_asset
    type: io.kestra.plugin.jdbc.duckdb.Query
    sql: |
      CREATE TABLE IF NOT EXISTS trips AS
      select VendorID, passenger_count, trip_distance from sample_data.nyc.taxi limit 10;
    assets:
      inputs:
        - id: sample_data.nyc.taxi
      outputs:
          - id: trips
            namespace: "{{flow.namespace}}"
            type: io.kestra.plugin.ee.assets.Table
            metadata:
              model_layer: staging

  - id: for_each
    type: io.kestra.plugin.core.flow.ForEach
    values:
      - passenger_count
      - trip_distance
    tasks:
      - id: create_mart_layer_asset
        type: io.kestra.plugin.jdbc.duckdb.Query
        sql: SELECT AVG({{taskrun.value}}) AS avg_{{taskrun.value}} FROM trips;
        assets:
          inputs:
              - id: trips
          outputs:
              - id: avg_{{taskrun.value}}
                type: io.kestra.plugin.ee.assets.Table
                namespace: "{{flow.namespace}}"
                metadata:
                  model_layer: mart
pluginDefaults:
  - type: io.kestra.plugin.jdbc.duckdb
    values:
      url: "jdbc:duckdb:md:my_db?motherduck_token={{ secret('MOTHERDUCK_TOKEN') }}"
      fetchType: STORE

This example demonstrates a complete data pipeline with asset tracking across multiple layers: staging tables created from source data, and mart layer aggregations built from staging assets. The DSL supports dynamic asset IDs (using template variables), metadata attachment for documentation (like model_layer), and clear lineage tracking through inputs and outputs. The design prioritizes clarity and discoverability while maintaining the declarative philosophy of Kestra workflows.

Working closely with the engineering team, I coordinated:

I led comprehensive QA efforts to ensure the feature met enterprise-grade quality standards:

• •Functional Testing: Validated all asset operations, lineage tracking, and metadata management across various scenarios
• •Performance Testing: Ensured lineage queries and asset lookups perform efficiently at scale
• •Edge Cases: Tested complex scenarios including circular dependencies, orphaned assets, and concurrent modifications
• •User Acceptance: Coordinated beta testing with select enterprise customers to gather feedback before general availability

I presented the Governance Assets feature to enterprise customers, demonstrating:

The feature received strong positive feedback from enterprise customers, with several adopting it immediately upon release. The presentations helped refine the feature based on real-world requirements and use cases.

Managing a feature of this scope required careful planning and coordination:

• •Roadmap Planning: Broken down the feature into manageable phases, prioritizing core functionality before advanced features
• •Sprint Coordination: Worked with engineering teams to plan sprints, manage dependencies, and ensure timely delivery
• •Stakeholder Management: Regular updates to leadership and sales teams on progress, timelines, and customer feedback
• •Documentation: Created comprehensive documentation, examples, and migration guides for users and customers

Governance Assets became a key differentiator for Kestra in enterprise sales conversations, addressing a critical gap in the data orchestration market. The feature enables:

Leading this project end-to-end provided valuable insights into product development at scale:

• •DSL Design Matters: The declarative syntax needed to feel intuitive while remaining powerful—this required multiple iterations and user feedback cycles
• •Enterprise Requirements: Understanding the specific governance needs of enterprise customers was crucial—generic solutions wouldn't have sufficed
• •Cross-Functional Leadership: Successfully coordinating design, engineering, QA, and customer success required clear communication and shared vision
• •User-Centric Design: Presenting to customers early and often helped validate assumptions and refine the feature before full release