Making AI Agents First-Class Citizens in Your Data Intelligence Platform

The Shift: From Code to Context

AI agents don’t struggle with intelligence. They struggle with missing or implicit context.

In traditional data platforms, context often lives in code, notebooks, and runtime parameters. Humans can reconstruct that context. Agents cannot. For an agent, anything not explicitly represented becomes guesswork.

This leads to a fundamental shift in data and AI solutions:

We are moving from code-driven systems to context-driven systems.

 

To build systems in which agents can operate reliably, we need to answer three questions:

• What does the agent know?
• How does the agent behave?
• Where does the agent act?

A Simple Model: Context, Behavior, Execution

Instead of thinking in tools or components, we can structure systems around three layers:

 

• Context defines the world
• Behavior defines the rules
• Execution defines how actions happen

Agents succeed when all three are explicit.

Part 1 — Context: Making the World Understandable

Context is the most important layer. Agents cannot reason about systems that are not clearly defined.

Context answers a set of fundamental questions:

• Where am I operating? — which domain defines the scope
• Who am I in this context? — what role and responsibility I have
• What does the data mean? — which concepts and definitions apply
• What exists and how is it used? — what metadata describes the system and can be used to drive behavior
• Which world am I in? — dev, test, or prod

Together, these elements form the world an agent can reason about.

Domains: Defining Scope

Modern lakehouses move in this direction through modular design. Data is organized into domains, each with clear ownership and boundaries. This structure is not just beneficial for humans — it is essential for agents.

An agent should never operate across undefined domains. Boundaries are are what make reasoning safe.

A well-defined domain provides:

• Clear scope — what data is relevant
• Ownership — who defines correctness
• Contracts — how data can be used

 

Roles: Defining Responsibility

Before defining what data means or how systems behave, we need to answer a more fundamental question:
Who is responsible for what?

A common mistake in agent design is treating agents as general-purpose assistants.

In practice, this leads to agents that try to do everything, require broad permissions, and behave inconsistently.

Instead, agents should be designed around roles.

A role does defines what the agent is responsible for and introduce clear boundaries. They prevent over-permissioned agents, eliminate “do everything” behavior, and avoid conflicts.

A role defines:

• What the agent owns?
• What decisions it can make?
• When it should escalate?

Examples include:

• Planner / Orchestrator — decomposes goals and assigns work
• Designer — Designs the models and defines the structure of the transformation logic.
• Analytics Agent — answers business questions using governed SQL
• Data Research Agent — explores datasets and validates assumptions
• ML Feature Agent — prepares and validates feature sets
• Builder — produces the final pipelines and artefacts

But structure and roles are not enough. Agents also need meaning.

Ontologies: Turning Metadata into Meaning

Ontologies are not built — they are grown. Most systems have metadata, but not all metadata carries meaning.

Ontologies provide that missing layer. They define business concepts, relationships, and shared definitions. Instead of treating “customer” or “revenue” as strings, ontologies make them explicit, structured, and reusable.

Without ontologies, agents rely on pattern matching. With ontologies, they can reason consistently across domains. This is the difference between recognizing data and understanding it.

Keep it minimal and high-leverage:

1. Core concepts — Not hundreds, just the business-critical ones

• Customer
• Order
• Revenue
• Product

 

2. A Few Canonical Definitions

Start with a small set of clear, shared definitions. For example:

Concept: Revenue
Definition: Income from completed orders

That alone is enough to anchor meaning across the system.

 

3. Key Relationships

Then define a few fundamental relationships:

Customer → Order
Order → Product

This may seem minimal, but it is powerful. With just a small foundation of concepts and relationships, agents gain enough structure to reason effectively.

Once this foundation exists, agents can extend it. They can generate column descriptions, summarize tables, and infer relationships. They can suggest meaningful connections — such as identifying a table as an OrderLine or recognizing that a column aligns with Revenue. They can also normalize inconsistencies, reconciling differences like amount versus total_price or cust_id versus customer_id.

The human-in-the-loop model

You’re not building ontology once, you’re evolving it continuously

Environment: Context vs Runtime Parameters

In data systems we often rely on runtime parameters to provide context.
Parameters such as run_date or flags passed into notebooks are:

• Hidden
• Ephemeral (changes or disappears)
• Tied to execution

In contrast, context, expressed through metadata and environment variables, is:

• Explicit
• Stable
• Visible

Runtime parameters describe a run.
Context describes a world.

Environment variables such as dev, test, or prod do not change the logic of a system. They define which world the system operates in.

Agents can reason about worlds. They struggle with runs.

If a system depends on hidden, run-specific inputs, its behavior becomes implicit. If context is explicit, the system itself becomes understandable.

That is exactly what separates imperative from declarative systems.

Part 2 - Behavior: From Imperative to Declarative Systems

Once context is clear, the next question is how agents should act.

Imperative systems describe how to compute something step by step. They depend on execution order and runtime state. This works well for humans but introduces implicit assumptions.

Declarative systems define outcomes — and that stability is critical for agents.

Imperative vs Declarative code

A notebook is an imperative program.
You run it with inputs: a date, a path, a flag. Same logic, different runs.

run_date = dbutils.widgets.get("run_date")
source_table = dbutils.widgets.get("source_table")
target_table = dbutils.widgets.get("target_table")

updates = spark.read.table(source_table) \
    .filter(f"order_date = '{run_date}'")
spark.sql(f"""
MERGE INTO {target_table} t
USING updates s
ON t.order_id = s.order_id
WHEN MATCHED THEN UPDATE SET *
WHEN NOT MATCHED THEN INSERT *
""")

Here, the behavior depends on runtime parameters.
The logic is correct — but the meaning of the result depends on the run.

A declarative pipeline is a definition.

from pyspark import pipelines as dp
  
@dp.table(  
name="sales",  
comment="Up-to-date sales table with incremental updates"  
)  
def sales():  
return dp.apply_changes(  
target="sales",  
source="raw.sales",  
keys=["order_id"],  
sequence_by="order_timestamp"  
)

No hidden parameters. No ambiguity.

This dataset is always computed like this from these inputs.

Why Declarative Systems Work for Agents

Data assistants love declarative systems because they rely on clear, stable metadata and predictable structures

Agents don’t struggle with complexity.
They struggle with implicit context.

But declarative systems introduce a new question:

If behavior is defined as metadata or configuration — how is it executed?

In imperative systems, execution is embedded in code.
In declarative systems, execution must be resolved.

This is where another key pattern emerges.

Registry Pattern: Resolving Intent into Action

A registry provides a mapping between intent and implementation.

Instead of embedding logic in control flow:

if write_type == "append":
    append_writer(...)
elif write_type == "scd2":
    scd2_writer(...)

We define behavior declaratively:

write_type: scd2
target_table: curated.sales

And resolve it through a registry:

writer = get_writer("scd2", declarative=True)
writer(df, "curated.sales", options)

The system knows how to execute scd2 because that capability has been registered:

@register_writer("scd2", declarative=True)
def scd2_declarative(...):
    ...

The registry separates:

• what should happen → metadata
• how it happens → implementation

This removes branching logic from pipelines and replaces it with a capability resolution layer.

With this pattern behavior becomes discoverable

• capabilities are enumerable (append, scd1, scd2)
• behavior is reusable
• execution is consistent

Dynamic Pipeline Generation

Declarative systems shift pipeline development from writing execution logic to defining intent.

Instead of hardcoding pipelines, the system can:

• scan metadata or configuration
• determine what should be built
• assemble pipelines dynamically

This is possible because declarative pipelines define structure independently of execution.

The registry complements this by resolving capabilities at runtime — mapping intent to implementation and enabling consistent, reusable behavior.

Together, they allow pipelines to be generated programmatically rather than written manually.

Example: Dynamic Pipeline Generation with Declarative + Registry

Instead of hardcoding pipelines, we define them through metadata:

pipelines:
 - name: sales_pipeline
 source: raw.sales
 target: curated.sales
 write_type: scd2
 keys: [order_id]
 sequence_by: order_timestamp

 - name: customer_pipeline
 source: raw.customers
 target: curated.customers
 write_type: scd1
 keys: [customer_id]
 sequence_by: updated_at

Pipeline builder: The system scans this configuration and builds pipelines dynamically:

from pyspark import pipelines as dp
from core.writers import get_writer #contains registered implementations(append, scd1, scd2, …)

def build_pipeline(config):
    for p in config["pipelines"]:
        
        writer = get_writer(p["write_type"], declarative=True)

        @dp.table(name=p["target"])
        def table():
            df = spark.read.table(p["source"])

            return writer(
                df,
                p["target"],
                {
                    "primary_key_columns": p["keys"],
                    "sequence_column": p["sequence_by"],
                    "source_view": p["source"]
                }
            )

• The declarative pipeline defines what should exist
• The metadata defines what type of behavior is needed
• The registry resolves how that behavior is executed

A declarative system does not contain behavior — it defines it through references and specifications.

Skills follow the same principle: they do not expose how an action is performed, only that the capability exists.

Skills: Structured Capabilities

Agents should not directly interact with systems. They should operate through well-defined capabilities.

Skills provide this abstraction.

A skill encapsulates an action — querying data, profiling a dataset, or explaining lineage — behind a structured interface.

This creates a clean separation:

• Agents decide what to do
• Skills define what can be done

For example , instead of allowing an agent to generate arbitrary SQL, we define a skill:

name: explain_lineage
description: Explain upstream sources and transformations for a dataset
inputs:
  table_name:
    type: string
outputs:
  - upstream_tables
  - transformation_steps
  - last_updated
constraints:
  - only_access: Unity Catalog lineage API

This means:

• The agent does not invent its own way of exploring lineage.
• It must use a defined capability with a known input and known outputs.
• The interaction is structured, predictable, and reusable.
• The agent is limited to metadata-level lineage access rather than arbitrary data access.

Guardrails: Safe Behavior by Design

Defining skills is not enough. Agents also need guardrails — explicit constraints that ensure actions are safe, compliant, and predictable.

Guardrails are not intelligence. They are rules encoded in metadata and enforced by the system.

Guardrails sit between intent and execution.

For example, a pipeline-generation skill might include these constraints:

name: generate_pipeline
inputs:
  source_name: string
constraints:
  - require_existing_metadata: true
  - require_human_approval_for_publish: t
  - allowed_output: draft_only
  - forbid_direct_production_deploy: true

That means:

• The agent can generate from defined metadata only
• It can prepare a draft, but not publish directly
• A human must approve before deployment
• It cannot push changes straight to production

Even if the agent tries, the system prevents unsafe behavior.

Part 3 — Execution: Choosing the Right Interface

Context and behavior define what should happen.
Execution defines how an agent interacts with the system.

CLI, MCP, and FastMCP

An agent should not work directly with raw implementation details such as notebook paths, SQL strings, or SDK calls. Instead, systems should expose a clear execution surface that turns internal logic into explicit capabilities.

This surface typically takes one of three forms:

• CLI → direct command execution
• MCP (Model Context Protocol) → structured, discoverable capability layer
• FastMCP (local MCP) → lightweight bridge between local tools and MCP-compatible agents

CLI represents a traditional interface. It is imperative, flexible, and well-suited for local tasks where context is already known. Developers use it for debugging, exploration, and targeted operations.

MCP (Model Context Protocol) is Anthropic’s standardized protocol for LLMs to communicate with external tools, services, and environments. Under the hood, it may be calling an API or a CLI, but it exposes that to the LLM in a structured, well-defined way.

FastMCP is a lightweight Python library that implements the Model Context Protocol (MCP) locally. It wraps existing functions, scripts, or CLI commands as structured tools, making them accessible to agents without requiring a deployed MCP server.

A CLI is a command surface. It is fast, direct, and familiar. The agent runs a command, reads the output, and moves on. That makes CLI a strong fit for the inner loop of development: writing code, running tests, validating configuration, and debugging locally.

MCP solves a different problem. It is not primarily about running commands. It is about exposing capabilities through a structured, discoverable interface. An agent can see which tools exist, what inputs they expect, and what outputs they return. That makes MCP a better fit for the outer loop, where the agent needs to interact with shared systems such as platform APIs.

Use CLI where the work is local, fast, and iterative.

Use MCP when the LLM needs to interact with external tools or systems — providing a structured bridge for requests, shared authentication, and consistent, reliable outputs.

The right question is not “CLI or MCP?”
It is: “What problem am I solving , and at which layer does it live?”

 

If the agent already understands the tools and has execution access, CLI is often enough.
If you are exposing capabilities across systems, need discoverability, or require structured interactions, MCP becomes the right choice.

Using both in the same product is not a compromise — it is often the natural architecture.

If an agent can already execute commands, use structured capabilities through a registry, and query governed data, then the system is already understandable.

MCP adds value when capabilities are hidden or need to be discovered.
If they are already explicit, it becomes optional — not required.

A well-designed CLI is often enough for agent interaction as long as the system behind it is structured and metadata-driven.

Where FastMCP Fits

FastMCP sits between CLI and hosted MCP. It allows you to expose local capabilities in a structured, MCP-compatible way, without introducing a full hosted MCP layer.

FastMCP does not replace CLI.
It makes local capabilities look like tools.

In many systems, local capabilities already exist:

• CLI commands
• Python functions
• registry-based abstractions (writers, validators, etc.)

FastMCP wraps these into explicit tools with defined inputs and outputs, making them easier for agents to use consistently.

This is particularly useful when:

• you want agents to interact with local developer environments
• you want to expose registry-based capabilities as tools
• you need structure and consistency, but not a hosted interface

At the same time, FastMCP does not introduce new capabilities. It changes how they are exposed.

If your system is already well-structured — through metadata, registry patterns, and a clean CLI — then FastMCP is optional. It becomes valuable when you want to make those capabilities more discoverable and standardized for agent interaction, especially across different agent frameworks. Agents can use a CLI directly, but they have to infer how to use it. Structured interfaces like MCP remove that guesswork.

When to Host MCP on the data platform

On platforms like Databricks, this becomes especially relevant when exposing capabilities such as Unity Catalog functions, Vector Search, or Genie.

In these cases, MCP provides a unified interface for agents, allowing them to interact with platform-native capabilities consistently instead of dealing with each service through separate access patterns.

Without MCP, these capabilities exist — but are fragmented:

• Unity Catalog functions → callable via SQL
• Vector Search → callable via API
• Genie → separate interface

Each capability has its own access pattern, interface, and integration model.

With MCP, they become part of a single, structured capability surface:

Agent → MCP → [UC Functions, Vector Search, Genie]

MCP exposes them, making them discoverable and consistent for agents across environments.

This is especially valuable when:

• capabilities are distributed across services
• access should be consistent across users and environments
• agents need structured, predictable interaction

When CLI or Local FastMCP Is Better

CLI and local FastMCP are best suited for local, iterative workflows where the agent already understands the system. This includes tasks like running notebooks, executing SQL, validating metadata, or generating and testing pipelines.

In these scenarios, capabilities are already explicit — defined through your registry, structured context, and CLI interface. The agent does not need an additional abstraction layer to discover or use them. It can act directly.

Rule of Thumb

Use CLI or local MCP when capabilities are local and already explicit. Use Databricks-hosted MCP when capabilities need to be exposed as a shared, discoverable interface.

Bringing It Together

In modular lakehouse architectures, context is often defined at the bundle (or project) level — a self-contained unit that defines a data product, its pipelines, and its metadata. Agents should be scoped accordingly.

This mirrors how human teams operate. An engineer working in one domain does not automatically gain context or authority in another.

By encoding these boundaries into structure and configuration, we enable least-privilege reasoning — agents operate with exactly the context they need, and nothing more.

Intelligent Scaffolding

As systems grow, productivity does not come from more flexibility — it comes from reducing ambiguity and cognitive load, the mental effort required to understand and navigate the system.

Scaffolding is how the system guides behavior using structure. It provides templates, conventions, and defaults that encode best practices directly into the system, guiding both humans and agents toward correct behavior without requiring constant decision-making.

In an agent-native architecture, scaffolding is what makes context actionable. It turns metadata, roles, and capabilities into a path the agent can follow.

Governance as a Boundary

Agents should not enforce governance.
Governance is an architectural boundary.

Systems like Unity Catalog define permissions, auditing, and lineage. Agents operate within these constraints — they do not interpret or redefine them.

This separation is critical. It keeps agents simple and systems trustworthy.

Final Thoughts

The Shift

We are moving from:

• code → context
• execution → definition
• implicit → explicit

From systems that run to systems that can be reasoned about

The Principle

The goal is not to describe everything.
It is to make the important things unambiguous.

The Outcome

When context is explicit, agents act predictably, systems remain composable, and governance becomes enforceable.

When context is hidden, behavior becomes inconsistent, logic becomes brittle, and systems stop scaling.

We are not building smarter agents.
We are building systems that can be understood.