Skip to content

Vertex AI hosted models for ADK agents

For enterprise-grade scalability, reliability, and integration with Google Cloud's MLOps ecosystem, you can use models deployed to Vertex AI Endpoints. This includes models from Model Garden or your own fine-tuned models.

Integration Method: Pass the full Vertex AI Endpoint resource string (projects/PROJECT_ID/locations/LOCATION/endpoints/ENDPOINT_ID) directly to the model parameter of LlmAgent.

Using Fully Qualified Resource Names

Supported in ADKPython v1.29.0

As a more direct alternative to setting environment variables, you can pass a fully qualified Vertex AI model or endpoint resource name directly into the model parameter. The ADK automatically parses the Project ID, location, and model name from the string, configuring the client to use the Vertex AI backend.

This method is convenient when working with models from Model Garden or when your application needs to access models across different Google Cloud projects.

When you use a fully qualified resource name, you do not need to set the following environment variables for that model: - GOOGLE_CLOUD_PROJECT - GOOGLE_CLOUD_LOCATION - GOOGLE_GENAI_USE_VERTEXAI

Examples of fully qualified names:

  • Google Model: projects/YOUR_PROJECT/locations/YOUR_REGION/publishers/google/models/gemini-1.5-flash
  • Third-Party Model: projects/YOUR_PROJECT/locations/YOUR_REGION/publishers/anthropic/models/claude-3-5-sonnet@20240620
  • Endpoint: projects/YOUR_PROJECT/locations/YOUR_REGION/endpoints/YOUR_ENDPOINT_ID

Vertex AI Setup

If you are not using fully qualified resource names, you must ensure your environment is configured for Vertex AI:

  1. Authentication: Use Application Default Credentials (ADC):

    gcloud auth application-default login

  2. Environment Variables: Set your project and location:

    export GOOGLE_CLOUD_PROJECT="YOUR_PROJECT_ID"
export GOOGLE_CLOUD_LOCATION="YOUR_VERTEX_AI_LOCATION" # e.g., us-central1

  3. Enable Vertex Backend: Crucially, ensure the google-genai library targets Vertex AI:

    export GOOGLE_GENAI_USE_VERTEXAI=TRUE

Model Garden Deployments

Supported in ADKPython v0.2.0Java v0.1.0

You can deploy various open and proprietary models from the Vertex AI Model Garden to an endpoint.

Example:

from google.adk.agents import LlmAgent
from google.genai import types # For config objects

# --- Example Agent using a Llama 3 model deployed from Model Garden ---

# Replace with your actual Vertex AI Endpoint resource name
llama3_endpoint = "projects/YOUR_PROJECT_ID/locations/us-central1/endpoints/YOUR_LLAMA3_ENDPOINT_ID"

agent_llama3_vertex = LlmAgent(
    model=llama3_endpoint,
    name="llama3_vertex_agent",
    instruction="You are a helpful assistant based on Llama 3, hosted on Vertex AI.",
    generate_content_config=types.GenerateContentConfig(max_output_tokens=2048),
    # ... other agent parameters
)

import com.google.adk.agents.LlmAgent;
import com.google.adk.models.Gemini;
import com.google.genai.types.GenerateContentConfig;

// ...

// Replace with your actual Vertex AI Endpoint resource name
String llama3Endpoint = "projects/YOUR_PROJECT_ID/locations/us-central1/endpoints/YOUR_LLAMA3_ENDPOINT_ID";

LlmAgent agentLlama3Vertex = LlmAgent.builder()
    .model(Gemini.builder()
        .modelName(llama3Endpoint)
        .build())
    .name("llama3_vertex_agent")
    .instruction("You are a helpful assistant based on Llama 3, hosted on Vertex AI.")
    .generateContentConfig(GenerateContentConfig.builder()
        .maxOutputTokens(2048)
        .build())
    // ... other agent parameters
    .build();

Fine-tuned Model Endpoints

Supported in ADKPython v0.2.0Java v0.1.0

Deploying your fine-tuned models (whether based on Gemini or other architectures supported by Vertex AI) results in an endpoint that can be used directly.

Example:

from google.adk.agents import LlmAgent

# --- Example Agent using a fine-tuned Gemini model endpoint ---

# Replace with your fine-tuned model's endpoint resource name
finetuned_gemini_endpoint = "projects/YOUR_PROJECT_ID/locations/us-central1/endpoints/YOUR_FINETUNED_ENDPOINT_ID"

agent_finetuned_gemini = LlmAgent(
    model=finetuned_gemini_endpoint,
    name="finetuned_gemini_agent",
    instruction="You are a specialized assistant trained on specific data.",
    # ... other agent parameters
)

import com.google.adk.agents.LlmAgent;
import com.google.adk.models.Gemini;

// ...

// Replace with your fine-tuned model's endpoint resource name
String finetunedGeminiEndpoint = "projects/YOUR_PROJECT_ID/locations/us-central1/endpoints/YOUR_FINETUNED_ENDPOINT_ID";

LlmAgent agentFinetunedGemini = LlmAgent.builder()
    .model(Gemini.builder()
        .modelName(finetunedGeminiEndpoint)
        .build())
    .name("finetuned_gemini_agent")
    .instruction("You are a specialized assistant trained on specific data.")
    // ... other agent parameters
    .build();

Anthropic Claude on Vertex AI

Supported in ADKPython v0.2.0Java v0.1.0

Some providers, like Anthropic, make their models available directly through Vertex AI.

Example:

Integration Method: You can use a fully qualified resource name (e.g., "projects/.../models/claude-3-sonnet@20240229") or a simple model string (e.g., "claude-3-sonnet@20240229"). Using the simple string requires manual registration within ADK.

Why Registration? ADK's registry automatically recognizes gemini-* strings and standard Vertex AI resource names (projects/.../endpoints/..., projects/.../models/...) and routes them via the google-genai library. For other model types used directly via Vertex AI (like Claude), if you use the simple model string, you must explicitly tell the ADK registry which specific wrapper class (Claude in this case) knows how to handle that model identifier string with the Vertex AI backend.

Setup (with simple model string):

  1. Vertex AI Environment: Ensure the consolidated Vertex AI setup (ADC, Env Vars, GOOGLE_GENAI_USE_VERTEXAI=TRUE) is complete.

  2. Install Provider Library: Install the necessary client library configured for Vertex AI.

    pip install "anthropic[vertex]"

  3. Register Model Class: Add this code near the start of your application, before creating an agent using the Claude model string:

    # Required for using Claude model strings directly via Vertex AI with LlmAgent
from google.adk.models.anthropic_llm import Claude
from google.adk.models.registry import LLMRegistry

LLMRegistry.register(Claude)

from google.adk.agents import LlmAgent
from google.adk.models.anthropic_llm import Claude # Import needed for registration
from google.adk.models.registry import LLMRegistry # Import needed for registration
from google.genai import types

# --- Register Claude class (do this once at startup) ---
LLMRegistry.register(Claude)

# --- Example Agent using Claude 3 Sonnet on Vertex AI ---

# Standard model name for Claude 3 Sonnet on Vertex AI
claude_model_vertexai = "claude-3-sonnet@20240229"

agent_claude_vertexai = LlmAgent(
    model=claude_model_vertexai, # Pass the direct string after registration
    name="claude_vertexai_agent",
    instruction="You are an assistant powered by Claude 3 Sonnet on Vertex AI.",
    generate_content_config=types.GenerateContentConfig(max_output_tokens=4096),
    # ... other agent parameters
)

Integration Method: Directly instantiate the provider-specific model class (e.g., com.google.adk.models.Claude) and configure it with a Vertex AI backend.

Why Direct Instantiation? The Java ADK's LlmRegistry primarily handles Gemini models by default. For third-party models like Claude on Vertex AI, you directly provide an instance of the ADK's wrapper class (e.g., Claude) to the LlmAgent. This wrapper class is responsible for interacting with the model via its specific client library, configured for Vertex AI.

Setup:

  1. Vertex AI Environment:

    • Ensure your Google Cloud project and region are correctly set up.
    • Application Default Credentials (ADC): Make sure ADC is configured correctly in your environment. This is typically done by running gcloud auth application-default login. The Java client libraries use these credentials to authenticate with Vertex AI. Follow the Google Cloud Java documentation on ADC for detailed setup.

  2. Provider Library Dependencies:

    • Third-Party Client Libraries (Often Transitive): The ADK core library often includes the necessary client libraries for common third-party models on Vertex AI (like Anthropic's required classes) as transitive dependencies. This means you might not need to explicitly add a separate dependency for the Anthropic Vertex SDK in your pom.xml or build.gradle.

  3. Instantiate and Configure the Model: When creating your LlmAgent, instantiate the Claude class (or the equivalent for another provider) and configure its VertexBackend.

import com.anthropic.client.AnthropicClient;
import com.anthropic.client.okhttp.AnthropicOkHttpClient;
import com.anthropic.vertex.backends.VertexBackend;
import com.google.adk.agents.LlmAgent;
import com.google.adk.models.Claude; // ADK's wrapper for Claude
import com.google.auth.oauth2.GoogleCredentials;
import java.io.IOException;

// ... other imports

public class ClaudeVertexAiAgent {

    public static LlmAgent createAgent() throws IOException {
        // Model name for Claude 3 Sonnet on Vertex AI (or other versions)
        String claudeModelVertexAi = "claude-3-7-sonnet"; // Or any other Claude model

        // Configure the AnthropicOkHttpClient with the VertexBackend
        AnthropicClient anthropicClient = AnthropicOkHttpClient.builder()
            .backend(
                VertexBackend.builder()
                    .region("us-east5") // Specify your Vertex AI region
                    .project("your-gcp-project-id") // Specify your GCP Project ID
                    .googleCredentials(GoogleCredentials.getApplicationDefault())
                    .build())
            .build();

        // Instantiate LlmAgent with the ADK Claude wrapper
        LlmAgent agentClaudeVertexAi = LlmAgent.builder()
            .model(new Claude(claudeModelVertexAi, anthropicClient)) // Pass the Claude instance
            .name("claude_vertexai_agent")
            .instruction("You are an assistant powered by Claude 3 Sonnet on Vertex AI.")
            // .generateContentConfig(...) // Optional: Add generation config if needed
            // ... other agent parameters
            .build();

        return agentClaudeVertexAi;
    }

    public static void main(String[] args) {
        try {
            LlmAgent agent = createAgent();
            System.out.println("Successfully created agent: " + agent.name());
            // Here you would typically set up a Runner and Session to interact with the agent
        } catch (IOException e) {
            System.err.println("Failed to create agent: " + e.getMessage());
            e.printStackTrace();
        }
    }
}

Open Models on Vertex AI

Supported in ADKPython v0.1.0Java v0.1.0

Vertex AI offers a curated selection of open-source models, such as Meta Llama, through Model-as-a-Service (MaaS). These models are accessible via managed APIs, allowing you to deploy and scale without managing the underlying infrastructure. For a full list of available options, see the Vertex AI open models for MaaS documentation.

You can use the LiteLLM library to access open models like Meta's Llama on VertexAI MaaS

Integration Method: Use the LiteLlm wrapper class and set it as the model parameter of LlmAgent. Make sure you go through the LiteLLM model connector for ADK agents documentation on how to use LiteLLM in ADK

Setup:

  1. Install LiteLLM: 
    pip install litellm

Example:

from google.adk.agents import LlmAgent
from google.adk.models.lite_llm import LiteLlm

# --- Example Agent using Meta's Llama 4 Scout ---
agent_llama_vertexai = LlmAgent(
    model=LiteLlm(model="vertex_ai/meta/llama-4-scout-17b-16e-instruct-maas"), # LiteLLM model string format
    name="llama4_agent",
    instruction="You are a helpful assistant powered by Llama 4 Scout.",
    # ... other agent parameters
)