Overview

Prompt Drivers are used by Gen AI Builder Structures to make API calls to the underlying LLMs. OpenAi Chat is the default prompt driver used in all structures.

You can instantiate drivers and pass them to structures:

Or use them independently:

You can pass images to the Driver if the model supports it:

from griptape.artifacts import ListArtifact, TextArtifact
from griptape.drivers.prompt.openai import OpenAiChatPromptDriver
from griptape.loaders import ImageLoader

driver = OpenAiChatPromptDriver(model="gpt-4.1", max_tokens=256)

image_artifact = ImageLoader().load("./tests/resources/mountain.jpg")
text_artifact = TextArtifact("Describe the weather in the image")

driver.run(ListArtifact([text_artifact, image_artifact]))

Structured Output

Some LLMs provide functionality often referred to as "Structured Output". This means instructing the LLM to output data in a particular format, usually JSON. This can be useful for forcing the LLM to output in a parsable format that can be used by downstream systems.

Warning

Each Driver may have a different default setting depending on the LLM provider's capabilities.

Prompt Task

The easiest way to get started with structured output is by using a PromptTask's output_schema parameter.

You can change how the output is structured by setting the Driver's structured_output_strategy to one of:

  • native: The Driver will use the LLM's structured output functionality provided by the API.
  • tool: The Driver will add a special tool, StructuredOutputTool, and will try to force the LLM to use the Tool.
  • rule: The Driver will add a JsonSchemaRule to the Task's system prompt. This strategy does not guarantee that the LLM will output JSON and should only be used as a last resort.

You can specify output_schema using either pydantic or schema libraries, though pydantic is recommended.

Info

PromptTasks will automatically validate the LLM's output against the output_schema you provide. If the LLM fails to output valid JSON, the Task will automatically re-prompt the LLM to try again. You can configure how many times it will retry by setting the max_subtasks parameter on the Task.

Prompt Drivers

Gen AI Builder offers the following Prompt Drivers for interacting with LLMs.

OpenAI Chat

The OpenAiChatPromptDriver connects to the OpenAI Chat API. This driver uses OpenAi function calling when using Tools.

Info

response_format and seed are unique to the OpenAI Chat Prompt Driver and Azure OpenAi Chat Prompt Driver.

OpenAi's reasoning models can also be used, and come with an additional parameter: reasoning_effort.

OpenAI Compatible

Many services such as LMStudio and OhMyGPT provide OpenAI-compatible APIs. You can use the OpenAiChatPromptDriver to interact with these services. Simply set the base_url to the service's API endpoint and the model to the model name. If the service requires an API key, you can set it in the api_key field.

For example, here is how we can connect to LMStudio's API:

from griptape.drivers.prompt.openai import OpenAiChatPromptDriver
from griptape.rules import Rule
from griptape.structures import Agent

agent = Agent(
    prompt_driver=OpenAiChatPromptDriver(
        base_url="http://127.0.0.1:1234/v1", model="lmstudio-community/Meta-Llama-3-8B-Instruct-GGUF", stream=True
    ),
    rules=[Rule(value="You are a helpful coding assistant.")],
)

agent.run("How do I init and update a git submodule?")

Or Groq's API:

Tip

Make sure to include v1 at the end of the base_url to match the OpenAI API endpoint.

Azure OpenAI Chat

The AzureOpenAiChatPromptDriver connects to Azure OpenAI Chat Completion APIs. This driver uses Azure OpenAi function calling when using Tools.

Gen AI Builder

The GriptapeCloudPromptDriver connects to the Gen AI Builder chat messages API.

Cohere

The CoherePromptDriver connects to the Cohere Chat API. This driver uses Cohere tool use when using Tools.

Info

This driver requires the drivers-prompt-cohere extra.

Anthropic

Info

This driver requires the drivers-prompt-anthropic extra.

The AnthropicPromptDriver connects to the Anthropic Messages API. This driver uses Anthropic tool use when using Tools.

Google

Info

This driver requires the drivers-prompt-google extra.

The GooglePromptDriver connects to the Google Generative AI API. This driver uses Gemini function calling when using Tools.

Amazon Bedrock

Info

This driver requires the drivers-prompt-amazon-bedrock extra.

The AmazonBedrockPromptDriver uses Amazon Bedrock's Converse API. This driver uses Bedrock tool use when using Tools.

All models supported by the Converse API are available for use with this driver.

Ollama

Info

This driver requires the drivers-prompt-ollama extra.

The OllamaPromptDriver connects to the Ollama Chat Completion API. This driver uses Ollama tool calling when using Tools.

from griptape.drivers.prompt.ollama import OllamaPromptDriver
from griptape.structures import Agent
from griptape.tools import CalculatorTool

agent = Agent(
    prompt_driver=OllamaPromptDriver(
        model="llama3.1",
    ),
    tools=[CalculatorTool()],
)
agent.run("What is (192 + 12) ^ 4")

Hugging Face Hub

Info

This driver requires the drivers-prompt-huggingface extra.

The HuggingFaceHubPromptDriver connects to the Hugging Face Hub API.

Warning

Not all models featured on the Hugging Face Hub are supported by this driver. Models that are not supported by Hugging Face serverless inference will not work with this driver. Due to the limitations of Hugging Face serverless inference, only models that are than 10GB are supported.

Text Generation Interface

The HuggingFaceHubPromptDriver also supports Text Generation Interface for running models locally. To use Text Generation Interface, just set model to a TGI endpoint.

import os

from griptape.drivers.prompt.huggingface_hub import HuggingFaceHubPromptDriver
from griptape.structures import Agent

agent = Agent(
    prompt_driver=HuggingFaceHubPromptDriver(
        model="http://127.0.0.1:8080",
        api_token=os.environ["HUGGINGFACE_HUB_ACCESS_TOKEN"],
    ),
)

agent.run("Write the code for a snake game.")

Hugging Face Pipeline

Info

This driver requires the drivers-prompt-huggingface-pipeline extra.

The HuggingFacePipelinePromptDriver uses Hugging Face Pipelines for inference locally.

Warning

Running a model locally can be a computationally expensive process.

Amazon SageMaker Jumpstart

Info

This driver requires the drivers-prompt-amazon-sagemaker extra.

The AmazonSageMakerJumpstartPromptDriver uses Amazon SageMaker Jumpstart for inference on AWS.

Amazon Sagemaker Jumpstart provides a wide range of models with varying capabilities. This Driver has been primarily chat-optimized models that have a Huggingface Chat Template available. If your model does not fit this use-case, we suggest sub-classing AmazonSageMakerJumpstartPromptDriver and overriding the _to_model_input and _to_model_params methods.

import os

from griptape.drivers.prompt.amazon_sagemaker_jumpstart import AmazonSageMakerJumpstartPromptDriver
from griptape.structures import Agent

agent = Agent(
    prompt_driver=AmazonSageMakerJumpstartPromptDriver(
        endpoint=os.environ["SAGEMAKER_LLAMA_3_INSTRUCT_ENDPOINT_NAME"],
        model="meta-llama/Meta-Llama-3-8B-Instruct",
    )
)

agent.run("What is a good lasagna recipe?")

Grok

The GrokPromptDriver uses Grok's chat completion endpoint.

Perplexity

The PerplexityPromptDriver uses Perplexity Sonar's chat completion endpoint.

While you can use this Driver directly, we recommend using it through its accompanying Web Search Driver.


Could this page be better? Report a problem or suggest an addition!