Tasks

Suggest edits

Overview

A Task is a purpose-built abstraction for the Large Language Model (LLM). Gen AI Builder offers various types of Tasks, each suitable for specific use cases.

Context

Tasks that take input have a field input which lets you define the Task objective. Within the input, you can access the following context variables:

  • args: an array of arguments passed to the .run() method.
  • structure: the structure that the task belongs to.
  • user defined context variables

Additional context variables may be added based on the Structure running the task.

Hooks

All Tasks implement RunnableMixin which provides on_before_run and on_after_run hooks for the Task lifecycle.

These hooks can be used to perform actions before and after the Task is run. For example, you can mask sensitive information before running the Task, and transform the output after the Task is run.

Prompt Task

For general-purpose interaction with LLMs, use the PromptTask:

Tools

You can pass in one or more Tools which the LLM will decide to use through Chain of Thought (CoT) reasoning. Because tool execution uses CoT, it is recommended to only use with very capable models.

Reflect On Tool Use

By default, Gen AI Builder will pass the results of Tool runs back to the LLM for reflection. This enables the LLM to reason about the results and potentially use additional tools.

However, there may be times where you may want the LLM to give you back the results directly, without reflection. You can disable this behavior by setting reflect_on_tool_use to False.

Important

Disabling reflection will prevent the LLM from using one Tool to inform the use of another Tool. Instead, you must coordinate the Tool uses yourself.

Images

If the model supports it, you can also pass image inputs:

Tool Task

Warning

ToolTask is deprecated and will be removed in a future version. Use Prompt Task with reflect_on_tool_use set to False instead.

Another way to use Gen AI Builder Tools, is with a Tool Task. This Task takes in a single Tool which the LLM will use without Chain of Thought (CoT) reasoning. Because this Task does not use CoT, it is better suited for less capable models.

Extraction Task

To extract information from text, use an ExtractionTask. This Task takes an Extraction Engine, and a set of arguments specific to the Engine.

CSV Extraction

JSON Extraction

Text Summary Task

To summarize a text, use the TextSummaryTask. This Task takes an Summarization Engine, and a set of arguments to the engine.

RAG Task

To query text, use the RagTask. This task takes a RAG Engine, and a set of arguments specific to the engine.

Code Execution Task

To execute an arbitrary Python function, use the CodeExecutionTask. This task takes a python function, and authors can elect to return a custom artifact.

In this example, the generate_title function combines a hero's name and setting from previous tasks with a random title, returning a TextArtifact that contains the generated fantasy title. The output of this task can then be referenced by subsequent tasks using the parent_outputs templating variable, as shown in the final PromptTask.

Branch Task

By default, a Workflow will only run a Task when all the Tasks it depends on have finished.

You can make use of BranchTask in order to tell the Workflow not to run all dependent tasks, but instead to pick and choose one or more paths to go down.

The BranchTask's on_run function can return one of three things:

  1. An InfoArtifact containing the id of the Task to run next.
  2. A ListArtifact of InfoArtifacts containing the ids of the Tasks to run next.
  3. An empty ListArtifact to indicate that no Tasks should be run next.

Image Generation Tasks

To generate an image, use one of the following Image Generation Tasks. All Image Generation Tasks accept an Image Generation Driver.

All successful Image Generation Tasks will always output an Image Artifact. Each task can be configured to additionally write the generated image to disk by providing either the output_file or output_dir field. The output_file field supports file names in the current directory (my_image.png), relative directory prefixes (images/my_image.png), or absolute paths (/usr/var/my_image.png). By setting output_dir, the task will generate a file name and place the image in the requested directory.

Prompt Image Generation Task

The Prompt Image Generation Task generates an image from a text prompt.

from griptape.drivers.image_generation.openai import OpenAiImageGenerationDriver
from griptape.structures import Pipeline
from griptape.tasks import PromptImageGenerationTask

# Create a driver configured to use OpenAI's DALL-E 3 model.
driver = OpenAiImageGenerationDriver(
    model="dall-e-3",
    quality="hd",
    style="natural",
)


# Instantiate a pipeline.
pipeline = Pipeline()

# Add a PromptImageGenerationTask to the pipeline.
pipeline.add_tasks(
    PromptImageGenerationTask(
        input="{{ args[0] }}",
        image_generation_driver=driver,
        output_dir="images/",
    )
)

pipeline.run("An image of a mountain on a summer day")

Variation Image Generation Task

The Variation Image Generation Task generates an image using an input image and a text prompt. The input image is used as a basis for generating a new image as requested by the text prompt.

from griptape.drivers.image_generation.amazon_bedrock import AmazonBedrockImageGenerationDriver
from griptape.drivers.image_generation_model.bedrock_stable_diffusion import (
    BedrockStableDiffusionImageGenerationModelDriver,
)
from griptape.loaders import ImageLoader
from griptape.structures import Pipeline
from griptape.tasks import VariationImageGenerationTask

# Create a driver configured to use Stable Diffusion via Bedrock.
driver = AmazonBedrockImageGenerationDriver(
    image_generation_model_driver=BedrockStableDiffusionImageGenerationModelDriver(),
    model="stability.stable-diffusion-xl-v1",
)


# Load input image artifact.
image_artifact = ImageLoader().load("tests/resources/mountain.png")

# Instantiate a pipeline.
pipeline = Pipeline()

# Add a VariationImageGenerationTask to the pipeline.
pipeline.add_task(
    VariationImageGenerationTask(
        input=("{{ args[0] }}", image_artifact),
        image_generation_driver=driver,
        output_dir="images/",
    )
)

pipeline.run("An image of a mountain landscape on a snowy winter day")

Inpainting Image Generation Task

The Inpainting Image Generation Task generates an image using an input image, a mask image, and a text prompt. The input image will be modified within the bounds of the mask image as requested by the text prompt.

from griptape.drivers.image_generation.amazon_bedrock import AmazonBedrockImageGenerationDriver
from griptape.drivers.image_generation_model.bedrock_stable_diffusion import (
    BedrockStableDiffusionImageGenerationModelDriver,
)
from griptape.loaders import ImageLoader
from griptape.structures import Pipeline
from griptape.tasks import InpaintingImageGenerationTask

# Create a driver configured to use Stable Diffusion via Bedrock.
driver = AmazonBedrockImageGenerationDriver(
    image_generation_model_driver=BedrockStableDiffusionImageGenerationModelDriver(),
    model="stability.stable-diffusion-xl-v1",
)

# Load input image artifacts.
image_artifact = ImageLoader().load("tests/resources/mountain.png")

mask_artifact = ImageLoader().load("tests/resources/mountain-mask.png")

# Instantiate a pipeline.
pipeline = Pipeline()

# Add an InpaintingImageGenerationTask to the pipeline.
pipeline.add_task(
    InpaintingImageGenerationTask(
        input=("{{ args[0] }}", image_artifact, mask_artifact), image_generation_driver=driver, output_dir="images/"
    )
)

pipeline.run("An image of a castle built into the side of a mountain")

Outpainting Image Generation Task

The Outpainting Image Generation Task generates an image using an input image, a mask image, and a text prompt. The input image will be modified outside the bounds of a mask image as requested by the text prompt.

from griptape.drivers.image_generation.amazon_bedrock import AmazonBedrockImageGenerationDriver
from griptape.drivers.image_generation_model.bedrock_stable_diffusion import (
    BedrockStableDiffusionImageGenerationModelDriver,
)
from griptape.loaders import ImageLoader
from griptape.structures import Pipeline
from griptape.tasks import OutpaintingImageGenerationTask

# Create a driver configured to use Stable Diffusion via Bedrock.
driver = AmazonBedrockImageGenerationDriver(
    image_generation_model_driver=BedrockStableDiffusionImageGenerationModelDriver(),
    model="stability.stable-diffusion-xl-v1",
)


# Load input image artifacts.
image_artifact = ImageLoader().load("tests/resources/mountain.png")

mask_artifact = ImageLoader().load("tests/resources/mountain-mask.png")

# Instantiate a pipeline.
pipeline = Pipeline()

# Add an OutpaintingImageGenerationTask to the pipeline.
pipeline.add_task(
    OutpaintingImageGenerationTask(
        input=("{{ args[0] }}", image_artifact, mask_artifact),
        image_generation_driver=driver,
        output_dir="images/",
    )
)

pipeline.run("An image of a mountain shrouded by clouds")

Structure Run Task

The Structure Run Task runs another Structure with a given input. This Task is useful for orchestrating multiple specialized Structures in a single run. Note that the input to the Task is a tuple of arguments that will be passed to the Structure.

Assistant Task

The Assistant Task enables Structures to interact with various "assistant" services using Assistant Drivers.

Text to Speech Task

This Task enables Structures to synthesize speech from text using Text to Speech Drivers.

Audio Transcription Task

This Task enables Structures to transcribe speech from text using Audio Transcription Drivers.

← Prev

Workflows

↑ Up

Agents

Next →

Task Memory and Off Prompt

Could this page be better? Report a problem or suggest an addition!