Managing Data with Libraries in Gen AI Builder

What are Libraries

Libraries are a fundamental component of Gen AI Builder. They enable you to connect, organize, and prepare your data for use in AI-powered applications.

Libraries serve as the primary mechanism for managing knowledge sources that your Griptape Structures — including Agents, Pipelines, and Assistants — leverage for Retrieval Augmented Generation (RAG) tasks.

In short: Libraries are the curated, organized collections of information your AI can "know" and retrieve.

Why use Libraries

Libraries allow you to:

• Centralize data for AI: Aggregate diverse data sources into unified, accessible knowledge pools.
• Power Retrieval Augmented Generation (RAG): Provide contextual data to AI structures for accurate, grounded responses.
• Fuel Assistants and other Structures: Make relevant content queryable and available to Griptape-based Agents and Pipelines.
• Simplify data management: Manage the full data lifecycle — connection, processing, indexing — through an intuitive UI.
• Support Knowledge Bases: Libraries feed content into the Knowledge Bases that your AI Assistants ultimately query.

How Libraries fit into Gen AI Builder

The content pipeline in Gen AI Builder looks like this:

Data Sources → Libraries → Knowledge Bases → Retrievers → Assistants → AI Applications

• Data Sources: Raw content (web pages, documents, databases, cloud storage, APIs)
• Libraries: Organize and index content from Data Sources
• Knowledge Bases: Searchable layer built from Libraries
• Retrievers: Define how Assistants query Knowledge Bases
• Assistants: Use retrieved content to generate grounded AI responses

For background on key concepts, see:

• AI Factory Concepts
• Knowledge Bases explained
• Retrievers explained
• Retrieval Augmented Generation (RAG)

To see how Libraries are used in Hybrid Manager deployments, visit:

• Hybrid Manager: Using Gen AI Builder

Components of Libraries

Data Sources

Libraries connect to one or more Data Sources, including:

• Web Pages
• PG.AI Data Lake
• Amazon S3
• Google Drive
• Atlassian Confluence
• Custom Data Sources (via Griptape Structures)

Learn how to configure Data Sources:

• Configure Data Sources

Knowledge Bases

Libraries feed content into Knowledge Bases, which index and optimize content for fast, semantic search.

Learn how to configure Knowledge Bases:

• Configure Knowledge Bases

Retrievers

Retrievers define how Assistants query Knowledge Bases — selecting and formatting retrieved content for AI use.

Learn how to configure Retrievers:

• Configure Retrievers

When to use Libraries

Use Libraries whenever:

• You want to manage content pipelines for AI applications.
• You need to organize content from multiple Data Sources for use in Knowledge Bases.
• You are building Assistants that require grounded responses based on your content.

Common examples:

• Centralizing product documentation for support chatbots.
• Building corporate policy Libraries to power HR Agents.
• Creating Libraries of financial data to enable advanced RAG-based analytics.

Best practices for Libraries

• Focused content: Create Libraries with a clear, specific scope.
• Data quality: Ensure source data is accurate, current, and well structured.
• Incremental additions: Start small, expand incrementally, test at each step.
• Security: Follow the principle of least privilege for Data Sources.
• Regular review: Periodically review and update Libraries to maintain relevance and quality.

Next steps

To start working with Libraries:

1. Configure Data Sources
2. Configure Knowledge Bases
3. Configure Retrievers
4. Connect Libraries and Retrievers to Assistants to enable Retrieval Augmented Generation.

For procedural steps, see the How-to Guides in this section.

Related topics

• AI Factory Concepts
• Knowledge Bases explained
• Retrievers explained
• Retrieval Augmented Generation (RAG)
• Structures explained
• Hybrid Manager: Using Gen AI Builder