web_scraper

Suggest edits
  • __all__ = ['BaseWebScraperDriver'] module-attribute

Bases:

ABC
Source Code in griptape/drivers/web_scraper/base_web_scraper_driver.py
class BaseWebScraperDriver(ABC):
    def scrape_url(self, url: str) -> TextArtifact:
        source = self.fetch_url(url)

        return self.extract_page(source)

    @abstractmethod
    def fetch_url(self, url: str) -> str: ...

    @abstractmethod
    def extract_page(self, page: str) -> TextArtifact: ...

extract_page(page)abstractmethod

Source Code in griptape/drivers/web_scraper/base_web_scraper_driver.py
@abstractmethod
def extract_page(self, page: str) -> TextArtifact: ...

fetch_url(url)abstractmethod

Source Code in griptape/drivers/web_scraper/base_web_scraper_driver.py
@abstractmethod
def fetch_url(self, url: str) -> str: ...

scrape_url(url)

Source Code in griptape/drivers/web_scraper/base_web_scraper_driver.py
def scrape_url(self, url: str) -> TextArtifact:
    source = self.fetch_url(url)

    return self.extract_page(source)

BaseWebScraperDriver

MarkdownifyWebScraperDriver

MarkdownifyWebScraperDriver

ProxyWebScraperDriver

ProxyWebScraperDriver

TrafilaturaWebScraperDriver

TrafilaturaWebScraperDriver

← Prev

redis

↑ Up

drivers

Next →

base_web_scraper_driver

Could this page be better? Report a problem or suggest an addition!