Basil Docs

v1.0.0

Basil Getting Started

Getting Started with Basil

Transform any website into a searchable knowledge base with AI-powered content scraping and semantic search.

What is Basil?

Basil is an AI-powered website content scraper and semantic search engine that transforms any website into a searchable knowledge base. With just one command, you can scrape, process, and deploy a powerful search interface for any website.

Intelligent Scraping

Automatic link discovery with rate limiting

AI Processing

Generate search patterns with Groq API

Semantic Search

Vector-based similarity search

Quick Start

One-Command Deployment

Get started with Basil in just one command. This will scrape, process, and serve your website:

python pipeline.py --url https://example.com --all

What this command does:

Scrapes the website

Discovers and downloads all linked pages with intelligent rate limiting

Processes content with AI

Generates search patterns and metadata using Groq API

Creates vector database

Builds ChromaDB collection for semantic search

Starts API server

Launches FastAPI server at http://localhost:8000

Step-by-Step Guide

Step 1: Scrape Website Content

Start by scraping the target website. This discovers all internal links and downloads page content:

python pipeline.py --url https://example.com --scrape

Performance: Basil scrapes at 1-2 pages per second with intelligent rate limiting to respect server resources.

Step 2: Process Content with AI

Use AI to generate search patterns and metadata for each page:

python pipeline.py --process

AI Power: Generates 10-50 natural language search patterns per page using the Groq API.

Step 3: Load into Vector Database

Create the ChromaDB vector database for semantic search:

python pipeline.py --load

Vector Search: Uses sentence transformers to create embeddings for sub-second search responses.

Step 4: Start the API Server

Launch the FastAPI server to begin searching:

python pipeline.py --serve

Ready to Search: API available at http://localhost:8000 with full OpenAPI documentation.

Test Your Setup

Once your server is running, test the search functionality:

Simple Search Query

curl "http://localhost:8000/search/query?q=website%20development&max_results=5"

JSON Search Request

                            curl -X POST "http://localhost:8000/search" \

  -H "Content-Type: application/json" \

  -d '{"query": "website development", "max_results": 5}'

API Documentation

Visit http://localhost:8000/docs for interactive API documentation powered by Swagger UI.

Next Steps

Installation Guide

Learn about Docker deployment, configuration options, and production setup.

Read Installation Guide

API Reference

Explore all available endpoints, parameters, and response formats.

View API Reference