Skip to main content

​
Creating the Embeddings Table

Each resource from the resources table will have to be chunked, embedded, and then stored in a new table called embeddings. Create a new file (lib/db/schema/embeddings.ts) and add the following code:
lib/db/schema/embeddings.ts
import { nanoid } from "@/lib/utils";
import { index, pgTable, text, varchar, vector } from "drizzle-orm/pg-core";
import { resources } from "./resources";

export const embeddings = pgTable(
  "embeddings",
  {
    id: varchar("id", { length: 191 })
      .primaryKey()
      .$defaultFn(() => nanoid()),
    resourceId: varchar("resource_id", { length: 191 }).references(
      () => resources.id,
      { onDelete: "cascade" }
    ),
    content: text("content").notNull(),
    embedding: vector("embedding", { dimensions: 1536 }).notNull(),
  },
  (table) => ({
    embeddingIndex: index("embeddingIndex").using(
      "hnsw",
      table.embedding.op("vector_cosine_ops")
    ),
  })
);

​
Structure of the embeddings table

This table has four columns:

id

Unique identifier - Primary key for each embedding record

resourceId

Foreign key relation - Links to the full source material in the resources table

content

Plain text chunk - The actual text content that was chunked

embedding

Vector representation - The vector representation of the plain text chunk
To perform similarity search, you also need to include an index (HNSW or IVFFlat) on the embedding column for better performance. The schema above includes an HNSW index using cosine similarity operations.
HNSW Index: Hierarchical Navigable Small World index for efficient approximate nearest neighbor search
Cosine Similarity: Measures the cosine of the angle between two vectors, commonly used for text similarity

​
Update the database

To push this change to the database, run the following command:
terminal
pnpm db:push
This command will create the new embeddings table in your database. Make sure your database is running and accessible.

​
Check the database

Check that your database has the new embeddings table as explained in theVerify Database Setup section.
⌘I