# Multimodal Knowledge Base Implementation Plan

> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.

**Goal:** Transform the text-only Knowledge Base into a multimodal system supporting documents, images, videos, and audio using Pinecone vector DB + Gemini Embedding 2, with per-tenant isolation and WhatsApp/voice agent integration.

**Architecture:** Files uploaded via dashboard are stored in Convex file storage (for serving). Gemini Embedding 2 generates embeddings for all media types. Embeddings stored in Pinecone (one shared serverless index, namespaces per tenant). Queries embed the question via Gemini Embedding 2, search Pinecone for top-K matches, retrieve media URLs from Convex, then pass context to Gemini 2.5 Flash for answer generation. Results include inline images/videos.

**Tech Stack:** Pinecone SDK (`@pinecone-database/pinecone`), Google GenAI SDK (`@google/genai`), Convex file storage, Next.js API routes, React components.

**Worktree:** `/Users/Asim/Desktop/mawidi_codex/mawidi-multimodal-kb`
**Branch:** `feature/multimodal-knowledge-base`

---

## Task 1: Install Dependencies + Environment Setup

**Files:**
- Modify: `mawidi-site/package.json`
- Verify: `mawidi-site/.env.local`

**Step 1: Install Pinecone SDK**

```bash
cd /Users/Asim/Desktop/mawidi_codex/mawidi-multimodal-kb/mawidi-site
npm install @pinecone-database/pinecone
```

**Step 2: Verify .env.local has required keys**

Confirm these exist:
```
PINECONE_API_KEY=pcsk_6rNr8Q_...
PINECONE_INDEX_NAME=mawidi-multimodal-kb
GOOGLE_API_KEY=AIzaSyC2Ns... (already exists)
```

**Step 3: Commit**

```bash
git add package.json package-lock.json
git commit -m "feat(kb): add Pinecone SDK dependency for multimodal knowledge base"
```

---

## Task 2: Extend Types + Constants for Multimodal Support

**Files:**
- Modify: `mawidi-site/lib/types/knowledge-base.types.ts`

**Step 1: Add missing RAG types and extend for multimodal**

Add these types/constants to `knowledge-base.types.ts`:

```typescript
// Media types for multimodal KB
export type KBMediaType = 'document' | 'image' | 'video' | 'audio';

export const MEDIA_TYPE_LABELS: Record<KBMediaType, { en: string; ar: string }> = {
  document: { en: 'Document', ar: 'مستند' },
  image: { en: 'Image', ar: 'صورة' },
  video: { en: 'Video', ar: 'فيديو' },
  audio: { en: 'Audio', ar: 'صوت' },
};

// Extended allowed file types (multimodal)
export const ALLOWED_MEDIA_TYPES = [
  // Documents
  'application/pdf',
  'application/msword',
  'application/vnd.openxmlformats-officedocument.wordprocessingml.document',
  'text/plain',
  'text/markdown',
  'text/csv',
  // Images
  'image/png',
  'image/jpeg',
  'image/webp',
  // Videos
  'video/mp4',
  'video/quicktime',
  // Audio
  'audio/mpeg',
  'audio/wav',
  'audio/mp4',
  'audio/x-m4a',
] as const;

export const MAX_MEDIA_SIZE = 50 * 1024 * 1024; // 50MB for video/audio
export const MAX_VIDEO_DURATION = 120; // seconds

export const MEDIA_FILE_TYPE_LABELS: Record<string, string> = {
  ...FILE_TYPE_LABELS,
  'image/png': 'PNG Image',
  'image/jpeg': 'JPEG Image',
  'image/webp': 'WebP Image',
  'video/mp4': 'MP4 Video',
  'video/quicktime': 'MOV Video',
  'audio/mpeg': 'MP3 Audio',
  'audio/wav': 'WAV Audio',
  'audio/mp4': 'M4A Audio',
  'audio/x-m4a': 'M4A Audio',
};

// Helper to determine media type from MIME
export function getMediaType(mimeType: string): KBMediaType {
  if (mimeType.startsWith('image/')) return 'image';
  if (mimeType.startsWith('video/')) return 'video';
  if (mimeType.startsWith('audio/')) return 'audio';
  return 'document';
}

// RAG types (were missing - needed by gemini-rag.service.ts)
export interface RAGCitation {
  documentId: string;
  documentTitle: string;
  excerpt: string;
  relevanceScore?: number;
  mediaType?: KBMediaType;
  mediaUrl?: string;
}

export interface RAGQueryResponse {
  success: boolean;
  answer: string;
  citations: RAGCitation[];
  mediaResults?: {
    type: KBMediaType;
    url: string;
    title: string;
    description?: string;
    relevanceScore: number;
  }[];
  metadata: {
    queryTimeMs: number;
    model: string;
    totalMatches?: number;
  };
}

// Extend KnowledgeBaseDocument for multimodal
export interface KnowledgeBaseMediaItem extends KnowledgeBaseDocument {
  mediaType: KBMediaType;
  mediaUrl?: string;         // Convex storage URL
  thumbnailUrl?: string;     // For video/image previews
  duration?: number;         // For video/audio in seconds
  dimensions?: { width: number; height: number }; // For images/videos
  embeddingId?: string;      // Pinecone vector ID
  aiDescription?: string;    // AI-generated description of media content
}
```

**Step 2: Commit**

```bash
git add mawidi-site/lib/types/knowledge-base.types.ts
git commit -m "feat(kb): extend types for multimodal support - media types, RAG interfaces, file constants"
```

---

## Task 3: Create Pinecone Multimodal Service

**Files:**
- Create: `mawidi-site/lib/services/pinecone-multimodal.service.ts`

This is the core service. It handles:
1. Pinecone index initialization (create if not exists)
2. Per-tenant namespace management
3. Embedding generation via Gemini Embedding 2
4. Upsert vectors with metadata
5. Multimodal query (text or image → similar results)
6. Delete vectors on media removal

**Key implementation details:**

- **Gemini Embedding 2 model**: `gemini-embedding-exp-03-07` (3072 dimensions)
- **Pinecone index**: serverless, cosine metric, 3072 dimensions
- **Namespace**: `tenant-{userId}` for isolation
- **Metadata stored in Pinecone**: mediaType, title, category, mimeType, convexFileId, mediaUrl, aiDescription, createdAt
- **For images/videos**: First generate an AI description using Gemini 2.5 Flash, then embed the description + visual content
- **For documents**: Extract text, chunk into ~1000 token segments, embed each chunk

The service should export:
- `initializeIndex()` — create Pinecone index if needed
- `embedAndUpsert(userId, file, metadata)` — embed media and store in Pinecone
- `queryMultimodal(userId, query, options)` — search by text or image
- `deleteVectors(userId, vectorIds)` — remove vectors
- `deleteNamespace(userId)` — remove all vectors for a tenant
- `getNamespaceStats(userId)` — count vectors per tenant

**Step 1: Write the service**

Full implementation ~200 lines. Key patterns:
- Singleton Pinecone client
- Lazy index creation
- Batch upsert for chunked documents
- Error handling with logger

**Step 2: Run typecheck**

```bash
cd mawidi-site && npx tsc --noEmit --pretty 2>&1 | head -30
```

**Step 3: Commit**

```bash
git add mawidi-site/lib/services/pinecone-multimodal.service.ts
git commit -m "feat(kb): add Pinecone multimodal service - embeddings, upsert, query, per-tenant namespaces"
```

---

## Task 4: Create Media Upload API Route (Extend Existing)

**Files:**
- Modify: `mawidi-site/app/api/dashboard/knowledge-base/documents/route.ts`
- Modify: `mawidi-site/lib/validators/knowledge-base.validators.ts`

**Changes:**
1. Accept multimodal file types (images, videos, audio) in addition to documents
2. On upload: store file buffer, call `pineconeMultimodalService.embedAndUpsert()`
3. Update document record with `mediaType`, `embeddingId`, `aiDescription`
4. For images/videos: generate AI description via Gemini before embedding

**Step 1: Update validators to accept new media types**

**Step 2: Update POST handler to process multimodal uploads**

**Step 3: Test with curl**

```bash
curl -X POST http://localhost:9000/api/dashboard/knowledge-base/documents \
  -H "x-csrf-token: TOKEN" \
  -F "file=@test-image.jpg" \
  -F "title=Test Image" \
  -F "category=general"
```

**Step 4: Commit**

```bash
git add mawidi-site/app/api/dashboard/knowledge-base/documents/route.ts mawidi-site/lib/validators/knowledge-base.validators.ts
git commit -m "feat(kb): extend upload API for multimodal files - images, videos, audio support"
```

---

## Task 5: Update Query API for Multimodal Results

**Files:**
- Modify: `mawidi-site/app/api/dashboard/knowledge-base/query/route.ts`

**Changes:**
1. Use Pinecone multimodal service instead of (or alongside) Gemini File Search
2. Return media results (images, videos) with URLs and relevance scores
3. Support image-based queries (upload image to find similar)
4. Pass retrieved context to Gemini 2.5 Flash for answer generation

**Step 1: Update query route to use Pinecone**
**Step 2: Test query returns media results**
**Step 3: Commit**

---

## Task 6: Update WhatsApp + N8N KB Query Routes

**Files:**
- Modify: `mawidi-site/app/api/whatsapp/knowledge-base/query/route.ts`
- Modify: `mawidi-site/app/api/n8n/kb/query/route.ts`

**Changes:**
1. Both routes now query Pinecone multimodal service
2. WhatsApp route: if image results found, include media URLs for Twilio MMS
3. N8N route: return structured media results for voice agent to describe

**Step 1: Update WhatsApp KB query**
**Step 2: Update N8N KB query**
**Step 3: Commit**

---

## Task 7: Create MediaUploadModal Component

**Files:**
- Create: `mawidi-site/components/dashboard/knowledge-base/MediaUploadModal.tsx`

Replace `DocumentUploadModal` with a new modal that:
1. Accepts all media types (drag-and-drop)
2. Shows file type icon (document/image/video/audio) based on what's dropped
3. Image preview before upload
4. Video thumbnail preview
5. Audio waveform or icon
6. Same metadata fields: title, description, category
7. Progress bar during upload + embedding
8. Bilingual labels (EN/AR)

**Step 1: Create component**
**Step 2: Verify it renders**
**Step 3: Commit**

---

## Task 8: Update KnowledgeBasePanel with Media Tabs + Grid

**Files:**
- Modify: `mawidi-site/components/dashboard/knowledge-base/KnowledgeBasePanel.tsx`

**Changes:**
1. Replace 2-tab layout (`Documents | FAQs`) with media-aware tabs:
   - `All | Documents | Images | Videos | Audio | FAQs`
2. Stats cards: update to show counts per media type
3. **Grid view for images**: thumbnails in a responsive grid (not list)
4. **List view for documents/audio**: keep existing list style
5. **Card view for videos**: thumbnail + duration badge
6. Each item shows: media type icon, title, category, status, relevance
7. Use `MediaUploadModal` instead of `DocumentUploadModal`
8. Add "Upload" button that auto-detects what tab you're on

**Step 1: Update panel component**
**Step 2: Verify tabs work**
**Step 3: Commit**

---

## Task 9: Update KnowledgeBaseQueryPanel for Inline Media Results

**Files:**
- Modify: `mawidi-site/components/dashboard/knowledge-base/KnowledgeBaseQueryPanel.tsx`

**Changes:**
1. Assistant messages can now include inline images and videos
2. Image results: show as clickable thumbnails in the response
3. Video results: show as embedded player or thumbnail with play button
4. Each media result shows relevance score (% match)
5. Sources section shows media type icon next to document title
6. Support image upload in query input (drag image to search for similar)

**Step 1: Update message rendering for media**
**Step 2: Add image upload to query input**
**Step 3: Commit**

---

## Task 10: Update Convex Schema for Media Tracking

**Files:**
- Modify: `mawidi-site/convex/schema/voice_agents.ts`

**Changes:**
Add fields to `knowledge_base_documents` table:
- `mediaType` (string: document|image|video|audio)
- `mediaUrl` (optional string - Convex storage URL)
- `thumbnailUrl` (optional string)
- `duration` (optional number - seconds)
- `dimensions` (optional object - width/height)
- `embeddingId` (optional string - Pinecone vector ID)
- `aiDescription` (optional string - AI-generated description)
- `pineconeNamespace` (optional string)

**Step 1: Update schema**
**Step 2: Push to Convex**

```bash
CONVEX_DEPLOYMENT=dev:tacit-chinchilla-978 npx convex dev --once
```

**Step 3: Commit**

---

## Task 11: Prisma Schema Update + Migration

**Files:**
- Modify: `mawidi-site/prisma/schema.prisma`

**Changes to KnowledgeBaseDocument model:**
Add columns:
- `mediaType String @default("document")`
- `mediaUrl String?`
- `thumbnailUrl String?`
- `duration Int?`
- `width Int?`
- `height Int?`
- `embeddingId String?`
- `aiDescription String? @db.Text`
- `pineconeNamespace String?`

**Step 1: Update schema**
**Step 2: Generate migration**

```bash
cd mawidi-site && npx prisma migrate dev --name add-multimodal-kb-fields
```

**Step 3: Commit**

---

## Task 12: Integration Tests

**Files:**
- Create: `mawidi-site/__tests__/services/pinecone-multimodal.test.ts`

**Test cases:**
1. `initializeIndex` creates index with correct dimensions
2. `embedAndUpsert` generates embedding and stores in Pinecone
3. `queryMultimodal` returns ranked results with media URLs
4. `deleteVectors` removes specific vectors
5. `getMediaType` helper returns correct type for MIME types
6. Per-tenant namespace isolation (tenant A can't see tenant B's data)

**Step 1: Write tests (mock Pinecone + Gemini APIs)**
**Step 2: Run tests**

```bash
cd mawidi-site && npx jest __tests__/services/pinecone-multimodal.test.ts --runInBand
```

**Step 3: Commit**

---

## Task 13: Final Verification + Cleanup

**Step 1: Full typecheck**
```bash
cd mawidi-site && npx tsc --noEmit
```

**Step 2: Lint**
```bash
cd mawidi-site && npm run lint
```

**Step 3: Run all KB tests**
```bash
cd mawidi-site && npx jest --testPathPattern="knowledge-base|pinecone" --runInBand
```

**Step 4: Final commit**

```bash
git add -A
git commit -m "feat(kb): multimodal knowledge base - complete implementation with Pinecone + Gemini Embedding 2"
```

---

## Execution Order (Parallelizable Groups)

**Group A (parallel - no dependencies):**
- Task 1: Install dependencies
- Task 2: Extend types
- Task 10: Convex schema
- Task 11: Prisma schema

**Group B (depends on Task 2 + 1):**
- Task 3: Pinecone multimodal service

**Group C (depends on Task 3):**
- Task 4: Upload API
- Task 5: Query API
- Task 6: WhatsApp/N8N routes

**Group D (depends on Task 2, parallel with Group C):**
- Task 7: MediaUploadModal
- Task 8: KnowledgeBasePanel
- Task 9: QueryPanel updates

**Group E (depends on all above):**
- Task 12: Integration tests
- Task 13: Final verification