Context & Embeddings¶

Embeddings are the foundation of COORDINATOR's AI capabilities. Files, session data, and other content are converted into vector embeddings that power chat responses and training.

Concepts¶

Concept	Description
Embedding	Vector representation of content for semantic search
Context	Accumulated embeddings available to the AI
Content Type	Category of the source content
Football Related	Whether content relates to sports data

Embedding Lifecycle¶

Upload → Processing → Completed/Failed

Status	Description
`pending`	Uploaded, awaiting processing
`processing`	Being converted to embeddings
`completed`	Ready for use
`failed`	Processing failed

Types¶

From types/embedding.ts:

interface Embedding {
  id: string;
  filename: string;
  contentType: string;
  status: "pending" | "processing" | "completed" | "failed";
  partIndex?: number;
  confidence?: number;
  footballRelated?: boolean;
  caption?: string;
  detailedDescription?: string;
  metadata?: Record<string, unknown>;
  createdAt: string;
}

interface EmbeddingsResponse {
  embeddings: Embedding[];
  total: number;
}

Hooks¶

useEmbeddings¶

hooks/embeddings/use-embeddings.ts

List embeddings and upload files.

const {
  embeddings,      // Embedding[]
  total,           // number
  isLoading,
  uploadFiles      // (files: File[]) => Promise<void>
} = useEmbeddings(organizationSlug, { params });

API Calls: - GET /embedding/data?limit=100 - List embeddings - POST /embedding/upload - Upload file (FormData)

Upload Implementation:

const uploadFiles = async (files: File[]) => {
  const results = await Promise.allSettled(
    files.map(async (file) => {
      const formData = new FormData();
      formData.append("file", file);
      return fetchJSON("/embedding/upload", token, {
        method: "POST",
        body: formData
      });
    })
  );

  // Report success/failure for each file
  results.forEach((result, i) => {
    if (result.status === "fulfilled") {
      toast.success(`Uploaded ${files[i].name}`);
    } else {
      toast.error(`Failed: ${files[i].name}`);
    }
  });

  queryClient.invalidateQueries(["embeddings"]);
};

useEmbedding¶

hooks/embeddings/use-embedding.ts

Manage a single embedding with media access.

const {
  embedding,       // Embedding
  mediaUrl,        // string (blob URL for preview)
  isLoading,
  isPolling,       // Still processing
  deleteEmbedding  // () => Promise<void>
} = useEmbedding(organizationSlug, embeddingId);

API Calls: - GET /embedding/search-by-file?filename={id}&limit=1 - Find embedding - GET /embedding/{id}/media - Fetch media blob - DELETE /embedding/{id} - Delete embedding

Polling:

// Polls every 5s while status is processing
refetchInterval: embedding?.status === "processing" ? 5000 : false

useSearchEmbeddings¶

hooks/embeddings/use-search-embedding.ts

Search embeddings with various strategies.

const { embeddings, isLoading } = useSearchEmbeddings(organizationSlug, {
  searchQuery,     // Text search
  filename,        // Filename search
  contentType,     // Filter by type
  footballRelated, // Filter by sports relevance
  limit
});

API Calls: - GET /embedding/search?query={text} - Semantic text search - GET /embedding/search-by-file?filename={name} - Filename match - GET /embedding/search-similar-content?contentType={type} - Similar content

Pages¶

Context Overview¶

/portal/[slug]/[teamID]/context

Tutorial cards introducing the context workflow: 1. Upload files 2. Process embeddings 3. Use in chat 4. Train models

Upload Files¶

/portal/[slug]/[teamID]/context/upload

File upload interface: - Drag-and-drop zone - File type validation - Progress indicators

Components: - FileDropzone - Drop area - FileUploadPopover - Upload trigger - SelectedFilesList - Pending uploads

Manage Embeddings¶

/portal/[slug]/[teamID]/context/manage

Tabbed interface for browsing embeddings:

Tab	Description
All Embeddings	Full list with pagination
Text Search	Semantic search by query
File Search	Search by filename
Similar Content	Find by content type

Embedding Detail¶

/portal/[slug]/[teamID]/context/manage/[embeddingID]

Displays: - Media preview (image/video/audio/PDF) - Metadata fields: - Content type - Part index - Confidence score - Football related flag - Caption - Detailed description - Raw metadata JSON - Delete action

Supported File Types¶

The system processes various content types:

Category	Types
Images	PNG, JPEG, GIF, WebP
Video	MP4, WebM, MOV
Audio	MP3, WAV, M4A
Documents	PDF, TXT, Markdown
Data	JSON, CSV

Components¶

FileDropzone¶

components/upload/file-dropzone.tsx

Drag-and-drop upload zone:

<FileDropzone
  onFilesSelected={(files) => setSelectedFiles(files)}
  accept={{ "image/*": [], "video/*": [], "application/*": [] }}
/>

EmbeddingsTable¶

components/tables/embeddings/embeddings-table.tsx

Data table with: - Filename, status, content type - Created date - Actions (view, delete)

EmbeddingCard¶

components/cards/embedding-card.tsx

Card view for embedding preview.

Media Preview¶

The embedding detail page renders media based on content type:

const renderMedia = () => {
  if (!mediaUrl) return null;

  if (contentType?.startsWith("image/")) {
    return <img src={mediaUrl} alt={filename} />;
  }
  if (contentType?.startsWith("video/")) {
    return <video src={mediaUrl} controls />;
  }
  if (contentType?.startsWith("audio/")) {
    return <audio src={mediaUrl} controls />;
  }
  if (contentType === "application/pdf") {
    return <iframe src={mediaUrl} />;
  }
  return <a href={mediaUrl} download>Download</a>;
};

Integration with Sessions¶

Session timelines can be embedded:

// From session detail page
const handleEmbedTimeline = async () => {
  const { uploadFiles } = useEmbeddings(orgSlug);

  // Convert timeline items to files
  const files = timeline.map(item => 
    new File([JSON.stringify(item.data)], `${item.id}.json`)
  );

  await uploadFiles(files);
};

Integration with Chat¶

Embeddings provide context for AI chat:

User sends message
Backend searches embeddings semantically
Relevant context injected into prompt
AI responds with context awareness

Integration with Training¶

Training jobs use embeddings as context:

Select embeddings for training set
Configure training parameters
Model fine-tuned on selected context
Deployed model uses new knowledge