Research

MSc Computer Science research papers, security analysis protocols, and AI curation summaries.

On the Reliability of AI Agent Curation Pipelines

2026-06-30AI Curation
Paper Abstract

Exploring prompt engineering limits and vector search indexing configurations to achieve deterministic structured JSON outputs from open-ended news data inputs.

1. The Challenge of Determinism Generative models are probabilistic. Achieving structured JSON news summaries (with mandatory keys, exact string arrays, and no markup wrapper) requires strict system formatting guidelines: ```json { "summary": "bulleted points", "category": "Technology | AI | Startups", "keywords": ["maximum 3 strings"] } ``` Enforcing this schema at the model layer is done using Gemini's structured output configuration: ```typescript responseSchema: Schema.json({ type: Type.OBJECT, properties: { ... } }) ```

2. Vector Deduplication Using `pgvector`, we convert titles into 768-dimension vectors and run cosine queries: ```sql SELECT title, 1 - (title_vector <=> candidate_vector) AS similarity FROM articles ORDER BY similarity DESC LIMIT 1; ``` Empirical testing shows that a cosine similarity threshold of `0.85` accurately filters duplicated content from different feeds while preserving sequels or continuous updates.