System architectures

Detailed schema specifications, database triggers, and workflow diagrams of automated platforms I have engineered.

StartupWire Crawler & Curation Pipeline

Autonomous cron-triggered system crawling verified sources, indexing entries, and executing LLM validation sweeps.

// PIPELINE DATA-FLOW MAP


[Verified RSS Feeds]
       |
       v  (cron: every 4h)
[Node.js Scraper Worker]
       |
       +---> [Zod Schema Validation]
       |
       v
[Google Gemini API] 
 (Summary, Taxonomy, Sentiment Curation)
       |
       v
[Vector Caching Module] <--> [Supabase pgvector] (Cosine similarity check)
       |
       v (Save if distance > 0.15)
[PostgreSQL Database]
       |
       v (Revalidation trigger)
[Next.js Edge Page]

Pipeline Nodes

Scraper WorkerNode.js, Axios, RSS-Parser

Fetches raw feed XML and extracts article nodes.

AI Curation AgentGemini 1.5 Flash API

Filters noise, writes bullet-point summaries and generates structured JSON content.

pgvector DeduplicationSupabase PostgreSQL

Checks cosine distance between candidate text embeddings and historical titles.

Edge Cached FrontendNext.js 16 Edge Runtime

Renders fully static semantic HTML with Edge middleware updates.

Engineering Rationale

Using Next.js edge caching combined with database-level triggers ensures the frontend loads in under 200ms while eliminating expensive real-time API calls for news articles.