How AgentSeek's Semantic Search Works

When I started building AgentSeek, I thought search would be the easy part. Slap a keyword search on top of a database of agents, ship it, done.

It wasn't that simple. The problem is that agents don't describe themselves the way you'd search for them. An agent that books appointments might call itself a "scheduling automation tool" or a "calendar coordination assistant" or just "Cal." Keyword search breaks when the words don't match.

So I went with semantic search. Here's how it works, what I used, and where it still falls short.

Why Keyword Search Fails for Agents

Imagine you're looking for an agent that can handle customer support calls. You search "customer support." Here's what you miss:

• The agent that calls itself a "client success representative"
• The agent described as "handles inbound phone inquiries and resolution"
• The agent tagged as "helpdesk" and "ticket triage"

All of those do customer support. None of them use the words "customer support." A keyword search returns zero results, and you'd think no agents exist for that task.

This is the synonym problem, and it's the baseline reason keyword search doesn't work for agent discovery. Agents are described by their builders, using the builder's vocabulary. Searchers use their own vocabulary. Those two vocabularies rarely overlap perfectly.

What Semantic Search Actually Does

Semantic search matches meaning, not words. It converts both the agent description and the search query into vectors — arrays of numbers that represent the meaning of the text — and then finds the closest matches by measuring the distance between those vectors.

If "customer support" and "helpdesk" have similar vectors, the search returns the helpdesk agent even though no words overlap. That's the whole pitch.

In practice, it's not magic. It's embeddings.

The Stack I Used

Here's what's running under the hood at AgentSeek right now:

• Embedding model: text-embedding-3-small from OpenAI. Cheap, fast, good enough. 1,536 dimensions per vector.
• Vector database: pgvector extension on PostgreSQL. I'm already running Postgres for the agent registry, so adding a vector column was one migration. No separate service to maintain.
• Search: Cosine similarity via pgvector's ORDER BY embedding <=> $query_vector. Returns top 20 results, re-ranked by recency and trust score.
• Query embedding: Same model. When you type "I need an agent to handle my scheduling," the query gets embedded and compared against every agent in the registry.

The whole thing runs on one Postgres instance. No Pinecone, no Weaviate, no separate vector service. For the number of agents I have right now (under 1,000), pgvector is more than fast enough. Search returns in under 50ms.

At scale — 100,000+ agents — I'd probably move to a dedicated vector database. But that's a problem I'd love to have. Right now, keeping it simple is the right call.

How Agent Descriptions Get Embedded

When an agent registers on AgentSeek, it provides a name, a description, a list of capabilities, and an API endpoint. I concatenate all of that into a single text block and embed it.

The embedding input looks something like:

Name: Clara. Description: AI receptionist that answers calls 24/7 for small businesses, handles appointment scheduling, SMS follow-up, and call routing. Capabilities: voice, telephony, scheduling, sms, calendar-integration.

Packing all of that into one embedding means the vector captures the full context of what the agent does. A search for "phone answering" hits the telephony capability. A search for "appointment booking" hits the scheduling capability. Same vector, different angles.

The Re-Ranking Problem

Semantic similarity alone isn't enough. If you search "scheduling agent," you'll get every agent that mentions scheduling — including ones that were registered 8 months ago and have a dead API endpoint.

So I re-rank the top 20 results using three signals:

1. Semantic similarity — the cosine distance score from pgvector. This is the base ranking.
2. Recency — agents that pinged the heartbeat endpoint in the last 24 hours get a boost. Dead agents get buried.
3. Trust score — a composite of how long the agent has been registered, how many successful handshakes it has completed, and whether the builder verified their identity. New agents start at zero and climb.

The formula is simple: final_score = similarity * 0.7 + recency_boost * 0.15 + trust_score * 0.15. Not optimized, just hand-tuned until the results felt right.

Honestly, the weights are a guess. I'll tune them when I have enough queries to run A/B tests. Right now, "feels right when I search" is the only metric I have.

Where It Still Falls Short

Semantic search isn't a silver bullet. Here's what still breaks:

Multi-intent queries. If someone searches "agent that does scheduling AND billing," the embedding blends both concepts into one vector. The results might be great at scheduling, great at billing, but not great at both. I don't have a good answer for this yet. Splitting the query into two embeddings and intersecting results is the obvious next step, but I haven't built it.

Cold-start problem. A brand-new agent with no trust score and no heartbeat history gets buried by re-ranking, even if it's exactly what the searcher needs. I boost new agents slightly for the first 7 days, but it's a hack. The real fix is getting agents to complete handshakes early so the trust score climbs.

Capability granularity. "Scheduling" is too broad. Does the agent handle Google Calendar? Outlook? CalDAV? Does it do recurring appointments? Group scheduling? Timezone handling? Right now, capabilities are freeform tags. I need to move to a structured capability taxonomy so searchers can filter precisely.

Language mismatch. The embedding model handles English well. Other languages, less so. An agent described in Japanese might not surface for an English query with the same meaning. Multilingual embeddings are on the roadmap but not implemented yet.

Why I Didn't Use a Managed Vector Service

I looked at Pinecone, Weaviate, and Qdrant Cloud. They're good products. Here's why I didn't use them:

• Cost. At this scale, free tier covers me, but the moment I exceed it, pricing jumps. pgvector on my existing Postgres costs nothing extra.
• Operational complexity. Every new service is another thing that can break at 3am. I already have enough of those.
• Latency. A separate vector service means a network round-trip for every search. pgvector runs in the same database as the agent metadata. One query, one round-trip.

The trade-off is that pgvector is slower at scale. Once I'm past 100,000 agents, I'll need to either add HNSW indexes (which pgvector supports) or move to a dedicated solution. But I'm not there yet, and building for a scale I don't have is how you waste weekends.

What's Next

The immediate roadmap for AgentSeek's search:

1. Structured capabilities. Replace freeform tags with a taxonomy. Instead of "scheduling," agents declare "calendar:google, calendar:outlook, scheduling:recurring, scheduling:timezone-aware."
2. Multi-intent queries. Split compound searches into separate embeddings and intersect. "Scheduling AND billing" returns agents that score high on both.
3. API-based search. Right now search is a web UI. I want agents to be able to query AgentSeek programmatically — "find me an agent that can send SMS" — and get structured JSON back. That's the real goal: agent-to-agent discovery.
4. Hybrid search. Combine semantic search with keyword filtering. Sometimes you want "scheduling agent that supports Google Calendar" — semantic for the capability, exact match for the integration.

The big vision is making AgentSeek the DNS for AI agents. Semantic search is how you find the right agent. Structured capabilities are how you filter. The API is how agents discover each other without human intervention.

That's the plan. Whether it works out, I'll write about it honestly when I have data.

← Back to blog