Running Search Quality Evaluations with the Cashmere Eval Template

Cashmere's eval query template gives your team a structured way to test search quality — logging queries, expected documents, connector responses, and scores in one place so you can track relevance and recency over time.

Before you start: You'll need access to your Cashmere search connector and a copy of the eval template. Make a copy of the Cashmere Eval Queries Template to your own Drive before filling it in.

Overview

• Log test queries alongside the specific documents you'd expect each one to return.
• Score each result on a 1–5 scale and capture notes on why a result succeeded or fell short.
• Track test dates so you can compare quality across connector updates or collection changes.
• Share the sheet across your team for collaborative scoring.

Important: Keep one row per query per test run. If you re-run the same query after a change, add a new row with a new date rather than overwriting the old one — this preserves your history.

Common Workflows

Setting up your copy

1. Open the Cashmere Eval Queries Template.
2. Click File → Make a copy and save it to your own Google Drive or a shared team folder.
3. Read the Instructions tab before starting — it covers the column definitions and scoring key.

Filling in the Eval Queries tab

Each row represents one query/test combination. Fill in the following columns:

• Query — The exact text you submitted to the connector (copy-paste from the search interface or API call).
• Expected Documents — The specific titles, authors, or editions you'd expect the query to surface. Be explicit: Smith, J. (2022). *Introduction to Machine Learning*, 3rd ed. is more useful than an ML textbook.
• Response — What the connector actually returned. Paste the top result(s) or a brief summary.
• Score (1–5) — Use the dropdown to select a score (see the scoring key below).
• Notes — Why did it score that way? Note any relevance gaps, unexpected results, or recency issues.
• Date — The date you ran this query. Use a consistent format (YYYY-MM-DD recommended).

Scoring key

Score	Meaning
5	Excellent — expected documents returned, highly relevant
4	Good — mostly relevant, minor gaps
3	Acceptable — partially relevant or missing some expected documents
2	Poor — mostly off-target or missing key documents
1	Failing — irrelevant or no useful results

Tracking quality over time

• Run your eval set before and after any significant collection or connector change.
• Use the Date column to filter or sort results in Sheets and compare scores across runs.
• For a quick health check, sort by Score ascending to surface your weakest queries first.

Tips

• Vary query types. Mix thematic queries ("books on climate policy"), bibliographic queries ("Smith carbon tax 2021"), and summary/topic queries to get a well-rounded picture.
• Use consistent scorers. If multiple people score the same queries, calibrate first — run a few together and align on what a 3 vs. a 4 looks like for your collection.
• Document your connector config. Add a note in the Instructions tab (or a separate tab) recording which connector version or collection you were testing against when you ran each batch.

Not getting the results you expect?

Before tuning your connector or reaching out for support, run through these two checks first — they account for the majority of "missing result" reports:

1. Is the document actually in Cashmere?

Search only returns content that has been ingested into Cashmere. If a title isn't in your corpus, it won't appear in results regardless of how the query is phrased. You can verify by calling GET /api/v2/omnipubs with your API key and checking whether the title appears. If it's missing, it needs to be ingested before it can be searched.

2. Is that content scoped to your API key?

Even if a document exists in Cashmere, your API key only searches collections it has been granted access to. Go to Settings → API Keys, open the key you're using, and confirm the relevant collection is included. If it isn't, update the key's collection scope.

If both checks pass and results still aren't what you expect, there are further things you can tune:

• Prompt patterns — How you phrase queries to the MCP connector has a significant impact on relevance. The Cashmere MCP Tools guide covers when to use search_publications vs. get_publication, and includes example prompts for better results.
• Search model and recency — The /search API supports multiple search models, including options for recency boosting and semantic-only retrieval. If your eval results suggest the connector is surfacing outdated content or missing recent titles, you may be able to improve this by changing the search model on your API key. See AI Search on Licensed Content for details.
• Collection and filter scope — Narrow filters can improve precision; too-narrow filters may exclude relevant content.

If you've worked through all of the above and are still seeing poor eval scores, contact support@cashmere.io with your eval sheet and a description of what you're seeing.