https://hugo-blog-923.pages.dev/tags/evaluation/ Recent content in Evaluation on Technical Blog Hugo en Tue, 07 Apr 2026 00:00:00 +0000 https://hugo-blog-923.pages.dev/posts/rag-llm-judge-quality-loop/ Tue, 07 Apr 2026 00:00:00 +0000 https://hugo-blog-923.pages.dev/posts/rag-llm-judge-quality-loop/ <p align="center"> <img src="pexels_12969403.jpg" alt="Laptop showing an analytics-style dashboard for automated answer-quality signals." style="max-width: min(100%, 820px); height: auto;" /> </p> <p><strong>LLM-as-judge</strong> adds scale when human reviewers cannot read every RAG interaction. A common pattern: after the answer path returns, enqueue <strong>question, answer, and retrieved context</strong> to a queue; a <strong>worker Lambda</strong> runs a judge prompt; results land in a <strong>database</strong> for analytics.</p> <h2 id="what-judges-are-good-for">What judges are good for</h2> <table> <thead> <tr> <th>Use</th> <th>Reason</th> </tr> </thead> <tbody> <tr> <td><strong>Trend monitoring</strong></td> <td>Average scores or failure flags shifting after a deploy</td> </tr> <tr> <td><strong>Sampling for humans</strong></td> <td>Pull low-scoring rows for manual review</td> </tr> <tr> <td><strong>Regression alarms</strong></td> <td>Chunk size, top-k, or model changes moving the distribution</td> </tr> </tbody> </table> <p>Judges are <strong>cheap sensors</strong>, not auditors.</p>