Treating the LLM Pipeline as Hyperparameter Tuning: Hands-On Bias Detection and Fact-checking

Most teams building with LLMs still rely on manual prompt tweaking and "vibe checks"—an approach that quickly breaks down in production. Building reliable generative AI systems requires applying core machine learning engineering principles: proper data splits, quantitative evaluation metrics, and automated optimization. In this three-hour hands-on workshop, inspired by the real-world engineering behind Debiasify, you will treat an end-to-end LLM pipeline as a measurable machine learning optimization problem. Using manipulation detection and automated fact-checking as our working case study, we will replace trial-and-error prompting with rigorous ML methodology in Python. The techniques you learn are directly transferable to classification, information extraction, and RAG applications in your own production systems. Workshop Structure (3 Hours): Part 1: Reproducible Baselines & Real Evaluation (45 min) Connect self-hosted open-weight models (vLLM) and cloud APIs through a unified LiteLLM gateway. Formulate manipulation and logical fallacy detection as multi-label and text-span classification tasks, evaluating model outputs against human-annotated ground truth using precision, recall, F1, and span overlap metrics. Part 2: Automated Pipeline & Prompt Tuning (60 min) Bridge traditional hyperparameter optimization (HPO) and LLM engineering. Treat system prompts, structured output schemas, and few-shot examples as tunable parameters, running automated optimization over training and validation splits while keeping the test set strictly held out. Part 3: Evidence-Grounded Fact-Checking with RAG (45 min) Build a claim extraction and retrieval pipeline that classifies claims as supported, contradicted, or unverifiable based on retrieved evidence. Quantitatively evaluate retrieval quality, citation faithfulness, and hallucination rates. Part 4: Production Trade-Offs & Pareto Optimization (30 min) Benchmark detection F1 against inference latency, token cost, and prompt-injection robustness across different models. Explore Pareto-optimal configurations to identify the best trade-offs for production deployment. What You Will Take Home: A working Python prototype, reusable evaluation tools, and a practical blueprint for optimizing LLM workflows using disciplined ML engineering practices. Target Audience & Prerequisites: Designed for ML engineers, data scientists, and AI developers with basic Python proficiency, familiarity with standard ML metrics (precision/recall/F1), and basic experience calling LLM APIs. Setup: Bring your laptop. Preconfigured cloud model endpoints, hosted notebooks, and annotated datasets will be provided. No local GPU, heavy downloads, or paid API keys are required.