Shlok Kumar (@sk2740) header
Shlok Kumar (@sk2740) profile picture

Shlok Kumar

@sk2740

Developer | Turning code into solutions

 India
linktr.ee/shlokkumar2303
Joined November 2017
  • Tweets449
  • Following538
  • Followers105
1/6:Q "How do you build an evaluation pipeline for a frontier model? What makes an eval trustworthy?" Building a strong eval pipeline starts with curated, held-out test data + contamination checks (e.g., detecting if test examples leaked into training).
1
15
more replies
6/6: How do you evaluate open-ended generation? No single ground truth—use rubrics for fluency/helpfulness/truthfulness. Combine LLM judges (with strong prompts), human raters, & rule-based metrics (n-grams). Test on varied prompts; measure consistency via inter-rater agreement.
1
8
#x #ai #interview
2
1
10
1/6: Q: "Compare chain-of-thought prompting vs training a reward model / verifier for selecting among multiple completions." CoT prompting: Generate step-by-step reasoning in one pass. Simple, no extra training. Higher latency/tokens per query due to longer output. Low data req.
1
10
more replies
6/6: Follow-up: How to evaluate if extra reasoning improved the answer? Use benchmarks (accuracy on held-out tests), compare final answer correctness, human eval of reasoning quality, or verifier scores on consistency/validity. Self-consistency across samples also helps.
1
6
#x #ai #interview
4
4/6 When it wastes resources: Simple queries (fact lookup, creative writing, fast chat). Diminishing returns kick in fast — after a point, extra compute gives tiny or zero gain but multiplies cost & latency.
1
17
more replies
6/6 "How would you decide at serving time whether to allocate extra compute for a given query?" Use a lightweight router/classifier (or confidence score) that looks at query difficulty, user tier, or early model uncertainty — then dynamically choose CoT depth, N sample
1
11
#x #ai #interview
11
2/6 Examples: • Chain-of-Thought (CoT) prompting • Best-of-N sampling (generate N answers, pick best) • Verifier-guided search / process reward models / tree search All trade more tokens/compute for better reasoning.
1
13
3/6 When it helps: Hard reasoning tasks (math, coding, science, complex planning). Accuracy often scales reliably with more compute (especially with good verifiers). Gains are biggest on problems where base model is uncertain.
1
9
1/6 Q: What is test-time compute scaling (e.g., chain-of-thought, best-of-N, verifier-guided search)? When does it help and when does it waste resources? Test-time compute scaling = spending extra inference-time FLOPs at serving to boost accuracy, instead of only training models.
1
14
Load more