Forays into Reliability, Efficiency, and Energy of Language Models

abstract: Language model based AI systems have given us much to admire: complex reasoning, ability to write code, use tools, agentic solutions and much more. Reliability and efficiency have become key hurdles to making the leap from admiration to impactful adoption. In this talk, I will present three threads of research we pursue in addressing these hurdles: 1. Reliability -- We have created a range of benchmarks over the years to assess reliable reasoning. I will briefly present AppWorld, our benchmark for interactive coding agents, and a recent extension AppWorld-UL that requires user interaction to reduce uncertainty. I will also discuss our recent work on self-expressed uncertainty in LLM reasoning and its calibration issues. 2. Efficiency -- Small Language Models present a viable alt