Towards Reliable Multi-step Reasoning in Question Answering

Multi-step has seen much empirical progress on many datasets recently, especially in Question Answering. However, training and evaluating on typical crowdsourced datasets is problematic because of the potential for shortcut reasoning based on artifacts. Even in large language models (LLMs) multi-step reasoning is challenging because of their tendency to hallucinate. What can we do about this? In this three part talk, I will first show how we can formalize and measure disconnected reasoning, a type of bad multihop reasoning. I will then discuss how we can construct new datasets using a bottom-up construction process, which allows us to better control for desired properties in the resulting dataset. In the second part, I will briefly present how synthetically generated data can be used to teach a broad range of multihop skills in a reliable manner and how to improve reliable multi-step reasoning in open-domain QA settings. And last, I will briefly discuss using iterative retrieval and reasoning as a means to reduce hallucination in LLM based QA.