Chain-of-Thought Reasoning

Chain-of-thought prompting was one of the most important prompt engineering discoveries. The core idea is simple: instead of asking the model to jump directly to an answer, you ask it to "think step by step" or provide examples that include intermediate reasoning. On multi-step tasks (arithmetic word problems, logical deductions, multi-hop questions), CoT can improve accuracy from near-random to near-human levels. The reason is that transformer models compute the answer in a fixed number of layers — reasoning steps in the output effectively give the model more "compute" by making each token a step in the reasoning chain.

There are two variants. Zero-shot CoT appends a trigger phrase like "Let's think step by step" or "Think through this carefully" to the prompt. This is simple and works surprisingly well. Few-shot CoT provides examples where the demonstrations include the full reasoning trace, not just the final answer. Few-shot CoT is more reliable because the examples demonstrate the expected depth and format of reasoning, preventing the model from producing shallow or tangential chains.

CoT has failure modes. Unfaithful reasoning — where the model produces a plausible-looking chain that leads to the wrong answer — is common when the task exceeds the model's capability. Long chains can drift off-topic or accumulate errors. And CoT adds significant token overhead, making it expensive for high-volume applications. The practical rule is: use CoT when the task requires more than one logical step, verify that the reasoning chain actually supports the final answer, and consider whether the latency and cost tradeoff is acceptable for your use case.