Chain Of Thought (COT)
Research Paper : Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Chain of thought in Prompt Engineering allows the LLM to achieve complex reasoning abilities through intermediate steps. Combining this this with Few Shot Prompting will help us achieve better results in tasks that require more complex reasoning
Example prompt
The odd numbers in this group add up to an even number: 4, 8, 9, 15, 12, 2, 1.
A: Adding all the odd numbers (9, 15, 1) gives 25. The answer is False.
The odd numbers in this group add up to an even number: 17, 10, 19, 4, 8, 12, 24.
A: Adding all the odd numbers (17, 19) gives 36. The answer is True.
The odd numbers in this group add up to an even number: 16, 11, 14, 4, 8, 13, 24.
A: Adding all the odd numbers (11, 13) gives 24. The answer is True.
The odd numbers in this group add up to an even number: 17, 9, 10, 12, 13, 4, 2.
A: Adding all the odd numbers (17, 9, 13) gives 39. The answer is False.
The odd numbers in this group add up to an even number: 15, 32, 5, 13, 82, 7, 1.
A:with this prompt we get a very good result
Adding all the odd numbers (15, 5, 13, 7, 1) gives 41. The answer is False.from the paper (which i haven’t read yet) tell that Chain Of Thought (COT) is something that the newer LLM will come with by default.
In the above image we have just asked the model to think step by step in a Zero Shot Prompting method and able to get the right answer.
An example prompt
I went to the market and bought 10 apples. I gave 2 apples to the neighbor and 2 to the repairman. I then went and bought 5 more apples and ate 1. How many apples did I remain with?Let's think step by step.The output
First, you started with 10 apples.You gave away 2 apples to the neighbor and 2 to the repairman, so you had 6 apples left.Then you bought 5 more apples, so now you had 11 apples.Finally, you ate 1 apple, so you would remain with 10 apples.It’s impressive that this simple prompt is effective at this task. This is particularly useful where you don’t have too many examples to use in the prompt.
Automatic Chain of Thought (Auto-CoT)
when applying Chain Of Thought (COT) prompting with demonstrations the process involves a lot of handcrafting effective and diverse examples. This manual effort could also lead to sub-optimal results.
Well you could think that we could generate all the examples with “Let’s think step by step”, but in doing so there is a possibility of generating wrong reasoning chains.
This work proposes Auto-CoT, which samples questions with diversity and generates reasoning chains to construct the demonstrations.
Auto - COT consists of two main stages:
- question clustering: partition questions of a given dataset into a few clusters
- demonstration sampling: select a representative question from each cluster and generate it’s reasoning using Zero Shot CoT with simple heuristics (instructions)
The simple heuristics could be length of questions (e.g., 60 Tokenizer) and number of steps in rationale (e.g., 5 reasoning steps). This encourages the model to use simple and accurate demonstrations.
References