AI Prompt Engineering - Use Code not Words
AI language models don’t actually reason in a human sense. For those interested in how these systems are trained, I recommend checking out Demystifying LLMs with Andrej Karpathy.
The Token Challenge
When processing text, language models work with “tokens” rather than complete words. The relationship between words and tokens isn’t always one-to-one. For instance, the term “LLM” gets split into two separate tokens in the paragraph below.
Similarly, longer or unusual strings can be divided into numerous tokens. The word “SuperCaliFragilisticExpialiDociouc” is broken down by GPT-4o into 11 distinct tokens.
It’s important to understand that AI responses are generated probabilistically, one token at a time, with deliberate randomness incorporated. This explains why asking the same question multiple times often yields different answers.
These fundamental characteristics create significant constraints when AI attempts text analysis tasks. For example, until recently, many language models would incorrectly count the number of ’r’s in “strawberry.” This error appears to have been addressed either through supervised fine-tuning or because countless web pages discussing this specific error (with the correct answer) have now been incorporated into training data.
The Code Solution
One prompt enhancement is to ask the AI to answer a question by writing and executing code (normally in Python) rather than by attempting to provide the answer by referencing its LLM model. The AI will:
- Generate appropriate code
- Execute that code
- Extract the result
- Present you with the answer
This approach doesn’t require you to know programming or have programming language installed on your system.
If the AI doesn’t automatically show its code, you can specifically request to see the code used to generate the answer. In one test case, Google’s Gemini was quite misleading when it displayed the correct code but still provided an incorrect final answer. Gemini includes a specific button that must be pressed to get the result directly from the code execution, which did produce the correct answer.
The code-based approach is most effective for topics involving logic, data processing, and procedural operations. Creative tasks like poetry generation typically benefit more from the language model’s native capabilities rather than code-based solutions.
Follow This, That and (Maybe), the Other:
Comments
Post a Comment