System Prompt

System Prompt refers to a special, typically developer-controlled message that precedes the user turn in an LLM API call. It sets the operating context: who the model is, what it can and cannot do, what format it should use, and what domain knowledge it should prioritize. System prompts are invisible to end users in most product implementations.

How it works

Chat-tuned LLMs are trained to treat the system role as authoritative instructions that override competing directions from user turns. In practice, the system prompt is concatenated with the conversation history before the model generates a reply. Providers like Anthropic and OpenAI expose a dedicated system field in their API message schema rather than prepending it manually.

Key facts

• Role hierarchy: System overrides user, which overrides assistant in most aligned chat models.
• Caching opportunity: Static system prompts are prime candidates for prompt caching, which can cut costs on long prompts by 80-90 percent.
• Injection risk: Malicious user input can attempt to override system prompt instructions, a class of attack called prompt injection.
• Length: System prompts in production apps routinely run 1,000 to 10,000 tokens.

For builders

A well-crafted system prompt is the primary tool for customizing a general-purpose LLM into a domain-specific assistant. Builders should treat system prompts with the same discipline as application code: version them, test changes against an eval suite, and store them in a configuration layer separate from business logic.

Sources

• Wei, J., et al. (2022). Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. arXiv:2201.11903. arxiv.org
• Brown, T., et al. (2020). Language Models are Few-Shot Learners. arXiv:2005.14165. arxiv.org
• Anthropic. Prompt engineering best practices. anthropic.com
• OpenAI. Prompt engineering guide. platform.openai.com
• NIST. (2023). AI Risk Management Framework. nist.gov