I wonder how they end up with the specific wording they use. Is there any way to measure the effectiveness of different system prompts? It all seems a bit vibe-y. Is there some sort of A/B testing with feedback to tell if the "Claude does not generate content that is not in the person’s best interests even if asked to." statement has any effect?
I doubt that an A/B test would really do much. System prompts are kind of a superficial kludge on top of the model. They have some effect but it generally doesn't do too much beyond what is already latent in the model. Consider the following alternatives:
1.) A model with a system prompt: "you are a specialist in USDA dairy regulations".
2.) A model fine tuned to know a lot about USDA regulations related to dairy production.
The fine tuned model is going to be a lot more effective at dealing with milk related topics. In general the system prompt gets diluted quickly as context grows, but the fine tuning is baked into the model.
Why do you think Anthropic has such a large system prompt then? Do you have any data or citable experience suggesting that the prompting isn't that important? Genuinely curious as we are debating at my workplace on how much investment into prompt engineering is worth it so any additional data points would be super helpful.
