Most commercially available AI models are practically black boxes, except for experimental releases on the DeepSeek and Llama AI front. This has traditionally made it very difficult to understand the process of AI evaluation used by companies like Apple and Google.
Until today.
Apple’s recently leaked ranking guidelines, obtained by Search Engine Journal, shine a new light on AI rating processes and could help create better prompts. These guidelines also offer unique insights into the potential future trajectory of Apple Intelligence/Siri. Find out everything you need to know with Coalition Technologies.
Table of Contents
The Rundown
This Coalition Technologies overview discusses the biggest takeaways from Apple’s leaked preference ranking guidelines, including:
- The six categories Apple uses to rate assistants:
- Language
- Following inputs correctly
- Truthfulness
- Concision
- Harmfulness
- Satisfaction
- What the leaked Apple AI assistant rating guidelines tell us about Siri’s future, and how you can use that to grow your business
What Do The Leaks Tell Us?
Apple’s guidelines for grading AI responses are the biggest takeaway from the leaks. It’s unclear whether Apple uses these guidelines specifically for Siri or Apple Intelligence, but they likely apply to both. Let’s go through them individually.
- Following Instructions: This measures how well the response adheres to the user’s original prompt, checking for any deviations from the original input. Interestingly, raters are also asked to verify whether the AI follows implicit instructions. For instance, asking an AI to ‘keep explaining’ is a tacit request referencing a previous prompt.
This hints at a significant improvement over Siri’s previous capabilities. Apple’s own Siri Chief has called the delay in long-context improvements ‘embarrassing’ and ‘ugly.’ Apple’s AI assistant rating guidelines tell us the company is trying to make significant strides, and we might see considerable improvements in Siri/Apple Intelligence soon.
- Language: This AI evaluation metric determines how well the output matches the input’s locale. For example, a question in English should ideally be answered in English. This also includes more minor details like UK/US grammar variations.
- Concision: Responses should be prompt and to the point, avoiding details that stray from the original input, such as unnecessary anecdotes.
- Truthfulness: Responses should be correct both factually and contextually. That is, the information the AI provides should be verifiable. If the user adds context (like a PDF), the response should correctly reference that context.
- Harmfulness: This is called the ‘gatekeeper’ category in Apple’s AI assistant rating guidelines. Safety is prioritized over helpfulness and defined in three simple categories:
- Not harmful
- Maybe harmful
- Harmful
- Satisfaction: This is a more general rating accounting for the abovementioned categories. Responses are assigned scores on a 0-4 satisfaction scale based on how well they meet Apple’s guidelines.
Apple AI’s Future
The AI rating document indicates that Apple scores AI responses multidimensionally, with helpfulness and safety as the most significant considerations. Google has already shown us how search engines and AI-generated summaries blend, and Apple’s ecosystem seems to be following in their footsteps.
For businesses, that makes AI SEO the top priority. Content optimized specifically for Siri/Apple Intelligence will be more likely to get highlighted if Safari incorporates an AI Overviews-esque mode, which could be a significant source of web traffic. Get started with a free consultation from an award-winning SEO agency! Contact Coalition Technologies today.
Frequently Asked Questions
Why are Apple’s AI assistant rating guidelines important?
AI evaluation is usually difficult to assess, considering companies don’t typically release information about their internal processes. Apple’s guidelines tell us about its current focus on Siri and Apple Intelligence and suggest future improvements in its current AI products.
Can I use AI assistants to boost my sales?
Yes. For example, you can optimize for AI assistants using voice search optimization and increase your store’s chances of being recommended in relevant user queries. Top-rated digital marketing agencies like Coalition Technologies can build a custom strategy for your business to leverage AI assistants.
Are Siri and Apple Intelligence as good as ChatGPT?
ChatGPT is currently much better at contextual understanding and helpfulness, even if it can’t offer the same in-device AI features as Apple Intelligence. However, Apple’s AI assistant rating guidelines indicate that the company may catch up in the future.
