The eighteen guidelines, and where each one sits

You are in a working session with designers from four portfolio companies. Someone is defending a decision to make their agent ask a clarifying question whenever the extraction confidence drops below a threshold, and they close the argument with: “the HAX guidelines say to scope services when in doubt.”

There are three available responses and only one of them is worth anything. You can nod, which costs you the room. You can disagree from instinct — “in my experience users hate that” — which is an opinion competing against a citation, and citations win those. Or you can say: yes, that’s guideline ten, and it sits under “when wrong” rather than during interaction, which is already telling you something about what it was written for. Then disagree.

The third response works because of its first half, not its second. This lesson is that first half: what the toolkit actually contains, how the eighteen are actually grouped, and where the evidence behind them came from. Get any of that wrong and the disagreement built on top of it is worth less than saying nothing.

The toolkit is four things, and three of them are not the guidelines

People say “HAX” and mean the eighteen guidelines. The HAX Toolkit is four separate artifacts, and Microsoft describes each one in a single line:

• Guidelines for Human-AI Interaction — “Best practices for how AI systems should behave during interaction.”
• HAX Design Library — “Learn the Guidelines for Human-AI Interaction and how to apply them, using patterns and examples.”
• HAX Workbook — “Work together with your team to prioritize which Guidelines to implement in your product.”
• HAX Playbook — “For applications using natural language processing, identify common failures so you can plan for mitigating them.”

Read those four descriptions again and notice the shape: one artifact states the guidelines, and the other three are all about getting a team to apply them. The Workbook is a prioritisation exercise, on the assumption that no product will implement all eighteen. That is a useful thing to know in a room, because it means Microsoft itself does not treat the eighteen as a checklist to be passed.

One more piece of trivia that stops you looking silly: the guidelines page itself does not print the list. It tells you there are eighteen, describes them as “a synthesis of more than 20 years’ research, introduced in this award-winning 2019 CHI paper,” and sends you to the Design Library or the paper for the actual text.

Four phases, and the boundary nobody expects

The eighteen are grouped by when in the interaction they apply. The toolkit page names the phases in running prose — “initial interaction, during interaction, when the AI system is wrong, and over time” — but only Table 1 of the paper prints which guideline sits in which phase. The membership is lopsided:

Two things fall out of that table immediately. The first is that the largest phase is Over time, with seven of the eighteen — memory, learning from behaviour, cautious updates, granular feedback, conveying consequences, global controls, notifying about changes. This is a framework substantially about a system that persists and adapts across sessions, which is not how most people who quote it in passing seem to hold it.

The second is the boundary that catches people out. These four all sit under When wrong, not during interaction:

• G7 Support efficient invocation. “Make it easy to invoke or request the AI system’s services when needed.”
• G8 Support efficient dismissal. “Make it easy to dismiss or ignore undesired AI system services.”
• G9 Support efficient correction. “Make it easy to edit, refine, or recover when the AI system is wrong.”
• G10 Scope services when in doubt. “Engage in disambiguation or gracefully degrade the AI system’s services when uncertain about a user’s goals.”

Invocation is filed under failure. So is dismissal. That only makes sense if you read the phase name as “the machinery you need because the system will be wrong,” rather than “what happens after a wrong answer appears.” The paper does not defend the placement; Table 1’s caption hedges the whole scheme as “roughly categorized by when they likely are to be applied during interaction with users.”

That hedge is useful to you in two directions. It means anybody who argues from the phase boundaries as if they were load-bearing is over-reading the paper. It also means you should not build your own disagreement on the boundaries either — go after what a guideline says, not where it was filed.

Where the eighteen came from

The reason a citation beats an instinct in a room is that the eighteen have an evidence base and your instinct does not have one yet. It is worth knowing the actual shape of that base, both so you can respect it and so you can see its edges. The paper describes four phases of work:

1. Consolidation. Industry guidelines, an audit of AI products, trade articles and scholarly papers were pooled into 168 potential design guidelines, affinity-diagrammed down to 35 concepts and then to a working set of 20.
2. Modified heuristic evaluation. Eleven team members each inspected an AI-infused product for about an hour, thirteen products in total. The set went from 20 to 18.
3. User study. 49 HCI practitioners tested the eighteen against 20 popular AI-infused products across ten product categories, two or three evaluators per product.
4. Expert evaluation of revisions. Eleven experts reviewed the nine guidelines that had been rewritten, and preferred the revised wording for all but one.

The paper’s own summary of that is worth memorising because it is the line that makes the guidelines quotable: they “were iteratively refined in four phases by a team of 11 researchers, and were applied or reviewed by an additional 60 designers and usability practitioners.”

Sixty practitioners is a real number for HCI work. It is also, notably, sixty people evaluating existing shipped products against a candidate list — not sixty people whose users were measured. The eighteen are a codification of expert judgement, validated for applicability and inter-rater agreement. They are not a set of findings about what users do. That distinction is the one that will matter when you start disagreeing with a specific guideline from your own shipped evidence, because your evidence is of a different kind: fewer people, but actual behaviour.

What this does not cover

Nothing here tells you whether any of the eighteen is right. It tells you what they say and what stands behind them, which is the precondition, not the argument. Two things follow directly. The Google lesson takes the other framework you will hear quoted and shows that it is not a competing list of interaction rules at all — it answers questions that come before an interface exists, which is why quoting one at the other usually misses. And the provenance lesson goes back to Table 2, the twenty products these guidelines were validated against, and asks what kind of machine they describe. That question turns out to be where most of your defensible disagreements are going to come from.