Skip to main content
Your project’s success depends on clear, detailed instructions. The instruction document is the single source of truth for the experts who will be generating data for you. It should be a living document that you update as the project evolves. We recommend structuring your instructions using the following framework:
  1. Context and Assignment
  2. Step-by-Step Workflow with a Checklist
  3. Examples: Golden Responses and Common Errors
  4. Grading rubric for Reviewers
An example template can be found here.

​
1. Context and Assignment

It’s best practice to cover:
  • Context – Why is an expert working on this project?
  • Assignment – What will the expert need to do and how will they submit their task
Experts are far more effective when they understand the “why” behind their work. Providing context helps them make better judgment calls on ambiguous cases that aren’t explicitly covered in the instructions.

​
2. Step-by-Step Workflow with a Checklist

Clearly lay out the exact process experts must follow. This minimizes guesswork and ensures the data you receive is consistent.
  • Use numbered steps with a checklist to make it easy to follow.
  • Include screenshots or short videos or other visual aids wherever possible to illustrate the process.
  • Highlight and define key terms. For example, if you are measuring sentiment, define what you mean (e.g., “Positive sentiment = friendly tone, helpful response”).

​
3. Examples: Golden Responses and Common Errors

Examples are one of the most effective ways to clarify your instructions. Provide a range of examples that show what to do and what to avoid. For each one, explain why it is a good or bad example.
  • Show an example of a good submission, highlighted in green.
  • Show an example of a bad submission, highlighted in red.
  • Include examples that cover edge cases, common mistakes, and frequently asked questions.Instruction Components

​
4. Grading Rubric for Reviewers

The grading rubric is the tool you will use to review and score the work submitted by experts. If you have experts review each other’s work (peer review), they will use this same rubric. It is critical that the rubric is comprehensive and as objective as possible. Your first version won’t be perfect; you should update it regularly as you review submissions and encounter new situations. A general best practice would be to make the grading rubric for reviewers public to all experts for transparency and alignment Best Practices for Instruction Documents
  • Document formatting should be clean, consistent and barebones. Use bold, italics, and emojis extremely sparingly. 
  • Create a table of contents with hyperlinks so that experts can easily navigate the document. 
  • Use green for good example andred for bad examples. However, try to not introduce additional colors as some people may have eyesight impairments and cannot read text in color 
  • Version Control: Keep your instruction document versioned and include a “change log” to track updates.
  • Establish a Communication Channel: Use a tool like Slack or Google Doc comments to create a space where experts can ask questions and flag edge cases as they arise.
  • Maintain a Clarification Log: Keep a running list of clarifications for edge cases that come up. This helps maintain consistency as the project grows.