# Types of Data Source: https://humandata.mercor.com/background-reading/types-of-data A deep-dive into the different types of data to be collected There are many steps to training large language models --- and their derivative systems, like agents --- with many types of relevant data involved. This isn't a checklist of sequential steps you have to complete to get from a raw, unaligned model to a fully autonomous AI agent. It's more of an overview of common training processes and the data workflows involved. ## Pre-training Pre-training involves feeding the model vast amounts of unstructured data to learn the structure and nuances of the language(s) in the dataset. The model predicts parts of the data withheld from it, and its parameters are updated based on a loss function, which measures prediction errors. This step is typically performed using diverse sources like web pages, books, conversations, scientific articles, and code repositories. Two commonly used locations of pre-taining data are ["Common Crawl"](https://commoncrawl.org/) and ["The Pile"](https://pile.eleuther.ai/). You generally won't involve expert annotators directly in creating this data, though they can help with curation. There are exceptions, like in this paper which explores using human preferences in pre-training data. ## Supervised fine-tuning (SFT) Supervised fine-tuning (SFT) trains the LLM on labeled datasets of input-output pairs for specific tasks, enhancing performance in those areas. However, extensive fine-tuning may trade off generalization capabilities for specialized task performance. An example prompt-response pair for arithmetic: * Prompt: Calculate the sum of 66231613267717 and 60882409809. * Response: 66231613267717 + 60882409809 = 66292495677526 Prompt-response pairs don't always need to have an objective right answer, though, the same technique can be applied to function calling, agent planning, or even writing amazing bedtime stories for children. Especially when trying to improve model performance in a particular task, it's critical to (1) ensure the pairs are high quality and (2) select for data at or beyond the frontier of current model capabilities. While you can use synthetic data for this purpose, you'll likely need to work with human experts when you want to push beyond current frontier model capabilities and collect out-of-distribution data. ## Instruction fine-tuning Instruction fine-tuning is a specific type of SFT that teaches the model to follow specific prompt instructions. OpenAI used this method to align their InstructGPT models to be much better at following human instructions than the original GPT-3 model. You can work with a team of human annotators to create instruction prompt-response pairs across the instruction-following use cases you care about. For example, OpenAI used the following instruction types: Generation, Open QA, Brainstorming, Chat, Rewrite, Summarization, Classification, Closed QA, and Extraction. Creating instruction prompt-response pairs often involves human annotators, though many open-source instruction datasets are now available. Especially when trying to improve model performance in a particular task, it's critical to (1) ensure the pairs are high quality and (2) select for data at or beyond the frontier of current model capabilities. While you can use synthetic data for this purpose, you'll likely need to work with human experts when you want to push beyond current frontier model capabilities and collect out-of-distribution data. ## Parameter-efficient fine-tuning (PEFT) PEFT techniques aim to make it much more efficient to fine-tune models by adjusting a smaller number of parameters. One common PEFT technique is Low-Rank Adaptation (LoRA). PEFT techniques still involve using prompt-response pairs to fine-tune the model. ## Reinforcement learning with human feedback (RLHF) RLHF is a form of fine-tuning that aligns model outputs with human preferences. At a high level, it involves having humans rate model outputs on certain prompts, and using those rating signals to fine-tune the model to generate more aligned outputs. It often follows an SFT stage. The RLHF training process has two primary steps: 1. Have the model generate multiple outputs for a given prompt, and have humans rank these outputs based on their preference. This forms a set of preferred prompt-response pairs. Preferences are typically measured on a Likert scale from 1-7 and can be broken out by specific qualities you want the model to prefer, such as helpfulness, correctness, and style. 2. Use the pairs to do one of the following: 1. Train a reward model (a model to replicate human preference), which is then used to update the target model's policy (the strategy used to generate responses), using an algorithm like proximal policy optimization (PPO). 2. Or directly adjust the model's policy with the human preference data. This is called Direct Preference Optimization (DPO), a newer approach that simplifies the process. RLHF can be complicated, expensive, and finicky. Many organizations don't have the resources needed to do it well, and it may not even be needed for many application-specific use cases. ## How do I know if it's working? Enter evals Training a model can be quite expensive between the compute and data costs. It's important to know if your model is actually improving in the areas you care about. One way to do this is to use a set of evaluation tasks, or evals, to measure your model's performance in specific capabilities. Evals would contain a question/prompt and a correct answer (or a way to at least know the answer is correct in the case of more subjective answers), against which we can compare the model's answer to the prompt. You can and should evaluate your model at each stage of fine-tuning to understand how effective your methods and datasets are. There are a few major challenges with model evaluations: 1. Public benchmarks used to evaluate model performance (MMLU, GSM8K, HumanEval, Chatbot Arena) are not sufficient to measure application-specific performance. Additionally, open benchmarks can sneakily contaminate your model via training data. 2. Your eval set needs to cover all of your actual use cases, be sufficiently challenging, and be accurate. As models get better, you'll need more and more task expertise to create high-quality eval sets. 3. Not all tasks can be automatically evaluated, so you might need human experts in the loop to evaluate results. In an ideal world, you have a comprehensive, high-quality eval set, and use it as a ground truth to judge dataset effectiveness whenever you fine-tune your model. We often recommend creating this eval set before you even start fine-tuning, so you can measure performance improvements. The alternative is to do model ranking, with humans comparing post-fine-tuning model outputs vs other model outputs (including your own model before fine-tuning). Good evals may seem expensive, but it's a lot cheaper in the long run than flying blind when it comes to real-world model performance. ## Data across workflows and modalities This is quite general, but one way to look at categories of data is to break them down by: 1. What the data will be used for (SFT, RLHF, evals, etc.) 2. The format of the data itself (prompt-response pairs, ranking, reasoning, etc.) 3. The domain/focus area of the data As seen in the above diagram, you might intake data from sources like external datasets or in-application user feedback, and use it across different fine-tuning or evaluation workflows. Let's go through some common data flows. ## Conversational data Conversational data is useful for fine-tuning your LLM to interact with humans. You can collect input data from user sessions with a chatbot, public chatbot/conversational datasets, or by having annotators create them from scratch. Conversations can be single-turn or multi-turn: * Single-turn: each conversation consists of a single prompt and response, without further context. It can be a good starting point when fine-tuning your model. * Examples: trivia questions, simple programming tasks, translations * Multi-turn: each conversation consists of multiple turns of back-and-forth discussions between parties. Multi-turn data is great for learning more realistic conversation patterns, but is more complex to collect, annotate, and train on. * Examples: customer support logs, chatbot interactions, roleplaying dialogues You can first have annotators create conversational data (e.g., instruction response pairs) from scratch to fine-tune your model. When your model is deployed, you can surface multiple responses during user conversations, ask users to select preferred responses with reasoning, and use this data for RLHF (after triaging to filter for valuable, high-quality data). ## Agentic flows Agentic workflows involve deploying LLMs as part of a system to autonomously plan and execute tasks. Agents can break down high-level tasks into plans, execute on steps, interface with external tools, and even collaborate with humans and other agents. Here are some types of data you might collect and use to improve your agent: 1. Planning data: Improve the LLM's capability to plan out workflows. This can look like breaking down directives into steps, updating plans based on task results, assigning work to agents, and self-reflecting. 2. Task-specific data: Fine-tune the LLM to perform better at specific tasks it will handle. Examples can include using external tools, writing code, and summarizing inputs. 3. Multi-turn conversation data: Train on long conversations to maintain coherence when interacting with humans or other agents. 4. User preference data: If your agent is directly user-facing, you can have the user select preferred outputs, either at each step or the end of a workflow. Your agent will collect a lot of data that's out of your training data distribution once launched. It's key to collect and triage this data. It can be used as the backbone for debugging your agent, fine-tuning your underlying model, and creating evals for both the model and system: 1. Create SFT data for agent runs that failed due to incorrect model outputs. This way, you can improve the underlying model's capability to perform core tasks, like planning, using tools, or having conversations. 2. Create model evals from agent runs that failed due to incorrect model outputs. These model evals will test your model's ability to execute on specific steps. 1. Since workflows can involve many sequential steps, and mistakes will compound downstream, it's particularly important to have good evals to evaluate each major step. 3. Create system evals from agent runs that failed due to overall system failures. These are useful to understand if your system performs consistently on specific workflows. 4. Collect user preference data on step-by-step or workflow outputs, and use this for RLHF to align your underlying model with human preferences on helpfulness, accuracy, and safety. ## Multimodal (image, video, audio) When working with multimodal data, you may encounter various data flows. Sources for multimodal data can include public datasets (e.g., YouTube-8M), user data collected in your application, compilations of data found online, and net new multimodal data created by annotators. ### Image data Pre-processing steps involve resizing, normalization, and augmentation techniques like rotation, flipping, and cropping to increase data diversity. Annotation processes include: * Classification: Assigning a label to an entire image (e.g., hot dog or not hot dog). * Captioning: Creating descriptive sentences about the content of an image. * Object Detection: Identifying and labeling objects within an image with bounding boxes. * Segmentation: Classifying each pixel of an image into categories. For generative AI applications, there are two primary workflows involving humans: 1. Captioning images to improve the representation of images to the model. 2. Having conversations with or about images to teach the model how to extract information from images and use it to generate a response. ### Video data Pre-processing can involve frame extraction, temporal segmentation, and potentially downsampling to manage large file sizes. Classic annotation processes can include: * Action Recognition: Labeling actions occurring in a video (e.g., walking, running). * Event Detection: Identifying specific events within a video sequence. * Tracking: Following objects or persons across frames in a video. * Captioning: Writing descriptions for video segments, including actions and events. Video's GenAI workflows involving humans are quite similar to images. ### Audio data You might pre-process audio data with steps like noise reduction and normalization. Annotation processes can include: * Speech Transcription: Transcribing spoken words into text. * Speaker Identification: Recognizing and labeling different speakers in an audio clip. * Speech Synthesis: Generating natural-sounding speech from text input. Language fluency is a common annotator requirement for handling audio speech data. For GenAI applications, creating audio data can involve collecting representative audio samples covering use cases, domains, and the range of emotions and vocal expressions you want to incorporate into your product. A key point is that handling multimodal data often requires distinct tooling, labeling pipelines, and specialized annotator teams. # When Humans Are Needed Source: https://humandata.mercor.com/background-reading/when-humans-are-needed Humans vs. Synthetic Data Once you know you need data and what [type of data](https://mercor.com/docs/types-of-data) you need specifically, you'll have a choice between collecting the data with humans or leveraging a synthetic data solution. This article provides a brief history of how humans got involved in the process in the first place and then discusses where humans and synthetic data shine. At a high level, you should be using humans when: * You want to push the overall frontier of model capabilities beyond best-in-class today * You want to improve or fine-tune specific capabilities beyond what a model can do today * You want to measure the performance of an AI system You may not need humans when: * Improvements in prompt engineering are still able to make a difference * You want to catch up to frontier models to have a solid baseline ## A brief history - making LLMs useful and intuitive A few years ago, it started to be possible to train incredibly large language models thanks to advances in model architecture (Transformers) and advancements in parallel computing (GPUs). Once the world started to have models that could reliably predict the next several words (the GPT-2 and 3 era), it became important to figure out how to actually make them useful for humans. Going back to 2020 when GPT-3 was first coming out, it was a bit challenging to work with. Consider a prompt such as > Can you translate from English to French: I want to go to the park. GPT-3 would often respond with something along the lines of > Can you translate from English to French: I want to go to the park because it is a nice day today. instead of doing the translation! ***"Because it is a nice day today”*** has a higher probability of being next than ***“Yes I can, your translated sentence is je veux aller au parc”*** This behavior actually extends to many other undesirable attributes, such as public safety and model hallucinations. We needed to train the model to not just predict the next word, but to actually follow our instructions. Enter Reinforcement Learning with Human Feedback, or RLHF. While there are many aspects of Reinforcement Learning with Human Feedback, the basic premise is that humans are used to help the model understand and reward models that align with human preferences. RLHF can be roughly boiled down into two key ingredients: 1. Having humans rank multiple model outputs for the same prompt effectively teaches the model how to rank its own responses. 2. Having humans write the "ideal" completion to create a fine-tuning dataset demonstrates to the model what a great response looks like. RLHF is very effective at aligning the largest foundational models in the world today. ## Fitting in Synthetic Data In its purest form, the idea of synthetic data is to start replacing the parts of the process where humans are contributing their insights with an AI model. A term that is gaining popularity is RLAIF (vs. RLHF where the "AI" and "H" for human are swapped). ### Less helpful for building frontier models While research is evolving quickly, one of the core principles of synthetic data is that it's generally thought to not be possible to exceed the current frontier model with synthetic data alone. This is because the model used to produce the synthetic data will still have errors and is not perfectly aligned. When using synthetic data, all of the existing biases and quirks in the "teacher" model will make their way into the model being trained. Furthermore, getting a bit technical, models tend to prefer their own style and response types over other models or human feedback, so simply letting the model rank itself on which outputs are best can add significant undesired bias into your training data. To push the frontier, you do need actual data from real human beings on topics, modalities, and styles that have not been covered before. ### Less helpful for building novel capabilities When you create synthetic data, you are distilling the abilities of the "teacher" model as it relates to the workflows you care about. If you are trying to build a better model than, say, ChatGPT, for your specific use case, then using ChatGPT to generate the synthetic data wouldn't be very helpful (not to mention, it would be violating OpenAI's acceptable use policy). As an example, consider AI "Agents" that do things in the world - a common challenge today is their ability to break down a problem and generate a plan. The state-of-the-art frontier models at this moment are not sufficiently good at doing this task, and generating sample trajectories from them without more processing or human review is not a feasible way to collect this. ### More helpful for catching up to frontier models There are dozens of models today that have surpassed ChatGPT's initial performance. A common technique in the industry has been to take a leading model and use it to generate the training data for smaller, more efficient models. This famously happened when Meta released their initial Llama model, only to be quickly outperformed by Alpaca, and then Vicuna which fine-tuned Llama's models on derivatives from OpenAI. ### AI should still be part of your human process It may seem like synthetic data is not that useful for model development given the caveats mentioned, but that doesn't mean AI models have no place in helping create human data, in fact, quite the contrary! Generative AI models are incredibly useful at triaging data, classifying types of errors, and providing sanity checks to their human counterparts. A best-in-class data annotation system makes thoughtful use of AI and understands the nuances of human and AI interaction. # A Guide to Evaluating and Improving Your LLM Source: https://humandata.mercor.com/how-to/design-a-data-pipeline Systematic evaluation is the only way to truly understand how your model performs. Without it, you are "flying blind" and cannot measure progress or identify areas for improvement. A strong evaluation framework helps you: * Measure performance on realistic, important tasks. * Pinpoint where the model excels and where it fails. * Compare your model to state-of-the-art (SOTA) systems. * Gain actionable insights to guide iteration and fine-tuning. # **The Evaluation Process: A Four-Step Cycle** Here is a high-level overview of the recommended evaluation process: 1. **Build** a realistic, diverse, and challenging evaluation set. 2. **Compare** your model’s responses against those from a state-of-the-art (SOTA) model. 3. **Analyze** the results to understand performance gaps. 4. **Improve** your model in a targeted way. ## **Step 1: Build a High-Quality Evaluation Set** - The quality of your evaluation depends entirely on the quality of the prompts you test it with. * **Realistic:** Your evaluation set should contain prompts and documents that are representative of real-world use cases. Avoid contrived prompts that are designed only to make the model fail but don't reflect real scenarios. * **Diverse:** To ensure broad coverage, build a taxonomy of different scenarios. Think in terms of "verticals" (sub-domains) and "horizontals" (workflows). For example, in finance: * Verticals could be investment banking, private equity, or corporate finance. * Horizontals could be market research, financial modeling, or deal structuring. * **Challenging:** The evaluation set should include tasks across a spectrum of difficulty to properly test your model's limits. **Example of Strong Prompt:** Your client is a private equity investor targeting Malaysian small to mid-sized companies (SMEs) for Southeast Asia expansion. Seeks financially healthy, high-growth private companies. Task objectives: 1. Using the "SME Corp\_MSMEs in 2015-2023” file, project the number of MSMEs in 2024 using the 2019-2023 CAGR. Round the calculated CAGR to two decimal places using normal rounding rules before calculating the number of MSMEs in 2024. Round the calculated number of MSMEs to the nearest whole number. 2. Using the "SC MTC Companies” file, determine the number of Malaysian MTCs; assume that the reported number is for the year 2024. 3. Using the “Orbis\_Malaysian Companies Data” file, identify the total number of companies meeting the following investment criteria: * a) Average EBITDA margin (rounded to one decimal) over the years 2022 to 2024 is at least 15%. Note: To calculate this average EBITDA margin, sum the EBITDA margins for all available years between 2022 and 2024 for a company and divide by the number of years for which data is available, and then round to one decimal. For example, if a company has EBITDA margin data only for 2023, and 2024 (missing 2022), calculate the average as (EBITDA\_2023 + EBITDA\_2024) ÷ 2. * b) Exclude companies that report any EBITDA margin greater than 90% (without rounding) during the 2022–2024 period to avoid unrealistic profitability outliers. * c) Report how many companies meet both criteria. 4. From the qualified companies from Task 3, identify the top three industries (use "BvD sectors" as an indicator of industry) by volume (e.g., how many times they are represented in the dataset). For each of the top three industries: * Calculate the median and the top quartile EBITDA margins (using the rounded company-level averages from 3a). * Round answers to one decimal place. * State the number of companies in each of the top three industries. \ **Why is this a good prompt?** 1. Stresses realistic reasoning for private equity professionals 2. Requires challenging instruction following and multi-step logical reasoning  3. Needs the model to integrate multiple source documents that are representative of real-world scenarios ## **Step 2: Compare Model Responses Using a Clear Taxonomy** To run evaluations, you will compare your model's output to another model's output (e.g., a SOTA model) on the same prompt. The following taxonomy provides a best-practice framework for scoring the responses. | **Taxonomy** | **Definition** | **Proposed Scale** | | :------------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :---------------------------------------------------------------------------------------------------------------------------------------------------- | | Overall Preference | The evaluator’s choice between multiple model outputs, reflecting which response is most useful, accurate, or satisfying overall. | 1 – 5
1 – A is much better
2 – A is slightly better
3 – No preference
4 – B is slightly better
5 – B is much better | | Overall Preference Justification | The explanation or reasoning behind why a specific output was preferred, highlighting strengths and weaknesses relative to alternatives. | Free text explanation | | Overall Quality Score | A holistic rating (often numerical or categorical) summarizing the evaluator’s judgment of the response’s quality, independent of comparison. | 1 – Terrible
2 – Pretty Bad
3 – Okay
4 – Minor Room for Improvements
5 – Cannot be Improved | | Explanation of Model improvement areas | If model scored 3 or lower, then the expert should explain in free text what could be improved about the model’s response. Sometimes, we provide dropdown options with set categories. | Free text explanation | | Truthfulness | The model’s factual accuracy and breadth of information, including recall of established facts and grounding of reference documents. | 1 – No Issues
2 – Minor Issues
3 - Major Issues | | Instruction Following | The degree to which the model adheres to the user’s prompt, constraints, and task requirements without deviation. | 1 – No Issues
2 – Minor Issues
3 - Major Issues | | Verbosity | The appropriateness of response length and level of detail, avoiding being too terse or unnecessarily wordy. | 1 – Too Short
2 – Just Right
3 - Too Long | | Writing Style and Tone | The clarity, fluency, and appropriateness of language, including tone, voice, and stylistic alignment with the intended audience. | 1 – No Issues
2 – Minor Issues
3 - Major Issues | ## **Step 3: Analyze the Results** Once scores are collected, the goal is to diagnose *why* your model performed the way it did. * If a response scored less than a perfect 5/5, focus on understanding the specific gaps. * You can use LLMs to help automatically classify results into failure categories (e.g., reasoning errors, factual inaccuracies). * Track performance over time to see if your interventions are fixing specific failure types. ## **Step 4: Continuously Improve Your Model (aka "Hillclimbing")** "Hillclimbing" is the process of continuous, systematic improvement. After analyzing the results, you can take targeted steps to address your model's weaknesses. * Treat each evaluation round as a feedback loop: fix the weakest areas, re-evaluate, and measure your progress. * Focus your efforts on the failure buckets where you consistently lag behind SOTA models. * Once your model reaches parity with SOTA models on your current evaluation set, it's time to increase the difficulty by creating a more challenging set of prompts. This ensures you are always pushing the limits of your model's capabilities. # **How to Improve the Quality of Your Evaluations** * **Source High-Quality Experts:** The quality of your evaluation data depends on the people providing it. Prioritize experts who are experienced in your subject matter. * **Calibrate Your Raters:** Create a set of "golden tasks" – example evaluations with pre-determined correct answers – to align your internal team and external experts on the grading criteria. * **Focus on Quality Before Quantity:** When an evaluator starts, review their first few graded tasks closely before allowing them to work on a larger volume. * **Use Consensus to Reduce Bias:** To improve reliability, many labs have multiple evaluators rate the same response. This reduces individual bias. You can measure the consistency between raters with a metric called **inter-rater agreement (IRA)**. A high IRA score means your instructions and rubric are clear, while a low score indicates ambiguity. * **Keep Comparisons Simple:** Studies show that side-by-side (pairwise) comparisons are less cognitively demanding and lead to more consistent ratings. Try to stick to comparing just two or three model responses at a time. * Oftentimes closing the batch at less than 100% completion is easier, faster, and preferred. If closing out batches early, you need to watch out for "cherrypicking", where annotators only do the easy tasks and you never get the diversity or complex cases you had in mind completed. # **Recommended Tooling** Good tooling is essential for maintaining high annotation quality. While you may eventually build custom tools, you can accomplish a great deal with off-the-shelf SaaS products. * **Google Forms:** A lightweight and easy-to-use option, great for getting started with simple annotation projects. * **Airtable:** A more powerful tool that allows for significant automation and customization, capable of supporting large-scale annotation pipelines with a user-friendly interface. # Creating Clear Project Instructions for Your Experts Source: https://humandata.mercor.com/how-to/write-great-instructions Your project's success depends on clear, detailed instructions. The instruction document is the single source of truth for the experts who will be generating data for you. It should be a living document that you update as the project evolves. We recommend structuring your instructions using the following framework: 1. Context and Assignment 2. Step-by-Step Workflow with a Checklist 3. Examples: Golden Responses and Common Errors 4. Grading Rubric for Reviewers ## 1. Context and Assignment It’s best practice to cover: * Context – Why is an expert working on this project? * Assignment – What will the expert need to do and how will they submit their task Experts are far more effective when they understand the "why" behind their work. Providing context helps them make better judgment calls on ambiguous cases that aren't explicitly covered in the instructions. ## 2. Step-by-Step Workflow with a Checklist Clearly lay out the exact process experts must follow. This minimizes guesswork and ensures the data you receive is consistent. * Use numbered steps with a checklist to make it easy to follow. * Include screenshots or short videos or other visual aids wherever possible to illustrate the process. * Highlight and define key terms. For example, if you are measuring sentiment, define what you mean (e.g., "Positive sentiment = friendly tone, helpful response"). ### 3. Examples: Golden Responses and Common Errors Examples are one of the most effective ways to clarify your instructions. Provide a range of examples that show what to do and what to avoid. For each one, explain *why* it is a good or bad example. * Show an example of a good submission, highlighted in green. * Show an example of a bad submission, highlighted in red. * Include examples that cover edge cases, common mistakes, and frequently asked questions. ### 4. Grading Rubric for Reviewers The grading rubric is the tool you will use to review and score the work submitted by experts. If you have experts review each other's work (peer review), they will use this same rubric. It is critical that the rubric is comprehensive and as objective as possible. Your first version won't be perfect; you should update it regularly as you review submissions and encounter new situations. A general best practice would be to make the grading rubric for reviewers public to all experts for transparency and alignment. Best Practices for Instruction Documents * **Document formatting should be clean, consistent and barebones.** Use bold, italics, and emojis extremely sparingly.  * **Create a table of contents with hyperlinks** so that experts can easily navigate the document.  * Use **green for good example** and **red for bad examples.** However, try to not introduce additional colors as some people may have eyesight impairments and cannot read text in color.  * **Version Control:** Keep your instruction document versioned and include a "change log" to track updates. * **Establish a Communication Channel:** Use a tool like Slack or Google Doc comments to create a space where experts can ask questions and flag edge cases as they arise. * **Maintain a Clarification Log:** Keep a running list of clarifications for edge cases that come up. This helps maintain consistency as the project grows. # Black Box vs. Open Box Source: https://humandata.mercor.com/mercors-approach/black-box-vs-open-box How the work gets done matters, a lot ## Core Tenets The way Mercor thinks about collecting high-quality human data for GenAI use cases: * **Data has become core product / IP for both foundation model and application layer companies.** It increasingly makes sense to build data collection and annotation processes in-house, both for improving core capabilities and data security. * **The quality of your annotator team is the biggest lever on the quality of your data.** QA processes and tooling used to be the biggest lever with non-specialized work, but now, annotators must truly be experts to push the frontier of model capabilities. * **Experimentation speed is critical for continuously improving your model.** Data annotation should not be a blocker for your experimentation cycles. ## Current State Unfortunately, neither of the two options research teams have today for creating human data are perfect: 1. Outsource the entire data collection/annotation to external vendors, resulting in: * Slower experimentation speed, as time is spent negotiating per-task prices and sharing feedback with the vendor. * Data security concerns, especially if the vendor prefers to use their own platform. Many vendors have already leaked data. * Lack of transparency around cost and quality of data annotators, since vendors can be incentivized to keep costs low by obfuscating time spent and annotator backgrounds. 2. Build a human data team themselves, which comes with: * Higher fixed costs, in the form of hiring a larger internal human data team and maintaining a team of annotators. * Having to constantly source, vet, hire, performance manage, and terminate contractors for your annotator team. * Spend time and resources on operations which are not core competencies. Our goal at Mercor is to help our clients achieve the best of both worlds with an **“Open Box”** approach, vs the **“Black Box”** approach commonly favored by data vendors. The Open Box is characterized by 1. Simple hourly pricing that enables quick iteration. 2. Optimizing for the highest quality expert annotators. 3. Allowing clients to maintain control over their data — all without spending time sourcing and fully managing annotators themselves. ## Black Box vs. Open Box A more detailed comparison of the Black Box and Open Box approaches: | **Criteria** | **Black Box** | **Open Box** | | ------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | | **Experimentation speed** | Spend time negotiating with vendors over price per task for each iteration - Play a game of telephones between your research team, the vendors, and annotators | No new pricing negotiations every time you kick off a new project, just a flat % - Work directly with annotators when you need to iterate quickly | | **Data security** | Risk data getting leaked by unknown annotators if stored or processed on vendor’s platform | Retain control of data and IP by keeping work on your own platform if preferred - Know exactly who handles your data | | **Cost structure** | Unknown vendor margins | Full visibility into annotator pay | | **Management overhead** | Vendor handles operations, but will likely still need a small team to coordinate with research teams, manage vendors, and assess quality | Build an internal human data team, but outsource sourcing and managing annotators, and setting up data pipelines and processes to vendors | ## How Mercor helps teams build the “Open Box” solution: 1. **Create a proposal on scope and methodology** * We come in with a custom proposal based on our initial understanding of their needs, and will collect input on specific clarifications on topics like data distribution, volume, and what they’re optimizing for. * There’s no lengthy pricing and scoping discussions each time they kick off a new project. We just take a flat percentage of the annotators’ pay rates. 2. **Source and vet high quality talent** * We have 300k+ experts in our talent pool, and will look at a combination of factors including interviews, work experience, education, GitHub profiles, Google Scholar citations, and more to find the best annotators for specific projects. * We surface all selected annotator profiles before moving forward, for full transparency. 3. **Design and set up pipelines** * Teams can choose to use their own platform or ours. We can set up the workflow and pipeline with our tooling if they don’t have a platform to start with. * We have a set of documents and processes to get started with as part of our Human Data Handbook — this includes guidelines, style guides, rubrics, and pipeline designs. We can help modify these for custom needs. * We also help set up QA processes, from automated metrics to peer review pipelines, based on the specifics of the type of data and required quality. 4. **Measure key metrics on an ongoing basis** * We define a set of metrics to monitor the quality, volume, and cost efficiency of data being created, on aggregate and for each annotator. * Based on the project, these metrics can be more or less custom. Quality in particular is usually quite context-specific, so we often create rubrics specific to each project to define quality for annotator work. 5. **Swap out talent based on performance and changing needs** * In cases where annotators are underperforming or no longer needed for projects, we will notify and off-board them. We do this proactively based on metrics. * If companies have new needs, we source, surface, and onboard new annotators within hours to days. Teams can also use our platform to easily search for specific top talent themselves. It’s our hope that research teams can get the best of both worlds (building an in-house data team and outsourcing to data vendors) with the Open Box approach.