Tiêu đề 2. [iMerit] i18n LiHA Candidate Evaluation Agent Evaluation Criteria.pdf ✅

i18n LiHA Candidate Evaluation Agent Evaluation Criteria [iMerit] [Confidential] Based on EN Guideline Version 1.4 Last Update: June 9th, 2025 1.Task Overview The goal of this annotation task is to evaluate the quality of output for a ‘Candidate Evaluation Agent’ which provides a recruiter (the user) with information on how good of a match a potential candidate’s profile is to a given job posting. 2.Evaluation Task You will be provided with the following reference data and model output fields for evaluation. Reference materials used in the evaluation Candidate Reference Data ● Candidate profile ● Screening questions (optional) ● Raw resume (optional) Job Requirement Reference Data ● Job requirement ● Job requirement type (required or preferred) Model output for evaluation ● Match classification ● Explanation why the candidate is a match/partial match/not a match ● Source of the supporting evidence for the match classification The evaluation task is made up of 5 interconnected question blocks. Begin the evaluation task by answering the questions in Step 1 and proceed through the remaining question blocks based on the instructions in the decision tree below. A visual summary of the decision flow for this evaluation is provided in the Candidate Evaluation Agent Task Summary document for reference.
Step 1: Candidate Reference Data Question Block Review the available candidate reference data and answer the following question about it. 1. Is the member profile accessible and do all available candidate reference data sources (member profile, resume, screening questions) refer to the same individual ? YES: Proceed to the next question. NO: Add a required comment. Submit the judgment - no further annotation is required. Step 2: Job Requirement Data Question Block Review The Job Requirement Input Data and answer the following questions about it. 1. Is the job requirement a grammatically well-formed, plausible and understandable job requirement in the target language? YES: Proceed to the next question. NO: Select applicable issue(s) from the list of job requirement data issue flags, add a required comment then submit the judgment - no further annotation is required. Examples of the job requirement issues that should be flagged. Issue Flag Explanation Example
The requirement is referring to outside context. The explanation contains words or phrases that would make sense in the context of a broader job posting, but which cannot be meaningfully interpreted in the job requirement string in isolation. ‘2 years of experience in a similar role.’ More context is needed to interpret what ‘a similar role’ is referring to. There is no way to assess whether the candidate is a match for this requirement without additional context. The string can’t reasonably be interpreted as a job qualification. This string is likely from a part of a job posting that is not related to the qualification requirements. It should not be evaluated. ‘Excellent perks and competitive benefits package’ This string was extracted in error from a part of a job posting that does not deal with desired or preferred qualifications of the candidate. It can’t be plausibly interpreted as something a candidate could be a match for or not. The requirement is garbled or contains grammatical errors. The string contains clear grammatical errors, is garbled or otherwise not linguistically well-formed. ‘Required experience in handle customers’ Although this string is interpretable, it contains serious grammatical errors and should be flagged. Other job requirement issue Any other issue with the job requirement which would prevent a meaningful match classification. Note: Job requirements for soft skills (e.g. “excellent teamwork”, “ability to work autonomously”) and vague requirements that may be difficult to assess are still considered valid skills and should NOT be marked as ‘Job Requirement Data Error.’ Even if they are underspecified. Step 3: Candidate/Job Requirement Match Classification Question Block Review the Job Requirement against the Candidate Reference Data and select the appropriate classification. Instructions for match classification decision-making and source evidence collection are provided in Section 3 Evaluation Dimensions: Requirement Match Classification, Evidence, and Source. 1. Does the information from the candidate reference data support the candidate being a match, partial match or not a match for the job requirement? Match or Partial Match Proceed to the next question.