Position Summary

The Director of Assessment Group will be responsible for overseeing the full cycle of certification exam development, new exam product development, exam and program maintenance, and evaluation of exam performance. This work will involve close cooperation with volunteer committees of subject matter experts. The goal will be to maintain the high level of integrity and reputation of the AWS Certification Examinations, and to improve the development, maintenance, and assessment functions as much as possible.

Essential Duties

• Lead and manage the Assessment Group consisting of Exam Developers, Data Analyst, and Scoring Team.
• Manage the Qualification and Certification Exam Bank (QCEB) Volunteer subcommittee and subject specialists responsible for personnel certification programs.
• Strategically plan the short-term and long-term assessment roadmaps for the Assessment Group and the Certification Department.
• Work with QCEB Subcommittee and subject specialists to develop and maintain personnel certification program requirements in order to uphold the AWS reputation, brand name, validity and reliability of the exam products following the Assessment Industry best practices. The activities include but not limited to:
  1. For new programs, lead or participate in subject matter expert task groups through JTA process for target personnel certification program. Institute valid test specifications, exam development, beta test, establish the psychometrics base for official exam and subsequent forms.
  2. For existing programs, reevaluate existing program requirements and participate in subject matter expert JTA task groups to modify requirements;
  3. Serve as committee secretary to exam bank committees to assist in the development of exam bank questions and new tests, and review the performance of current exam bank questions;
  4. Participate in the development of personnel qualification standards and personnel certification standards; and
  5. Lead the exam program subcommittee in defining the KSAs for exam questions.
  6. Re-establish historical standards of various existing certification programs to align with best practices;
• Conduct analyses of certification tests with the goals of:
  1. Identifying breaches of exam security;
  2. Identifying questions and test components not performing within designed performance parameters for future form publishing or exam update, as well as key validation;
  3. Summarizing exam and question performances for review by exam bank committee; and
  4. Summarizing exam quality for the stake-holders.
• Oversee and provide responses to candidate inquiries pertaining to their exam performance, and how to interpret their score reports;
• Manage the relationship with vendors and collaborate with them for smooth running of computer-based testing. This includes regularly meetings to communicate and solve issues related to online item authoring system, secure share of confidential materials, exam setup, publishing, new features of systems, score report and meet project timeline.
• Manage the contract of vendors for psychometric services.
• Manage the contracting and receipt of exam materials to be translated into foreign languages including developing and securing confidentiality agreements with translators;
• Oversee exam scoring.
  1. Ensure all exams are scored accurately and in a timely manner; and
  2. Resolve scoring issues in a timely manner
• Oversee the creation of Standard Operation Procedures and Policies.
• Always be mindful of AWS business needs and keep the AWS business needs as a priority when making decisions with assessment work, vendors, and projects. Ensure all job duties are carried out while reducing cost;
• Ascertain the impact of Department goals and Corporate goals and how it impacts the Assessment Group.
• Support the Managing Director, other groups of the Certification, as well as departmental projects.
  1. Budgeting;
  2. Qualification and Certification Main Committee activities;
  3. Setting SMART Department Goals and achievement them.
  4. Evaluation and choice of third-party testing agencies;
  5. Resolve or assist in resolving exam administration issues or nonconformities;
  6. Support Certification Operations with their administration and implementation of the certification programs:
  7. Provide advice and guidance to co-workers in the Certification Department on development of administrative documents, e.g., applications, test instructions;
• Develop and coach direct reports and foster a team environment.
• Actively participate in the assessment industry conferences such as annual meetings by Institute of Credentialing Excellence and Association of Test Publishing.
• Work with IT and other departments.
  1. Maintain and improve existing scoring, analysis, and archiving of test results;
  2. Be the principal architect in the transition from paper-based testing to computer-based testing;
  3. Continuously evaluate systems in order to obtain best psychometric results
  4. Review of old and new IT systems, including Scantron paper-based result files, and Prometric computer based testing results, including exam performance and candidate experience; and
  5. Respond to candidate inquiries pertaining to their exam performance and how to interpret their score reports.

 Supervision Exercised

• Exam Developer I
• Exam Developer II
• Program Manager, Scoring and Policies
• Data Analyst, Certification

 Working Conditions and Required Physical Abilities

When working on-site, this position is located in a quiet to moderately noisy indoor office environment.

Individuals in this type of position must have the use of sensory skills in order to effectively communicate with other employees and the public through use of the telephone and personal contact as normally defined by the ability to see, read, talk, sit, stand, hear, use of fingers in order to handle, feel or operate objects and write.  Must have the physical capabilities to move about the office and to effectively use and operate various items of office related equipment, such as but not limited to personal computer, calculator, copy and fax machines.  Must be able to remain seated at a keyboard or desk for extended periods. Must be able to lift, carry, and/or push/pull articles weighing up to 20 lbs.

To perform this job successfully, an individual must be able to perform each essential duty satisfactorily.  Reasonable accommodations may be made to enable qualified individuals to perform the essential functions of the job.

Required Knowledge, Skills & Abilities

Required Qualifications include:

• Quantitative, statistical, and psychometric data analysis.
• Critical thinking.
• Statistical test analysis.
• Project management.
• Test development.
• Group consensus building.
• Strong communication skills;
• Software: statistical software, Office 365 suite.
• Foreign language skills (e.g., Spanish and Chinese) is an asset.

Required Education and Experience

• Advanced degree in psychometrics, educational measurement, quantitative psychology, statistics, or related field.
• Minimum 7 years’ experience in the Assessment Industry, preferably experienced in industry credentialing organization.
• Experience with leading subject matter expert groups, designing testing programs, and leading psychometric activities including job task analysis and standard setting.

Special Requirements, Licenses, or Certifications

• Certificates in psychometrics and credentialing