Session Overview
Session
PA2: Organizational Assessment 1
Time:
Thursday, 23/Jul/2015:
9:45am - 11:15am

Session Chair: Bengt Jansson
Location: KOL-G-209 (Ⅲ)
capacity: 82

Presentations

Prediction of job performance, based on leaders' ratings, with a personality test: A correlational approach

Bengt Jansson

University of Gothenburg, Sweden; psybenja@gmail.compsybenja@gmail.com

Predictive criterion validity was examined in this study based on leaders' ratings of job performance. Employees were recruited with the personality test JobmatchTalent (JMT). A number of 258 recruited individuals was included in the analysis, and the ratings were performed by 95 leaders. The ratings were based on three dimensions (criteria): (1) productivity and motivation, (2) quality and structure of performed work, and (3) interpersonal skills. Nine main scales of JMT were used as predictors of the criteria (i.e. Work Structure, Personnel Drive, Decision characteristics, Activity, Drive, Acting, Tolerance, Social interest, and Communication). The predictive validity was based on pairwise correlations between criteria and predictors. By use of conceptual similarity, the 27 possible correlations were categorized as convergent (expected) or divergent (non-expected). A clear support was found for the assumed relationships. The average convergent correlation was .37 (N = 258). For a subgroup of recruited managers (n = 120), the corresponding value was .46. Convergent values differed significantly from divergent values. The predictive validity of the JMT was rated as good. The values found were compared to an extensive international meta-analysis, in which a correlation of just less than .40 was considered to be a very high value.

Predicting job performance of financial representatives based on Harrison Assessment Talent Management System (HATS)

Kristie Ann Craigen

Blurton Family Development Center, China; kcraigen@blurton-fdc.comkcraigen@blurton-fdc.com

The goal of this research was to empirically evaluate the predictive utility of Harrison Assessment Talent Management System (HATS) for use as a selection instrument within organizational settings. This was done by investigating the strength of the relationship between the job fit percentages generated by HATS, and the performance ratings (i.e., number of life insurance policies sold within the first 6 months of employment) of 238 employees (201 men and 37 women) working for a large American insurance company. The resulting correlation between HATS prediction scores and performance ratings yielded significant findings, r (236) = .599, p> .0001, indicating HATS is a useful instrument for predicting employee performance. Further ancillary analysis revealed 7 of the 156 predictor variables significantly correlated with performance ratings. As such, it is highly probable that many traits measured in the workplace (and by HATS) are best conceptualized as sharing a curvilinear relationship with performance. Moreover, the strength of the correlation coefficient between employee performance ratings and HATS prediction scores, in addition to its innovative methodology makes it a strong and rather unique instrument among the psychological tests currently available for use in organizational settings.

Predictive validity of an asynchronous video interview assessing leadership competences in a real teamwork situation

Anne-Kathrin Schock, Tuulia Ortner, Eva Traut-Mattausch

University of Salzburg, Austria; anne-kathrin.schock@sbg.ac.atanne-kathrin.schock@sbg.ac.at

Interviews are the most commonly used assessment tool in personnel selection. Nevertheless their application often lacks procedural standards and systematic scoring, which in turn can reduce fairness, comparability, and transparency. We sought to investigate the predictive validity of a new and promising tool in regard to standardisation: asynchronous video interviewing. Asynchronicity implies that candidates and interviewers do not need to be present at the same place and time, as recording and scoring form independent steps. To validate an interview for the assessment of leadership competences, a sample of 120 students completed the interview and held presentations in teams of 10 persons. Afterwards, students were asked to name and rank the team members according to their leadership potential during teamwork. First analyses of the interview revealed that the number of practical competences relevant to leadership success in small newly formed groups specified by participants was significantly associated with the frequency of assigned leadership. Further results regarding the interviews’ predictive validity will be discussed.

Measurement invariance and differential item functioning investigation of FVAT-S across gender and age groups in Turkish sample

Levent Sevinç, Hayri Eren Suna

Assessment Systems; erens@assessment.com.trerens@assessment.com.tr

The main purpose of the study was to investigate the measurement invariance of FVAT-S (FVAT-Short Form) across gender and age groups. FVAT-S is an ability test, which is widely-used in employee election procedures in Turkey. The sample that was used consisted of 2464 applicants (1273 females and 1191 males). The measurement invariance test was performed with Multiple Group Confirmatory Factor Analysis (MG-CFA). To decide the invariance, the change in the Comparative Fit Index (ΔCFI) was considered and additionally RMSEA, SRMR, and TLI were reviewed. At the first stage, FVAT-S showed configural and metric invariance across gender groups and then, partial scalar invariance was attained with the modifications. In terms of age groups (aged between 20-25 and 26-32), FVAT-S showed, and metric, and scalar invariance. Items which disrupting the scalar invariance across gender groups were tested against Differential Item Functioning. All items which decrease the invariance included gender-based DIF. These results showed that FVAT-S is a useful tool to compare age groups; however amounts of error may be changed between age groups. The DIF-included items in FVAT-S are needed to be reviewed to achieve full scalar invariance in gender groups.

Test order effects in high stakes assessment: An experiment using objective personality tests and cognitive ability tests in personnel selection

Alexander Leonard Schünemann1, Lale Khorramdel-Ameri2,3

1Schuhfried GmbH, Austria; 2University of Vienna, Austria; 3Educational Testing Service, USA; schuenemann@schuhfried.atschuenemann@schuhfried.at

For various reasons, varying test orders within test batteries is a common practice. Different test orders are used for organizational reasons, to avoid cheating, or to prevent a decrease in motivation if test-takers realize that others are performing faster. However, this practice is not well explored or proven to be without consequences for test results and therefore might decrease fairness and comparability in assessment processes. An experiment is presented that aims to extend a study from Khorramdel and Frebort (2011) by examining the influence of test order effects in personnel selection. The data comes from 70 applicants for a management trainee program of an Austrian railway company who had to take part in a computerized assessment of their professional potential as part of the selection process. Objective personality tests and cognitive ability tests were administered within a computer test battery and the administration order was systematically varied. The results are compared to the findings of Khorramdel and Frebort (2011) and are discussed with respect to subjects’ differences in cognitive ability and educational level, achievement motivation, and resilience.