How to Assess Manager AI Skills with SJT

Enterprise software budgets have directed substantial capital into generative AI copilots and autonomous agent platforms over the past twenty-four months. Yet operating leaders encounter an execution bottleneck across the managerial layer. According to the vendor-published, self-reported Section AI's The AI Proficiency Report (2026), surveying 5,026 US knowledge workers, 82% of C-suite executives are excited about artificial intelligence, compared to 30% of individual contributors. More revealingly, only 33% of business managers report using artificial intelligence daily.

This middle-management lag affects frontline operational momentum. Section AI's survey indicates that 65% of managers either set no expectations regarding AI adoption or encourage informal exploration without holding direct reports accountable for deliverables. Only 7% of managers require AI utilization and tie it directly to employee performance reviews. While executive leadership assumes software procurement translates into operational velocity, the supervisors directing day-to-day delivery often lack personal familiarity with AI workflows and structured governance frameworks to evaluate automated outputs.

The resulting tension shapes corporate talent priorities. In the Microsoft & LinkedIn 2024 Work Trend Index, surveying 31,000 workers across 31 countries, 79% of business leaders agreed that their company must adopt AI to remain competitive. Concurrently, 59% of leaders noted difficulty in quantifying productivity gains, and 60% stated their organization's leadership lacked a plan and vision to implement AI. The hiring landscape reflects this urgency: 66% of leaders say they would not hire an applicant lacking AI skills, and 71% prefer hiring a less experienced candidate with AI skills over an experienced professional without them.

To address this capability gap, human resources leaders have introduced workforce assessments. However, traditional evaluation tools—multiple-choice vocabulary exams, coding screens, and self-reported surveys—measure recall rather than managerial competence. An engineering manager who correctly defines attention mechanisms on a test may still struggle to review pull requests generated by autonomous agents. A marketing director reporting high confidence on a questionnaire may approve synthetic research incorporating flawed competitive assumptions.

Developing an effective AI competency assessment requires shifting focus from technical trivia to contextual judgment under uncertainty. Situational Judgment Tests (SJTs)—measurement instruments that evaluate candidate choices across realistic workplace dilemmas—offer a grounded psychometric foundation for evaluating how leaders manage delegation boundaries, review probabilistic outputs, establish quality gates, and organize team capacity.

The Structural Failure of Knowledge Quizzes and Self-Assessments

When evaluating emerging technology capabilities, organizations often rely on standardized vocabulary tests featuring questions such as "Define temperature in large language models" or "What is a vector embedding?" This equates technical terminology with supervisory capability, creating an evaluation mismatch.

Technical Recall Disconnected from Practical Execution

Testing terminology fails to capture operational capability. In the Section AI (2026) survey, researchers evaluated knowledge workers on conceptual definitions and practical instruction writing. On structured multiple-choice tests, 82% of respondents could not correctly identify the definition of an AI agent. When asked to define an agent in their own words, fewer than 10% provided a directionally accurate response.

A pronounced deficit appeared in practical execution. When asked to construct configuration instructions for an autonomous assistant (such as a Custom GPT or Copilot agent), only 3.8% of respondents delivered workable, production-ready specifications. Memorizing definitions provides minimal insight into whether a manager can establish the context fences, operational authority limits, and delivery criteria necessary to guide autonomous systems. Supervisory roles focus on governing workflows rather than training models, making terminology quizzes ineffective for leadership evaluation.

Vulnerability of Self-Reported Questionnaires to Social Desirability

To bypass the constraints of knowledge quizzes, some organizations use self-assessment surveys. Employees and managers rate their own proficiency on five-point Likert scales, evaluating statements such as "I effectively incorporate generative AI into my team's workflow."

These self-reported measures carry substantial methodological limitations. In a meta-analysis published in Frontiers in Psychology, Martínez & Salgado (2021) examined how individuals modify responses on non-cognitive self-inventories under evaluative stakes. When respondents recognize that promotion or performance ratings depend on presenting digital fluency, self-ratings tilt toward perceived organizational expectations. While forced-choice questionnaire formats have been studied in experimental settings to reduce response distortion, evaluating managerial competence requires observing practical decisions rather than relying on self-descriptions. Self-administered surveys measure internal confidence or corporate positioning instead of supervisory capability.

Redefining AI Competency Across Human-Machine Collaboration

A valid evaluation examines how professionals navigate human-machine workflows. From the perspective of workplace evaluation providers such as Canditech (Sernoff, 2026), assessing AI skills across technical and non-technical domains requires focusing on three functional competencies: structured prompt specification, critical evaluation of automated outputs, and balanced human-AI collaboration.

For managers, these competencies govern supervisory actions:

• Setting precise authority boundaries so autonomous agents do not execute unauthorized operational actions.
• Applying critical scrutiny to identify logical flaws or unsubstantiated claims in polished synthetic reports.
• Rebalancing workload when automated drafting accelerates draft generation while downstream verification pipelines remain constrained.

These competencies cannot be evaluated through static recall. They require assessing decisions across scenarios where actions carry distinct trade-offs.

The Psychometric Foundation: Why SJTs Predict Managerial Performance

A Situational Judgment Test presents candidates with realistic workplace dilemmas accompanied by multiple potential responses. Instead of defining a concept or rating personal confidence, the candidate analyzes a scenario involving competing priorities, resource constraints, or ambiguous data, selecting the most effective management intervention.

Meta-Analytic Evidence of Criterion-Related Validity

In an influential meta-analysis published in Personnel Psychology, Christian, Edwards, & Bradley (2010) synthesized 85 studies and 136 independent effect sizes to assess the criterion-related validity of situational judgment tests across various behavioral constructs.

Their meta-analysis reported that SJTs assessing leadership skills demonstrate an operational validity of $M_\rho = .28$ ($M_r = .21, k=51, N=7,589$) against overall job performance, where $M_\rho$ represents the mean validity corrected for criterion unreliability and $M_r$ is the uncorrected correlation. In comparison, SJTs assessing job knowledge constructs yielded an operational validity of $M_\rho = .19$ ($M_r = .15, k=4, N=695$), though the authors noted that this estimate was based on only four studies and its confidence interval (.07 to .32) overlaps with leadership measures.

Data source: Christian, Edwards, & Bradley (2010), Table 3 in Personnel Psychology. $M_\rho$ reflects corrections for criterion measurement unreliability.

When Christian, Edwards, and Bradley examined managerial performance specifically, construct alignment proved essential. SJTs assessing leadership competencies showed an operational validity of $M_\rho = .29$ ($M_r = .24, k=17, N=3,769$) and interpersonal skills showed $M_\rho = .36$ ($M_r = .27, k=2, N=321$, small sample caveat), compared to heterogeneous composite SJTs at $M_\rho = .12$ ($M_r = .10, k=5, N=1,282$). On this specific managerial criterion, construct-aligned leadership tests showed an estimated validity 2.4 times higher than unaligned composites. Across overall job performance in all occupations, however, heterogeneous composites achieved $M_\rho = .28$ ($k=54, N=8,767$), matching leadership tests. Grounding questions in defined supervisory constructs remains essential for managerial roles.

Evidence reviews by workforce research bodies support these findings. An evidence summary from the Quality Improvement Center for Workforce Development (Paul, 2020) concluded that SJTs function most effectively when developed from thorough job analyses and when the targeted construct aligns conceptually with the performance dimensions being evaluated. The review also noted that SJT scores correlate with general cognitive ability and personality traits, underscoring that scenario design and construct alignment govern predictive utility.

Navigating the Jagged Frontier: Anti-Hallucination and Supervisory Defense

Evaluating managerial AI capability differs from assessing proficiency in deterministic business software such as ERP or CRM systems. Deterministic software operates under fixed rules: valid inputs yield consistent outputs. Generative models, however, are probabilistic systems that produce fluent language while lacking causal reasoning.

The Jagged Frontier: Performance Divergence in Complex Work

Generative AI exhibits an uneven capability boundary known as the "jagged technological frontier." Complex tasks such as synthesizing regulatory documents may fall inside model capabilities, while tasks requiring cross-table arithmetic or nuanced context reconciliation may fall outside.

This boundary dynamic was documented in a randomized controlled field experiment conducted across 758 management consultants by Dell'Acqua et al. (Harvard Business School / Boston Consulting Group, 2026/2023). On tasks inside the frontier, consultants using GPT-4 finished 12.2% more subtasks, completed work 25% faster, and produced deliverables evaluated 32% higher in quality by expert evaluators compared to the control group.

On a strategic business analysis task situated outside the frontier—involving subtle brand positioning and conflicting financial exhibits—the outcome reversed:

• Consultants in the control group working without AI achieved an accuracy rate of 84.5%.
• Consultants using GPT-4 reached an accuracy rate of 70.6% (a difference of 13.9 percentage points).
• Consultants using GPT-4 who also received prompt engineering training achieved an accuracy rate of 60.0% (a difference of 24.5 percentage points; the study noted participants were 19% less likely to produce correct solutions).

The consultants with AI access fell behind because the model missed critical business insights and presented misleading conclusions within convincing prose. Consultants over-relied on synthetic explanations, reducing their independent verification of primary source exhibits.

As highlighted in leadership research by Development Dimensions International (DDI, 2023), institutional workflows require clear checks and balances between artificial intelligence tools and human decision-makers. AI-assisted analyses should undergo structured verification to ensure validity, accuracy, and fairness.

The matrix below compares how different evaluation modalities perform across core parameters:

Comparing Four Assessment Modalities for Managerial AI Fluency

Red-Teaming SJTs: Addressing Methodological Caveats and Construct Dilution

A balanced evaluation approach must address methodological constraints. In a review of personnel selection meta-analyses published in Applied Developmental Science, Richardson & Norgate (2015) noted that statistical adjustments for range restriction and unreliability in classical meta-analyses adjusted raw correlations upward by 61%, reaching .51, compared to more conservative adjustments yielding .22. This finding cautions against over-interpreting corrected laboratory estimates. A test composed of generic vignettes will experience validity decay without authentic operational grounding.

A second vulnerability originates within the SJT literature itself. Christian, Edwards, and Bradley (2010) observed that 33% of evaluated SJTs did not report underlying constructs, resulting in heterogeneous composite scores that yielded an operational validity of only $M_\rho = .12$ ($k=5$) for managerial performance. Combining miscellaneous administrative questions without construct mapping dilutes predictive utility.

To address composite dilution, the AI-Ready Leader diagnostic maps each dilemma to five defined capability dimensions:

1. D1: Task Delegation & Boundary: Setting structured machine contracts, context fences, and verifiable Definition of Done (DoD) criteria.
2. D2: Decision Quality & Anti-Hallucination: Conducting provenance audits, reviewing source data, and identifying jagged frontier risks.
3. D3: Review & Quality Assurance: Implementing multi-stage review gates to prevent junior generation surges from overwhelming senior staff.
4. D4: Team Enablement & Knowledge Asset: Curation of shared prompt assets, documentation of workflows, and team-wide practices.
5. D5: Workforce Capacity & New KPIs: Replacing volume-based metrics with review queue ratios, cycle times, and first-pass yield.

The diagnostic evaluates responses across four defined maturity levels: M0 Traditional, M1 AI-Enabled, M2 AI-Ready, and M3 AI-Native (Target State). While the empirical criterion validity of this specific instrument awaits independent organizational studies, its architecture builds on established construct alignment principles. For a detailed breakdown of these dimensions, review our five capability dimensions overview, or read the companion definition guide on what AI readiness means for modern leaders.

Practical Implementation: A Four-Step Architecture for AI Managerial Assessment

Organizations can implement a four-step framework to audit and develop managerial AI capabilities across business units.

Step 1: Ground Dilemmas in Operational Limiting Steps

Ground scenarios in authentic operational friction points where automated generation meets human supervisory review. Drawing on Andrew Grove's concept of the "limiting step" from High Output Management, identify the inspection points where unverified automated errors create downstream disruption:

• A junior analyst uses an automated agent to summarize regulatory documents, but the agent omits compliance exceptions.
• An engineering lead adopts code generation tools, but the resulting microservices bypass data privacy standards.
• A support manager deploys an autonomous customer agent that begins issuing unauthorized billing credits.

Step 2: Formulate Options Across M0–M3 Maturity Baselines

An effective SJT avoids obvious correct answers. Each option should reflect a coherent management approach corresponding to a defined maturity tier (the observable behaviors behind each tier are detailed in our manager AI maturity model):

### Sample Managerial SJT Scenario: The Autonomous Competitive Intelligence Dilemma

**Context**: A senior strategy analyst submits a 40-page competitive intelligence brief analyzing three industry rivals. The report was generated using an autonomous research agent that synthesized public financial filings and market commentary. The document appears thorough, but a spot-check reveals that market share figures for Competitor B cite non-existent publications, and pricing projections contradict historical filings. The executive strategy review begins in 36 hours.

**Question**: How should the business leader handle this deliverable?

- **Option A (M0: Traditional)**: 
  Instruct the analyst to discard the draft and prohibit AI tools for executive briefs. Require manual compilation of figures from verified annual reports over the next 36 hours.
  *Evaluation: Rejects automated leverage entirely, responding to errors through blanket tool prohibition.*

- **Option B (M1: AI-Enabled)**: 
  Accept the draft and spend the weekend manually checking every figure, citation, and table across all 40 pages before the executive meeting.
  *Evaluation: Relies on personal managerial heroics, creating an unsustainable review bottleneck.*

- **Option C (M2: AI-Ready)**: 
  Return the report to the analyst. Institute a Citation Provenance Protocol requiring auditable primary source links for every market claim before entering review pipelines.
  *Evaluation: Sets clear delegation boundaries and enforces non-negotiable verification gates before human review.*

- **Option D (M3: AI-Native)**: 
  Enforce the provenance gates in Option C. Concurrently, lead a post-incident review to codify automated retrieval-augmented generation (RAG) cross-checks against internal pricing databases into shared repositories.
  *Evaluation: Converts operational friction into an institutional asset, combining automated verification with shared team enablement.*

Step 3: Establish Multi-Tiered Verification Gates

To prevent senior reviewers from becoming overwhelmed by synthetic drafts, establish layered verification gates:

• Tier-1 Programmatic Validation: Deliverables must pass automated linting, schema validation, and citation ping checks before entering human queues.
• Tier-2 Adversarial Self-Check: The workflow requires the model to identify its own assumptions: "Identify the top three unverified empirical assumptions in these recommendations."
• Tier-3 Human Architectural Audit: Experienced managers concentrate attention on strategic trade-offs, policy boundaries, and business logic rather than formatting.

Step 4: Redesign Performance Metrics Around Throughput

Volume-based metrics such as prompts logged, slides generated, or lines written encourage teams to produce unverified intermediate material. Instead, align evaluation with systemic throughput:

• Review Queue Ratio: Ratio of active work-in-progress to senior review capacity.
• First-Pass Yield: Percentage of AI-assisted deliverables that pass quality gates without rework.
• Change Lead Time: Calendar time from initial problem definition to verified operational deployment.

For the system-level view of the queue dynamics these metrics surface, see why AI adoption stalls at middle managers.

Action Guide for Talent Leaders: What to Do This Week

For CHROs, heads of organizational development, and business leaders evaluating assessment strategies, the following action steps provide immediate traction:

1. Executive Alignment: Share the four-modality comparison matrix and the BCG field experiment (84.5% versus 60.0% accuracy) in leadership meetings to illustrate why prompt quizzes fail to protect decision quality.
2. Vendor Due Diligence: When reviewing commercial assessment providers, evaluate three criteria:
   • Does the vendor demonstrate job analysis grounding for leadership roles?
   • Can the tool resist gaming under promotion stakes?
   • Are scenarios construct-aligned to prevent composite dilution?
3. Internal Pilot: Deploy an 8-to-10 manager pilot using the 28-question SJT to review aggregate capability radar patterns before enterprise-wide rollout.
4. Recognize Boundaries: Apply SJTs specifically to multi-step supervisory trade-offs. For entry-level tool mechanics or deterministic software features, simpler practical task simulations remain appropriate.

The Strategic Imperative: Upgrading from AI Awareness to Behavioral Mastery

Enterprise AI adoption is maturing beyond preliminary tool exploration. Procuring seat licenses and hosting introductory workshops does not generate sustainable operational advantage. Competitive performance depends on whether managers possess the behavioral judgment to govern autonomous systems, challenge flawed synthetic assertions, and organize operational workflows.

Traditional assessment models centered on multiple-choice quizzes and subjective self-reports struggle to diagnose these supervisory competencies. By grounding leadership evaluations in the psychometric principles of Situational Judgment Tests, organizations can benchmark managerial maturity, reduce organizational blind spots, and develop capable leadership pipelines.

To explore how supervisory competencies evolve across maturity tiers, explore the broader AI-Ready Leader Insights series. For a comparison of how major frameworks divide governance, literacy, and supervision, read our analysis of nine AI competency frameworks. When you are ready to evaluate your managerial decision profile across authentic workplace dilemmas, take the 28-question Situational Judgment Test to receive your capability radar and development roadmap.