Virtual Reality Simulation-Based
Robotic Technician Assessment
Objective
To demonstrate the potential offered by VR technology for credentialing and training of the robotics workforce.
Key Features
Overview
Siminsights partnered with ARM Institute to develop a VR based robotics technician assessment. The program focuses on analyzing a potential hire’s expertise in handling the maintenance of a palletizing robot in a virtual environment.
The assessment simulation, authored in HyperSkill, presents the user with a fault and asks the user to troubleshoot. SimInsights created a high-fidelity simulation capable of producing observations of users actions in a manner that supports gathering of detailed evidence for assessment of their proficiency on specific standards.
Problem Statement
Several technicians have years of experience but do not have recognized/formal training or credentials. Therefore this may be a limiting factor for their career progression. Formal training is often undesirable because it is too time consuming and expensive. Therefore it is desirable to have a scalable method of assessment and credentialing.
In the conventional assessment method (i.e., not VR-based), the subject matter expert (SME) would take a potential hire into the manufacturing environment, with the robot nearby and taken out of production, and provide them with a potential error prompt. This conventional assessment of taking an expert and a machine out of a field to evaluate a potential hire is
- Costly
- Time consuming
- Limited to few users
- Potentially unsafe
VR Solutions : Create 6 DOF VR immersive manufacturing environment
Siminsights project team worked with Dr. Ayesha Madni, Professor Mark Steyvers, the SMEs and the ARM Institute team to conduct the Cognitive Task Analysis. Analysis Assessment design was based on inputs from SME, Jonathan Clingan, on tasks related to troubleshooting and maintenance for the machine. Once the standards were set, the Scoring Model scheme was implemented. Based on the Scoring Model scheme, Hyperskill allows the user to proceed with the simulation to demonstrate their competency with other steps, even if they may not have completed a preceding step successfully.
Ultimately, when providing feedback to the user on their completion of the entire simulation, Hyperskill indicates which steps and substeps they completed successfully for partial and total scores. It also provides the requisite remediating information. In other words, a user gets feedback about particular content aspects and steps (e.g., safety procedures) where they may require additional practice, or where they have achieved adequate competence.
AI methods were used to implement conversational capabilities. Users can give voice commands to virtual humans in the simulation to obtain information. Users’ voice commands are logged and can be used in the assessment.
Delivery
SimInsights’ primary deliverable was a high fidelity simulation of a virtual reality manufacturing training environment. This environment contained highly realistic graphics and interactions, capable of producing observations of users’ knowledge, skills and abilities (KSA). Such information is used to gather evidence for valid and reliable assessment for certification and credentialing.
A tutorial was created and iteratively refined with pilot testing feedback. Another highlight of the simulation was the inclusion of conversational artificial intelligence (AI) to simulate dialogue between the worker and two virtual human assistants who played the role of a supervisor and a work cell technician.
Benefits
Credentialing for technicians who have years of experience but have not had recognized/formal training and assessment
● There are many competencies for technicians, so VR work cells are more scalable
● The user would be able to actually perform the tasks rather than verbally describe what actions they would do.
● The company conducting the assessment would not need to take an otherwise functioning robot out of the production line to conduct the assessment.
● A subject matter expert (SME) is not required to assess the performance of every user.
The net impact of these benefits offered by VR credentialing and assessment is a substantial cost saving to the company, improved assessment of potential hires and a notable increase in the number of potential hires taking this assessment to qualify for open positions.
Conclusion – Constructive Feedback from the Customer
Successful completion of this project demonstrates the potential offered by HyperSkill for credentialing and training the robotics workforce. All users who went through the experience were universally impressed by the environment and the implementation of the robot. It was interesting to note that the robotics expert users quickly made the correct inferences as new information was revealed to them by the simulation.
Participants were also asked about their overall experience with the VR task. All participants indicated that their experience was positive ranging from challenging and educational, to enjoyable and interesting.
The HyperSkill platform and the methodology followed herein are general purpose and applicable to a variety of assessments and VR training simulations. This case study lays out a clear path to scaling up a large collection of VR simulation based assessments for a broad range of careers in robotics and other industries. Such a collection of robotic enabled assessments has the potential to expand access, cut travel costs and promote equity to equip the workforce with in-demand skills today and in the future.