OaSIS Canadian Ratings Guide Update #2

[Francois Paradis]

Hello dear colleagues,

I am writing to provide you with an update on the development of the Canadian Ratings Guide (CRG), which is used by ESDC to assign ratings to occupational factors in OaSIS. As you may know, ESDC plans on releasing the CRG to the public at some point in the future and it is a crucial document to understand occupational ratings in OaSIS. CAVEWAS has put together an Ad Hoc committee to review a draft of the CRG and provide feedback. We had another consultation with ESDC on September 12, 2023. You will find below a list of our recommendations and ESDC’s response:

1. We need a better understanding on how ratings were assigned to each factor (i.e. ability levels, physical demands). Are ratings based on field surveys, expert ratings or other methods?

For each occupational profile of the OaSIS, the following process was used:

1. Data was imported from O*NET
2. Preliminary ratings (based on O*NET data) were reviewed by at least two independent occupational analysts. Each rating was converted from a 7-point scale (O*NET) to a 5-point scale (OaSIS).
3. Revised ratings were discussed among the two (or more) independent analysts
4. A senior analyst then reviewed the ratings

Following steps 1 to 4, for each descriptor:

1. A final revision for each descriptor was done to review ratings across all occupations.

2. Some descriptors can easily be confused with one another. For example, reading comprehension (Skill) and written comprehension (ability).

Some definitions are similar but the key difference is in the nature of the descriptor; some apply to abilities (latent, natural propensity) or skills (acquired). Abilities are the foundation upon which skills can be developed.

3. Under general common elements, difficulty levels are usually defined by complexity and duration. CAVEWAS recommends clear criteria to define complexity levels. Clear criteria are important to understand each complexity level. Otherwise, we have to rely on task examples provided, which are more subjective. There also needs to be a better consistency of criteria between the Canadian rating guide and the skills for success taxonomy. We also found inconsistencies of ratings between oasis and the Career Handbook.

ESDC agreed that the complexity criterion could be better defined. CAVEWAS provided criterion examples from the Revised Handbook for Analyzing Jobs (RHAJ) and the Skills for Success to better define complexity levels.

ESDC indicated that there is ongoing consultation with Skill for Success to align, as much as possible, the information between the two frameworks.

4. Some ability definitions have been changed from the GATB (i.e. numerical ability). The GATB defines aptitude in terms of speed and accuracy. The speed element is no longer part of the definition in the Canadian rating guide for abilities. This not only creates a disconnect between the career handbook and oasis but also puts in question whether it will be possible to use existing tests to measure ability factors in oasis.

ESDC acknowledges that speed has been removed from several ability definitions but stated that a new ability in the OaSIS captures the speed at which fingers, hands and or wrist most perform some tasks (Finger-hand-wrist motion).

Other discrepancies in definitions, scales and ratings between the Career Handbook and OaSIS were noted. ESDC pointed out that Career Handbook ratings (i.e. aptitudes) was based on the NOC 2001 framework. These ratings have not been revised since the early 2000’s. Occupations have evolved since then and the NOC 2021 and OaSIS reflect this.

There are differences in how ratings were assigned in OaSIS, which leads to discrepancies with Career Handbook ratings. For example, the Career Handbook assigns a high average level of numerical ability to dentists (NOC 3113.0) to “administer anesthetics, and to fill cavities with amalgam, silver, gold and other materials” and the same rating for Brokers (NOC 1113.2) “to contact trading departments of other investment firms, pension-fund managers and financial analysts, to complete details of sales on exchange trade tickets, and to prepare reviews of portfolios’ positions”. OaSIS takes into account the complexity and the quantity of the information being analyzed. The more information there is to analyse, the faster (more efficient) one must be. As such, for the numerical ability, brokers have a 5 (the highest level) and dentists a 3 (the average level).

ESDC also reported that it is conducting preliminary discussions with consulting firms to undertake a comparative review of the aptitudes of the Career Handbook and abilities in OaSIS, to determine their degree of similarity and whether existing aptitude tests can still be used. Should this review project go ahead, ESDC has offered to arrange a consultation between CAVEWAS and the selected firm, so that we can explain how aptitudes are measured and used in vocational evaluation.

Academic achievement skills are an important consideration in vocational evaluation and CAVEWAS mentioned that the scale for measuring essential skills has been changed in OaSIS. Essential skills profiles were using the International Adult Literacy Survey (IALS) scale of 0-500 and provided minimum scores for hundreds of occupations in the NOC. The TOWES (http://www.towes.com/en/home/home) allowed to measure a person’s Reading, Document Use and Numeracy levels and compare those with occupational requirements in essential skills profiles. The OECD Survey of Adult Skills (PIAAC) (https://www.oecd.org/skills/piaac/onlineassessmment/) is another test that measures literacy, numeracy and problem solving in technology-rich environments, using the IALS scale. All results are comparable to NOC essential skills profiles. These profiles are now being replaced by the Skills for Success framework and it remains to be seen how and when appropriate tests will be developed. ESDC stated they would forward our concerns to the department in charge of the Skills for Success framework for further discussion.

Bow Valley College (developer of the TOWES) has been commissioned in January 2022 by the Government of Canada’s Skills for Success Program to conduct a validation study of the Skills for Success framework through the piloting of evaluations and training resources aligned with the new model. You can find more details on this project here: https://bowvalleycollege.ca/about/media-relations-centre/news-media-releases/2022/bow-valley-college-receives-funding-for-skills-research-in-canada. This project is expected to be completed in 2025.

5. Although form perception is supposed to be included in the skills and competencies taxonomy, we do not find it listed in OaSIS.

ESDC stated that it will continue expanding OaSIS content as follow:

Expand the content found in OaSIS and on its website:

Descriptors will be added

• Knowledge – summer 2023
• General learning ability, form perception and motor coordination – early/mid 2024
• Tools and technology – late 2024 early 2025
• Tasks – mid/late 2025

Addition of new tools and information online

• A Skills match tool – June 2023
• Advanced Search Function – June 2023
• Addition of Main characteristics section – sometime in 2024
• Datasets available online (Open data portal) – in the coming weeks

6. The OaSIS scale used for physical demands (i.e. static strength) does not align with the Career Handbook or the American RHAJ. We recommend using a method such as the Matheson table to quantify weight and frequency in a manner consistent with other classifications such as the DOT.

ESDC stated that they are considering integrating the Leonard Matheson framework for physical demands, which classifies strength demands on a 5-point scale and considers exertional duration. 

7. The duration element should be better defined. Instead of using labels such as moderate or short duration, a percentage of the workday should be used.

 It was agreed that the duration element would be best suited to exertional demands. This will be further reviewed by ESDC.

All of the above points were discussed and will be addressed in future updates of OaSIS.

As always, your comments and questions are encouraged!

• This topic was modified 1 year, 3 months ago by 3986. Reason: Typos

[Jennifer Griffiths]

This was a great meeting and a solid partnership between the ESDC staff and the CAVEWAS group!  Francois has done an amazing job of stickhandling the project and keeping us all well-informed.  From the update provided by the ESDC staff on the OaSIS project, there is much more information and functionality being integrated into the system, with stakeholder feedback, including the excellent work done by the committee team, headed by Francois.  Looking forward to more good things for our profession with this valuable tool!

[Tamara Damini]

Thank you for the timeline of development for OaSIS.

[Author]

Posts