Controversy erupted this month around the quality of hiring demand data used by the Government of Canada to identify labour market trends. The data in question had been harvested from online job postings, leading some commenters to attack the viability of the practice.
Although Vicinity Jobs was not involved in preparing the reports in the centre of this controversy, we are one of the companies that pioneered the concept of using online job postings to analyze hiring demand trends. The practice is used widely by Economic Development professionals, and is based on a simple and very straight-forward premise: To find out what employers require, you need to look at their job postings.
I have been following the debate very closely. Many strong and valuable points were made in the discussions, but so were some incorrect and sometimes misleading statements. So I felt compelled to address the major misconceptions that I came across.
- There is no need to analyze job postings: Statistics Canada has the data to provide hiring demand analysis.
As part of its monthly labour market survey, Statistics Canada breaks down employment numbers by industry. It could measure how employment levels change month over month in each industry. Such changes would be a proxy indicator of hiring demand.
Such approach, however, would involve a major blind spot: Since Statistics Canada only gathers information about jobs once they are filled, it has no information about job vacancies that never got filled.
Why does this matter? Consider this example: An employer needs certain skills but cannot find people who have them. What will they do? A larger employer may move the job opening to another location (for multinational companies this may be abroad and may involve moving a whole business division, not just the job that cannot be filled). Another option may be to simply not carry on the business requiring these skills. Either way, the shortage of skills that would never be detected will cause the local economy to lose a potential opportunity.
Another problem results from the fact that that skills shortages are much more likely to exist at a regional than at a national level. For example, employers in Alberta’s Fort McMurray may have difficulties finding gas fitters, whereas there may be unemployed gas fitters in Victoria BC. Statistics Canada produces very good reports when it comes to larger regions, but does not always provide granular reporting at a micro-regional level. There is a good reason for this: Statistics Canada reports are based on survey interviews and therefore require certain minimum sample size. Providing more granular regional reporting would mean interviewing more people in smaller communities to increase the sample sizes there. This is doable but could involve a significant extra cost, due to the large manual effort involved.
- Statistics Canada can get the same data by just analyzing postings published through the Service Canada’s Job Bank.
The Service Canada job bank is, indeed, a government-funded service that is provided free to employers and is one of Canada’s most popular job boards (if not the single most popular). Anyone who wants to post a job opening on Service Canada’s web site must first prove that they are a legitimate employer, by providing a valid payroll account registration number with Revenue Canada.
However, the Service Canada job bank data cannot be taken in isolation. It accounts for less than 25% of all jobs advertised in most regional markets. More importantly, it is not representative of the job postings published on other web sites. Vacancies in certain occupational categories are less likely to be found on the Service Canada job bank, as are jobs in certain industries. For example, most healthcare and professional service providers prefer to advertise their job openings through other channels (including their own web sites), whereas manufacturers are more likely to use the job bank. So analysis that uses only Service Canada will be skewed and subject to potentially significant gaps.
- Statistics Canada can just interview employers
If you want to know what employers are up to, why not just ask them, then analyze their responses?
This is actually a commonly used approach. It works well for larger geographic regions and is aligned with the methodology used by Statistics Canada to produce its monthly labour market survey. It provides a good snapshot in time. But identifying trends requires repeating the interviews over regular periods of time – which could involve a significant cost and relies on the cooperation of employers.
The limitation of this approach, aside from its significantly higher cost (compared to analyzing online job postings), is that it does not work very well in smaller communities. Imagine a town where the bulk of all manufacturing jobs are in 3 companies, 2 of which are laying off people while one is expanding. Without interviewing all 3 employers, you can never be sure that you are getting the full picture.
What makes this even more challenging is the fact that not all employers are willing to respond to surveys. If an employer does not want to talk about their needs and requirements, you would not be able to find out what they are looking for. However, most employers will advertise jobs when they need to hire.
- Anyone can post anything on the Internet, so it is impossible to tell bogus postings from legitimate ones.
It is worth mentioning that the whole controversy this month began when it was discovered that postings from Kijiji had been used in the analysis. I do find this problematic myself: Many free classified web sites like Kijiji and Craigslist lack proper validation of the identity of their ad publishers.
This is part of the reason why we exclude Craigslist and Kijiji postings in our analysis (another reason is that we found that many legitimate jobs there are also advertised elsewhere). However, this is not a reason to ignore postings advertised on other web sites with strict validation policies around advertisers’ identities. The Service Canada job bank is one example… Another example are employers’ own corporate web sites (it is hard to argue that postings published there are illegitimate). Leading paid national job boards also have policies in place to validate the identity of their advertisers: For example, employers are required to pay hundreds of dollars to post a job there, and processing the payment requires submission of valid billing information (usually credit card and billing name / address). We have found that, between themselves, credible sources of job postings contain a significant volume of data even in relatively small communities.
In a nutshell: When it comes to the credibility of job postings, not all web sources are created equal. For a report based on online job postings to be credible, it must use postings from credible sources. This is actually no different from any other type of statistical analysis: Analysis based on interviewing employers, for example, would be unreliable without safeguards to make sure that the right people at the right companies are being interviewed. This is why, at Vicinity Jobs, we put a lot of emphasis on establishing and following strict data source management policies.
- Employers may post the same job many times, so analysis based on job postings tends to count postings multiple times.
It is true that a single job may be advertised on different web sites, while different job openings may have similar sounding job descriptions. To address this, we have developed complex deduplication algorithms as part of our data processing methodologies, which remove as many duplicate postings as possible. Because computer algorithms are not perfect, at Vicinity Jobs we further augments the automated deduplication process by having a QA analyst review postings from known employers to remove duplicates missed by the computers.
In reality, even the best deduplication procedure may miss a few duplicate job postings. However, those are only a very small percentage of the total. So when we see the number of legitimate job postings requiring certain skills is growing over a period of time, this is a clear indication that demand for these skills is growing (just as a decreasing number of postings is a proof of declining demand). Together with known information from Statistics Canada about current employment levels, this information can be used to forecast future employment levels.
As the popularity of social media and online publishing is growing, so is the volume of information people and businesses share online. This has created a unique opportunity to tap into a wealth of knowledge that would not have been available to us a decade ago. The opportunity comes with its challenges. Other reporting methodologies have their limitations as well. Relevant, reliable reports can be produced from online data, as long as the proper tools and techniques and sound methodologies are used to ensure credibility. Ignoring online data because a single report may have been produced using an inadequate methodology would be a costly mistake.
written by Strac Ivanov, MBA, President of Vicinity Jobs Inc