Maintain search historyAt first, the problem sounds simple: find a company's careers page, follow the job links and extract the vacancies.
That works for some websites. But across hundreds or thousands of companies it gets complicated quickly. A careers link may send you to another domain. A shared recruitment platform may contain jobs from many employers. JavaScript may control how vacancies appear. Following the wrong links can take a crawler outside the company you meant to track.
This guide explains the main problems you need to solve.
Start with the companies you actually care about
Before thinking about crawling, define the employer universe. For example: UK fintech companies, European cybersecurity startups, your recruitment clients, companies in a particular sector, or employers selected by users of your product.
Find where the company publishes jobs
The main company website is only the starting point. Jobs may sit on a path or subdomain — or the careers link may take you somewhere completely different. That external page may be the correct source for the company's vacancies.
Misses legitimate jobs hosted on external platforms.
Leaves the employer entirely within a few hops.
Understand how the company relates to each job source.
The careers page may not contain the jobs
Finding a careers page doesn't mean you've found the vacancies. It may contain employer branding, employee stories, benefits, office locations, graduate programmes — and a button leading to another system. The actual listings can be several steps away.
A reliable system needs to follow this journey without treating every page it encounters as job data.
Shared recruitment platforms create another problem
Many recruitment systems host jobs for more than one employer. If a careers link lands on a platform with thousands of vacancies, the crawler can't simply collect everything it sees.
Sometimes the relationship is obvious from an employer identifier or URL. Sometimes it isn't. Without getting this right, jobs from unrelated companies leak into your dataset.
The problem becomes more than page discovery. It becomes source attribution.
Different companies publish jobs in very different ways
Career websites generally fall into a few broad groups. A single crawling strategy won't work equally well across all of them.
Standard ATS platforms
Established systems with relatively predictable structures, often shared by many companies.
Custom career sites
Navigation, listings and job-detail pages may be unique to the employer.
Jobs inside the main website
Vacancies published through the normal CMS, looking much like ordinary web pages.
JavaScript applications
Initial HTML holds almost no vacancy information; jobs appear after scripts run.
Bars indicate how much structure varies from one employer to the next.
Don't assume structured metadata will be available
Some job pages expose structured data such as schema.org/JobPosting. When it exists and is accurate, it's very useful. But many employers don't provide it. A valid job page might simply contain:
<h1>Senior Backend Engineer</h1> <p>London · Hybrid</p> <div> We are looking for an experienced engineer to join our platform team... </div>
A job-data system needs to work with the page itself, not only with structured feeds.
Keep the crawler inside the right boundaries
Once a crawler starts following links, it's surprisingly easy for it to wander. Following everything, it could end up traversing a large part of the web.
employer
As we saw in section 2, real jobs may legitimately live elsewhere. The system has to decide which external destinations belong to the hiring journey and which don't.
This matters most at scale: a small navigation mistake repeated across thousands of employers creates a huge amount of irrelevant crawling.
Some websites make automated access difficult
Not every career website is equally easy to crawl. Some sites limit automated requests, require JavaScript, use bot-detection systems, rate-limit traffic, or behave differently depending on how they're accessed.
The important point isn't how to get around those systems. It's that reliable job collection is an operational problem: sources behave differently and sometimes become temporarily inaccessible.
Identify the actual job pages
Once you reach a listing area, you still need to work out which links represent vacancies. Destinations vary widely — and sometimes there's no obvious URL until the user clicks. URL patterns alone aren't enough; look at the page content and its relationship to the listing.
Turn different job pages into one structure
Discovering the vacancy is only half the work. Employers describe the same information in completely different ways — one in labelled fields, another buried in the description. Your system needs to normalize the result.
- Salary
- £70,000 – £85,000
- Location
- London
- Working pattern
- Hybrid
- Contract
- Permanent
…join our London team on a permanent, hybrid basis. We offer £70–85k for a senior engineer…
{ "job_title": "Senior Software Engineer", "company_name": "Example Company", "location": "London", "location_type": "Hybrid", "job_types": ["Permanent", "FullTime"], "seniority_level": "SeniorLevel", "min_salary": 70000, "max_salary": 85000, "salary_currency": "GBP" }
The goal: your application sees one consistent structure, however the employer built its careers website.
Avoid duplicates
The same vacancy can be discovered through several routes — and rediscovered through filters, pagination or different navigation paths. If every discovery becomes a new record, users quickly see duplicates.
Deduplication isn't an optional cleanup step. It's part of producing usable job data.
Crawling once is not enough
Even a perfect crawl goes stale. Companies continuously publish new roles and remove old ones, so a maintained job index keeps asking:
If an employer leaves an old vacancy online, a crawler can't know they've stopped hiring for it. The safest source of truth is what the employer currently publishes.
That's why maintaining a job index is an ongoing process, not a one-time scrape.
Crawling is only the first layer
Once jobs are discovered and structured, most products still need another layer on top. At this point the architecture looks more like this:
The crawler itself is only one component.
The hard part isn't the first ten companies
For a handful of employers, building this internally may be completely reasonable. The calculation changes as the number and variety of companies grows.
A few known sources you can check by hand when something breaks.
It's keeping hundreds or thousands of sources working over time — while you're also trying to build the product customers actually use.
Build on the jobs, not the crawling infrastructure
Recuity starts with the companies that matter to your product. Bring your own employer list or discover companies in Recuity.
Recuity then handles the job-data layer behind those companies: discovering their career sources, finding the jobs and turning them into a consistent structure.