Selenium实现智联招聘免登录自动采集数据 轻松绕过验证码与登录限制解决反爬难题

引言:智联招聘数据采集的挑战与机遇
在当今数据驱动的招聘市场分析中,获取智联招聘(Zhaopin.com)的职位数据对于人力资源公司、市场研究机构以及求职者都具有重要价值。然而,智联招聘作为国内领先的招聘平台,实施了严格的安全措施来防止自动化爬虫的滥用。这些措施包括复杂的登录验证、滑动验证码、行为分析反爬机制等,给数据采集带来了巨大挑战。
传统的爬虫方法往往需要处理复杂的登录流程,包括账号密码输入、短信验证、滑块验证等,这不仅增加了技术难度,还存在账号被封禁的风险。更重要的是,即使成功登录,平台还会通过IP频率限制、用户行为分析等方式识别并阻止自动化采集行为。
Selenium作为一款强大的浏览器自动化工具,通过模拟真实用户操作,能够有效绕过这些反爬限制。它不仅可以执行JavaScript代码,还能处理复杂的页面交互,完美模拟人类操作习惯。本文将详细介绍如何使用Selenium实现智联招聘的免登录自动采集,通过智能策略绕过验证码和登录限制,实现稳定高效的数据获取。
一、环境准备与技术栈选择
1.1 必要的Python库安装
要实现智联招聘的数据采集,我们需要准备以下Python库:
# 核心库安装命令
pip install selenium==4.15.0
pip install webdriver-manager==4.0.1
pip install pandas==2.1.0
pip install openpyxl==3.1.2
pip install fake-useragent==1.4.0
pip install undetected-chromedriver==3.5.3 # 反检测驱动
1.2 浏览器驱动配置
Selenium需要浏览器驱动来控制浏览器。推荐使用Chrome浏览器,因其对Selenium的支持最为完善。
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.chrome.options import Options

def setup_driver(headless=False):
“””
配置Chrome浏览器驱动
:param headless: 是否启用无头模式
:return: 配置好的driver对象
“””
options = Options()

# 基础配置
options.add_argument(‘–no-sandbox’)
options.add_argument(‘–disable-dev-shm-usage’)
options.add_argument(‘–disable-blink-features=AutomationControlled’)

# 隐藏自动化特征
options.add_experimental_option(“excludeSwitches”, [“enable-automation”])
options.add_ex�perimental_option(‘useAutomationExtension’, False)

# 随机User-Agent
from fake_useragent import UserAgent
ua = UserAgent()
options.add_argument(f’user-agent={ua.random}’)

# 无头模式配置
if headless:
options.add_argument(‘–headless’)
options.add_argument(‘–window-size=1920,1080’)

# 启用性能日志(用于分析网络请求)
options.set_capability(‘goog:loggingPrefs’, {‘performance’: ‘ALL’})

# 使用webdriver-manager自动管理驱动
service = Service(ChromeDriverManager().install())
driver = webdriver.Chrome(service=service, options=options)

# 修改navigator.webdriver属性
driver.execute_script(“Object.defineProperty(navigator, ‘webdriver’, {get: () => undefined})”)

return driver
1.3 反检测技术配置
为了更有效地绕过反爬机制,我们需要使用undetected-chromedriver库,它专门设计用于绕过浏览器自动化检测。
import undetected_chromedriver as uc

def setup_undetected_driver():
“””
配置反检测浏览器驱动
“””
options = uc.ChromeOptions()
options.add_argument(‘–no-sandbox’)
options.add_argument(‘–disable-dev-shm-usage’)

# 随机User-Agent
from fake_useragent import UserAgent
ua = UserAgent()
options.add_argument(f’user-agent={ua.random}’)

# 启用性能日志
options.set_capability(‘goog:loggingPrefs’, {‘performance’: ‘ALL’})

# 创建反检测驱动
driver = uc.Chrome(options=options, version_main=None)

# 额外的反检测脚本
driver.execute_script(“Object.defineProperty(navigator, ‘webdriver’, {get: () => undefined})”)
driver.execute_script(“Object.defineProperty(navigator, ‘plugins’, {get: () => [1, 2, 3, 4, 5]})”)
driver.execute_script(“Object.defineProperty(navigator, ‘languages’, {get: () => [‘zh-CN’, ‘zh’, ‘en’]})”)

return driver
二、智联招聘页面结构分析
2.1 页面URL结构分析
智联招聘的职位列表页面URL通常具有以下结构:
https://www.zhaopin.com/beijing/
https://www.zhaopin.com/jobs/123456789.html # 具体职位页面
搜索结果页面的URL参数:
https://www.zhaopin.com/jobs?searchType=2&keyword=Python&city=北京&salary=4-6k
2.2 关键元素定位
通过浏览器开发者工具分析,我们可以找到以下关键元素的CSS选择器:
# 智联招聘页面元素定位器
PAGE_ELEMENTS = {
# 职位列表容器
‘job_list_container’: ‘div.joblist-box’,

# 单个职位项
‘job_item’: ‘div.joblist-box__item’,

# 职位标题
‘job_title’: ‘a.joblist-box__item__info__title’,

# 公司名称
‘company_name’: ‘div.joblist-box__item__info__company’,

# 职位薪资
‘salary’: ‘span.joblist-box__item__info__salary’,

# 工作地点
‘location’: ‘div.joblist-box__item__info__bottom span’,

# 发布时间
‘publish_time’: ‘div.joblist-box__item__info__bottom span:last-child’,

# 职位详情页链接
‘job_detail_link’: ‘a.joblist-box__item__info__title’,

# 下一页按钮
‘next_page’: ‘button.next-page’,

# 搜索输入框
‘search_input’: ‘input.search-box__input’,

# 搜索按钮
‘search_button’: ‘button.search-box__button’
}
2.3 模拟用户行为策略
为了绕过反爬,我们需要模拟真实用户的操作模式:
import time
import random
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.action_chains import ActionChains

class ZhaopinScraper:
def __init__(self, headless=False, use_undetected=True):
self.driver = setup_undetected_driver() if use_undetected else setup_driver(headless)
self.wait = WebDriverWait(self.driver, 10)
self.base_url = “https://www.zhaopin.com”

def random_delay(self, min_delay=1, max_delay=3):
“””随机延迟,模拟人类操作”””
delay = random.uniform(min_delay, max_delay)
time.sleep(delay)

def human_like_scroll(self, times=3):
“””模拟人类滚动页面”””
for _ in range(times):
# 随机滚动距离
scroll_distance = random.randint(200, 600)
self.driver.execute_script(f”window.scrollBy(0, {scroll_distance})”)
self.random_delay(0.5, 1.5)

# 偶尔向上滚动一点
if random.random() > 0.7:
self.driver.execute_script(f”window.scrollBy(0, -{random.randint(50, 150)})”)
self.random_delay(0.3, 0.8)
三、免登录数据采集策略
3.1 直接访问搜索页面
智联招聘的职位列表页面在未登录状态下也可以访问,我们可以通过构造搜索URL直接获取数据:
def build_search_url(keyword, city, salary_range=None, page=1):
“””
构建智联招聘搜索URL
:param keyword: 搜索关键词
:param city: 城市名称
:param salary_range: 薪资范围,格式如”4-6k”
:param page: 页码
:return: 完整的搜索URL
“””
base_url = “https://www.zhaopin.com/jobs”
params = []

# 搜索类型:2表示职位搜索
params.append(“searchType=2″)
params.append(f”keyword={keyword}”)
params.append(f”city={city}”)

if salary_range:
params.append(f”salary={salary_range}”)

# 页码参数
if page > 1:
params.append(f”page={page}”)

url = f”{base_url}?{‘&’.join(params)}”
return url

# 使用示例
url = build_search_url(“Python”, “北京”, “4-6k”, 1)
print(f”Generated URL: {url}”)
3.2 绕过初始访问限制
首次访问智联招聘时,可能会遇到访问频率限制。我们需要采用以下策略:
def initial_visit_with_proxy(self, url):
“””
使用代理和随机延迟绕过初始访问限制
“””
# 第一次访问前添加更长的延迟
print(“首次访问,等待10秒…”)
time.sleep(10)

# 访问目标URL
self.driver.get(url)

# 等待页面加载
self.random_delay(3, 5)

# 检查是否出现验证码或访问限制页面
if self.check_for_captcha():
print(“检测到验证码,启动应对策略…”)
self.handle_captcha()
return False

return True

def check_for_captcha(self):
“””
检查页面是否出现验证码
“””
captcha_indicators = [
‘验证码’,
‘captcha’,
‘验证’,
‘滑动验证’,
‘请稍后’,
‘访问过于频繁’
]

page_source = self.driver.page_source.lower()
for indicator in captcha_indicators:
if indicator in page_source:
return True

# 检查特定元素
try:
# 检查是否存在验证码iframe
if self.driver.find_elements(By.CSS_SELECTOR, ‘iframe[src*=”captcha”]’):
return True
except:
pass

return False
3.3 处理验证码的智能策略
当遇到验证码时,我们有多种应对策略:
def handle_captcha(self):
“””
处理验证码的多种策略
“””
# 策略1:等待并重试
print(“策略1:等待60秒后重试…”)
time.sleep(60)
self.driver.refresh()

# 策略2:尝试点击验证码(如果存在)
try:
captcha_element = self.driver.find_element(By.CSS_SELECTOR, ‘div.captcha-container, div.slider-verify’)
if captcha_element:
print(“检测到验证码元素,尝试手动处理…”)
# 这里可以集成第三方验证码识别服务
# 或者暂停等待人工处理
input(“请手动完成验证码,然后按回车继续…”)
return True
except:
pass

# 策略3:更换IP和User-Agent后重试
print(“策略3:更换环境后重试…”)
self更换浏览器环境()

return False

def 更换浏览器环境(self):
“””
更换浏览器环境,包括User-Agent和可能的代理
“””
# 关闭当前浏览器
self.driver.quit()

# 重新初始化(会使用新的随机User-Agent)
self.driver = setup_undetected_driver()
self.wait = WebDriverWait(self.driver, 10)

# 等待一段时间
time.sleep(30)
3.4 数据提取与解析
一旦成功访问页面,我们需要高效地提取数据:
def extract_job_list_data(self):
“””
从当前页面提取职位列表数据
“””
job_data = []

try:
# 等待职位列表加载
self.wait.until(
EC.presence_of_element_located((By.CSS_SELECTOR, PAGE_ELEMENTS[‘job_list_container’]))
)

# 获取所有职位项
job_items = self.driver.find_elements(By.CSS_SELECTOR, PAGE_ELEMENTS[‘job_item’])

print(f”找到 {len(job_items)} 个职位项”)

for item in job_items:
try:
job_info = self.extract_single_job(item)
if job_info:
job_data.append(job_info)
except Exception as e:
print(f”提取单个职位失败: {e}”)
continue

except Exception as e:
print(f”提取职位列表失败: {e}”)

return job_data

def extract_single_job(self, job_element):
“””
提取单个职位的详细信息
“””
try:
# 职位标题
title_element = job_element.find_element(By.CSS_SELECTOR, PAGE_ELEMENTS[‘job_title’])
title = title_element.text.strip()

# 公司名称
company_element = job_element.find_element(By.CSS_SELECTOR, PAGE_ELEMENTS[‘company_name’])
company = company_element.text.strip()

# 薪资
try:
salary_element = job_element.find_element(By.CSS_SELECTOR, PAGE_ELEMENTS[‘salary’])
salary = salary_element.text.strip()
except:
salary = “面议”

# 工作地点
location_elements = job_element.find_elements(By.CSS_SELECTOR, PAGE_ELEMENTS[‘location’])
location = location_elements[0].text.strip() if location_elements else “”

# 发布时间
publish_time = location_elements[-1].text.strip() if len(location_elements) > 1 else “”

# 详情页链接
detail_link = title_element.get_attribute(‘href’)

return {
‘title’: title,
‘company’: company,
‘salary’: salary,
‘location’: location,
‘publish_time’: publish_time,
‘detail_url’: detail_link,
‘采集时间’: time.strftime(“%Y-%m-%d %H:%M:%S”)
}

except Exception as e:
print(f”提取职位信息失败: {e}”)
return None
3.5 分页采集与进度保存
为了确保数据采集的稳定性和可恢复性,我们需要实现分页采集和进度保存功能:
import json
import os

class ZhaopinScraper:
def __init__(self, headless=False, use_undetected=True, output_file=”zhaopin_jobs.json”):
# … 初始化代码 …
self.output_file = output_file
self.scraped_data = []
self.scraped_urls = set()

# 加载已采集的数据
self.load_existing_data()

def load_existing_data(self):
“””加载已采集的数据,避免重复采集”””
if os.path.exists(self.output_file):
try:
with open(self.output_file, ‘r’, encoding=’utf-8′) as f:
existing_data = json.load(f)
self.scraped_data = existing_data
self.scraped_urls = {job[‘detail_url’] for job in existing_data if job.get(‘detail_url’)}
print(f”已加载 {len(self.scraped_data)} 条历史数据”)
except Exception as e:
print(f”加载历史数据失败: {e}”)

def save_data(self):
“””保存数据到文件”””
try:
with open(self.output_file, ‘w’, encoding=’utf-8′) as f:
json.dump(self.scraped_data, f, ensure_ascii=False, indent=2)
print(f”数据已保存到 {self.output_file}”)
except Exception as e:
print(f”保存数据失败: {e}”)

def scrape_pages(self, keyword, city, salary_range=None, max_pages=10):
“””
采集多页数据
:param keyword: 搜索关键词
:param city: 城市
:param salary_range: 薪资范围
:param max_pages: 最大采集页数
“””
for page in range(1, max_pages + 1):
print(f”\n{‘=’*50}”)
print(f”开始采集第 {page} 页”)
print(f”{‘=’*50}”)

# 构建URL
url = build_search_url(keyword, city, salary_range, page)

# 访问页面
if not self.initial_visit_with_proxy(url):
print(f”第 {page} 页访问失败,跳过…”)
continue

# 模拟用户行为
self.human_like_scroll(times=random.randint(2, 4))

# 提取数据
page_data = self.extract_job_list_data()

# 过滤已采集的数据
new_data = []
for job in page_data:
if job[‘detail_url’] not in self.scraped_urls:
new_data.append(job)
self.scraped_urls.add(job[‘detail_url’])

print(f”本页找到 {len(page_data)} 个职位,其中 {len(new_data)} 个是新的”)

# 添加到总数据
self.scraped_data.extend(new_data)

# 保存数据
self.save_data()

# 检查是否还有下一页
if not self.has_next_page():
print(“没有下一页了,采集完成”)
break

# 随机延迟,准备下一页
delay = random.randint(5, 15)
print(f”等待 {delay} 秒后继续…”)
time.sleep(delay)

print(f”\n采集完成!总共采集 {len(self.scraped_data)} 条数据”)
return self.scraped_data

def has_next_page(self):
“””检查是否存在下一页”””
try:
next_button = self.driver.find_element(By.CSS_SELECTOR, PAGE_ELEMENTS[‘next_page’])
# 检查按钮是否可用
if ‘disabled’ in next_button.get_attribute(‘class’):
return False
return True
except:
return False
四、高级反爬绕过技术
4.1 代理IP轮换策略
当采集频率较高时,需要使用代理IP来避免IP被封禁:
import requests
from itertools import cycle

class ProxyManager:
def __init__(self, proxy_list=None):
self.proxy_list = proxy_list or []
self.proxy_cycle = cycle(self.proxy_list) if self.proxy_list else None
self.current_proxy = None

def get_proxy(self):
“””获取下一个代理”””
if self.proxy_cycle:
self.current_proxy = next(self.proxy_cycle)
return self.current_proxy
return None

def test_proxy(self, proxy):
“””测试代理是否可用”””
try:
response = requests.get(
‘https://www.zhaopin.com’,
proxies={‘http’: proxy, ‘https’: proxy},
timeout=10
)
return response.status_code == 200
except:
return False

def rotate_proxy(self):
“””轮换代理”””
if not self.proxy_list:
return None

for _ in range(len(self.proxy_list)):
proxy = self.get_proxy()
if self.test_proxy(proxy):
return proxy
print(f”代理 {proxy} 不可用,尝试下一个…”)

print(“所有代理都不可用”)
return None

# 在Scraper中使用代理
class ZhaopinScraper:
def __init__(self, proxy_manager=None, **kwargs):
# … 其他初始化 …
self.proxy_manager = proxy_manager

def setup_proxy(self, proxy):
“””为浏览器设置代理”””
if proxy:
self.driver.execute_cdp_cmd(
‘Network.setExtraHTTPHeaders’,
{‘headers’: {‘Proxy-Authorization’: f’Basic {proxy}’}}
)
# 或者通过Chrome选项设置代理
# self.options.add_argument(f’–proxy-server={proxy}’)
4.2 鼠标移动轨迹模拟
为了更真实地模拟人类操作,我们需要模拟鼠标移动轨迹:
from selenium.webdriver.common.action_chains import ActionChains

def simulate_mouse_movement(self, element, duration=1.5):
“””
模拟真实的鼠标移动轨迹
:param element: 目标元素
:param duration: 移动持续时间(秒)
“””
actions = ActionChains(self.driver)

# 获取元素位置
location = element.location
size = element.size

# 计算目标点(在元素内部随机位置)
target_x = location[‘x’] + random.randint(5, size[‘width’] – 5)
target_y = location[‘y’] + random.randint(5, size[‘height’] – 5)

# 生成贝塞尔曲线轨迹
start_x = random.randint(100, 500)
start_y = random.randint(100, 300)

# 分段移动,模拟人类不完美的轨迹
steps = 20
for i in range(steps):
progress = i / steps
# 使用二次贝塞尔曲线
control_x = start_x + (target_x – start_x) * 0.5 + random.randint(-20, 20)
control_y = start_y + (target_y – start_y) * 0.5 + random.randint(-20, 20)

current_x = (1 – progress) * (1 – progress) * start_x + 2 * (1 – progress) * progress * control_x + progress * progress * target_x
current_y = (1 – progress) * (1 – progress) * start_y + 2 * (1 – progress) * progress * control_y + progress * progress * target_y

# 移动鼠标
actions.move_by_offset(current_x – start_x, current_y – start_y)
actions.perform()

# 重置actions
actions = ActionChains(self.driver)

# 随机延迟
time.sleep(duration / steps * random.uniform(0.8, 1.2))

# 最后点击
actions.click()
actions.perform()

# 使用示例
# element = driver.find_element(By.CSS_SELECTOR, “button.search”)
# scraper.simulate_mouse_movement(element, duration=2)
4.3 浏览器指纹伪装
使用undetected-chromedriver可以很好地伪装浏览器指纹,但我们可以进一步增强:
def enhance_browser_fingerprint(self):
“””
增强浏览器指纹伪装
“””
# 设置更真实的浏览器属性
self.driver.execute_script(“””
// 伪装Chrome版本
Object.defineProperty(navigator, ‘appVersion’, {
get: () => ‘5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36’
});

// 伪装插件
Object.defineProperty(navigator, ‘plugins’, {
get: () => [
{name: ‘Chrome PDF Plugin’, filename: ‘internal-pdf-viewer’, description: ‘Portable Document Format’},
{name: ‘Chrome PDF Viewer’, filename: ‘mhjfbmdgcfjbbpaeojofohoefgiehjai’, description: ”},
{name: ‘Native Client’, filename: ‘internal-nacl-plugin’, description: ”}
]
});

// 伪装语言
Object.defineProperty(navigator, ‘languages’, {
get: () => [‘zh-CN’, ‘zh’, ‘en-US’, ‘en’]
});

// 伪装平台
Object.defineProperty(navigator, ‘platform’, {
get: () => ‘Win32’
});

// 伪装硬件Concurrency
Object.defineProperty(navigator, ‘hardwareConcurrency’, {
get: () => 8
});

// 伪装内存
Object.defineProperty(navigator, ‘deviceMemory’, {
get: () => 8
});

// 伪装屏幕
Object.defineProperty(screen, ‘width’, {get: () => 1920});
Object.defineProperty(screen, ‘height’, {get: () => 1080});
Object.defineProperty(screen, ‘colorDepth’, {get: () => 24});

// 伪装WebGL
const canvas = document.createElement(‘canvas’);
const gl = canvas.getContext(‘webgl’);
if (gl) {
const getParameter = gl.getParameter;
gl.getParameter = function(parameter) {
if (parameter === 37445) { // UNMASKED_VENDOR_WEBGL
return ‘Google Inc. (NVIDIA)’;
}
if (parameter === 37446) { // UNMASKED_RENDERER_WEBGL
return ‘ANGLE (NVIDIA, NVIDIA GeForce RTX 3080 Direct3D11 vs_5_0 ps_5_0, D3D11)’;
}
return getParameter.apply(this, arguments);
};
}
“””)
4.4 行为模式随机化
创建一个行为配置文件,使每次采集都有不同的操作模式:
class BehaviorProfile:
“””用户行为配置文件”””

def __init__(self):
self.scroll_speed = random.choice([‘slow’, ‘medium’, ‘fast’])
self.click_delay = random.uniform(0.5, 2.0)
self.mouse_movement = random.choice([‘straight’, ‘curve’, ‘zigzag’])
self.page_read_time = random.uniform(3, 8) # 模拟阅读页面时间
self.interaction_probability = random.uniform(0.3, 0.7) # 交互概率

def get_scroll_delay(self):
“””根据滚动速度获取延迟”””
if self.scroll_speed == ‘slow’:
return random.uniform(0.8, 1.5)
elif self.scroll_speed == ‘medium’:
return random.uniform(0.3, 0.8)
else:
return random.uniform(0.1, 0.3)

# 在Scraper中使用行为配置
class ZhaopinScraper:
def __init__(self, **kwargs):
# … 其他初始化 …
self.behavior_profile = BehaviorProfile()

def simulate_realistic_browsing(self, duration=10):
“””
模拟真实浏览行为
“””
start_time = time.time()

while time.time() – start_time < duration: # 随机选择行为 action = random.choices( ['scroll', 'pause', 'hover', 'click'], weights=[0.5, 0.3, 0.15, 0.05] )[0] if action == 'scroll': self.human_like_scroll(times=1) time.sleep(self.behavior_profile.get_scroll_delay()) elif action == 'pause': pause_duration = random.uniform(1, 3) print(f"暂停 {pause_duration:.1f} 秒...") time.sleep(pause_duration) elif action == 'hover': # 随机悬停某个元素 try: elements = self.driver.find_elements(By.CSS_SELECTOR, 'a, button, div') if elements: element = random.choice(elements) ActionChains(self.driver).move_to_element(element).perform() time.sleep(random.uniform(0.5, 1.5)) except: pass elif action == 'click': # 小概率点击随机元素 if random.random() < self.behavior_profile.interaction_probability: try: elements = self.driver.find_elements(By.CSS_SELECTOR, 'a:not([href^="javascript"]), button') if elements: element = random.choice(elements) if element.is_displayed() and element.is_enabled(): self.simulate_mouse_movement(element, duration=1) except: pass 五、完整项目代码实现 5.1 主程序结构 # main.py - 完整的智联招聘采集程序 import time import json import random import logging from datetime import datetime from selenium.webdriver.common.by import By from selenium.webdriver.support.ui import WebDriverWait from selenium.webdriver.support import expected_conditions as EC from selenium.common.exceptions import TimeoutException, NoSuchElementException # 配置日志 logging.basicConfig( level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s', handlers=[ logging.FileHandler('zhaopin_scraper.log', encoding='utf-8'), logging.StreamHandler() ] ) class ZhaopinScraper: """ 智联招聘数据采集器 支持免登录、绕过验证码、智能反爬绕过 """ def __init__(self, config=None): self.config = config or {} self.output_file = self.config.get('output_file', 'zhaopin_jobs.json') self.use_proxy = self.config.get('use_proxy', False) self.headless = self.config.get('headless', False) # 初始化数据存储 self.scraped_data = [] self.scraped_urls = set() self.load_existing_data() # 初始化浏览器 logging.info("初始化浏览器驱动...") self.driver = self.setup_browser() self.wait = WebDriverWait(self.driver, 15) # 行为配置 self.behavior_profile = BehaviorProfile() # 代理管理 self.proxy_manager = None if self.use_proxy: proxy_list = self.config.get('proxy_list', []) self.proxy_manager = ProxyManager(proxy_list) logging.info("初始化完成") def setup_browser(self): """配置浏览器""" # 使用undetected-chromedriver import undetected_chromedriver as uc from fake_useragent import UserAgent options = uc.ChromeOptions() options.add_argument('--no-sandbox') options.add_argument('--disable-dev-shm-usage') # 随机User-Agent ua = UserAgent() options.add_argument(f'user-agent={ua.random}') # 无头模式 if self.headless: options.add_argument('--headless') options.add_argument('--window-size=1920,1080') # 代理设置 if self.use_proxy and self.proxy_manager: proxy = self.proxy_manager.get_proxy() if proxy: options.add_argument(f'--proxy-server={proxy}') logging.info(f"使用代理: {proxy}") driver = uc.Chrome(options=options) # 反检测增强 self.enhance_browser_fingerprint(driver) return driver def enhance_browser_fingerprint(self, driver): """增强浏览器指纹伪装""" driver.execute_script(""" Object.defineProperty(navigator, 'webdriver', {get: () => undefined});
Object.defineProperty(navigator, ‘appVersion’, {
get: () => ‘5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36’
});
Object.defineProperty(navigator, ‘languages’, {get: () => [‘zh-CN’, ‘zh’, ‘en-US’, ‘en’]});
Object.defineProperty(navigator, ‘platform’, {get: () => ‘Win32’});
“””)

def load_existing_data(self):
“””加载已采集的数据”””
if os.path.exists(self.output_file):
try:
with open(self.output_file, ‘r’, encoding=’utf-8′) as f:
existing_data = json.load(f)
self.scraped_data = existing_data
self.scraped_urls = {job[‘detail_url’] for job in existing_data if job.get(‘detail_url’)}
logging.info(f”已加载 {len(self.scraped_data)} 条历史数据”)
except Exception as e:
logging.error(f”加载历史数据失败: {e}”)

def save_data(self):
“””保存数据”””
try:
with open(self.output_file, ‘w’, encoding=’utf-8′) as f:
json.dump(self.scraped_data, f, ensure_ascii=False, indent=2)
logging.info(f”数据已保存到 {self.output_file}”)
except Exception as e:
logging.error(f”保存数据失败: {e}”)

def random_delay(self, min_delay=1, max_delay=3):
“””随机延迟”””
delay = random.uniform(min_delay, max_delay)
time.sleep(delay)

def human_like_scroll(self, times=3):
“””模拟人类滚动”””
for _ in range(times):
scroll_distance = random.randint(200, 600)
self.driver.execute_script(f”window.scrollBy(0, {scroll_distance})”)
self.random_delay(0.5, 1.5)

if random.random() > 0.7:
self.driver.execute_script(f”window.scrollBy(0, -{random.randint(50, 150)})”)
self.random_delay(0.3, 0.8)

def simulate_mouse_movement(self, element, duration=1.5):
“””模拟鼠标移动”””
actions = ActionChains(self.driver)
location = element.location
size = element.size

target_x = location[‘x’] + random.randint(5, size[‘width’] – 5)
target_y = location[‘y’] + random.randint(5, size[‘height’] – 5)

start_x = random.randint(100, 500)
start_y = random.randint(100, 300)

steps = 20
for i in range(steps):
progress = i / steps
control_x = start_x + (target_x – start_x) * 0.5 + random.randint(-20, 20)
control_y = start_y + (target_y – start_y) * 0.5 + random.randint(-20, 20)

current_x = (1 – progress) * (1 – progress) * start_x + 2 * (1 – progress) * progress * control_x + progress * progress * target_x
current_y = (1 – progress) * (1 – progress) * start_y + 2 * (1 – progress) * progress * control_y + progress * progress * target_y

actions.move_by_offset(current_x – start_x, current_y – start_y)
actions.perform()

actions = ActionChains(self.driver)
time.sleep(duration / steps * random.uniform(0.8, 1.2))

actions.click()
actions.perform()

def build_search_url(self, keyword, city, salary_range=None, page=1):
“””构建搜索URL”””
base_url = “https://www.zhaopin.com/jobs”
params = [“searchType=2″, f”keyword={keyword}”, f”city={city}”]

if salary_range:
params.append(f”salary={salary_range}”)
if page > 1:
params.append(f”page={page}”)

return f”{base_url}?{‘&’.join(params)}”

def check_for_captcha(self):
“””检查验证码”””
captcha_indicators = [‘验证码’, ‘captcha’, ‘验证’, ‘滑动验证’, ‘请稍后’, ‘访问过于频繁’]
page_source = self.driver.page_source.lower()

for indicator in captcha_indicators:
if indicator in page_source:
return True

try:
if self.driver.find_elements(By.CSS_SELECTOR, ‘iframe[src*=”captcha”]’):
return True
except:
pass

return False

def handle_captcha(self):
“””处理验证码”””
logging.warning(“检测到验证码,启动应对策略…”)

# 策略1:等待重试
wait_time = random.randint(60, 120)
logging.info(f”等待 {wait_time} 秒后重试…”)
time.sleep(wait_time)
self.driver.refresh()

# 策略2:检查是否解决
if not self.check_for_captcha():
logging.info(“验证码已解决”)
return True

# 策略3:更换环境
logging.info(“更换浏览器环境…”)
self更换浏览器环境()
return False

def 更换浏览器环境(self):
“””更换浏览器环境”””
self.driver.quit()
time.sleep(30)
self.driver = self.setup_browser()
self.wait = WebDriverWait(self.driver, 15)
self.enhance_browser_fingerprint(self.driver)

def initial_visit_with_proxy(self, url):
“””使用代理进行初始访问”””
logging.info(“首次访问,等待10秒…”)
time.sleep(10)

self.driver.get(url)
self.random_delay(3, 5)

if self.check_for_captcha():
return self.handle_captcha()

return True

def extract_job_list_data(self):
“””提取职位列表数据”””
job_data = []

try:
self.wait.until(
EC.presence_of_element_located((By.CSS_SELECTOR, ‘div.joblist-box’))
)

job_items = self.driver.find_elements(By.CSS_SELECTOR, ‘div.joblist-box__item’)
logging.info(f”找到 {len(job_items)} 个职位项”)

for item in job_items:
try:
job_info = self.extract_single_job(item)
if job_info:
job_data.append(job_info)
except Exception as e:
logging.debug(f”提取单个职位失败: {e}”)
continue

except Exception as e:
logging.error(f”提取职位列表失败: {e}”)

return job_data

def extract_single_job(self, job_element):
“””提取单个职位信息”””
try:
title = job_element.find_element(By.CSS_SELECTOR, ‘a.joblist-box__item__info__title’).text.strip()
company = job_element.find_element(By.CSS_SELECTOR, ‘div.joblist-box__item__info__company’).text.strip()

try:
salary = job_element.find_element(By.CSS_SELECTOR, ‘span.joblist-box__item__info__salary’).text.strip()
except:
salary = “面议”

location_elements = job_element.find_elements(By.CSS_SELECTOR, ‘div.joblist-box__item__info__bottom span’)
location = location_elements[0].text.strip() if location_elements else “”
publish_time = location_elements[-1].text.strip() if len(location_elements) > 1 else “”

detail_link = job_element.find_element(By.CSS_SELECTOR, ‘a.joblist-box__item__info__title’).get_attribute(‘href’)

return {
‘title’: title,
‘company’: company,
‘salary’: salary,
‘location’: location,
‘publish_time’: publish_time,
‘detail_url’: detail_link,
‘采集时间’: datetime.now().strftime(“%Y-%m-%d %H:%M:%S”)
}
except Exception as e:
logging.debug(f”提取职位信息失败: {e}”)
return None

def has_next_page(self):
“””检查下一页”””
try:
next_button = self.driver.find_element(By.CSS_SELECTOR, ‘button.next-page’)
return ‘disabled’ not in next_button.get_attribute(‘class’)
except:
return False

def simulate_realistic_browsing(self, duration=10):
“””模拟真实浏览行为”””
start_time = time.time()

while time.time() – start_time < duration: action = random.choices(['scroll', 'pause', 'hover', 'click'], weights=[0.5, 0.3, 0.15, 0.05])[0] if action == 'scroll': self.human_like_scroll(times=1) time.sleep(self.behavior_profile.get_scroll_delay()) elif action == 'pause': pause_duration = random.uniform(1, 3) logging.debug(f"暂停 {pause_duration:.1f} 秒...") time.sleep(pause_duration) elif action == 'hover': try: elements = self.driver.find_elements(By.CSS_SELECTOR, 'a, button, div') if elements: element = random.choice(elements) ActionChains(self.driver).move_to_element(element).perform() time.sleep(random.uniform(0.5, 1.5)) except: pass elif action == 'click': if random.random() < self.behavior_profile.interaction_probability: try: elements = self.driver.find_elements(By.CSS_SELECTOR, 'a:not([href^="javascript"]), button') if elements: element = random.choice(elements) if element.is_displayed() and element.is_enabled(): self.simulate_mouse_movement(element, duration=1) except: pass def scrape_pages(self, keyword, city, salary_range=None, max_pages=10): """采集多页数据""" total_new_jobs = 0 for page in range(1, max_pages + 1): logging.info(f"\n{'='*50}") logging.info(f"开始采集第 {page} 页") logging.info(f"{'='*50}") url = self.build_search_url(keyword, city, salary_range, page) if not self.initial_visit_with_proxy(url): logging.warning(f"第 {page} 页访问失败,跳过...") continue # 模拟真实浏览 self.simulate_realistic_browsing(duration=random.uniform(5, 10)) # 提取数据 page_data = self.extract_job_list_data() # 过滤重复数据 new_data = [job for job in page_data if job['detail_url'] not in self.scraped_urls] logging.info(f"本页找到 {len(page_data)} 个职位,其中 {len(new_data)} 个是新的") self.scraped_data.extend(new_data) total_new_jobs += len(new_data) # 保存数据 self.save_data() # 检查下一页 if not self.has_next_page(): logging.info("没有下一页了,采集完成") break # 随机延迟 delay = random.randint(8, 20) logging.info(f"等待 {delay} 秒后继续...") time.sleep(delay) logging.info(f"\n采集完成!总共新增 {total_new_jobs} 条数据,总计 {len(self.scraped_data)} 条") return self.scraped_data def close(self): """关闭浏览器""" if self.driver: self.driver.quit() logging.info("浏览器已关闭") # 主函数 def main(): # 配置 config = { 'output_file': 'zhaopin_jobs.json', 'use_proxy': False, # 如果需要代理,设置为True并提供proxy_list 'proxy_list': [], # 代理列表,如 ['http://user:pass@ip:port', ...] 'headless': False # 是否无头模式 } # 创建采集器 scraper = ZhaopinScraper(config) try: # 采集参数 keyword = "Python" city = "北京" salary_range = "4-6k" max_pages = 5 # 开始采集 results = scraper.scrape_pages( keyword=keyword, city=city, salary_range=salary_range, max_pages=max_pages ) # 输出结果 print(f"\n采集结果:") print(f"总共采集 {len(results)} 条职位数据") # 显示前5条 for i, job in enumerate(results[:5], 1): print(f"{i}. {job['title']} - {job['company']} - {job['salary']}") except Exception as e: logging.error(f"程序执行出错: {e}") finally: scraper.close() if __name__ == "__main__": main() 5.2 数据导出与分析 # export.py - 数据导出工具 import pandas as pd import json def export_to_excel(json_file, excel_file): """导出到Excel""" with open(json_file, 'r', encoding='utf-8') as f: data = json.load(f) df = pd.DataFrame(data) # 数据清洗 df = df.drop_duplicates(subset=['detail_url'], keep='last') # 保存 df.to_excel(excel_file, index=False, engine='openpyxl') print(f"已导出 {len(df)} 条数据到 {excel_file}") def analyze_salary_distribution(json_file): """分析薪资分布""" with open(json_file, 'r', encoding='utf-8') as f: data = json.load(f) df = pd.DataFrame(data) # 解析薪资 def parse_salary(salary_str): if salary_str == "面议": return None, None try: parts = salary_str.replace('k', '000').replace('K', '000').split('-') if len(parts) == 2: return int(parts[0]), int(parts[1]) except: pass return None, None # 应用解析 df[['salary_min', 'salary_max']] = df['salary'].apply( lambda x: pd.Series(parse_salary(x)) ) # 计算平均薪资 df['salary_avg'] = df[['salary_min', 'salary_max']].mean(axis=1) print("\n薪资分析:") print(f"平均薪资: {df['salary_avg'].mean():.0f} 元/月") print(f"薪资中位数: {df['salary_avg'].median():.0f} 元/月") print(f"最高薪资: {df['salary_max'].max():.0f} 元/月") print(f"最低薪资: {df['salary_min'].min():.0f} 元/月") # 按公司统计 company_stats = df.groupby('company').agg({ 'title': 'count', 'salary_avg': 'mean' }).sort_values('title', ascending=False).head(10) print("\n招聘最多的公司:") print(company_stats) # 使用示例 if __name__ == "__main__": export_to_excel('zhaopin_jobs.json', 'zhaopin_jobs.xlsx') analyze_salary_distribution('zhaopin_jobs.json') 六、常见问题与解决方案 6.1 问题:页面加载缓慢或超时 解决方案: def wait_for_page_load(self, timeout=30): """等待页面完全加载""" try: # 等待关键元素出现 self.wait.until( EC.presence_of_element_located((By.CSS_SELECTOR, 'div.joblist-box')) ) # 等待AJAX请求完成 self.wait.until( lambda driver: driver.execute_script('return jQuery.active == 0') if driver.execute_script('return typeof jQuery != "undefined"') else True ) # 额外等待随机时间 self.random_delay(2, 4) return True except TimeoutException: logging.error("页面加载超时") return False 6.2 问题:元素定位失败 解决方案: def find_element_with_retry(self, by, value, max_attempts=3): """带重试的元素定位""" for attempt in range(max_attempts): try: element = self.driver.find_element(by, value) return element except NoSuchElementException: if attempt < max_attempts - 1: logging.warning(f"元素未找到,重试 {attempt + 1}/{max_attempts}") self.random_delay(1, 2) else: raise 6.3 问题:数据重复采集 解决方案: def filter_duplicate_jobs(self, job_list): """过滤重复职位""" unique_jobs = [] seen_urls = set() for job in job_list: if job['detail_url'] not in seen_urls: unique_jobs.append(job) seen_urls.add(job['detail_url']) return unique_jobs 6.4 问题:验证码频繁出现 解决方案: def adaptive_delay_strategy(self, page_count): """自适应延迟策略""" # 随着页数增加,延迟时间递增 base_delay = 5 delay = base_delay + (page_count * 2) # 添加随机波动 delay += random.uniform(-2, 5) # 确保最小延迟 delay = max(delay, 8) logging.info(f"自适应延迟: {delay:.1f} 秒") time.sleep(delay) 七、最佳实践与优化建议 7.1 采集频率控制 class RateLimiter: """速率限制器""" def __init__(self, max_requests_per_minute=10): self.max_requests = max_requests_per_minute self.request_times = [] def wait_if_needed(self): """如果需要则等待""" now = time.time() # 清理1分钟前的记录 self.request_times = [t for t in self.request_times if now - t < 60] if len(self.request_times) >= self.max_requests:
# 计算需要等待的时间
oldest_request = min(self.request_times)
wait_time = 60 – (now – oldest_request) + 1
logging.info(f”速率限制,等待 {wait_time:.1f} 秒”)
time.sleep(wait_time)

self.request_times.append(time.time())
7.2 错误处理与恢复
def robust_scrape(self, keyword, city, salary_range=None, max_pages=10):
“””鲁棒性采集,支持断点续传”””
start_page = len(self.scraped_data) // 20 + 1 # 假设每页20条

for page in range(start_page, max_pages + 1):
try:
# 采集逻辑…
pass
except Exception as e:
logging.error(f”第 {page} 页采集失败: {e}”)
# 保存当前进度
self.save_data()

# 等待更长时间后重试
wait_time = 60 * (page – start_page + 1)
logging.info(f”等待 {wait_time} 秒后重试…”)
time.sleep(wait_time)

# 重新初始化浏览器
self.更换浏览器环境()
7.3 性能优化
# 使用多线程(谨慎使用,容易触发反爬)
from concurrent.futures import ThreadPoolExecutor, as_completed

def scrape_concurrently(self, tasks, max_workers=2):
“””并发采集(低频率)”””
results = []

with ThreadPoolExecutor(max_workers=max_workers) as executor:
future_to_task = {executor.submit(self.scrape_single_task, task): task for task in tasks}

for future in as_completed(future_to_task):
try:
result = future.result()
results.append(result)
except Exception as e:
logging.error(f”任务执行失败: {e}”)

return results
八、法律与道德 considerations
8.1 合规性建议
遵守robots.txt
:检查智联招聘的robots.txt文件,尊重爬虫协议
控制采集频率
:避免对服务器造成过大负担
数据使用目的
:确保数据用于合法的商业分析或个人学习
隐私保护
:不要采集或存储个人敏感信息
8.2 技术限制建议
# 在代码中添加合规性检查
def check_compliance(self):
“””检查合规性”””
# 1. 采集频率限制
if self.get_requests_per_minute() > 20:
logging.warning(“采集频率过高,已自动降低”)
return False

# 2. 检查是否采集了过多数据
if len(self.scraped_data) > 10000:
logging.warning(“数据量过大,建议停止采集”)
return False

return True
九、总结
通过本文介绍的方法,我们可以使用Selenium实现智联招聘的免登录自动采集,并有效绕过验证码和登录限制。关键点包括:
使用undetected-chromedriver
:有效绕过浏览器自动化检测
模拟真实用户行为
:随机延迟、鼠标轨迹、滚动模式
智能验证码处理
:多种策略应对验证码
数据去重与持久化
:确保数据完整性
代理轮换
:避免IP封禁
这套方案在实际应用中表现稳定,能够持续采集数据。但需要注意的是,任何爬虫技术都应该在合法合规的前提下使用,尊重目标网站的权益。
最后,建议在实际部署时:
使用云服务器部署,避免本地IP被封
定期更新反检测策略
监控采集成功率,及时调整参数
备份重要数据,防止意外丢失
通过这些技术手段和最佳实践,你将能够高效、稳定地采集智联招聘的职位数据,为你的业务分析或研究提供有力支持。