引言：XPath在现代Web数据提取中的核心地位
XPath（XML Path Language）作为一种强大的查询语言，专门用于在XML和HTML文档中导航和选择节点。在当今数据驱动的时代，XPath已成为Web爬虫、数据抓取、自动化测试和内容管理等领域的核心技术。与CSS选择器相比，XPath提供了更精细的控制能力，能够处理复杂的文档结构、执行条件判断和实现多层嵌套查询。
本文将系统性地解析XPath的语法体系，从基础选择器到高级函数应用，再到复杂的实战案例，帮助读者全面掌握XPath的精髓。无论您是初学者还是希望提升技能的开发者，本文都将提供实用的指导和深入的见解。
第一部分：XPath基础语法体系
1.1 节点选择基础
XPath通过路径表达式来选择文档中的节点。理解节点关系是掌握XPath的第一步。
基本节点类型：
根节点
：文档的顶级节点
元素节点
：HTML/XML标签
属性节点
：标签的属性
文本节点
：标签内的文本内容
注释节点
：文档中的注释
基本路径表达式示例：

基础选择器示例：
# Python中使用lxml库演示XPath
from lxml import etree

html = “””

“””

tree = etree.HTML(html)

# 1. 绝对路径选择
absolute_path = tree.xpath(‘/html/body/div’)
print(f”绝对路径选择: {[elem.tag for elem in absolute_path]}”)

# 2. 相对路径选择
relative_path = tree.xpath(‘//div’)
print(f”相对路径选择: {[elem.tag for elem in relative_path]}”)

# 3. 通配符选择
wildcard = tree.xpath(‘//div/*’)
print(f”通配符选择: {[elem.tag for elem in wildcard]}”)

# 4. 选择所有节点
all_nodes = tree.xpath(‘//*’)
print(f”所有节点数量: {len(all_nodes)}”)
1.2 轴（Axes）的使用
轴定义了相对于当前节点的节点集合，是XPath强大功能的核心。
常用轴说明：
轴名称
说明
示例
child
当前节点的子节点
child::div
parent
当前节点的父节点
parent::div
ancestor
当前节点的所有祖先节点
ancestor::div
descendant
当前节点的所有后代节点
descendant::a
following-sibling
当前节点之后的所有同级节点
following-sibling::li
preceding-sibling
当前节点之前的所有同级节点
preceding-sibling::li
attribute
当前节点的属性
attribute::class
轴的实战应用：
# 继续使用上面的HTML文档

# 1. child轴：选择所有子节点
children = tree.xpath(‘//div[@id=”container”]/child::*’)
print(f”container的子节点: {[elem.tag for elem in children]}”)

# 2. following-sibling轴：选择后续同级节点
following = tree.xpath(‘//h1[@class=”title”]/following-sibling::div’)
print(f”h1之后的div兄弟: {[elem.tag for elem in following]}”)

# 3. ancestor轴：选择祖先节点
ancestors = tree.xpath(‘//a[@href=”/home”]/ancestor::div’)
print(f”a元素的祖先div: {[elem.tag for elem in ancestors]}”)

# 4. attribute轴：选择属性
attrs = tree.xpath(‘//h1/@class’)
print(f”h1的class属性: {attrs}”)

# 5. 组合轴使用
complex_query = tree.xpath(‘//ul[@class=”nav”]/child::li[1]/following-sibling::li/child::a/@href’)
print(f”导航链接的href: {complex_query}”)
1.3 谓词（Predicates）与过滤
谓词用于在路径表达式中添加条件，实现精确筛选。
基本谓词语法：
# 示例HTML – 包含复杂结构
html_complex = “””

“””

tree_complex = etree.HTML(html_complex)

# 1. 索引谓词
first_product = tree_complex.xpath(‘//product[1]/name/text()’)
print(f”第一个产品: {first_product}”)

# 2. 条件谓词
electronics = tree_complex.xpath(‘//product[@category=”electronics”]/name/text()’)
print(f”电子产品: {electronics}”)

# 3. 比较运算符
expensive = tree_complex.xpath(‘//product[@price > 1000]/name/text()’)
print(f”价格大于1000的产品: {expensive}”)

# 4. 逻辑运算符
in_stock = tree_complex.xpath(‘//product[@price < 1000 and stock > 0]/name/text()’)
print(f”价格小于1000且有库存: {in_stock}”)

# 5. 位置函数
last_product = tree_complex.xpath(‘//product[last()]/name/text()’)
print(f”最后一个产品: {last_product}”)

# 6. 包含函数
contains_name = tree_complex.xpath(‘//product[contains(name, “书”)]/name/text()’)
print(f”名称包含’书’的产品: {contains_name}”)
1.4 通配符与多路径选择
通配符使用：
# 示例HTML – 动态类名
html_wildcard = “””

“””

tree_wildcard = etree.HTML(html_wildcard)

# 1. 匹配所有div
all_divs = tree_wildcard.xpath(‘//div’)
print(f”所有div数量: {len(all_divs)}”)

# 2. 匹配特定前缀的类名
header_divs = tree_wildcard.xpath(‘//div[starts-with(@class, “header”)]’)
print(f”header类div: {[elem.text for elem in header_divs]}”)

# 3. 匹配特定后缀的类名
footer_divs = tree_wildcard.xpath(‘//div[ends-with(@class, “footer-1”) or ends-with(@class, “footer-2″)]’)
print(f”footer类div: {[elem.text for elem in footer_divs]}”)

# 4. 多路径选择（使用 | 运算符）
multiple = tree_wildcard.xpath(‘//div[@class=”header-1″] | //span[@class=”special”]’)
print(f”多路径选择结果: {[elem.text for elem in multiple]}”)
第二部分：XPath高级函数与技巧
2.1 字符串处理函数
XPath提供了丰富的字符串处理函数，用于数据清洗和格式化。
常用字符串函数：
# 示例HTML – 包含复杂文本
html_text = “””

“””

tree_text = etree.HTML(html_text)

# 1. normalize-space() – 去除首尾空白和压缩中间空白
name = tree_text.xpath(‘normalize-space(//span[@class=”name”])’)
print(f”标准化名称: ‘{name}’”)

# 2. concat() – 连接字符串
full_info = tree_text.xpath(‘concat(//span[@class=”name”], ” – “, //span[@class=”email”])’)
print(f”连接信息: {full_info}”)

# 3. substring() – 截取子串
phone_partial = tree_text.xpath(‘substring(//span[@class=”phone”], 5, 3)’)
print(f”手机号中间三位: {phone_partial}”)

# 4. string-length() – 字符串长度
bio_length = tree_text.xpath(‘string-length(normalize-space(//span[@class=”bio”]))’)
print(f”简介长度: {bio_length}”)

# 5. contains() – 包含判断
has_python = tree_text.xpath(‘//span[contains(normalize-space(.), “Python”)]’)
print(f”包含Python的元素: {[elem.text for elem in has_python]}”)

# 6. starts-with() – 开头判断
is_zhang = tree_text.xpath(‘//span[starts-with(normalize-space(.), “张”)]’)
print(f”以’张’开头的元素: {[elem.text for elem in is_zhang]}”)

# 7. translate() – 字符替换
translated = tree_text.xpath(‘translate(//span[@class=”phone”], “-“, “”)’)
print(f”去除横杠的电话: {translated}”)
2.2 数值与数学函数
数值处理函数示例：
# 示例HTML – 包含数值数据
html_number = “””

“””

tree_number = etree.HTML(html_number)

# 1. sum() – 求和
total_amount = tree_number.xpath(‘sum(//order/@amount)’)
print(f”总金额: {total_amount}”)

# 2. round() – 四舍五入
avg_price = tree_number.xpath(’round(sum(//order/@amount) div count(//order))’)
print(f”平均价格: {avg_price}”)

# 3. floor() – 向下取整
floor_avg = tree_number.xpath(‘floor(sum(//order/@amount) div count(//order))’)
print(f”向下取整平均: {floor_avg}”)

# 4. ceiling() – 向上取整
ceiling_avg = tree_number.xpath(‘ceiling(sum(//order/@amount) div count(//order))’)
print(f”向上取整平均: {ceiling_avg}”)

# 5. 复杂数学计算
total_value = tree_number.xpath(‘sum(//order/@amount * //order/@quantity)’)
print(f”总销售额: {total_value}”)

# 6. 比较运算
high_value_orders = tree_number.xpath(‘//order[@amount * @quantity > 1000]/@id’)
print(f”高价值订单ID: {high_value_orders}”)
2.3 日期与时间函数
虽然XPath 1.0的日期函数有限，但可以通过字符串处理实现基本操作。
# 示例HTML – 包含日期
html_date = “””

“””

tree_date = etree.HTML(html_date)

# 1. 提取日期部分
dates = tree_date.xpath(‘//event/@date’)
print(f”所有日期: {dates}”)

# 2. 按日期排序（需要额外处理）
events_by_date = tree_date.xpath(‘//event’)
sorted_events = sorted(events_by_date, key=lambda x: x.get(‘date’))
print(f”按日期排序: {[elem.get(‘date’) for elem in sorted_events]}”)

# 3. 日期范围筛选
jan_events = tree_date.xpath(‘//event[starts-with(@date, “2024-01″)]/@date’)
print(f”1月份事件日期: {jan_events}”)

# 4. 时间提取
times = tree_date.xpath(‘//event/@time’)
print(f”所有时间: {times}”)

# 5. 组合日期时间
datetime_info = tree_date.xpath(‘concat(//event[1]/@date, ” “, //event[1]/@time)’)
print(f”第一个事件的日期时间: {datetime_info}”)
2.4 节点集函数
节点集处理函数：
# 示例HTML – 包含多个列表
html_lists = “””

“””

tree_lists = etree.HTML(html_lists)

# 1. count() – 计数
ul_count = tree_lists.xpath(‘count(//ul)’)
print(f”ul元素数量: {ul_count}”)

li_count = tree_lists.xpath(‘count(//li)’)
print(f”li元素总数: {li_count}”)

# 2. position() – 位置
first_li = tree_lists.xpath(‘//ul[1]/li[position()=1]/text()’)
print(f”第一个ul的第一个li: {first_li}”)

# 3. last() – 最后一个位置
last_li = tree_lists.xpath(‘//ul[last()]/li[last()]/text()’)
print(f”最后一个ul的最后一个li: {last_li}”)

# 4. 复杂位置筛选
middle_items = tree_lists.xpath(‘//ul[2]/li[position() > 1 and position() < 3]/text()') print(f"第二个ul的中间项: {middle_items}") # 5. 节点集比较 has_three_items = tree_lists.xpath('//ul[count(li) = 3]/@class') print(f"包含3个li的ul: {has_three_items}") 第三部分：复杂查询实战案例 3.1 案例1：电商网站商品筛选系统 场景描述： 从复杂的电商页面中提取符合特定条件的商品信息。 # 模拟电商页面HTML ecommerce_html = """

“””

tree_ecommerce = etree.HTML(ecommerce_html)

# 复杂查询1：获取所有有库存且评分>4.7的电子产品
query1 = tree_ecommerce.xpath(”’
//div[@class=”product-item” and
@data-category=”electronics” and
@data-stock > 0 and
.//span[@class=”score”] > 4.7]
/div[@class=”product-header”]/h3/text()
”’)
print(f”高评分有库存电子产品: {query1}”)

# 复杂查询2：获取折扣大于20%的商品名称和折扣率
query2 = tree_ecommerce.xpath(”’
//div[@class=”product-item”]
[number(translate(.//span[@class=”discount”], “%-“, “”)) > 20]
/div[@class=”product-header”]/h3/text()
”’)
discount_rates = tree_ecommerce.xpath(”’
//div[@class=”product-item”]
[number(translate(.//span[@class=”discount”], “%-“, “”)) > 20]
/div[@class=”product-body”]//span[@class=”discount”]/text()
”’)
print(f”折扣大于20%的商品: {list(zip(query2, discount_rates))}”)

# 复杂查询3：获取Apple品牌的所有商品ID和库存
query3 = tree_ecommerce.xpath(”’
//div[@class=”product-item” and .//span[@class=”brand”] = “Apple”]/@data-id
”’)
query3_stock = tree_ecommerce.xpath(”’
//div[@class=”product-item” and .//span[@class=”brand”] = “Apple”]/@data-stock
”’)
print(f”Apple商品ID和库存: {list(zip(query3, query3_stock))}”)

# 复杂查询4：获取所有包含”热销”或”畅销”标签的商品
query4 = tree_ecommerce.xpath(”’
//div[@class=”product-item” and
(.//span[@class=”tag”] = “热销” or .//span[@class=”tag”] = “畅销”)]
/div[@class=”product-header”]/h3/text()
”’)
print(f”热销/畅销商品: {query4}”)

# 复杂查询5：计算所有电子产品的平均价格
query5 = tree_ecommerce.xpath(”’
sum(//div[@class=”product-item” and @data-category=”electronics”]/@data-price) div
count(//div[@class=”product-item” and @data-category=”electronics”])
”’)
print(f”电子产品平均价格: {query5}”)
3.2 案例2：新闻网站内容提取
场景描述：
从新闻网站提取结构化信息，包括标题、作者、发布时间、分类和正文。
# 模拟新闻页面HTML
news_html = “””

“””

tree_news = etree.HTML(news_html)

# 查询1：提取完整的新闻元数据
title = tree_news.xpath(‘normalize-space(//h1[@class=”news-title”])’)
author = tree_news.xpath(‘normalize-space(//span[@class=”author”])’)
publish_date = tree_news.xpath(‘normalize-space(//span[@class=”publish-date”])’)
category = tree_news.xpath(‘normalize-space(//span[@class=”category”])’)
source = tree_news.xpath(‘normalize-space(//span[@class=”source”])’)

print(f”新闻标题: {title}”)
print(f”作者: {author}”)
print(f”发布时间: {publish_date}”)
print(f”分类: {category}”)
print(f”来源: {source}”)

# 查询2：提取正文内容（处理段落和列表）
paragraphs = tree_news.xpath(‘//div[@class=”news-content”]//p/text()’)
list_items = tree_news.xpath(‘//div[@class=”news-content”]//ul/li/text()’)
image_caption = tree_news.xpath(‘//div[@class=”news-content”]//span[@class=”caption”]/text()’)

print(f”段落: {paragraphs}”)
print(f”列表项: {list_items}”)
print(f”图片说明: {image_caption}”)

# 查询3：提取所有标签
tags = tree_news.xpath(‘//div[@class=”news-tags”]//span[@class=”tag”]/text()’)
print(f”新闻标签: {tags}”)

# 查询4：提取评论信息
comment_authors = tree_news.xpath(‘//div[@class=”comment”]//span[@class=”comment-author”]/text()’)
comment_times = tree_news.xpath(‘//div[@class=”comment”]//span[@class=”comment-time”]/text()’)
comment_contents = tree_news.xpath(‘//div[@class=”comment”]//p[@class=”comment-content”]/text()’)

comments = list(zip(comment_authors, comment_times, comment_contents))
print(f”评论信息: {comments}”)

# 查询5：提取图片信息
image_src = tree_news.xpath(‘//div[@class=”news-content”]//img/@src’)
image_alt = tree_news.xpath(‘//div[@class=”news-content”]//img/@alt’)
print(f”图片信息: {list(zip(image_src, image_alt))}”)

# 查询6：统计信息
paragraph_count = tree_news.xpath(‘count(//div[@class=”news-content”]//p)’)
list_item_count = tree_news.xpath(‘count(//div[@class=”news-content”]//ul/li)’)
comment_count = tree_news.xpath(‘count(//div[@class=”news-comments”]//div[@class=”comment”])’)

print(f”段落数: {paragraph_count}”)
print(f”列表项数: {list_item_count}”)
print(f”评论数: {comment_count}”)
3.3 案例3：论坛帖子分析
场景描述：
从论坛页面提取帖子信息，包括用户信息、回复内容、时间线和互动数据。
# 模拟论坛页面HTML
forum_html = “””

“””

tree_forum = etree.HTML(forum_html)

# 查询1：提取帖子基本信息
thread_title = tree_forum.xpath(‘normalize-space(//h2[@class=”thread-title”])’)
author = tree_forum.xpath(‘normalize-space(//span[@class=”author”])’)
author_id = tree_forum.xpath(‘//span[@class=”author”]/@data-userid’)
post_time = tree_forum.xpath(‘normalize-space(//span[@class=”post-time”])’)
views = tree_forum.xpath(‘//span[@class=”views”]/text()’)
replies = tree_forum.xpath(‘//span[@class=”replies”]/text()’)

print(f”帖子标题: {thread_title}”)
print(f”作者: {author} (ID: {author_id[0]})”)
print(f”发布时间: {post_time}”)
print(f”浏览量: {views[0]}, 回复数: {replies[0]}”)

# 查询2：提取所有回复的详细信息
reply_authors = tree_forum.xpath(‘//div[@class=”reply”]//span[@class=”reply-author”]/text()’)
reply_author_ids = tree_forum.xpath(‘//div[@class=”reply”]//span[@class=”reply-author”]/@data-userid’)
reply_times = tree_forum.xpath(‘//div[@class=”reply”]//span[@class=”reply-time”]/text()’)
reply_floors = tree_forum.xpath(‘//div[@class=”reply”]//span[@class=”reply-floor”]/text()’)
reply_contents = tree_forum.xpath(‘//div[@class=”reply”]//div[@class=”reply-content”]//p/text()’)
reply_likes = tree_forum.xpath(‘//div[@class=”reply”]//span[@class=”likes”]/text()’)
reply_dislikes = tree_forum.xpath(‘//div[@class=”reply”]//span[@class=”dislikes”]/text()’)
reply_levels = tree_forum.xpath(‘//div[@class=”reply”]/@data-level’)

# 组合回复信息
replies_info = []
for i in range(len(reply_authors)):
replies_info.append({
‘author’: reply_authors[i],
‘author_id’: reply_author_ids[i],
‘time’: reply_times[i],
‘floor’: reply_floors[i],
‘content’: reply_contents[i] if i < len(reply_contents) else '', 'likes': reply_likes[i], 'dislikes': reply_dislikes[i], 'level': reply_levels[i] }) print("\n回复详情:") for reply in replies_info: print(f" {reply['floor']} - {reply['author']}: {reply['content'][:30]}... (👍{reply['likes']} 👎{reply['dislikes']})") # 查询3：统计互动数据 total_likes = tree_forum.xpath('normalize-space(//span[@class="total-likes"])') total_dislikes = tree_forum.xpath('normalize-space(//span[@class="total-dislikes"])') engagement_rate = tree_forum.xpath('normalize-space(//span[@class="engagement-rate"])') print(f"\n互动统计: 总点赞 {total_likes}, 总点踩 {total_dislikes}, 互动率 {engagement_rate}") # 查询4：筛选高互动回复（点赞>20）
high_engagement = tree_forum.xpath(”’
//div[@class=”reply” and .//span[@class=”likes”] > 20]
//span[@class=”reply-author”]/text()
”’)
print(f”高互动回复作者: {high_engagement}”)

# 查询5：提取楼主的所有回复
op_replies = tree_forum.xpath(”’
//div[@class=”reply” and .//span[@class=”reply-author”] = “资深用户”]
//span[@class=”reply-floor”]/text()
”’)
print(f”楼主回复楼层: {op_replies}”)

# 查询6：按层级统计回复数
level1_count = tree_forum.xpath(‘count(//div[@class=”reply” and @data-level=”1″])’)
level2_count = tree_forum.xpath(‘count(//div[@class=”reply” and @data-level=”2″])’)
print(f”一级回复: {level1_count}, 二级回复: {level2_count}”)
3.4 案例4：表格数据提取
场景描述：
从HTML表格中提取结构化数据，处理合并单元格和复杂表头。
# 模拟复杂表格HTML
table_html = “””

ID	姓名	销售数据		部门
		Q1	Q2
001	张三	12000	15000	销售部
002	李四	18000	22000	市场部
003	王五	9000	11000	技术部
004	赵六	15000	18000	销售部
总计		54000	66000	–

“””

tree_table = etree.HTML(table_html)

# 查询1：提取表头信息
header_rows = tree_table.xpath(‘//table/thead/tr’)
print(f”表头行数: {len(header_rows)}”)

# 获取合并表头的完整信息
headers = []
for row in header_rows:
row_headers = []
for th in row.xpath(‘.//th’):
rowspan = th.get(‘rowspan’, ‘1’)
colspan = th.get(‘colspan’, ‘1’)
text = th.text_content().strip()
row_headers.append({
‘text’: text,
‘rowspan’: int(rowspan),
‘colspan’: int(colspan)
})
headers.append(row_headers)

print(“表头结构:”)
for i, row in enumerate(headers):
print(f” 行{i+1}: {row}”)

# 查询2：提取所有数据行
data_rows = tree_table.xpath(‘//table/tbody/tr’)
print(f”\n数据行数: {len(data_rows)}”)

# 提取每行的数据
table_data = []
for row in data_rows:
cells = row.xpath(‘.//td/text()’)
table_data.append(cells)

print(“表格数据:”)
for row in table_data:
print(f” {row}”)

# 查询3：按条件筛选数据
# 筛选销售部员工
sales_dept = tree_table.xpath(‘//table/tbody/tr[td[5] = “销售部”]/td[2]/text()’)
print(f”\n销售部员工: {sales_dept}”)

# 筛选Q1销售额>10000的员工
high_q1 = tree_table.xpath(‘//table/tbody/tr[td[3] > 10000]/td[2]/text()’)
print(f”Q1销售额>10000: {high_q1}”)

# 筛选Q2比Q1增长超过20%的员工
growth = tree_table.xpath(”’
//table/tbody/tr[
(number(td[4]) – number(td[3])) div number(td[3]) > 0.2
]/td[2]/text()
”’)
print(f”Q2增长>20%: {growth}”)

# 查询4：计算统计信息
total_q1 = tree_table.xpath(‘sum(//table/tbody/tr/td[3])’)
total_q2 = tree_table.xpath(‘sum(//table/tbody/tr/td[4])’)
avg_q1 = tree_table.xpath(‘sum(//table/tbody/tr/td[3]) div count(//table/tbody/tr)’)
avg_q2 = tree_table.xpath(‘sum(//table/tbody/tr/td[4]) div count(//table/tbody/tr)’)

print(f”\n统计信息:”)
print(f” Q1总和: {total_q1}, 平均: {avg_q1:.2f}”)
print(f” Q2总和: {total_q2}, 平均: {avg_q2:.2f}”)

# 查询5：按部门分组统计
departments = tree_table.xpath(‘distinct-values(//table/tbody/tr/td[5])’)
print(f”\n部门列表: {departments}”)

for dept in departments:
dept_total = tree_table.xpath(f’sum(//table/tbody/tr[td[5] = “{dept}”]/td[3])’)
print(f” {dept} Q1总和: {dept_total}”)

# 查询6：提取表尾总计
footer_data = tree_table.xpath(‘//table/tfoot/tr/td/text()’)
print(f”\n表尾数据: {footer_data}”)
3.5 案例5：动态内容与嵌套结构
场景描述：
处理深度嵌套的动态内容，如多级菜单、树形结构等。
# 模拟嵌套结构HTML
nested_html = “””

“””

tree_nested = etree.HTML(nested_html)

# 查询1：获取所有节点名称和值
node_names = tree_nested.xpath(‘//span[@class=”node-name”]/text()’)
node_values = tree_nested.xpath(‘//span[@class=”node-value”]/text()’)
node_ids = tree_nested.xpath(‘//li/@data-id’)

print(“所有节点:”)
for i in range(len(node_names)):
print(f” ID: {node_ids[i]}, 名称: {node_names[i]}, 值: {node_values[i]}”)

# 查询2：获取所有叶子节点（没有子ul的li）
leaf_nodes = tree_nested.xpath(‘//li[not(ul)]’)
print(f”\n叶子节点数量: {len(leaf_nodes)}”)

for leaf in leaf_nodes:
name = leaf.xpath(‘.//span[@class=”node-name”]/text()’)[0]
value = leaf.xpath(‘.//span[@class=”node-value”]/text()’)[0]
print(f” {name}: {value}”)

# 查询3：获取所有层级为2的节点（直接子节点）
level2_nodes = tree_nested.xpath(‘//ul[@class=”root”]/li/ul/li’)
print(f”\n层级2节点数量: {len(level2_nodes)}”)

# 查询4：计算每个根节点的总值
root_items = tree_nested.xpath(‘//ul[@class=”root”]/li’)
for item in root_items:
name = item.xpath(‘.//span[@class=”node-name”]/text()’)[0]
# 获取该节点及其所有后代的值
all_values = item.xpath(‘.//span[@class=”node-value”]/text()’)
total = sum(map(int, all_values))
print(f”\n根节点 {name} 及其后代总值: {total}”)

# 查询5：查找特定值的节点路径
target_value = “50”
nodes_with_value = tree_nested.xpath(f’//li[.//span[@class=”node-value”] = “{target_value}”]’)
print(f”\n值为{target_value}的节点:”)
for node in nodes_with_value:
name = node.xpath(‘.//span[@class=”node-name”]/text()’)[0]
# 获取完整路径
path = node.xpath(‘string(ancestor-or-self::li//span[@class=”node-name”])’)
print(f” {name} – 路径: {path}”)

# 查询6：获取节点深度信息
def get_node_depth(node):
“””计算节点深度”””
depth = node.xpath(‘count(ancestor::li)’)
return int(depth)

all_li = tree_nested.xpath(‘//li’)
print(f”\n节点深度信息:”)
for li in all_li:
name = li.xpath(‘.//span[@class=”node-name”]/text()’)[0]
depth = get_node_depth(li)
print(f” {name}: 深度 {depth}”)

# 查询7：查找所有包含子节点的节点
parent_nodes = tree_nested.xpath(‘//li[ul]’)
print(f”\n包含子节点的节点数量: {len(parent_nodes)}”)
for parent in parent_nodes:
name = parent.xpath(‘.//span[@class=”node-name”]/text()’)[0]
child_count = parent.xpath(‘count(ul/li)’)
print(f” {name}: 有 {child_count} 个子节点”)

# 查询8：获取所有路径信息
def get_full_path(node):
“””获取节点的完整路径”””
path_parts = node.xpath(‘.//span[@class=”node-name”]/text()’)
return ” > “.join(path_parts)

print(f”\n所有节点的完整路径:”)
for li in all_li:
path = get_full_path(li)
print(f” {path}”)
第四部分：性能优化与最佳实践
4.1 XPath查询性能优化
优化策略：
# 性能对比示例
import time

# 优化前：使用通配符和深层遍历
def slow_query(tree):
return tree.xpath(‘//*//div//span[@class=”target”]’)

# 优化后：使用具体标签和轴
def fast_query(tree):
return tree.xpath(‘//div[@class=”container”]/child::span[@class=”target”]’)

# 性能测试
def performance_test():
# 创建测试数据
test_html = “””

“”” * 1000 # 重复1000次增加数据量

tree = etree.HTML(test_html)

# 测试慢查询
start = time.time()
result1 = slow_query(tree)
time1 = time.time() – start

# 测试快查询
start = time.time()
result2 = fast_query(tree)
time2 = time.time() – start

print(f”慢查询耗时: {time1:.4f}秒, 结果数: {len(result1)}”)
print(f”快查询耗时: {time2:.4f}秒, 结果数: {len(result2)}”)
print(f”性能提升: {time1/time2:.2f}倍”)

# 性能优化技巧：
# 1. 避免使用通配符 *，使用具体标签
# 2. 避免使用 //，使用具体路径
# 3. 使用轴替代深层遍历
# 4. 使用谓词提前过滤
# 5. 避免在谓词中使用复杂函数
4.2 常见陷阱与解决方案
陷阱1：命名空间问题
# XML命名空间处理
xml_with_ns = “””


测试


“””

tree_ns = etree.fromstring(xml_with_ns)

# 错误方式：直接查询
# result = tree_ns.xpath(‘//name’) # 返回空

# 正确方式：注册命名空间
ns = {‘ns’: ‘http://example.com/ns’}
result = tree_ns.xpath(‘//ns:name’, namespaces=ns)
print(f”命名空间查询结果: {[elem.text for elem in result]}”)
陷阱2：文本节点处理
# 文本节点处理示例
html_text = “””

“””

tree_text = etree.HTML(html_text)

# 错误：只获取直接文本
direct_text = tree_text.xpath(‘//p[1]/text()’)
print(f”直接文本: {direct_text}”) # [‘第一行’]

# 正确：使用text-content()
all_text = tree_text.xpath(‘normalize-space(//p[1])’)
print(f”所有文本: {all_text}”) # ‘第一行第二行’

# 获取所有文本节点
all_text_nodes = tree_text.xpath(‘//p//text()’)
print(f”所有文本节点: {all_text_nodes}”) # [‘第一行’, ‘第二行’, ‘第三行 ‘, ‘内联’, ‘ 第四行’]
陷阱3：动态类名处理
# 动态类名处理
html_dynamic = “””

“””

tree_dynamic = etree.HTML(html_dynamic)

# 使用starts-with和contains
header = tree_dynamic.xpath(‘//div[starts-with(@class, “header-“)]’)
footer = tree_dynamic.xpath(‘//div[contains(@class, “footer”)]’)
main = tree_dynamic.xpath(‘//div[contains(@class, “main”)]’)

print(f”动态头部: {[elem.text for elem in header]}”)
print(f”动态底部: {[elem.text for elem in footer]}”)
print(f”主内容: {[elem.text for elem in main]}”)
4.3 调试技巧
调试XPath表达式：
# 调试函数
def debug_xpath(tree, xpath_expr):
“””调试XPath表达式”””
print(f”\n调试XPath: {xpath_expr}”)
try:
result = tree.xpath(xpath_expr)
print(f”结果数量: {len(result)}”)
if len(result) > 0:
print(f”第一个结果类型: {type(result[0])}”)
if hasattr(result[0], ‘tag’):
print(f”第一个结果标签: {result[0].tag}”)
print(f”第一个结果文本: {result[0].text}”)
else:
print(f”第一个结果值: {result[0]}”)
else:
print(“未找到结果”)
return result
except Exception as e:
print(f”错误: {e}”)
return None

# 使用示例
test_html = “””

“””
tree_test = etree.HTML(test_html)

# 调试各种表达式
debug_xpath(tree_test, ‘//span’)
debug_xpath(tree_test, ‘//div[@class=”test”]/span’)
debug_xpath(tree_test, ‘//span[@id=”s1″]/text()’)
debug_xpath(tree_test, ‘count(//span)’)
第五部分：实战应用与集成
5.1 与Python爬虫集成
import requests
from lxml import etree

class XPathScraper:
def __init__(self):
self.session = requests.Session()
self.session.headers.update({
‘User-Agent’: ‘Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36’
})

def fetch_page(self, url):
“””获取页面内容”””
try:
response = self.session.get(url, timeout=10)
response.raise_for_status()
return response.text
except Exception as e:
print(f”获取页面失败: {e}”)
return None

def extract_with_xpath(self, html, xpath_dict):
“””
使用XPath字典批量提取数据
xpath_dict格式: {‘字段名’: ‘XPath表达式’}
“””
tree = etree.HTML(html)
result = {}

for field, xpath in xpath_dict.items():
try:
elements = tree.xpath(xpath)
if elements:
# 根据元素类型处理
if hasattr(elements[0], ‘text’):
result[field] = [elem.text.strip() if elem.text else ” for elem in elements]
elif isinstance(elements[0], str):
result[field] = elements
else:
result[field] = elements
else:
result[field] = []
except Exception as e:
result[field] = f”错误: {e}”

return result

# 使用示例
scraper = XPathScraper()

# 定义提取规则
xpath_rules = {
‘title’: ‘//h1/text()’,
‘prices’: ‘//span[@class=”price”]/text()’,
‘links’: ‘//a/@href’,
‘descriptions’: ‘//div[@class=”desc”]/text()’,
‘stock_status’: ‘//span[@class=”stock”]/@data-stock’
}

# 模拟页面内容
sample_html = “””

商品列表

“””

# 提取数据
data = scraper.extract_with_xpath(sample_html, xpath_rules)
print(“提取结果:”)
for key, value in data.items():
print(f” {key}: {value}”)
5.2 与Selenium集成
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from lxml import etree
import time

class SeleniumXPathScraper:
def __init__(self, headless=True):
options = webdriver.ChromeOptions()
if headless:
options.add_argument(‘–headless’)
self.driver = webdriver.Chrome(options=options)
self.wait = WebDriverWait(self.driver, 10)

def get_page_source(self, url):
“””获取动态页面源码”””
self.driver.get(url)
# 等待页面加载
time.sleep(2)
return self.driver.page_source

def find_elements_by_xpath(self, xpath):
“””使用Selenium的XPath查找元素”””
return self.driver.find_elements(By.XPATH, xpath)

def extract_dynamic_content(self, url, xpath_dict):
“””提取动态内容”””
html = self.get_page_source(url)
tree = etree.HTML(html)

result = {}
for field, xpath in xpath_dict.items():
elements = tree.xpath(xpath)
result[field] = [elem.text.strip() if elem.text else ” for elem in elements]

return result

def wait_and_extract(self, url, wait_xpath, extract_dict):
“””等待特定元素出现后提取”””
self.driver.get(url)
self.wait.until(EC.presence_of_element_located((By.XPATH, wait_xpath)))

html = self.driver.page_source
tree = etree.HTML(html)

result = {}
for field, xpath in extract_dict.items():
elements = tree.xpath(xpath)
result[field] = [elem.text.strip() if elem.text else ” for elem in elements]

return result

def close(self):
self.driver.quit()

# 使用示例（需要安装ChromeDriver）
# scraper = SeleniumXPathScraper()
# data = scraper.wait_and_extract(
# ‘https://example.com’,
# ‘//div[@class=”loaded”]’,
# {
# ‘items’: ‘//div[@class=”item”]/text()’,
# ‘prices’: ‘//span[@class=”price”]/text()’
# }
# )
# scraper.close()
5.3 与数据库集成
import sqlite3
import json

class XPathDataStorage:
def __init__(self, db_path=’xpath_data.db’):
self.conn = sqlite3.connect(db_path)
self.create_tables()

def create_tables(self):
“””创建数据表”””
cursor = self.conn.cursor()
cursor.execute(”’
CREATE TABLE IF NOT EXISTS scraped_data (
id INTEGER PRIMARY KEY AUTOINCREMENT,
source_url TEXT,
field_name TEXT,
field_value TEXT,
timestamp DATETIME DEFAULT CURRENT_TIMESTAMP
)
”’)
self.conn.commit()

def store_extracted_data(self, source_url, data_dict):
“””存储提取的数据”””
cursor = self.conn.cursor()

for field, values in data_dict.items():
if isinstance(values, list):
for value in values:
cursor.execute(”’
INSERT INTO scraped_data (source_url, field_name, field_value)
VALUES (?, ?, ?)
”’, (source_url, field, str(value)))
else:
cursor.execute(”’
INSERT INTO scraped_data (source_url, field_name, field_value)
VALUES (?, ?, ?)
”’, (source_url, field, str(values)))

self.conn.commit()
print(f”已存储 {len(data_dict)} 个字段的数据”)

def query_data(self, field_name=None, source_url=None):
“””查询存储的数据”””
cursor = self.conn.cursor()

if field_name and source_url:
cursor.execute(”’
SELECT * FROM scraped_data
WHERE field_name = ? AND source_url = ?
”’, (field_name, source_url))
elif field_name:
cursor.execute(”’
SELECT * FROM scraped_data
WHERE field_name = ?
”’, (field_name,))
elif source_url:
cursor.execute(”’
SELECT * FROM scraped_data
WHERE source_url = ?
”’, (source_url,))
else:
cursor.execute(‘SELECT * FROM scraped_data’)

return cursor.fetchall()

def export_to_json(self, output_file=’exported_data.json’):
“””导出为JSON”””
cursor = self.conn.cursor()
cursor.execute(‘SELECT * FROM scraped_data’)
rows = cursor.fetchall()

data = []
for row in rows:
data.append({
‘id’: row[0],
‘source_url’: row[1],
‘field_name’: row[2],
‘field_value’: row[3],
‘timestamp’: row[4]
})

with open(output_file, ‘w’, encoding=’utf-8′) as f:
json.dump(data, f, ensure_ascii=False, indent=2)

print(f”数据已导出到 {output_file}”)

def close(self):
self.conn.close()

# 使用示例
storage = XPathDataStorage()

# 模拟提取的数据
extracted_data = {
‘title’: [‘商品A’, ‘商品B’],
‘prices’: [‘¥99’, ‘¥199’],
‘stock’: [’15’, ‘0’]
}

# 存储数据
storage.store_extracted_data(‘https://example.com/products’, extracted_data)

# 查询数据
results = storage.query_data(source_url=’https://example.com/products’)
print(“查询结果:”)
for row in results:
print(f” {row[2]}: {row[3]}”)

# 导出JSON
storage.export_to_json()

storage.close()
第六部分：高级技巧与进阶应用
6.1 XPath 2.0+ 新特性（如果环境支持）
# 注意：标准Python lxml不支持XPath 2.0
# 但可以使用第三方库如saxonche

# XPath 2.0+ 新特性示例（概念性展示）
“””
# 1. 序列处理
//item[position() le 3]

# 2. 类型转换
xs:integer(//price)

# 3. 条件表达式
if (//stock > 0) then “有货” else “缺货”

# 4. 正则表达式
matches(//email, ‘^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$’)

# 5. 分组和聚合
for $i in //item
group by $category := $i/@category
return
“””
6.2 组合查询模式
# 组合查询模式示例
html_complex = “””

“””

tree_complex = etree.HTML(html_complex)

# 组合查询1：提取所有数据点并计算总和
data_points = tree_complex.xpath(‘//div[@class=”data-point”]/@value’)
total_sales = sum(map(int, data_points))
print(f”总销售额: {total_sales}”)

# 组合查询2：提取用户统计数据并格式化
user_stats = {}
stats = tree_complex.xpath(‘//span[@class=”count”]’)
for stat in stats:
stat_type = stat.get(‘data-type’)
count = stat.text
user_stats[stat_type] = count

print(f”用户统计: {user_stats}”)

# 组合查询3：多条件筛选和计算
# 获取销售数据点中值大于150的季度
high_sales = tree_complex.xpath(”’
//div[@class=”data-point” and @value > 150]/text()
”’)
print(f”高销售季度: {high_sales}”)

# 组合查询4：跨section查询
# 获取所有section的标题和对应的数据
sections = tree_complex.xpath(‘//section’)
for section in sections:
title = section.xpath(‘.//h2/text()’)[0]
# 根据section内容提取不同类型数据
if section.get(‘class’) == ‘sales’:
values = section.xpath(‘.//div[@class=”data-point”]/@value’)
data = f”销售数据: {values}”
else:
counts = section.xpath(‘.//span[@class=”count”]/text()’)
data = f”用户统计: {counts}”
print(f”{title} – {data}”)
6.3 错误处理与健壮性
def robust_xpath_query(tree, xpath_expr, default=None):
“””
健壮的XPath查询函数
“””
try:
result = tree.xpath(xpath_expr)
if result:
return result
return default if default is not None else []
except Exception as e:
print(f”XPath查询失败: {xpath_expr}, 错误: {e}”)
return default if default is not None else []

# 使用示例
html_robust = “””

“””

tree_robust = etree.HTML(html_robust)

# 安全查询
header = robust_xpath_query(tree_robust, ‘//div[@class=”header”]/text()’, [‘默认头部’])
content = robust_xpath_query(tree_robust, ‘//div[@class=”content”]/text()’, [‘默认内容’])
missing = robust_xpath_query(tree_robust, ‘//div[@class=”missing”]’, [‘默认值’])

print(f”头部: {header}”)
print(f”内容: {content}”)
print(f”缺失: {missing}”)
总结
XPath作为一种强大的查询语言，在Web数据提取、自动化测试和内容管理中发挥着不可替代的作用。通过本文的系统学习，您应该已经掌握了：
基础语法
：节点选择、轴、谓词和通配符的使用
高级函数
：字符串、数值、日期和节点集处理函数
复杂查询
：电商、新闻、论坛、表格和嵌套结构的实战案例
性能优化
：查询优化、陷阱避免和调试技巧
实际应用
：与爬虫、Selenium和数据库的集成
关键要点：
始终优先使用具体的选择器而非通配符
合理使用轴可以避免深层遍历
谓词是精确筛选的核心工具
性能优化需要在实际场景中不断实践
健壮的错误处理是生产环境必备的
随着Web技术的不断发展，XPath也在持续演进。掌握XPath不仅能够提升数据提取的效率，更能帮助您更好地理解DOM结构和数据关系。建议在实际项目中不断练习和应用这些技巧，逐步形成自己的XPath查询模式库。
记住，优秀的XPath查询应该是：
精确、高效、可维护
。祝您在XPath的学习和应用中取得成功！