镜像站点 · 本页由第三方 GitHub 只读镜像提供,非 GitHub 官方站点,不接受任何登录或凭据输入。前往 github.com
Skip to content

Use ASCII case folding for :lang() and attribute values - #201

Merged
AdrianAtZyte merged 1 commit into
scrapy:masterfrom
itecz26:ascii-fold-lang-attr-values
Oct 14, 2026
Merged

AdrianAtZyte merged 1 commit into
scrapy:masterfrom
itecz26:ascii-fold-lang-attr-values

Conversation

@itecz26

@itecz26 itecz26 commented Oct 14, 2026

Copy link
Copy Markdown
Contributor

#200 moved element and attribute names to ASCII case folding, but two translation sites still fold the selector side of a comparison with full Unicode str.lower() while the document side only folds A-Z: the :lang() argument in HTMLTranslator.xpath_lang_function(), and attribute values under the lower_case_attribute_values extension point. str.lower() maps exactly two non-ASCII code points into ASCII (U+212A KELVIN SIGN to "k", U+0130 to "i" plus U+0307), so a :lang() argument containing U+212A compiles to starts-with(..., 'k-') and matches lang="K" or lang="k", while an element whose lang is actually U+212A is missed:

>>> from lxml.html import fromstring
>>> from cssselect import HTMLTranslator
>>> doc = fromstring('<p lang="K">x</p>')
>>> doc.xpath(HTMLTranslator().css_to_xpath(":lang(\u212a)"))
[<Element p at 0x...>]

Language-range matching is ASCII case-insensitive (RFC 4647 section 3.3.1, and the HTML specification's definition of :lang), so the argument must be folded the same way as the translate() already emitted for the @lang side. Same reasoning for attribute values: that knob's contract is that the document parser lower-cased the values, an ASCII transformation.

The document side of the :lang() comparison only folds A-Z through
translate(), but the argument was folded with str.lower(), which maps
U+212A (KELVIN SIGN) to "k" and U+0130 to "i" + U+0307. A :lang()
argument containing U+212A therefore matched lang="K" while missing
an actual U+212A language tag. Language-range matching is ASCII
case-insensitive (RFC 4647), so fold the argument with ascii_lower(),
and do the same for attribute values under the
lower_case_attribute_values extension point, completing the name
folding from scrapy#200.
@codecov

codecov Bot commented Oct 14, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 99.79%. Comparing base (99ec093) to head (51ca8bf).

Additional details and impacted files
@@           Coverage Diff           @@
##           master     #201   +/-   ##
=======================================
  Coverage   99.79%   99.79%           
=======================================
  Files           3        3           
  Lines         987      987           
  Branches      161      161           
=======================================
  Hits          985      985           
  Misses          1        1           
  Partials        1        1           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@AdrianAtZyte
AdrianAtZyte merged commit 6a859d5 into scrapy:master Oct 14, 2026
23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants