text separated by <br> is joined by get_text() without spaces in between
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Beautiful Soup |
New
|
Undecided
|
Unassigned |
Bug Description
There are cases when you find something like this on html pages:
URL 1: www.example-
You can find a real example on this link (I am also attaching the html) (look for "Blog" in the html): https:/
(when I see that page with the inspector tool, the html is prettified, so the <br> is surrounded by new lines, but when downloading the html, it is a case like the above).
The current implementation of text acquisition (get_text()) ignores <br> tags, resulting in a string like the following (for the example above): URL 1: www.example-
Here is a code to reproduce it:
```
from bs4 import BeautifulSoup
bs = BeautifulSoup("URL 1: www.example-
print(bs.text)
# URL 1: www.example-
```
I did a merge proposal trying to fix this: https:/
Additional information:
- Python version: 3.12.2
- html5lib installed: no
- lxml installed: no
- parser used: 'html.parser'
Thank you for your review
Thanks for taking the time to file this issue and write a patch.
The problem you've found is that certain markup tags should have whitespace appended when converting them to a string. In HTML, this includes the <br> tag and possibly others. In XML, it depends on the XML vocabulary in use--there are probably no tags with this property. So we can't solve this problem by changing the Tag class: a specific <br> tag might or might not be equivalent to a newline.
This is similar to other situations where certain HTML tags or attributes have special semantics that don't apply to XML. For instance, the <br> tag is an empty-element tag in HTML, but in XML it's generally a regular tag that can contain other tags.
We deal with this by customizing the TreeBuilder object, either through a constructor argument (as with multi_valued_ attributes and preserve_ whitespace_ tags) or as something that can be overridden in a subclass (as with empty_element_ tags).
So, there are two open questions:
1. Is this customizable often enough that it should be made a constructor argument? Or is the difference mainly between HTML semantics and a default set of XML semantics? Probably the latter, so I think treating it like empty_element_tags is fine.
2. What are the relevant semantics as defined in the HTML spec, and what other HTML tags have semantics of the same sort?
The section on the <br> element itself says "The br element represents a line break." /html.spec. whatwg. org/#the- br-element
https:/
There's a step specific to <br> in the algorithm for generating innerText and outerText (the closest thing in the HTML spec to get_text()) /html.spec. whatwg. org/#the- innertext- idl-attribute
https:/
"If node is a br element, then append a string containing a single U+000A LF code point to items."
(U+000A is the newline character)
Are there any other elements like this? Superficially, I don't think so; otherwise the HTML spec would define a type of element that should be treated this way, rather than giving <br> special treatment in the innerText algorithm. The <wbr> tag "represents a line break opportunity" but you don't have to take that opportunity.
You could argue that the <hr> element should also be turned into a newline, but I think that is covered by these other steps in the innerText algorithm:
8. If node is a p element, then append 2 (a required line break count) at the beginning and end of items.
9. If node's used value of 'display' is block-level or 'table-caption', then append 1 (a required line break count) at the beginning and end of items.
That's saying that a <p> tag should be preceded and followed by a newline, and any other block-level tag should be followed by a newline. (As usual, Beautiful Soup is ignoring the effects of CSS on these rules.) <hr> is a block-level tag, for example. So the real special case is that <br> is followed by a newline even though it's *not* a block-level tag.
Right now get_text() doesn't go through those steps:
>>> from bs4 import BeautifulSoup "<p>foo< /p><hr> <p>bar< /p>").get_ text()
>>> BeautifulSoup(
'foobar'
Beautiful Soup knows which HTML tags are defined as block-level tags (or at least which ones were defined as block-leve...