Proposal: Add new method to `HTMLParser`: `flush(self) -> str: ...`

my-dog-webdevs · September 2, 2026, 6:38am

I’d like to propose adding a method named flushto HTMLParserwhich flushes any buffered unparsed data so that all complete elements are processed and then returns the remaining unprocessed data, a str, to the caller.

Before closing an HTMLParserI would like to be able to make inferences about the remaining unprocessed input data before it is “forced” to be processed upon reaching end of file. These inferences primarily involve providing helpful messages when it appears that the input was not complete.

New changes were made that were released in CPython 3.14.7that make the current status of how much input data has actually been parsed ambiguous. HTMLParser.getpos() is still available but its not clear from a public API if and when the internal buffers will be flushed/parsed/processed. The concept of flush was not as applicable previously because the parsing was always “greedy” in that anything fed to feed was immediately parsed and processed to the full extent that the elements were complete.

Before going further I want to clarify that the parser is technically working correctly, as far as I know, in the domain of this proposal. When the parser transitions to the end of the file it is forced to internally flush buffers and decide if incomplete data should be discarded or treated as something else. There are many of these “what to do at sudden end of file” cases.

The case most pertinent to this proposal is when a start tag appears to have been partially begun but then the end of file is reached and the parser must discard that input data. Although other cases might be covered by this proposal and improve the usefulness of HTMLParser.

Problems and Solutions

This proposal solves 2 slightly different problems together:

How can a user know if HTMLParser has parsed all the “complete elements” in the data it has fed to it via feed before calling HTMLParser.close()?
How can a user know what data was not considered “complete” in the data fed to HTMLParserbefore it was forced into the context of the end of the input, ie. calling HTMLParser.close()?

The solutions in the proposal to those problems would be:

The caller will know that any “complete elements” in the data fed to the parser have been parsed and processed before proceeding.
The caller can receive whatever the remaining unprocessed data and handle it as it sees fit.

Transition To EOF Examples

In all the eof-in-tag examples I would like to show the user a friendly error based on the results of rawdata = parser.flush().

Column 1	Column 2	Column 3	Column 4
case	rawdata	expecting more input	it’s the end of the file
less than char (eof-before-tag-name)	`feed("<")`	leave in buffer, `rawdata="<"`	treat as text `"<"`, `rawdata = ""`
incomplete start tag (eof-in-tag)	`feed("<div id=1")`	leave in buffer, `rawdata="<div id=1"`	discard partial starttag, `rawdata = ""`
incomplete start tag (eof-in-tag) (dangling quote)	`feed("<div id='1></div>")`	leave in buffer, `rawdata="<div id='1></div>"`	discard partial starttag, `rawdata = ""`
incomplete start tag (eof-in-tag) (mismatched quotes)	`feed("""<div id='1"></div>""")`	leave in buffer, `rawdata="""<div id='1"></div>"""`	discard partial starttag, `rawdata = ""`

Draft Implementation

This implementation tries to remain consistent with the larger class and not over-expose the internals.

    def flush(self) -> str:
        """
        Flush buffered data.  Data will be processed insofar as it contains
        complete elements.  Any remaining unprocessed data will
        be returned to the caller.

        Call close to force this remainder to be processed.
        """
        # Flush any remaining buffered data
        if self._pending:
            self.rawdata += ''.join(self._pending)
            self._pending.clear()
            self._pending_len = 0
        # Perform incremental parsing
        self.goahead(0)
        # Return any unparsed rawdata to the caller
        return self.rawdata

Full Example

I’ve included the draft implementation as a method of a subclass of HTMLParser but it clearly uses the internals of the HTMLParserto do its work.

from html.parser import HTMLParser


class FlushableHTMLParser(HTMLParser):
    """
    Parser with flush implemented.

    This would be meant to be added to the existing HTMLParser.
    """
    def flush(self) -> str:
        """
        Flush buffered data.  Data will be processed insofar as it contains
        complete elements.  Any remaining unprocessed data will
        be returned to the caller.

        Call close to force this remainder to be processed.
        """
        # Flush any remaining buffered data
        if self._pending:
            self.rawdata += ''.join(self._pending)
            self._pending.clear()
            self._pending_len = 0
        # Perform incremental parsing
        self.goahead(0)
        # Return any unparsed rawdata to the caller
        return self.rawdata


class UserHTMLParser(FlushableHTMLParser):
    """
    Parser uses heuristics to try to determine if the forced processing
    of incomplete elements was intentional or not.

    This is *not* meant to be added to Python because it is subjective.
    """

    def close(self):
        rawdata = self.flush()
        # decide how/if this matters
        # '<' might ok, becomes entity, but '<t' is silently discarded
        if rawdata and rawdata != '<':
            raise ValueError(f'Incomplete fragment:{rawdata}')
        super().close()

def parse(html_str_parts: list[str]):
    parser = UserHTMLParser()
    for part in html_str_parts:
        parser.feed(part)
    parser.close()


if __name__ == '__main__':
    def remainder(e):
        return str(e).split(':', 1)[1] # extract rawdata from exception msg

    parts = ["<div", " x="]
    try:
        _ = parse(parts)
    except ValueError as e:
        assert 'Incomplete fragment' in str(e) # eof-in-tag: likely mistake, incomplete tag
        print (f'ERROR: "{''.join(parts)}" {remainder(e)=}')
    else:
        raise AssertionError('This should have failed.')

    parts = ["<div", ''' x='1">''', "</div>"]
    try:
        _ = parse(parts)
    except ValueError as e:
        assert 'Incomplete fragment' in str(e) # eof-in-tag: likely mistake, mismatched quotes
        print (f'ERROR: "{''.join(parts)}" {remainder(e)=}')
    else:
        raise AssertionError('This should have failed.')

    parts = ["<div", ''' x='1>''', "</div>"]
    try:
        _ = parse(parts)
    except ValueError as e:
        assert 'Incomplete fragment' in str(e) # eof-in-tag: likely mistake, lopsided quote
        print (f'ERROR: "{''.join(parts)}" {remainder(e)=}')
    else:
        raise AssertionError('This should have failed.')

    parts = ["<div", " x=1>", "</div>"]
    try:
        _ = parse(parts)
    except ValueError as e:
        raise AssertionError('Buffering occurs but this should not raise.')
    else:
        print (f'OK: "{"".join(parts)}"')

    parts = ["<div", " x=1>", "</div>", "<"]
    try:
        _ = parse(parts)
    except ValueError as e:
        raise AssertionError('Buffering occurs but trailing "<" might be intended as text.') # This is app-specific.
    else:
        print (f'OK: "{"".join(parts)}"')

The output when the script is run in CPython 3.14.7 :

ERROR: "<div x=" remainder(e)='<div x='
ERROR: "<div x='1"></div>" remainder(e)='<div x=\'1"></div>'
ERROR: "<div x='1></div>" remainder(e)="<div x='1></div>"
OK: "<div x=1></div>"
OK: "<div x=1></div><"

Workaround

A work around proposed to me by another developer was to feed the entire block of text into the parser at once via feed() which based on the internal implementation would not trigger the internal pending buffer. Then getpos() could be used to deduce what was left from that initial feed(). Then after deciding if there was an error or not close()could be called as usual.

This was a good idea and appears to work but it is yet another implicit dependency on when buffering might occur. If the algorithm changed again in the future this could also break if the pending buffer was applied preemptively during the first feed at some point.