# Proposal: Add new method to \`HTMLParser\`: \`flush(self) -\> str: ...\`

**URL:** <https://discuss.python.org/t/proposal-add-new-method-to-htmlparser-flush-self-str/108855>\
**Category:** Ideas\
**Created:** [September 2, 2026, 6:38am UTC](https://discuss.python.org/t/proposal-add-new-method-to-htmlparser-flush-self-str/108855 "2026-09-02T06:38:05Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![my-dog-webdevs](https://avatars.discourse-cdn.com/v4/letter/m/d07c76/32.png) [@my-dog-webdevs](https://discuss.python.org/u/my-dog-webdevs)\
**Post date:** [September 2, 2026, 6:38am UTC](https://discuss.python.org/t/proposal-add-new-method-to-htmlparser-flush-self-str/108855/1 "2026-09-02T06:38:05Z")

</div>

I’d like to propose adding a method named `flush`to `HTMLParser`which flushes any buffered unparsed data so that all complete elements are processed and then returns the remaining unprocessed data, a `str`, to the caller.

Before closing an `HTMLParser`I would like to be able to make inferences about the remaining unprocessed input data before it is “forced” to be processed upon reaching end of file. These inferences primarily involve providing helpful messages when it appears that the input was not complete.

[New changes](https://github.com/python/cpython/issues/153030) were made that were released in `CPython 3.14.7`that make the current status of how much input data has actually been parsed ambiguous. `HTMLParser.getpos()` is still available but its not clear from a public API if and when the internal buffers will be flushed/parsed/processed. The concept of `flush` was not as applicable previously because the parsing was always “greedy” in that anything fed to `feed` was immediately parsed and processed to the full extent that the elements were complete.

Before going further I want to clarify that the parser is technically working correctly, as far as I know, in the domain of this proposal. When the parser transitions to the end of the file it is forced to internally flush buffers and decide if incomplete data should be discarded or treated as something else. There are many of these “what to do at sudden end of file” cases.

The case most pertinent to this proposal is when a start tag appears to have been partially begun but then the end of file is reached and the parser must discard that input data. Although other cases might be covered by this proposal and improve the usefulness of `HTMLParser`.

### Problems and Solutions

This proposal solves 2 slightly different problems together:

1. How can a user know if `HTMLParser` has parsed all the “complete elements” in the data it has fed to it via `feed` before calling `HTMLParser.close()`?
2. How can a user know what data was not considered “complete” in the data fed to `HTMLParser`before it was forced into the context of the end of the input, ie. calling `HTMLParser.close()`?

The solutions in the proposal to those problems would be:

1. The caller will know that any “complete elements” in the data fed to the parser have been parsed and processed before proceeding.
2. The caller can receive whatever the remaining unprocessed data and handle it as it sees fit.

### Transition To EOF Examples

In all the `eof-in-tag` examples I would like to show the user a friendly error based on the results of `rawdata = parser.flush()`.

| Column 1 | Column 2 | Column 3 | Column 4 |
| --- | --- | --- | --- |
| case | rawdata | expecting more input | it’s the end of the file |
| less than char (eof-before-tag-name) | `feed("<")` | leave in buffer, `rawdata="<"` | treat as text `"&lt;"`, `rawdata = ""` |
| incomplete start tag (eof-in-tag) | `feed("<div id=1")` | leave in buffer, `rawdata="<div id=1"` | discard partial starttag, `rawdata = ""` |
| incomplete start tag (eof-in-tag)   
(dangling quote) | `feed("<div id='1></div>")` | leave in buffer, `rawdata="<div id='1></div>"` | discard partial starttag, `rawdata = ""` |
| incomplete start tag (eof-in-tag)  
(mismatched quotes) | `feed("""<div id='1"></div>""")` | leave in buffer, `rawdata="""<div id='1"></div>"""` | discard partial starttag, `rawdata = ""` |
| | | | |

### Draft Implementation

This implementation tries to remain consistent with the larger class and not over-expose the internals.

```python
    def flush(self) -> str:
        """
        Flush buffered data. Data will be processed insofar as it contains
        complete elements. Any remaining unprocessed data will
        be returned to the caller.

        Call close to force this remainder to be processed.
        """
        # Flush any remaining buffered data
        if self._pending:
            self.rawdata += ''.join(self._pending)
            self._pending.clear()
            self._pending_len = 0
        # Perform incremental parsing
        self.goahead(0)
        # Return any unparsed rawdata to the caller
        return self.rawdata

```

### Full Example

I’ve included the draft implementation as a method of a subclass of `HTMLParser` but it clearly uses the internals of the `HTMLParser`to do its work.

```python
from html.parser import HTMLParser

class FlushableHTMLParser(HTMLParser):
    """
    Parser with flush implemented.

    This would be meant to be added to the existing HTMLParser.
    """
    def flush(self) -> str:
        """
        Flush buffered data. Data will be processed insofar as it contains
        complete elements. Any remaining unprocessed data will
        be returned to the caller.

        Call close to force this remainder to be processed.
        """
        # Flush any remaining buffered data
        if self._pending:
            self.rawdata += ''.join(self._pending)
            self._pending.clear()
            self._pending_len = 0
        # Perform incremental parsing
        self.goahead(0)
        # Return any unparsed rawdata to the caller
        return self.rawdata

class UserHTMLParser(FlushableHTMLParser):
    """
    Parser uses heuristics to try to determine if the forced processing
    of incomplete elements was intentional or not.

    This is *not* meant to be added to Python because it is subjective.
    """

    def close(self):
        rawdata = self.flush()
        # decide how/if this matters
        # '<' might ok, becomes entity, but '<t' is silently discarded
        if rawdata and rawdata != '<':
            raise ValueError(f'Incomplete fragment:{rawdata}')
        super().close()

def parse(html_str_parts: list[str]):
    parser = UserHTMLParser()
    for part in html_str_parts:
        parser.feed(part)
    parser.close()

if __name__ == ' __main__':
    def remainder(e):
        return str(e).split(':', 1)[1] # extract rawdata from exception msg

    parts = ["<div", " x="]
    try:
        _ = parse(parts)
    except ValueError as e:
        assert 'Incomplete fragment' in str(e) # eof-in-tag: likely mistake, incomplete tag
        print (f'ERROR: "{''.join(parts)}" {remainder(e)=}')
    else:
        raise AssertionError('This should have failed.')

    parts = ["<div", ''' x='1">''', "</div>"]
    try:
        _ = parse(parts)
    except ValueError as e:
        assert 'Incomplete fragment' in str(e) # eof-in-tag: likely mistake, mismatched quotes
        print (f'ERROR: "{''.join(parts)}" {remainder(e)=}')
    else:
        raise AssertionError('This should have failed.')

    parts = ["<div", ''' x='1>''', "</div>"]
    try:
        _ = parse(parts)
    except ValueError as e:
        assert 'Incomplete fragment' in str(e) # eof-in-tag: likely mistake, lopsided quote
        print (f'ERROR: "{''.join(parts)}" {remainder(e)=}')
    else:
        raise AssertionError('This should have failed.')

    parts = ["<div", " x=1>", "</div>"]
    try:
        _ = parse(parts)
    except ValueError as e:
        raise AssertionError('Buffering occurs but this should not raise.')
    else:
        print (f'OK: "{"".join(parts)}"')

    parts = ["<div", " x=1>", "</div>", "<"]
    try:
        _ = parse(parts)
    except ValueError as e:
        raise AssertionError('Buffering occurs but trailing "<" might be intended as text.') # This is app-specific.
    else:
        print (f'OK: "{"".join(parts)}"')

```

The output when the script is run in `CPython 3.14.7` :

```plaintext
ERROR: "<div x=" remainder(e)='<div x='
ERROR: "<div x='1"></div>" remainder(e)='<div x=\'1"></div>'
ERROR: "<div x='1></div>" remainder(e)="<div x='1></div>"
OK: "<div x=1></div>"
OK: "<div x=1></div><"

```

### Workaround

A work around proposed to me by another developer was to feed the entire block of text into the parser at once via `feed()` which based on the internal implementation would not trigger the internal pending buffer. Then `getpos()` could be used to deduce what was left from that initial `feed()`. Then after deciding if there was an error or not `close()`could be called as usual.

This was a good idea and appears to work but it is yet another implicit dependency on when buffering might occur. If the algorithm changed again in the future this could also break if the pending buffer was applied preemptively during the first feed at some point.

---

<div class="post-metadata">

**Author:** ![my-dog-webdevs](https://avatars.discourse-cdn.com/v4/letter/m/d07c76/32.png) [@my-dog-webdevs](https://discuss.python.org/u/my-dog-webdevs)\
**Post date:** [September 4, 2026, 5:41pm UTC](https://discuss.python.org/t/proposal-add-new-method-to-htmlparser-flush-self-str/108855/2 "2026-09-04T17:41:27Z")

</div>

This proposal is a little specific. Maybe it would be better to just make a GitHub issue for it? Let me know if this is the wrong platform for this. Thanks.

---

<div class="post-metadata">

**Author:** ![storchaka](https://sea2.discourse-cdn.com/flex002/user_avatar/discuss.python.org/storchaka/32/217_2.png) [@storchaka](https://discuss.python.org/u/storchaka)\
**Post date:** [September 5, 2026, 8:18am UTC](https://discuss.python.org/t/proposal-add-new-method-to-htmlparser-flush-self-str/108855/3 "2026-09-05T08:18:22Z")

</div>

You are right that there is a gap, and it is worse than “ambiguous”: the deferral is observable in the callbacks, not only in `rawdata`.

```python
p.feed("<div") # nothing parsed
p.feed(">") # buffered -- handle_starttag() is *not* called
p.close() # only now: starttag div

```

There is currently no way to say “process everything you can right now” except `close()`, which also applies the EOF rules. `flush()` is a reasonable fix for that.

But it should not return `rawdata`. `rawdata` is not the incomplete tail: `feed("a&copy")` leaves `rawdata == "a&copy"`, yet that input is already complete – `&copy` is a valid character reference even without the semicolon, and `close()` emits `a©`. The parser holds it back only because it could still be the start of a longer reference: a following `";"` gives `a©`, a following `"sr;"` gives `a℗`. So your example would report `Incomplete fragment: a&copy` for input that is not incomplete at all. Documenting a return value would freeze such details as public API.

So: `flush()` returning `None`, documented as “process all data fed so far that forms complete constructs; incomplete trailing data stays buffered”. You can still read `rawdata` yourself afterwards.

Please open an issue referencing issue [#153030](https://github.com/python/cpython/issues/153030).

---

<div class="post-metadata">

**Author:** ![my-dog-webdevs](https://avatars.discourse-cdn.com/v4/letter/m/d07c76/32.png) [@my-dog-webdevs](https://discuss.python.org/u/my-dog-webdevs)\
**Post date:** [September 5, 2026, 6:25pm UTC](https://discuss.python.org/t/proposal-add-new-method-to-htmlparser-flush-self-str/108855/4 "2026-09-05T18:25:01Z")

</div>

Thank you for your feedback. I created an issue:

> <https://github.com/python/cpython/issues/157003>
>
> \# Feature or enhancement
> 
> \### Proposal:
> 
> With recent changes it isn't clear when… the the parser actually does the parsing/processing anymore on the data given to \`feed()\`.  
> 
> Changes were made in #153030 , (seen in \`3.14.7\`), which leave this gap in the \`HTMLParser\`.  
> 
> Propose adding a flush method as seen below ( I reset the parse threshold to \`1\` which differs from my proposal on discuss, seems appropriate?):
> 
> \`\`\`python
> class PrototypeHTMLParser(HTMLParser):
> 
> def flush(self):
> """
> Process all data fed insofar as it contains complete elements any remaining data remains buffered.
> """
> if self.\_pending:
> self.rawdata += ''.join(self.\_pending)
> self.\_pending.clear()
> self.\_pending\_len = 0
> self.\_parse\_threshold = 1
> # Perform incremental parsing but its not EOF.
> self.goahead(0)
> \`\`\`
> 
> \### Example
> 
> \`\`\`python
> from html.parser import HTMLParser
> 
> 
> class FlushableHTMLParser(HTMLParser):
> """
> Parser with flush implemented.
> 
> This would be meant to be added to the existing HTMLParser.
> """
> def flush(self):
> if self.\_pending:
> self.rawdata += ''.join(self.\_pending)
> self.\_pending.clear()
> self.\_pending\_len = 0
> self.\_parse\_threshold = 1
> self.goahead(0)
> 
> 
> class DemoHTMLParser(FlushableHTMLParser):
> 
> def handle\_starttag(self, tag, attrs):
> print (f'Found starttag {tag=} {len(attrs)=}')
> 
> def handle\_endtag(self, tag):
> print (f'Found endtag {tag=}')
> 
> 
> def parse(html\_str\_parts: list\[str\]):
> parser = DemoHTMLParser()
> for part in html\_str\_parts:
> print (f'Feed {len(part)}')
> parser.feed(part)
> print ('')
> print ('Calling flush()')
> parser.flush()
> print ('Calling close()')
> parser.close()
> 
> 
> def demo\_attrs(attrs\_len):
> print (f'\\nDemo with {attrs\_len=}')
> print ('='\*20)
> attrs = \[f" a{i}='1'" for i in range(attrs\_len)\]
> parse(\["\<div", \*attrs, "\>", "content", "\</div\>"\])
> 
> 
> if \_\_name\_\_ == '\_\_main\_\_':
> for attrs\_len in (2, 8):
> demo\_attrs(attrs\_len)
> \`\`\`
> 
> \`\`\`text
> 
> Demo with attrs\_len=2
> ====================
> Feed 4
> Feed 7
> Feed 7
> Feed 1
> Feed 7
> Found starttag tag='div' len(attrs)=2
> Feed 6
> Found endtag tag='div'
> 
> Calling flush()
> Calling close()
> 
> Demo with attrs\_len=8
> ====================
> Feed 4
> Feed 7
> Feed 7
> Feed 7
> Feed 7
> Feed 7
> Feed 7
> Feed 7
> Feed 7
> Feed 1
> Feed 7
> Feed 6
> 
> Calling flush()
> Found starttag tag='div' len(attrs)=8
> Found endtag tag='div'
> Calling close()
> \`\`\`
> \#### Other Doc Changes
> 
> I think the \`feed()\` doc string also needs a cleanup to match these changes. Should that also be part of this issue?
> 
> \#### Handling EOF
> 
> There is a related issue about how can a stdlib user determine what should be done when \`close()\` is called and data is discarded. Specifically from the living spec, \[\`eof-in-tag\` error case\](https://html.spec.whatwg.org/multipage/parsing.html#parse-error-eof-in-tag). I'm going to link it but include the text here as well:
> 
> \> This error occurs if the parser encounters the end of the input stream in a start tag or an end tag (e.g., \<div id=). Such a tag is ignored.
> 
> 
> Should I start that in discuss in the same thread as before or make another issue about that? Maybe it is beyond the scope of the python stdlib parser? It seems like a hard problem but it kind of makes the parser into partial solution in certain use cases.
> 
> \### Has this already been discussed elsewhere?
> 
> I have already discussed this feature proposal on Discourse
> 
> \### Links to previous discussion of this feature:
> 
> https://discuss.python.org/t/proposal-add-new-method-to-htmlparser-flush-self-str/108855

I mostly agree with your conclusion about the return value being `None` but there is a “missing” piece when using the parser for processing application templates regarding providing feedback about “what happened” when `EOF` occurs. I would like to explore if that is in the parser’s scope and if anything could be done about that situation if it is. Would that be in the same issue or in another issue ? I also asked that question in the issue itself. Sorry a little redundant but that’s how I got here; Hoping for a longterm solution. Thanks for your time.