Streaming data from a file while only reading 1 line once

im streaming data from a file and i want to exculde the data printed if its the same as the previous output how would i go around doing that

from time import sleep

logLocation = "console.log"
buffer = ""


while True:
    sleep(0.25)

    with open(logLocation, "r") as log:
        currentLine = log.readlines()[-1]
        
        if buffer == currentLine:
            print(True)

        print("False")
        buffer == currentLine

im streaming data from a file and i want to exculde the data printed if
its the same as the previous output how would i go around doing that

Well you seem to have the relevant if-statement in your code, just
print(currentLine) in the (missing) else: branch of the
if-statement.

But note this line:

 buffer == currentLine

I expect you meant to stored currentLine in buffer for the next
comparison. That is written:

 buffer = currentLine

What you have written is just a comparison expression, comparing
buffer against currentLine, and just not assigned it to anything.
That is legal, but not useful and not what you probably intended.

Returning to the code, it looks as if you’re trying to read lines from
the end of the file as they appear, based on this:

 with open(logLocation, "r") as log:
     currentLine = log.readlines()[-1]

This is a very inefficient way to read the last line. It reads the
entire file (log.readlines()), then takes the last line present. For
large files this will be Very Slow.

Also, your process seems to be polling-based: you sleep for 0.25s then
grab the current last line. Maybe that’s the origin of your “exculde the
data printed if its the same as the previous output” requirement,
because if the log file is not growing you’ll get the same previous
line.

This has 2 problems:

  • if the log genuinely gets another identical line, should it be printed
    or not? that is a policy decision you need to decide because you know
    the larger context
  • if the log gets more than one line in the 0.25s interval, only the
    last line will be considered; the other lines will be ignored

I would be inclined to use the file pointer to notice if the log has
grown. You get that from its .tell() method i.e. log.tell(). You
could keep that around. To get new lines, seek
(log.seek(previous_pointer_value) to the old position and try to read
lines.

On a UNIX system you can keep the file open between polls, so you might
not need to use seek() at all. On Windows this is not normally the
case because the default behaviour is often to only let one thing access
the file at a time.

Cheers,
Cameron Simpson cs@cskk.id.au