Logging to XML file

Python Help
1

Hello :wave:!

I’d like to write data coming from a socket (after some processing) to an file in XML format. This should be rotated each day and x days should be kept in a given directory.

The structure of that XML is pretty straight forward

<CommunicationLog xmlns="http://knx.org/xml/telegrams/01">
  <Telegram Timestamp="2021-01-25T07:47:41.3745655Z" Service="L_Data.ind" FrameFormat="CommonEmi" RawData="2900BCD011352332010081" />
  <Telegram Timestamp="2021-01-25T07:47:44.9591475Z" Service="L_Data.ind" FrameFormat="CommonEmi" RawData="2900BCD01122341E0300800076" />
  <Telegram Timestamp="2021-01-25T07:47:47.0888991Z" Service="L_Data.ind" FrameFormat="CommonEmi" RawData="2900BCD01151295103008007B7" />
  <!-- new log entries shall be inserted here -->
</CommunicationLog>

Can someone recommend a good way to achieve this?

I was thinking to use logging.handlers.TimedRotatingFileHandler for that, but I don’t see how I could insert new entries before the closing tag </CommunicationLog> for each emit().

2

Which part of this are you stuck on. Is it simply the appending new data part, or the entire project?

3

I am just looking at the logging module and was thinking
"this has everything I need

  • creates new files on midnight
  • cleanup of old files
  • collect some data and write in bunch

why reinvent the wheel?"

But I’m stuck at how to insert data to the penultimate line of a file (instead of appending).

If it is not possible, I’ll have to come up with my custom RotatingFileHandler:grimacing:

4

The starting point is, at the least, an empty log file:

<CommunicationLog xmlns="http://knx.org/xml/telegrams/01">
</CommunicationLog>

Now each time this script is run, the data held in new_data will be appended to the log file. As this is simply a POC, you’ll need to take care of the new_data part, so that you have new data before the script is run.

# simulate incoming data
new_data = '<Telegram Timestamp="2021-01-25T07:47:47.0888991Z" Service="L_Data.ind" FrameFormat="CommonEmi" RawData="2900BCD01151295103008007B7" />'
# -------------------------

xml_close = "</CommunicationLog>\n"
line_read = ""
log_data = []
xml_file = "log.xml"
# create a generator for the file read
log_file = (line for line in open(xml_file, mode='r', encoding='UTF-8'))

xml_open = next(log_file).strip()

while line_read != xml_close:
    line_read = next(log_file)
    if line_read and line_read != xml_close:
        log_data.append(line_read.strip())

log_data.append(new_data)

with open(xml_file, mode='w', encoding='UTF-8') as f:
    print(xml_open, file=f)
    for item in log_data:
        print(f"  {item}", file=f)
    print(xml_close.strip(), file=f)
1 Like
5

TimedRotatingFileHandler looks like the right basis (with your own format).

I was thinking of this the other way around: that you could provide a rotator that puts the tail on the file and tops the next one with your outer tags. However, does the file being written have to be a valid instance, with a closing tag, at all times?

If so, we seem to run into the problem that TimedRotatingFileHandler inherits from FileHandler its idea of what it means to emit() a record. However, FileHandler is a StreamHandler, and has a setStream() method, so I wonder if you can use that to give it a behaviour that positions before the closing tag and then writes new data?

6

This is to post-process the file. It means a valid instance, with start and end CommunicationLog tags is only obtained for a finished log. (One surely wouldn’t do this with each emit()?)

However, this could be the basis of a rotator callable.

1 Like
7

Yes, I’ve made some assumptions (such as the close tag), but given a valid log file, there’s no reason that the close tag could not be read from said file, rather than being hard-coded: I simply focused on appending new data before the close tag.

Thinking about this some more, the close tag could be (and maybe should be) constructed from the open tag.

Maybe this will be of help to @farmio , maybe not.

1 Like
8

Thank you both for your inputs!

Sleeping over this issue, I now think it is a better solution for my problem to not use the logging module, but implement my own “file handler class” (even if that means more work). This will make it more flexible in naming and storing these log files (eg. subfolders for year, month, filename representing the day) and I really do not need additional logging features like log levels.

I’d also like the files to be useable (have the end tag written) at any time - so a copy can even be used between emits (which I plan to do ~every 10 minutes), not only after a rollover.

In my case the opening tag, as well as the closing tag are constants. But I see the point.

I guess I’ll try 2 methods and see what works best / uses less resources :thinking:

  • “Hacky version”: open the file; delete last line; append new entries; add closing tag; close file (similar to rob42’s answer)
  • “Proper version”: use xml.etree.ElementTree.write()