I’m writing a small application that will ideally query an SQL Server database and write the results to file, then read an additional set of (6+) files, comparing those results against the initial output of the query, correct any notable errors (some of the files have line-length formatted fields), then transfer the entire set to an upload directory. It may be necessary to make additional SQL queries to the database during error correction.
I’ve got a test sample working with pyodbc and have been playing with pandaSQL; roughly speaking, I intend to create a set of dataframes to work with the data and update the existing files.
Outside of the tools I’ve already got can anyone suggest anything else that I might look into to make life easier - in particular, which design pattern makes the most sense here?