I want to write my own application for something like a home accounting system.
Overview
Daikokuten - the application - is supposed to be a graphical application to store and manage household expenses. Every “document” should be scanned and as a picture imported to Daikokuten. Such a document inside the application shall have some attributes e.g. date, place, category of expense, expense and so forth.
My objectives
It should be entirely autonomous from the system (no dependencies needed)
All files should be stored on an external hard drive formatted with zfs
It should be fast, responsive and not getting in my way while using it
My idea
I think that the first objective will be quite easy to accomplish with nix packages. The only thing I’m worrying about is that I don’t know how to import a python module to interact with zfs drive sot that Daikokuten won’t rely on installed system wide driver. Any idea how can I do it? Or is there a better way of doing it?
Well that’s really easy. I think…
Here I have some doubts. For example what kind of a database should I use? SQLite? Or something more robust like mariadb running in a container?
Thank you for that insightful response. Although I should guess that did you read my post you’re probably aware that I won’t get my answers in documentation.
My problem is: “which way should I go”, not “what are the ways”. I’m quite aware of ways I can accomplish my idea. What I need is a discussion concerning downsides of each approach. I won’t get it reading documentation.
So, let me rephrase my questions:
What are the downsides of using SQLite in my application (I’m namely worried about speed. I plan to store thousands documents, every document shall have to be referenced in db)
Should I burry my dream of running Daikokuten autonomously from system and install system wide driver for zfs or using some kind of a module will be quite easy?
Be assured that I have done my research ere I wrote here and I am still researching.
If you let an AI write your posts for you, a lot of us aren’t going to bother reading them. Write your own words, even if your spelling and grammar aren’t perfect.
There are some oddities here which lead me to agree with the guess that these posts are being run through an LLM. Using one is not against forum rules, but if you do, please inform your readers.
Regarding your questions… I think you need to build a prototype and start exploring and learning. The questions themselves suggest that you’re focusing on the wrong things right now.
Python itself is a dependency.
Use sqlite until it no longer works for your use case. Sqlite is often vastly underestimated. By the time you outgrow it, you’ll have learned enough to judge the various alternatives for yourself.
“Thousands” is so far below the threshold for considering anything more complex, I would not worry.
This is also a good place to learn and think about application architecture. Your storage system can be made pluggable, so that you can swap it out later.
Why is the choice of filesystem (ZFS) directly relevant to your application? Start with reading and writing files, and not worry about whether the filesystem is zfs, fat, ext, etc.
Because I’m concerned that you’re relying on an LLM, I’ll call out that I think you may be doing too much research and not enough learning.
Rather than spending a lot of time reading about sqlite, make yourself a sqlite db. Put some data in it. Run some queries.
Don’t worry about filesystem technologies. Get acquainted with the basics of files and IO first.
I’m actually familiar with SQL and managing files well enough.
I decided to go with ZFS because I want redundance. So my question is: is there a python module which will allow me interact with zfs without the need of installed system wide driver?
Strange then that you described an “insightful response” when the only response you got was some very basic links. That response may have been exactly what you needed, and it may have been extremely helpful, but I wouldn’t have called it “insightful”.
Maybe you’re not using AI to write your posts, but it certainly doesn’t seem like a normal way to respond as a human. And if you’re getting an AI to “polish” your posts, that’s still using AI to write for you, even if you feel like it’s all your content - because it isn’t your content any more, and we have to contend with the very real possibility that errors were added during the polishing.
Anyhow. You’re asking about a bunch of individual technologies, many of which aren’t really Python-specific, but if you actually want something that’s part of Python, check PyPI. A simple search there will answer your questions about whether there is a Python module for X. [1]
And yes, there is, in fact, a Python module for X, which I quite like. But it’s not what you’re after.) ↩︎
The reason I responded the way I did was because the way you worded your post sounded a lot like a homework assignment. Lots of people can address your issues better than I, but it appeared you hadn’t tried anything yet.
I’m sorry that I sound weird. I try to use best English I can. I don’t converse in that language with anybody and my knowledge is limited in that regard.
SQLite should be fine. As you get into 1000s of records you will need SQLite. I use JSON to store data for one of my programs, I think I’m up to 4000 records. But since it runs in the background, I don’t ever see it run, so I don’t know how long it takes to run, and read and write the JSON to disk.
Don’t worry OP. I used to talk like you are writing. I had a very large vocabulary which I found precise and interesting. I had good English teachers in school that really pushed us, a lot. But in the real world I got complaints that most people didn’t understand me because I used words that aren’t used much. So I changed and adapted.
I had a fascination with precise words but using them alienated my coworkers so it was me who had to change.
Your English is very formal so don’t blame others if they think you use AI to write it as most English speakers don’t know how to write like you do. So it’s statistically likely you did use AI, which we now know is not true.
English conversation and writing is much more casual today.
Please DM me if you have more questions about English.
I will agree that SQLite should work just fine for your application. SQLite works very well when all the data is stored in one place and is read and written by one process. There shouldn’t be many simultaneous accesses to your database, and your data is definitely small enough that it doesn’t have to be spread across multiple servers. If you do find your SQLite database file becoming larger than you want, an easy thing to do is to move large items like photos or documents out of the database and into regular files, and have the database store the path to the file.
About ZFS, I would recommend thinking of the file system separately from your application. This makes your application less complicated. Your application can be concerned with just writing files as normal, and ZFS can worry about replicating things. I am not an expert on ZFS it as far as I know it is usually used as a “kernel-mode” file system, so you do have to install and configure it as part of the operating system where you run your app. I don’t think it can be used as a “portable” dependency. Someone else who knows more on that topic may correct me.
The thing which has seemed most unusual to me in this thread is focusing on ZFS as a particular technology. All filesystems are abstractions, and they all provide the same interfaces[1] to let applications be written which are loosely coupled and don’t depend upon which filesystem is in use.
You can write an application which depends specifically on ZFS. But for the use case, that sounds both more difficult and much worse than one which uses files and works on ext4 and other less exotic filesystems.
Regarding the suspicion of LLM usage, I don’t attribute it to strange or archaic word choices. Native speakers rarely use “ere”, but neither do LLMs.
The things that made me suspect an LLM were fourfold:
An extremely broad topic question (“how to build an app”), suggestive of a beginner
Very specific technologies being considered (ZFS, sqlite, mariadb), which seems at odds with (1)
Requirements/goals that are unclear or nonspecific (“autonomous from the system”?[2] “doesn’t get in the way”?)
Calling a reply with a doc link an “insightful response”
Regarding that last one, it’s quite normal to thank people for offering help. Even when the type of help they offer isn’t what you’re looking for. But an enthusiastic thanks – to my eye, somewhat too enthusiastic – which calls a single link response “insightful” rings of LLM behavior. LLMs have, among other faults, a habit of excessively praising their interlocutors. In the future, I recommend only thanking people for what they’ve actually given you. If it was a link, rather than an insight, thank them for a useful link.
I hope this explanation clarifies why I gave a response primarily tailored towards a beginner relying on an LLM. I’m still not sure of your programming background and expertise, but I feel confident that you haven’t used sqlite very much based on that part of the topic. As above, I recommend playing with it to see how capable it is – it’s a fantastic bit of technology.
filesystem experts will take issue with this, I’m sure! ↩︎
Possibly this is only a wording issue, but assuming you mean “independent” or “isolated”, it’s too vague to give any advice. Independent from what? Bundled as an application? No package dependencies? Able to run on multiple platforms? There are too many options for how to answer without more explanation. ↩︎
