Hi everyone.
I am a kind of python novice, sorry for dumb question here in advance…
I recently watched a few conference talks about functional programming.
(for example, Scott Wlaschin’s Domain Modeling Made Functional, Composition, Functional design pattern etc using F#)
Without jargons (functor, monoid, monads etc.), the basic concepts and approaches are very attractive for me and make me not to be scared about type systems…So I want to give a try using Python in a kind of functional way.
Python is an interpreted, interactive, object-oriented programming language. It incorporates modules, exceptions, dynamic typing, very high level dynamic data types, and classes. It supports multiple programming paradigms beyond object-oriented programming, such as procedural and functional programming.
Although Python is not designed for “functional programming languages” at first (to be honest I only have shallow understanding about difference in approach, philosophy, toolkit between OOP, and FP…), Is it okay and possible(even recommended sometimes) to write software fully in functional way using Python?
Is it possible? - It depends on what you mean by “functional”. If you want to be “strictly” or “purely” functional, it’s not possible. But it’s definitely possible to apply a functional style. Perhaps the single most important thing there is avoiding side-effects in functions. Functions with side-effects often are troublesome and make analyzing programs a lot harder, so this is sth that is good to always apply, imo.
Avoiding statefullness is also sth that I think should generally be applied - again as far as possible. But here the question again is - how far do you want to go? (Python generators and coroutines for instance are very useful but maintain state.)
Is it okay? – Nobody is going to forbid you (I hope)
If you want to look at examples:
The pandas APIs for the most part apply (and invite) a functional programming style; the pyspark API is similar.
In the stdlib functools and itertools use a functional style (see docs - itertools explicitly mentions it’s inspired by Haskell); functools is a bunch of convenience functions to create higher-order functions
Python is described as an object-oriented language (and, in my view, it takes this idea more seriously than many other languages like Java that are commonly held up as archetypes of “object-oriented languages”); but it is still designed very much as a pragmatic language that supports multiple programming paradigms and allows for writing code in different styles. In that regard I think it is similar to C++ - only, far more pleasant to use
Because of Python’s commitment to the “everything is an object” principle of OO, functions (as well as other “callables”, such as classes) are first-class objects. This makes it natural to design and use higher-order functions.
Python has a lambda (anonymous function) syntax, using the keyword lambda. It is restricted: the body of a lambda must be a single expression. (However, Python expressions can have side effects, and there is no general way to prevent this.) This syntax produces a function object (the same type as the def statement does); in the corresponding bytecode, the result of the expression is returned.
The decorator mechanism is implemented using higher-order functions, in fact. For example:
def spam(func):
return lambda: ' '.join(func() for _ in range(10))
@spam
def what_the_knights_say():
return 'Ni!'
# The `@call_twice_and_sum` before the function is roughly equivalent to doing
# `what_the_knights_say = spam(what_the_knights_say)` afterwards.
As mentioned, the standard library functools and itertools are invaluable here. In particular, functools.reduce implements a fold (a left fold, if I’m thinking clearly). map and filter are builtins, and there is a basic list-comprehension syntax that generalizes to set and tuple comprehensions and generator expressions. Most evaluation in Python is eager, but there are several lazy constructs available. Typically we use generator expressions to create generators (a kind of lazy iterable), and functionality from itertools is designed to accommodate lazy iterables.
Function composition is not supported out of the box, but it’s easy enough to implement simple cases using the above tools, and there are third-party packages for more robust solutions.
It is a maxim of Pythonistas that “practicality beats purity”. I personally have a fair affinity for functional style, and use list comprehensions in places that many other developers would write an explicit for loop. I even advocate for teaching list comprehensions to beginners first, as I sincerely believe they are easier to understand and analyze for most common cases than explicit iteration. However, when I write code I don’t try to avoid side effects and I certainly don’t succeed in doing so. Nor do I try to implement monads etc. or find myself wanting them. In the Python world, the assumption of imperative style for I/O is rarely questioned; it’s generally considered abusive to run a list comprehension (and discard the result) simply to cause I/O as a side effect, for example.
My entire Python career and open source oeuvre (finally a chance to use this word) is based on prioritizing functions and function composition over subclassing and OOP, so it’s indeed possible albeit going against the grain somewhat.
That said, you probably won’t find a ton of support for monads and stuff like that. But just favoring functions as first class objects, falling back to classes when you must, can get you very far.
Thanks for your kind and helpful comments! It’s awesome.
The fact that function in python is first-class as you mentioned make me think it can be possible to write code in functional way.
Yeah… composition is key to functional programming I think.
Composing functions, Composing types(algebraic type system?) bla bla…
It’s mind-blowing experience for me to watch pipeline-oriented and railway-oriented programming talks in Youtube (it’s all about composing functions), and see Rust’s official books about error and null? handling such as Result, Option.
I found the open source projects related to above things such as
I am a quite novice, so all thoses are not easy for me to understanding implementations and I want to build it by myself using plain, standard library python as possible as I can. I hope that It can be valuable for me to learn basic and core concepts about programming and python itself.
Anyway, thanks for your reply. I have learned a lot from you!!
My primary language is definitely not English.
So I checked to oxford learner’s dictionary… Cool!
I envy you for making open source oeuvre, I hope I can do it like you in the future.
It’s totally understandable although it’s sad for me.
Oh… I see. I wrote down your saying in my engineering daybook…!
I have actually written embedded SW for microcontrollers using ‘C’. Now that I am learning Python,
it appears that there is a strong emphasis for the OOP approach (especially for the application of the Tkinter module - I am sure it applies to other modules and Python in general). The main difference that I notice is that for python, you generally are strongly encouraged to create classes for everything (objects - inheritance, composition, etc.) with an emphasis on encapsulation.
With procedural programming for embedded applications, I never considered this in general. Writing programs for small embedded applications was generally written in a linear, procedural way. The closest I came to creating an object was creating C structures when I wanted to create multiple one to a few bit flags from one 16-bit structure (no reason to use a 16 bit/8 bit variable for example, if the flag’s value is only a ‘1’ or a ‘0’). Embedded applications tend to be memory (quantity) sensitive.
So, in a nutshell, kind of, in functional/procedural programming, we don’t worry about classes/encapsulation.
If you want to learn functional/procedural programming, try learning ‘C’.
IMO
Python is so powerful you can use everytime and everywhere a functional style, that’s one of the most impressive skills this languages has.
Anyway, if you know OOP you’d probably never use a fully-functional paradigm approach (with Python), because of encapsulation, design patterns, low-level abstraction, etc.
I typically try to use a functional programming style in Python as much as possible. But I think it is important not to force it. As others have already mentioned, itertools and functools are your friend. Iterators (although they’re stateful) can also support a functional programming style (just be aware of the iteration side effect). And try to limit side effects and make them explicit in docstrings. Some other things that have not been mentioned yet that fit well in the functional programming style in my opinion:
more-itertools, more functions like the ones in itertools
Frozen dataclasses (dataclass package) are (pseudo) immutable
If you’re using type annotations, use the abstract types from collections.abc (e.g. Sequence) for function parameters rather than the concrete built-ins (e.g. list) to communicate clearly to the users of your functions that arguments will not be mutated.
These present a nice toolbox for functional programming, but in my opinion the other side is just as important: don’t force it. Python is not a purely functional programming language, and though it pains me sometimes, I will prefer an imperative style over a functional one in places where it just fits the language better. A good example are the built-in collection types (list, set, dict). They don’t have a functional manipulation API (at least not without creating an expensive copy), these operations have an inherent side effect. This makes it difficult to construct e.g. a dict by folding (functools.reduce) over another iterable. In these situations I will typically opt for a for loop, because it’s just more natural and more ideomatic, even if it still bothers me every time I have to do so.
So my advice would be to use the tools that are available and you’ll be able to use a functional programming style for a large part of your code. But whenever it feels like you have to go against the grain of the language itself, just make your peace with the imperative style and don’t force it.
Thanks for sharing your opinion!
I totally agreed with you that Python is so versatile, (making me lost sometimes…)
I know same problem can be solved using OOP if well coded, but that’s hard for me now.
Maybe I am not good at OOP in python way. I think I have to put my efforts more deliberately into reading open source projects.
Although I don’t get 100% understanding in your comment (becuase of my lack of python experience and skill issues), I get to the point of what you said in general.
I will write your saying below in my engineering daybook by hand.
It’ll take some time for me to get good taste on code written in pythonic way.
As others have said, you won’t be able to do “pure” functional
programming in Python the way you can in a language designed
specifically for it, e.g. LISP. I’m not sure that really matters
much for the utility of the language, though, and in many cases
being able to mix in constructs that aren’t purely functional opens
up the possibility to create superior solutions…
I started learning rudimentary computer programming in 1980. I’ve
had no formal training, but simply used the tools available to me in
order to solve problems that needed solving. The first languages I
picked up were essentially procedural in nature, interpreters that
came with my computer’s operating system. Later as I needed the
performance boost of compiled programs, I experimented with
functional programming because those were some of the compilers I
could get my hands on. Over time, as portability became a concern
for me I switched languages more and started to write in some that
leaned into object oriented design.
I never really thought about procedural vs. functional vs. object
oriented design paradigms back then, I simply looked at examples of
source code similar to what I wanted to accomplish, played around
with them until I gained an understanding of how they worked, and
internalized that knowledge. These days Python is my go-to language,
and a major reason is that it can be used in any of those ways (and
more). You’re not limited to using one specific kind of programming,
even within the same body of source code.
While the vast majority of problems you’ll encounter can be solved
with a program written in any of those styles, no one style is
superior to the others, and different styles are often better for
different kinds of tasks. Unless your field is actually language
theory though, my recommendation is to not worry too much about what
style you’re writing in. Learn to use the language hands-on through
solving problems that need solving, and when you come across a
construct that you don’t understand play around with it until it
makes sense to you.
As you learn, you’ll find cleaner and more elegant solutions to
problems. Sometimes that will look “functional” from an academic
perspective, sometimes it might look “procedural” or “object
oriented” or a mix of those and more. You’ll internalize the
algorithms, data models and relationships that work well in
different situations, but you won’t really waste time thinking about
whether what you’ve written is “proper” functional style. Making
software that works as intended and is easy for others to understand
is usually far more important than language theory.
I completely agree with the balanced opinions above: it is completely possible (and quite easy) to write parts of your Python program in a functional style, but if one clings blindly to the ways things are done in languages where the compilers/interpreters are specifically designed for that, the resulting code may turn out to be a) difficult to read by others, and b) inefficient. @sanderr did an excellent “pros and cons” list; I’d just like to add that the expensive copies problem does not only affect lists and dictionaries, but also immutable values. I love dataclasses, I love frozen dataclasses with a passion, but the compiler is simply not designed to optimize for methods that consume the original object and return a slightly modified version - the original object still remains in memory for a while, the modified version is a copy. If one tries to write a complex program with thousands of such consumed-and-thrown-away objects, things start to pile up (of course, in a well-designed program each individual consumed object will be garbage-collected pretty quickly, but still).
Note that the above is NOT meant as a criticism of Python; as I wrote, it is not designed that way, and the things that it is designed for it does very well indeed.
This is a non-trivial problem even for languages that are explicitly designed for the paradigm. Of course, they generally do put a lot more effort into trying to solve it
Thanks for sharing your stories.
I have learned a lot from reading your comment more than twice.
It seems like a cheating to get invaluable lessons from others by just reading comments.
However it’s useless if I failed to internalize good things by programming a lot with hard efforts on solving real world problems as you mentioned.
