Any interest in working out a standard API for calling linters?

[brettcannon]

I need to preface this with the fact that I have not spoken to a single linter team about this, but since this involves coordinating between two different groups I need to start somewhere.

I think pretty much all of us support some linter like flake8. I think that means we have all written our own regex to parse flake8’s output. What I would like to know is who here would be interested in working with the various linter teams over in PyCQA to see if we can come up with a simple API that could be provided as an entry point by linters so editors can stop parsing stdout and instead call a Python API? That way if flake8 implemented said entry point then editors could use that API to get diagnostic details instead of parsing stdout.

But there is also a knock-on effect that if the API becomes something that people can agree to then support in our editors can occur for new/other linters as fast as they implement the API. So instead of all of us having to implement yet another regex when some new linter comes on the scene we could say instead, “if you implement this entry point we can support you immediately”. Linters get better, faster, more controlled support from us and vice-versa; win-win!

And I am purposefully not suggesting LSP here since we don’t need to be language-agnostic as the linters we all seem to care about are already written in Python (not to say that LSP implementations or some other wrapping mechanism couldn’t take advantage of this work).

Anyway, as a rep for the Python extension for VS Code I’m willing to say we are interested in this. Any other editors interested in working with the PyCQA to design such an API? If there’s enough interest, my plan would be:

1. Find out which editors are interested.
2. Tell PyCQA what we are thinking about doing here so they can choose to participate now or wait to see what we come up with
3. Come up with a design we can all agree with here (I have already done research on editors and linters and have a draft API)
4. Take the design to PyCQA.
5. Resolve any requested changes.
6. ???
7. Profit! (South Park reference for those that aren’t familiar.)

[isidentical (Batuhan Taskaya)]

@brettcannon

What I would like to know is who here would be interested in working with the various linter teams over in PyCQA to see if we can come up with a simple API that could be provided as an entry point by linters so editors can stop parsing stdout and instead call a Python API?

I’m not a member of PyCQA, but rather maintain 2 python linters (unimport and teyit), if needed would love to help on forming such a format!

One thing that I didn’t understand here is that, whether we are debating a format of our own or rather using something existing (like SARIF)?

[brettcannon]

@isidentical

I’m not a member of PyCQA, but rather maintain 2 python linters (unimport and teyit), if needed would love to help on forming such a format!

That’s great to hear!

@isidentical

One thing that I didn’t understand here is that, whether we are debating a format of our own or rather using something existing (like SARIF)?

So I’m thinking more of an API and not a log/result format, e.g. teyit.editors:call() would be registered as an entry point and editors would call that callable directly to get the linting results. My hope is the API would work well enough that it would just sit in between the UI of the linter and the logic code of the linter.

Asking linters to support SARIF is definitely an interesting idea, although I’m not sure how annoying it would be to run a linter, read a file, parse it, and then display the results compared to just calling a function from Python and getting the results directly.

[brettcannon]

@brettcannon

Asking linters to support SARIF is definitely an interesting idea, although I’m not sure how annoying it would be to run a linter, read a file, parse it, and then display the results compared to just calling a function from Python and getting the results directly.

And to make it explicit, that’s an open question to other editors whether parsing a SARIF file would work for them. SARIF support is a generic thing that other systems support to asking for it to help us out might be a motivator for linters (although by itself it doesn’t solve the disoverability or unified API problem if linters don’t all use the same CLI flag to control this).

[smontanaro (Skip Montanaro)]

Wouldn’t a standardized output format be sufficient? Is an actual callable API required? One reason i find a standardized API problematic is that the command line interfaces of the few linters I’ve used seem to be pretty different. I assume current CLI usage would inform any API creation.

Forgive me if I’m putting the cart before the horse here. I realize that’s not quite what you’re asking in this thread.

[brettcannon]

@smontanaro

Wouldn’t a standardized output format be sufficient?

It might be sufficient, just not ideal for users. I’m just imagining trying to explain to a new user “to use your linter, find out how to call it, add the appropriate flag to generate a SARIF file, and then tell us where that file is” (and that’s even if they were provided with a placeholder to let us tell you where to write the SARIF file).

@smontanaro

Is an actual callable API required?

Required? Probably not. Would make the experience for users smoother? Yes.

@smontanaro

One reason i find a standardized API problematic is that the command line interfaces of the few linters I’ve used seem to be pretty different. I assume current CLI usage would inform any API creation.

So if this wasn’t in an editor where setting and/or changing CLI flags to a linter wasn’t a pain I might agree. But since this is from an editor perspective you typically want users to use configuration files, which means the linter itself will pick up the settings via its own default configuration file search/load mechanism.

[brettcannon]

The alternative, low tech solution is we have a way for linters to provide the regex they want us to use to parse their output. If that object was specified as an entry point then that would also be discoverable.

[terrdavis (Terry Davis)]

Or provide a JSON output option? (Is that what you’re referring to, @smontanaro?)

[smontanaro (Skip Montanaro)]

Yeah, JSON or output standardization.

I had in the back of my mind that somehow text editors and IDEs have managed to deal with the output of compilers for a long time. I’m an Emacs user and have probably encountered a dozen different C/C++ compilers on a number of different operating systems over the years. Parsing output? Yeah, you can call it that, but just barely. I imagine the Emacs approach has changed a bit over the years, but in the old days (when I still grubbed around with Emacs LISP), the code that dealt with compiler output had an alist of (I think) (compiler regex) pairs. (Yup, it’s still there, though it’s grown significantly over the years.) Depending on the particular compiler, the appropriate regex was a matter of a one-time selection. This method of operation was established long before JSON was a gleam in anybody’s eye, but if you had JSON available, I could easily see the compiler called with a --format=json flag to produce structured input to the caller, in much the same way that you can just toss “&_rss=1” on an eBay search URL to get RSS instead of HTML output. Having JSON as an output option might reduce the work necessary for such a project to settling on the structure of the output.

Brett, i assume none of this is news to you though. As you’re not someone i think of as one who casually tosses out suggestions for more work, maybe you can elaborate a bit more, maybe offer up a scenario or two where current practice is insufficient. Also, FWIW, as an old curmudgeon I will tell you I am almost certainly not going to give up Emacs as my programming editor/IDE (I know a few people who feel the same way about vim), so there are two examples of non-Python callers of flake8, pylint, etc.

[brettcannon]

@smontanaro

i assume none of this is news to you though.

Nope. VS Code has the concept of tasks for this exact purpose of parsing compiler output.

@smontanaro

As you’re not someone i think of as one who casually tosses out suggestions for more work

I try not to.

@smontanaro

FWIW, as an old curmudgeon I will tell you I am almost certainly not going to give up Emacs as my programming editor/IDE (I know a few people who feel the same way about vim),

Same for any editor really. Hence why I’m trying to find consensus around this idea instead of just saying, “VS Code is doing this for us and anyone who follows what we say will get support” as I feel like that’s throwing our weight around a bit too much.

@smontanaro

there are two examples of non-Python callers of flake8, pylint, etc.

So I would disagree with that characterization for two reasons. One, VS Code extensions are written in JavaScript, so you can add us to your list. Two, you’re still calling Python to run flake8, pylint, etc., so it isn’t that you have “non-Python callers”, you’re just saying “direct callers of the linters”. You could conceivably call a wrapper command/library like flake8-standardized or something, but you’re still calling Python regardless from a non-Python app probably for a decent chunk of folks.

@smontanaro

maybe you can elaborate a bit more, maybe offer up a scenario or two where current practice is insufficient.

So running a linter requires three things:

1. What linters does the user want to run?
2. When does the user want to run the linter(s)?
3. How do you run the linter(s)?
4. How do you get results from the linter(s)?

For knowing what linter(s) a user wants to be using, typically that’s done via a configuration of some sort for the editor (e.g. for VS Code it’s your settings.json). The typically problem with that, though, is it can trip up beginners who have simply been told to “use flake8”. It’s also a nicer experience if you can just detect any and all installed linters in a virtual environment and then automatically use them.

As for when to run linters, that doesn’t necessarily plug into when you want to run a compiler. Linters usually run either manually, on every save, or on every edit. Various editors like VS Code thus have specific triggers for running linters which do not overlap with when e.g. compilers run. That also means that linters are expected to feed through a specific system which does not necessarily align with the “just give us a regex and we will automatically handle surfacing errors” system.

Assuming you do know what linter(s) you want to run and when to run them, the next question is how do you do that? Every linter has its own approach to execution. E.g. some may lint all files if you don’t specify an argument while some may require at least . to start a linting. So even with standardized output you need to know how to execute the linter(s) appropriately.

And then there’s the suggestion of standardizing the output. That’s for obviously reasons, else we wouldn’t be having this conversation.

So to answer my own questions:

1. What linters does the user want to run? Use an entry point for automatic discovery.
2. When does the user want to run the linter(s)? VS Code takes care of this for me.
3. How do you run the linter(s)? Standardized API which the entry point specifies the location of.
4. How do you get results from the linter(s)? Standardized API unifies the results.

[smontanaro (Skip Montanaro)]

there are two examples of non-Python callers of flake8, pylint, etc.

So I would disagree with that characterization for two reasons. One, VS Code extensions are written in JavaScript, so you can add us to your list.

(attempting reply by email with quote trimming - I’m skeptical this will work properly)

Got it. I misunderstood what you originally wrote, thinking you meant that only Python programs would be calling these tools.

[tjreedy (Terry Jan Reedy)]

For people as ignorant as I was: https://github.com/PyCQA.

There have been proposals that IDLE users be able to run either one particular code checker (which I vetoed) or generically any locally installed code analysis program over an editor’s contents. A GSOC student worked on the latter, but users would be required to register an import name and subprocess command line. We did not finish, and one of the issues was variation in line output formats.

Since IDLE is all Python, my personal preference would be for a lint API (LAPI), similar to the DBAPI to get an iterable of tuples such as (filename, line, column, comment), to use in a ‘for’ statement. For general callers, a generic format for output lines would be really nice. The pycodestyle ‘filename:line:col: comment’ layout looks decent. But I don’t care about the particulars. To not break current users, a standard command-line option could be required by programs with a different form, and accepted by programs that already use that form.

I would like there to be a PEP defining the standard so I would be allowed to use it in IDLE.

[isidentical (Batuhan Taskaya)]

@tjreedy

Since IDLE is all Python, my personal preference would be for a lint API (LAPI), similar to the DBAPI to get an iterable of tuples such as (filename, line, column, comment), to use in a ‘for’ statement. For general callers, a generic format for output lines would be really nice. The pycodestyle ‘filename:line:col: comment’ layout looks decent. But I don’t care about the particulars. To not break current users, a standard command-line option could be required by programs with a different form, and accepted by programs that already use that form.

This sounds like the simplest solution, though would probably nicer if we would extend it a bit. Since 3.8, the AST has position start-end location, instead of just the start, and this is extremely useful when you are highlighting a particular expression in a long line.

I am not sure how much of the linters actually support it (as of right now) (due to they still support older versions), but since this is going to be long term API, it might be nicer to consider a bit of extended format. Maybe (filename: PathLike, position: Tuple[int, int, Optionalint], Optional[int]], message: str, extra_properties: Dict[str, Any]).

I would like there to be a PEP defining the standard so I would be allowed to use it in IDLE.

+1! If anyone interested in starting to work, please let me know. I’d be happy to contribute to the efforts.

[brettcannon]

@tjreedy

Since IDLE is all Python, my personal preference would be for a lint API (LAPI), similar to the DBAPI to get an iterable of tuples such as (filename, line, column, comment), to use in a ‘for’ statement.

That’s basically what I’m after as well (maybe a little more structured than a tuple, but same idea that you get an iterable back to loop over).

@tjreedy

The pycodestyle ‘filename:line:col: comment’ layout looks decent. But I don’t care about the particulars. To not break current users, a standard command-line option could be required by programs with a different form, and accepted by programs that already use that form.

If we can get an API then doing a wrapper to normalize output wouldn’t be difficult. And using entry point would allow for automatic tool discovery.

@tjreedy

I would like there to be a PEP defining the standard so I would be allowed to use it in IDLE.

My assumption is this would become a PEP to make sure linters in general were aware of it.

@isidentical

Maybe (filename: PathLike, position: Tuple[int, int, Optionalint], Optional[int]], message: str, extra_properties: Dict[str, Any]) .

My current thinking of an API is:

Class Diagnostic:
    path: os.PathLike[str]
	position: ((int, Optional[int]), (Optional[int], Optional[int]))  # To support col/row range.
	severity: SeverityEnum  # Or string with predefined values.
	message: str
	message_id: str
    tool: str

entry_point(root: Optional[os.PathLike[str]] = None, path: Optional[os.PathLike[str]] = None, source: Optional[str] = None) -> Iterable[Diagnostic]

@isidentical

+1! If anyone interested in starting to work, please let me know. I’d be happy to contribute to the efforts.

At this point I’m waiting to see who speaks up with interest and then going to PyCQA to see what the interest is over there.

[AWhetter (Ashley Whetter)]

As a member of the PyCQA, but by no means a spokesperson for them, this is an exciting proposal that I think the PyCQA is in a good position to be involved with.

If you want to float the idea to the group then I would suggest getting in contact with the group directly: https://meta.pycqa.org/en/latest/getting-in-touch.html

[brettcannon]

@AWhetter

As a member of the PyCQA, but by no means a spokesperson for them, this is an exciting proposal

That’s great to hear!

@AWhetter

I think the PyCQA is in a good position to be involved with.

I fully expect to pull them in as this whole idea isn’t useful without PyCQA buy-in.

@AWhetter

If you want to float the idea to the group then I would suggest getting in contact with the group directly

My plan is to email PyCQA in the new year to give any other editor folks time to chime in before I fork this conversation.

[sigmavirus24 (Ian Stapleton Cordasco)]

I have two thoughts here really:

First: What have I created? The ‘A’ in PyCQA was intended as a joke cue Frankenstein-esque horror music

Second, and more seriously: A programmatic API makes sense. If it was just a matter of formatting, then this would already be done (flake8 has pluggable formatters and flake8-json prints JSON and has a CodeClimate formatter as well). This, however, creates a few areas of worry:

1. Thinking specifically about Flake8 which has a configuration option merging based on “hierarchy” (CLI overrides local project config file which overrides the user config). How would the API handle the configuration there?

2. Flake8, pycodestyle, and others don’t really have severity levels today (despite naming of the violation codes), without wanting to grow that, would there be some way of skipping that?

3. Flake8, pylint, and others can use multiprocessing for parallelism to speed things up. Is that something that needs to be considered while thinking about this API?

4. What are the output constraints besides the dataclass you outlined? Today, Flake8 goes to great lengths to report things in order top-to-bottom but could provide faster results with unordered output. Is that something the editors would be comfortable with?

5. What happens when the LSP changes (as I’m certain it eventually will)? What will editors have to do to conform? What about linters?

6. I’d rather wait on finalizing the PEP until we’ve had implementations (similar to how the IETF works) where linters go through a spike to ensure that everything’s doable. I’d rather us have the right APIs rather than have a PEP that mostly works.

[pf_moore (Paul Moore)]

@sigmavirus24

What have I created? The ‘A’ in PyCQA was intended as a joke

PyPA member nods sympathetically…

[brettcannon]

@brettcannon

My plan is to email PyCQA in the new year to give any other editor folks time to chime in before I fork this conversation.

I have made the post to code-quality@.

@sigmavirus24

Thinking specifically about Flake8 which has a configuration option merging based on “hierarchy” (CLI overrides local project config file which overrides the user config). How would the API handle the configuration there?

It wouldn’t because that instantly becomes tool-specific. My assumption is users will set things in the appropriate config file and the tool’s normal setting resolution will be relied upon from those config files.

@sigmavirus24

Flake8, pycodestyle, and others don’t really have severity levels today (despite naming of the violation codes), without wanting to grow that, would there be some way of skipping that?

Could have a default value if not specified. (BTW, the most common complaint about flake8 that we get is the lack of that specification as we then have to come up with it and everyone inevitably disagrees ).

@sigmavirus24

Flake8, pylint, and others can use multiprocessing for parallelism to speed things up. Is that something that needs to be considered while thinking about this API?

I don’t think so. If it’s transparent then it shouldn’t be a concern.

@sigmavirus24

What are the output constraints besides the dataclass you outlined? Today, Flake8 goes to great lengths to report things in order top-to-bottom but could provide faster results with unordered output. Is that something the editors would be comfortable with?

I personally couldn’t care less about report order. We have to parse the output to use the appropriate VS Code hooks to surface the results. I think a key thing here is the API would be driving the linters as libraries, not tools with UIs; the editors are the UI in this case, so you can leave that stuff to us.

@sigmavirus24

What happens when the LSP changes (as I’m certain it eventually will)? What will editors have to do to conform? What about linters?

Don’t worry about LSP stuff. That’s a stretch goal for me. And my proposal above is already over-engineered for the future since none of the linters I know about specify start/stop positions and half of them only specify the line and no column.

But to directly answer your question, LSP has mechanisms for communicating protocol support between client and server. As long as this API provides the data necessary for linters to express what they need to, then whatever LSP server uses it can do the translation to support what LSP needs.

This is one potential bonus to having a SARIF JSON object as the return type since that itself is versioned and has been worked through.

@sigmavirus24

I’d rather wait on finalizing the PEP until we’ve had implementations (similar to how the IETF works) where linters go through a spike to ensure that everything’s doable. I’d rather us have the right APIs rather than have a PEP that mostly works.

Fine by me!

[sigmavirus24 (Ian Stapleton Cordasco)]

@brettcannon

Could have a default value if not specified. (BTW, the most common complaint about flake8 that we get is the lack of that specification as we then have to come up with it and everyone inevitably disagrees ).

That makes sense. I’ve thought about it a bunch and I think that the problem started with pycodestyle (then pep8) calling things “errors” and “warnings”. I’ve thought about allowing plugins to report that, but it doesn’t fit in to the default format we’re using and that’s part of the API we provide folks anyway so we wouldn’t want to break it. I think if we’re going to report severity though, it’s not as useful without confidence (e.g., how confident is the tool that this is actually a big problem. Bandit has these concepts already and they’re quite useful together) so if anything, I’d argue that flake8 should grow that ability to collect/report severity and confidence and then the plugins need to provide those values.

@brettcannon

It wouldn’t because that instantly becomes tool-specific. My assumption is users will set things in the appropriate config file and the tool’s normal setting resolution will be relied upon from those config files.

That’s exactly what I wanted to hear, honestly.

@brettcannon

I don’t think so. If it’s transparent then it shouldn’t be a concern.

Perfect. I didn’t want us stepping on toes for VS Code, that said, if I remember correctly, flake8’s parallelism is per-file so this is moot anyway as call happens per file.

@brettcannon

I personally couldn’t care less about report order. We have to parse the output to use the appropriate VS Code hooks to surface the results. I think a key thing here is the API would be driving the linters as libraries, not tools with UIs; the editors are the UI in this case, so you can leave that stuff to us.

Just wanted to make sure there wouldn’t be assumptions. That’s something I’d like to make sure ends up in the PEP when one is written.

@brettcannon

Fine by me!

To be clear, I just want the group to have the flexibility to iterate on the API if necessary. I’m not against writing the PEP up-front. Just want to make sure we’re not all finalizing a PEP before we have had an opportunity to kick the tires