Greetings all,
For the last six months, I’ve been fuzzing CPython using a tool called fusil, created by @vstinner about 20 years ago. This campaign resulted in 52 issues filed, and in these six months that corresponds to a little less than 30% of all crash issues created.
Now that this effort is coming to a (temporary!) end, I’m writing a technical report about it. For that report, core developers’ feedback would be very important to assess this campaign’s:
- usefulness (e.g. did it help CPython development in any tangible way?);
- cost/benefit (e.g. was it worth doing, and are the results compatible with the resources used?);
- impact (e.g. were any significant issues fixed?);
- quality of reports (e.g. is filing issues without understanding the cause and before having diagnosed what’s happening a problem?);
- disruption of normal development flow (e.g. how bad is it to have a constant trickle of issues being filed?);
- suggestions for any future efforts (e.g. file a single issue with all findings and let developers create issues from that?);
- approach to getting help triaging found issues (e.g. was it a nuisance to have constant questions in the community Python Discord about what did a crash mean?).
That is, any positive or negative feedback would be very welcome. Suggestions for improvements and constructive criticism would be wonderful, but if all you have to contribute is something like “I don’t think it really helps”, “never heard of it”, “I liked some issues”, that would still be valuable.
This thread is for free-form feedback, to gather different opinions.If you want your response to be quotable in the report, please indicate that quoting with attribution (or anonymously if you’d rather) is fine. Depending on response/interest, I might create a poll too.
I’d like to thank @vstinner, @ZeroIntensity, @Jelle, @picnixz, @sobolevn, @kumaraditya303, and everybody else who helped triage, diagnose and fix the issues.
Thank you for your time!
Daniel
P.S.: As a thank you for reading this far, here’s a draft visualization of the temporal pattern of issue filing, in which it’s clear success is clustered. The reason is that after a while all fusil known tricks have paid-off and stopped uncovering new crashes, until a new feature is added and it starts finding new hits again.