Confirm collected pystats are reliable #511

Activity

1. mdboom

   self-assigned this

   on Dec 9, 2022
2. mdboom commented on Dec 9, 2022

   

   mdboom

   on Dec 9, 2022

   ContributorAuthor

   More actions

   I think I've found the crux of the problem: when the process is in the "stats off" state (after calling sys._stats_off), stats aren't dumped. When we run a benchmark, we surround the benchmark code with sys._stats_on() and sys._stats_off(), and then there's nothing dumped out at the end of the process. The fact that we are getting any stats at all is a sign of leakage from non-benchmark code (which I'll consider a separate issue that I haven't yet solved).

   There's a couple ways to solve this:

   • Modify pyperf to call sys._stats_dump(); sys._stats_off() after running benchmark code. This will leak the operation of calling sys._stats_dump() itself, but maybe not a huge deal.
   • Modify CPython to always dump whatever is left, even if stats is turned off. I think this is probably the better, less surprising solution.

   @markshannon Thoughts?
3. markshannon commented on Dec 9, 2022

   

   markshannon

   on Dec 9, 2022

   Member

   More actions

   Stats should be dumped, even if stats are off.

   Because stats are on at startup, any non-benchmark process will dump stats, so that also needs to be fixed.

   Maybe start with stats off by default, and turn them on during startup if an -Xstats option is enabled?

   That way the stats should work for pyperformance as it is now, but we still have the option to gather stats for a whole program.
4. mdboom commented on Dec 9, 2022

   

   mdboom

   on Dec 9, 2022

   ContributorAuthor

   More actions

   See python/cpython#100144
5. sweeneyde commented on Dec 12, 2022

   

   sweeneyde

   on Dec 12, 2022

   More actions

   It seems the .json files are not comparable to each other because they have keys like "opcode[157].pair_count[145]", which changes meanings as bytecodes change.

   Couldn't _Py_GetSpecializationStats be modified to output _PyOpcode_OpName[i] instead of the integer i? I wonder to what degree that would clean up the various scripts. It would also make it so that a different python could run the summary script than the one whose stats are being taken.
6. mdboom commented on Dec 12, 2022

   

   mdboom

   on Dec 12, 2022

   ContributorAuthor

   More actions

   The .json files are just shorter summaries of the raw output, which is designed to be as easy as possible to generate from C. (The C code doesn't have access to the opcode names, or would have to add a table for that or import opcode.py etc...)

   I'm not sure there's a lot of value in making this data format well-defined and permanent. The use case for comparing really only makes sense with adjacent revisions anyway. Personally, I see the requirement to use a matching version of the script as a feature -- we can change the semantics of how this works as we understand the problems better without concern for backward compatibility.

   All that said, we should probably agree on a policy about this and document it somewhere.
7. mdboom

   closed this as completedon Dec 20, 2022