Performance Analysis Methodology

A performance analysis methodology is a procedure that you can follow to analyze system or application performance. These generally provide a starting point and then guidance to root cause, or causes. Different methodologies are suited for solving different classes of issues, and you may try more than one before accomplishing your goal. Analysis without a methodology can become a fishing expedition, where metrics are examined ad hoc, until the issue is found – if it is at all.
Methodologies can help with another difficult issue: when to give up. At any complex environment, spending too much time analyzing one performance problem takes time away from other problems and other possible wins. Performance engineers can spend six months working a single issue without success, when it would be better to try for a week and then turn their attention elsewhere. My recommendation is to try several methodologies on a given issue or theory before moving on, for example: 1) Problem Statement Method, 2) Workload Characterization Method, 3) USE Method, 4) CPU Profile Method, 5) Off-CPU Analysis, 6) Time Division Method.
Methodologies documented in more detail on this site (beyond this summary page) are:
The following summarizes methodologies I've either created or encountered. You can print these all out as a cheetsheet/reminder.
I first summarized and named various performance methodologies (mostly developed by me) for my USENIX LISA 2012 talk: Performance Analysis Methodology (PDF, slideshare, youtube, USENIX), then later documented them in my Systems Performance book, and the ACMQ article Thinking Methodically about Performance, which was also published in Communications of the ACM, Feb 2013. More detailed references are at the end of this page.
The following is my most up to date summary list, with methodologies enumerated. These begin with anti-methods, which are included for comparison, and not to follow.
(Read The Fine Manual) How to research performance tools and metrics:
(I previously called this the "Latency Analysis Method")
Measure latency in detail (eg, as a histogram) from:
Investigate the lowest layer that latency is introduced
For every resource, check:
For every service or microservice, check: