The TSA Method

There are two basic performance analysis methodologies you can use for most performance issues. The first is the resource-oriented USE Method, which provides a checklist for identifying common bottlenecks and errors. The second is the thread-oriented TSA Method, for identifying issues causing poor thread performance. I summarized the TSA Method in my Stop The Guessing talk at Velocity conf this year, and it is also covered in the Applications chapter of my Systems Performance book.
The TSA Method is complementary to the USE Method as it has a different perspective: threads instead of resources. Like the USE Method, it provides a starting point for analysis, and then narrows the investigation to the problem area. It can also be applied to any OS, as it is developed from the questions we'd like like answered: where is each thread spending its time?
I've taught this method in recent systems performance classes, where it has helped my students – mostly system administrators – to quickly solve a range of issues. I've also been using it for years, in both enterprise and cloud production environments, where it has solved countless performance issues.
The Thread State Analysis (TSA) Method can be summarized as:
I use the term "thread" to denote an operating system runnable entity, which may be a thread, task, or process. The threads of interest may be application threads or kernel threads.
Thread time is divided into several exclusive states. The following set of six are based on a generic operating system kernel, and have been chosen to identify key sources of performance issues:
These can then be adjusted for your OS, depending on the tools it provides for measurement and analysis. Some additional states may also be added, if the OS provides them easily and they are useful, and the number of states remains small enough to be practically applied.
In preparing to use the TSA method, you can identify the set of states to measure for your OS, document the OS tools available to measure them, and how to further analysis each state. This can be shared with your team, providing everyone with a short procedure for analyzing performance issues. In a moment I'll provide a summary of this for Solaris as an example.
The following short example describes a common performance issue in cloud computing, and how the TSA Method provides a starting point and then direction for further investigation.
An application has a performance issue. Following the TSA Method, thread state time is measured for the application threads, in each of the six states. About 50% of their time is in the Runnable state: waiting for a turn on-CPU. Tools for investigating that state are now used, including mpstat(1) to see if the CPUs are overloaded, and specific tools for checking CPU resource controls. It is quickly found that the application is throttled by a CPU limit, which is normally present in this cloud computing environment. The CPU limit is increased, and performance is resolved.
Some monitoring software provide a breakdown of time by application component, with metrics for "time spent in MySQL", "time spent in PHP", and so on. This usually measures time from when the application component began servicing work, to when it completed. Instead of being thread-oriented, this approach is request-oriented, breaking down the request time into the application components involved. This approach can also be misleading, but can be clarified by using the TSA Method to investigate each component.
For example, to better understand "time spent in MySQL", use the TSA Method to examine MySQL threads. You may find that MySQL is spending significant time in the Runnable state, due to another application or tenant that is stealing CPU cycles. Reporting this as "time spent in MySQL" is misleading: this is not time spent executing MySQL, instead, it may be time spent in another application or tenant entirely. The TSA Method can save time that would otherwise be wasted with a misdirected investigation, and can also solve problems that may have remained unsolved.
The six states introduced earlier are inspired by the Unix process state transition diagram [Bach 86]. A similar thread state diagram is shown here, with the six states highlighted:
This can be applied to all application threads, or kernel threads of interest.
Here is the six state breakdown in detail, which should be applicable to most operating systems. This table includes suggestions for how to investigate each state:
Operating system kernels manage most of these states directly. This means there are code paths that can be instrumented – if they aren't already – providing state timing information for any thread.
In operating systems of today, there is usually no easy way to identify the Idle state. Applications threads wait for work in various ways, including waiting for a new network connection, network I/O, a lock, or a timer. From the kernel's perspective, the application thread is in the Sleeping or Lock state. After some analysis, you will see that the time is really Idle time.
In many cases, the Idle state can be skipped in analysis, even though it is the most common state. An application may have a performance issue while it is processing requests 10% of the time, and be Idle 90% of the time. For the TSA Method, the next most common states should be investigated, after Idle.
It may be tempting to add more states to narrow the initial investigation further. Here are some suggestions for further subdivision:
It's a little difficult to implement, but it is tempting to split out Memory wait from the Execution state. ie, memory stall cycles, leaving the Execution state as a measure of non-stalled instruction cycles.
Check what thread states your OS currently provides, how they can be used, and if any additional states are useful to include. For example, Mac OS X provides various thread states in Instruments (based on DTrace), including "Waiting" and "Waiting uninterruptibly".
In practic