Server Metrics & Counters¶
This reference lists all server metrics available through the Advanced Server Monitoring agent. These go far beyond basic CPU and memory: disk I/O saturation, memory paging, network packet loss, TCP connection failures. The metrics that tell you why the server is struggling, not just that it is.
How to Use This Reference¶
This page is organized into two sections:
- Metric Definitions: What each metric means and how to interpret it (CPU, Memory, Disk, Network groups)
- Counter Locations: Where the agent reads each metric from Windows Performance Monitor and Linux /proc files
Use the metric definitions to understand what you're seeing in your load test results. The counter locations section is for when you need to verify metric sources or troubleshoot agent configuration.
Note: Not all metrics are available on all operating systems. The agent automatically detects your platform and collects only the metrics your system supports.
For client-side metrics (response times, throughput, errors), see Load Test Metrics.
Metric Definitions¶
Below are all metrics collected by the Server Monitoring Agent in advanced mode. (Basic mode collects only CPU % and % Memory.)
Each metric includes: - What it measures: The resource or activity being monitored - What it tells you: How to interpret the values - When to investigate: Patterns that indicate potential bottlenecks
CPU group¶
CPU % (Processor Time for all processors) - the percentage of elapsed time that the processor spends executing non-Idle threads. This metric is the primary indicator of processor activity, and displays the average percentage of busy time observed during the sample interval. This metric is expected to increase proportionally to the load applied to the system.
Context switches/sec - the rate of switches from one thread to another. Thread switches can occur either inside of a single process or across processes. A thread switch can be caused either by one thread asking another for information, or by a thread being preempted by another, higher priority thread becoming ready to run. This metric is expected to increase proportionally to the load applied to the system.
Process Queue Length - the number of processes waiting to be scheduled to run. On Windows, a sustained processor queue of less than 10 threads per processor is normally acceptable, depending on the workload.
Memory group¶
\% Memory - The percentage of available memory that is in use. Large values of this metric suggest that the frequency of page swaps to disk will be high. For Windows servers, this is the percentage of virtual memory currently committed. For Linux servers, this is the percentage of physical memory in use (non-free, cached, or buffered).
Cache Memory Allocation - The amount of memory reserved by the Operating System for cache usage. Decreases in this value can be used as indicators that the Operating System requires memory for other purposes, which might not cause an immediate increase in memory usage.
Cache Memory Allocation Ratio - The percentage of physical memory reserved by the Operating System for cache usage. Decreases in this value can be used as indicators that the Operating System requires memory for other purposes, which might not cause an immediate increase in memory usage.
Page reads/sec - the rate at which the disk was read to resolve hard page faults. Hard page faults occur when a process references a page in virtual memory that is not in the working set or elsewhere in physical memory, and must be retrieved from disk. This counter is a primary indicator of the kinds of faults that cause system-wide delays. It includes read operations to satisfy faults in the file system cache (usually requested by applications) and in non-cached mapped memory files. Large increases in this metric can degrade system performance. Increasing physical memory can alleviate the problem.
Page Writes/sec - the rate at which pages are written to disk to free up space in physical memory. Pages are written to disk only if they are changed while in physical memory, so they are likely to hold data, not code. Large increases in this metric can degrade system performance. Increasing physical memory can alleviate the problem.
Disk group¶
\% I/O Time Utilized - The percentage of time during the sample interval that the disk was executing I/O.
Service time: The average amount of time required for each I/O transfer.
Reads/sec - the rate of read operations on the disk. A plateau in this metric could indicate a performance bottleneck.
Writes/sec - the rate of write operations on the disk. A plateau in this metric could indicate a performance bottleneck.
Queue Length - The average number of write requests that were queued for the disk during the sample interval.
Network group¶
Packets Received/sec - the rate at which packets are received on the network interfaces. This metric is expected to increase proportionally to the applied load. A greater-than-linear increase could indicate less efficient operation of the network. A less-than-linear increase indicates a limitation of network and/or server capacity.
Packets Sent/sec - the rate at which packets are sent on the network interfaces. This metric is expected to increase proportionally to the applied load. A greater-than-linear increase could indicate less efficient operation of the network. A less-than-linear increase indicates a limitation of network and/or server capacity.
Bytes received/sec - the rate at which data is received on the network interfaces. This metric is expected to increase proportionally to the applied load. A greater-than-linear increase could indicate less efficient operation of the network. A less-than-linear increase indicates a limitation of network and/or server capacity.
Bytes sent/sec - the rate at which data is sent on the network interfaces. This metric is expected to increase proportionally to the applied load. A greater-than-linear increase could indicate less efficient operation of the network. A less-than-linear increase indicates a limitation of network and/or server capacity.
Packets Received Errors - the number of inbound packets that contained errors preventing them from being deliverable to a higher-layer protocol. These errors are considered a serious network degradation.
Packets Sent Errors - the number of outbound packets that contained errors preventing them from being deliverable to a higher-layer protocol. These errors are considered a serious network degradation.
Collisions/sec - the rate at which outgoing ethernet packets must be re-transmitted. When this metric exceeds 5% of packets sent/sec, this indicates a network problem or network capacity limit.
Connections Established - the number of TCP connections for which the current state is either ESTABLISHED or CLOSE-WAIT. This metric is expected to increase proportionally to the load applied to the system.
Connection Failures - the number of times TCP connections have gone from SYN-SENT or SYN-RCVD to CLOSED.
TCP Segments Retransmitted - the number of TCP segments which were previously transmitted, but had to be retransmitted.
External Evaluation¶
Developers or advanced users who need to see sources that these metrics are measured from should refer to the Locating Server Metric Counters section below.
Locating Server Metric Counters¶
In some scenarios, it may be necessary to monitor server metrics outside of the Advanced Server Monitoring Agent. Many of these metrics can be replicated by following the reference table below.
For a description of the various metrics collected, see the Server Metrics sections above.
Windows¶
Most performance counters for Windows machines are also available to developers using Perfmon.
+--------------------------------------------------------------------------------+--------------------------------------------------------------------------------+ | ##### Counter Name {#counter-name style="font-size: 10pt;"} | ##### Perfmon Counter Path {#perfmon-counter-path style="font-size: 10pt;"} | +--------------------------------------------------------------------------------+--------------------------------------------------------------------------------+ | ###### Processor Counters {#processor-counters style="font-size: 10pt;"} | +--------------------------------------------------------------------------------+--------------------------------------------------------------------------------+ | CPU % | \Processor(_Total)\% Processor Time | +--------------------------------------------------------------------------------+--------------------------------------------------------------------------------+ | Context switches/sec | \System\Context Switches/sec | +--------------------------------------------------------------------------------+--------------------------------------------------------------------------------+ | Process Queue Length | \System\Processor Queue Length | +--------------------------------------------------------------------------------+--------------------------------------------------------------------------------+ | ###### Memory Counters {#memory-counters style="font-size: 10pt;"} | +--------------------------------------------------------------------------------+--------------------------------------------------------------------------------+ | Memory % | \Memory\% Committed Bytes In Use | +--------------------------------------------------------------------------------+--------------------------------------------------------------------------------+ | Page reads/sec | \Memory\Page Reads/sec | +--------------------------------------------------------------------------------+--------------------------------------------------------------------------------+ | [Page Writes/sec] | [\Memory\Page Writes/sec] | +--------------------------------------------------------------------------------+--------------------------------------------------------------------------------+ | [Cache Memory Allocation] | \Memory\Cache Bytes | +--------------------------------------------------------------------------------+--------------------------------------------------------------------------------+ | ###### Disk Counters {#disk-counters style="font-size: 10pt;"} | +--------------------------------------------------------------------------------+--------------------------------------------------------------------------------+ | \% I/O Time Utilized | \PhysicalDisk(_Total)\% Idle Time | | | | | | (Calculation taken as 100 - counter value) | +--------------------------------------------------------------------------------+--------------------------------------------------------------------------------+ | Service time | \PhysicalDisk(_Total)\Avg. Disk sec/Transfer | +--------------------------------------------------------------------------------+--------------------------------------------------------------------------------+ | Queue Length | \PhysicalDisk(_Total)\Current Disk Queue Length | +--------------------------------------------------------------------------------+--------------------------------------------------------------------------------+ | Reads/sec | \PhysicalDisk(_Total)\Disk Reads/sec | +--------------------------------------------------------------------------------+--------------------------------------------------------------------------------+ | Writes/sec | \PhysicalDisk(_Total)\Disk Writes/sec | +--------------------------------------------------------------------------------+--------------------------------------------------------------------------------+ | ###### Network Counters {#network-counters style="font-size: 10pt;"} | | | | Note: for values coming from a Network Interface, the Server Monitoring Agent will record the sum of all instances, excluding the local loop-back interface. | +--------------------------------------------------------------------------------+--------------------------------------------------------------------------------+ | Packets Received/sec | \Network Interface(interface name)\Packets Received/sec | +--------------------------------------------------------------------------------+--------------------------------------------------------------------------------+ | Packets Sent/sec | \Network Interface(interface name)\Packets Sent/sec | +--------------------------------------------------------------------------------+--------------------------------------------------------------------------------+ | Bytes received/sec | \Network Interface(interface name)\Bytes Received/sec | +--------------------------------------------------------------------------------+--------------------------------------------------------------------------------+ | Bytes sent/sec | \Network Interface(interface name)\Bytes Sent/sec | +--------------------------------------------------------------------------------+--------------------------------------------------------------------------------+ | Packets Received Errors | \Network Interface(interface name)\Packets Received Errors | +--------------------------------------------------------------------------------+--------------------------------------------------------------------------------+ | Packets Sent Errors | \Network Interface(interface name)\Packets Outbound Errors | +--------------------------------------------------------------------------------+--------------------------------------------------------------------------------+ | Connections Established | \TCP\Connections Established | +--------------------------------------------------------------------------------+--------------------------------------------------------------------------------+ | Connection Failures | \TCP\Connection Failures | +--------------------------------------------------------------------------------+--------------------------------------------------------------------------------+ | TCP Segments Retransmitted | \TCP\Segments Retransmitted/sec | +--------------------------------------------------------------------------------+--------------------------------------------------------------------------------+
Linux¶
Performance counters for Linux are calculated by examining the following values, if available. Some counters may be measured from multiple sources, in which case a source will be selected which appears applicable to the current kernel.
+---------------------------------------------------------------+------------------------------------------------------+----------------------------------------------------------------------------------+ | ##### Counter Name {#counter-name-1 style="font-size: 10pt;"} | ##### File {#file style="font-size: 10pt;"} | ##### Relevant lines, columns {#relevant-lines-columns style="font-size: 10pt;"} | +---------------------------------------------------------------+------------------------------------------------------+----------------------------------------------------------------------------------+ | ###### Processor Counters {#processor-counters-1 style="font-size: 10pt;"} | +---------------------------------------------------------------+------------------------------------------------------+----------------------------------------------------------------------------------+ | CPU % | /proc/stat | Line: cpu | | | | | | | | 4th numeric column is idle time | +---------------------------------------------------------------+------------------------------------------------------+----------------------------------------------------------------------------------+ | Context switches/sec | /proc/stat | Line: ctxt | | | | | | | | | +---------------------------------------------------------------+------------------------------------------------------+----------------------------------------------------------------------------------+ | Process Queue Length | /proc/stat | Line: procs_running | | | | | | | | | +---------------------------------------------------------------+------------------------------------------------------+----------------------------------------------------------------------------------+ | ###### Memory Counters {#memory-counters-1 style="font-size: 10pt;"} | +---------------------------------------------------------------+------------------------------------------------------+----------------------------------------------------------------------------------+ | Memory % | /proc/meminfo | Lines: MemTotal, MemFree, Buffers, Cached | +---------------------------------------------------------------+------------------------------------------------------+----------------------------------------------------------------------------------+ | Cache Memory Allocation Ratio | /proc/meminfo | Lines: MemTotal, Cached, SwapCached | +---------------------------------------------------------------+------------------------------------------------------+----------------------------------------------------------------------------------+ | Major Page Faults | /proc/vmstat | Line: pgmajfault | +---------------------------------------------------------------+------------------------------------------------------+----------------------------------------------------------------------------------+ | Page ins/sec | /proc/stat | Line: page, Column: 1 | | +------------------------------------------------------+----------------------------------------------------------------------------------+ | | | /proc/vmstat | +---------------------------------------------------------------+------------------------------------------------------+----------------------------------------------------------------------------------+ | Page outs/sec | /proc/stat | Line: page, Column: 2 | | +------------------------------------------------------+----------------------------------------------------------------------------------+ | | | /proc/vmstat | +---------------------------------------------------------------+------------------------------------------------------+----------------------------------------------------------------------------------+ | ###### Disk Counters {#disk-counters-1 style="font-size: 10pt;"} | | | | Note: Counters are measured by summing values from all physical disks, excluding logical disks | +---------------------------------------------------------------+------------------------------------------------------+----------------------------------------------------------------------------------+ | \% I/O Time Utilized | /proc/partitions | Column: 14 | | +------------------------------------------------------+----------------------------------------------------------------------------------+ | | | /proc/diskstats | +---------------------------------------------------------------+------------------------------------------------------+----------------------------------------------------------------------------------+ | Average Service time | /proc/partitions | Columns: 5, 9, 13, 15 | | +------------------------------------------------------+----------------------------------------------------------------------------------+ | | | /proc/diskstats | +---------------------------------------------------------------+------------------------------------------------------+----------------------------------------------------------------------------------+ | Queue Length | /proc/partitions | Column: 13 | | +------------------------------------------------------+----------------------------------------------------------------------------------+ | | | /proc/diskstats | +---------------------------------------------------------------+------------------------------------------------------+----------------------------------------------------------------------------------+ | Reads/sec | /proc/partitions | Column: 5 | | +------------------------------------------------------+----------------------------------------------------------------------------------+ | | | /proc/diskstats | +---------------------------------------------------------------+------------------------------------------------------+----------------------------------------------------------------------------------+ | Writes/sec | /proc/partitions | Column: 9 | | +------------------------------------------------------+----------------------------------------------------------------------------------+ | | | /proc/diskstats | +---------------------------------------------------------------+------------------------------------------------------+----------------------------------------------------------------------------------+ | ###### Network Counters {#network-counters-1 style="font-size: 10pt;"} | | | | Note: for values coming from a Network Interface, the Server Monitoring Agent will record the sum of all instances, excluding the local loop-back interface. | +---------------------------------------------------------------+------------------------------------------------------+----------------------------------------------------------------------------------+ | Packets Received/sec | /proc/net/dev | Column: 3 | +---------------------------------------------------------------+------------------------------------------------------+----------------------------------------------------------------------------------+ | Packets Sent/sec | /proc/net/dev | Column: 11 | +---------------------------------------------------------------+------------------------------------------------------+----------------------------------------------------------------------------------+ | Bytes received/sec | /proc/net/dev | Column: 2 | +---------------------------------------------------------------+------------------------------------------------------+----------------------------------------------------------------------------------+ | Bytes sent/sec | /proc/net/dev | Column: 10 | +---------------------------------------------------------------+------------------------------------------------------+----------------------------------------------------------------------------------+ | Packets Received Errors | /proc/net/dev | Column: 4 | +---------------------------------------------------------------+------------------------------------------------------+----------------------------------------------------------------------------------+ | Packets Sent Errors | /proc/net/dev | Column: 12 | +---------------------------------------------------------------+------------------------------------------------------+----------------------------------------------------------------------------------+ | Collisions/sec | /proc/net/dev | Column: 15 | +---------------------------------------------------------------+------------------------------------------------------+----------------------------------------------------------------------------------+ | Connections Established | /proc/net/snmp | Line: Tcp, Column 9 | +---------------------------------------------------------------+------------------------------------------------------+----------------------------------------------------------------------------------+ | Connection Failures | /proc/net/snmp | Line: Tcp, Column 8 | +---------------------------------------------------------------+------------------------------------------------------+----------------------------------------------------------------------------------+ | TCP Segments Retransmitted | /proc/net/snmp | Line: Tcp, Column 12 | +---------------------------------------------------------------+------------------------------------------------------+----------------------------------------------------------------------------------+