Modern processors have several power saving modes called C-states. They reflect the capability of an idle processor to turn off unused components in order to save power. Whereas C-states have been available for laptops for some time, they are a more recent trend in the server market. For example, Intel* server processors have supported C-modes starting with the Nehalem architecture.

When a processor is in the C0 state, it is executing instructions. A processor running in any other C-state is idle. The higher the C number, the deeper the CPU sleep mode: more components are shut down to save power. Deeper sleep states can save large amounts of energy. Their downside is that they introduce latency. This means, it takes more time for the CPU to go back to C0). Depending on workload (threads waking up, triggering CPU usage and then going back to sleep again for a short period of time) and hardware (for example, interrupt activity of a network device), disabling the deepest sleep states can significantly increase overall performance. For details on how to do so, refer to Section 10.3.2.2, “Viewing and Modifying Kernel Idle Statistics with cpupower”.

Some states also have submodes with different power saving latency levels. Which C-states and submodes are supported depends on the respective processor. However, C1 is always available.

Table 10.1, “C-States” gives an overview of the most common C-states.

To avoid needless power consumption, it is recommended to test your workloads with deep sleep states enabled versus deep sleep states disabled. For more information, refer to Section 10.3.2.2, “Viewing and Modifying Kernel Idle Statistics with cpupower” or the cpupower-idle-set(1) man page.

While a processor operates (in C0 state), it can be in one of several CPU performance states (P-states). Whereas C-states are idle states (all but C0), P-states are operational states that relate to CPU frequency and voltage.

The higher the P-state, the lower the frequency and voltage at which the processor runs. The number of P-states is processor-specific and the implementation differs across the various types. However, P0 is always the highest-performance state. Higher P-state numbers represent slower processor speeds and lower power consumption. For example, a processor in P3 state run more slowly and uses less power than a processor running in the P1 state. To operate at any P-state, the processor must be in the C0 state, which means that it is working and not idling. The CPU P-states are also defined in the ACPI specification, see http://www.acpi.info/spec.htm.

C-states and P-states can vary independently of one another.

T-states refer to throttling the processor clock to lower frequencies in order to reduce thermal effects. This means that the CPU is forced to be idle a fixed percentage of its cycles per second. Throttling states range from T1 (the CPU has no forced idle cycles) to Tn, with the percentage of idle cycles increasing the greater n is.

However, throttling does not reduce voltage. Also, since the CPU is forced to idle part of the time, processes will take longer to finish and will consume more power.

T-states are only useful if reducing thermal effects is the primary goal. Since T-states can interfere with C-states (preventing the CPU from reaching higher C-states), they can even increase power consumption in a modern CPU capable of C-states.

Modern processors use a combination of different means to achieve a balance between performance and power savings: deep sleep states, traditional dynamic frequency scaling and hidden boost frequencies. Newer AMD* and Intel* CPUs have turbo features (Turbo CORE* and Turbo Boost*, respectively). Turbo features allow dynamically increasing the clock speed of active CPU cores while other cores are in deep sleep states. This increases the performance of active threads while still complying with Thermal Design Power (TDP) limits.

However, the conditions under which a CPU core can use turbo frequencies are architecture-specific. Learn how to evaluate the efficiency of those new features in Section 10.3.2, “Using the cpupower Tools”.

You can think of the in-kernel governors as a sort of preconfigured power scheme for the CPU. The CPUfreq governors use P-states to change frequencies and lower power consumption. The dynamic governors can switch between CPU frequencies, based on CPU usage, to allow for power savings while not sacrificing performance. These governors also allow for some tuning so you can customize and change the frequency scaling behavior.

The following governors are available with the CPUfreq subsystem:

If the CPUfreq subsystem in enabled on your system (which it is by default with SUSE Linux Enterprise Server), you can find the relevant files and directories under /sys/devices/system/cpu/. If you list the contents of this directory, you will find a cpu{0..x} subdirectory for each processor, and several other files and directories. A cpufreq subdirectory in each processor directory holds a number of files and directories that define the parameters for CPUfreq. Some of them are writable (for root), some of them are read-only. If your system currently uses the on-demand or conservative governor, you will see a separate subdirectory for those governors in cpufreq, containing the parameters for the governors.

Note: Different Processor Settings

The settings under the cpufreq directory can be different for each processor. If you want to use the same policies across all processors, you need to adjust the parameters for each processor. Instead of looking up or modifying the current settings manually (in /sys/devices/system/cpu*/cpufreq), we advise to use the tools provided by the cpupower package or by the older cpufrequtils package for that.

Note: cpupower and cpufrequtils

All functions of cpufrequtils are also covered by cpupower—a new set of tools that is more powerful and provides additional features. As cpupower will replace cpufrequtils sooner or later, we advise to switch to cpupower soon and to adjust your scripts accordingly.

After you have installed the cpufrequtils package, you can make use of the cpufreq-info and cpufreq-set command line tools.

cpufrequtils 004: cpufreq-info (C) Dominik Brodowski 2004-2006
Report errors and bugs to http://bugs.opensuse.org, please.
analyzing CPU 0:
  driver: acpi-cpufreq
  CPUs which need to switch frequency at the same time: 0
  hardware limits: 2.80 GHz - 3.40 GHz
  available frequency steps: 3.40 GHz, 2.80 GHz
  available cpufreq governors: conservative, userspace, powersave, ondemand, performance
  current policy: frequency should be within 2.80 GHz and 3.40 GHz.
                  The governor "performance" may decide which speed to use
                  within this range.
  current CPU frequency is 3.40 GHz.
analyzing CPU 1:
 driver: acpi-cpufreq
 CPUs which need to switch frequency at the same time: 1
 hardware limits: 2.80 GHz - 3.40 GHz
 available frequency steps: 3.40 GHz, 2.80 GHz
 available cpufreq governors: conservative, userspace, powersave, ondemand, performance
 current policy: frequency should be within 2.80 GHz and 3.40 GHz.
                 The governor "performance" may decide which speed to use
                 within this range.
current CPU frequency is 3.40 GHz.
    

After installing the cpupower package, view the available cpupower subcommands with cpupower --help. Access the general man page with man cpupower, and the man pages of the subcommands with man cpupower-subcommand.

The subcommands frequency-info and frequency-set are mostly equivalent to cpufreq-info and cpufreq-set, respectively. However, they provide extended output and there are small differences in syntax and behavior:

analyzing CPU 0:
  driver: acpi-cpufreq
  CPUs which run at the same hardware frequency: 0 1 2 3
  CPUs which need to have their frequency coordinated by software: 0
  maximum transition latency: 10.0 us.
  hardware limits: 2.00 GHz - 2.83 GHz
  available frequency steps: 2.83 GHz, 2.34 GHz, 2.00 GHz
  available cpufreq governors: conservative, userspace, powersave, ondemand, performance
  current policy: frequency should be within 2.00 GHz and 2.83 GHz.
                  The governor "ondemand" may decide which speed to use
                  within this range.
  current CPU frequency is 2.00 GHz (asserted by call to hardware).
  boost state support:
    Supported: yes
    Active: yes
    

CPUidle driver: acpi_idle
CPUidle governor: menu

Analyzing CPU 0:
Number of idle states: 3
Available idle states: C1 C2
C1:
Flags/Description: ACPI FFH INTEL MWAIT 0x0
Latency: 1
Usage: 3156464
Duration: 233680359
C2:
Flags/Description: ACPI FFH INTEL MWAIT 0x10
Latency: 1
Usage: 273007117
Duration: 103148860538

cpupower idle-set -d 4

|Mperf               || Idle_Stats
 1                      2 
CPU | C0   | Cx   | Freq || POLL | C1   | C2   | C3
   0|  3.71| 96.29|  2833||  0.00|  0.00|  0.02| 96.32
   1| 100.0| -0.00|  2833||  0.00|  0.00|  0.00|  0.00
   2|  9.06| 90.94|  1983||  0.00|  7.69|  6.98| 76.45
   3|  7.43| 92.57|  2039||  0.00|  2.60| 12.62| 77.52
     

Another useful tool for monitoring system power consumption is powerTOP. It helps you to identify the reasons for unnecessary high power consumption (for example, processes that are mainly responsible for waking up a processor from its idle state) and to optimize your system settings to avoid these. It supports both Intel and AMD processors. The powertop package is available from the SUSE Linux Enterprise SDK. For information on how to access the SDK, refer to About This Guide.

powerTOP combines various sources of information (analysis of programs, device drivers, kernel options, amounts and sources of interrupts waking up processors from sleep states) and shows them in one screen. Example 10.5, “Example powerTOP Output” shows which information categories are available:

Cn               Avg  residency       P-states   (frequencies)
1                 2      3              4            5
C0 (cpu running)        (11.6%)       2.00 Ghz       0.1%
polling         0.0ms   ( 0.0%)       2.00 Ghz       0.0%
C1              4.4ms   (57.3%)       1.87 Ghz       0.0%
C2             10.0ms   (31.1%)       1064 Mhz      99.9%


Wakeups-from-idle per second : 11.2     interval: 5.0s 6
no ACPI power usage estimate available 7



Top causes for wakeups: 8
96.2% (826.0)       <interrupt> : extra timer interrupt
 0.9% (  8.0)     <kernel core> : usb_hcd_poll_rh_status (rh_timer_func)
 0.3% (  2.4)       <interrupt> : megasas
 0.2% (  2.0)     <kernel core> : clocksource_watchdog (clocksource_watchdog)
 0.2% (  1.6)       <interrupt> : eth1-TxRx-0
 0.1% (  1.0)       <interrupt> : eth1-TxRx-4

[...]

Suggestion: 9 Enable SATA ALPM link power management via:
echo min_power > /sys/class/scsi_host/host0/link_power_management_policy
or press the S key.

For more information, refer to the powerTOP project page at https://01.org/powertop.

The CPUfreq subsystem offers several tuning options for P-states: You can switch between the different governors, influence minimum or maximum CPU frequency to be used or change individual governor parameters.

To switch to another governor at runtime, use cpupower frequency-set (or cpufreq-set) with the -g option. For example, running the following command (as root) will activate the on-demand governor:

cpupower frequency-set -g ondemand

If you want the change in governor to persist also after a reboot or shutdown, use tuned described in man 8 tuned.

To set values for the minimum or maximum CPU frequency the governor may select, use the -d or -u option, respectively.

Apart from the governor settings that can be influenced with cpupower or cpufreq*, you can also tune further governor parameters manually, for example, Ignoring Nice Values in Processor Usage.

Tip: Using the Same Value for All Cores

When setting the ignore_nice_load value for cpu0, the same value is automatically used for all cores. In this case, you do not need to repeat the steps above for each of the processors where you want to modify this governor parameter.

Another parameter that significantly impacts the performance loss caused by dynamic frequency scaling is the sampling rate (rate at which the governor checks the current CPU load and adjusts the processor's frequency accordingly). Its default value depends on a BIOS value and it should be as low as possible. However, on modern systems, an appropriate sampling rate is set without manual intervention.

By default, SUSE Linux Enterprise Desktop uses C-states appropriately. The only parameter that is relevant for optimization is the sched_mc_power_savings scheduler. Instead of distributing a workload across all cores with the effect that all cores are used only at a minimum level, the kernel can try to schedule processes on as few cores as possible, so that all others can go idle. This helps to save power, as it allows some processors to be idle for a longer time so they can reach a higher C-state. However, the actual savings depend on a number of factors. For example, how many processors are available and which C-states they support (especially deeper ones such as C3 to C6).

If sched_mc_power_savings is set to 0 (default value), no special scheduling is done. If it is set to 1, the scheduler tries to consolidate the work onto the fewest number of processors possible in the case that all processors are a little busy. To modify this parameter, proceed as follows: