Help can be accessed in several ways:

Almost all output of the help subcommand (do not mix it up with the --help option) opens a text viewer. This text viewer allows you to scroll up or down and read the help text more comfortably. To leave the text viewer, press the Q key.

Tip: Use Tab Completion in Bash and Interactive Shell

The crmsh supports full tab completion in Bash directly, not only for the interactive shell. For example, typing crm help config→| will complete the word just like in the interactive shell.

The crm command itself can be used in the following ways:

In addition to previously explained methods, crmsh also supports synchronous command execution. Use the -w option to activate it. If you have started crm without -w, you can enable it later with the user preference's wait set to yes (options wait yes). If this option is enabled, crm waits until the transition is finished. Whenever a transaction is started, dots are printed to indicate progress. Synchronous command execution is only applicable for commands like resource start.

Note: Differentiate Between Management and Configuration Subcommands

The crm tool has management capability (the subcommands resource and node) and can be used for configuration (cib, configure).

The following subsections give you an overview about some important aspects of the crm tool.

As you have to deal with resource agents in your cluster configuration all the time, the crm tool contains the ra command to get information about resource agents and to manage them (for additional information, see also Section 4.2.2, “Supported Resource Agent Classes”):

root # crm ra
crm(live)ra# 

The command classes gives you a list of all classes and providers:

crm(live)ra# classes
 lsb
 ocf / heartbeat linbit lvm2 ocfs2 pacemaker
 service
 stonith
 systemd
 

To get an overview about all available resource agents for a class (and provider) use the list command:

crm(live)ra# list ocf
AoEtarget           AudibleAlarm        CTDB                ClusterMon
Delay               Dummy               EvmsSCC             Evmsd
Filesystem          HealthCPU           HealthSMART         ICP
IPaddr              IPaddr2             IPsrcaddr           IPv6addr
LVM                 LinuxSCSI           MailTo              ManageRAID
ManageVE            Pure-FTPd           Raid1               Route
SAPDatabase         SAPInstance         SendArp             ServeRAID
...

An overview about a resource agent can be viewed with info:

crm(live)ra# info ocf:drbd:linbit
This resource agent manages a DRBD* resource
as a master/slave resource. DRBD is a shared-nothing replicated storage
device. (ocf:linbit:drbd)

Master/Slave OCF Resource Agent for DRBD

Parameters (* denotes required, [] the default):

drbd_resource* (string): drbd resource name
    The name of the drbd resource from the drbd.conf file.

drbdconf (string, [/etc/drbd.conf]): Path to drbd.conf
    Full path to the drbd.conf file.

Operations' defaults (advisory minimum):

    start         timeout=240
    promote       timeout=90 
    demote        timeout=90 
    notify        timeout=90 
    stop          timeout=100
    monitor_Slave_0 interval=20 timeout=20 start-delay=1m
    monitor_Master_0 interval=10 timeout=20 start-delay=1m

Leave the viewer by pressing Q.

Tip: Use crm Directly

In the former example we used the internal shell of the crm command. However, you do not necessarily have to use it. You get the same results, if you add the respective subcommands to crm. For example, you can list all the OCF resource agents by entering crm ra list ocf in your shell.

Configuration templates are ready-made cluster configurations for crmsh. Do not confuse them with the resource templates (as described in Section 6.4.2, “Creating Resource Templates”). Those are templates for the cluster and not for the crm shell.

Configuration templates require minimum effort to be tailored to the particular user's needs. Whenever a template creates a configuration, warning messages give hints which can be edited later for further customization.

The following procedure shows how to create a simple yet functional Apache configuration:

It is possible to simplify the commands even more, if you know the details. The above procedure can be summarized with the following command on the shell:

root # crm configure template \
   new g-intranet apache params \
   configfile="/etc/apache2/httpd.conf" ip="192.168.1.101"

If you are inside your internal crm shell, use the following command:

crm(live)configure template# new intranet apache params \
   configfile="/etc/apache2/httpd.conf" ip="192.168.1.101"

However, the previous command only creates its configuration from the configuration template. It does not apply nor commit it to the CIB.

A shadow configuration is used to test different configuration scenarios. If you have created several shadow configurations, you can test them one by one to see the effects of your changes.

The usual process looks like this:

Before loading your configuration changes back into the cluster, it is recommended to review your changes with ptest. The ptest command can show a diagram of actions that will be induced by committing the changes. You need the graphviz package to display the diagrams. The following example is a transcript, adding a monitor operation:

root # crm configure
crm(live)configure# show fence-bob 
primitive fence-bob stonith:apcsmart \
        params hostlist="bob"
crm(live)configure# monitor fence-bob 120m:60s
crm(live)configure# show changed
primitive fence-bob stonith:apcsmart \
        params hostlist="bob" \
        op monitor interval="120m" timeout="60s"
crm(live)configure# ptest
crm(live)configure# commit

To output a cluster diagram as shown in Figure 5.2, “Hawk—Cluster Diagram”, use the command crm configure graph. It displays the current configuration on its current window, therefore requiring X11.

If you prefer Scalable Vector Graphics (SVG), use the following command:

root # crm configure graph dot config.svg svg

There are three types of RAs (Resource Agents) available with the cluster (for background information, see Section 4.2.2, “Supported Resource Agent Classes”). To add a new resource to the cluster, proceed as follows:

If you want to create several resources with similar configurations, a resource template simplifies the task. See also Section 4.4.3, “Resource Templates and Constraints” for some basic background information. Do not confuse them with the “normal” templates from Section 6.1.4, “Using Configuration Templates”. Use the rsc_template command to get familiar with the syntax:

root # crm configure rsc_template
usage: rsc_template <name> [<class>:[<provider>:]]<type>
        [params <param>=<value> [<param>=<value>...]]
        [meta <attribute>=<value> [<attribute>=<value>...]]
        [utilization <attribute>=<value> [<attribute>=<value>...]]
        [operations id_spec
            [op op_type [<attribute>=<value>...] ...]]

For example, the following command creates a new resource template with the name BigVM derived from the ocf:heartbeat:Xen resource and some default values and operations:

crm(live)configure# rsc_template BigVM ocf:heartbeat:Xen \
   params allow_mem_management="true" \
   op monitor timeout=60s interval=15s \
   op stop timeout=10m \
   op start timeout=10m

Once you defined the new resource template, you can use it in primitives or reference it in order, colocation, or rsc_ticket constraints. To reference the resource template, use the @ sign:

crm(live)configure# primitive MyVM1 @BigVM \
   params xmfile="/etc/xen/shared-vm/MyVM1" name="MyVM1"

The new primitive MyVM1 is going to inherit everything from the BigVM resource templates. For example, the equivalent of the above two would be:

crm(live)configure# primitive MyVM1 ocf:heartbeat:Xen \
   params xmfile="/etc/xen/shared-vm/MyVM1" name="MyVM1"
   params allow_mem_management="true" \
   op monitor timeout=60s interval=15s \
   op stop timeout=10m \
   op start timeout=10m

If you want to overwrite some options or operations, add them to your (primitive) definition. For instance, the following new primitive MyVM2 doubles the timeout for monitor operations but leaves others untouched:

crm(live)configure# primitive MyVM2 @BigVM \
   params xmfile="/etc/xen/shared-vm/MyVM2" name="MyVM2" \
   op monitor timeout=120s interval=30s    

A resource template may be referenced in constraints to stand for all primitives which are derived from that template. This helps to produce a more concise and clear cluster configuration. Resource template references are allowed in all constraints except location constraints. Colocation constraints may not contain more than one template reference.

From the crm perspective, a STONITH device is just another resource. To create a STONITH resource, proceed as follows:

Having all the resources configured is only one part of the job. Even if the cluster knows all needed resources, it might still not be able to handle them correctly. For example, try not to mount the file system on the slave node of DRBD (in fact, this would fail with DRBD). Define constraints to make these kind of information available to the cluster.

For more information about constraints, see Section 4.4, “Resource Constraints”.

crm(live)configure# location loc-fs1 fs1 100: alice

crm(live)configure# primitive pingd pingd \
    params name=pingd dampen=5s multiplier=100 host_list="r1 r2"
crm(live)configure# location loc-node_pref internal_www \
    rule 50: #uname eq alice \
    rule pingd: defined pingd

crm(live)configure# primitive vip1 ocf:heartbeat:IPaddr2 params ip=192.168.1.5
crm(live)configure# primitive vip1 ocf:heartbeat:IPaddr2 params ip=192.168.1.6
crm(live)configure# location loc-alice { vip1 vip2 } inf: alice 

crm(live)configure# location  loc-alice /vip.*/ inf: alice

crm(live)configure# colocation nfs_on_filesystem inf: nfs_group filesystem_resource

root # crm configure assist weak-bond RES1 RES2

crm(live)configure# order nfs_after_filesystem mandatory: filesystem_resource nfs_group

To determine a resource failover, use the meta attribute migration-threshold. In case failcount exceeds migration-threshold on all nodes, the resource will remain stopped. For example:

crm(live)configure# location rsc1-alice rsc1 100: alice

Normally, rsc1 prefers to run on alice. If it fails there, migration-threshold is checked and compared to the failcount. If failcount >= migration-threshold then it is migrated to the node with the next best preference.

Start failures set the failcount to inf depend on the start-failure-is-fatal option. Stop failures cause fencing. If there is no STONITH defined, the resource will not migrate at all.

For an overview, refer to Section 4.4.4, “Failover Nodes”.

A resource might fail back to its original node when that node is back online and in the cluster. If you want to prevent a resource from failing back to the node it was running on prior to failover, or if you want to specify a different node for the resource to fail back to, you must change its resource stickiness value. You can either specify resource stickiness when you are creating a resource, or afterwards.

For an overview, refer to Section 4.4.5, “Failback Nodes”.

Some resources may have specific capacity requirements such as minimum amount of memory. Otherwise, they may fail to start completely or run with degraded performance.

To take this into account, the High Availability Extension allows you to specify the following parameters:

For detailed background information about the parameters and a configuration example, refer to Section 4.4.6, “Placing Resources Based on Their Load Impact”.

To configure the resource's requirements and the capacity a node provides, use utilization attributes. You can name the utilization attributes according to your preferences and define as many name/value pairs as your configuration needs. In certain cases, some agents update the utilization themselves, for example the VirtualDomain.

In the following example, we assume that you already have a basic configuration of cluster nodes and resources and now additionally want to configure the capacities a certain node provides and the capacity a certain resource requires.

The following example demonstrates a three node cluster of equal nodes, with 4 virtual machines:

crm(live)configure# node alice utilization memory="4000"
crm(live)configure# node bob utilization memory="4000"
crm(live)configure# node charly utilization memory="4000"
crm(live)configure# primitive xenA ocf:heartbeat:Xen \
    utilization memory="3500" meta priority="10"
crm(live)configure# primitive xenB ocf:heartbeat:Xen \
    utilization memory="2000" meta priority="1"
crm(live)configure# primitive xenC ocf:heartbeat:Xen \
    utilization memory="2000" meta priority="1"
crm(live)configure# primitive xenD ocf:heartbeat:Xen \
    utilization memory="1000" meta priority="5"
crm(live)configure# property placement-strategy="minimal"

With all three nodes up, xenA will be placed onto a node first, followed by xenD. xenB and xenC would either be allocated together or one of them with xenD.

If one node failed, too little total memory would be available to host them all. xenA would be ensured to be allocated, as would xenD; however, only one of xenB or xenC could still be placed, and since their priority is equal, the result is not defined yet. To resolve this ambiguity as well, you would need to set a higher priority for either one.

To monitor a resource, there are two possibilities: either define a monitor operation with the op keyword or use the monitor command. The following example configures an Apache resource and monitors it every 60 seconds with the op keyword:

crm(live)configure# primitive apache apache \
  params ... \
  op monitor interval=60s timeout=30s

The same can be done with:

crm(live)configure# primitive apache apache \
   params ...
crm(live)configure# monitor apache 60s:30s

For an overview, refer to Section 4.3, “Resource Monitoring”.

One of the most common elements of a cluster is a set of resources that needs to be located together. Start sequentially and stop in the reverse order. To simplify this configuration we support the concept of groups. The following example creates two primitives (an IP address and an e-mail resource):

To change the order of a group member, use the modgroup command from the configure subcommand. Use the following commands to move the primitive Email before Public-IP. (This is just to demonstrate the feature):

crm(live)configure# modgroup g-shortcut add p.lsb-exim before p.public-ip

In case you want to remove a resource from a group (for example, Email), use this command:

crm(live)configure# modgroup g-shortcut remove p.lsb-exim

For an overview, refer to Section 4.2.5.1, “Groups”.

Clones were initially conceived as a convenient way to start N instances of an IP resource and have them distributed throughout the cluster for load balancing. They have turned out to quite useful for a number of other purposes, including integrating with DLM, the fencing subsystem and OCFS2. You can clone any resource, provided the resource agent supports it.

Learn more about cloned resources in Section 4.2.5.2, “Clones”.

To start a new cluster resource you need the respective identifier. Proceed as follows:

A resource will be automatically restarted if it fails, but each failure raises the resource's failcount. If a migration-threshold has been set for that resource, the node will no longer be allowed to run the resource as soon as the number of failures has reached the migration threshold.

Proceed as follows to remove a cluster resource:

Although resources are configured to automatically fail over (or migrate) to other nodes of the cluster in the event of a hardware or software failure, you can also manually move a resource to another node in the cluster using either Hawk or the command line.

Use the migrate command for this task. For example, to migrate the resource ipaddress1 to a cluster node named bob, use these commands:

root # crm resource
crm(live)resource# migrate ipaddress1 bob

Tags are a way to refer to multiple resources at once, without creating any colocation or ordering relationship between them. This can be useful for grouping conceptually related resources. For example, if you have a number of resources related to a database, create a tag called tag-db and add all resources related to the database to this tag:

root # crm configure tag-db db1 db2 db3

This allows you to start them all with a single command:

root # crm resource start tag-db

Similarly, you can stop them all too:

root # crm resource stop tag-db

Every now and then, you will need to perform testing or maintenance tasks on individual cluster components or the whole cluster—be it changing the cluster configuration, updating software packages for individual nodes, or upgrading the cluster to a higher product version.

With regards to that, High Availability Extension provides maintenance options on several levels:

Warning: Risk of Data Loss

If you need to execute any testing or maintenance tasks while services are running under cluster control, make sure to follow this outline:

1. Before you start, set the individual resource, node or the whole cluster to maintenance mode. This helps to avoid unwanted side effects like resources not starting in an orderly fashion, the risk of unsynchronized CIBs across the cluster nodes or data loss.

2. Execute your maintenance task or tests.

3. After you have finished, remove the maintenance mode to start normal cluster operation.

For more details on what happens to the resources and the cluster while being in maintenance mode, see Section 4.7, “Maintenance Mode”.

The “health” status of a cluster or node can be displayed with so called scripts. A script can perform different tasks, they are not targeted to health at all. However, for this subsection, we focus on how getting the health status.

To get all the details about the health command, use describe:

root # crm script describe health

It shows a description and a list of all parameters and their default values. To execute a script, use run:

root # crm script run health verbose=true

If you prefer to run only one step from the suite, the describe command shows a list of all available steps in the Steps category.

For example, the following command executes the first step of the health command. The output is stored in the health.json file for further investigation:

root # crm script run health \
    step='Collect cluster information' \
    statefile='health.json'

For additional information regarding scripts, see http://crmsh.github.io/scripts/.