Quick Start

1. To install the packages from both patterns via command line, use zypper:

   sudo zypper in -t pattern ha_sles ha_geo

2. Alternatively, use YaST for a graphical installation:

   1. Start YaST as root user and select Software › Software Management.

   2. Click View › Patterns and activate the following patterns:

      • High Availability

      • GEO Clustering for High Availability

   3. Click Accept to start installing the packages.

1. Make sure that GEO Clustering for SUSE Linux Enterprise High Availability Extension has been added as add-on product to the machines to serve as arbitrators.

2. Log in to each arbitrator and install the packages with the following command:

   sudo zypper in -t pattern ha_geo

   Alternatively, use YaST to install the GEO Clustering for High Availability pattern.

1. Log in to a cluster node as root or equivalent.

2. Copy the example booth configuration file /etc/booth/booth.conf.example to /etc/booth/booth.conf.

3. Edit /etc/booth/booth.conf according to Example 1, “A Booth Configuration File”.

4. Verify your changes and save the file.

5. On all cluster nodes and arbitrators, open the port in the firewall that you have configured for booth. See Example 1, “A Booth Configuration File”, position 2.

1. Log in to a cluster node as root or equivalent.

2. Start the YaST Geo Cluster module.

3. Choose to Edit an existing booth configuration file or click Add to create a new booth configuration file:

   1. In the screen that appears configure the following parameters:

      • Configuration File. A name for the booth configuration file. YaST suggests booth by default. This results in the booth configuration being written to /etc/booth/booth.conf. Only change this value if you need to set up multiple booth instances for different GEO clusters as described in Section 6.2, “Booth Setup for Multiple Tenants”.

      • Transport. The transport protocol used for communication between the sites. Only UDP is supported, but other transport layers will follow in the future. See also Example 1, “A Booth Configuration File”, position 1.

      • Port. The port to be used for communication between the booth instances at each site. See also Example 1, “A Booth Configuration File”, position 2.

      • Arbitrator.  The IP address of the machine to use as arbitrator. See also Example 1, “A Booth Configuration File”, position 3.

        To specify an Arbitrator, click Add. In the dialog that opens, enter the IP address of your arbitrator and click OK.

      • Site. The IP address used for the boothd on a site. See also Example 1, “A Booth Configuration File”, position 4.

        To specify a Site of your GEO cluster, click Add. In the dialog that opens, enter the IP address of one site and click OK.

      • Ticket. The ticket to be managed by booth. See also Example 1, “A Booth Configuration File”, position 5.

        To specify a Ticket, click Add. In the dialog that opens, enter a unique Ticket name. If you need to define multiple tickets with the same parameters and values, you can save configuration effort by creating a “ticket template” which specifies the defaults parameters and values for all tickets. To do so, enter __default__ as Ticket name.

        Additionally, you can specify optional parameters for your ticket. For an overview, see Example 1, “A Booth Configuration File”, positions 6 to 13.

        Click OK to confirm your changes.

      

      Figure 2: Example Ticket Dependency #

   2. Click OK to close the current booth configuration screen. YaST shows the name of the booth configuration file that you just defined.

4. Before closing the YaST module, switch to the Firewall Configuration category.

5. To open the port you have configured for booth, enable Open Port in Firewall.

   

   Important: Firewall Setting for Local Machine Only

   The firewall setting is only applied to the current machine. It will open the UDP/TCP ports for all ports that have been specified in /etc/booth/booth.conf or any other booth configuration files (see Section 6.2, “Booth Setup for Multiple Tenants”).

   Make sure to open the respective ports on all other cluster nodes and arbitrators of your GEO cluster setup, too. Do so either manually or by synchronizing the following files with Csync2:

   • /etc/sysconfig/SuSEfirewall2

   • /etc/sysconfig/SuSEfirewall2.d/services/booth

6. Click Finish to confirm all settings and close the YaST module. Depending on the NAME of the Configuration File specified in Step 3.a, the configuration is written to /etc/booth/NAME.conf.

1. Create different booth configuration files in /etc/booth as shown in Example 2, “ /etc/booth/apac.conf ” and Example 3, “ /etc/booth/emea.conf ”. Do so either manually or with YaST, as outlined in Procedure 2, “Setting Up Booth with YaST”.

2. On the arbitrator, open the ports that are defined in any of the booth configuration files in /etc/booth.

3. On the nodes belonging to the individual GEO clusters that the arbitrator is used for, open the port that is used for the respective booth instance.

4. Synchronize the respective booth configuration files across all cluster nodes and arbitrators that use the same booth configuration. For details, see Section 6.3, “Synchronizing the Booth Configuration Across All Sites and Arbitrators”.

5. On the arbitrator, start the individual booth instances as described in Starting the Booth Services on Arbitrators for multi-tenancy setups.

6. On the individual GEO clusters, start the booth service as described in Starting the Booth Services on Cluster Sites.

1. Adjust the booth configuration files as desired.

2. Synchronize the updated booth configuration files to all cluster nodes and arbitrators that are part of your GEO cluster. For details, see Section 6.3, “Synchronizing the Booth Configuration Across All Sites and Arbitrators”.

3. Restart the booth services on the arbitrators and cluster sites as described in Section 6.4, “Enabling and Starting the Booth Services”. This does not have any effect on tickets that have already been granted to sites.

For GEO clusters, you can specify which resources depend on a certain ticket. Together with this special type of constraint, you can set a loss-policy that defines what should happen to the respective resources if the ticket is revoked. The attribute loss-policy can have the following values:

1. On one of the cluster nodes, start a shell and log in as root or equivalent.

2. Enter crm configure to switch to the interactive crm shell.

3. Configure a constraint that defines which resources depend on a certain ticket. For example:

   crm(live)configure# rsc_ticket rsc1-req-ticketA ticketA: rsc1 \
     loss-policy="fence"

   This creates a constraint with the ID rsc1-req-ticketA. It defines that the resource rsc1 depends on ticketA and that the node running the resource should be fenced in case ticketA is revoked.

   Alternatively, you can configure resource rsc1 not as a primitive, but a multi-state resource that can run in master or slave mode. In that case, make only rsc1's master mode depend on ticketA. With the following configuration, rsc1 is automatically demoted to slave mode if ticketA is revoked:

   crm(live)configure# rsc_ticket rsc1-req-ticketA ticketA: rsc1:Master \
     loss-policy="demote"

4. If you want other resources to depend on further tickets, create as many constraints as necessary with rsc_ticket.

5. Review your changes with show.

6. If everything is correct, submit your changes with commit and leave the crm live configuration with exit.

   The constraints are saved to the CIB.

Each site needs to run one instance of boothd that communicates with the other booth daemons. The daemon can be started on any node, therefore it should be configured as primitive resource. To make the boothd resource stay on the same node, if possible, add resource stickiness to the configuration. As each daemon needs a persistent IP address, configure another primitive with a virtual IP address. Group booth primitives:

1. On one of the cluster nodes, start a shell and log in as root or equivalent.

2. Enter crm configure to switch to the interactive crm shell.

3. Enter the following to create both primitive resources and to add them to one group, g-booth:

   crm(live)configure# primitive booth-ip ocf:heartbeat:IPaddr2 \
     params ip="IP_ADDRESS"
   crm(live)configure# primitive booth ocf:pacemaker:booth-site \
     meta resource-stickiness="INFINITY" \
     op monitor interval="10s"
     group g-booth booth-ip booth

4. Review your changes with show.

5. If everything is correct, submit your changes with commit and leave the crm live configuration with exit.

6. Repeat the resource group configuration on the other cluster sites, using a different IP address for each boothd resource group.

   With this configuration, each booth daemon will be available at its individual IP address, independent of the node the daemon is running on.

If a ticket has been granted to a site but all nodes of that site should fail to host the boothd resource group for any reason, a “split-brain” situation among the geographically dispersed sites could occur. In that case, no boothd instance would be available to safely manage fail-over of the ticket to another site. To avoid a potential concurrency violation of the ticket (the ticket is granted to multiple sites simultaneously), add an ordering constraint:

1. On one of the cluster nodes, start a shell and log in as root or equivalent.

2. Enter crm configure to switch to the interactive crm shell.

3. Create an ordering constraint:

   crm(live)configure# order order-booth-rsc1 inf: g-booth rsc1

   This defines that rsc1 (that depends on ticketA) can only be started after the g-booth resource group.

   In case rsc1 is not a primitive, but a special clone resource and configured as described in Step 3 of Procedure 5, “Configuring Ticket Dependencies”, the ordering constraint should be configured as follows:

   crm(live)configure# order order-booth-rsc1 inf: g-booth rsc1:promote

   This defines that rsc1 can only be promoted to master mode after the g-booth resource group has started.

4. Review your changes with show.

5. For any other resources that depend on a certain ticket, define further ordering constraints.

6. If everything is correct, submit your changes with commit and leave the crm live configuration with exit.

For GEO clusters, you can specify which resources depend on a certain ticket. Together with this special type of constraint, you can set a loss-policy that defines what should happen to the respective resources if the ticket is revoked. The attribute loss-policy can have the following values:

The following example shows two alternatives to configure the constraint: One with the resource being a primitive and loss-policy="fence", the other one with the resource being a multi-state resource that can run in master or slave mode and with loss-policy="demote".

1. Start a Web browser and log in to Hawk.

2. In the left navigation bar, select Constraints. The Constraints screen shows categories for all types of constraints and lists all defined constraints.

3. To add a new ticket dependency, click the plus icon in the Ticket category.

   To modify an existing constraint, click the wrench icon next to the constraint and select Edit Constraint.

4. Enter a unique Constraint ID. When modifying existing constraints, the ID is already defined.

5. Set a Loss Policy.

6. Enter the ID of the ticket that the resources should depend on.

7. Select a resource from the list Add resource to constraint. The list shows the IDs of all resources and all resource templates configured for the cluster.

8. To add the selected resource, click the plus icon next to the list. A new list appears beneath, showing the remaining resources. Add as many resources to the constraint as you would like to depend on the ticket.

   

   Figure 4: Hawk—Ticket Dependency with loss-policy="fence" #

   Figure 4, “Hawk—Ticket Dependency with loss-policy="fence"” shows a constraint with the ID rsc1-req-ticketA. It defines that the resource rsc1 depends on ticketA and that the node running the resource should be fenced in case ticketA is revoked.

   If resource rsc1 was not a primitive, but a multi-state resource, define that only rsc1's master mode depends on ticketA. With the configuration shown in Figure 5, “Hawk—Ticket Dependency with loss-policy="demote"”, rsc1 is automatically demoted to slave mode if ticketA is revoked:

   

   Figure 5: Hawk—Ticket Dependency with loss-policy="demote" #

9. Click Create Constraint to finish the configuration. A message at the top of the screen shows if the constraint was successfully created.

Each site needs to run one instance of boothd that communicates with the other booth daemons. The daemon can be started on any node, therefore it should be configured as primitive resource. To make the boothd resource stay on the same node, if possible, add resource stickiness to the configuration. As each daemon needs a persistent IP address, configure another primitive with a virtual IP address. Group booth primitives:

1. Start a Web browser and log in to Hawk.

2. In the left navigation bar, select Resources. The Resources screen shows categories for all types of resources. It lists any resources that are already defined.

3. Select the Primitive category and click the plus icon.

4. To specify the resource for boothd:

   1. Enter a unique Resource ID, for example: booth-ip.

   2. Set Class to ocf, Provider to heartbeat and Type to IPaddr2.

      Hawk automatically shows any required parameters for the resource plus an empty drop-down list that you can use to specify additional parameters.

   3. Define the following Parameters (instance attributes) for the resource and enter values for them:

      • ip

      • cidr_netmask

   4. Click Create Resource to finish the configuration. A message at the top of the screen shows if the resource was successfully created or not.

5. Click Back to return to the list of configured resources.

6. Select the Primitive category and click the plus icon.

7. To specify the resource for boothd:

   1. Enter a unique Resource ID, for example: booth.

   2. Set Class to ocf, Provider to pacemaker and Type to booth-site.

      Hawk automatically shows any required parameters for the resource plus an empty drop-down list that you can use to specify additional parameters.

   3. In the Operations category, select monitor. Hawk proposes a timeout value of 20 and an interval of 10 seconds. Keep the proposed values and add this monitoring operation by clicking the plus icon next to it.

   4. In the Meta-Attributes category, select resource-stickiness and enter INFINITY as value. Click the plus icons next to the value to add this meta attribute.

   5. Click Create Resource to finish the configuration. A message at the top of the screen shows if the resource was successfully created or not.

8. Click Back to return to the list of configured resources.

9. To create the group and add booth primitives to it:

   1. Select the Group category and click the plus icon.

   2. Enter a unique Group ID, for example: g-booth.

   3. To define the group members, select booth-ip and booth in the list of Available Primitives and click the < icon to add them to the Group Children list. To define the order of the group members, you currently need to add and remove them in the order you desire.

   4. Hawk automatically proposes the meta attribute target-role. Set its value to Started.

      

      Figure 6: Hawk—Resource Group for boothd #

   5. Click Create Group to finish the configuration. A message at the top of the screen shows if the group was successfully created.

10. Repeat the resource group configuration on the other cluster sites, using a different IP address for each boothd resource group.

    With this configuration, each booth daemon will be available at its individual IP address, independent of the node the daemon is running on.

If a ticket has been granted to a site but all nodes of that site should fail to host the boothd resource group for any reason, a “split-brain” situation among the geographically dispersed sites could occur. In that case, no boothd instance would be available to safely manage fail-over of the ticket to another site. To avoid a potential concurrency violation of the ticket (the ticket is granted to multiple sites simultaneously), add an ordering constraint:

1. Start a Web browser and log in to Hawk.

2. In the left navigation bar, select Constraints. The Constraints screen shows categories for all types of constraints and lists all defined constraints.

3. Select the Order category and click the plus icon to create a new ordering constraint.

4. Enter a unique Constraint ID, for example order-booth-rsc1.

5. Set the Score to INFINITY.

   For colocation constraints, the score determines the location relationship between the resources. Setting the score to INFINITY forces the resources to run on the same node. For order constraints, the constraint is mandatory if the score is greater than zero, otherwise it is only a suggestion. The default value is INFINITY.

6. Keep the option Symmetrical enabled. This specifies that resources are stopped in reverse order.

7. To define the resources for the constraint:

   1. Select the resource group g-booth from the list Add resource to constraint and click the plus icon next to the list to add the resource to the ordering constraint.

      

      Figure 7: Hawk—Ordering Constraint with Multi-state Resource #

   2. Select the resource rsc1 from the list Add resource to constraint and click the plus icon next to the list to add the resource to the ordering constraint.

      Now you have both resources in a dependency chain. The topmost (g-booth) will start first, then the next one (rsc1). Usually the resources will be stopped in reverse order.

   3. In case rsc1 is not a primitive, but a multi-state resource and configured as described in Step 8 of Procedure 8, “Configuring Ticket Dependencies”, select the following entry from the empty drop-down box next to rsc1: promote. This defines that rsc1 can only be promoted to master mode after the g-booth resource group has started.

   4. Click Create Constraint.

      A message at the top of the screen shows if the constraint was successfully created.

8. Click Back to return to the list of constraints.

9. For any other resources that depend on a certain ticket, define further ordering constraints.

Assuming that you want to manually move ticketA from site 147.2.207.14 to 192.168.1.110, proceed as follows:

1. Set ticketA to standby with the following command:

   root # crm_ticket -t ticketA -s

2. Wait for any resources that depend on ticketA to be stopped or demoted cleanly.

3. Revoke ticketA from its current site with:

   root # booth revoke -s 147.2.207.14 ticketA

4. After the ticket has been revoked from its original site, grant it to the new site with:

   booth grant -s 192.168.1.110 ticketA

1. For an upgrade of the cluster nodes from SUSE Linux Enterprise High Availability Extension 11 SP3 to SUSE Linux Enterprise High Availability Extension 12, follow the instructions in the Administration Guide for SUSE Linux Enterprise High Availability Extension 12, section Upgrading from SLE HA 11 SP3 to SLE HA 12.

2. If you use arbitrators outside of the cluster sites:

   1. Upgrade them from SUSE Linux Enterprise Server 11 SP3 to SUSE Linux Enterprise Server 12, too.

   2. Add the GEO Clustering for SUSE Linux Enterprise High Availability Extension add-on and install the packages as described in Section 1.2, “Installing the Packages on Arbitrators”.

3. Because the syntax and the consensus algorithm for booth has changed, you need to update the booth configuration files to match the latest requirements. Whereas previously the optional expiry time and weights could be specified by appending them to the ticket name with a semicolon (;) as separator, the new syntax has separate tokens for all ticket options. See Section 6, “Setting Up the Booth Services” for details. If you did not specify expiry time or weights different from the defaults and do not want to make use of the multi-tenancy feature, you can still use the old /etc/booth/booth.conf.

4. Synchronize the updated booth configuration files across all cluster sites and arbitrators.

5. Start the booth service on the cluster sites and the arbitrators as described in Section 6.4, “Enabling and Starting the Booth Services”.