Unicast Configuration Example

The steps showed in this guide are the same followed in the demonstration video. Please adapt them to match your cluster configuration. The Ansible inventory file for any playbook/task showed in this exercise is:

[master]
master.hpc              ansible_connection=local

[clusters]
c01
c02
c03

Of course, if you have configured the dynamic inventory with Cobbler, no inventory file is needed.

Server Configuration (gmetad)

Ganglia’s metadata monitor gmetad can be installed in the master node by executing:

yum install ganglia-gmetad

After the installation is complete, edit the configuration file in /etc/ganglia/gmetad.conf:

# data_source "my cluster" [polling interval (seconds)] address1:port [address2 ]
data_source "Master 7" 1 localhost 192.168.16.1 127.0.0.1

The polling interval is given in seconds. In this example, gmetad will try to pull information from the nodes every second. If the pull is successful, a single data point (SDP) will be stored to be consolidated in the RRA (Round-Robin Archives). To define the RRAs use:

# RRA:CF:xff:steps:rows (24h at 1s res. 2 weeks at 1min res)
RRAs "RRA:AVERAGE:0.5:1:86400" "RRA:AVERAGE:0.5:60:20160"

In this declaration RRA:CF:xff:steps:rows:

  • steps are the number of SDP used to create one consolidated data point (CDP). It is also known as the resolution of the RRA. In our two RRAs, one and 60 SDP respectively.

  • xff (the x files factor) refers to the minimum of valid SDP required to write a valid CDP. It can be set from 0 (0%) to 1 (100%). In this example, 0.5 (50%) means that for the second RRA, at least 30 SDP are required to create one CDP.

  • CF (the consolidation function) is used to create the CDPs. It can be set to AVERAGE, MIN, MAX or LAST. In our case, AVERAGE.

  • rows are the amount of CDPs the RRA can hold. In this example, 86400 one second CDPs for a total of 24 hours, and 20160 CDPs consolidated every 60 times the polling interval (one minute) for a total of two weeks.

Additional variables to be set:

setuid_username ganglia
case_sensitive_hostnames 0

Once the configuration is done, enable and start the gmetad service:

[root@master ~]# systemctl enable gmetad
[root@master ~]# systemctl start gmetad

Note

A polling interval of one second is extreme and can affect the network in the cluster. It was chosen to show the short-term results of the demonstration. A 60-second polling interval can be a good start point in production systems.

Task

Calculate the resolution and capacity (in time) for the following RRAs: RRAs "RRA:AVERAGE:0.5:1:5856" "RRA:AVERAGE:0.5:4:20160" "RRA:AVERAGE:0.5:40:52704" "RRA:AVERAGE:0.5:1440:14600" for a polling interval of 60.

Client Configuration (gmond)

Note

If you want the master node to report to gmetad, you must follow these steps in the master and compute nodes.

For the client configuration, we have to install the ganglia-gmond package:

yum install ganglia-gmond

Next, we adjust the configuration file. First, the cluster declaration:

cluster {
  name = "Master 7"
  owner = "Temple University"
  latlong = "unspecified"
  url = "unspecified"
}

Then, the UDP send and receive channels, remember this is for the UNICAST case:

udp_send_channel {
  host = 192.168.16.1
  port = 8649
  ttl = 1
}

udp_recv_channel {
  port = 8649
  retry_bind = true
}

For this basic configuration, set the TCP to accept channel:

tcp_accept_channel {
  port = 8649
  # If you want to gzip XML output
  gzip_output = no
}

Leave the remaining configuration setting with the default values. Note that this is the most basic configuration. Ganglia can be fine-tuned to your needs, including MULTICAST mode and custom modules. The default modules can be found in the gmond configuration file under the modules keyword:

modules {
  module {
    name = "core_metrics"
  }
  module {
    name = "cpu_module"
    path = "modcpu.so"
  }
...

Finally, enable and start the service gmond:

[root@master ~]# systemctl enable gmond
[root@master ~]# systemctl start gmond

Web Front-end Configuration

To configure the web interface provided by Ganglia (you can also create your own), you need to install the ganglia-web package in master:

yum install ganglia-web

We have to add an alias in the Apache webserver to /usr/share/ganglia. Usually, this alias is ganglia so you can reach the web interface at http://192.168.16.1/ganglia, but you can choose any alias you like. To do this and to add some permissions to access this particular location, edit the configuration file /etc/httpd/conf.d/ganglia.conf:

Alias /ganglia /usr/share/ganglia

<Location /ganglia>
  Require all granted
</Location>

To apply this configuration, restart the httpd service:

[root@master ~]# systemctl restart httpd

You should now be able to see your cluster stats in your web browser. It is recommended to create a SOCKS proxy with SSH. To do this, add -D [port number] to your ssh root@master[1-10] command. You have to configure your browser to use the SOCKS proxy. See the video for more information.

Full Ganglia Configuration as a Playbook

You can do all the steps by executing the following playbook (you must change it to match your cluster.)

---
- name: Install gmond in the master node
  hosts: master

  tasks:
    - name: Install Ganglia gmetad and webfrontend
      yum:
        name: ['ganglia-gmond', 'ganglia-gmetad', 'ganglia-web']
        state: latest

    - name: Configure Ganglia gmetad daemon
      copy:
        src: gmetad.conf
        dest: /etc/ganglia/gmetad.conf

    - name: Configure Ganglia gmond daemon
      copy:
        src: gmond.conf
        dest: /etc/ganglia/gmond.conf

    - name: Configure /ganglia in webserver
      copy:
        src: ganglia.conf
        dest: /etc/httpd/conf.d/ganglia.conf

    - name: Enable and Restart Ganglia gmetad daemon
      service: name={{item}} enabled=yes state=restarted
      with_items:
        - gmond
        - gmetad
        - httpd

- name: Install ganglia in masters
  hosts: compute

  tasks:
    - name: Install Ganglia gmond daemon
      yum:
        name: ganglia-gmond
        state: latest

    - name: Configure Ganglia gmond daemon
      copy:
        src: gmond.conf
        dest: /etc/ganglia/gmond.conf
        owner: root
        group: root
        mode: "u=rw,g=r,o=r"
        force: yes

    - name: Enable and Restart Ganglia gmond daemon
      service:
        name: gmond
        enabled: yes
        state: restarted

...

The files gmetad.conf, gmond.conf and ganglia.conf must be in the playbook’s path.