Load Balancing VMware Log Insight With KEMP’s LoadMaster Part 3 – Load Balancing Log Insight with KEMP


This article will show you how to configure a KEMP Load Master for VMware Log Insight.

Log Insight supports receipt and ingestion of Syslog messages that are sent over UDP, TCP, TCP with SSL Encryption and API.

The Load Master provides Log Insight-aware Services to optimise high availability and scalability of Log Insight deployments.

One Concern that KEMP has addressed, is when Syslog Messages are sent using methods other than UDP. Clients will often open long-lived connections that are then used for large amounts of messages. The issue is that traffic is not distributed on a close-to-even fashion across the cluster of available nodes. The Load Master provides a solution to this problem and allows messages to be parsed with a connection to allow a more even distribution across servers in a cluster, as well as simplified scalability of log Insight environments.

I have provided a logical diagram showing how the solution works.

Load Balancing Log insight

Installing and Configuring the Load Master

In this demo we are only going to use one Load Master, however you can deploy KEMP Load Masters in a HA Pair. For configuring a KEMP HA Solution please see the following Link: KEMP High Availability Configuration Guide

The first thing we need to do is Download a Load Master from KEMP’s Site. KEMP Technologies VLM Download Page

Import the OVF or (VMDK & VMX)  into your VMware Infrastructure.

KEMP VLM

You will then need to license the Load Master using your KEMP ID (Email Address) Please see the following Link: Licensing Feature Description

Adding the Log Insight Add on and Templates

Before we can start creating the Virtual Services for Log Insight, we will need to import the Add-on pack for Log Insight. you can find this at the following link: Log Insight Add-on pack

Once Downloaded, go to System Administration > Update Software

adding the Loginsight Addin packUnder the Installed Addon Packages, you can browse the Addon file, once selected, click Install Addon Package.

Finished added the addon

You will now see the Addon Package for Log Insight (Log_Insight). Ensure that you obtain the same Addon version as your Firmware Version. For Example if you have a Firmware Version 7.1-20a,  you will need a Addon Version of 7.1-20a.

You will need to reboot the Load Master once the install is complete for the Addon to become active.

Adding the Log Insight Templates

KEMP have simplyfyed the Deployment of Log Insight by create Templates that can be imported into the Load Master.

You can find the Template file at the following Link: Log Insight Template

The template contains the following Virtual Services: UDP, TCP, SSL, and API Services

To import the Template, Navigate to Virtual Services > Manage Templates

KEMP Log Insight Template

Select the Template File you want to Import, then click “Add New Template”

KEMP Log Insight Template after

Once imported you will see the Templates for all four Log Insight Virtual Services.

Configure Log Insight Message Split Interval

The Log Insight Split Interval Value Controls how many Syslog messages should be sent to each server in the cluster before moving to the next server. If you set the Split interval to 1, then it would allow a single message to be sent Server A before sending a message to Server B , then C, then again back to Server A.

Configure the Log Insight Split Interval:

Go to the Main Menu of WUI, > System Configuration > Miscellaneous Options > L7 Configuration

Setting the Split interval

Layer 7

I have set the Log Insight Message Split Interval to “1”

Configuring the Log Insight Virtual Services

Create the TCP Syslog Virtual Service

A TCP Syslog Virtual Service Must be create if clients will send syslog messages to Log insight TCP.

To create the Service, Navigate to Virtual Services > Add New

TCP Virtual ServiceEnter the Shared Virtual Service and Select the Template Log Insight TCP, This will then Populate all the required fields.

Standard Options Section:

It is recommended that you use Round Robin as the Scheduling Method as Round Robin is typically best to accomplish even traffic distribution. Avoid using Least Connection as the Scheduling Method as the Log Insight Split Interval will not behave as expected.

Standard Options TCP

Add the Real Servers (Master Node & Worker Node)

Firstly Configure the Real Server Check Parameters and set the Port “514”

Then add the Real Servers.

TCP real Server Add

TCP Log Insight rule

Create the UDP Syslog Virtual Service

A UDP Syslog Virtual Service Must be create if clients will send syslog messages to Log insight UDP.

To create the Service, Navigate to Virtual Services > Add New

Virtual Service UDP

Configuring Standard Options

Ensure that Force L7 (Layer 7) is selected and Ensure that transparency Check Box is Selected.

UDP Standard Options

Transparency allows the client’s IP address to be presented to the Log Insight Servers.  Depending on your Network Topology, Transparency may not be supported. You can disable this option and set the persistence to Source IP if it is not supported.

It is recommended that you set the Idle Connection Timeout value to 1 as it has been tested and shows the best performance for Log Insight’s UDP Service.

Real Servers

The UDP Real Server Check Parameter should use ICMP Ping

UDP real Servers

Add all the real Servers.

UDP Health Service

Once Complete, you will see the Virtual Service showing as up.

Create the SSL Syslog Virtual Service

A SSL Syslog Virtual Service Must be create if clients will send syslog messages to Log insight SSL.

To create the Service, Navigate to Virtual Services > Add New

SSL Virtual Service

For this Service, we will need to configure the Certificate from Log Insight:

Import the Log Insight Certificate:

Certificates > Import Certificate >

SSL Cert1

SSL Cert2

SSL Cert3

Assign the Certificate to the Virtual Service.

Real Servers:

enter the Checked port 514

SSL Virtual Service real server

SSL Real Servers

Ensure that the Real Servers are configured with the port 514.

Log Insight API Ingest Service

“if HTTP POST requests” are used to programmatically send log information to the Log Insight cluster. A “Log Split” Content rule is required and an accompanying Virtual Service Must be created. Content rules interrogate incoming client connections and make decisions as well as header modification based on the contents of the requests.

We need to create this rule to ensure even distribution of messages across the cluster of Log Insight nodes when the API Ingest Service is Utilised.

Creating the Log Split Content Rule:

A “Log Split” content rule is required to minimise “Lumpiness” (as KEMP Call it) to accomplish a more even distribution of messages that are posted.

Navigate to Rules & Checking > Content Rules

The Log Insight template has the content rule configured, you can check the configuration by selecting modify.

Content Rule

Content Rule2

Creating the Log Insight API Ingest Virtual Service

Add the IP for the Virtual Server and select the Template for Log Insight API.

api service

API Transparancy

Transparency Should be Selected. If not Supported, remember to select Source IP as the selected persistence.

api real server

Advance Settings API

Check that the Header Rules

Header rule

There you have it. The Load Master configured for VMware Log Insight.

Services

 

One thought on “Load Balancing VMware Log Insight With KEMP’s LoadMaster Part 3 – Load Balancing Log Insight with KEMP

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s