How to configure Apache Kafka with TIBCO AMX BusinessWorks-6.x

In this tutorial, we will discuss how to integrate TIBCO ActiveMatrix BusinessWorks Plug-in for Apache Kafka with TIBCO ActiveMatrix BusinessWorks.

To achieve that we will first discuss the components we require for the setup, the steps required to install and configure the environment along with subsequent options to create and deploy an application and finally will test the entire message flow execution.

For this purpose, the below index shall be followed:

1.Integrate Anything

With TIBCO Software, you can easily integrate virtually any endpoint across your IT ecosystem with a wide variety of commercial and community developed connectors for modern business suites and platforms as well as legacy technologies. TIBCO offers 200+ connectors that are supported across the integration capabilities within TIBCO Cloud Integration: Connect, Integrate, and Develop. 

TIBCO’s broad connector portfolio allows you to focus more on developing business logic, and less on connectivity details.

1.1 Components we require for the setup

• TIBCO ActiveMatrix BusinessWorks [Ver. 6.7.0/6.5.1]: Download
• TIBCO ActiveMatrix BusinessWorks™ is TIBCO’s leading integration platform. It allows organizations to create, orchestrate services, capture, and publish events from internal or external applications and technologies. Developers can rapidly develop, and test graphically, integration processes in a no-code, standards based IDE. These processes are then deployed in a reliable, highly available and scalable architecture to support mission critical applications.
• Apache Kafka [Ver. 2.6.0]: Download:Mac  A distributed streaming platform published by the Apache Foundation (see https://kafka.apache.org/), is a component of TIBCO(R) Messaging.
• TIBCO ActiveMatrix BusinessWorks Plug-in for Apache Kafka[Ver.6.4.1] Download TIBCO ActiveMatrix BusinessWorks Plug-in for Apache Kafka integrates Apache Kafka with TIBCO ActiveMatrix BusinessWorks. It addresses the need of the TIBCO ActiveMatrix BusinessWorks customer who wants a user-friendly way to integrate with Apache Kafka to create producers, consumers and perform send message and receive message operations.
• TIBCO Messaging-Schema Repo for Apache Kafka [Ver. 1.2.0]: Download:Mac A component of TIBCO(R) Messaging for managing Apache Avro schemas.

1.2 Apache Kafka Plug-in Palette

Activities in this palette can be used to create producers, consumers, and perform send message and receive message operations. 

• Kafka Send Message Activity:

This activity is used to send messages to the Kafka consumer. It performs as a Kafka producer, which sends the message to a specified topic, and consumers can fetch the message from the specified topics.

• Kafka Receive Message Activity:

This activity is a process starter activity that starts the process execution on receiving a Kafka message.

• Avro Schema Registry:

Avro schema is used with or without registry to configure Send and Receive activities to use supplied AVRO schema as a message format.

• Kafka Transactions:

Kafka Transactions is used to support exactly once semantics and Kafka Transaction API. 

• Kafka Get Messages Activity:

This activity is used to consume messages from Kafka topics. The activity also allows you to seek partition offset to beginning, end or custom offset value. 

2. Architecture Overview

The following figure describes the relationship between the Apache Kafka Server, TIBCO ActiveMatrix BusinessWorksTM Plug-in for Apache Kafka, and TIBCO ActiveMatrix BusinessWorksTM

The following list describes the relationship between different products:

• TIBCO ActiveMatrix BusinessWorksTM Plug-in for Apache Kafka communicates with the Kafka server instance.
• TIBCO ActiveMatrix BusinessWorksTM Plug-in for Apache Kafka plugs into TIBCO ActiveMatrix BusinessWorksTM and connects to a Kafka server instance.
• TIBCO ActiveMatrix BusinessWorksTM is an easy-to-use integration product suite for enterprise applications.
• TIBCO Business StudioTM is the graphical user interface (GUI) used by TIBCO ActiveMatrix BusinessWorksTM and the plug-in to design business processes, and it is also the process engine used to execute them.
• TIBCO Enterprise AdministratorTM provides a centralized administrative interface to manage and monitor the plug-in applications deployed in an enterprise.

3.TIBCO ActiveMatrix BusinessWorksTM Plug-in installation Overview

This section is designed to get users started with TIBCO ActiveMatrix BusinessWorksTM Plug-in for Apache Kafka in TIBCO Business Studio. You can install TIBCO ActiveMatrix BusinessWorksTM Plug-in for Apache Kafka either by using TIBCO Universal Installer or an Update Site.

3.1 Installation using Update Site

You can install the plug-in from within TIBCO Business StudioTM using the public update site maintained by TIBCO or using a local update site managed by your organization.

• Installation Prerequisites:

The software package must be available on the update site, http://update.tibco.com/eclipse/bw/version (where the ‘version’ at the end of the address reflects the version of BusinessWorks you are using). Typically, the package name has the format <plugin_name>_<version>_p2installer.zip.

• Installation Procedure:

1. Start TIBCO Business StudioTM if it has not already been started.

• On Microsoft Windows, click Start > All Programs > TIBCO > TIBCO_HOME > TIBCO Business Studio version_number > Studio for Designers.
• On Linux or Mac OS, run the TIBCO Business StudioTM executable located in the TIBCO_HOME/studio/version_number/eclipse directory.

2. From menu, select Help>Install New Software to open Eclipse Update Manager.

 

3. In the Install dialog, enter the URL of the public update site, http://update.tibco.com/eclipse/bw/<version>, in the ‘Work with’ field.

For ex. TIBCO BusinessWorks Update Site – http://update.tibco.com/eclipse/bw/6.5/

You can save the URL by adding it to the list of available software sites using the TIBCO Business StudioTM menu

Window > Preferences > Install/Update > Available Software Sites.

4. From the list of available plug-ins, select the one you want to install. Click Next.

You can select plug-in/s from the list to install multiple plug-ins at the same time.

(Optional) You can also find more software by clicking “Available Software Sites”.

5. In the Install dialog, Select the components you want to install. Click Next.

6. Click Next.

7. In the Review Licenses dialog, review the licenses, and click I accept the terms of the license agreement.

8. Click Finish to start the installation.

• What to do next:

After installing the software, restart of TIBCO Business Studio is necessary.

3.2 Using TIBCO Universal Installer

TIBCO ActiveMatrix BusinessWorks™ Plug-in for Apache Kafka – Community Edition is available now.(https://community.tibco.com/questions/apache-kafka-600-community-edition-plug-available-now)

• Plug-in is currently supported with
  TIBCO ActiveMatrix BusinessWorks™ 6.3.4 – 6.3.5 & 
  TIBCO BusinessWorks™ Container Edition 2.2.x
• Plugin can be downloaded from here
• Install the plug-in into an existing installation environment where TIBCO ActiveMatrix BusinessWorks is already installed. 
• The installation environment of TIBCO ActiveMatrix BusinessWorksTM is referred to as TIBCO_HOME.

Installation Modes:

You can run TIBCO Universal Installer in GUI, console, or silent mode. 

Each mode is supported on all platforms. 

• GUI Mode: (We are covering only this option)

In GUI mode, the installer displays panels where you can select a product, product location, and so on. Double-click the executable file to invoke the installer in GUI mode. 

• Console Mode: (Out of scope)

In console mode, you can run the installer on a command line. This is useful if your machine does not have a GUI environment.

• Silent Mode: (Out of scope)

In silent mode, the installer installs the product without prompting you for information. The installer uses either default or custom settings that are saved in a response file.

Installing in GUI Mode:

In GUI mode, the installer prompts you for information regarding the installation environment, installation profile, and other installation information.

Installation Procedure: 

1.Download the installation package from the TIBCO eDelivery site,

(https://edelivery.tibco.com)….will require authentication.

2. Extract the content of the installation package to a temporary directory.

Use the extracted package to start the installation in one of the following ways: 

• On Windows, double-click TIBCOUniversalInstaller-x86-64.exe.
• On Linux, run TIBCOUniversalInstaller-platform_acronym.bin.
• On Mac, double-click TIBCOUniversalInstaller-mac.command.

3. On the Welcome screen, click Next.

4. On the License Agreement screen, read through the license agreement and click

 ‘I accept the terms of the license agreement.’ & Click Next.

5. On the TIBCO Installation Home screen, click Use an existing TIBCO_HOME to select an existing 
installation environment where TIBCO ActiveMatrix BusinessWorks is installed. Click Next.

6. On the Installation Profile Selection screen, select an installation profile to specify the installation components to be installed. Make sure Apache Kafka Design & Runtime options have been selected.

Or select the Customize Installation check box to select the installation components. Click Next.

7. On the Pre-Install Summary screen, review the list of products selected for the installation. Click Next & Install.

8. On the Post Install Summary screen, review the list of the installation information. 

9. Click Finish to complete the installation process and exit the installer.

• Installation Logging: 

If you encounter problems with the installation, first ensure that your system meets all requirements. Next, check the installer log for potential problems. 

The installer log file, tibco_universal_installer.username_install.log, is available in the following locations: 

• On Microsoft Windows: C:\Users\username\.TIBCO\install_timestamp
• On Linux: user_home_dir/.TIBCO/install_timestamp
• On Mac OS: Users/user_home_dir/.TIBCO/install_timestamp

The installer log file captures the following information: 

• Detailed information regarding the user that invoked the installer, hostname, operating system details, and the list of assemblies installed
• To change the location of the installer log file, specify the option -V logFile=”myLogFile” when you run the installer. 

3.3 E2E Process Flow

1.Create a Project:

• Open the TIBCO Businessworks Studio & create BusinessWorks Application Module

2. Create a Process:

Create a process flow using Kafka activities from palette as shown

3. Create a Kafka Connection resource:

• Populate Bootstrap Servers details in Kafka shared resource and click on ‘Validate Bootstrap Servers’ to confirm connection with Kafka Broker Server is successful.

Note: Make sure, Kafka Zookeeper & Broker Server are running in background consoles.

4. Configure Publish activity resource:

•  Eventually, make sure both the Kafka Activities are referring to same ‘Kafka shared resource’ which we configured in the previous step.
• In KafkaSendMessage activity, populate field Topic as ‘test’ & input message as “Hello Kafka World!!”

5. Configure Subscribe activity resource:

6. Configure Log activity:

• Then log the incoming message from activity KafkaGetMessage

3.4 Steps to follow to test E2E Process Flow

Here are the steps to follow:

1.Execute the BW Project:

• Start the process in debug mode as shown below,

2. Publish a Message:

• KafkaSendMessage activity publishes message to topic ‘test’,

3. Subscribe a Message:

• Activity KafkaGetMessage consumes the same message which is eventually getting logged in console using Log activity,
• Also, will see the published message at Kafka server by running the following command,

 “bin/kafka-console-consumer.sh –bootstrap-server localhost:9092 –topic test —  from-beginning”

17:43:04.553 INFO  [main] com.tibco.thor.frwk - TIBCO ActiveMatrix BusinessWorks version 6.5.1, build V9, 2019-02-06
17:43:18.638 INFO  [main] com.tibco.bw.frwk.engine.BWEngine - TIBCO-BW-FRWK-300002: BW Engine [Main] started successfully.
17:43:18.725 INFO  [Framework Event Dispatcher: Equinox Container: a0e5d296-d967-001b-10f5-bf94366cd4af] com.tibco.thor.frwk.Deployer - TIBCO-THOR-FRWK-300001: Started OSGi Framework of AppNode [BWEclipseAppNode] in AppSpace [BWEclipseAppSpace] of Domain [BWEclipseDomain]
17:43:18.891 INFO  [Framework Event Dispatcher: Equinox Container: a0e5d296-d967-001b-10f5-bf94366cd4af] com.tibco.thor.frwk.Application - TIBCO-THOR-FRWK-300018: Deploying BW Application [Kafka_GeneralSample.application:1.0].
17:43:22.046 INFO  [Job_Executor0] com.tibco.thor.frwk.Application - TIBCO-THOR-FRWK-300021: All Application dependencies are resolved for Application [Kafka_GeneralSample.application:1.0]
log4j:WARN No appenders could be found for logger (org.apache.kafka.clients.CommonClientConfigs).
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
17:43:25.020 INFO  [Thread-18] com.tibco.thor.frwk.Application - TIBCO-THOR-FRWK-300006: Started BW Application [Kafka_GeneralSample.application:1.0]
17:43:45.873 INFO  [bwEngThread:In-Memory Process Worker-3]- Received message from GetMessage Activity : <?xml version="1.0" encoding="UTF-8"?><ns0:KafkaGetMessagesOutput xmlns:ns0="http://www.tibco.com/namespaces/tnt/plugins/getmessages+-bc77-+KafkaGetMessagesOutput"><Records><topic>getmessage</topic><partition>0</partition><offset>0</offset><message>Hello Kafka World!!</message></Records></ns0:KafkaGetMessagesOutput>

4. Containerization option from TIBCO for Kafka plug-in

4.1 TIBCO BusinessWorks Container Edition [Ver. 2.6.0]

TIBCO BusinessWorks™ Container Edition allows customers to leverage the power and functionality of TIBCO ActiveMatrix BusinessWorks™ in order to build cloud-native applications with an API-first approach and deploy it to container-based PaaS platforms such as Cloud Foundry™. 

TIBCO ActiveMatrix BusinessWorks™ is TIBCO’s flagship Enterprise integration offering. Deployed by thousands of customers worldwide, it serves as an integration foundation for mission critical IT environments. Powered by a next-generation foundation that includes an Eclipse-based design-time, powerful process engine and a modular OSGI-based run-time, BusinessWorks enables developers to create new services, orchestrate business processes and integrate applications in the shortest time possible.

5. Conclusions about Apache Kafka with TIBCO AMX BW6.x

So first, we have installed & configured TIBCO BW Kafka plugin with TIBCO AMX BW6.x & then we have developed a BW process to publish and subscribe a message to Kafka Topic ‘test’.

We have also made a consumer client listening on given Topic at Apache Kafka. When we executed a BW process, it first published a message on the Kafka topic (‘test’) using a Kafka activity, ‘Publish’.

Then we received the same message at Kafka consumer which was listening on the same topic, ‘test’. Also, the Kafka Subscribe activity in the same BW process got the same message.

This way, we can confirm, using ‘Publish’ activity, message is getting published to Kafka Topic and using ‘Subscribe’ activity, the same message can be consumed by client.