How to consume message from Kafka in OIC

How to consume message from Kafka Topic in OIC

In this example, I have configured a scheduled orchestrated integration to read records using the Apache Kafka Adapter and publish them to a file location using the FTP write operation.

Drag and drop the kafka adapter into the flow.
Configuring kafka adapter connection is shown in my previous post.

Select Consume records.

Select the topic and partition.

And specify the option for consuming messages as “Read latest”.

Specify Message structure as “Yes”

Select the message structure.

Next and finish.

Below is the map activity between kafka response and FTP input.

Kafka adapter in OIC

Kafka adapter in OIC

How to Create Connection.?

Navigate to Designer->Connections.

 Select the Apache Kafka adapter.

Name the Connection. And please note, role can be only selected as Invoke.

I used localhost:9092* – because the actual connectivity is handled by the agent, so in reality we are connecting to the Kafka server as if we were inside the machine where it runs.
*9092 is the default Kafka port, but you can verify the one you are using in <Kafka_Home>/config/server.properties

Note: Please visit my previous post on how to create Agent and run it in system where kafka is running.

 How to Produce Message into Kafka Topic.?

In this example, I have configured a scheduled orchestrated integration to read records using the FTP Adapter and publish them to an Apache Kafka topic using the Apache Kafka Adapter produce operation.

Final look of the integration will be something like below,

I have not mentioned the steps about reading the message using FTP adapter. Assume your FTP adapter will read the CSV file and we are going to use the same data for publishing into Kafka queue.
Now drag and drop the Kafka connection created in previous step.

You can visit my previous post to understand how to configure FTP adapters.

 Select the publish option.

Select the Topic name and partition. If we do not select a specific partition and use the Default selection, Kafka considers all available partitions and decides which one to use.
And select the Specify the Message structure as “Yes”.

You can specify the message structure with one of the below,
1. XML Schema (XSD) document
2. Sample XML document
3. Sample JSON document

Just have a simple Schema with 3 fields.

Select next and finish.
Below is the mapping details from File response to kafka request.

Now Activate the integration and test it. You can see the message in kafka Queue.

In my next post, we can see how to consume message from kafka topic.

How to Create Agent in OIC

How to Create Agent in OIC

When you install the connectivity agent in your environment, you associate the connectivity agent with the agent group identifier. Only one connectivity agent can be associated with an agent group. For a single Oracle Integration instance, you can create up to five agent groups. Creating the agent group also creates the necessary artifacts required for message exchange.

To create an agent group:

1. In the navigation pane, click Integrations, then click Agents.
2. Click Create Agent Group.
3. Enter the following information, then click Create

Note: Just keep note of the Identifier.

Now download the Connectivity Agent jar from console.

Unzip the file and edit InstallerProfile.cfg in it.

Enter OIC environment details and agent Identifier.

Note: you have to add oic_USER/oic_PASSWORD, it will not be there by default since it’s an optional parameter.

You must place the file in Linux server using winscp/filezilla.

Now navigate to the folder location in putty and run below command to run the connectivity agent. I have used nohup & to run this in in background.

nohup java -jar connectivityagent.jar &

Once you see the above message, you can confirm agent its running successfully. You can also check the agent status in OIC, Navigate to Monitoring, Agents and look for the status.