IBM API Connect Tutorial: Part-5 Create user defined policy in IBM API Connect

IBM API Connect comes with a some very useful policies for development, but they are not always sufficient. In an enterprise, there will always be need for more and more reusable policies. IBM has provided us with a framework, using which we can create user defined policy suited for our own need. In my previous blog, I have covered how to integrate MQ with IBM API Connect. I have also shown how to drop a message into MQ queue using gateway script. In this article, I’ll extend that particular POC. I’ll now create IBM API Connect user defined policy UDP. Let’s learn how to create user defined policy in IBM API Connect.

Use Case

Let’s consider that an organization has a requirement of auditing various metadata about the transaction and payload at various points of the API assembly. The audit information need to be put to IBM MQ. To achieve this, we can use a custom gateway extension and gateway script at various points. But it means that, we have to copy paste the same GW script multiple times across. It’s also a hassle to use this framework in different APIs; as we have to copy the script from another API assembly etc. Instead of this, we can package the gateway script and custom gateway extension to create a user defined policy. This policy will be available in the policy pallet for all the API developers. The developers can just drag and drop this user defined policy at any point of the API assembly.

This policy will extract the transaction metadata and message/request body to create an audit payload. Thereafter, it’ll put the message to MQ.

The functionalities of this policy are listed below:

• It reads the parameters set by developer in the policy
• The policy has a default queue manager and queue tied up with it. However should there be a requirement, it also provides an option for developers to provide a different queue manager and queue name.
• User can log the target application name (backend) in the policy itself. However, if it is not set in policy level, consumer can pass the backend application name in the http header. The http header settings can be useful for internally exposed APIs.
• Transaction Id and correlation Id can be set by user in the content variables (custom.transactionId and custom.correlId) with relevant values. However, if not set in the assembly, these values can be sent by consumers in http headers as well.
• Using all these information, it creates an audit payload and puts to MQ queue.

You can send this audit feed from MQ queue to any data store for storage and visualization. However, our tutorial will be limited to just putting the payload to MQ.

Basic Constructs of IBM API Connect User Defined Policy

Following image shows the basic construct and their respective description of user defined policy.

Let’s build AuditFW User Defined Policy

We’ll name our policy AuditFW. This policy will take few input parameters instructing the policy:

1. whether to log the payload or not
2. Logging sequence number
3. Whether to log the payload from request.body or message.body
4. Any custom message from the audit point
5. Whether to use a different Queue manager other than the default one
6. Whether to use a different queue other than the default one.

Build the MQ Queue Manager Object

We need to build a configuration file for the MQ Queue Manager object. We’ll reuse the same configuration file that we created in the previous article.

top; configure terminal;

mq-qm "AUDITQM"
  hostname x.x.x.x(1414)
  queue-manager "QM1"
  ccsid 819
  channel-name "DEV.APP.SVRCONN"
  heartbeat 300
  maximum-message-size 1048576
  cache-timeout 60
  no automatic-backout 
  total-connection-limit 250
  initial-connections 1
  sharing-conversations 0
  no share-single-conversation 
  no permit-insecure-servers 
  no permit-ssl-v3 
  ssl-cipher none
  no auto-recovery 
  convert 
  auto-retry 
  retry-interval 10
  retry-attempts 6
  long-retry-interval 600
  reporting-interval 10
  no alternate-user 
  polling-tolerance 10
  xml-manager default
  ssl-client-type proxy
exit

Build the policy configuration file

Next, we will create a configuration file that’ll define the UDP. It will have following components:

• assembly-setvar (to set the input parameters): This is the first assembly policy of our UDP. It will have few parameters (as explained above) that will be set by the API developer at the development time. These parameters will drive the policy behavior. As this is the first step to be executed, you will notice that we have used correlation-path “$.x-ibm-configuration.assembly.execute[0]”

assembly-setvar auditfw_1.0.0_set-variable_0
  reset
  title "set-variable"
  correlation-path "$.x-ibm-configuration.assembly.execute[0]"
  variable
    action set
    name "logsequenceparam"
    type string
    value "$(local.parameter.logsequence)"
  exit
  variable
    action set
    name "logpayloadparam"
    type string
    value "$(local.parameter.logpayload)"
  exit
  variable
    action set
    name "logpayloadfromparam"
    type string
    value "$(local.parameter.logpayloadfrom)"
  exit
  variable
    action set
    name "custommsgparam"
    type string
    value "$(local.parameter.custommsg)"
  exit
  variable
    action set
    name "targetappparam"
    type string
    value "$(local.parameter.targetapp)"
  exit
  variable
    action set
    name "queuemanagerparam"
    type string
    value "$(local.parameter.queuemanager)"
  exit
  variable
    action set
    name "queueparam"
    type string
    value "$(local.parameter.queue)"
  exit
exit

• assembly-gatewayscript: This is second assembly policy of our UDP. It’ll refer to the gateway script that will construct the audit message and put it to the queue.

assembly-gatewayscript auditfw_1.0.0_gatewayscript_1
  reset
  title "gatewayscript"
  correlation-path "$.x-ibm-configuration.assembly.execute[1]"
  gatewayscript-location temporary:///filestores/extensions/gateway-extension/auditfw-persist-to-mq.js
exit

• api-rule: The API rule will combine two policies that we created above:

api-rule auditfw_1.0.0_main
  reset
  action auditfw_1.0.0_set-variable_0
  action auditfw_1.0.0_gatewayscript_1
exit

• assembly: Now, we will create an assembly using the above rule:

assembly auditfw_1.0.0
  reset
  rule auditfw_1.0.0_main
exit

• assembly-function: Finally, we will create an assembly function that will wrap the input params and also the above created assembly:

assembly-function "auditfw_1.0.0"
  reset
  summary "auditfw-policy_1.0.0"
  title "Audit Framework"
  parameter
    name "logsequence"
    description "Mention the log seuence"
    value-type string
  exit
  parameter
    name "logpayload"
    description "Will you the log the payload? Type either Yes or No"
    value-type string
  exit
  parameter
    name "logpayloadfrom"
    description "Mention message.body or request.body based on requirement. Default is message.body"
    value-type string
  exit
  parameter
    name "custommsg"
    description "Type your custom message"
    value-type string
  exit
  parameter
    name "targetapp"
    description "Type the traget app name"
    value-type string
  exit
  parameter
    name "queuemanager"
    description "Mention the queue manager object name if you don't want to use the default. Leave it empty otherwise"
    value-type string
  exit
  parameter
    name "queue"
    description "Mention the queue  name if you don't want to use the default. Leave it empty otherwise"
    value-type string
  exit
  assembly auditfw_1.0.0
exit

apic-gw-service
  user-defined-policies auditfw_1.0.0
exit

Putting it all together, our configuration will look like below. Let’s save this configuration in a file named auditfw.cfg.

assembly-setvar auditfw_1.0.0_set-variable_0
  reset
  title "set-variable"
  correlation-path "$.x-ibm-configuration.assembly.execute[0]"
  variable
    action set
    name "logsequenceparam"
    type string
    value "$(local.parameter.logsequence)"
  exit
  variable
    action set
    name "logpayloadparam"
    type string
    value "$(local.parameter.logpayload)"
  exit
  variable
    action set
    name "logpayloadfromparam"
    type string
    value "$(local.parameter.logpayloadfrom)"
  exit
  variable
    action set
    name "custommsgparam"
    type string
    value "$(local.parameter.custommsg)"
  exit
  variable
    action set
    name "targetappparam"
    type string
    value "$(local.parameter.targetapp)"
  exit
  variable
    action set
    name "queuemanagerparam"
    type string
    value "$(local.parameter.queuemanager)"
  exit
  variable
    action set
    name "queueparam"
    type string
    value "$(local.parameter.queue)"
  exit
exit

assembly-gatewayscript auditfw_1.0.0_gatewayscript_1
  reset
  title "gatewayscript"
  correlation-path "$.x-ibm-configuration.assembly.execute[1]"
  gatewayscript-location temporary:///filestores/extensions/gateway-extension/auditfw-persist-to-mq.js
exit

api-rule auditfw_1.0.0_main
  reset
  action auditfw_1.0.0_set-variable_0
  action auditfw_1.0.0_gatewayscript_1
exit

assembly auditfw_1.0.0
  reset
  rule auditfw_1.0.0_main
exit

assembly-function "auditfw_1.0.0"
  reset
  summary "auditfw-policy_1.0.0"
  title "Audit Framework"
  parameter
    name "logsequence"
    description "Mention the log seuence"
    value-type string
  exit
  parameter
    name "logpayload"
    description "Will you the log the payload? Type either Yes or No"
    value-type string
  exit
  parameter
    name "logpayloadfrom"
    description "Mention message.body or request.body based on requirement. Default is message.body"
    value-type string
  exit
  parameter
    name "custommsg"
    description "Type your custom message"
    value-type string
  exit
  parameter
    name "targetapp"
    description "Type the traget app name"
    value-type string
  exit
  parameter
    name "queuemanager"
    description "Mention the queue manager object name if you don't want to use the default. Leave it empty otherwise"
    value-type string
  exit
  parameter
    name "queue"
    description "Mention the queue  name if you don't want to use the default. Leave it empty otherwise"
    value-type string
  exit
  assembly auditfw_1.0.0
exit

apic-gw-service
  user-defined-policies auditfw_1.0.0
exit

Build the gateway script

Now, we will build the gateway script that will read the user input, context variables and build audit payload. Thereafter, it will put the payload into the queue. Let’s take a look at the gateway script file. We’ll name the configuration file as auditfw-persist-to-mq.js as mentioned in the auditfw.cfg file.

var apim = require('apim');
var urlopen  = require('urlopen');

var logsequenceparam = context.get('logsequenceparam');
var logpayloadparam = context.get('logpayloadparam');
var custommsgparam = context.get('custommsgparam');
var logpayloadfromparam = context.get('logpayloadfromparam');
var queuemanagerparam = context.get('queuemanagerparam');
var queueparam = context.get('queueparam');
var targetappparam = context.get('targetappparam');

if (!logsequenceparam){
	logsequenceparam = 0;
}
if (!queuemanagerparam){
	queuemanagerparam = 'AUDITQM';
}
if (!queueparam){
	queueparam = 'DEV.QUEUE.1';
}
var queuemanagerparam = 'dpmq://'+queuemanagerparam+'/?';

//get the input data
var data;
if(logpayloadfromparam.toLowerCase() == 'request.body'){
	data = apim.getvariable('request.body');
}else{
	data = apim.getvariable('message.body');
}
if (data){
	data = data.toString();
}

var tapp;
console.log("targetappparam:"+targetappparam);
if (targetappparam){
	tapp=targetappparam;
}else if(apim.getvariable('request.headers.x-tabadul-targetapp')){
	tapp=apim.getvariable('request.headers.x-tabadul-targetapp');
}

var transId = context.get('custom.transactionId');
if (!(transId)){
	transId=apim.getvariable('request.headers.x-tabadul-transactionId');
}

var correlId = context.get('custom.correlId');
if (!(correlId)){
	correlId=apim.getvariable('request.headers.x-tabadul-correlid');
}

var body={};
body.type='apicevent';
body.log_sequence=logsequenceparam;
body.correlation_Id=correlId;//context.get('custom.correlId');
body.transaction_Id=transId;
body.porg_details={};
body.porg_details.api_name=apim.getvariable('api.name');
body.porg_details.api_version=apim.getvariable('api.version');
body.porg_details.target_app=tapp;
body.corg_details = {};
body.corg_details.developer_org_name=apim.getvariable('client.org.name');
body.corg_details.app_name=apim.getvariable('client.app.name');
body.corg_details.plan_name=apim.getvariable('plan.name');
body.corg_details.client_id=apim.getvariable('client.app.id');
body.api_end_point= {};
body.api_end_point.resource_url=apim.getvariable('request.uri');
body.api_end_point.resource_verb=apim.getvariable('request.verb');
body.api_end_point.query_string=apim.getvariable('request.querystring');
body.custom_audit_message=context.get('custommsg');;
body.status_code=apim.getvariable('message.status.code')+" "+apim.getvariable('message.status.reason');

var contenttype = apim.getvariable('request.content-type');

if (data){
	if(logpayloadparam){
		var logpayloadparam = ""+ logpayloadparam.toUpperCase();
		if(logpayloadparam.includes("YES")){
			body.payload=data;
		}
	}
}

body.datetime=apim.getvariable('system.datetime')+apim.getvariable('system.timezone');

var inpMsg = JSON.stringify(body);
var dpmqurl = { target: queuemanagerparam,
			requestQueue: queueparam,
			transactional: false,
			sync: false,
			timeOut: 10000,
			data: inpMsg };

urlopen.open (dpmqurl, function (error, response) {
    // handle the error when connecting to MQ
	if (error) {
		var msg = error + 'errorCode=' + error.errorCode;
		console.error("MQ error is %s", msg);
		//throw error;
	}else{
		var responseCode = response.statusCode;
		var successMsg = "SUCCSS!! MQ status code is: " + responseCode;    
		//console.log(successMsg);
	}
});

Now, we will package all 3 files (queue manager configuration, policy configuration and gateway script) in a .zip file; we’ll name it as gwext.zip

Deploy the IBM API Connect user defined policy

We’ll deploy the policy in our local API connect LTE setup. So, let’s start the LTE by running following commands. Just to note, the commands are same for actual APIC setup as well; just that the argument values will change as per your environment setup.

apic-lte.exe start

Once the LTE starts, we will login to the API manager using admin user

$ apic login –server 127.0.0.1:2000 –username admin –password 7iron-hide –realm admin/default-idp-1

Thereafter, we’ll issue following command to apply the user defined policy in our API manager setup

apic gateway-extensions:create /C/gwext/gwext.zip –scope org –org admin –gateway-service datapower-api-gateway –availability-zone availability-zone-default –server 127.0.0.1:2000

If the command is executed successfully, we’ll see a result something like below

gateway-extension https://127.0.0.1:2000/api/orgs/42535da0-4c43-45cc-bed4-d190f4789235/availability-zones/4f8893e7-569b-4cb9-976a-9776d4abca1d/gateway-services/6236ded1-aff8-43ee-b159-545fd0c5b5eb/gateway-extension

To verify that, it is indeed applied, we will run following commands

$ apic gateway-extensions:get –scope org -o admin –gateway-service datapower-api-gateway –availability-zone availability-zone-default –server 127.0.0.1:2000 –output –

The result of the command will be as follows

type: gateway_extension
api_version: 2.0.0
name: gateway-extension
gateway_service_url: >-
https://127.0.0.1:2000/api/orgs/42535da0-4c43-45cc-bed4-d190f4789235/availability-zones/4f8893e7-569b-4cb9-976a-9776d4abca1d/gateway-services/6236ded1-aff8-43ee-b159-545fd0c5b5eb
scope: org
created_at: ‘2021-06-09T19:26:47.114Z’
updated_at: ‘2021-06-09T19:26:47.114Z’
org_url: ‘https://127.0.0.1:2000/api/orgs/42535da0-4c43-45cc-bed4-d190f4789235’
url: >-
https://127.0.0.1:2000/api/orgs/42535da0-4c43-45cc-bed4-d190f4789235/availability-zones/4f8893e7-569b-4cb9-976a-9776d4abca1d/gateway-services/6236ded1-aff8-43ee-b159-545fd0c5b5eb/gateway-extension

To delete the gateway extension, we can issue following command

apic gateway-extensions:delete –scope org -o admin –gateway-service datapower-api-gateway –availability-zone availability-zone-default –server 127.0.0.1:2000 –output –

Apply the policy to the gateway server by restarting the API Connect gateway service object. Complete the following steps on each Gateway server in the Gateway service.

• Log in to the Datapower (https://localhost:9091/dp/login.xml) using username admin and password admin; switch to the apiconnect domain
• Search for API Connect Gateway Service
• Set the administrative state to disabled
• Apply the changes
• Set the administrative state to enabled
• Apply the changes
• Save the configuration

Verify IBM API Connect User Defined Policy

In the user defined policy section of API Connect Gateway Service, we’ll notice the newly created policy

We’ll check the queue manager object is created as part of the user defined policy:

We’ll also check the gateway script file is present in the file management view:

Test IBM API Connect User Defined Policy

It’s the easy part. We’ll build an API with the newly created user defined policy.

• Launch the designer toolkit.
• Build a simple API, go to assemble section and notice the last policy in the policy pane. This is the UDP we just created. We can drag and drop this policy in the assembly and configure.
• PFB the swagger of the test API

swagger: '2.0'
info:
  title: UDP Test API
  x-ibm-name: udp-test-api
  version: 1.0.0
x-ibm-configuration:
  cors:
    enabled: true
  gateway: datapower-api-gateway
  type: rest
  phase: realized
  enforced: true
  testable: true
  assembly:
    execute:
      - gatewayscript:
          version: 2.0.0
          title: gatewayscript
          source: "var apim = require('apim');\r\nvar successMsg = \"Hello World\";\r\ncontext.message.body.write(successMsg);\r\ncontext.message.header.set('Content-Type', \"text/plain\");\r\ncontext.set('custom.correlId','CR123456');\r\ncontext.set('custom.transactionId','TR123456');"
      - auditfw:
          version: 1.0.0
          title: auditfw
          logsequence: '1'
          logpayload: 'yes'
          logpayloadfrom: ''
          custommsg: udp test
          targetapp: mybackend
          queuemanager: ''
          queue: ''
  properties:
    target-url:
      value: 'http://example.com/operation-name'
      description: The URL of the target service
      encoded: false
  application-authentication:
    certificate: false
basePath: /udp-test-api
paths:
  /:
    get:
      responses:
        '200':
          description: success
          schema:
            type: string
      consumes: []
      produces: []
    put:
      responses:
        '200':
          description: success
          schema:
            type: string
      consumes: []
      produces: []
    post:
      responses:
        '200':
          description: success
          schema:
            type: string
      consumes: []
      produces: []
    delete:
      responses:
        '200':
          description: success
          schema:
            type: string
      consumes: []
      produces: []
    options:
      responses:
        '200':
          description: success
          schema:
            type: string
      consumes: []
      produces: []
    head:
      responses:
        '200':
          description: success
          schema:
            type: string
      consumes: []
      produces: []
    patch:
      responses:
        '200':
          description: success
          schema:
            type: string
      consumes: []
      produces: []
securityDefinitions:
  clientIdHeader:
    type: apiKey
    in: header
    name: X-IBM-Client-Id
schemes:
  - https
security:
  - clientIdHeader: []

Conclusion

That was a pretty long tutorial. But I hope to have shared some knowledge.

Reference

• https://www.ibm.com/docs/en/api-connect/2018.x?topic=apdag-defining-packaging-your-user-defined-policydatapower-api-gateway
• https://www.ibm.com/docs/en/api-connect/2018.x?topic=apdag-publishing-your-user-defined-policy-datapower-apigateway
• https://github.com/ozairs/apiconnectpolicies/tree/master/udp-managed-obj