IBM API Connect comes with a some very useful policies for development, but they are not always sufficient. In an enterprise, there will always be need for more and more reusable policies. IBM has provided us with a framework, using which we can create user defined policy suited for our own need. In my previous blog, I have covered how to integrate MQ with IBM API Connect. I have also shown how to drop a message into MQ queue using gateway script. In this article, I’ll extend that particular POC. I’ll now create IBM API Connect user defined policy UDP. Let’s learn how to create user defined policy in IBM API Connect.
Use Case
Let’s consider that an organization has a requirement of auditing various metadata about the transaction and payload at various points of the API assembly. The audit information need to be put to IBM MQ. To achieve this, we can use a custom gateway extension and gateway script at various points. But it means that, we have to copy paste the same GW script multiple times across. It’s also a hassle to use this framework in different APIs; as we have to copy the script from another API assembly etc. Instead of this, we can package the gateway script and custom gateway extension to create a user defined policy. This policy will be available in the policy pallet for all the API developers. The developers can just drag and drop this user defined policy at any point of the API assembly.
This policy will extract the transaction metadata and message/request body to create an audit payload. Thereafter, it’ll put the message to MQ.
The functionalities of this policy are listed below:
- It reads the parameters set by developer in the policy
- The policy has a default queue manager and queue tied up with it. However should there be a requirement, it also provides an option for developers to provide a different queue manager and queue name.
- User can log the target application name (backend) in the policy itself. However, if it is not set in policy level, consumer can pass the backend application name in the http header. The http header settings can be useful for internally exposed APIs.
- Transaction Id and correlation Id can be set by user in the content variables (custom.transactionId and custom.correlId) with relevant values. However, if not set in the assembly, these values can be sent by consumers in http headers as well.
- Using all these information, it creates an audit payload and puts to MQ queue.
You can send this audit feed from MQ queue to any data store for storage and visualization. However, our tutorial will be limited to just putting the payload to MQ.
Basic Constructs of IBM API Connect User Defined Policy
Following image shows the basic construct and their respective description of user defined policy.
Let’s build AuditFW User Defined Policy
We’ll name our policy AuditFW. This policy will take few input parameters instructing the policy:
- whether to log the payload or not
- Logging sequence number
- Whether to log the payload from request.body or message.body
- Any custom message from the audit point
- Whether to use a different Queue manager other than the default one
- Whether to use a different queue other than the default one.
Build the MQ Queue Manager Object
We need to build a configuration file for the MQ Queue Manager object. We’ll reuse the same configuration file that we created in the previous article.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 |
top; configure terminal; mq-qm "AUDITQM" hostname x.x.x.x(1414) queue-manager "QM1" ccsid 819 channel-name "DEV.APP.SVRCONN" heartbeat 300 maximum-message-size 1048576 cache-timeout 60 no automatic-backout total-connection-limit 250 initial-connections 1 sharing-conversations 0 no share-single-conversation no permit-insecure-servers no permit-ssl-v3 ssl-cipher none no auto-recovery convert auto-retry retry-interval 10 retry-attempts 6 long-retry-interval 600 reporting-interval 10 no alternate-user polling-tolerance 10 xml-manager default ssl-client-type proxy exit |
Build the policy configuration file
Next, we will create a configuration file that’ll define the UDP. It will have following components:
- assembly-setvar (to set the input parameters): This is the first assembly policy of our UDP. It will have few parameters (as explained above) that will be set by the API developer at the development time. These parameters will drive the policy behavior. As this is the first step to be executed, you will notice that we have used correlation-path “$.x-ibm-configuration.assembly.execute[0]”
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 |
assembly-setvar auditfw_1.0.0_set-variable_0 reset title "set-variable" correlation-path "$.x-ibm-configuration.assembly.execute[0]" variable action set name "logsequenceparam" type string value "$(local.parameter.logsequence)" exit variable action set name "logpayloadparam" type string value "$(local.parameter.logpayload)" exit variable action set name "logpayloadfromparam" type string value "$(local.parameter.logpayloadfrom)" exit variable action set name "custommsgparam" type string value "$(local.parameter.custommsg)" exit variable action set name "targetappparam" type string value "$(local.parameter.targetapp)" exit variable action set name "queuemanagerparam" type string value "$(local.parameter.queuemanager)" exit variable action set name "queueparam" type string value "$(local.parameter.queue)" exit exit |
- assembly-gatewayscript: This is second assembly policy of our UDP. It’ll refer to the gateway script that will construct the audit message and put it to the queue.
|
1 2 3 4 5 6 |
assembly-gatewayscript auditfw_1.0.0_gatewayscript_1 reset title "gatewayscript" correlation-path "$.x-ibm-configuration.assembly.execute[1]" gatewayscript-location temporary:///filestores/extensions/gateway-extension/auditfw-persist-to-mq.js exit |
- api-rule: The API rule will combine two policies that we created above:
|
1 2 3 4 5 |
api-rule auditfw_1.0.0_main reset action auditfw_1.0.0_set-variable_0 action auditfw_1.0.0_gatewayscript_1 exit |
- assembly: Now, we will create an assembly using the above rule:
|
1 2 3 4 |
assembly auditfw_1.0.0 reset rule auditfw_1.0.0_main exit |
- assembly-function: Finally, we will create an assembly function that will wrap the input params and also the above created assembly:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 |
assembly-function "auditfw_1.0.0" reset summary "auditfw-policy_1.0.0" title "Audit Framework" parameter name "logsequence" description "Mention the log seuence" value-type string exit parameter name "logpayload" description "Will you the log the payload? Type either Yes or No" value-type string exit parameter name "logpayloadfrom" description "Mention message.body or request.body based on requirement. Default is message.body" value-type string exit parameter name "custommsg" description "Type your custom message" value-type string exit parameter name "targetapp" description "Type the traget app name" value-type string exit parameter name "queuemanager" description "Mention the queue manager object name if you don't want to use the default. Leave it empty otherwise" value-type string exit parameter name "queue" description "Mention the queue name if you don't want to use the default. Leave it empty otherwise" value-type string exit assembly auditfw_1.0.0 exit apic-gw-service user-defined-policies auditfw_1.0.0 exit |
Putting it all together, our configuration will look like below. Let’s save this configuration in a file named auditfw.cfg.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 |
assembly-setvar auditfw_1.0.0_set-variable_0 reset title "set-variable" correlation-path "$.x-ibm-configuration.assembly.execute[0]" variable action set name "logsequenceparam" type string value "$(local.parameter.logsequence)" exit variable action set name "logpayloadparam" type string value "$(local.parameter.logpayload)" exit variable action set name "logpayloadfromparam" type string value "$(local.parameter.logpayloadfrom)" exit variable action set name "custommsgparam" type string value "$(local.parameter.custommsg)" exit variable action set name "targetappparam" type string value "$(local.parameter.targetapp)" exit variable action set name "queuemanagerparam" type string value "$(local.parameter.queuemanager)" exit variable action set name "queueparam" type string value "$(local.parameter.queue)" exit exit assembly-gatewayscript auditfw_1.0.0_gatewayscript_1 reset title "gatewayscript" correlation-path "$.x-ibm-configuration.assembly.execute[1]" gatewayscript-location temporary:///filestores/extensions/gateway-extension/auditfw-persist-to-mq.js exit api-rule auditfw_1.0.0_main reset action auditfw_1.0.0_set-variable_0 action auditfw_1.0.0_gatewayscript_1 exit assembly auditfw_1.0.0 reset rule auditfw_1.0.0_main exit assembly-function "auditfw_1.0.0" reset summary "auditfw-policy_1.0.0" title "Audit Framework" parameter name "logsequence" description "Mention the log seuence" value-type string exit parameter name "logpayload" description "Will you the log the payload? Type either Yes or No" value-type string exit parameter name "logpayloadfrom" description "Mention message.body or request.body based on requirement. Default is message.body" value-type string exit parameter name "custommsg" description "Type your custom message" value-type string exit parameter name "targetapp" description "Type the traget app name" value-type string exit parameter name "queuemanager" description "Mention the queue manager object name if you don't want to use the default. Leave it empty otherwise" value-type string exit parameter name "queue" description "Mention the queue name if you don't want to use the default. Leave it empty otherwise" value-type string exit assembly auditfw_1.0.0 exit apic-gw-service user-defined-policies auditfw_1.0.0 exit |
Build the gateway script
Now, we will build the gateway script that will read the user input, context variables and build audit payload. Thereafter, it will put the payload into the queue. Let’s take a look at the gateway script file. We’ll name the configuration file as auditfw-persist-to-mq.js as mentioned in the auditfw.cfg file.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 |
var apim = require('apim'); var urlopen = require('urlopen'); var logsequenceparam = context.get('logsequenceparam'); var logpayloadparam = context.get('logpayloadparam'); var custommsgparam = context.get('custommsgparam'); var logpayloadfromparam = context.get('logpayloadfromparam'); var queuemanagerparam = context.get('queuemanagerparam'); var queueparam = context.get('queueparam'); var targetappparam = context.get('targetappparam'); if (!logsequenceparam){ logsequenceparam = 0; } if (!queuemanagerparam){ queuemanagerparam = 'AUDITQM'; } if (!queueparam){ queueparam = 'DEV.QUEUE.1'; } var queuemanagerparam = 'dpmq://'+queuemanagerparam+'/?'; //get the input data var data; if(logpayloadfromparam.toLowerCase() == 'request.body'){ data = apim.getvariable('request.body'); }else{ data = apim.getvariable('message.body'); } if (data){ data = data.toString(); } var tapp; console.log("targetappparam:"+targetappparam); if (targetappparam){ tapp=targetappparam; }else if(apim.getvariable('request.headers.x-tabadul-targetapp')){ tapp=apim.getvariable('request.headers.x-tabadul-targetapp'); } var transId = context.get('custom.transactionId'); if (!(transId)){ transId=apim.getvariable('request.headers.x-tabadul-transactionId'); } var correlId = context.get('custom.correlId'); if (!(correlId)){ correlId=apim.getvariable('request.headers.x-tabadul-correlid'); } var body={}; body.type='apicevent'; body.log_sequence=logsequenceparam; body.correlation_Id=correlId;//context.get('custom.correlId'); body.transaction_Id=transId; body.porg_details={}; body.porg_details.api_name=apim.getvariable('api.name'); body.porg_details.api_version=apim.getvariable('api.version'); body.porg_details.target_app=tapp; body.corg_details = {}; body.corg_details.developer_org_name=apim.getvariable('client.org.name'); body.corg_details.app_name=apim.getvariable('client.app.name'); body.corg_details.plan_name=apim.getvariable('plan.name'); body.corg_details.client_id=apim.getvariable('client.app.id'); body.api_end_point= {}; body.api_end_point.resource_url=apim.getvariable('request.uri'); body.api_end_point.resource_verb=apim.getvariable('request.verb'); body.api_end_point.query_string=apim.getvariable('request.querystring'); body.custom_audit_message=context.get('custommsg');; body.status_code=apim.getvariable('message.status.code')+" "+apim.getvariable('message.status.reason'); var contenttype = apim.getvariable('request.content-type'); if (data){ if(logpayloadparam){ var logpayloadparam = ""+ logpayloadparam.toUpperCase(); if(logpayloadparam.includes("YES")){ body.payload=data; } } } body.datetime=apim.getvariable('system.datetime')+apim.getvariable('system.timezone'); var inpMsg = JSON.stringify(body); var dpmqurl = { target: queuemanagerparam, requestQueue: queueparam, transactional: false, sync: false, timeOut: 10000, data: inpMsg }; urlopen.open (dpmqurl, function (error, response) { // handle the error when connecting to MQ if (error) { var msg = error + 'errorCode=' + error.errorCode; console.error("MQ error is %s", msg); //throw error; }else{ var responseCode = response.statusCode; var successMsg = "SUCCSS!! MQ status code is: " + responseCode; //console.log(successMsg); } }); |
Now, we will package all 3 files (queue manager configuration, policy configuration and gateway script) in a .zip file; we’ll name it as gwext.zip
Deploy the IBM API Connect user defined policy
We’ll deploy the policy in our local API connect LTE setup. So, let’s start the LTE by running following commands. Just to note, the commands are same for actual APIC setup as well; just that the argument values will change as per your environment setup.
apic-lte.exe start
Once the LTE starts, we will login to the API manager using admin user
$ apic login –server 127.0.0.1:2000 –username admin –password 7iron-hide –realm admin/default-idp-1
Thereafter, we’ll issue following command to apply the user defined policy in our API manager setup
apic gateway-extensions:create /C/gwext/gwext.zip –scope org –org admin –gateway-service datapower-api-gateway –availability-zone availability-zone-default –server 127.0.0.1:2000
If the command is executed successfully, we’ll see a result something like below
gateway-extension https://127.0.0.1:2000/api/orgs/42535da0-4c43-45cc-bed4-d190f4789235/availability-zones/4f8893e7-569b-4cb9-976a-9776d4abca1d/gateway-services/6236ded1-aff8-43ee-b159-545fd0c5b5eb/gateway-extension
To verify that, it is indeed applied, we will run following commands
$ apic gateway-extensions:get –scope org -o admin –gateway-service datapower-api-gateway –availability-zone availability-zone-default –server 127.0.0.1:2000 –output –
The result of the command will be as follows
type: gateway_extension
api_version: 2.0.0
name: gateway-extension
gateway_service_url: >-
https://127.0.0.1:2000/api/orgs/42535da0-4c43-45cc-bed4-d190f4789235/availability-zones/4f8893e7-569b-4cb9-976a-9776d4abca1d/gateway-services/6236ded1-aff8-43ee-b159-545fd0c5b5eb
scope: org
created_at: ‘2021-06-09T19:26:47.114Z’
updated_at: ‘2021-06-09T19:26:47.114Z’
org_url: ‘https://127.0.0.1:2000/api/orgs/42535da0-4c43-45cc-bed4-d190f4789235’
url: >-
https://127.0.0.1:2000/api/orgs/42535da0-4c43-45cc-bed4-d190f4789235/availability-zones/4f8893e7-569b-4cb9-976a-9776d4abca1d/gateway-services/6236ded1-aff8-43ee-b159-545fd0c5b5eb/gateway-extension
To delete the gateway extension, we can issue following command
apic gateway-extensions:delete –scope org -o admin –gateway-service datapower-api-gateway –availability-zone availability-zone-default –server 127.0.0.1:2000 –output –
Apply the policy to the gateway server by restarting the API Connect gateway service object. Complete the following steps on each Gateway server in the Gateway service.
- Log in to the Datapower (https://localhost:9091/dp/login.xml) using username admin and password admin; switch to the apiconnect domain
- Search for API Connect Gateway Service
- Set the administrative state to disabled
- Apply the changes
- Set the administrative state to enabled
- Apply the changes
- Save the configuration
Verify IBM API Connect User Defined Policy
In the user defined policy section of API Connect Gateway Service, we’ll notice the newly created policy
We’ll check the queue manager object is created as part of the user defined policy:
We’ll also check the gateway script file is present in the file management view:
Test IBM API Connect User Defined Policy
It’s the easy part. We’ll build an API with the newly created user defined policy.
- Launch the designer toolkit.
- Build a simple API, go to assemble section and notice the last policy in the policy pane. This is the UDP we just created. We can drag and drop this policy in the assembly and configure.
- PFB the swagger of the test API
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 |
swagger: '2.0' info: title: UDP Test API x-ibm-name: udp-test-api version: 1.0.0 x-ibm-configuration: cors: enabled: true gateway: datapower-api-gateway type: rest phase: realized enforced: true testable: true assembly: execute: - gatewayscript: version: 2.0.0 title: gatewayscript source: "var apim = require('apim');\r\nvar successMsg = \"Hello World\";\r\ncontext.message.body.write(successMsg);\r\ncontext.message.header.set('Content-Type', \"text/plain\");\r\ncontext.set('custom.correlId','CR123456');\r\ncontext.set('custom.transactionId','TR123456');" - auditfw: version: 1.0.0 title: auditfw logsequence: '1' logpayload: 'yes' logpayloadfrom: '' custommsg: udp test targetapp: mybackend queuemanager: '' queue: '' properties: target-url: value: 'http://example.com/operation-name' description: The URL of the target service encoded: false application-authentication: certificate: false basePath: /udp-test-api paths: /: get: responses: '200': description: success schema: type: string consumes: [] produces: [] put: responses: '200': description: success schema: type: string consumes: [] produces: [] post: responses: '200': description: success schema: type: string consumes: [] produces: [] delete: responses: '200': description: success schema: type: string consumes: [] produces: [] options: responses: '200': description: success schema: type: string consumes: [] produces: [] head: responses: '200': description: success schema: type: string consumes: [] produces: [] patch: responses: '200': description: success schema: type: string consumes: [] produces: [] securityDefinitions: clientIdHeader: type: apiKey in: header name: X-IBM-Client-Id schemes: - https security: - clientIdHeader: [] |
Conclusion
That was a pretty long tutorial. But I hope to have shared some knowledge.
Reference
- https://www.ibm.com/docs/en/api-connect/2018.x?topic=apdag-defining-packaging-your-user-defined-policydatapower-api-gateway
- https://www.ibm.com/docs/en/api-connect/2018.x?topic=apdag-publishing-your-user-defined-policy-datapower-apigateway
- https://github.com/ozairs/apiconnectpolicies/tree/master/udp-managed-obj
raj devershetty
June 9, 2021Nice Article! I think the below
var logsequenceparam = context.get(‘logsequenceparam’); should be
var logsequenceparam = context.get(‘logsequence’);
should match with value “$(local.parameter.logsequence)”
Sadruddin Md
June 10, 2021Hi, no. It’ll be logsequenceparam. The input in parameter local.parameter.logsequence is set to context variable logsequenceparam