Showing posts with label Python. Show all posts
Showing posts with label Python. Show all posts

Thursday, April 7, 2022

My takeaway from SHARE Dallas 2022 Hackathon for IBM Z - Part 1

Hiya! 👋

It's been some time since I wrote on my blog. Recently, I participated in SHARE Dallas 2022 Hackathon that happened between Mar 28, 2022 and Mar 29, 2022 and it was a great experience that I wanted to treasure. Hence this post 😀.

This Hackathon showed the modern ways of working on a Mainframe and it, for sure, fascinates the Mainframers as well as those who think of IBM Mainframes as legacy and old stuff.   

⚠ Before we start, I would like to let you know that this is going to be a long post because the takeaway was pretty huge for me. So, I'll be writing couple of posts to cover the entire experience (this being the first one)

The Hackathon was made available to participate for virtual attendees as well as in-person attendees of SHARE Dallas 2022 Event and no registration cost was involved. I participated virtually from India. 

How things began on the day of event? 

I received an email from the Hackathon's organizers with instructions to install the necessary tools  (given below): 

Discord - Most of the communication with organizers and fellow Hackers took place on Discord. 

VSCode with Zowe Explorer extension - This is an IDE. Zowe Explorer extension on VSCode was used throughout the Hackathon to interact with Mainframe and even with an IoT device. 

Postman - This tool allows to easily test and call REST-based APIs and web-enabled microservices. Though this tool was optional to install, I had it installed on my system and used it to GET and POST some API's. This gave an outlook on the structure of response after posting a request (more about this in the later part of this post) 

OpenVPN - The Hackathon wanted our personal system to be connected to the Montpellier VPN to access the Mainframe. 

In another email from the Organizers, I received my Credentials for the VPN and Z System. There was also a link provided in the same email to access the Challenge documentation. 

I just grabbed a coffee 🥤 and started reading out the 40 page Challenge documentation. 

The Challenge 💪

We were walked through the Catalog Manager Application running inside CICS and written in COBOL. The application had three basic functions:
  • Display all the items in a catalog 
  • Display detailed information of an item
  • Order an item.
There were some screenshots on the documentation which showed the CICS Screen interface to access the Catalog Manager application and its functionalities.

CICS Interface showing the Main Menu of the Catalog Manager Application.

CICS Interface showing the list of items in the Catalog, when '1. List Items' is selected from the Main Menu.

CICS Interface showing the detailed information of an item when an item is selected from the previous screen. 

CICS Interface showing the status of the order at the left side bottom of the screen. 

The overall challenge was to re-build this application using Python on z/OS and REST API's. The transactions still be handled by the CICS Transaction sever as they had enabled a technology (z/OS Connect Enterprise Edition) on the Hackathon LPAR to make the CICS functions available by means of REST API's. Don't fret if all these jargons doesn't make too much sense right now. You'll understand once they are unfolded in the later part of this post. 

Hacking through the tasks for Day 1

Task 1:

After connecting to the Montpellier VPN and setting up the profile for Hackathon in Zowe Explorer on VS Code, I started doing the first task, which was to implement the Traditional "Hello World" application. I used the Terminal to type Unix commands to create a new file (helloWorld.py) on the home directory of Unix System Services. 
There is a UNIX interface within z/OS called UNIX System Services, or USS, so you can log in through ssh and hammer out commands.

For writing the Python code, I used the IBM Z Open Editor instead of Unix terminal. To view/edit the files on IBM Z Open Editor, all you need to do is to locate the file that you just created, from Unix System Services (USS) section under Zowe, and click on it. The contents of the file will be shown on the right side on IBM Z Open Editor. 

Viewing the contents of helloWorld.py file on IBM X Open Editor.

I just wrote a one line command using the print() function in Python to display a message. To finish the Task 1, I used the python3 command on the Terminal of VS Code to run the Python script I had created. 

Output displayed after invoking the Python script using python3 command from USS Terminal.

It is essential that we get inside the directory where our Python code is saved, before running the code with the python3 command. 

This marked the completion of Task 1 ✅ 

Task 2:

To take things to the next level, Task 2 was all about creating our First web application on z/OS. I was introduced to the usage of Flask on Python. 

Flask is a popular web framework and it provides functions on Python for rendering an HTML page and handling incoming HTTP requests. 

I installed Flask on Python using the pip3 command. Post installation, I created a new directory under my Home directory (/u/ztecu85) and named it as webApp. Under webApp directory, two sub-directories namely static and templates were created. 

webApp directory is where all my project's files resided. As the challenge unfolded, we had to create some HTML files and those were stored under the templates sub directory. 

To complete the task, I created a new file named as webserver.py under webApp directory and wrote the code as shown below. 


Let me walk you through some of the important sections of this code.

from flask import Flask - This line imports Flask module to the project. 

app = Flask(__name__) - __name__ is a special variable in Python which returns the name of the current module (webserver.py).  This line of code helps Flask look for the relevant files needed by the application, such as static and template files. Flask manages to find the root path based on the value stored in __name__

@app.route('/') - This tells the Flask which URL should call the user defined Python function, index()that is defined in the next line. By passing '/' as an argument to the route() function of the Flask Class, we're binding the URL so that when we load the URL on the browser, we will be greeted with a message returned by index() function. 

As we have associated the URL - 10.3.20.138 and Port - 24320 with the index() function defined in Python through app.run() method, this 👇 is what happened when I ran this code from the Unix Terminal. 


Upon the execution of the code, there were several messages shown on the Terminal. One message showed a URL where the application was running. Pressing Ctrl + Clicking on the link took me to the URL where the message returned by the index() function was displayed. 

This task was a nice twist to the traditional 'Hello World' program that we printed the same message on a web application. 

Task 3: 

This task introduced me to the CICS part. As mentioned earlier, the three CICS functions were made available to us by means of three REST API's. The Challenge documentation for this task contained a table with detailed description of the API's. 

Though testing these API's were optional, I used the Postman tool to post the API's with the URL given for each function and receive responses to get familiarized with them. 

Following are the GIF's showing the responses received on Postman after posting the three API's. 

1. API to list all the items in Catalog

HTTP Verb:  GET

URL given to access this service: http://10.3.20.1:50780/catalogManager/v1.0/items?startItemRef={ }

Note that there were 21 items in the Catalog and each item had a unique Item Reference number (itemRef), starting from # 0010 up until 0210, in the increment of +10. The curly braces (highlighted in Red color) at the end of the URL means that it accepts a query parameter, which is an Integer between 10 and 210 (in the increment of +10). In the following GIF, I've assigned 10 to startItemRef at the end of the URL, so that the CICS application use the startItemRef value to get the next 15 items from the Catalog. 



2. API to get the detailed information on one Item:

HTTP Verb: GET

URL given to access this service: http://10.3.20.1:50780/catalogManager/v1.0/items/{ }

Posting this URL fetched the detailed information on one Item with itemRef that we have passed as query parameter at the end of the URL (The curly braces highlighted in Red color at the end of the URL means that it accepts query parameter). 



3. API to Order an Item:

HTTP Verb: POST

URL given to access this service: http://10.3.20.1:50780/catalogManager/v1.0/orders 

This service was different from the other two that while posting this service, we had to send along a Request in JSON format. This Request contained information about the item that we wish to order and the User ID with which we are ordering. CICS Application used this information to update the Catalog and pass the Order's status message. 

Upon posting the URL, following response was received.  


Task 3a:

In this task, we were asked to invoke the API, that list items from the Catalog, using Python. As the response had only 15 items, participants were asked to use the information in the lastItemRef key to create some code in Python that lists all 21 items. FYI, each Item in the response would have a lastItemRef key that holds the value of the last item's reference number. So, the idea was to post the API and get the first response. Then, make use of the lastItemRef key value from the 15th item of the first response and use this value as a query parameter and create the URL to be posted for the second response. 

It was required to combine the two results from the API calls. A clue that was given to the participants was to convert the responses into a Python Dictionary with the json() method; then add two dictionaries together using the extend() method. 

This is where I got stuck 😕 and couldn't get past this task on the Day 1. I slept over this problem and was able to fare better on the Day 2.

Hacking through the tasks for Day 2

Task 3a (continued): 

The response after posting the  API to list the items in the Catalog looked like the below:

 {  
   "data": {  
     "returnCode": 0,  
     "responseMessage": "+15 ITEMS RETURNED",  
     "inquireCatalog": {  
       "startItemRef": 10,  
       "lastItemRef": 150,  
       "items": [  
         {  
           "itemRef": 10,  
           "cost": "089.90",  
           "onOrder": 0,  
           "description": "Man's Waterproof Rain Jacket",  
           "department": 10,  
           "stock": 989  
         },  
         {  
           "itemRef": 20,  
           "cost": "089.90",  
           "onOrder": 99,  
           "description": "Women Waterproof Rain Jacket",  
           "department": 10,  
           "stock": 996  
         },  
         {  
           "itemRef": 30,  
           "cost": "040.90",  
           "onOrder": 0,  
           "description": "Sunglasses",  
           "department": 10,  
           "stock": 992  
         },  
         {  
           "itemRef": 40,  
           "cost": "079.90",  
           "onOrder": 0,  
           "description": "Windy Umbrella",  
           "department": 10,  
           "stock": 996  
         },  
         {  
           "itemRef": 50,  
           "cost": "004.99",  
           "onOrder": 0,  
           "description": "Hot Coffee",  
           "department": 10,  
           "stock": 990  
         },  
         {  
           "itemRef": 60,  
           "cost": "004.99",  
           "onOrder": 40,  
           "description": "Hot Chocolate",  
           "department": 10,  
           "stock": 999  
         },  
         {  
           "itemRef": 70,  
           "cost": "049.99",  
           "onOrder": 20,  
           "description": "Woman swim suit with fancy colors",  
           "department": 10,  
           "stock": 992  
         },  
         {  
           "itemRef": 80,  
           "cost": "049.99",  
           "onOrder": 0,  
           "description": "Man's swim suit also with fancy colors",  
           "department": 10,  
           "stock": 998  
         },  
         {  
           "itemRef": 90,  
           "cost": "019.99",  
           "onOrder": 0,  
           "description": "Thermoflask double stainless steel",  
           "department": 10,  
           "stock": 996  
         },  
         {  
           "itemRef": 100,  
           "cost": "009.90",  
           "onOrder": 20,  
           "description": "Icecream Family Pack",  
           "department": 10,  
           "stock": 994  
         },  
         {  
           "itemRef": 110,  
           "cost": "004.40",  
           "onOrder": 0,  
           "description": "Iced Tea",  
           "department": 10,  
           "stock": 995  
         },  
         {  
           "itemRef": 120,  
           "cost": "025.99",  
           "onOrder": 0,  
           "description": "Mineral Sunscreen",  
           "department": 10,  
           "stock": 997  
         },  
         {  
           "itemRef": 130,  
           "cost": "014.85",  
           "onOrder": 0,  
           "description": "Rain Cap Size Small",  
           "department": 10,  
           "stock": 998  
         },  
         {  
           "itemRef": 140,  
           "cost": "014.85",  
           "onOrder": 0,  
           "description": "Rain Cap Size Medium",  
           "department": 10,  
           "stock": 999  
         },  
         {  
           "itemRef": 150,  
           "cost": "014.85",  
           "onOrder": 45,  
           "description": "Rain Cap Size Large",  
           "department": 10,  
           "stock": 996  
         }  
       ]  
     }  
   }  
 }  

Per Python, this data is structured as a nested Dictionary. 

💡 Python Dictionaries store data values in key:value pair and are written with Curly braces. 

"data" is a dictionary with three key:value pairs viz. "returnCode", "responseMessage" and "inquireCatalog". 

"inquireCatalog" itself is a Dictionary with three key:value pairs (again 😐). Make note of the third key, "items" for the value of this key is a Python List of all the 15 items of the response. 

💡 Python Lists are created using square brackets and are used to store multiple items in a single variable. 

Responses from the two API calls resulted in two Python Dictionaries but I was not able to combine these dictionaries together. 

Discord came as savior 🙏. I posted this issue there and the organizers quickly helped me understand that most of the info from the Dictionary won't be used besides to make sure the request went well and the focus should be only on the Python List with the details of the items in the Catalog. 

I created a new file named as webRequestHandler.py under the webApp directory and wrote the following code to get the result needed to mark this task as complete ✅.


I've added comments to each line of the code for your understanding. Still, I would like to highlight few sections of the code. 

Line #13 is used to create a Python dictionary called header which stores the authentication token given by the Organizers of Hackathon. Requests to the back-end systems only with this Auth token are deemed as legal. Without the Auth token, we will not be able to send/post API's and get the responses from the back-end systems. 

This 👇 is what happened after running this Python script. 



The output was a Python List of all the 21 items that were on the Catalog and this marked the task as complete ✅.


In the next task, we will make this response look nicer 👌 using Flask HTML rendering capabilities. 

I'm hitting the pause button on this experience journey for now. Remaining tasks will be covered on the next post titled as "My takeaway from SHARE Dallas 2022 Hackathon for IBM Z - Part 2". Keep watching this space 👀

Thanks for reading! Should you have any questions/feedback, please post them in Comments section below. 


References:
1. Challenge Documentation provided for the Hackathon. 
2. GeeksforGeeks, Programiz and W3Schools for Python related information.

Screenshot/GIF courtesy: Mainframe access obtained for SHARE Dallas 2022 Hackathon for IBM Z.


Friday, February 26, 2021

How to sort on the bits of a byte using IBM DFSORT?

Recently, I came across a DFSORT coding challenge titled as "Odds & Evens".

The problem statement goes like this - Given a file with valid sequence numbers in columns 1 thru 6, sort the file so the corresponding output has all the even numbered records first, followed by all the odd numbered records.

I put on my thinking cap 🎩 for a while and came up with the following answer:

 ----+----1----+----2----+----3----+----4----+----5----+----6----+----7--  
 ***************************** Top of Data ******************************  
 //Z01071A JOB 1,NOTIFY=&SYSUID                       
 //STEP01  EXEC PGM=SORT                           
 //SORTIN  DD *                               
 000001                                   
 000002                                   
 000003                                   
 000004                                   
 000005                                   
 000006                                   
 000007                                   
 000008                                   
 000009                                   
 000010                                   
 000011                                   
 000012                                   
 000013                                   
 000014                                   
 000015                                   
 000016                                   
 000017                                   
 000018                                   
 000019                                   
 000020                                   
 //SORTOUT DD SYSOUT=*                            
 //SYSOUT  DD SYSOUT=*                            
 //SYSIN   DD *                               
   INREC IFTHEN=(WHEN=GROUP,RECORDS=2,PUSH(10:SEQ=1))            
   SORT FIELDS=(10,1,CH,D,1,6,CH,A)                     
   OUTREC FIELDS=(1,6)                            
 /*                                     
I just formed a group of 2 records and PUSH'ed sequence numbers (of 1 byte) for each record of the group. As there are only 2 records in a group, the sequence number will be 1 for the first record and 2 for the second record. The sequence number will be restarted from 1 when a new group is started. 

Then, I used the sequence number field (at col 10) in the SORT statement to sort it in the descending order so that all the records with sequence number as 2 will be at the top.  A secondary sort was applied on the first 6 bytes. 

Submitting this job, I got the following output,
  COMMAND INPUT ===>                                            SCROLL ===> CSR   
 ********************************* TOP OF DATA **********************************  
 000002                                       
 000004                                       
 000006                                       
 000008                                       
 000010                                       
 000012                                       
 000014                                       
 000016                                       
 000018                                       
 000020                                       
 000001                                       
 000003                                       
 000005                                       
 000007                                       
 000009                                       
 000011                                       
 000013                                       
 000015                                       
 000017                                       
 000019                                       
 ******************************** BOTTOM OF DATA ********************************  
WHEN=GROUP is one amazing feature in DFSORT, thanks to Frank Yaeger from IBM DFSORT Development team, as he is one of the brains behind the invention of WHEN=GROUP.

We got the answer. Are we done here?

Nope, I'm just done with the Intro. 

The main reason behind the idea of writing this blog post was that when I was looking at other answers, I stumbled upon a solution which had a syntax that I've never seen before. It goes like this: 
SORT FIELDS=(6.7,0.1,BI,A),EQUALS
Most of us would use the SORT control statement to specify the control field based on which the sorting should take place. We provide,
  1. the position of the field within the record
  2. the length of the field (in bytes)
  3. the format of the data in control field
  4. the order in which field must be sorted (ascending or descending)
Let's take the first 2 items. The position of the field within the record is the byte positon relative to the beginning of the record. The length of the field is usually expressed in integer numbers of bytes. We deal with Bytes (and a pet lover has to deal with bites 🐶 sometimes).

Let's take a look under the hood 🔧


A byte consists of 2 nibbles and each nibble is 4 bits long. A bit is either 0 or 1.

IBM Mainframe uses the EBCDIC character encoding. Each character is represented by its 8 bit EBCDIC Code. When we turn on the Hex mode, we will be able to see an hex value for each byte. When the hex value of each byte is converted to binary, we'll get the corresponding bits. 

For example, 

SRINI becomes,
E2        D9        C9        D5        C9                  Hexadecimal
11100010  11011001  11001001  11010101  11001001    Binary

📣IBM DFSORT allows us to sort on the bits of a byte with "bytes.bits" notation. 

How to sort on the bits of a byte?

Now, we know that each character has got an 8 bit binary value, we can use the bytes.bits notation to sort using bits.
  • First, specify the byte location relative to the beginning of the record and follow it with a period.
  • Then, specify the bit location relative to the beginning of that byte. Remember that the first (high-order) bit of a byte is bit 0 (not bit 1); the remaining bits are numbered 1 through 7.
In SORT FIELDS=(6.7,0.1,BI,A),EQUALS statement,
6.7 - says that the starting postion is the last bit in byte 6. 
0.1 - says that the length is 1 bit. 
BI  - for Binary format as we want to sort on bits
A  - for Ascending order. 

But why 6.7 as the start position of the control field? 

That's because by looking at the 6th byte of every sequence number, we can say whether that's an even number or odd number.

Example:
000001 - 👀 -> that's an odd number
000002 - 👀 -> that's an even
000003 - 👀 -> that's an odd
000004 - 👀 -> that's an even
000005 - 👀 -> that's an odd
000006 - 👀 -> that's an even. I'm tired😑
....
....
.... and so on.

Another significance is that for each even number, the Least significant bit (the last bit) is 0 and for each odd number, it's 1. 

Example:
1         EBCDIC character
F1        Hexadecimal  
11110001  Binary

2         EBCDIC character
F2        Hexadecimal
11110010  Binary

3         EBCDIC character   
F3        Hexadecimal
11110011  Binary

4         EBCDIC character
F4        Hexadecimal
11110100  Binary

Hence, if we sort the last bit of 6th byte in ascending order, we would get all the even numbered records first, followed by the odd numbered records. 

The EQUALS parameter is coded in the SORT statement to preserve the original sequene in the output. If EQUALS is not coded, then the output will have all the even numbered records first, followed by the odd numbered record but the even/odd numbered records will not be in sorted order.

Let's try running this SORT operation using Python 🐍


We can make use of the Python API's provided by ZOAU to run the SORT operation. Z Open Automation Utilities (abbreviated to ZOAU) lets you perform many tasks on z/OS without needing to get into JCL. IBM has developed a bridge between Python and z/OS by creating API's for Python which allow Pythonistas to access z/OS resources. 

Before we start, we need the following stuff to run the SORT operation from Python:
  1. VS Code with Zowe explorer and IBM Z Open Editor extensions.
  2. Access to Zowe explorer.
  3. Access to USS (Unix System Services). 
  4. Little bit of Python Skills.
Note: Access to Zowe explorer and USS can be obtained when you sign up for MTM2020.

First, we need to create a new file under your home directory (/z/zxxxxx) in Unix Sytem Services. Use the touch command to create a new file.

I created one using this command, touch run_sort.py. Then, I used the IBM Z Open Editor to write the following code inside this file.

 
I've used Trinket to embed the Python code in this blog post. Note that you may not be able to RUN 🏃 this script as the ZOAU utilities for Python aren't available in Trinket.

Let's walk through the code.

Lines 1 thru 3: The required ZOAU libraries for Python are imported so that you can use them in your code. 

Lines 5 thru 9: Line #5 uses the os.getenv() method in Python with 'USER' as argument. As the operating system that Python is running under is z/OS, USERID variable is assigned with your TSO user ID. 
Lines 6 thru 9 has got 3 variables of string type to store the dataset names. 

Lines 11 thru 28: Lines 11 thru 28 mimics the functionality of IEFBR14 utility. These lines delete the datasets before creation. We make use of the zoautil_py.Datasets module which has got several dataset related functions like create, delete, exists and so on. 

Lines 30 thru 41: Writes data into the SORTIN and SYSIN datasets. 
Line #31 defines an empty list called num.This list is created to store the sequence numbers from 1 to 20. Read more about lists and how to access the elements in a list 👉 here
Lines 34 and 35 creates sequence numbers from 1 to 20 with the help of for loop and range() function in Python. zfill() method is used to populate leading zeros. As zfill() method can be applied only on string data, the numbers are type converted to string using str() function (in line #35). All the sequence numbers are appended to the list, num. The list is then written to the SORTIN dataset.
Lines 40 and 41 writes the SORT statements to the SYSIN dataset. The write functionality is achieved via zoautil_py.Datasets module.

Lines 43 thru 53: Line 44 creates an empty list called dd_names to store the DD names that are needed for the SORT program to run. 
In Line 53, MVSCmd.execute API is called to run the program SORT with arguments MSGPRT=CRITICAL,LIST (which goes to the PARM parameter in EXEC statement) and the list of DDStatements created in lines 47 thru 50. When this instruction is executed, a job might be submitted on z/OS in the background. 

Line #55 checks the return code from MVSCmd.execute API call. If it's zero, a message is displayed in the terminal and the output dataset (SORTOUT) created from the previous execution is read. 

When the program is run with python3 run_sort.py command in terminal, we get the following output.

Note: Click on the picture to get an enlarged view. 

The records in the output dataset are displayed in the terminal after running the Python code. The even numbered records are at the top, followed by the odd numbered records.

The datasets that were created from Python can also be accessed from the terminal. The datasets are shown below:

The input dataset to the SORT program. The sequence numbers were generated in Python.


The SYSIN input to the SORT program. The SORT statements were written from Python.


The output dataset. 

We have reached the bottom of this post and we discussed about two things:
  1. How to sort on the bits of a byte using IBM DFSORT?
  2. How to perform DFSORT operation using Python? 
I hope the content in this post was helpful to you. Please post your questions/suggestions in the Comments section of this post. 

Thx and Happy Weekend!


Wednesday, October 7, 2020

Master the Mainframe 2020 | Level 2.9: ZOAU2: Power Through Python

Hiya! Welcome to my blog😃

Master the Mainframe has been a lot interesting this year (2020) as we are using Visual Studio Code (for the first time) to establish a connection with z/OS and work with data sets and jobs, thanks to Zowe™ and IBM Z Open Editor extensions. We're now able to use the mouse to scroll through the contents of a data set😎.

Picture 9.1: Viewing the contents of a data set using IBM Z Open Editor in VS Code.

In this blog post, I want to share my impressions on using Python🐍 and Z Open Automation Utilities (abbreviated to ZOAU) to build an app to validate credit card data (This is the challenge in Level 2.9 of MTM2020). 

💡: ZOAU lets you perform many tasks on z/OS without needing to get into JCL. 

Before getting started with Zowe and Python, we will be building the logic using COBOL; we will then compile and run the COBOL program with the help of  JCL.  Then, we will be looking at Python and ZOAU way of building the same logic. In this way, I believe you will be able to understand the differences better. You'll also get answers to  some questions like 'Why Python?', 'Why ZOAU?', 'Why not JCL and COBOL?' and so on..

Fasten your seat belts, let's take-off now! 

The requirement is to read an input file (MTM2020.PUBLIC.CUST16 is the input data set name in MTM2020 system) and print a report with list of records that has got an invalid Credit card number (aka Payment Card Number). 

The input data set conforms to Track 1 Format B of ISO/IEC 7813 format 👉(https://en.wikipedia.org/wiki/ISO/IEC_7813). 

Picture 9.2: A snap from Wikipedia that shows the structure of Track 1 Format B. I've used this structure to come up with the record layout of the input file for COBOL.

Using COBOL program and JCL:

The COBOL program prepared by this Author😉 is as follows: 
Picture 9.3: The COBOL program reads the input file; implements Luhn Algorithm to find if a credit card number is vaild or not; if invalid, the record is written to the output file as a report. 

What the COBOL program does? 

  • An input file is being read (the record layout of the input file, INFILE-REC is based on Track 1 Format B structure of ISO/IEC 7813). 
  • For each record that is being read, control is passed to 100-IS-INVALID-PARA. This para implements the Luhn Algorithm. 
  • The first PERFORM loop within 100-IS-INVALID-PARA computes the sum of digits present in the odd number positions in PYMT-CARD-NUMBER. For example, if the value in the PYMT-CARD-NUMBER data item is 1234567890223457846, then 1 + 3 + 5 + 7 + 9 + 2 + 3 + 5 + 8 + 6 = 49 will be stored in WS-CHECKSUM at the end of the first perform loop. 
  • The second PERFORM loop within 100-IS-INVALID-PARA multiply the digits present in the even number positions, by 2; if the resulting value is greater than 9, then 9 is subtracted from the resulting value (Please don't ask ME why we have to do this. IBM Scientist Hans Peter Luhn, created this algorithm😊); accumulate the resulting value by adding it to WS-CHECKSUM data-item. 
  • After the 2 PERFORM loops, divide the final value in WS-CHECKSUM data item by 10. You should assign a data item for holding the remainder. If the remainder is 0, the value in PYMT-CARD-NUMBER is valid. Else, it is invalid. 
  • When an Invalid credit card number is found, control is passed to 200-WRITE-OUTPUT-PARA to write the whole record with the invalid credit card number, to the output file. When the control is passed to 200-WRITE-OUTPUT-PARA for the first time, the header record and an empty line are written to the output file. 

I was not aware of Luhn algorithm until I started this challenge. A pretty interesting one. The Luhn Algorithm is an efficient method of checking if a credit card number is valid, locally, without needing to have the bank or financial institute process it. This way, cards can be checked directly on a web page for mistakes in typing or copying digits. More information about Luhn algortihm can be found 👉 here.

After finishing the COBOL program, we have to prepare a JCL to compile the program and run it by providing input and output files. The following JCL does that.


After the compilation and execution of the program, we get the following output.

To read an input data set with magnetic stripe data that you might find on a credit card (if it doesn’t have a chip) and to print a report out of it with the list of invalid credit card numbers, what all we did? We wrote a COBOL program, prepared a JCL to compile the program and execute it. I call this the  usual Mainframe Way (not The Milky Way😉). 

Let's do the same task using Zowe™ and Python. This is what Level 2.9 in MTM2020 is all about. 

Using Zowe™ and Python🐍:

ZOAU supports shell scripts, Python and Node.js. The commands available through Z Open Automation Utilities in Python can be found 👉 here. In Layman's terms, IBM has developed a bridge between z/OS and Python by making these commands available for us to use within the Python scripts. We use 👉 zoautil_py.Datasets module extensively in this challenge. Modules in Python will have a set of functions, classes and variables defined in it. zoautil_py.Datasets module has got several functions like, 
  • create(name, type, size, format, class_name, length, offset) - Used to create a data set in z/OS
  • delete(dataset) - Used to delete a data set in z/OS
  • copy(source, target) - Used to copy the source data set into destination data set
You just have to refer to the corresponding function and pass necessary arguments in your Python script to perform the task it is intended to do. 

Building the logic using Python:

Note: In order to access python scripts from your home directory, /z/zxxxxxyou should be signed up to 👉 MTM2020 and should've finished the challenges till Level 2.4. 
There's already a python script (cc_check.py) available under your home directory (/z/zxxxxx) in Unix Sytem Services (abbreviated to USS). There are LOTS of comments in the script to help you understand what the code is doing. You just have to build, upon this foundation, a logic which would write the invalid credit card number to the output file.  
💡: A comment in Python starts with the hash character, #
I've used Trinket to embed Interactive Python in this blog post and I've copy pasted the contents of cc_check.py file in Trinket (And nope! The script shown below isn't the solution for Level 2.9😐). I'll do my best here to make you understand what needs to be done to complete Level 2.9. 

Note that you may not be able to RUN 🏃 this script in Trinket as the ZOAU utilities for Python aren't available in Trinket.
 
 
Lines 1 thru 7: The necessary ZOAU libraries for Python are imported so that you can use them in the script. 

Lines 9 thru 16: The input data set, MTM2020.PUBLIC.CUST16 is being read into a variable called cc_contents. Line #12 uses the os.getenv() method in Python with 'USER' as argument. As the operating system that Python is running under, is z/OS, USERID variable is assigned with your TSO user ID. In Line #13, 2 strings are concatenated with '+' operator and the result is stored in output_dataset variable. Some functions (like exists, delete) in the Datasets module are being used to check if the output data set is already existing in the Mainframe. If yes, then it is deleted.  

Lines 17 and 18: Read👀 those comments in the 2 lines as they help you figure out what needs to be done.

Lines 21 thru 28: def keyword in Python is used to define functions. You all know that function is a group of statements which are intended to perform a specific task. This function's name is is_even and this function receives one argument from the caller. The purpose of this function is to check if the number passed as an argument by the caller, is even or odd. If it is even, the function returns 'True' to the caller. Else, it returns 'False'. Your task is to edit this script so that instead of finding the even numbers, it implements the Luhn algorithm to find the six invalid entries in the input file. You may use the sample code in luhn.py file to perform the checking logic. 

Lines 30 thru 35: Each record in the data set is stored as an element in a List named as cc_list. Like how we read records one by one in COBOL, a for loop in Python is being used to read the elements of cc_list, one by one. cc_line variable is used to hold the element in any given iteration of the for loop. Like how we use Reference Modification in COBOL to extract a portion of the string, slicing technique (in Line 33) is being used on the List element to extract the Payment card number. The extracted card number is then passed as an argument to is_even function.
💡: List is one of the most frequently used datatype in Python. The elements are stored in a list within a square brackets ([ ]). Lists can have any number of items in it and the elements may be of different data types. Read more about lists and how to access the elements in a list 👉 here.

Lines 38 thru 57: Pretty much self explanatory as there are a lot of comments. Once you're ready to write the 6 invalid credit card numbers to the output file, uncomment line 57. 

And, there you go! refer luhn.py file and implement the Luhn algorithm in the place of is_even function. Bonus: Pay attention whenever you find the word hint in a comment line.

With Python and ZOAU utilities, it required only ~30 lines of code to build the logic that will look for the invalid credit card numbers from a Mainframe Input data set and write a report consisting of the invalid entries to an output data set. Power Through Python💪
You just witnessed one of the newest innovations in z/OS, the introduction of IBM Z Open Automation Utilities (ZOA Utilities) in ACTION 💥. ZOAU is an alternative way of interacting with tasks on z/OS through scripting instead of writing JCL's and submitting jobs. 

Hope you liked this post. Feel free to add your comments below. Thx👍

Screenshot Courtesy: Mainframe access💻 obtained via MTM2020 contest run by IBM.


Friday, July 13, 2018

Static and Dynamic call in COBOL and their compile JCL's

Much information on this topic is available in Google, but I personally felt that they are all scattered here and there. In this blog post, I've tried my best to explain the difference between Static and Dynamic call in COBOL and how to compile the COBOL programs that involve CALL statements.

We use CALL statements in COBOL to call sub-program(s).


What happens during a call?
Control from the Main program/Calling program is passed to the sub-program/Called program. The control however is returned back to the Main program, once the sub-program is done. EXIT/GO BACK statements are coded in the sub program to return the control back to the main program. 

Main program: Calling program.
Sub program: Called program. 


CALL statement in COBOL is usually coded inside the PROCEDURE DIVISION and in AREA B, as like the other statements. If you wonder what I just wrote in the previous line, then I strongly recommend you to check this link. If you wish to check the link later, PROCEDURE DIVISION in COBOL is where all the statements that does the processing, resides. AREA B is nothing but a rule that you should follow while writing COBOL programs. Certain entries must begin in AREA A (Columns 8 - 11) and others like CALL, must begin in  AREA B (Columns 12 - 72).


Right. Here we go!


What are the advantages in calling sub programs? 
Modularity: Anything which is modular is cool! Take a Desktop computer as example. There are various modules like RAM, Graphics Card, hard drive etc. If something doesn't work the way you want, you can just change that module. The sub programs that are being called from the Main program are the modules here. We can do changes to the sub program without modifying the main program. 

Reusability: Avoids duplication of effort.

Let's do a deep dive into COBOL's static and dynamic calls:
To explain calls, I'll be using a sample COBOL program to calculate the Tax amount (CGST+SGST) and the total amount payable while taking Amount and Tax percentage (GST) as Input to the main program. The tax amount and total amount payable will be calculated inside the Sub program. 

Static call and dynamic call both does the same stuff - to call the sub program. But, there are slight differences between the two and I've listed them below.



Static Call:
  • The called sub program is link-edited along with main program. 
  • The sub program's name will be enclosed within single quotes in the CALL statement. 
      • Ex: CALL 'SUBPGM' USING A B C. SUBPGM is the sub program's name. A, B and C are the arguments. 
  • Any changes made to the sub program will require the main program to be compiled along with the sub program. 
The main program is shown in Picture 3.1. Amount and GST percentage are passed as Input to the main program from the run JCL. 

Notice the CALL statement in line #17. This is CALL literal, where literal is the explicit name of the sub program, in this case, PGMB. 3 arguments are passed to the sub program; Amount, GST percentage and Total amount (for holding the total amount value i.e., Amount + Tax).

Note: Click on the picture to get an enlarged view.

Picture 3.1: Main program


The sub program is shown in Picture 3.2. Linkage section is defined to handle the data items that are passed as arguments to the sub program from the main program. In the PROCEDURE DIVISION, tax amount is calculated using Amount and GST percentage. GST is split into 2 components, CGST and SGST. Tax percentage is calculated for each component. Then, tax percentage is applied to the amount to derive total tax amount. Total amount payable is computed by adding amount and tax amount. LS-TOT-AMT data item in the sub program holds the final value. The corresponding data item of LS-TOT-AMT is used in the main program to display the Total amount payable. 
Picture 3.2: Sub program

GST helped me write a memory efficient program!
The Goods and Service Tax (GST) is a value-added tax levied on most goods and services sold for domestic consumption. The GST  shall have two components: one levied by the Centre (referred to as Central GST or CGST), and the other levied by the States (referred to as State GST or SGST). Since tax will be shared equally between the Central and State Government, I thought of using REDEFINES in COBOL for one of the tax data item. In the Picture 3.2, WS-SGST-P redefines WS-CGST-P, so it is enough to calculate the tax percentage for one data item i.e., WS-CGST-P. The other one (WS-SGST-P) will hold the same tax percentage as it shares the same memory as that of WS-CGST-P data item. 

Compile and link-edit the sub program(PGMB) as shown in Picture 3.3.
Picture 3.3: Compile and Link edit the Sub program


In COBOL's Static Call, Main program and Sub program are tied together in a same Load module:
When compiling the main program(PGMA), do a composite link by adding INCLUDE statements in the SYSIN of IEWL program, as shown in Picture 3.4. We are instructing the Link-edit program (in lines 35 thru 37) to include the load of PGMB from a private library (PRIVLIB) while creating the load module for PGMA. 
Picture 3.4: Do a Composite link of multiple COBOL programs and create an executable. 

Compile and Link-editing are done. It's time to run the main program.
Picture 3.5: Run JCL

The result, Picture 3.6. So much of tax!
Picture 3.6: Result!


Dynamic call:
  • The main program and the called program are part of different load modules i.e.sub program is not link-edited with the main program. 
  • If you make any changes to the sub program, you will only compile and link-edit the sub program. 
Notice the CALL statement in Picture 3.7, line #17. This is CALL identifier, where identifier is the data item, in this case, it is WS-PGM which contains the name of the sub program. 
Picture 3.7: Changes in the Main program for Dynamic call



Main program and sub program are compiled and link-edited separately. While compiling the main program, Compiler option DYNAM is provided in the PARM list, as shown in Picture 3.8. 
Picture 3.8: Compile Main program with compiler option as DYNAM. 

I obtained same results as shown in Picture 3.6, when calling the sub program dynamically. 

Key points to be noted: 
  • Default compiler option is NODYNAM. All calls made are static.
  • With DYNAM as compiler option, CALL literal statement calls the sub program dynamically. 
  • CALL identifier type calls are always dynamic.  

Same Stuff using Python:
I'm learning Python and I've just coded the example program in Python too. Feel free to use the interactive console on the right side of Trinket's tool to run Python. You have to call the main function and pass two arguments, Amount & GST %, as input. 
Ex: >>> main(3000,12)



Hope you found this post useful. Should you have any questions or suggestions, please share it in the Comments section below. thx.