Compressing SMF Data for Transfer

Navigation:  Table of Contents > Using EasySMF > Loading SMF Data >

Compressing SMF Data for Transfer

Previous pageReturn to chapter overviewNext page

Data can be compressed on z/OS to decrease the time required to transfer data to the PC. Typical compression ratios are about 10:1, so transfer time could be reduced by around 90%. Compressed data needs to be in Gzip format when using the EasySMF FTP client.

 

Zip format is supported if you transfer the data to a PC file and load the data from the file.

 

Any product that produces Gzip format can be used to compress the data for loading via FTP into EasySMF. The compressed data must include the RDW in the SMF records.

 

Sample Jobs

 

Sample jobs are provided to use Gzip from the IBM Ported Tools with Dovetailed Technologies Dataset Pipes, and to use a Java program with the JZOS Toolkit.

 

Gzip with Dovetailed Technologies Dataset Pipes

 

Installation

 

Install the Dovetailed Technologies Co:Z Toolkit using the instructions on their website. SSH setup is not required for this job, but can be used to load data remotely over SSH using a job running on z/OS.

 

Gzip can be downloaded from the IBM Ported Tools site, and extracted into a HFS directory.

 

Sample Jobs

 

Sample jobs are delivered in the EasySMF samples directory. They can be accessed via the Windows Start Menu using Start -> All Programs -> EasySMF -> Samples.

 

gziptodataset.txt

 

gziptodataset.txt contains JCL to compress data with Gzip into a dataset that can be loaded via FTP in EasySMF

 

//S1        EXEC PGM=COZBATCH                          

//STEPLIB   DD DISP=SHR,DSN=SYS1.COZ.LOADLIB           

//SMFIN     DD DISP=SHR,DSN=smf.dataset                

//* Output must be fixed records, any LRECL            

//* for EasySMF Load via FTP                           

//SMFOUT    DD DISP=(NEW,CATLG),DSN=smf.daatset.gzip,  

//       UNIT=SYSDA,                                   

//       RECFM=FB,LRECL=1024,SPACE=(TRK,(100,100),RLSE)

//SYSOUT    DD SYSOUT=*                                

//STDOUT    DD SYSOUT=*                                

//STDERR    DD SYSOUT=*                                

//STDIN     DD *                                       

/usr/local/coz/bin/fromdsn -l rdw DD:SMFIN | \         

  /usr/local/bin/gzip  | \                             

  /usr/local/coz/bin/todsn -b DD:SMFOUT                

//                                                     

gziptodataset.txt

 

Gzip output is a stream format rather than records, so the actual LRECL of the output file doesn't matter. Compressed data must use fixed length records rather than variable length, to prevent Gzip dataset RDWs being included with the data when it is transferred by EasySMF.

Typically, the last record in the FB dataset will not be filled and will be padded with binary zeros. This extra data will be ignored.

 

gziptofile.txt

 

gziptofile.txt contains JCL to compress data with Gzip into a HFS file that can be loaded via FTP in EasySMF

 

//S1        EXEC PGM=COZBATCH                 

//STEPLIB   DD  DISP=SHR,DSN=SYS1.COZ.LOADLIB 

//SMFIN     DD DISP=SHR,DSN=smf.dataset       

//SYSOUT    DD SYSOUT=*                       

//STDOUT    DD SYSOUT=*                       

//STDERR    DD SYSOUT=*                       

//STDIN     DD *                              

/usr/local/coz/bin/fromdsn -l rdw DD:SMFIN | \

  /usr/local/bin/gzip > /tmp/smfdata.gz       

//                                            

gziptofile.txt

 

Java with JZOS Toolkit

 

A Java program to compress SMF data is supplied in the EasySMF samples folder at Start -> All Programs -> EasySMF -> Samples, file SMFCompress.jar. The program is compiled for Java V7. The jar file needs to be transferred using binary mode to a HFS file on z/OS.

 

Java source for the program is provided in file Compress.java.

 

JZOS Toolkit Installation

 

The JZOS Toolkit should be set up according to instructions in the Installation and User's Guide. This involves copying the load module from <JAVA_HOME>/mvstools into a PDS/E, and copying and customizing the PROC and JCL from <JAVA_HOME>/mvstools/samples/jcl.

 

Running the Job

 

JCL to run Java programs is distributed with the JZOS Toolkit in <JAVA_HOME>/mvstools/samples/jcl/JVMJCL70.

 

Make the following changes to JVMJCL70:

 

Change JAVACLS on the EXEC statement to '-jar /<directory>/SMFCompress.jar' where <directory> is the location of the jar file uploaded from the PC.

 

//JAVA EXEC PROC=JVMPRC70,                                 

// JAVACLS='-jar /<directory>/SMFCompress.jar'             

 

Add INPUT and OUTPUT DD statements to the step:

 

//INPUT     DD DISP=SHR,DSN=smf.dataset                    

//* Output must be fixed records, any LRECL                

//* for EasySMF Load via FTP                               

//OUTPUT    DD DISP=(NEW,CATLG),DSN=compressed.smf.dataset,

//       UNIT=SYSDA,                                       

//       RECFM=FB,LRECL=1024,SPACE=(TRK,(100,100),RLSE)    

 

Run the job.

 

Source

 

Source for the Java compression program is provided in case you wish to make changes, or compile it to run with a different version of Java.

 

import java.io.ByteArrayOutputStream;

import java.io.IOException;

import java.io.OutputStream;

import java.util.zip.GZIPOutputStream;

import com.ibm.jzos.ZFile;

 

public class Compress {

  public static void main(String[] args) throws Exception {

       ZFile zFileIn = new ZFile("//DD:INPUT", "rb,type=record,noseek");

       ZFile zFileOut = new ZFile("//DD:OUTPUT", "wb,noseek");

      long count = 0;

      long bytesIn = 0;

      long bytesOut = 0;

      try {

          byte[] recBuf = new byte[zFileIn.getLrecl()];

          int nRead;

           ByteArrayOutputStream byteStream = new ByteArrayOutputStream();

           GZIPOutputStream zipStream = new GZIPOutputStream(byteStream);

          while ((nRead = zFileIn.read(recBuf)) >= 0) {

               count++;

               bytesIn += nRead + 4;

              writeRDW(nRead + 4, zipStream);

               zipStream.write(recBuf, 0, nRead);

              if (byteStream.size() > 32 * 1024) // arbitrary size for chunks

                                                  // to write

                                                  // actual value will be

                                                  // slightly greater

               {

                   zFileOut.write(byteStream.toByteArray(), 0,

                           byteStream.size());

                   bytesOut += byteStream.size();

                   byteStream.reset();

               }

           }

           ;

           zipStream.close();

          if (byteStream.size() > 0) {

               zFileOut.write(byteStream.toByteArray(), 0, byteStream.size());

               bytesOut += byteStream.size();

           }

           System.out.format("Records     : %,15d%n", count);

           System.out.format("Bytes In    : %,15d%n", bytesIn);

           System.out.format("Bytes Out   : %,15d%n", bytesOut);

           System.out.format("Compression : %,15.1f%%%n",

                   ((double) bytesIn - bytesOut) / bytesIn * 100);

       } finally {

           zFileIn.close();

           zFileOut.close();

       }

   }

 

  private static void writeRDW(int nRead, OutputStream os) throws IOException {

       os.write(nRead >> 8); // writes 1 byte from 8 low order bits of argument

       os.write(nRead);

       os.write(0);

       os.write(0);

   }

}

Compress.java