Black Hill Software

  • Home
  • Products
    • EasySMF
      • Online Manual
      • Release Notes and Latest Version
      • License Agreement
    • EasySMF:JE
      • EasySMF:JE Java Quickstart
      • Release Notes and Latest Version
      • Javadoc
      • License Agreement
    • EasySMF:RTI
      • EasySMF:RTI – SMF Real Time Interface
      • Javadoc
    • 30 Day Trial
  • Purchase
    • How to Buy
    • Purchase a License
  • Support
    • Documentation
      • EasySMF Desktop Online Manual
      • EasySMF:JE Javadoc
      • EasySMF RTI Javadoc
      • EasySMF JSON Javadoc
      • z/OS Utilities Javadoc
    • EasySMF Support
    • Get the latest version of EasySMF
    • EasySMF:JE Support
    • Get the latest version of EasySMF:JE
  • News
  • Contact
    • Support
    • Sales

Using zEDC compression for SMF data

April 29, 2024 by Andrew

Should you use zEDC to compress SMF data? Should you use both zEDC and CICS software compression?

There was a recent discussion on the IBM-MAIN email list about exploiting zEDC. One of the topics was SMF data, including whether to turn off the software compression for CICS SMF records.

I ran some experiments to compare the processing of SMF records with and without the various compression options. The tests were run using a dataset containing 6.6 GB of CICS SMF data compressed with CICS software compression. For the tests without software compression, the data was uncompressed and copied to another dataset compressed with zEDC. The uncompressed data size was about 39 GB.

Disclaimer: As always, your mileage may vary. This isn’t intended as a benchmark – it’s just one set of tests run against one dataset. This is a very lightly loaded system running under VM. It might bear no resemblance to a busy production system. CPU times etc. might not be accurately reported.

Data size

CICS software compression124000 tracks
CICS software compression + zEDC18500 tracks
zEDC compression35000 tracks

The data is very compressible. The software compressed data compresses further using zEDC and the final size is smaller than compression by zEDC alone.

Reading data

To test the overhead of zEDC when reading SMF data, I used the IFASMFDP program with the OUTDD pointing to DUMMY. This reads the data and produces a report on the numbers of each SMF record type, showing that individual records were read.

Elapsed (seconds)CPU (seconds)
CICS software compression
(read 6.6GB)
30.10.86
CICS software compression + zEDC
(read 6.6GB, ~1GB compressed)
2.21.2
zEDC compression only
(read 39GB, ~1.9GB compressed)
6.55.6
Chart showing comparative times to read data

This program doesn’t do anything with the CICS data so there is no software decompression. It’s just a indication of the cost of reading SMF records with and without zEDC compression.

I was surprised how slow it was to read the records without zEDC. Is this a result of e.g. running under VM, or is it typical? zEDC was very fast, relative to the uncompressed data – better than 10x speed-up.

Copying Data

I ran the IFASMFDP jobs but this time with OUTDD specifying a real dataset. The difference between the read jobs and the copy jobs should give an indication of the cost of writing the data using zEDC.

Elapsed (seconds)CPU (seconds)
CICS software compression
(read/write 6.6GB)
621.58
CICS software compression + zEDC
(read/write 6.6GB, ~1GB compressed)
8.32.31
zEDC compression only
(read/write 39GB, ~1.9GB compressed)
19.411.88
Chart showing comparative times to copy data

Again zEDC was much faster but unsurprisingly there was a CPU cost (less than 0.2s/GB).

SMF Reporting using Java

I ran the CICS Transaction Summary report from the EasySMF Java samples:

https://github.com/BlackHillSoftware/easysmf-samples/blob/main/sample-reports/src/main/java/com/smfreports/cics/CicsTransactionSummary.java

This report produces a basic summary of CICS transactions grouped by APPLID and transaction name.

Software decompression is performed in Java by EasySMF as required. The program is processing 39GB of data after decompression.

This gives some idea of the overhead of compression on reporting, however this is very dependent on the reporting software you use and your mileage will definitely vary.

Elapsed (seconds)CP (seconds)zIIP (seconds)
CICS software compression
(6.6GB compressed)
64.51.4940.6
CICS software compression + zEDC
(read 6.6GB, ~1GB compressed)
39.51.2545.4
zEDC compression only
(read 39GB, ~1.9GB compressed)
35.02.0238.6
Chart showing comparative times to run a report

One interesting thing here is that zEDC with Java seems to allow more work to move from the CP to the zIIP. The time on CP reduces from 1.49s to 1.25s processing the 6.6GB input. The CP time processing the 39GB compressed with zEDC is only 2 seconds. Reading the same data with IFASMFDP took 5.6 seconds of CPU time. Presumably under Java the rest is included in the zIIP time.

CICS software compression increased elapsed and zIIP times by about 10-15% compared to zEDC compression only, but reduced CP time.

Conclusions

Based on these tests, zEDC works very well with SMF data.

  • Elapsed times were greatly reduced in all cases
  • The CPU time required was fractions of a second per GB
  • CICS software compression appears to be still worthwhile. Reducing the amount of data being processed by zEDC significantly reduces the elapsed and CPU times for SMF copy operations e.g. SMF dump, weekly/monthly processing.

Don’t forget to measure and verify the results on your own systems!

Filed Under: Java, Java SMF

Java mapping for CICS SMF records

August 3, 2017 by Andrew

The EasySMF Java API now has experimental support for CICS records.

Experimental, because I want to get some feedback from CICS users about class names, usage etc. before locking down the design. In particular:

  • Do the class names and organization make sense to a CICS person? Would other names or a different organization make more sense?
  • Are the examples of how to process data clear and useful?
  • Are there areas where terminology is used incorrectly?

The complete Javadoc is here with an overview of the CICS functionality here.

If you have any comments, you can leave feedback in the comments box below, send it to support@blackhillsoftware.com, or give feedback in person at booth 323 at SHARE in Providence, Rhode Island.

You can try out the API using the 30 day trial available here: 30 Day Trial.

Installation information is available here: EasySMF:JE Java Quickstart

Using the API

EasySMF:JE aims to provide a consistent interface across different SMF record types and sections, and converts values to standard Java types for simple programming.

Dates and Times

Dates and times are converted to java.time classes. Java.time can represent dates and times with a precision of 1 nanosecond.

Times representing a duration e.g. CPU or elapsed time are converted to Duration.
Dates and times of day are converted to LocalDate, LocalTime, LocalDateTime or ZonedDateTime depending on exactly what information is in the field. Typically, times based on UTC(GMT) are converted to ZonedDateTime with ZoneOffset.UTC. Other dates and times are converted to LocalDate/Times.
Java has time zone rules so it is possible to apply a ZoneId to a LocalDateTime and perform date aware conversions between time zones.

Numeric Values

1, 2 and 3 byte integer values and 4 byte signed integer values are converted to int (32 bit signed) values.
4-7 byte integer values and 8 byte signed values are converted to long (64 bit signed).

8 byte unsigned values are available as both long (64 bit signed) and as a BigInteger. The long value may provide better performance if the value will not exceed the maximum value for a long. If a value does exceed the maximum value (i.e. the high order bit is set) an exception will be thrown. If the field value might exceed the maximum value for a long, use the BigInteger version.

Integer values greater than 8 bytes are converted to BigInteger.

Floating point values are converted to Java double.

String Values

EBCDIC and UTF8 string/character values are converted to String. Java uses Unicode internally – values are converted from EBCDIC or UTF8.

Flags

Flag bits within a byte are converted to a boolean value indicating whether the bit is set.

CICS Statistics

Reading CICS statistics is very done the same way as reading sections from other records using the API. Sections of a specific type are returned in a List<E> of that type. If there are no sections of the type in the record an empty List is returned. This allows you to iterate over the sections without explicitly checking whether the sections exist in the record – an empty list will iterate 0 times.

Example

The following code reads all FileControlStatistics sections from type 110 SMF records from the DD INPUT.

 try (SmfRecordReader reader = 
         SmfRecordReader
             .fromDD("INPUT")
             .include(110, Smf110Record.SMFSTSTY))
 {
     for (SmfRecord record : reader)
     {
         Smf100Record r110 = new Smf110Record(record);
         for (FileControlStatistics fc : 
             r110.fileControlStatistics())
         {
             //...   process FileControlStatistics sections here
         }
     }
 }

CICS Performance Monitoring

Accessing data from CICS monitoring performance records is slightly different to other SMF records because the data needs to be accessed using a Dictionary.

Dictionary records are handled automatically, however you cannot access the data from a record before a related dictionary record has been seen. You can check whether a dictionary record is available using Smf110Record.haveDictionary() or simply concatenate all required dictionary records ahead of the data records in the input data.

Specific fields are defined by name and type. Then Performance records are read from the SMF record, and specific fields accessed using getField(…) methods or variations.

Example

 ByteStringField transactionField = ByteStringField.define("DFHTASK","C001");
 TimestampField startField = TimestampField.define("DFHCICS","T005");
 TimestampField stopField = TimestampField.define("DFHCICS","T006");
 ClockField dispatchField = ClockField.define("DFHTASK","S007");

 try (SmfRecordReader reader = 
         SmfRecordReader
             .fromDD("INPUT")
             .include(110, Smf110Record.SMFMNSTY))
 {
     for (SmfRecord record : reader)
     {
         Smf100Record r110 = new Smf110Record(record); 
         if (r110.haveDictionary())
         {
             for (PerformanceRecord perfdata :
                 r110.performanceRecords())
             {
                 String txName = perfdata.getField(transactionField);
                 ZonedDateTime start = perfdata.getField(startField);
                 ZonedDateTime stop = perfdata.getField(stopField);
                 double dispatch = perfdata.getFieldTimerSeconds(dispatchField);

                 //...  process data
             }
         }
     }
 }

Complete CICS Statistics reporting sample

These samples are designed to show how to use the API, not to suggest items that you should specifically be reporting. However comments about their relevance are welcome.

import java.io.*;
import java.util.*;
import static java.util.Comparator.comparing;

import com.blackhillsoftware.smf.*;
import com.blackhillsoftware.smf.cics.*;
import com.blackhillsoftware.smf.cics.statistics.FileControlStatistics;

public class CicsFileStatistics 
{
    public static void main(String[] args) throws IOException 
    {
        Map<String, Map<String, FileData>> applids = 
                new HashMap<String, Map<String, FileData>>();

        try (SmfRecordReader reader = 
                args.length == 0 ? 
                SmfRecordReader.fromDD("INPUT") :
                SmfRecordReader.fromStream(new FileInputStream(args[0]))) 
        {
            reader.include(110, Smf110Record.SMFSTSTY);
            for (SmfRecord record : reader) 
            {
                Smf110Record r110 = new Smf110Record(record);

                Map<String, FileData> applidFiles = 
                        applids.computeIfAbsent(r110.stProductSection().smfstprn(),
                        files -> new HashMap<String, FileData>());

                for (FileControlStatistics fileStats : r110.fileControlStatistics()) 
                {
                    String entryName = fileStats.a17fnam();
                    applidFiles.computeIfAbsent(entryName, 
                            x -> new FileData(entryName)).add(fileStats);
                }
            }
        }
        writeReport(applids);
    }

    private static void writeReport(Map<String, Map<String, FileData>> applidFiles) 
    {

        applidFiles.entrySet().stream()
            .filter(applid -> !applid.getValue().isEmpty())
            .sorted((a, b) -> a.getKey().compareTo(b.getKey()))
            .forEachOrdered(applid -> 
            {
                // Headings
                System.out.format("%n%-8s", applid.getKey());

                System.out.format("%n%-8s %12s %12s %12s %12s %12s %12s %12s %12s%n%n", 
                        "ID", 
                        "Gets", 
                        "Get Upd",
                        "Browse", 
                        "Adds", 
                        "Updates", 
                        "Deletes", 
                        "Data EXCP", 
                        "Index EXCP");

                applid.getValue().entrySet().stream()
                    .map(x -> x.getValue())
                    .sorted(comparing(FileData::getTotalExcps)
                            .reversed())
                    .forEachOrdered(fileInfo -> 
                    {
                        // write detail line
                        System.out.format("%-8s %12d %12d %12d %12d %12d %12d %12d %12d%n", 
                                fileInfo.getId(),
                                fileInfo.getGets(), 
                                fileInfo.getGetUpd(), 
                                fileInfo.getBrowse(),
                                fileInfo.getAdds(), 
                                fileInfo.getUpdates(), 
                                fileInfo.getDeletes(),
                                fileInfo.getDataExcps(), 
                                fileInfo.getIndexExcps());
                    });
                });

    }

    private static class FileData 
    {
        public FileData(String fileId)
        {
            this.id = fileId;
        }

        public void add(FileControlStatistics fileStatistics) 
        {
            gets += fileStatistics.a17dsrd();
            getupd += fileStatistics.a17dsgu();
            browse += fileStatistics.a17dsbr();
            add = fileStatistics.a17dswra();
            update = fileStatistics.a17dswru();
            delete = fileStatistics.a17dsdel();
            dataexcp = fileStatistics.a17dsxcp();
            indexexcp = fileStatistics.a17dsixp();
            totalexcp += fileStatistics.a17dsxcp() 
                    + fileStatistics.a17dsixp();
        }

        public String getId() 
        {
            return id;
        }

        public long getGets() 
        {
            return gets;
        }

        public long getGetUpd() 
        {
            return getupd;
        }

        public long getBrowse() 
        {
            return browse;
        }

        public long getAdds() 
        {
            return add;
        }

        public long getUpdates() 
        {
            return update;
        }

        public long getDeletes() 
        {
            return delete;
        }

        public long getDataExcps() 
        {
            return dataexcp;
        }

        public long getIndexExcps() 
        {
            return indexexcp;
        }

        public long getTotalExcps() 
        {
            return totalexcp;
        }

        private String id;
        private long gets = 0;
        private long getupd = 0;
        private long browse = 0;
        private long add = 0;
        private long update = 0;
        private long delete = 0;
        private long dataexcp = 0;
        private long indexexcp = 0;
        private long totalexcp = 0;
    }
}

Complete CICS Transaction Monitoring reporting sample

import java.io.*;
import java.time.*;
import java.util.*;
import static java.util.Collections.reverseOrder;
import static java.util.Comparator.comparing;

import com.blackhillsoftware.smf.*;
import com.blackhillsoftware.smf.cics.*;
import com.blackhillsoftware.smf.cics.monitoring.*;
import com.blackhillsoftware.smf.cics.monitoring.fields.*;

public class CicsTransactionSummary 
{

    public static void main(String[] args) throws IOException 
    {
        Map<String, Map<String, TransactionData>> applids = 
                new HashMap<String, Map<String, TransactionData>>();

        ByteStringField transaction = ByteStringField.define("DFHTASK", "C001");

        int noDictionary = 0;

        try (SmfRecordReader reader = 
                args.length == 0 ? 
                SmfRecordReader.fromDD("INPUT") :
                SmfRecordReader.fromStream(new FileInputStream(args[0]))) 
        {     
            reader.include(110, Smf110Record.SMFMNSTY);
            for (SmfRecord record : reader) 
            {
                Smf110Record r110 = new Smf110Record(record);

                if (r110.haveDictionary()) 
                {
                    Map<String, TransactionData> applidTransactions = 
                        applids.computeIfAbsent(
                            r110.mnProductSection().smfmnprn(), 
                            transactions -> new HashMap<String, TransactionData>());

                    for (PerformanceRecord mn : r110.performanceRecords()) 
                    {
                        String txName = mn.getField(transaction);
                        applidTransactions.computeIfAbsent(
                                txName, 
                                x -> new TransactionData(txName)).add(mn);
                    }
                } else 
                {
                    noDictionary++;
                }
            }
        }

        writeReport(applids);

        if (noDictionary > 0) 
        {
            System.out.format(
                    "%n%nSkipped %s records because no applicable dictionary was found.", 
                    noDictionary);
        }

    }

    private static void writeReport(Map<String, Map<String, TransactionData>> transactions) 
    {
        transactions.entrySet().stream()
            .sorted((a, b) -> a.getKey().compareTo(b.getKey()))
            .forEachOrdered(applid -> 
            {
                // Headings
                System.out.format("%n%-8s", applid.getKey());

                System.out.format("%n%-4s %15s %15s %15s %15s %15s %15s %15s %15s %15s%n%n", 
                        "Name", 
                        "Count", 
                        "Elapsed",
                        "Avg Elapsed", 
                        "CPU", 
                        "Avg CPU", 
                        "Dispatch", 
                        "Avg Disp.", 
                        "Disp Wait", ""
                                + "Avg Disp Wait");

                applid.getValue().entrySet().stream()
                    .map(x -> x.getValue())
                    .sorted(comparing(TransactionData::getCpu, reverseOrder())
                            .thenComparing(TransactionData::getCount, reverseOrder()))
                    .forEachOrdered(txInfo -> 
                    {
                        // write detail line
                        System.out.format("%-4s %15d %15f %15f %15f %15f %15f %15f %15f %15f%n", 
                                txInfo.getName(),
                                txInfo.getCount(), 
                                txInfo.getElapsed(), 
                                txInfo.getAvgElapsed(), 
                                txInfo.getCpu(),
                                txInfo.getAvgCpu(), 
                                txInfo.getDispatch(), 
                                txInfo.getAvgDispatch(),
                                txInfo.getDispatchWait(), 
                                txInfo.getAvgDispatchWait());

                    });
            });

    }

    private static class TransactionData 
    {
        public TransactionData(String name) 
        {
            this.name = name;
        }

        public void add(PerformanceRecord perfdata) 
        {
            count++;
            elapsed += Utils.ToSeconds(
                    Duration.between(perfdata.getField(start), perfdata.getField(stop)));
            dispatch += perfdata.getFieldTimerSeconds(dispatchField);
            dispatchWait += perfdata.getFieldTimerSeconds(dispatchWaitField);
            cpu += perfdata.getFieldTimerSeconds(cpuField);
        }

        public String getName() 
        {
            return name;
        }

        public int getCount() 
        {
            return count;
        }

        public double getElapsed() 
        {
            return elapsed;
        }

        public double getDispatch() 
        {
            return dispatch;
        }

        public double getDispatchWait() 
        {
            return dispatchWait;
        }

        public double getCpu() 
        {
            return cpu;
        }

        public Double getAvgElapsed() 
        {
            return count != 0 ? elapsed / count : null;
        }

        public Double getAvgDispatch() 
        {
            return count != 0 ? dispatch / count : null;
        }

        public Double getAvgDispatchWait() 
        {
            return count != 0 ? dispatchWait / count : null;
        }

        public Double getAvgCpu() 
        {
            return count != 0 ? cpu / count : null;
        }

        static TimestampField start = TimestampField.define("DFHCICS", "T005");
        static TimestampField stop = TimestampField.define("DFHCICS", "T006");
        static ClockField dispatchField = ClockField.define("DFHTASK", "S007");
        static ClockField dispatchWaitField = ClockField.define("DFHTASK", "S102");
        static ClockField cpuField = ClockField.define("DFHTASK", "S008");

        private String name;
        private int count = 0;
        private double elapsed = 0;
        private double dispatch = 0;
        private double dispatchWait = 0;
        private double cpu = 0;
    }
}

Filed Under: EasySMF News, Java, Java SMF

Java SMF API Version 1.3.0

September 7, 2016 by Andrew

Verison 1.3.0 of the EasySMF:JE Java API is now available.

Major Changes

EZSMFKEY DD

The program will now look for the key in the EZSMFKEY DD statement as well as the EASYSMFKEY environment variable. The EASYSMFKEY environment variable is no longer required for z/OS batch jobs.

API messages

API messages are now issued via the SLF4J logging framework instead of to STDOUT. This allows the destination to be customized, and avoids mixing them with program output written to STDOUT. The default destination for messages is STDERR.

SMF Section Constructors marked private

The SMF section class constructors were always documented as “intended for internal use only”. Now they are private. This allows more flexibilty in construction, which in turn allows more transparent handling of changes to sections.

Sample JCL changes

The sample JCL procedures have been modified to eliminate the extra step to concatenate user additions and changes to the environment variables. This is now done in the main program execution step.

QWHCTOKN

The DB2 Accounting Token QWHCTOKN in the Qwhc section now has its own class. It was initially unclear what type of data appeared in the QWHCTOKN. While it was frequently readable characters, it also often contained binary data. Since the purpose is presumably to match data by token, a string representation was no good because binary data might result in different tokens converting to the same String. So initially a BigInteger was chosen as a class that could uniquely represent 22 bytes of data, and be compared and used as a key etc.

However, since there is character data there, people might reasonably want to see it as a string. BigInteger doesn’t make for a simple EBCDIC to UTF8 conversion. So I created a specific class for the QWHCTOKN. Now they can be reliably compared and used as a key, and there is a custom toString() implementation that does the EBCDIC to UTF8 conversion on the character portion.

Reminder: the DB2 SMF record support is still in experimental status so there is a greater likelihood of changes to the API. In particular, while the status is experimental, improving the API is likely to take precedence over mainitaining compatibility with existing programs.

Filed Under: Java SMF Tagged With: DB2, Java

Processing z/OS SMF data using Java

November 4, 2014 by Andrew

Black Hill Software is currently developing an API for processing SMF data using Java.

The API builds on experience gained developing EasySMF to create a powerful and easy to use API for working with SMF data.

The API will run on z/OS and other platforms such as Linux and Windows. The API is 100% Java which makes it zAAP eligible on z/OS.

[Read more…]

Filed Under: Java SMF

EasySMF:JE Installation

November 4, 2014 by Andrew

[Read more…]

Filed Under: Java SMF

  • 1
  • 2
  • Next Page »

30 Day Trial

EasySMF and EasySMF:JE are available for a free 30 day trial. Download now and start using them immediately.
30 Day Trial

Information

EasySMF:JE Java API for SMF: Quickstart

Java vs C++ : Drag Racing on z/OS

News

  • Using zEDC compression for SMF data
  • Text message alerts using the z/OS SMF Real Time Interface
  • DCOLLECT Reports and DCOLLECT to JSON using Java

Black Hill Software

Suite 10b, 28 University Drive, Mt Helen, VIC 3350, Australia
PO Box 2214, Bakery Hill, VIC 3354, Australia
+61 3 5331 8201
+1 (310) 634 9882
info@blackhillsoftware.com

News

  • Using zEDC compression for SMF data
  • Text message alerts using the z/OS SMF Real Time Interface
  • DCOLLECT Reports and DCOLLECT to JSON using Java

Copyright © 2025 · Enterprise Pro Theme on Genesis Framework · WordPress · Log in