Black Hill Software

  • Home
  • Products
    • EasySMF
      • Online Manual
      • Release Notes and Latest Version
      • License Agreement
    • EasySMF:JE
      • EasySMF:JE Java Quickstart
      • Release Notes and Latest Version
      • Javadoc
      • License Agreement
    • EasySMF:RTI
      • EasySMF:RTI – SMF Real Time Interface
      • Javadoc
    • 30 Day Trial
  • Purchase
    • How to Buy
    • Purchase a License
  • Support
    • Documentation
      • EasySMF Desktop Online Manual
      • EasySMF:JE Javadoc
      • EasySMF RTI Javadoc
      • EasySMF JSON Javadoc
      • z/OS Utilities Javadoc
    • EasySMF Support
    • Get the latest version of EasySMF
    • EasySMF:JE Support
    • Get the latest version of EasySMF:JE
  • News
  • Contact
    • Support
    • Sales

Understanding z/OS Unix Work with EasySMF

June 20, 2019 by Andrew

Unix work running under z/OS can be difficult to track.

A Unix process can create thousands of child processes, each running in another address space. They may only exist for fractions of a second and produce no job output, so you don’t see them in SDSF. Child processes create their own SMF type 30 records for job accounting, so their resource usage doesn’t appear in the parent’s accounting records. The child processes may have a different jobname and can even run in different service and report classes to the parent.

I have seen a job running BPXBATCH where the job itself used virtually no CPU time, but it spawned over 10,000 sub tasks. More than 99% of the CPU time used by that batch job appeared in child process SMF records with different job names, service and report classes.

EasySMF has reports to help you understand your z/OS Unix work.

EasySMF uses the Unix process and parent process id information from the type 30 SMF records to build a tree view of your Unix work.

  • You can see the parent – child relationships between different address spaces.
  • You can see whether the child service and report classes are the same as the original job.
  • When you collapse the tree, usage information from all the children is rolled up into the parent job. When you expand the tree, usage information is shown for the individual entries.

The expanded view of Unix work, showing the relationship between work running in different address spaces:

The collapsed view of the same work. Resources like CPU time show the total for all the related address spaces:

Related Processes

EasySMF can also help you find where work came from. If Related Processes is selected, EasySMF will search for and show parent and child tasks that do not match the main selection criteria. Here we can see a task with the job name ANDREWR, and Related Processes shows that particular task came in through SSH, and it had a number of Unix sub tasks.

Related Processes can help you find out where a Unix process came from.
More Detailed Information about Unix Work

The Unix Work report shows you more detail about work that uses z/OS Unix. As the name suggests, it only shows work that had a Unix component – other batch jobs and started tasks are ignored. This report shows information from the step and substep end records, which include some information about the program that was executed. This give clues to the processing taking place, although the information included in SMF is limited and may not show every program.

The Unix Work report shows information from Step End records for work with a Unix component, including the program information recorded in SMF.

Unix processes don’t always run in a different address space under z/OS. Sometimes processes will share an address space. This gives multiple Unix Process sections in SMF, with most of the statistics reported at the shared address space level. You can see this in the Unix Process report where you have processes that do not report the address space level information.

When multiple processes share an address space only one will show address space level information.

You can filter the Unix Work report to find a particular process ID – in this case the selected process was one of multiple processes that shared one address space.

Detailed Unix information is included in EasySMF since version 3.2.

Upgrades are free for existing customers. If you are not a customer, a 30 day trial is available.

Filed Under: EasySMF News

Java mapping for CICS SMF records

August 3, 2017 by Andrew

The EasySMF Java API now has experimental support for CICS records.

Experimental, because I want to get some feedback from CICS users about class names, usage etc. before locking down the design. In particular:

  • Do the class names and organization make sense to a CICS person? Would other names or a different organization make more sense?
  • Are the examples of how to process data clear and useful?
  • Are there areas where terminology is used incorrectly?

The complete Javadoc is here with an overview of the CICS functionality here.

If you have any comments, you can leave feedback in the comments box below, send it to support@blackhillsoftware.com, or give feedback in person at booth 323 at SHARE in Providence, Rhode Island.

You can try out the API using the 30 day trial available here: 30 Day Trial.

Installation information is available here: EasySMF:JE Java Quickstart

Using the API

EasySMF:JE aims to provide a consistent interface across different SMF record types and sections, and converts values to standard Java types for simple programming.

Dates and Times

Dates and times are converted to java.time classes. Java.time can represent dates and times with a precision of 1 nanosecond.

Times representing a duration e.g. CPU or elapsed time are converted to Duration.
Dates and times of day are converted to LocalDate, LocalTime, LocalDateTime or ZonedDateTime depending on exactly what information is in the field. Typically, times based on UTC(GMT) are converted to ZonedDateTime with ZoneOffset.UTC. Other dates and times are converted to LocalDate/Times.
Java has time zone rules so it is possible to apply a ZoneId to a LocalDateTime and perform date aware conversions between time zones.

Numeric Values

1, 2 and 3 byte integer values and 4 byte signed integer values are converted to int (32 bit signed) values.
4-7 byte integer values and 8 byte signed values are converted to long (64 bit signed).

8 byte unsigned values are available as both long (64 bit signed) and as a BigInteger. The long value may provide better performance if the value will not exceed the maximum value for a long. If a value does exceed the maximum value (i.e. the high order bit is set) an exception will be thrown. If the field value might exceed the maximum value for a long, use the BigInteger version.

Integer values greater than 8 bytes are converted to BigInteger.

Floating point values are converted to Java double.

String Values

EBCDIC and UTF8 string/character values are converted to String. Java uses Unicode internally – values are converted from EBCDIC or UTF8.

Flags

Flag bits within a byte are converted to a boolean value indicating whether the bit is set.

CICS Statistics

Reading CICS statistics is very done the same way as reading sections from other records using the API. Sections of a specific type are returned in a List<E> of that type. If there are no sections of the type in the record an empty List is returned. This allows you to iterate over the sections without explicitly checking whether the sections exist in the record – an empty list will iterate 0 times.

Example

The following code reads all FileControlStatistics sections from type 110 SMF records from the DD INPUT.

 try (SmfRecordReader reader = 
         SmfRecordReader
             .fromDD("INPUT")
             .include(110, Smf110Record.SMFSTSTY))
 {
     for (SmfRecord record : reader)
     {
         Smf100Record r110 = new Smf110Record(record);
         for (FileControlStatistics fc : 
             r110.fileControlStatistics())
         {
             //...   process FileControlStatistics sections here
         }
     }
 }

CICS Performance Monitoring

Accessing data from CICS monitoring performance records is slightly different to other SMF records because the data needs to be accessed using a Dictionary.

Dictionary records are handled automatically, however you cannot access the data from a record before a related dictionary record has been seen. You can check whether a dictionary record is available using Smf110Record.haveDictionary() or simply concatenate all required dictionary records ahead of the data records in the input data.

Specific fields are defined by name and type. Then Performance records are read from the SMF record, and specific fields accessed using getField(…) methods or variations.

Example

 ByteStringField transactionField = ByteStringField.define("DFHTASK","C001");
 TimestampField startField = TimestampField.define("DFHCICS","T005");
 TimestampField stopField = TimestampField.define("DFHCICS","T006");
 ClockField dispatchField = ClockField.define("DFHTASK","S007");

 try (SmfRecordReader reader = 
         SmfRecordReader
             .fromDD("INPUT")
             .include(110, Smf110Record.SMFMNSTY))
 {
     for (SmfRecord record : reader)
     {
         Smf100Record r110 = new Smf110Record(record); 
         if (r110.haveDictionary())
         {
             for (PerformanceRecord perfdata :
                 r110.performanceRecords())
             {
                 String txName = perfdata.getField(transactionField);
                 ZonedDateTime start = perfdata.getField(startField);
                 ZonedDateTime stop = perfdata.getField(stopField);
                 double dispatch = perfdata.getFieldTimerSeconds(dispatchField);

                 //...  process data
             }
         }
     }
 }

Complete CICS Statistics reporting sample

These samples are designed to show how to use the API, not to suggest items that you should specifically be reporting. However comments about their relevance are welcome.

import java.io.*;
import java.util.*;
import static java.util.Comparator.comparing;

import com.blackhillsoftware.smf.*;
import com.blackhillsoftware.smf.cics.*;
import com.blackhillsoftware.smf.cics.statistics.FileControlStatistics;

public class CicsFileStatistics 
{
    public static void main(String[] args) throws IOException 
    {
        Map<String, Map<String, FileData>> applids = 
                new HashMap<String, Map<String, FileData>>();

        try (SmfRecordReader reader = 
                args.length == 0 ? 
                SmfRecordReader.fromDD("INPUT") :
                SmfRecordReader.fromStream(new FileInputStream(args[0]))) 
        {
            reader.include(110, Smf110Record.SMFSTSTY);
            for (SmfRecord record : reader) 
            {
                Smf110Record r110 = new Smf110Record(record);

                Map<String, FileData> applidFiles = 
                        applids.computeIfAbsent(r110.stProductSection().smfstprn(),
                        files -> new HashMap<String, FileData>());

                for (FileControlStatistics fileStats : r110.fileControlStatistics()) 
                {
                    String entryName = fileStats.a17fnam();
                    applidFiles.computeIfAbsent(entryName, 
                            x -> new FileData(entryName)).add(fileStats);
                }
            }
        }
        writeReport(applids);
    }

    private static void writeReport(Map<String, Map<String, FileData>> applidFiles) 
    {

        applidFiles.entrySet().stream()
            .filter(applid -> !applid.getValue().isEmpty())
            .sorted((a, b) -> a.getKey().compareTo(b.getKey()))
            .forEachOrdered(applid -> 
            {
                // Headings
                System.out.format("%n%-8s", applid.getKey());

                System.out.format("%n%-8s %12s %12s %12s %12s %12s %12s %12s %12s%n%n", 
                        "ID", 
                        "Gets", 
                        "Get Upd",
                        "Browse", 
                        "Adds", 
                        "Updates", 
                        "Deletes", 
                        "Data EXCP", 
                        "Index EXCP");

                applid.getValue().entrySet().stream()
                    .map(x -> x.getValue())
                    .sorted(comparing(FileData::getTotalExcps)
                            .reversed())
                    .forEachOrdered(fileInfo -> 
                    {
                        // write detail line
                        System.out.format("%-8s %12d %12d %12d %12d %12d %12d %12d %12d%n", 
                                fileInfo.getId(),
                                fileInfo.getGets(), 
                                fileInfo.getGetUpd(), 
                                fileInfo.getBrowse(),
                                fileInfo.getAdds(), 
                                fileInfo.getUpdates(), 
                                fileInfo.getDeletes(),
                                fileInfo.getDataExcps(), 
                                fileInfo.getIndexExcps());
                    });
                });

    }

    private static class FileData 
    {
        public FileData(String fileId)
        {
            this.id = fileId;
        }

        public void add(FileControlStatistics fileStatistics) 
        {
            gets += fileStatistics.a17dsrd();
            getupd += fileStatistics.a17dsgu();
            browse += fileStatistics.a17dsbr();
            add = fileStatistics.a17dswra();
            update = fileStatistics.a17dswru();
            delete = fileStatistics.a17dsdel();
            dataexcp = fileStatistics.a17dsxcp();
            indexexcp = fileStatistics.a17dsixp();
            totalexcp += fileStatistics.a17dsxcp() 
                    + fileStatistics.a17dsixp();
        }

        public String getId() 
        {
            return id;
        }

        public long getGets() 
        {
            return gets;
        }

        public long getGetUpd() 
        {
            return getupd;
        }

        public long getBrowse() 
        {
            return browse;
        }

        public long getAdds() 
        {
            return add;
        }

        public long getUpdates() 
        {
            return update;
        }

        public long getDeletes() 
        {
            return delete;
        }

        public long getDataExcps() 
        {
            return dataexcp;
        }

        public long getIndexExcps() 
        {
            return indexexcp;
        }

        public long getTotalExcps() 
        {
            return totalexcp;
        }

        private String id;
        private long gets = 0;
        private long getupd = 0;
        private long browse = 0;
        private long add = 0;
        private long update = 0;
        private long delete = 0;
        private long dataexcp = 0;
        private long indexexcp = 0;
        private long totalexcp = 0;
    }
}

Complete CICS Transaction Monitoring reporting sample

import java.io.*;
import java.time.*;
import java.util.*;
import static java.util.Collections.reverseOrder;
import static java.util.Comparator.comparing;

import com.blackhillsoftware.smf.*;
import com.blackhillsoftware.smf.cics.*;
import com.blackhillsoftware.smf.cics.monitoring.*;
import com.blackhillsoftware.smf.cics.monitoring.fields.*;

public class CicsTransactionSummary 
{

    public static void main(String[] args) throws IOException 
    {
        Map<String, Map<String, TransactionData>> applids = 
                new HashMap<String, Map<String, TransactionData>>();

        ByteStringField transaction = ByteStringField.define("DFHTASK", "C001");

        int noDictionary = 0;

        try (SmfRecordReader reader = 
                args.length == 0 ? 
                SmfRecordReader.fromDD("INPUT") :
                SmfRecordReader.fromStream(new FileInputStream(args[0]))) 
        {     
            reader.include(110, Smf110Record.SMFMNSTY);
            for (SmfRecord record : reader) 
            {
                Smf110Record r110 = new Smf110Record(record);

                if (r110.haveDictionary()) 
                {
                    Map<String, TransactionData> applidTransactions = 
                        applids.computeIfAbsent(
                            r110.mnProductSection().smfmnprn(), 
                            transactions -> new HashMap<String, TransactionData>());

                    for (PerformanceRecord mn : r110.performanceRecords()) 
                    {
                        String txName = mn.getField(transaction);
                        applidTransactions.computeIfAbsent(
                                txName, 
                                x -> new TransactionData(txName)).add(mn);
                    }
                } else 
                {
                    noDictionary++;
                }
            }
        }

        writeReport(applids);

        if (noDictionary > 0) 
        {
            System.out.format(
                    "%n%nSkipped %s records because no applicable dictionary was found.", 
                    noDictionary);
        }

    }

    private static void writeReport(Map<String, Map<String, TransactionData>> transactions) 
    {
        transactions.entrySet().stream()
            .sorted((a, b) -> a.getKey().compareTo(b.getKey()))
            .forEachOrdered(applid -> 
            {
                // Headings
                System.out.format("%n%-8s", applid.getKey());

                System.out.format("%n%-4s %15s %15s %15s %15s %15s %15s %15s %15s %15s%n%n", 
                        "Name", 
                        "Count", 
                        "Elapsed",
                        "Avg Elapsed", 
                        "CPU", 
                        "Avg CPU", 
                        "Dispatch", 
                        "Avg Disp.", 
                        "Disp Wait", ""
                                + "Avg Disp Wait");

                applid.getValue().entrySet().stream()
                    .map(x -> x.getValue())
                    .sorted(comparing(TransactionData::getCpu, reverseOrder())
                            .thenComparing(TransactionData::getCount, reverseOrder()))
                    .forEachOrdered(txInfo -> 
                    {
                        // write detail line
                        System.out.format("%-4s %15d %15f %15f %15f %15f %15f %15f %15f %15f%n", 
                                txInfo.getName(),
                                txInfo.getCount(), 
                                txInfo.getElapsed(), 
                                txInfo.getAvgElapsed(), 
                                txInfo.getCpu(),
                                txInfo.getAvgCpu(), 
                                txInfo.getDispatch(), 
                                txInfo.getAvgDispatch(),
                                txInfo.getDispatchWait(), 
                                txInfo.getAvgDispatchWait());

                    });
            });

    }

    private static class TransactionData 
    {
        public TransactionData(String name) 
        {
            this.name = name;
        }

        public void add(PerformanceRecord perfdata) 
        {
            count++;
            elapsed += Utils.ToSeconds(
                    Duration.between(perfdata.getField(start), perfdata.getField(stop)));
            dispatch += perfdata.getFieldTimerSeconds(dispatchField);
            dispatchWait += perfdata.getFieldTimerSeconds(dispatchWaitField);
            cpu += perfdata.getFieldTimerSeconds(cpuField);
        }

        public String getName() 
        {
            return name;
        }

        public int getCount() 
        {
            return count;
        }

        public double getElapsed() 
        {
            return elapsed;
        }

        public double getDispatch() 
        {
            return dispatch;
        }

        public double getDispatchWait() 
        {
            return dispatchWait;
        }

        public double getCpu() 
        {
            return cpu;
        }

        public Double getAvgElapsed() 
        {
            return count != 0 ? elapsed / count : null;
        }

        public Double getAvgDispatch() 
        {
            return count != 0 ? dispatch / count : null;
        }

        public Double getAvgDispatchWait() 
        {
            return count != 0 ? dispatchWait / count : null;
        }

        public Double getAvgCpu() 
        {
            return count != 0 ? cpu / count : null;
        }

        static TimestampField start = TimestampField.define("DFHCICS", "T005");
        static TimestampField stop = TimestampField.define("DFHCICS", "T006");
        static ClockField dispatchField = ClockField.define("DFHTASK", "S007");
        static ClockField dispatchWaitField = ClockField.define("DFHTASK", "S102");
        static ClockField cpuField = ClockField.define("DFHTASK", "S008");

        private String name;
        private int count = 0;
        private double elapsed = 0;
        private double dispatch = 0;
        private double dispatchWait = 0;
        private double cpu = 0;
    }
}

Filed Under: EasySMF News, Java, Java SMF

Sending Email from z/OS using Java

June 21, 2017 by Andrew

How do you send email from your z/OS system?

Java on z/OS makes this very easy. Java on z/OS can use Java libraries developed for other platforms which means there are powerful, free libraries available and plenty of documentation available via Google.

Instead of setting up a SMTP server on z/OS, you can define a user (or more than one) for z/OS in your corporate mail system, and create batch jobs that log in and send mail just like any other email user.

This example uses the JavaMail package (part of Java EE and available for download for Java SE) to send email from z/OS. It uses the JZOS Batch Launcher so you can define the input using JCL DD statements.

You can send an email using JCL like this:

//JAVAG    EXEC PROC=JAVAG,
// JAVACLS='''ZMail''',
// CLASPATH='''java/lib/javax.mail.jar'''
//TO       DD *
someone@example.com
//SUBJECT  DD *
Test Message from batch job
//MESSAGE  DD *
Message line 1
Message line 2
...
Message Line n
//

The JCL uses the JAVAG JCL PROC described in this post: Systems Programmer Friendly Java

The following code is the complete Java program. The sample uses Gmail as an example to show how to use TLS and SMTP authentication. Customize the Properties used to create the Session to suit your own mail server.

To, Subject and the Message Body are read from DD statements defined in the JCL. You could use the same pattern to allow other information to be input from DD statements, e.g. CC, BCC, From, or even the properties used to log in to the mail server.

import java.io.*;
import java.util.*;
import javax.mail.*;
import javax.mail.internet.*;

import com.ibm.jzos.ZFile;
import com.ibm.jzos.ZFileException;

public class ZMail 
{
    public static void main(String[] args) 
            throws IOException, MessagingException 
    {   
        List<String> messagelines = readLinesFromDD("MESSAGE");
        String subject = readLinesFromDD("SUBJECT").get(0);
        List<String> recipients = readLinesFromDD("TO");        

        StringBuilder messagetext = new StringBuilder();
        for (String line : messagelines)
        {
            messagetext.append(line);
            messagetext.append(System.lineSeparator());
        }

        final String username = "smtp-username";
        final String password = "smtp-password";

        Properties props = new Properties();
        props.put("mail.smtp.auth", "true");
        props.put("mail.smtp.starttls.enable", "true");
        props.put("mail.smtp.host", "smtp.gmail.com");
        props.put("mail.smtp.port", "587");
        props.put("mail.ssl.checkserveridentity", "true");

        Session session = Session.getInstance(props,
                new javax.mail.Authenticator() 
                {
                    protected PasswordAuthentication getPasswordAuthentication() 
                    {
                        return new PasswordAuthentication(username, password);
                    }
                });

        Message message = new MimeMessage(session);
        message.setFrom(new InternetAddress("me@example.com"));
        for (String recipient : recipients)
        {
            message.addRecipients(Message.RecipientType.TO, 
                    InternetAddress.parse(recipient));

        }
        message.setSubject(subject);
        message.setText(messagetext.toString());

        Transport.send(message);        
    }

    private static List<String> readLinesFromDD(String dd) 
            throws ZFileException, IOException 
    {
        List<String> lines = new ArrayList<String>();
        ZFile input = null;
        try
        {
            input = new ZFile("//DD:" + dd, "rt");
            BufferedReader reader = 
                new BufferedReader(
                    new InputStreamReader(input.getInputStream()));

            String line;     
            while ((line = reader.readLine()) != null) 
            {   
                lines.add(line);
            }

        }
        finally
        {
            if (input != null)
            {
                input.close();
            }
        }
        return lines;
    }
}

Filed Under: Java

Java SMF API Version 1.3.0

September 7, 2016 by Andrew

Verison 1.3.0 of the EasySMF:JE Java API is now available.

Major Changes

EZSMFKEY DD

The program will now look for the key in the EZSMFKEY DD statement as well as the EASYSMFKEY environment variable. The EASYSMFKEY environment variable is no longer required for z/OS batch jobs.

API messages

API messages are now issued via the SLF4J logging framework instead of to STDOUT. This allows the destination to be customized, and avoids mixing them with program output written to STDOUT. The default destination for messages is STDERR.

SMF Section Constructors marked private

The SMF section class constructors were always documented as “intended for internal use only”. Now they are private. This allows more flexibilty in construction, which in turn allows more transparent handling of changes to sections.

Sample JCL changes

The sample JCL procedures have been modified to eliminate the extra step to concatenate user additions and changes to the environment variables. This is now done in the main program execution step.

QWHCTOKN

The DB2 Accounting Token QWHCTOKN in the Qwhc section now has its own class. It was initially unclear what type of data appeared in the QWHCTOKN. While it was frequently readable characters, it also often contained binary data. Since the purpose is presumably to match data by token, a string representation was no good because binary data might result in different tokens converting to the same String. So initially a BigInteger was chosen as a class that could uniquely represent 22 bytes of data, and be compared and used as a key etc.

However, since there is character data there, people might reasonably want to see it as a string. BigInteger doesn’t make for a simple EBCDIC to UTF8 conversion. So I created a specific class for the QWHCTOKN. Now they can be reliably compared and used as a key, and there is a custom toString() implementation that does the EBCDIC to UTF8 conversion on the character portion.

Reminder: the DB2 SMF record support is still in experimental status so there is a greater likelihood of changes to the API. In particular, while the status is experimental, improving the API is likely to take precedence over mainitaining compatibility with existing programs.

Filed Under: Java SMF Tagged With: DB2, Java

Java SMF API Version 1.2.1

July 15, 2016 by Andrew

Version 1.2.1 of the EasySMF:JE Java API is now available.

Major Changes

  • Add SMF Type 121 (Java Statistics) record.

    This record has some interesting information, but has one major drawback. The ID running the Java job needs access to write SMF records. This is probably a security issue in many shops, because in general you don’t want to allow unauthorized users to write SMF records. It would be much nicer if this information was written by the JVM, without requiring special access for the user.

  • Modify samples to demonstrate use of Java 8 Streams functions. Streams are very nice for processing SMF data, and in many cases allow you to greatly simplify your code. It is unclear at the moment what the performance implications are. There are suggestions in some places that the code generated by Streams is not as well optimized as loops and iterators. That may be the case, but I don’t know yet whether the difference is detectable.

Filed Under: Uncategorized Tagged With: Java

  • « Previous Page
  • 1
  • 2
  • 3
  • 4
  • 5
  • …
  • 7
  • Next Page »

30 Day Trial

EasySMF and EasySMF:JE are available for a free 30 day trial. Download now and start using them immediately.
30 Day Trial

Information

EasySMF:JE Java API for SMF: Quickstart

Java vs C++ : Drag Racing on z/OS

News

  • Using zEDC compression for SMF data
  • Text message alerts using the z/OS SMF Real Time Interface
  • DCOLLECT Reports and DCOLLECT to JSON using Java

Black Hill Software

Suite 10b, 28 University Drive, Mt Helen, VIC 3350, Australia
PO Box 2214, Bakery Hill, VIC 3354, Australia
+61 3 5331 8201
+1 (310) 634 9882
info@blackhillsoftware.com

News

  • Using zEDC compression for SMF data
  • Text message alerts using the z/OS SMF Real Time Interface
  • DCOLLECT Reports and DCOLLECT to JSON using Java

Copyright © 2025 · Enterprise Pro Theme on Genesis Framework · WordPress · Log in