Black Hill Software

  • Home
  • Products
    • EasySMF
      • Online Manual
      • Release Notes and Latest Version
      • License Agreement
    • EasySMF:JE
      • EasySMF:JE Java Quickstart
      • Release Notes and Latest Version
      • Javadoc
      • License Agreement
    • EasySMF:RTI
      • EasySMF:RTI – SMF Real Time Interface
      • Javadoc
    • 30 Day Trial
  • Purchase
    • How to Buy
    • Purchase a License
  • Support
    • Documentation
      • EasySMF Desktop Online Manual
      • EasySMF:JE Javadoc
      • EasySMF RTI Javadoc
      • EasySMF JSON Javadoc
      • z/OS Utilities Javadoc
    • EasySMF Support
    • Get the latest version of EasySMF
    • EasySMF:JE Support
    • Get the latest version of EasySMF:JE
  • News
  • Contact
    • Support
    • Sales

Text message alerts using the z/OS SMF Real Time Interface

April 15, 2024 by Andrew

In this post, I’ll show how you can send SMS text messages from z/OS for failed jobs using Twilio and the z/OS SMF Real Time Interface.

Twilio provides a Java API to use their service, and that can be combined with the EasySMF Real Time Interface to send SMS messages based on real time SMF data.

Sending text messages from z/OS

Twilio is a paid service for production messaging, but you can sign up for a free trial account and send a limited number of messages to a verified phone number.

Twilio provides quickstart documentation here:

https://www.twilio.com/docs/messaging/quickstart/java

You can ignore the stuff about the CLI, all we are going to do is send an outbound message which requires a Twilio account, the Java program and dependencies i.e. the fat jar.

This is the Twilio quickstart Java program with a couple of minor tweaks:

import com.twilio.Twilio;
import com.twilio.rest.api.v2010.account.Message;
import com.twilio.type.PhoneNumber;

public class TwilioTest {
    // Find your Account SID and Auth Token at twilio.com/console
    // and set the environment variables. See http://twil.io/secure
    public static final String ACCOUNT_SID = System.getenv("TWILIO_ACCOUNT_SID");
    public static final String AUTH_TOKEN = System.getenv("TWILIO_AUTH_TOKEN");

    public static void main(String[] args) {
        Twilio.init(ACCOUNT_SID, AUTH_TOKEN);
        Message message = Message.creator(
                new com.twilio.type.PhoneNumber("+14159352345"), // to
                new com.twilio.type.PhoneNumber("+14158141829"), // from
                args[0])
            .create();

        System.out.println(message.getSid());
    }
}

Upload the Twilio fat jar twilio-x.x-jar-with-dependencies.jar to z/OS. It is available from e.g. https://repo1.maven.org/maven2/com/twilio/sdk/twilio/10.1.3/

Java 11 will run single file Java programs without a separate compilation step, so we can just run the program under BPXBATCH:

//ANDREWRG JOB CLASS=A,
//             MSGCLASS=H,
//             NOTIFY=&SYSUID
//*
//BPXBATCH EXEC PGM=BPXBATCH,REGION=512M
//STDPARM  DD *
sh /usr/lpp/java/J11.0_64/bin/java
 /home/andrewr/java/src/TwilioTest.java
 "Hello from z/OS"
//STDENV   DD *
CLASSPATH=/home/andrewr/java/lib/twilio-10.1.3-jar-with-dependencies.jar
TWILIO_ACCOUNT_SID=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
TWILIO_AUTH_TOKEN=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
//SYSOUT   DD SYSOUT=*
//STDOUT   DD SYSOUT=*
//STDERR   DD SYSOUT=*

Twilio can also send the messages to WhatsApp using the same functionality.

Sending notifications for failed jobs

Prerequisite: z/OS SMF Real Time Interface

The z/OS SMF Real Time Interface must be active. This requires:

  • SMF running in logstream mode
  • Define an in-memory resource to receive the required records (type 30 in this case)
  • Setup RACF definitions to allow the user running Java program to access the SMF in memory resource

The program

The complete source code for this program can be found here:

https://github.com/BlackHillSoftware/easysmf-samples/tree/main/easysmf-rti/rti-notifications

The following is an overview of the major sections of the program.

Reading the in-memory resource

The first thing to do is set up the connection to the in-memory resource.

public class RtiNotifications
{
    public static void main(String[] args) throws IOException
    {
        try (SmfConnection connection = 
                 SmfConnection.forResourceName("IFASMF.MYRECS")
                     .onMissedData(RtiNotifications::handleMissedData)
                     .disconnectOnStop()
                     .connect();

             // Set up SmfrecordReader to read type 30 subtypes 4 and 5    
             SmfRecordReader reader = 
                 SmfRecordReader.fromByteArrays(connection)
                     .include(30,4)
                     .include(30,5))
        {
            // process data here
        }
    }

    private static void handleMissedData(MissedDataEvent e)
    {
        System.out.println("Missed Data!");
        e.throwException(false);    
    }
}

This code:

  • creates a real time connection to resource name IFASMF.MYRECS
  • sets up an action to be taken if data is written faster than the program can read it and the real time buffer wraps (write a message and suppress the exception)
  • sets up a command handler to disconnect from the resource when the STOP command is received
  • creates a SmfRecordReader to read SMF 30 subtypes 4 and 5 (step and job end) records

The connection and reader will be closed automatically when the program exits the try block.

The connection can be simulated in a development environment by setting environment variables:

SIMULATE_RTI=Y

and

IFASMF_MYRECS=/your/smf/filename

The SmfConnection will read SMF data from the file indicated by the environment variable matching the resource name.

Processing the SMF 30 Records

We want to send the SMS message when the job ends, which is indicated by a subtype 5 record. However, if steps are skipped due to e.g. COND processing or an ABEND, the information in the subtype 5 record might be from a step that didn’t run. These steps have the “flushed” indicator set in the Completion Section.

We need to keep the step end information for steps that did run and check the last executed step when we see the job end record. The Java HashMap provides a convenient way to store information for failed steps.

We can create a class to use as a HashMap key to identify specific jobs. The class has fields for the identifying information (system, jobname, job number, read date and time) which are populated in the constructor. Eclipse can generate the hashCode and equals methods which are required for a HashMap key.

Map<JobKey, Smf30Record> failedSteps = new HashMap<>();

...

private static class JobKey
{
    String system;
    String jobname;
    String jobnumber;
    long readtime;
    int readdate;

    JobKey(Smf30Record r30)
    {
        system = r30.system();
        jobname = r30.identificationSection().smf30jbn();
        jobnumber = r30.identificationSection().smf30jnm();
        readtime = r30.identificationSection().smf30rstRawValue();
        readdate = r30.identificationSection().smf30rsdRawValue();
    }

    // hashCode and equals generated using Eclipse
    @Override
    public int hashCode() {
        return Objects.hash(jobname, jobnumber, readdate, readtime, system);
    }

    @Override
    public boolean equals(Object obj) {
        if (this == obj)
            return true;
        if (obj == null)
            return false;
        if (getClass() != obj.getClass())
            return false;
        JobKey other = (JobKey) obj;
        return Objects.equals(jobname, other.jobname) 
                && Objects.equals(jobnumber, other.jobnumber)
                && readdate == other.readdate 
                && readtime == other.readtime 
                && Objects.equals(system, other.system);
    }
}

SMF 30 Subtype 4: Step End

When processing step end records, if the step failed we save the record for later. If it ran successfully, we remove any previous failed step.

if (failed(r30)) 
{  
    // this replaces existing entry if present 
    failedSteps.put(new JobKey(r30), r30);
}
// else it didn't fail, check it wasn't flushed
else if (!r30.completionSection().smf30flh()) 
{
    // If the step wasn't flushed, remove any earlier failed step
    failedSteps.remove(new JobKey(r30));
}

failed is a method which checks whether the step failed:

private static boolean failed(Smf30Record r30) 
{
    return 
            // abended except for S222 (cancelled)
            (r30.completionSection().smf30abd() 
                    && r30.completionSection().smf30scc() != 0x222)
            // or condition code > 8
            || r30.completionSection().smf30scc() > 8
            // or post execution error
            || r30.completionSection().smf30sye();
}

The criteria can be adjusted as required.

SMF 30 Subtype 5: Job End

When the job ends, we check whether we previously saved a failed step.

JobKey key = new JobKey(r30);                        
if (failedSteps.containsKey(key))
{
    // send notification using information from failed step 
    sendNotification(failedSteps.get(key));                            
    failedSteps.remove(key); // finished with this
}
// or if the type 5 record indicates failure
else if(failed(r30))
{
    // send notification using information from job
    sendNotification(r30);                            
}

Sending the SMS message

The job information is extracted from the SMF record, the Twilio information is supplied in environment variables, and sending the SMS message uses the same process as the Twilio quickstart.

private static final String ACCOUNT_SID = System.getenv("TWILIO_ACCOUNT_SID");
private static final String AUTH_TOKEN = System.getenv("TWILIO_AUTH_TOKEN");
private static final String TO_PHONE = System.getenv("TO_PHONE");
private static final String FROM_PHONE = System.getenv("FROM_PHONE");

private static void sendNotification(Smf30Record r30) 
{
    String messagetext = 
            String.format("%s Job failed: %s %s Step: %d %s Program: %s CC: %s",
                    r30.smfDateTime().toString(),
                    r30.identificationSection().smf30jbn(), // job name
                    r30.identificationSection().smf30jnm(), // job number
                    r30.identificationSection().smf30stn(), // step number
                    r30.identificationSection().smf30stm(), // step name
                    r30.identificationSection().smf30pgm(), // program
                    r30.completionSection().completionDescription());
    
    // Send a SMS notification through Twilio
    Twilio.init(ACCOUNT_SID, AUTH_TOKEN);
    Message message = Message.creator(
            new com.twilio.type.PhoneNumber(TO_PHONE),
            new com.twilio.type.PhoneNumber(FROM_PHONE),
            messagetext)
            .create();
    System.out.println("Message Sent: " + message.getSid());
}

What Next?

Browse and build the EasySMF:RTI sample code:

https://github.com/BlackHillSoftware/easysmf-samples/tree/main/easysmf-rti/rti-notifications

The Github sample has some additional functionality not covered here. You can limit the notifications to jobs running in specific job classes. You can use the same functionality to include jobs by job name or create custom failure criteria for specific jobs.

For simplicity, the sample doesn’t include error handling. If sending the message fails, the program will disconnect from the SMF in memory resource and end with an exception. A production version should probably catch certain errors and retry.

Filed Under: Java, Uncategorized

DCOLLECT Reports and DCOLLECT to JSON using Java

October 27, 2023 by Andrew

EasySMF:JE version 2.2.1 added support for DCOLLECT records.

DCOLLECT reports using Java are fast and powerful. Programs can run on z/OS on zIIP processors, or DCOLLECT data can be downloaded to another platform for reporting.

Sample reports are provided for several different scenarios:

  • Space by SMS storage group and volume
  • Space by high level qualifier, on primary volumes and HSM ML1 and ML2
  • Space by time since last reference (7 days, 1 month, 6 months, 1 year, 5 years)
  • Aged datasets – datasets on primary volumes with a last reference data older than a cutoff e.g. 5 years
  • Datasets by migration date (7 days, 1 month, 6 months, 1 year, 5 years)
  • Frequently migrated datasets – datasets that are being migrated and recalled frequently
  • zEDC compression statistics by high level qualifier

Samples are also provided to compare 2 different DCOLLECT runs and find what changed:

  • Change in space by storage group
  • Change in space by the top 50 high level qualifiers
  • Change in space for selected dataset names

Convert DCOLLECT Records to JSON format

DCOLLECT records can also be converted to JSON format, for use with Splunk, Excel or other JSON reporting tools.

See the Samples

Sample code is available on Github:

https://github.com/BlackHillSoftware/easysmf-samples/tree/main/dcollect

30 Day Trial

Get a 30 Day Trial, or request further information:

Loading

Filed Under: Uncategorized

Java SMF API Version 1.2.1

July 15, 2016 by Andrew

Version 1.2.1 of the EasySMF:JE Java API is now available.

Major Changes

  • Add SMF Type 121 (Java Statistics) record.

    This record has some interesting information, but has one major drawback. The ID running the Java job needs access to write SMF records. This is probably a security issue in many shops, because in general you don’t want to allow unauthorized users to write SMF records. It would be much nicer if this information was written by the JVM, without requiring special access for the user.

  • Modify samples to demonstrate use of Java 8 Streams functions. Streams are very nice for processing SMF data, and in many cases allow you to greatly simplify your code. It is unclear at the moment what the performance implications are. There are suggestions in some places that the code generated by Streams is not as well optimized as loops and iterators. That may be the case, but I don’t know yet whether the difference is detectable.

Filed Under: Uncategorized Tagged With: Java

Java API for DB2 SMF Records

May 19, 2016 by Andrew

Version 1.2.0 of the EasySMF:JE Java API is now available.

Major Changes

  • SMF Type 89 Support
  • SMF type 42 Support
  • DB2 SMF type 100 and 101 support

Java API for DB2 SMF Records

This support is currently experimental while we learn more about the specifics of DB2 SMF data. Breaking changes to the API are more likely while it is in experimental status.

Using the API

Typical steps to use the package are:

  1. Read SMF records using the SmfRecordReader class
  2. Check the record type and other relevant attributes of each SmfRecord to see whether it is one you are interested in
  3. Create the specific record type from the SmfRecord e.g. new Smf100Record(record)
  4. Process information from the specific record as required
DB2 Record Compression

Records compressed with CSRCESRV will be automatically expanded when the SmfDb2Record is constructed.

IFCID

The IFCID of the record can be determined using the SmfDb2Record.ifcid() method.

Accessing Record Data

DB2 records are typically made up of a varying number of sections of different types. Sections of a specific type are returned in a List<E> of that type. If there are no sections of the type in the record an empty List is returned. This allows you to iterate over the sections without explicitly checking whether the sections exist in the record – an empty list will iterate 0 times. However, you may need to check the IFCID if the section is present in multiple IFCIDs but you want data from a specific IFCID.

Example

The following code reads all Qwos sections from type 100 SMF records from the DD INPUT.

 try (SmfRecordReader reader = 
         SmfRecordReader.fromDD("INPUT").include(100))  // include only type 100 records
 {
     for (SmfRecord record : reader)                    // read each record
     {
         Smf100Record r100 = new Smf100Record(record);  // construct a Smf100record
         for (Qwos qwos : r100.qwos())                  // process 0 or more Qwos sections
         {
             //...                                      // process Qwos sections here
         }
     }
 }                                                      // reader automatically closed at end
                                                        // of try with resources block.

Of course the inner loop can be replaced with whatever processing you need to do with the record.

Data Conversion

The API aims to provide a consistent interface across different types and sections, and converts values to standard Java types for simple programming. Conversions are the same as for other SMF record types.

Dates and Times

Dates and times are converted to java.time classes. Java.time can represent dates and times with a precision of 1 nanosecond.

  • Times representing a duration e.g. CPU or elapsed time are converted to Duration.
  • Dates and times of day are converted to LocalDate, LocalTime, LocalDateTime or ZonedDateTime depending on exactly what information is in the field. Typically, times based on UTC(GMT) are converted to ZonedDateTime with ZoneOffset.UTC. Other dates and times are converted to LocalDate/Times.
    Java has time zone rules so it is possible to apply a ZoneId to a LocalDateTime and perform date aware conversions between time zones.

The raw numeric value of date and time fields is also provided.

Numeric Values
  • 1, 2 and 3 byte integer values and 4 byte signed integer values are converted to int (32 bit signed) values.
  • 4-7 byte integer values and 8 byte signed values are converted to long (64 bit signed).
  • 8 byte unsigned values are available as both long (64 bit signed) and as a BigInteger. The long value may provide better performance if the value will not exceed the maximum value for a long. If a value does exceed the maximum value (i.e. the high order bit is set) an exception will be thrown.
    If the field value might exceed the maximum value for a long, use the BigInteger version.
  • Integer values greater than 8 bytes are converted to BigInteger.
  • Floating point values are converted to Java double.

Filed Under: Uncategorized

30 Day Trial

EasySMF and EasySMF:JE are available for a free 30 day trial. Download now and start using them immediately.
30 Day Trial

Information

EasySMF:JE Java API for SMF: Quickstart

Java vs C++ : Drag Racing on z/OS

News

  • Using zEDC compression for SMF data
  • Text message alerts using the z/OS SMF Real Time Interface
  • DCOLLECT Reports and DCOLLECT to JSON using Java

Black Hill Software

Suite 10b, 28 University Drive, Mt Helen, VIC 3350, Australia
PO Box 2214, Bakery Hill, VIC 3354, Australia
+61 3 5331 8201
+1 (310) 634 9882
info@blackhillsoftware.com

News

  • Using zEDC compression for SMF data
  • Text message alerts using the z/OS SMF Real Time Interface
  • DCOLLECT Reports and DCOLLECT to JSON using Java

Copyright © 2025 · Enterprise Pro Theme on Genesis Framework · WordPress · Log in