In this post, I’ll show how you can send SMS text messages from z/OS for failed jobs using Twilio and the z/OS SMF Real Time Interface.

Twilio provides a Java API to use their service, and that can be combined with the EasySMF Real Time Interface to send SMS messages based on real time SMF data.

Sending text messages from z/OS

Twilio is a paid service for production messaging, but you can sign up for a free trial account and send a limited number of messages to a verified phone number.

Twilio provides quickstart documentation here:

https://www.twilio.com/docs/messaging/quickstart/java

You can ignore the stuff about the CLI, all we are going to do is send an outbound message which requires a Twilio account, the Java program and dependencies i.e. the fat jar.

This is the Twilio quickstart Java program with a couple of minor tweaks:

import com.twilio.Twilio;
import com.twilio.rest.api.v2010.account.Message;
import com.twilio.type.PhoneNumber;

public class TwilioTest {
    // Find your Account SID and Auth Token at twilio.com/console
    // and set the environment variables. See http://twil.io/secure
    public static final String ACCOUNT_SID = System.getenv("TWILIO_ACCOUNT_SID");
    public static final String AUTH_TOKEN = System.getenv("TWILIO_AUTH_TOKEN");

    public static void main(String[] args) {
        Twilio.init(ACCOUNT_SID, AUTH_TOKEN);
        Message message = Message.creator(
                new com.twilio.type.PhoneNumber("+14159352345"), // to
                new com.twilio.type.PhoneNumber("+14158141829"), // from
                args[0])
            .create();

        System.out.println(message.getSid());
    }
}

Upload the Twilio fat jar twilio-x.x-jar-with-dependencies.jar to z/OS. It is available from e.g. https://repo1.maven.org/maven2/com/twilio/sdk/twilio/10.1.3/

Java 11 will run single file Java programs without a separate compilation step, so we can just run the program under BPXBATCH:

//ANDREWRG JOB CLASS=A,
//             MSGCLASS=H,
//             NOTIFY=&SYSUID
//*
//BPXBATCH EXEC PGM=BPXBATCH,REGION=512M
//STDPARM  DD *
sh /usr/lpp/java/J11.0_64/bin/java
 /home/andrewr/java/src/TwilioTest.java
 "Hello from z/OS"
//STDENV   DD *
CLASSPATH=/home/andrewr/java/lib/twilio-10.1.3-jar-with-dependencies.jar
TWILIO_ACCOUNT_SID=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
TWILIO_AUTH_TOKEN=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
//SYSOUT   DD SYSOUT=*
//STDOUT   DD SYSOUT=*
//STDERR   DD SYSOUT=*

Twilio can also send the messages to WhatsApp using the same functionality.

Sending notifications for failed jobs

Prerequisite: z/OS SMF Real Time Interface

The z/OS SMF Real Time Interface must be active. This requires:

• SMF running in logstream mode
• Define an in-memory resource to receive the required records (type 30 in this case)
• Setup RACF definitions to allow the user running Java program to access the SMF in memory resource

The program

The complete source code for this program can be found here:

https://github.com/BlackHillSoftware/easysmf-samples/tree/main/easysmf-rti/rti-notifications

The following is an overview of the major sections of the program.

Reading the in-memory resource

The first thing to do is set up the connection to the in-memory resource.

public class RtiNotifications
{
    public static void main(String[] args) throws IOException
    {
        try (SmfConnection connection = 
                 SmfConnection.forResourceName("IFASMF.MYRECS")
                     .onMissedData(RtiNotifications::handleMissedData)
                     .disconnectOnStop()
                     .connect();

             // Set up SmfrecordReader to read type 30 subtypes 4 and 5    
             SmfRecordReader reader = 
                 SmfRecordReader.fromByteArrays(connection)
                     .include(30,4)
                     .include(30,5))
        {
            // process data here
        }
    }

    private static void handleMissedData(MissedDataEvent e)
    {
        System.out.println("Missed Data!");
        e.throwException(false);    
    }
}

This code:

• creates a real time connection to resource name IFASMF.MYRECS
• sets up an action to be taken if data is written faster than the program can read it and the real time buffer wraps (write a message and suppress the exception)
• sets up a command handler to disconnect from the resource when the STOP command is received
• creates a SmfRecordReader to read SMF 30 subtypes 4 and 5 (step and job end) records

The connection and reader will be closed automatically when the program exits the try block.

SIMULATE_RTI=Y

IFASMF_MYRECS=/your/smf/filename

The SmfConnection will read SMF data from the file indicated by the environment variable matching the resource name.

Processing the SMF 30 Records

We want to send the SMS message when the job ends, which is indicated by a subtype 5 record. However, if steps are skipped due to e.g. COND processing or an ABEND, the information in the subtype 5 record might be from a step that didn’t run. These steps have the “flushed” indicator set in the Completion Section.

We need to keep the step end information for steps that did run and check the last executed step when we see the job end record. The Java HashMap provides a convenient way to store information for failed steps.

We can create a class to use as a HashMap key to identify specific jobs. The class has fields for the identifying information (system, jobname, job number, read date and time) which are populated in the constructor. Eclipse can generate the hashCode and equals methods which are required for a HashMap key.

Map<JobKey, Smf30Record> failedSteps = new HashMap<>();

...

private static class JobKey
{
    String system;
    String jobname;
    String jobnumber;
    long readtime;
    int readdate;

    JobKey(Smf30Record r30)
    {
        system = r30.system();
        jobname = r30.identificationSection().smf30jbn();
        jobnumber = r30.identificationSection().smf30jnm();
        readtime = r30.identificationSection().smf30rstRawValue();
        readdate = r30.identificationSection().smf30rsdRawValue();
    }

    // hashCode and equals generated using Eclipse
    @Override
    public int hashCode() {
        return Objects.hash(jobname, jobnumber, readdate, readtime, system);
    }

    @Override
    public boolean equals(Object obj) {
        if (this == obj)
            return true;
        if (obj == null)
            return false;
        if (getClass() != obj.getClass())
            return false;
        JobKey other = (JobKey) obj;
        return Objects.equals(jobname, other.jobname) 
                && Objects.equals(jobnumber, other.jobnumber)
                && readdate == other.readdate 
                && readtime == other.readtime 
                && Objects.equals(system, other.system);
    }
}

SMF 30 Subtype 4: Step End

When processing step end records, if the step failed we save the record for later. If it ran successfully, we remove any previous failed step.

if (failed(r30)) 
{  
    // this replaces existing entry if present 
    failedSteps.put(new JobKey(r30), r30);
}
// else it didn't fail, check it wasn't flushed
else if (!r30.completionSection().smf30flh()) 
{
    // If the step wasn't flushed, remove any earlier failed step
    failedSteps.remove(new JobKey(r30));
}

failed is a method which checks whether the step failed:

private static boolean failed(Smf30Record r30) 
{
    return 
            // abended except for S222 (cancelled)
            (r30.completionSection().smf30abd() 
                    && r30.completionSection().smf30scc() != 0x222)
            // or condition code > 8
            || r30.completionSection().smf30scc() > 8
            // or post execution error
            || r30.completionSection().smf30sye();
}

The criteria can be adjusted as required.

SMF 30 Subtype 5: Job End

When the job ends, we check whether we previously saved a failed step.

JobKey key = new JobKey(r30);                        
if (failedSteps.containsKey(key))
{
    // send notification using information from failed step 
    sendNotification(failedSteps.get(key));                            
    failedSteps.remove(key); // finished with this
}
// or if the type 5 record indicates failure
else if(failed(r30))
{
    // send notification using information from job
    sendNotification(r30);                            
}

Sending the SMS message

The job information is extracted from the SMF record, the Twilio information is supplied in environment variables, and sending the SMS message uses the same process as the Twilio quickstart.

private static final String ACCOUNT_SID = System.getenv("TWILIO_ACCOUNT_SID");
private static final String AUTH_TOKEN = System.getenv("TWILIO_AUTH_TOKEN");
private static final String TO_PHONE = System.getenv("TO_PHONE");
private static final String FROM_PHONE = System.getenv("FROM_PHONE");

private static void sendNotification(Smf30Record r30) 
{
    String messagetext = 
            String.format("%s Job failed: %s %s Step: %d %s Program: %s CC: %s",
                    r30.smfDateTime().toString(),
                    r30.identificationSection().smf30jbn(), // job name
                    r30.identificationSection().smf30jnm(), // job number
                    r30.identificationSection().smf30stn(), // step number
                    r30.identificationSection().smf30stm(), // step name
                    r30.identificationSection().smf30pgm(), // program
                    r30.completionSection().completionDescription());
    
    // Send a SMS notification through Twilio
    Twilio.init(ACCOUNT_SID, AUTH_TOKEN);
    Message message = Message.creator(
            new com.twilio.type.PhoneNumber(TO_PHONE),
            new com.twilio.type.PhoneNumber(FROM_PHONE),
            messagetext)
            .create();
    System.out.println("Message Sent: " + message.getSid());
}

What Next?

Browse and build the EasySMF:RTI sample code:

https://github.com/BlackHillSoftware/easysmf-samples/tree/main/easysmf-rti/rti-notifications

The Github sample has some additional functionality not covered here. You can limit the notifications to jobs running in specific job classes. You can use the same functionality to include jobs by job name or create custom failure criteria for specific jobs.

For simplicity, the sample doesn’t include error handling. If sending the message fails, the program will disconnect from the SMF in memory resource and end with an exception. A production version should probably catch certain errors and retry.