Falling off an MLOG

The EQQMLOG is a valuable source of diagnostic information for the Workload Automation z/OS product set.  Every task has one, every job that interacts with the product has one, it is the go-to source to find out what is going off.

The most important MLOG is for the controller, by far the busiest, which can get very long.  Originally the only way to manage the length of the MLOG was to bounce the controller and archive the log, but in all the currently supported releases of the product it is possible to switch the MLOG whilst the controller is up.

The basic premise is that in both the controller and tracker tasks you can have 2 DD statements, EQQMLOG and EQQMLOG2.  These must both point to datasets, and the dataset names must be almost identical with the EQQMLOG2 having an additional 2 on the end.  You cannot for example have one that ends with MLOG1 and the other ends with MLOG2, as this will cause JCL errors in the switching process.

You activate the product in the task OPCOPTS parameters, telling it the record limit for the log and the name of the procedure to handle the switching.  The number can be from 0 to 999,999,999 with 0 meaning log switching is disabled.  Do not be tempted to think that 0 just means that it will not switch at a record limit, but allows manual switching as this is NOT the case.  Specifying 0 means that even if you issue a SWITCHMLOG command it will be ignored.

Once activated when the task starts it will start to write to EQQMLOG until a switch event occurs, then all subsequent log entries will be directed to EQQMLOG2 and the task will run a procedure to copy the EQQMLOG off to a GDG.  This will continue until the next switch event, at which point it goes back to writing to EQQMLOG and EQQMLOG2 is copied to a GDG.

A switch event can happen in one of 3 circumstances –

1. When the MLOG dataset reaches the record limit. If you would prefer to control exactly when the log switch event happens specify 999,999,999 as the limit which you are unlikely to hit with regular forced log switching.
2. When the task is stopped, this ensures that whenever a controller or tracker is started it always begins in a new EQQMLOG file, making it easier to find the startup messages.
3. When the SWITCHMLOG modify command is issued against the task

e.g. F IOPC,SWITCHMLOG

Bear this in mind when trying to find the GDG entry you need as on days when a task is stopped you may have unexpected extra generations.

Notes:

• The first time you use log switching, in odd circumstances, you can see some unusual behaviour until the first limit or SWITCHMLOG event occurs, in that the current MLOG is reused on restart without being cleared out, so if the previous MLOG was longer you can see older records on the end of the MLOG. This resolves itself once a true switch has taken place.
• If you have used EQQJOBS to setup your system, be aware that it does not quite do things correctly for Controller and Tracker in that both and up sharing the same datasets and switch procedure. The controller and each tracker must have their own unique pairs of MLOG datasets and the trackers must use a separate switching procedure from the controller.

Setting up log switching on a controller

You do not have to implement log switching on both the controller and trackers, you can choose to do one or the other or both.  Log switching for trackers is rare, it tends to only be applied to the controller.

First thing to do is to change the controller JCL to have an EQQMLOG2 dataset as well as EQQMLOG.  As mentioned before the only difference between the dataset names is that EQQMLOG2 must have 2 appended to the name that EQQMLOG uses, otherwise JCL errors will occur.  The datasets should have been pre-allocated in the EQQPCSxx setup jobs, but if not, allocate EQQMLOG2 the same as EQQMLOG.

Next, add the SWITCHMLOGLIM and MLOGPROCNAME keywords to the OPCOPTS statement for your controller.  Use the maximum 999999999 if you intend to control when the switch occurs by command.

You must also create the switch log procedure.  This should be placed in a procedure library that is known to JES.

Here you can see exactly why the EQQMLOG and EQQMLOG2 must have almost identical dataset names.  When the procedure is submitted for EQQMLOG symbol P1 is not set and for EQQMLOG2 it is set to 2, so the only difference can be 2 or nothing.  There is no way to influence this behaviour so we are stuck with this convention.

To switch the MLOG on demand you need to issue the SWITCHMLOG modify command to the controller at the time you wish the log switch to occur e.g. F IOPC,SWITCHMLOG.  You can do this either by a timed event in whatever automation tool you use, or you could issue a command from JCL.

JCL has a little-known COMMAND statement which can be used to issue an operator command from a batch job.  This allows simple commands to be scheduled at any time you need within a time dependent batch job.

To run this job the following considerations must be made –

• The job class in which the job runs must have the COMMAND attribute set to EXECUTE.
• The user under which the job runs must have the authority to issue the command
• The job must run on the same machine as the controller. To do this schedule the job from a workstation with an empty destination field, which will cause the job to be submitted directly from the controller and SYSAFF=* will ensure it runs on the LPAR it was submitted to.
• The job must contain a real step, it cannot contain COMMAND statements alone.

Setting up log switching on a tracker

If you want to set up log switching on a tracker, the process is largely the same, but you need to consider that you will have the same tracker task running across multiple LPARs at the same time, so each will need its own pair of MLOG datasets.  This is easy to do using the SYSCLONE system symbol which should have a unique 2 character value for each LPAR in the SYSPLEX.

In your tracker task it would look something like this –

Because of this you will also need to point to a different switch procedure in the tracker OPCOPTS statement –

Which would look something like this, noting that as well as a unique pair of MLOG datasets, each tracker would also need its own GDG base –

Forcing the switch to happen would use a similar job, but in this case the modify command would be directed at each tracker, running a job for each tracker on the appropriate workstation for that LPAR.

Considerations for running with Hot Standby

Assuming you have the necessary subsystem entries and tasks defined on each LPAR, you can actually start the controller on any LPAR in the SYSPLEX.  This allows for control to be moved between LPARs to allow rolling IPLs, keeping workload running 24×7.  Often referred to as a Standby Controller, the standby can either be COLD or HOT.

COLD simply means that you take the controller down on one LPAR and start it on another.  During this time existing workload continues to run, but no new jobs will be submitted.  This process is normally managed by an automation product.

HOT means that a controller is started on more than one LPAR at once.  Only one controller is fully functioning, the other has performed basic startup and is watching the other controller for failure, to takeover automatically.  Takeover can also be forced by a command.  During the takeover existing workload continues to run, but no new jobs will be submitted.

The MLOG consideration for HOT stand by is more than one controller is simultaneously writing to EQQMLOG, so each standby controller needs its own unique pair of MLOG datasets.

This can be handled using SYSCLONE similar to how you handle tracker instances, the only difference being you may choose to archive all controller logs to a single GDG, to give one continuous flow of controller information.  The choice is yours.

Datasets versus SYSOUT

To use the log switching process you MUST send EQQMLOG and EQQMLOG2 to datasets.  Whilst this might be the preferred approach to the workload support personnel, as they can more accurately target the messages they need, many Systems Programmers prefer the messages to appear in the SYSOUT.  This makes it quicker and easier to confirm the health or diagnose issues of a task that has just been restarted.

The good news is that it is possible to do both at the same time.  Whilst EQQMLOG and EQQMLOG2 must be written to disk, there is an additional undocumented optional DD statement that can be used called EQQLOOP.

You can add an EQQLOOP DD statement to your controller or tracker JCL and point it to SYSOUT, this will then display the EQQMLOG from the point of startup to the current time, just like EQQMLOG would do without log switching enabled.