Journaling - Using APYJRNCHG for Disaster Recovery

What are the requirements to use APYJRNCHG?

To use the APYJRNCHG command to recover lost transactions, a few things are needed up front:

• All objects that will need to be recovered must be saved regularly.
• All objects need to be journaled. When objects are journaled, all changes made to them are recorded in journal receivers.
• Journal receivers must be saved. Recovery can only be accomplished as far as the last available journal entry, so save journal receivers often. We recommend that journal receivers be saved as soon as they are detached from the journal or sent to a remote system via remote journal.

At the backup site:

• All objects from the last save must be restored, along with all of the journal receivers saved since then.
• The objects must be journaled to the same journal, and the restored receivers must be associated with that journal.

It is best if the journal is restored before restoring any other objects or journal receivers. If this restore order is used, the objects will start journaling automatically and the journal receivers will associate with the journal automatically.

Finally:

• Practice disaster recovery scenarios.

This is not a requirement to get APYJRNCHG to work, but the process of restoring many objects and calling APYJRNCHG correctly is a major operation and prone to error. Also, sometimes normal business applications perform operations that will cause APYJRNCHG to fail (for example, ending and restarting journaling). Errors in your disaster recovery process and problematic operations can be addressed, but you do not want to discover these things when you are desperately trying to recover your business.

What parameters should be specified on APYJRNCHG?

There are three things that APYJRNCHG needs to know:

• What journal and journal receivers to apply from
• What are the starting and ending points
• Which objects to apply entries for

We discuss these points in more detail in the following sections.

What journal and journal receivers to apply from
For this requirement, specify the journal that the objects are journaled to. The default value for the RCVRNG (Range of journal receivers) parameter, *LASTSAVE, should be sufficient, as long as you are specifying *LASTSAVE on the FROMENTLRG parameter as well (see below). Otherwise, specify the first and last journal receivers in the range of journal receivers that were restored.

What are the starting and ending points
This requirement is deceptively simple. If *LASTSAVE is specified on the FROMENTLRG (Starting large sequence number) parameter, which is the default, the operating system will seek out the "Object saved" journal entries for every object. Each of these journal entries will be used as the starting point for applying journaled changes. Essentially, this means all changes that occurred after the objects were saved will be applied. This even works if save-while-active is used. For the ending point, on the TOENTLRG (Ending sequence number) parameter, specify *LAST. This will apply changes up to the end of the last journal receiver. Note that the default value, *LASTRST, will not work because the "Object restored" journal entries are on a different journal receiver chain than the journal entries you wish to apply.

Which objects to apply entries for
This requirement is either the simplest or the most complicated. The simple answer is to specify *ALLJRNOBJ on the OBJ (Objects) parameter. This will apply changes to all objects journaled to that journal. However, there may be occasions where you want to apply changes only to specific objects. To limit the selection of objects, use the OBJ (Objects), OBJPATH (Object paths), SUBTREE (Subtree), PATTERN (Pattern), and APYLF (Apply changes to logical files) parameters.

There are a few other parameters that are worth considering. The OBJERROPT (Object error option) parameter tells the operating system how to react to failures. The default value, *CONTINUE, will continue processing other objects even if a failure occurs for one object. The other value, *END, will cause APYJRNCHG to stop processing immediately if it fails to apply any journal entry. *CONTINUE is the default and is usually preferred. If *CONTINUE is used, be sure to check the joblog, the journal, and/or the outfile to see which, if any, objects encountered problems.

Regarding the outfile, another useful parameter is the OUTPUT (Output) parameter. If you specify *OUTFILE on the OUTPUT parameter and specify an outfile to send the results to, a file will be created with details on which objects where applied to, how many entries were applied to each, which objects encountered problems, and more. Sending the results to an outfile is highly recommended, especially when applying to a large number of objects.

APYJRNCHG versus APYJRNCHGX

There are actually two Apply Journaled Changes commands, APYJRNCHG and APYJRNCHGX (Apply Journaled Changes Extend). Why are there two? Historically, these two commands handled different types of journaled changes and had different restrictions on which objects they would apply changes for. Because APYJRNCHGX was introduced, changes have been made to incorporate APYJRNCHGX functionality into APYJRNCHG.

Starting in V5R3M0, most of the functionality of APYJRNCHGX has been incorporated into APYJRNCHG, with two notable exceptions:

• APYJRNCHGX is able to replay the creation of new database files, but APYJRNCHG cannot.
• APYJRNCHGX can also apply changes to logical files, but APYJRNCHG cannot.

This added functionality, however, comes at a price:

• APYJRNCHGX can only apply changes to database files. In contrast, APYJRNCHG can also apply to data areas, data queues, and Integrated File System objects.
• APYJRNCHGX is only capable of applying entries to all files in a library or a list of libraries; it does not allow individual object selection. APYJRNCHG allows individual objects to be selected.

Depending on your applications, you may need to choose one command over the other, or you may need to use a combination of both commands.

In IBM i 6.1, all of the functionality of APYJRNCHGX has been incorporated into APYJRNCHG. APYJRNCHG is able to apply the creation of new database files and is able to apply changes to logical files. In addition, APYJRNCHG is also able to apply the creates of data areas and data queues.

Preparation is required to use this support, however. To apply "Object created" journal entries, the library that the objects are created into must be journaled. That library must also be included in the list of objects on your APYJRNCHG command. If the library is not journaled, the object creation journal entries will only be applied if *ALLJRNOBJ is specified on the OBJ (Objects) parameter. Of course, this results in a loss of flexibility in choosing which objects to apply entries for. To apply entries for logical files, simply specify *YES on the APYLF (Apply changes to logical files) parameter. This will apply changes to all logical files that are associated with any of the physical files in the list of objects to apply entries for.

It is worth noting that APYJRNCHG will not be able to apply object creation entries or logical file entries for journal receivers created prior to IBM i 6.1 unless OBJ(*ALLJRNOBJ) is specified. If you have journal receivers from a V5R4M0 or earlier system, either specify OBJ(*ALLJRNOBJ) or use APYJRNCHGX to apply file creation and logical file entries.

Troubleshooting

Replaying a day's worth of activity to a large set of objects of various types is a complex operation, and many things can go wrong. For example, if a record of a database file is deleted before calling APYJRNCHG, the command will fail if it tries to update that record. Or if a user profile does not exist on the backup system, APYJRNCHG will fail if it attempts to grant authority to that user profile. Another potential problem arises if your application routinely ends and restarts journaling. APYJRNCHG will fail for an object if journaling was ended for that object in the range of journal entries being applied.

So what actions do you need to take if APYJRNCHG fails? The joblog will contain information about the failure that occurred, including which journal entry failed to apply. Inspect the error, address the problem, and then attempt the APYJRNCHG again. If OBJERROPT(*CONTINUE) was specified, it is possible to apply entries for just that object on a subsequent APYJRNCHG. Restore the object again and attempt the APYJRNCHG from *LASTSAVE for just that object. If OBJERROPT(*END) was specified, you will need to apply to the entire list of objects again. It is sometimes possible to pick up your APYJRNCHG command where it left off, but this is not always the case. For example, any journal entries created under commitment control that were not committed get rolled back (depending on the CMTBDY parameter).Therefore, your restart point may be earlier than the failing journal entry, and may be different for different objects. It is safest to simply restore the objects anew and call the same APYJRNCHG command.

Here is an example. In this example, the original application does not use commitment control. The initial APYJRNCHG command succeeds for all objects except one (OBJERROPT(*CONTINUE) was specified). The outfile indicates which journal entry failed to apply, and the joblog contains a message indicating what the failure was. In this example, the entry had a journal code of D and an entry type of GT, which indicates a Grant authority journal entry. The message in the joblog indicates that the user profile in the journal entry does not exist on the system. To recover, create or restore that user profile onto the disaster recovery system. Then, use the APYJRNCHG command again, this time specifying only the object that failed and specifying the failing journal entry as the new starting point (on the FROMENTLRG parameter).

Summary

The Apply Journaled Changes (APYJRNCHG) command is a powerful tool that can be used to recover all transactions lost in a disaster. All changes made to database files, data areas, data queues, and directories and stream files in the Integrated File System can be replayed with this command. Use of this command can be a key component of a disaster recovery process. However, like all disaster recovery procedures, it should be tested and practiced regularly to avoid unpleasant surprises if a real disaster occurs.