Piper Push Cache Documentation Download Samples Contact  

Change Log

The change log, in rough terms, contains the changes made the system. The contents are controlled by record groups described in the 'groups' help section. If a record arrives and matches a group that writes to the change log it is entered in the log along with the record identifier for the record. If that record is deleted another entry in the log is made.

The change log is a set of separate JSON objects separated by a newline. Each JSON object has one member. The member key is the record identifier for the record and the member value is the JSON record itself. A change in the record will result in another entry in the change log. If a record is deleted an entry is made with the record identifier key and a null value.

Entries are not one single JSON array but rather a sequence of separate JSON entities. This allows for easier recovery and lower memory use when extracting data from the log files.

The change log is a rolling log. When a maximum size is exceeded the log is moved aside and a new log is started. The maximum number of files can be configured with the configuration parameters.

The change_log configuration parameters describe the options available for the change log as follows.

{
    "change_log": {
        "active": true,
        "dir": "log",
        "filename": "piper-changes.json",
        "max_files": 5,
        "max_size": 100000000,
        "stdout": false,
        "with_from": true
    }
}

The parameter descriptions are:

active Flag indicating whether the change log should be written to. True turns on the feature and false turns it off.
dir The directory to create the change log files in.
filename The name of the active change log file in the change log directory. This is also the template for archived change log file names. If the filename is 'changes.json' then the first archive file will be named 'changes.1.json'. Note the file extension is preserved.
max_files The maximum number of files to keep before removing old change log files.
max_size The size that when exceeded will cause a roll over of the change log to a new file.
stdout A flag that is set to true will also write the change log to stdout.
with_from A flag that if set to true will cause the 'from' field to be placed in the change log.