2020-10-30 - Job messages and payloads


In case of a structural change in a dataset you might need to migrate serialized data in a way to make in work with the latest code. The data that is affected of the structural change can still be within a message queue provider and is often out of access until message handling. You could unintentionally send duplicated messages to drain performance and increase I/O operations overall.


  • Extract job actions from the messages into the storage.
  • Separate job payloads from their actions.
  • Prevent sending of duplicate messages.
  • Normalize the structure of a job regarding the current message structure.


This change adds a little overhead on the message dispatching but adds multiple benefits regarding upcoming structure changes and future I/O operations that can be prevented.


  • Prevention of duplicate messages reduces handler calls and follow-up I/O operations.
  • Message payloads can be accessed without knowledge about the used message provider.
  • When emptying a message queue the messages can be reconstructed in a plausible order.
  • The message provider has less data to store than before.
  • It is easy to add new job types in the core without changing the storage.


  • More I/O operations have to be done when the message handlers have to hydrate the message with the payload storage before processing can be continued.
  • It is difficult to add new job types that are not similar to the existing jobs when there is a structural mismatch.

Datasets could use some sort of migration pattern which can be applied by the storage implementations onto their stored payload to react to a structural change. When a new job has to be introduced that is not related to this job pattern which is tied to mapping components a new message type can be introduced instead.