Stream Processor Overview

Stream Processor is a service for transforming and manipulating messages between Kafka topics.

To transform input messages into one or multiple output messages the services leverages a Javascript sandbox to run the processing function.
During service startup the sandbox loads a user-defined function from a file, which can be written as a Javascript ES6 module, and it exposes the module as a function that can be called by the service code.

Common use cases for this plugin include:

• Stateless Stream Processing: generate a new stream starting from a source one. Input events may be transformed and/or filtered before being produced on the output topic. The output messages are generated based only on the input message.
• Stateful Stream Processing: similar to the previous use case, the service is able to transform an input stream of events into an output one, leveraging a stream state which can be employed to perform computations that span across multiple messages. Consequently, output messages are generated by a combination of the current input event and the actual stream state.

To enable the latter use case, the processing function can access both the input message and a cache object, which allows reading the current state of the stream and update it as needed.