Streaming Data Framework for MATLAB Production Server Basics

Use Streaming Data Framework for MATLAB^® Production Server™ to read from and write to event streaming platforms, such as Kafka^®. Using this framework, you can:

Develop a streaming analytic function in MATLAB that filters, transforms, records, or processes event stream data.
Connect to a streaming source and test how the analytic function reads from and writes to event streams by using Streaming Data Framework for MATLAB Production Server functions.
Simulate the production environment for testing your streaming analytic algorithms (requires MATLAB Compiler SDK™).
Package the analytic function (requires MATLAB Compiler SDK) and deploy it to MATLAB Production Server.

Install Streaming Data Framework for MATLAB Production Server

Install the Streaming Data Framework for MATLAB Production Server support package from the MATLAB Add-On Explorer. For information about installing add-ons, see Get and Manage Add-Ons (MATLAB).

After your installation is complete, find examples in support_package_root\toolbox\mps\streaming\Examples, where support_package_root is the root folder of support packages on your system. To get the path to this folder, use this command:

fullfile(matlabshared.supportpkg.getSupportPackageRoot,'toolbox','mps','streaming','Examples')

System Requirements

Streaming Data Framework for MATLAB Production Server has the same system requirements as MATLAB. For more information, see System Requirements for MATLAB.

Write Streaming Analytic MATLAB Function

The event stream analytic function typically consumes a stream of input events and can produce a stream of output events. It can filter, transform, record, or process the stream of events by using any MATLAB functionality that is deployable to MATLAB Production Server.

Event stream analytic functions process windows or batches of events. An event consists of three parts:

Key — Identifies the event source
Timestamp — Indicates the time at which the event occurred
Body — Contains event data, specified as an unordered set of (name, value) pairs

Analytic functions read events into a timetable. Each row of the timetable represents a streaming event, typically in chronological order. If the analytic function produces results, they must also be timetables.

When processing a stream, you can call an analytic function several times, because the window size is typically much smaller than the number of messages in the stream. The stateless execution model of MATLAB Production Server isolates the processing of each window, so the processing of one window does not affect the processing of the next. Stateful functions that require interaction between the processing of consecutive windows specify a MATLAB structure that is preserved between windows and passed to the next invocation of the analytic function.

An analytic function can have one of three signatures:

Function Signature	Description
results = `analyticFcn`(data)	Stateless analytic function that emits a stream of results
[ results, state ] = `analyticFcn`(data, state)	Stateful analytic function that preserves state between batches and emits a stream of results
`analyticFcn`(data)	Stateless analytic function that does not emit a stream of results

Stateless Analytic Function

The following plotSierpinski function is an example of a stateless analytic function. plotSierpinski plots the X and Y columns of the input timetable. The source code for this function and a script to run it is located in the \Examples\ExportOptions folder.

function howMany = plotSierpinski(xyData)

    hold on
    arrayfun(@(x,y)plot(x,y,'ro-', 'MarkerSize', 2), [xyData.X], [xyData.Y]);
    hold off
    drawnow
    count = height(xyData);
    howMany = timetable(xyData.Properties.RowTimes(end), count);
end

Stateful Analytic Function

The following recamanSum function is an example of a stateful analytic function. In stateful functions, the data state is shared between events and past events can influence the way current events are processed. recamanSum computes the cumulative sum of a numeric sequence. In returns two values:

cSum — A table that contains the cumulative sum of the elements in the stream
state — A structure that contains the final value of the sequence

The source code for the recamanSum function, it initialization function initRecamanSum, and the scripts used to run the analytic function are located in the \Examples\Numeric folder.

function [cSum, state] = recamanSum(data, state)
    timestamp = data.Properties.RowTimes; 
    key = data.key;
    
    sum = cumsum(data.R) + state.cumsum;
    
    state.cumsum = sum(end);

    cSum = timetable(timestamp, key, sum);
end

Process Kafka Events Using MATLAB

To process events from a stream, you create an object to connect to the stream, read events from the stream, iterate the streaming analytic function to process the several windows of events, and, if the analytic function produces results, create a different stream object to write the results to stream.

The following code sample gives an overview of processing one window of events using the framework. Assume that you have a Kafka host running at the network address kafka.host.com:9092 that has a topic recamanSum_data. Also, assume that the recamanSum_data topic contains the first 1000 elements of the Recamán sequence..

Create a KafkaStream object for reading from and writing to the recamanSum_data topic.
```
inKS = kafkaStream("kafka.host.com",9092,"recamanSum_data");
```
Read events from the recamanSum_data topic into a timetable tt.
```
tt = readtimetable(inKS);
```
Call the recamanSum function and calculate the cumulative sum of a part of Recamán's sequence in tt. Since recamanSum is a stateful function, first call the initRecamSum function, which initializes state.
```
state = initRecamanSum();
[results, state] = recamanSum(tt,state);
```

For a detailed example of how to process several windows of events, see Process Kafka Events Using MATLAB.

Simulate Production Using Development Version of MATLAB Production Server

Before deploying to MATLAB Production Server, you can test the streaming analytic function using the development version of MATLAB Production Server, which acts as a local test server. For a detailed example, see Test Streaming Analytic Function Using Local Test Server.

Deploy Streaming Analytic to MATLAB Production Server

You can also package the analytic function and deploy it to MATLAB Production Server. For a detailed example, see Deploy Streaming Analytic Function to MATLAB Production Server.