Main Content

detectImportOptions

Create import options based on event stream content

Since R2022b

collapse all in page

    This function requires Streaming Data Framework for MATLAB® Production Server™.

    Syntax

    opts = detectImportOptions(stream)
    opts = detectImportOptions(stream,Event=e)

    Description

    opts = detectImportOptions(stream) obtains the event schema from an event stream and based on that schema, returns data import options in the ImportOptions object opts. The schema includes the names and data types of variables in the event body.

    You can modify opts later and use it with readtimetable to control how MATLAB imports stream events into MATLAB timetables.

    example

    opts = detectImportOptions(stream,Event=e) analyzes the event structure array e to determine the event schema. Use this syntax only if the event stream that stream connects to is unavailable. You can either manually create this array, which can be complex, or use one that was previously obtained from the stream, such as from a call to the readevents function.

    Examples

    collapse all

    Assume that you have a Kafka® server running at the network address kafka.host.com:9092 that has a topic Triangles. Each event has one data column, "triangle", which contains a structure with three fields, "x", "y", "z", containing the integer side lengths of a triangle.

    Create a KafkaStream object connected to the Triangles topic.

    ks = kafkaStream("kafka.host.com",9092,"Triangles");

    Create an ImportOptions object from the Kafka stream object. The data type of the length of each side is string.

    opts = detectImportOptions(ks)
    opts = 

  ImportOptions with properties:

            VariableNames: ["triangle/x"    "triangle/y"    "triangle/z"]
            VariableTypes: ["string"    "string"    "string"]
              KeyVariable: "key"
    SelectedVariableNames: ["triangle/x"    "triangle/y"    "triangle/z"]

    To perform mathematical operations on the imported data, update the data type of variables to double. Because the side length variables are nested within "triangle", use a forward slash ("/") to specify the path to these variables.

    opts = setvartype(opts, ["triangle/x", "triangle/y", "triangle/z"], "double")
    opts = 

  ImportOptions with properties:

            VariableNames: ["triangle/x"    "triangle/y"    "triangle/z"]
            VariableTypes: ["double"    "double"    "double"]
              KeyVariable: "key"
    SelectedVariableNames: ["triangle/x"    "triangle/y"    "triangle/z"]

    Update the ImportOptions property of the KafkaStream object.

    ks.ImportOptions = opts
    ks = 

  KafkaStream with properties:

                  Topic: "Triangles"
                  Group: "85c42e39-695d-467a-86f0-f0095792e7de"
                  Order: EventTime
                   Host: "kafka.host.com"
                   Port: 9092
      ConnectionTimeout: 30
         RequestTimeout: 61
          ImportOptions: "Import to MATLAB types"
          ExportOptions: "Source: function eventSchema"
          PublishSchema: "true"
             WindowSize: 50
            KeyVariable: "key"
            KeyEncoding: "utf16"
                KeyType: "text"
           KeyByteOrder: "BigEndian"
           BodyEncoding: "utf8"
             BodyFormat: "JSON"
              ReadLimit: "Size"
    TimestampResolution: "Milliseconds"

    When importing the triangles, readtimetable converts the side lengths to double values.

    tt = readtimetable(ks);
tt(1,:).triangle
    ans = 

  struct with fields:

    x: 3
    y: 4
    z: 5

    Input Arguments

    collapse all

    Object connected to an event stream, specified as a KafkaStream or TestStream object.

    Event information, specified as a structure array. This array is in the format returned by the readevents function.

    Each structure in the array has these fields.

    Event key as stored in Kafka, returned as a string array or integer. The key identifies the event source.

    Data Types: single | double | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64 | char | string

    Event value, specified as a byte array with a format and encoding determined by the BodyFormat and BodyEncoding properties of the stream object. The event value does not undergo schema processing and appears exactly as is stored in Kafka, for example, as a JSON string.

    Data Types: string | uint8 | uint16

    Timestamp of event occurrence or timestamp of event ingestion in Kafka, specified as a datetime scalar.

    Data Types: datetime

    Output Arguments

    collapse all

    Event stream import options, returned as an ImportOptions object.

    Version History

    Introduced in R2022b

    See Also

    | | | | |