Detect Human Presence Using WLAN Signals and Deep Learning

Introduction

WLAN sensing (also known as Wi-Fi™ sensing) uses Wi-Fi networks for applications such as gesture recognition, location tracking, monitoring vital signs, and human presence detection. WLAN sensing is currently being standardized by the IEEE® 802.11bf task group [1]. This technology can sense presence, range, velocity, and location of objects in various environments through subtle changes in the communication channel. CSI is the most common metric for determining the changes in the channel [2]. However, CSI is not typically exposed to you as a user of commercial Wi-Fi devices.

Using WLAN Toolbox™ with a supported SDR, you can capture WLAN waveforms, detect beacon packets, and extract CSI. From this CSI, you create a collection of periodogram images to use as the training and validation datasets for deep learning. Using Deep Learning Toolbox™, you train a CNN and use the trained network for live detection of human presence. Alternatively, you can use the prerecorded dataset to train the CNN and evaluate the presence detection performance against the test dataset. Beacon packets are useful for WLAN sensing as access points (APs) transmit beacon packets periodically. As a result, the sensing resolution is independent of the traffic load at the AP in the temporal domain.

SDR Setup

This section explains how to set up an SDR for capturing the data that will be used in training and live detection of human presence. If you do not have SDR hardware and plan to use the prerecorded data, you can skip this section.

This example supports these SDRs:

1) Ensure that you have installed the appropriate support package for the SDR that you intend to use and that you have configured the hardware accordingly.

2) To use your SDR as the data source for training the neural network and then inferencing with the trained CNN, select the useSDR check box.

useSDR = false; % You can skip this section if unchecked

3) Specify your capture device (rxsim.DeviceName), radio gain (rxsim.RadioGain), and a channel number (rxsim.ChannelNumber) for your operation frequency band (rxsim.FrequencyBand). These settings determine the center frequency from which your SDR captures beacon packets. If you do not know a channel number, use the OFDM Beacon Receiver Using Software-Defined Radio example to determine which channel(s) contain beacon packets.

if useSDR
    % User-defined parameters
    rxsim.DeviceName    = "Pluto";
    rxsim.RadioGain     = 15; % Can be 'AGC Slow Attack', 'AGC Fast Attack', or an integer value. See relevant documentation for selected radio for valid values of radio gain.
    rxsim.ChannelNumber = 36; % Valid values for 5 GHz band are integers in range [1, 200]
    rxsim.FrequencyBand = 5; % in GHz

    % Set up SDR receiver object
    rx = hSDRReceiver(rxsim.DeviceName);
    rx.SampleRate = 20e6; % Configured for 20 MHz since this is beacon transmission bandwidth
    rx.Gain = rxsim.RadioGain;
    rx.CenterFrequency = wlanChannelFrequency(rxsim.ChannelNumber,rxsim.FrequencyBand);
    rx.ChannelMapping = 1;
    rx.OutputDataType = 'single';

    rxsim.SDRObj = rx;
end

4) Configure these capture parameters:

if useSDR %#ok<*UNRCH>
    % User defined parameters
    rxsim.NumCaptures          = 10;
    rxsim.NumPacketsPerCapture = 8;
    rxsim.BeaconInterval       = 100; % in time units (TUs). The default value in typical APs is 100.
    rxsim.BeaconSSID           = ""; % Optional

    % Calculated parameters
    rxsim.CaptureDuration = rxsim.BeaconInterval*milliseconds(1.024)*rxsim.NumPacketsPerCapture + milliseconds(5.5); % Add 5.5 ms for beacons located at the end of the waveform

    % SDR setup is complete
    msgbox("SDR object configuration is complete! Run steps 1 and 2 to capture data.")
    return % Stop execution
end

System Description

This example uses the following process to obtain CSI images for human presence detection:

When you use an SDR, the system captures an over-the-air waveform and performs receiver processing in WLAN Toolbox to extract the CSI from beacon packets. It does the same for all the packets in a capture to form a numSubcarriers-by-rxsim.NumPacketsPerCapture raw CSI array. Repeat this process for multiple captures to create an array of images. When you create the training data, the system assigns "no-presence" or "presence" labels to the captures. It obtains the periodogram representation of the raw CSI image array by using 2D-FFT, applying [0 1] range scaling, and removing DC components along the $$x$$ and $$y$$ axes. CSI periodogram images make it easier to focus on the CSI changes caused by human motion by discarding the effects of imperfections in the hardware, software, or environment. This focus also improves the training performance. Next, you use the periodogram images as an input dataset for CNN training and validation. Finally, you use the trained CNN with live SDR captures to detect presence or test the performance of the CNN by using a subset of the prerecorded dataset. When capturing training data or performing live detection, you can visualize the magnitude spectrum and periodogram representation of the raw CSI.

This process was applied to obtain the prerecorded CSI dataset, which consists of 500 samples, with 250 labeled "no-presence" and 250 labeled "presence". When this dataset was being created, the rxsim.NumCaptures parameter was set to 250. An ADALM-Pluto SDR captured over-the-air beacon frames in a meeting room that contained furniture and had floor dimensions of 3m-by-3m. The beacon interval (rxsim.BeaconInterval) of the AP was set to 20 TUs. Two different human presence states were recorded: no presence (empty room) and presence (one person walking a random path).

Step 1: Create "no-presence" Data

In this section, you generate the "no-presence" data (dataNoPresence) and associated labels vector (labelNoPresence).

To generate the data, click Create "no-presence" Data. Two plots are generated for each capture:

In "no-presence" data, the power content of the CSI spreads over both axes of the periodogram representation due to the lack of time-domain disturbance induced by the motion.

 

if useSDR % Capture live CSI with SDR
    [dataNoPresence,labelNoPresence,timestampNoPresence] = captureCSIDataset(rxsim,"no-presence");
else % No SDR hardware, load CSI dataset
    [dataNoPresence,labelNoPresence,timestampNoPresence] = loadCSIDataset("dataset-no-presence.mat","no-presence");
end

Dimensions of the no-presence dataset (numSubcarriers x numPackets x numCaptures): [52    8  250]

Step 2: Create "presence" Data

In this section, you generate the "presence" data (dataPresence) and associated labels vector (labelPresence).

The capture visualization techniques are the same as in step 1. For the "presence" data, the periodogram plot shows focus of the CSI power content in both axes near the low frequency region that corresponds to the center of the image.

 

if useSDR % Capture CSI with SDR
    [dataPresence,labelPresence,timestampPresence] = captureCSIDataset(rxsim,"presence");
else % No SDR hardware, load CSI dataset
    [dataPresence,labelPresence,timestampPresence] = loadCSIDataset("dataset-presence.mat","presence");
end

Dimensions of the presence dataset (numSubcarriers x numPackets x numCaptures): [52    8  250]

Step 3: Create and Train CNN

CNNs are one of the most computationally efficient, accurate, and scalable approaches to problems involving deep learning. Their predominant application is to deep learning tasks based on computer vision [3]. In this section, you define the CNN architecture and divide the data into training (trainingData) and validation (validationData) datasets to evaluate the training and final network performance.

To create, train and validate the CNN click Create CNN and Train.

 

trainingRatio =  0.8;
numEpochs = 10;
sizeMiniBatch = 8;

Use the trainValTestSplit function to create array datastore objects trainingData. This process is based on the trainingRatio parameter. A training dataset (trainingData), which comprises 80% of the whole dataset, updates the weights of the CNN to fit the general characteristics of the data. This process is based on the training labels contained in the array datastore object. A validation dataset (validationData), which comprises 10% of the whole dataset, ensures that the model is learning the general behavior of the system rather than memorizing the patterns in it. Finally, a test dataset, which also comprises 10% of the whole dataset, evaluates the performance of the trained CNN using unseen samples of the data. If you are using an SDR, the size of validationData becomes 20% of the whole dataset, because the testing is based on the human presence status inferences of the trained CNN.

[trainingData,validationData,testData,imgInputSize,numClasses] = trainValTestSplit(useSDR,trainingRatio,dataNoPresence,labelNoPresence,dataPresence,labelPresence);

CSI image input size: [51   7]
Number of training images: 400

The example uses CSI periodogram images with size (numSubcarriers-1)-by-(rxsim.NumPacketsPerCapture-1) as the training input. Use the Create Simple Deep Learning Neural Network for Classification (Deep Learning Toolbox) example to learn more about the individual layers and their roles in the architecture.

cnn = [
    imageInputLayer(imgInputSize,'Normalization','none')
    convolution2dLayer(3,8,'Padding','same')
    batchNormalizationLayer
    reluLayer

    maxPooling2dLayer(2,'Stride',2)
    
    convolution2dLayer(3,16,'Padding','same')
    batchNormalizationLayer
    reluLayer

    maxPooling2dLayer(2,'Stride',2)

    convolution2dLayer(3,32,'Padding','same')
    batchNormalizationLayer
    reluLayer
    
    dropoutLayer(0.1)
    fullyConnectedLayer(numClasses)
    softmaxLayer
    classificationLayer];

cnn = layerGraph(cnn);
disp(cnn.Layers)

  16x1 Layer array with layers:

     1   'imageinput'    Image Input             51x7x1 images
     2   'conv_1'        2-D Convolution         8 3x3 convolutions with stride [1  1] and padding 'same'
     3   'batchnorm_1'   Batch Normalization     Batch normalization
     4   'relu_1'        ReLU                    ReLU
     5   'maxpool_1'     2-D Max Pooling         2x2 max pooling with stride [2  2] and padding [0  0  0  0]
     6   'conv_2'        2-D Convolution         16 3x3 convolutions with stride [1  1] and padding 'same'
     7   'batchnorm_2'   Batch Normalization     Batch normalization
     8   'relu_2'        ReLU                    ReLU
     9   'maxpool_2'     2-D Max Pooling         2x2 max pooling with stride [2  2] and padding [0  0  0  0]
    10   'conv_3'        2-D Convolution         32 3x3 convolutions with stride [1  1] and padding 'same'
    11   'batchnorm_3'   Batch Normalization     Batch normalization
    12   'relu_3'        ReLU                    ReLU
    13   'dropout'       Dropout                 10% dropout
    14   'fc'            Fully Connected         2 fully connected layer
    15   'softmax'       Softmax                 softmax
    16   'classoutput'   Classification Output   crossentropyex

% Definition of the training options
options = trainingOptions('adam', ...
    LearnRateSchedule='piecewise', ...
    InitialLearnRate=0.001, ... % learning rate
    MaxEpochs=numEpochs, ...
    MiniBatchSize=sizeMiniBatch, ...
    ValidationData=validationData, ...
    Shuffle='every-epoch', ...
    ExecutionEnvironment='auto',...
    Verbose=true);

Train the deep network architecture cnn using the training dataset trainingData and the training options defined by options.

% Train the network
trainedCNN = trainNetwork(trainingData,cnn,options);

Training on single CPU.
|======================================================================================================================|
|  Epoch  |  Iteration  |  Time Elapsed  |  Mini-batch  |  Validation  |  Mini-batch  |  Validation  |  Base Learning  |
|         |             |   (hh:mm:ss)   |   Accuracy   |   Accuracy   |     Loss     |     Loss     |      Rate       |
|======================================================================================================================|
|       1 |           1 |       00:00:00 |       37.50% |       84.00% |       0.8120 |       0.6155 |          0.0010 |
|       1 |          50 |       00:00:01 |      100.00% |      100.00% |       0.0085 |       0.0150 |          0.0010 |
|       2 |         100 |       00:00:02 |      100.00% |      100.00% |       0.0059 |       0.0055 |          0.0010 |
|       3 |         150 |       00:00:04 |      100.00% |      100.00% |       0.0134 |       0.0179 |          0.0010 |
|       4 |         200 |       00:00:05 |      100.00% |      100.00% |       0.0029 |       0.0024 |          0.0010 |
|       5 |         250 |       00:00:06 |      100.00% |      100.00% |       0.0123 |       0.0047 |          0.0010 |
|       6 |         300 |       00:00:08 |      100.00% |      100.00% |       0.0012 |       0.0007 |          0.0010 |
|       7 |         350 |       00:00:10 |      100.00% |      100.00% |       0.0008 |       0.0005 |          0.0010 |
|       8 |         400 |       00:00:13 |      100.00% |      100.00% |       0.0006 |       0.0192 |          0.0010 |
|       9 |         450 |       00:00:14 |      100.00% |      100.00% |       0.0001 |       0.0006 |          0.0010 |
|      10 |         500 |       00:00:15 |      100.00% |      100.00% |       0.0002 |       0.0063 |          0.0010 |
|======================================================================================================================|
Training finished: Max epochs completed.

Step 4: Evaluate Performance

In this section, you evaluate the performance of the neural network using training data or live inference with an SDR.

To run the evaluation, click Evaluate Performance.

 

if useSDR % Predict movement from the live SDR captures
    sensingResults = livePresenceDetection(rxsim,trainedCNN,20);
else % Predict movement by using the test set
    sensingResults = testPresenceDetection(testData,trainedCNN);
end

References

[1] IEEE P802.11 - TASK GROUP BF (WLAN SENSING), https://www.ieee802.org/11/Reports/tgbf_update.htm.

[2] IEEE 802.11-21/0407r2 - Multi-Band WiFi Fusion for WLAN Sensing, March 2021, https://mentor.ieee.org/802.11/dcn/21/11-21-0407-00-00bf-multi-band-wifi-fusion-for-wlan-sensing.pptx.

[3] Y. Lecun, L. Bottou, Y. Bengio and P. Haffner, "Gradient-based learning applied to document recognition," in Proceedings of the IEEE, Vol. 86, No. 11, pp. 2278–2324, Nov. 1998, doi: 10.1109/5.726791.