Remove non-finite values while decimating

2023 10 31
2 个回答
回答已采纳
更新时间：2023 11 9
2 次查看（30 天）

0 个投票

I was able to decimate 8 other identical (or so I thought) files, but this one hit an error. Code and error below:
%% Set path
cd 'C:\folder'
%% Load data
data = readtable('data.csv');
%% Decimate and then convert to table format
data50_X = decimate(data{:,4},2);
Error using filtfilt
Expected input to be finite.
Error in filtfilt>efiltfilt (line 123)
validateattributes(x,{'double','single'},{'finite','nonempty'},'filtfilt');
Error in filtfilt (line 102)
y = efiltfilt(b,a,x);
Error in decimate (line 157)
odata = filtfilt(b,a,idata);

请先登录，再回答此问题。

 采纳的回答

Image Analyst
Image Analyst 2023-11-1

0 个投票

Try this:
data = readtable('data.csv');
goodRows = isfinite(data{:, 4});
data = data(goodRows, :); % Extract only the good/finite values rows.

9 个评论
显示 7更早的评论 隐藏 7更早的评论

Taylor Azizeh
Taylor Azizeh 2023-11-1
Thank you! This fixed it, but what if I needed to keep the finite values from columns 4, 5, and 6??
Image Analyst
Image Analyst 2023-11-2
If you want to throw out any rows in which a nan or infinite value occurs in either column 4, 5, or 6, you could try
data = readtable('data.csv');
goodRows = all(isfinite(data{:, 4:6}));
data = data(goodRows, :); % Extract only the good/finite values rows.
Taylor Azizeh
Taylor Azizeh 2023-11-8
This did something funny to the "data" table.
Image Analyst
Image Analyst 2023-11-9
@Taylor Azizeh, if you need more help, please attach your csv file and your code, or:
Taylor Azizeh
Taylor Azizeh 2023-11-9
编辑：Taylor Azizeh 2023-11-9
My CSV file is huge, so I can't attach it, but I will include a snippet below. But my code is as follows. Those last rows of code are when it seems to mess up the data variable.
%% First, set pathway
cd 'C:\path'
%% Load in your data (this may take a while, depdending on the size of the file)
data = readtable('data.csv');
beep
%% Decimate and then convert to table format
goodRows = all(isfinite(data{:, 4:6}));
data = data(goodRows, :); % Extract only the good/finite values rows
Data sample of what I'm trying to decimate (within the larger file, these are columns 4:6
data_XYZ = 124826901×3
-0.1250 -0.0630 1.0000
-0.1250 -0.0630 0.9390
-0.1230 -0.0630 1.0000
-0.0660 -0.0630 0.9970
-0.1250 -0.0630 0.9430
-0.1190 -0.0630 1.0000
-0.0700 -0.0630 0.9930
-0.1160 -0.0630 0.9470
-0.0630 -0.0630 1.0000
-0.0630 -0.0630 1.0000
Image Analyst
Image Analyst 2023-11-9
Just edit it so that it's less than 5 MB and attach it. I'll check for it tomorrow.
Taylor Azizeh
Taylor Azizeh 2023-11-9
编辑：Taylor Azizeh 2023-11-9
Ran in:
Ran in:
Here you go! It's only 1000 rows. But you can see here that despite the finite values in XYZ, the code only keeps three rows...
%% Load in your data (this may take a while, depdending on the size of the file)
data = readtable('small_data.csv');
beep
%% Decimate and then convert to table format
goodRows = all(isfinite(data{:, 4:6}));
data = data(goodRows, :) % Extract only the good/finite values rows
data = 3×18 table
TagID Date Time X Y Z Activity Depth Temp___C_ location_lat location_lon height_msl ground_speed satellites hdop signal_strength SensorRaw Metadata _______________________ ___________ ____________ ______ ______ _ ______________ _____ _________ ____________ ____________ __________ ____________ __________ ____ _______________ _________ ________ {'EP19_PEN19_EP9_B_S1'} 30-Oct-2019 19:49:07.000 -0.063 -0.063 1 {'Active/Dry'} 0 18.6 NaN NaN NaN NaN NaN NaN NaN 997 NaN {'EP19_PEN19_EP9_B_S1'} 30-Oct-2019 19:49:07.010 -0.063 -0.063 1 {0×0 char } NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN {'EP19_PEN19_EP9_B_S1'} 30-Oct-2019 19:49:07.020 -0.063 -0.063 1 {0×0 char } NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
Image Analyst
Image Analyst 2023-11-9
Ran in:
Ran in:
Sorry, you need another argument for all to make it apply across columns. Try this:
%% Load in your data (this may take a while, depdending on the size of the file)
data = readtable('small_data.csv');
%% Decimate and then convert to table format
goodRows = all(isfinite(data{:, 4:6}), 2);
data = data(goodRows, :) % Extract only the good/finite values rows
data = 1000×18 table
TagID Date Time X Y Z Activity Depth Temp___C_ location_lat location_lon height_msl ground_speed satellites hdop signal_strength SensorRaw Metadata _______________________ ___________ ____________ ______ ______ _ ______________ _____ _________ ____________ ____________ __________ ____________ __________ ____ _______________ _________ ________ {'EP19_PEN19_EP9_B_S1'} 30-Oct-2019 19:49:07.000 -0.063 -0.063 1 {'Active/Dry'} 0 18.6 NaN NaN NaN NaN NaN NaN NaN 997 NaN {'EP19_PEN19_EP9_B_S1'} 30-Oct-2019 19:49:07.010 -0.063 -0.063 1 {0×0 char } NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN {'EP19_PEN19_EP9_B_S1'} 30-Oct-2019 19:49:07.020 -0.063 -0.063 1 {0×0 char } NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN {'EP19_PEN19_EP9_B_S1'} 30-Oct-2019 19:49:07.030 -0.063 -0.063 1 {0×0 char } NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN {'EP19_PEN19_EP9_B_S1'} 30-Oct-2019 19:49:07.040 -0.003 -0.063 1 {0×0 char } NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN {'EP19_PEN19_EP9_B_S1'} 30-Oct-2019 19:49:07.050 -0.059 -0.063 1 {0×0 char } NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN {'EP19_PEN19_EP9_B_S1'} 30-Oct-2019 19:49:07.060 -0.004 -0.063 1 {0×0 char } NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN {'EP19_PEN19_EP9_B_S1'} 30-Oct-2019 19:49:07.070 0 -0.063 1 {0×0 char } NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN {'EP19_PEN19_EP9_B_S1'} 30-Oct-2019 19:49:07.080 -0.058 -0.063 1 {0×0 char } NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN {'EP19_PEN19_EP9_B_S1'} 30-Oct-2019 19:49:07.090 -0.063 -0.063 1 {0×0 char } NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN {'EP19_PEN19_EP9_B_S1'} 30-Oct-2019 19:49:07.100 -0.063 -0.063 1 {0×0 char } NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN {'EP19_PEN19_EP9_B_S1'} 30-Oct-2019 19:49:07.110 -0.063 -0.007 1 {0×0 char } NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN {'EP19_PEN19_EP9_B_S1'} 30-Oct-2019 19:49:07.120 -0.008 -0.055 1 {0×0 char } NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN {'EP19_PEN19_EP9_B_S1'} 30-Oct-2019 19:49:07.130 -0.054 -0.063 1 {0×0 char } NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN {'EP19_PEN19_EP9_B_S1'} 30-Oct-2019 19:49:07.140 -0.063 -0.063 1 {0×0 char } NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN {'EP19_PEN19_EP9_B_S1'} 30-Oct-2019 19:49:07.150 -0.063 -0.063 1 {0×0 char } NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
Taylor Azizeh
Taylor Azizeh 2023-11-9
Fixed!! Thank you so much for your help, you're a lifesaver.

请先登录，再进行评论。

更多回答（1 个）

Matt J
Matt J 2023-10-31
编辑：Matt J 2023-10-31

0 个投票

Likely, there are NaNs in your data that you have to remove.
find(~isfinite(data{:,4}))

5 个评论
显示 3更早的评论 隐藏 3更早的评论

Taylor Azizeh
Taylor Azizeh 2023-10-31
That creates a new variable (407 x 1) with an output like this:
81501
1028902
1032803
1075404
1080505
1081706
1081807
1084408
1713709
1761010
1762411
2260910
2260912
But after that, my decimation code still doesn't run.
Matt J
Matt J 2023-10-31
编辑：Matt J 2023-10-31
Presumably you haven't removed any of the bad values from your data.
Taylor Azizeh
Taylor Azizeh 2023-10-31
Can you explain how I would remove the bad values after running the
find(~isfinite(data{:,4}))
code? Thanks for your patience! I am still new to MATLAB.
Matt J
Matt J 2023-11-1
find(~isfinite(data{:,4})) gives the locations of the bad data. You can go there and see what those values are, and either remove them or replace them with something else.
Taylor Azizeh
Taylor Azizeh 2023-11-1
Ohhhh gotcha. Thank you so much!

请先登录，再进行评论。

类别

在 帮助中心 和 File Exchange 中查找有关 Numeric Types 的更多信息

产品

版本

R2023a

标签

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by