## Data Transformation and Frequency Conversion

The data transformation and the frequency conversion functions convert a data series into a different format.

Data Transformation Functions

Function

Purpose

`boxcox`

Box-Cox transformation

`diff`

Differencing

`fillts`

Fill missing values

`filter`

Filter

`lagts`

Lag time series object

`leadts`

`peravg`

Periodic average

`smoothts`

Smooth data

`tsmovavg`

Moving average

Frequency Conversion Functions

As an example look at `boxcox`, the Box-Cox transformation function. This function transforms the data series contained in a financial time series object into another set of data series with relatively normal distributions.

First create a financial time series object from the supplied `whirlpool.dat` data file.

```whrl = ascii2fts('whirlpool.dat', 1, 2, []) ```
```Warning: FINTS will be removed in a future release. Use TIMETABLE instead. > In ascii2fts (line 64) Warning: FINTS will be removed in a future release. Use TIMETABLE instead. > In fints/display (line 66) whrl = desc: Whirlpool (WHR) freq: Unknown (0) 'dates: (1108)' 'Open: (1108)' 'High: (1108)' 'Low: (1108)' 'Close: (1108)' 'Volume: (1108)' '03-Jan-1995' [ 50.25] [ 50.88] [ 50.13] [ 50.38] [ 159000.00] '04-Jan-1995' [ 50.00] [ 50.13] [ 49.63] [ 49.75] [ 365800.00] '05-Jan-1995' [ 50.00] [ 50.00] [ 49.25] [ 49.88] [ 410500.00] '06-Jan-1995' [ 50.00] [ 50.25] [ 49.88] [ 50.25] [ 192900.00] '09-Jan-1995' [ 49.88] [ 50.38] [ 49.63] [ 50.38] [ 180800.00] '10-Jan-1995' [ 51.25] [ 51.25] [ 50.25] [ 50.25] [ 157800.00] '11-Jan-1995' [ 50.38] [ 50.63] [ 49.75] [ 49.75] [ 266000.00] '12-Jan-1995' [ 49.88] [ 50.13] [ 49.25] [ 49.88] [ 70000.00] '13-Jan-1995' [ 50.00] [ 50.25] [ 49.75] [ 50.13] [ 128000.00] '16-Jan-1995' [ 50.25] [ 50.38] [ 50.00] [ 50.00] [ 228200.00] ...```

Fill any missing values denoted with `NaN`s in `whrl` with values calculated using the linear method:

```f_whrl = fillts(whrl) ```
```Warning: FINTS will be removed in a future release. Use TIMETABLE instead. > In fints/fillts (line 213) Warning: FINTS will be removed in a future release. Use TIMETABLE instead. > In fints/display (line 66) f_whrl = desc: Filled Whirlpool (WHR) freq: Unknown (0) 'dates: (1108)' 'Open: (1108)' 'High: (1108)' 'Low: (1108)' 'Close: (1108)' 'Volume: (1108)' '03-Jan-1995' [ 50.25] [ 50.88] [ 50.13] [ 50.38] [ 159000.00] '04-Jan-1995' [ 50.00] [ 50.13] [ 49.63] [ 49.75] [ 365800.00] '05-Jan-1995' [ 50.00] [ 50.00] [ 49.25] [ 49.88] [ 410500.00] '06-Jan-1995' [ 50.00] [ 50.25] [ 49.88] [ 50.25] [ 192900.00] '09-Jan-1995' [ 49.88] [ 50.38] [ 49.63] [ 50.38] [ 180800.00] '10-Jan-1995' [ 51.25] [ 51.25] [ 50.25] [ 50.25] [ 157800.00] '11-Jan-1995' [ 50.38] [ 50.63] [ 49.75] [ 49.75] [ 266000.00] '12-Jan-1995' [ 49.88] [ 50.13] [ 49.25] [ 49.88] [ 70000.00] '13-Jan-1995' [ 50.00] [ 50.25] [ 49.75] [ 50.13] [ 128000.00] '16-Jan-1995' [ 50.25] [ 50.38] [ 50.00] [ 50.00] [ 228200.00] '17-Jan-1995' [ 51.00] [ 51.00] [ 50.00] [ 50.63] [ 196200.00] '18-Jan-1995' [ 50.50] [ 51.00] [ 50.13] [ 50.13] [ 369700.00] '19-Jan-1995' [ 50.25] [ 50.50] [ 50.00] [ 50.50] [ 254300.00] '20-Jan-1995' [ 50.50] [ 51.25] [ 50.50] [ 50.50] [ 183400.00] ...```

Transform the nonnormally distributed filled data series `f_whrl` into a normally distributed one using Box-Cox transformation:

```bc_whrl = boxcox(f_whrl) ```
```Warning: FINTS will be removed in a future release. Use TIMETABLE instead. > In fints/boxcox (line 36) Warning: FINTS will be removed in a future release. Use TIMETABLE instead. > In fints/display (line 66) bc_whrl = desc: Box-Cox: Filled Whirlpool (WHR) freq: Unknown (0) 'dates: (1108)' 'Open: (1108)' 'High: (1108)' 'Low: (1108)' 'Close: (1108)' 'Volume: (1108)' '03-Jan-1995' [ 0.56] [ 0.51] [ 0.60] [ 0.60] [ 9.15] '04-Jan-1995' [ 0.56] [ 0.51] [ 0.60] [ 0.60] [ 9.62] '05-Jan-1995' [ 0.56] [ 0.51] [ 0.60] [ 0.60] [ 9.68] '06-Jan-1995' [ 0.56] [ 0.51] [ 0.60] [ 0.60] [ 9.26] '09-Jan-1995' [ 0.56] [ 0.51] [ 0.60] [ 0.60] [ 9.23] '10-Jan-1995' [ 0.56] [ 0.51] [ 0.60] [ 0.60] [ 9.15] '11-Jan-1995' [ 0.56] [ 0.51] [ 0.60] [ 0.60] [ 9.44] '12-Jan-1995' [ 0.56] [ 0.51] [ 0.60] [ 0.60] [ 8.68] '13-Jan-1995' [ 0.56] [ 0.51] [ 0.60] [ 0.60] [ 9.03] '16-Jan-1995' [ 0.56] [ 0.51] [ 0.60] [ 0.60] [ 9.36] '17-Jan-1995' [ 0.56] [ 0.51] [ 0.60] [ 0.60] [ 9.27] '18-Jan-1995' [ 0.56] [ 0.51] [ 0.60] [ 0.60] [ 9.62] '19-Jan-1995' [ 0.56] [ 0.51] [ 0.60] [ 0.60] [ 9.42]...```

Compare the result of the `Close` data series with a normal (Gaussian) probability distribution function and the nonnormally distributed `f_whrl`:

```subplot(2, 1, 1); hist(f_whrl.Close); grid; title('Nonnormally Distributed Data'); subplot(2, 1, 2); hist(bc_whrl.Close); grid; title('Box-Cox Transformed Data'); ```
```Warning: FINTS will be removed in a future release. Use TIMETABLE instead. > In fints/subsref (line 106) Warning: FINTS will be removed in a future release. Use TIMETABLE instead. > In fints/hist (line 37) Warning: FINTS will be removed in a future release. Use TIMETABLE instead. > In fints/subsref (line 106) Warning: FINTS will be removed in a future release. Use TIMETABLE instead. > In fints/hist (line 37) ```

Box-Cox Transformation

The bar chart on the top represents the probability distribution function of the filled data series, `f_whrl`, which is the original data series `whrl` with the missing values interpolated using the linear method. The distribution is skewed toward the left (not normally distributed). The bar chart on the bottom is less skewed to the left. If you plot a Gaussian probability distribution function (PDF) with similar mean and standard deviation, the distribution of the transformed data is very close to normal (Gaussian).

When you examine the contents of the resulting object `bc_whrl`, you find an identical object to the original object `whrl` but the contents are the transformed data series. If you have the Statistics and Machine Learning Toolbox™ software, you can generate a Gaussian PDF with mean and standard deviation equal to those of the transformed data series and plot it as an overlay to the second bar chart. In the next figure, you can see that it is an approximately normal distribution.

Overlay of Gaussian PDF

The next example uses the `smoothts` function to smooth a time series.

To begin, transform `ibm9599.dat`, a supplied data file, into a financial time series object:

```ibm = ascii2fts('ibm9599.dat', 1, 3, 2) ```
```Warning: FINTS will be removed in a future release. Use TIMETABLE instead. > In ascii2fts (line 64) Warning: FINTS will be removed in a future release. Use TIMETABLE instead. > In fints/display (line 66) ibm = desc: International Business Machines Corporation (IBM) freq: Unknown (0) 'dates: (1108)' 'OPEN: (1108)' 'HIGH: (1108)' 'LOW: (1108)' 'CLOSE: (1108)' 'VOLUME: (1108)' '03-Jan-1995' [ 36.75] [ 36.91] [ 36.66] [ 36.88] [ 1167900.00] '04-Jan-1995' [ 37.00] [ 37.28] [ 36.63] [ 37.16] [ 1994700.00] '05-Jan-1995' [ 37.13] [ 37.63] [ 36.88] [ 37.00] [ 2270900.00] '06-Jan-1995' [ 37.25] [ 37.88] [ 37.03] [ 37.53] [ 3040500.00] '09-Jan-1995' [ 37.50] [ 37.91] [ 37.41] [ 37.75] [ 1713000.00] '10-Jan-1995' [ 38.00] [ 38.63] [ 37.78] [ 38.28] [ 3420800.00] '11-Jan-1995' [ 38.53] [ 38.53] [ 37.63] [ 38.00] [ 2450300.00] '12-Jan-1995' [ 37.91] [ 38.16] [ 37.75] [ 38.00] [ 2275900.00] '13-Jan-1995' [ 38.25] [ 38.41] [ 37.88] [ 38.16] [ 2030300.00] '16-Jan-1995' [ 38.16] [ 39.00] [ 38.16] [ 38.75] [ 2716000.00] '17-Jan-1995' [ 38.63] [ 39.00] [ 38.53] [ 38.75] [ 1596300.00] '18-Jan-1995' [ 38.88] [ 38.91] [ 38.28] [ 38.63] [ 1842600.00] '19-Jan-1995' [ 38.53] [ 38.78] [ 38.16] [ 38.25] [ 1713900.00]...```

Fill the missing data for holidays with data interpolated using the `fillts` function and the `Spline` fill method:

```f_ibm = fillts(ibm, 'Spline') ```
```Warning: FINTS will be removed in a future release. Use TIMETABLE instead. > In fints/fillts (line 213) Warning: FINTS will be removed in a future release. Use TIMETABLE instead. > In fints/display (line 66) f_ibm = desc: Filled International Business Machines Corporation (IBM) freq: Unknown (0) 'dates: (1108)' 'OPEN: (1108)' 'HIGH: (1108)' 'LOW: (1108)' 'CLOSE: (1108)' 'VOLUME: (1108)' '03-Jan-1995' [ 36.75] [ 36.91] [ 36.66] [ 36.88] [ 1167900.00] '04-Jan-1995' [ 37.00] [ 37.28] [ 36.63] [ 37.16] [ 1994700.00] '05-Jan-1995' [ 37.13] [ 37.63] [ 36.88] [ 37.00] [ 2270900.00] '06-Jan-1995' [ 37.25] [ 37.88] [ 37.03] [ 37.53] [ 3040500.00] '09-Jan-1995' [ 37.50] [ 37.91] [ 37.41] [ 37.75] [ 1713000.00] '10-Jan-1995' [ 38.00] [ 38.63] [ 37.78] [ 38.28] [ 3420800.00] '11-Jan-1995' [ 38.53] [ 38.53] [ 37.63] [ 38.00] [ 2450300.00] '12-Jan-1995' [ 37.91] [ 38.16] [ 37.75] [ 38.00] [ 2275900.00] '13-Jan-1995' [ 38.25] [ 38.41] [ 37.88] [ 38.16] [ 2030300.00] '16-Jan-1995' [ 38.16] [ 39.00] [ 38.16] [ 38.75] [ 2716000.00] '17-Jan-1995' [ 38.63] [ 39.00] [ 38.53] [ 38.75] [ 1596300.00] '18-Jan-1995' [ 38.88] [ 38.91] [ 38.28] [ 38.63] [ 1842600.00] '19-Jan-1995' [ 38.53] [ 38.78] [ 38.16] [ 38.25] [ 1713900.00] '20-Jan-1995' [ 38.13] [ 38.25] [ 37.50] [ 37.66] [ 3333400.00]...```

Smooth the filled data series using the default Box (rectangular window) method:

```sm_ibm = smoothts(f_ibm) ```
```Warning: FINTS will be removed in a future release. Use TIMETABLE instead. > In fints/smoothts (line 44) Warning: FINTS will be removed in a future release. Use TIMETABLE instead. > In fints/display (line 66) sm_ibm = desc: Box-smoothed of Filled International Business Machines Corporation (IBM) freq: Unknown (0) 'dates: (1108)' 'OPEN: (1108)' 'HIGH: (1108)' 'LOW: (1108)' 'CLOSE: (1108)' 'VOLUME: (1108)' '03-Jan-1995' [ 22.18] [ 22.36] [ 22.03] [ 22.21] [ 1086700.00] '04-Jan-1995' [ 29.63] [ 29.94] [ 29.44] [ 29.71] [ 1694800.00] '05-Jan-1995' [ 37.13] [ 37.52] [ 36.92] [ 37.26] [ 2037400.00] '06-Jan-1995' [ 37.38] [ 37.86] [ 37.14] [ 37.54] [ 2487980.00] '09-Jan-1995' [ 37.68] [ 38.11] [ 37.34] [ 37.71] [ 2579100.00] '10-Jan-1995' [ 37.84] [ 38.22] [ 37.52] [ 37.91] [ 2580100.00] '11-Jan-1995' [ 38.04] [ 38.33] [ 37.69] [ 38.04] [ 2378060.00] '12-Jan-1995' [ 38.17] [ 38.54] [ 37.84] [ 38.24] [ 2578660.00] '13-Jan-1995' [ 38.29] [ 38.62] [ 37.99] [ 38.33] [ 2213760.00] '16-Jan-1995' [ 38.36] [ 38.69] [ 38.12] [ 38.46] [ 2092220.00] '17-Jan-1995' [ 38.49] [ 38.82] [ 38.20] [ 38.51] [ 1979820.00]...```

Now, plot the original and smoothed closing price series for IBM® stock:

```plot(f_ibm.CLOSE('11/01/97::02/28/98'), 'r') datetick('x', 'mmmyy') hold on plot(sm_ibm.CLOSE('11/01/97::02/28/98'), 'b') hold off datetick('x', 'mmmyy') legend('Filled', 'Smoothed') title('Filled IBM Close Price vs. Smoothed Series') ```
```Warning: FINTS will be removed in a future release. Use TIMETABLE instead. > In fints/subsref (line 106) Warning: FINTS will be removed in a future release. Use TIMETABLE instead. > In fints/plot (line 63) Warning: FINTS will be removed in a future release. Use TIMETABLE instead. > In fints/subsref (line 106) Warning: FINTS will be removed in a future release. Use TIMETABLE instead. > In fints/plot (line 63) ```

Smoothed Data Series

These examples give you an idea of what you can do with a financial time series object. This toolbox provides some MATLAB® functions that have been overloaded to work directly with these objects. The overloaded functions are those most commonly needed to work with time series data.