MapReduce

用于分析无法载入内存的数据集的编程方法

mapreduce 是一种编程方法，适用于分析无法以其他方法载入计算机内存的大型数据集。该方法使用 datastore，以小分块的方式处理数据，包含映射阶段（用于对数据进行格式化或执行前兆计算）和化简阶段（用于对来自映射阶段的所有结果进行聚合）。有关详细信息，请参阅MapReduce 快速入门。

有关将其他产品与 mapreduce 结合使用的信息，请参阅Speed Up and Deploy MapReduce Using Other Products。

函数

全部展开

常规

`mapreduce`	用于分析无法载入内存的数据集的编程方法
`datastore`	为大型数据集合创建数据存储

辅助对象方法

`add`	向 KeyValueStore 中添加单个键-值对组
`addmulti`	向 KeyValueStore 中添加多个键-值对组
`hasnext`	确定 ValueIterator 是否具有一个或多个可用值
`getnext`	从 ValueIterator 获取下一个值

执行环境

`mapreducer`	为 mapreduce 或 tall 数组定义执行环境
`gcmr`	获取当前的 mapreducer 配置

对象

`KeyValueStore`	存储用于 mapreduce 的键-值对组
`ValueIterator`	用于 mapreduce 的中间值迭代器

主题

MapReduce 快速入门
了解 MapReduce 编程方法并运行示例计算。
编写 map 函数
创建在 mapreduce 算法中使用的 map 函数。
Write a Reduce Function
Create a reduce function for use in a mapreduce algorithm.
Build Effective Algorithms with MapReduce
Summary of mapreduce example files.
Speed Up and Deploy MapReduce Using Other Products
Capabilities of other products to speed up and share mapreduce algorithms.
Find Maximum Value with MapReduce
This example shows how to find the maximum value of a single variable in a data set using mapreduce.
Compute Mean Value with MapReduce
This example shows how to compute the mean of a single variable in a data set using mapreduce.
Create Histograms Using MapReduce
This example shows how to visualize patterns in a large data set without having to load all of the observations into memory simultaneously.
Compute Mean by Group Using MapReduce
This example shows how to compute the mean by group in a data set using mapreduce.
Simple Data Subsetting Using MapReduce
This example shows how to extract a subset of a large data set.
Using MapReduce to Compute Covariance and Related Quantities
This example shows how to compute the mean and covariance for several variables in a large data set using mapreduce.
Compute Summary Statistics by Group Using MapReduce
This example shows how to compute summary statistics organized by group using mapreduce.
Using MapReduce to Fit a Logistic Regression Model
This example shows how to use mapreduce to carry out simple logistic regression using a single predictor.
Tall Skinny QR (TSQR) Matrix Factorization Using MapReduce
This example shows how to compute a tall skinny QR (TSQR) factorization using mapreduce.
Compute Maximum Average HSV of Images with MapReduce
This example shows how to use ImageDatastore and mapreduce to find images with maximum hue, saturation and brightness values in an image collection.

疑难解答

Debug MapReduce Algorithms

This example shows how to debug your mapreduce algorithms in MATLAB^® using a simple example file, MaxMapReduceExample.m. Debugging enables you to follow the movement of data between the different phases of mapreduce execution and inspect the state of all intermediate variables.

精选示例

Analyze Big Data in MATLAB Using MapReduce

Use the mapreduce function to process a large amount of file-based data. The MapReduce algorithm is a mainstay of many modern "big data" applications. This example operates on a single computer, but the code can scale up to use Hadoop®.

打开实时脚本