Main Content

本页对应的英文页面已更新,但尚未翻译。 若要查看最新内容,请点击此处访问英文页面。

MapReduce

用于分析无法载入内存的数据集的编程方法

mapreduce 是一种编程方法,适用于分析无法以其他方法载入计算机内存的大型数据集。该方法使用 datastore,以小分块的方式处理数据,包含映射阶段(用于对数据进行格式化或执行前兆计算)和化简阶段(用于对来自映射阶段的所有结果进行聚合)。有关详细信息,请参阅MapReduce 快速入门

有关将其他产品与 mapreduce 结合使用的信息,请参阅Speed Up and Deploy MapReduce Using Other Products

函数

全部展开

mapreduce用于分析无法载入内存的数据集的编程方法
datastore为大型数据集合创建数据存储
add向 KeyValueStore 中添加单个键-值对组
addmulti向 KeyValueStore 中添加多个键-值对组
hasnext确定 ValueIterator 是否具有一个或多个可用值
getnext从 ValueIterator 获取下一个值
mapreducer为 mapreduce 或 tall 数组定义执行环境
gcmr获取当前的 mapreducer 配置

对象

KeyValueStore存储用于 mapreduce 的键-值对组
ValueIterator用于 mapreduce 的中间值迭代器

主题

MapReduce 快速入门

了解 MapReduce 编程方法并运行示例计算。

编写 map 函数

创建在 mapreduce 算法中使用的 map 函数。

Write a Reduce Function

Create a reduce function for use in a mapreduce algorithm.

Build Effective Algorithms with MapReduce

Summary of mapreduce example files.

Speed Up and Deploy MapReduce Using Other Products

Capabilities of other products to speed up and share mapreduce algorithms.

Find Maximum Value with MapReduce

This example shows how to find the maximum value of a single variable in a data set using mapreduce.

Compute Mean Value with MapReduce

This example shows how to compute the mean of a single variable in a data set using mapreduce.

Create Histograms Using MapReduce

This example shows how to visualize patterns in a large data set without having to load all of the observations into memory simultaneously.

Compute Mean by Group Using MapReduce

This example shows how to compute the mean by group in a data set using mapreduce.

Simple Data Subsetting Using MapReduce

This example shows how to extract a subset of a large data set.

Using MapReduce to Compute Covariance and Related Quantities

This example shows how to compute the mean and covariance for several variables in a large data set using mapreduce.

Compute Summary Statistics by Group Using MapReduce

This example shows how to compute summary statistics organized by group using mapreduce.

Using MapReduce to Fit a Logistic Regression Model

This example shows how to use mapreduce to carry out simple logistic regression using a single predictor.

Tall Skinny QR (TSQR) Matrix Factorization Using MapReduce

This example shows how to compute a tall skinny QR (TSQR) factorization using mapreduce.

Compute Maximum Average HSV of Images with MapReduce

This example shows how to use ImageDatastore and mapreduce to find images with maximum hue, saturation and brightness values in an image collection.

疑难解答

Debug MapReduce Algorithms

This example shows how to debug your mapreduce algorithms in MATLAB® using a simple example file, MaxMapReduceExample.m. Debugging enables you to follow the movement of data between the different phases of mapreduce execution and inspect the state of all intermediate variables.

特色示例