Incompatible padding for RSA algorithm operation

Cryptography operation is not supported by the padding type set in context

Description

This defect occurs when you perform an RSA algorithm operation on a context object that is not compatible with the padding previously associated with the object.

For instance, you associate the OAEP padding scheme with a context object but later use the context for signature verification, an operation that the padding scheme does not support.

ret = EVP_PKEY_CTX_set_rsa_padding(ctx, RSA_PKCS1_OAEP_PADDING);
...
ret = EVP_PKEY_verify(ctx, out, out_len, in, in_len);

Risk

Padding schemes remove determinism from the RSA algorithm and protect RSA operations from certain kinds of attack.

When you use an incorrect padding scheme, the RSA operation can fail or result in unexpected ciphertext.

Fix

Before performing an RSA operation, associate the context object with a padding scheme that is compatible with the operation.

Encryption: Use the OAEP padding scheme.
For instance, use the EVP_PKEY_CTX_set_rsa_padding function with the argument RSA_PKCS1_OAEP_PADDING or the RSA_padding_add_PKCS1_OAEP function.
```
ret = EVP_PKEY_CTX_set_rsa_padding(ctx, RSA_PKCS1_OAEP_PADDING);
```
You can also use the PKCS#1v1.5 or SSLv23 schemes. Be aware that these schemes are considered insecure.
You can then use functions such as EVP_PKEY_encrypt / EVP_PKEY_decrypt or RSA_public_encrypt / RSA_private_decrypt on the context.
Signature: Use the RSA-PSS padding scheme.
For instance, use the EVP_PKEY_CTX_set_rsa_padding function with the argument RSA_PKCS1_PSS_PADDING.
```
ret = EVP_PKEY_CTX_set_rsa_padding(ctx, RSA_PKCS1_PSS_PADDING);
```
You can also use the ANSI X9.31, PKCS#1v1.5, or SSLv23 schemes. Be aware that these schemes are considered insecure.
You can then use functions such as the EVP_PKEY_sign-EVP_PKEY_verify pair or the RSA_private_encrypt-RSA_public_decrypt pair on the context.

If you perform two kinds of operation with the same context, after the first operation, reset the padding scheme in the context before the second operation.

Examples

expand all

OAEP Padding for Signature Operation

#include <stddef.h>
#include <openssl/rsa.h>

#define fatal_error() exit(-1)

int ret;
unsigned char *out_buf;

int func(unsigned char *src, size_t len, RSA* rsa){
  if (rsa == NULL) fatal_error();
  return RSA_private_encrypt(len, src, out_buf, rsa, RSA_PKCS1_OAEP_PADDING);
}

In this example, the function RSA_private_encrypt performs a signature operation by using the OAEP padding scheme, which supports encryption operations only.

Correction — Use Padding Scheme That Supports Signature

One possible correction is to use the RSA-PSS padding scheme. The corrected example uses the function RSA_padding_add_PKCS1_PSS to associate the padding scheme with the context.

#include <stddef.h>
#include <openssl/evp.h>
#include <openssl/rsa.h>

#define fatal_error() exit(-1)

int ret;
unsigned char *msg_pad;
unsigned char *out_buf;

int func(unsigned char *src, size_t len, RSA* rsa){
  if (rsa == NULL) fatal_error(); 

  ret = RSA_padding_add_PKCS1_PSS(rsa, msg_pad, src, EVP_sha256(), -2); 
  if (ret <= 0) fatal_error();

  return RSA_private_encrypt(len, msg_pad, out_buf, rsa, RSA_NO_PADDING); 
}

Result Information

Group: Cryptography

Language: C | C++

Default: Off

Command-Line Syntax: CRYPTO_RSA_BAD_PADDING

Impact: Medium

Version History

Introduced in R2018a