Generation
Fuzz Test
Fuzz Testing
This guide will walk you through generating fuzz tests with the Skyramp CLI. Throughout this guide, we are using Skyramp’s Demo Shop API as an example REST API. You can find all relevant information on the Demo Shop here.
If you haven’t already installed Skyramp, follow the instructions here.
Overview
Fuzz testing (or fuzzing) uncovers bugs and vulnerabilities by injecting random, invalid, or unexpected inputs into an application. It excels at revealing edge cases and security flaws that traditional testing often misses, ensuring software remains robust and secure even under unpredictable conditions.
Generate a fuzz test for REST APIs
This section explains how you can use Skyramp to generate a fuzz test for a specific method of a REST API. To reliably generate test cases, we require at least one of the following inputs:
OpenAPI schema (JSON or YAML file)
Sample request data (JSON blob or JSON file)
These instructions are for using an API schema as input. If you want more control over the generated body values, you can also do test generation from sample data.
Generate Fuzz Test for a Single Method
To create a fuzz test for a single method, specify the method you want to test against in the command. In this example, we are using the https://demoshop.skyramp.dev/api as the URL to our service. When testing your service, replace it with the URL to the endpoint you want to test.
You can find the used API specification here.
Fuzz Testing
This guide will walk you through generating fuzz tests with the Skyramp CLI. Throughout this guide, we are using Skyramp’s Demo Shop API as an example REST API. You can find all relevant information on the Demo Shop here.
If you haven’t already installed Skyramp, follow the instructions here.
Overview
Fuzz testing (or fuzzing) uncovers bugs and vulnerabilities by injecting random, invalid, or unexpected inputs into an application. It excels at revealing edge cases and security flaws that traditional testing often misses, ensuring software remains robust and secure even under unpredictable conditions.
Generate a fuzz test for REST APIs
This section explains how you can use Skyramp to generate a fuzz test for a specific method of a REST API. To reliably generate test cases, we require at least one of the following inputs:
OpenAPI schema (JSON or YAML file)
Sample request data (JSON blob or JSON file)
These instructions are for using an API schema as input. If you want more control over the generated body values, you can also do test generation from sample data.
Generate Fuzz Test for a Single Method
To create a fuzz test for a single method, specify the method you want to test against in the command. In this example, we are using the https://demoshop.skyramp.dev/api as the URL to our service. When testing your service, replace it with the URL to the endpoint you want to test.
You can find the used API specification here.
Fuzz Testing
This guide will walk you through generating fuzz tests with the Skyramp CLI. Throughout this guide, we are using Skyramp’s Demo Shop API as an example REST API. You can find all relevant information on the Demo Shop here.
If you haven’t already installed Skyramp, follow the instructions here.
Overview
Fuzz testing (or fuzzing) uncovers bugs and vulnerabilities by injecting random, invalid, or unexpected inputs into an application. It excels at revealing edge cases and security flaws that traditional testing often misses, ensuring software remains robust and secure even under unpredictable conditions.
Generate a fuzz test for REST APIs
This section explains how you can use Skyramp to generate a fuzz test for a specific method of a REST API. To reliably generate test cases, we require at least one of the following inputs:
OpenAPI schema (JSON or YAML file)
Sample request data (JSON blob or JSON file)
These instructions are for using an API schema as input. If you want more control over the generated body values, you can also do test generation from sample data.
Generate Fuzz Test for a Single Method
To create a fuzz test for a single method, specify the method you want to test against in the command. In this example, we are using the https://demoshop.skyramp.dev/api as the URL to our service. When testing your service, replace it with the URL to the endpoint you want to test.
You can find the used API specification here.
Python
skyramp generate fuzz rest https://demoshop.skyramp.dev/api/v1/products \
-X POST \
--language python \
--framework pytest \
--api-schema
This command generates one file:
products_POST_fuzz_test.py
The content of the generated test is explained here.
Explanation of Command
https://demoshop.skyramp.dev/api/v1/products: Defines the URL to the endpoint we aim to test.-X: Specifies the API method to test.--language: Specifies the test output language. For fuzz testing, we currently support Python and TypeScript.--framework: Specify the test execution framework of choice.--api-schema: Points to the OpenAPI schema used to generate the test. We also support sample data as an input for fuzz test generation.
Adjustments
Below are a few flags to customize the test generation. Additional flags are explained here.
--auth-header: This flag allows you to specify the key of your authentication header, e.g.--auth-header X-API-KEY. By default, we assumeBearer.--response-status-code: Specify the expected status code. For fuzz tests, we default to40x.--path-params: This flag allows you to override path parameters from your endpoint URL or the pre-defined values in the API schema, e.g.--path-params id=3fa85f64-5717-4562-b3fc-2c963f66afa6--output: Specify the name of the generated test file.--output-dir: Specify the directory to store the generated test file in.
Execute the Fuzz Test
You can execute the generated tests without any additional adjustments to the code. However, based on the application you want to test, you can pass your authentication token to Skyramp Tests via an environment variable.
Set environment variable for authentication (if applicable)
Skyramp’s sample application doesn't require any authentication.
Ensure proper authentication for test execution. To test against an application that does require authentication, pass your token using an environment variable. By default, Skyramp expects a Bearer Token but we support additional authentication methods. If your API does not require any authentication, you can skip this step and just run the test.
export SKYRAMP_TEST_TOKEN=$your_auth_tokenRun the test
Run the test using Pytest. If you don’t have Pytest, install it with pip by running the following command in your terminal:
# Prerequisites
pip3 install pytest
# Execution of fuzz test for products/POST
python3 -m
Review Test Results
Pytest’s default test output will print a line for each test that is being run and listing all failures at the end.
We ran the above Pytest command using a shorter test output, which prints a line for each test that is run but prints only the failed assertions at the end. You can adjust Pytest’s output behavior following this documentation.
Test failure
The execution of this specific fuzz test as generated will result in a failure, as we have not adjusted the status codes to reflect the expected behavior of the service.
Successful test
To fix the test, we need to update the expected status codes for each request body tested.
In general, fuzz tests loop through each field in the request body and test two variations of the field:
A variation of the request with that field set to some fuzzed value
A variation of the request with that field set to
None
Looking at products_POST_fuzz_test.py, the following fuzzed values are tested:
products_post_fuzzed_body = {
"category": "0123456789",
"description": "0123456789",
"image_url": "0123456789",
"in_stock": True,
"name": "0123456789",
"price": -10
}In Python, any non-null value can evaluate to the boolean true, so the result when the request is run with a fuzzed value is a 20x instead of a 40x.
To make this test pass, we can update line 48 from 40X to 20X so that the endpoint returns a success when a fuzzed value of in_stock is passed.
expected_products_post_status_code = {
"category": "40x",
"description": "40x",
"image_url": "40x",
"in_stock": "20x",
"name": "40x",
"price": "40x"
}Running the test with these changes yields the following output:
Next Steps
Congratulations, you have just generated your first fuzz test! To learn more about how to adjust the the test file, please go to the Test File Anatomy page.
Related topics
Python
skyramp generate fuzz rest https://demoshop.skyramp.dev/api/v1/products \
-X POST \
--language python \
--framework pytest \
--api-schema
This command generates one file:
products_POST_fuzz_test.py
The content of the generated test is explained here.
Explanation of Command
https://demoshop.skyramp.dev/api/v1/products: Defines the URL to the endpoint we aim to test.-X: Specifies the API method to test.--language: Specifies the test output language. For fuzz testing, we currently support Python and TypeScript.--framework: Specify the test execution framework of choice.--api-schema: Points to the OpenAPI schema used to generate the test. We also support sample data as an input for fuzz test generation.
Adjustments
Below are a few flags to customize the test generation. Additional flags are explained here.
--auth-header: This flag allows you to specify the key of your authentication header, e.g.--auth-header X-API-KEY. By default, we assumeBearer.--response-status-code: Specify the expected status code. For fuzz tests, we default to40x.--path-params: This flag allows you to override path parameters from your endpoint URL or the pre-defined values in the API schema, e.g.--path-params id=3fa85f64-5717-4562-b3fc-2c963f66afa6--output: Specify the name of the generated test file.--output-dir: Specify the directory to store the generated test file in.
Execute the Fuzz Test
You can execute the generated tests without any additional adjustments to the code. However, based on the application you want to test, you can pass your authentication token to Skyramp Tests via an environment variable.
Set environment variable for authentication (if applicable)
Skyramp’s sample application doesn't require any authentication.
Ensure proper authentication for test execution. To test against an application that does require authentication, pass your token using an environment variable. By default, Skyramp expects a Bearer Token but we support additional authentication methods. If your API does not require any authentication, you can skip this step and just run the test.
export SKYRAMP_TEST_TOKEN=$your_auth_tokenRun the test
Run the test using Pytest. If you don’t have Pytest, install it with pip by running the following command in your terminal:
# Prerequisites
pip3 install pytest
# Execution of fuzz test for products/POST
python3 -m
Review Test Results
Pytest’s default test output will print a line for each test that is being run and listing all failures at the end.
We ran the above Pytest command using a shorter test output, which prints a line for each test that is run but prints only the failed assertions at the end. You can adjust Pytest’s output behavior following this documentation.
Test failure
The execution of this specific fuzz test as generated will result in a failure, as we have not adjusted the status codes to reflect the expected behavior of the service.
Successful test
To fix the test, we need to update the expected status codes for each request body tested.
In general, fuzz tests loop through each field in the request body and test two variations of the field:
A variation of the request with that field set to some fuzzed value
A variation of the request with that field set to
None
Looking at products_POST_fuzz_test.py, the following fuzzed values are tested:
products_post_fuzzed_body = {
"category": "0123456789",
"description": "0123456789",
"image_url": "0123456789",
"in_stock": True,
"name": "0123456789",
"price": -10
}In Python, any non-null value can evaluate to the boolean true, so the result when the request is run with a fuzzed value is a 20x instead of a 40x.
To make this test pass, we can update line 48 from 40X to 20X so that the endpoint returns a success when a fuzzed value of in_stock is passed.
expected_products_post_status_code = {
"category": "40x",
"description": "40x",
"image_url": "40x",
"in_stock": "20x",
"name": "40x",
"price": "40x"
}Running the test with these changes yields the following output:
Next Steps
Congratulations, you have just generated your first fuzz test! To learn more about how to adjust the the test file, please go to the Test File Anatomy page.
Related topics
Python
skyramp generate fuzz rest https://demoshop.skyramp.dev/api/v1/products \
-X POST \
--language python \
--framework pytest \
--api-schema
This command generates one file:
products_POST_fuzz_test.py
The content of the generated test is explained here.
Explanation of Command
https://demoshop.skyramp.dev/api/v1/products: Defines the URL to the endpoint we aim to test.-X: Specifies the API method to test.--language: Specifies the test output language. For fuzz testing, we currently support Python and TypeScript.--framework: Specify the test execution framework of choice.--api-schema: Points to the OpenAPI schema used to generate the test. We also support sample data as an input for fuzz test generation.
Adjustments
Below are a few flags to customize the test generation. Additional flags are explained here.
--auth-header: This flag allows you to specify the key of your authentication header, e.g.--auth-header X-API-KEY. By default, we assumeBearer.--response-status-code: Specify the expected status code. For fuzz tests, we default to40x.--path-params: This flag allows you to override path parameters from your endpoint URL or the pre-defined values in the API schema, e.g.--path-params id=3fa85f64-5717-4562-b3fc-2c963f66afa6--output: Specify the name of the generated test file.--output-dir: Specify the directory to store the generated test file in.
Execute the Fuzz Test
You can execute the generated tests without any additional adjustments to the code. However, based on the application you want to test, you can pass your authentication token to Skyramp Tests via an environment variable.
Set environment variable for authentication (if applicable)
Skyramp’s sample application doesn't require any authentication.
Ensure proper authentication for test execution. To test against an application that does require authentication, pass your token using an environment variable. By default, Skyramp expects a Bearer Token but we support additional authentication methods. If your API does not require any authentication, you can skip this step and just run the test.
export SKYRAMP_TEST_TOKEN=$your_auth_tokenRun the test
Run the test using Pytest. If you don’t have Pytest, install it with pip by running the following command in your terminal:
# Prerequisites
pip3 install pytest
# Execution of fuzz test for products/POST
python3 -m
Review Test Results
Pytest’s default test output will print a line for each test that is being run and listing all failures at the end.
We ran the above Pytest command using a shorter test output, which prints a line for each test that is run but prints only the failed assertions at the end. You can adjust Pytest’s output behavior following this documentation.
Test failure
The execution of this specific fuzz test as generated will result in a failure, as we have not adjusted the status codes to reflect the expected behavior of the service.
Successful test
To fix the test, we need to update the expected status codes for each request body tested.
In general, fuzz tests loop through each field in the request body and test two variations of the field:
A variation of the request with that field set to some fuzzed value
A variation of the request with that field set to
None
Looking at products_POST_fuzz_test.py, the following fuzzed values are tested:
products_post_fuzzed_body = {
"category": "0123456789",
"description": "0123456789",
"image_url": "0123456789",
"in_stock": True,
"name": "0123456789",
"price": -10
}In Python, any non-null value can evaluate to the boolean true, so the result when the request is run with a fuzzed value is a 20x instead of a 40x.
To make this test pass, we can update line 48 from 40X to 20X so that the endpoint returns a success when a fuzzed value of in_stock is passed.
expected_products_post_status_code = {
"category": "40x",
"description": "40x",
"image_url": "40x",
"in_stock": "20x",
"name": "40x",
"price": "40x"
}Running the test with these changes yields the following output:
Next Steps
Congratulations, you have just generated your first fuzz test! To learn more about how to adjust the the test file, please go to the Test File Anatomy page.