skyramp generate fuzz rest https://demoshop.skyramp.dev/api/v1/products \
--language python \
--framework pytest \
--api-schema

This command generates one file:

• products_fuzz_test.py.

Explanation of Command

• https://demoshop.skyramp.dev/api/v1/products: Defines the URL to the endpoint we aim to test.

• NOTE: No API method is specified in this generation command. When paired with an OpenAPI spec, Skyramp will generate tests for all endpoints at the URL, its direct parent (if any), and its direct children.

• --language: Specifies the test output language. For fuzz testing, we currently support Python and TypeScript.

• --framework: Specify the test execution framework of choice.

• --api-schema: Points to the OpenAPI schema used to generate the test. We also support sample data as an input for fuzz test generation.

Adjustments

Below are a few flags to customize the test generation. Additional flags are explained here.

• --auth-header: This flag allows you to specify the key of your authentication header, e.g. --auth-header X-API-KEY. By default, we assume Bearer.

• --response-status-code: Specify the expected status code. For fuzz tests, we default to 40x.

• --path-params: This flag allows you to override path parameters from your endpoint URL or the pre-defined values in the API schema, e.g. --path-params id=3fa85f64-5717-4562-b3fc-2c963f66afa6

• --output: Specify the name of the generated test file.

• --output-dir: Specify the directory to store the generated test file in.

Execute the Fuzz Test

You can execute the generated tests without any additional adjustments to the code. However, based on the application you want to test, you can pass your authentication token to Skyramp Tests via an environment variable. In a later section, we will elaborate on how to make changes to the code, if needed.

Set environment variable for authentication (if applicable)

Skyramp’s sample application doesn't require any authentication.

To test against an application that does require authentication, pass your token using an environment variable. By default, Skyramp expects a Bearer Token but we support additional authentication methods as outlined in the Examples page.

export SKYRAMP_TEST_TOKEN=$your_auth_token

Run the test

Run the tests using Pytest. If you don’t have Pytest, install it with pip by running the following command in your terminal:

# Prerequisites 
pip3 install pytest

# Execution of fuzz test for products/POST 
python3 -m pytest products_fuzz_test.py --tb

Review Test Results

Pytest’s default test output will print a line for each test that is being run and listing all failures at the end.

We ran the above Pytest command using a shorter test output, which prints a line for each test that is run but prints only the failed assertions at the end. You can adjust Pytest’s output behavior following this documentation.

Test failure

We can see that the test checks 5 methods that one out of five test cases pass.

Successful test

To fix the test, we need to update the expected status codes for each request body tested.

Here is the fix for each test:

• test_products_post

  • Problem: Python evaluates True fuzzed value to true

  • Solution: Update the fuzzed expected response code body for in_stock field (line 132) from 40x to 20x

    expected_products_post_status_code = {
            "category": "40x",
            "description": "40x",
            "image_url": "40x",
            "in_stock": "20x",
            "name": "40x",
            "price": "40x"
    
    

  • test_products_product_id_get, test_products_product_id_put, test_products_product_id_delete

    • Problem: Product 0 does not exist in Demo Shop

    • Solution: Change the expected_code of the original non-fuzzed request (lines 194, 254, 304) to 40x

      products_product_id_PUT_response = client.send_request(
              url=URL,
              path="/api/v1/products/{product_id}",
              method="PUT",
              body=products_product_id_PUT_request_body,
              headers=headers,
              path_params={"product_id": product_id},
              expected_code="40x"
      
      

Adding these fixes will cause the fuzz test to pass.

Next Steps

Congratulations! You have successfully generated a full suite of fuzz tests for a REST endpoint. You can adjust the expected status code and fuzzing strategies directly in the output file (see Test File Anatomy link for an example with a single method).

Related topics

• Test File Anatomy

• Skyramp Demo Shop

• Flags Glossary