E2E Testing

Introduction

This guide will show you how to generate E2E tests for complex scenarios with Skyramp using a combination of Skyramp and Playwright traces. We'll demonstrate using Skyramp’s Demo Shop API, a simple e-commerce API for product and order management. Learn more about the Demo Shop API.

Prerequisites

• Skyramp CLI installed

• Relevant libraries installed (ie Python, TypeScript libraries)

• Docker installed and running (for trace collection only)

Refer to the Installation Guide if you haven't installed Skyramp yet.

Overview

An end-to-end (E2E) test validates an application's entire workflow from start to finish, simulating real-world user scenarios. It ensures that all integrated components, including front-end, back-end, and external systems, work together seamlessly to deliver the expected functionality and performance. 

To generate E2E tests that span across multiple endpoints or services, Skyramp requires a trace to generate the test case reliably.

What is a trace?

A trace is a log of a set of actions executed in a particular window or scenario in your system. Traces can be used to simulate real-world scenarios of product usage, and those traces can in turn be used to generate tests that give you confidence that said scenario will be resilient in production traffic.

Skyramp can function as a “Man-in-the-Middle” proxy that monitors your traffic and allows you to simply record the scenario you want to test. Skyramp can collect traffic from 3 different sources:

• HTTP traffic from a browser UI

• Backend traffic through the CLI

• UI traces through the Playwright recorder

By following these simple steps, you'll be able to collect a trace that you can directly supply for the Skyramp test generation.

Start Trace Collection

Want to skip straight to test generation? Download the following Skyramp trace and Playwright trace and skip to “Generate E2E Test from Trace”

To start collecting traces, you need to run the following command. This command will spawn a new shell configured to collect all relevant backend network traffic as well as a Playwright Browser and Inspector. All visited web links traffic will also be part of the collected trace. The first time you run this command, it might take a while to bring up Skyramp.

skyramp generate trace \
--playwright \
--include "https://demoshop.skyramp.dev/api/*" \
--runtime docker \
--output e2e_test_trace.json \
--playwright-output

Explanation of Command

• --playwright: Enable Playwright to capture UI activities (default: false)

• --include: Specify a comma-separated list of fully qualified domain names (FQDNs) for Skyramp to collect traces for.

• --runtime: Select the environment in which the Skyramp trace collector gets deployed. [Docker, Kubernetes]

• --output: Specify the name of the generated trace file.

Additional Flags:

• --docker-network: Specify the name (string) of your Docker network.

• --docker-skyramp-port: Specify a port number for Skyramp (default=35142).

A Note on Fully Qualified Domain Names (FQDNs)

FQDNs need to be specified either as an exact path to the endpoint or in combination with a wildcard:

• demoshop.skyramp.dev/api/v1/products will only filter for POST and GET of the products endpoint.

• demoshop.skyramp.dev/products* will filter for all products endpoints + any nested endpoints.

• demoshop.skyramp.dev/api/v1/products/* will filter only for any nested endpoints that require a product ID. It will not collect the traces of POST and GET products.

For endpoints that require path parameters, please replace the path parameter with the wildcard:

• Convert demoshop.skyramp.dev/products/{product_id}/reviews to demoshop.skyramp.dev/products/*/reviews

• --exclude: Specify a comma-separated list of FQDNs for which you do not want Skyramp to collect traces. This flag takes precedence over --include if a FQDN is specified in both.

• --docker-network: Specify the name (string) of your Docker network.

• --docker-skyramp-port: Specify a port number for Skyramp (default=35142).

• --playwright-output: Specify the name of the generated playwright trace file (default: skyramp_playwright.zip)

Record Scenario

We now leverage the freshly spawned shell and Playwright Browser to collect the traces for our test scenario. Here is a quick summary of the scenario:

1. Create new product (via cURL)

2. Find product in product list (via UI)

3. Create new order containing the new product (via cURL)

4. Find order in order list (via UI)

5. View newly created order (via UI)

6. Delete newly created order (UI)

Let's get started!

1. Create new product (via cURL)

• To start, let’s create a new product with the Demo Shop API

  curl -X POST https://demoshop.skyramp.dev/api/v1/products \
    -H "Content-Type: application/json" \
    -k \
    -d '{
      "name": "SkyPad Pro",
      "description": "High-performance tablet powered by Skyramp",
      "price": 1299.99,
      "image_url": "https://shorturl.at/sZeO8",
      "category": "Tablets",
      "in_stock": true
    }'

2. Find product in product list (via UI)

• In the newly spawned Chromium browser, go to https://skyramp.dev/demoshop/products. This will open the Demo Shop UI.

  

• Find your newly created product on the list

• Assert that the new product name and price appear correctly

  • Click on the 'ab' button

  • Click on the name of the newly created product

  • This will ask you to generate an assertion that the element contains the text “SkyPad Pro”. Click the checkmark.

  

  • Repeat for price: Click on the 'ab' button

  • Click on the price of the newly created product

  • This will ask you to generate an assertion that the element contains the text “$1299.99”. Click the checkmark.

  

3. Create new order containing the new product (via cURL)

• To switch to a backend call from a UI action, you will need to add a breakpoint. To do so, click on the 'Add marker' button.

  

• From the original POST products API call, copy the product_id field that is returned in the response

• We will use this product_id to create a new order. Replace <placeholder> in the below command with your product_id:

  curl -X POST https://demoshop.skyramp.dev/api/v1/orders \
    -H "Content-Type: application/json" \
    -k \
    -d '{
      "customer_email": "sahil@skyramp.dev",
      "items": [
        {
          "product_id": <placeholder>,
          "quantity": 2,
          "unit_price": 1299.99
        }
      ]
    }'

4. Find order in order list (via UI)

• From the Demo Shop UI in the Chromium browser, click on “Orders” at the top right of the page.

  

• Find your newly created order on the list

• Assert that the new order details and total price appear correctly

  • Click on the 'ab' button

  • Click on the name of the newly created order

  • This will ask you to generate an assertion that the element contains the text “sahil@skyramp.dev - 1 item”. Click the checkmark.

  

  • Repeat for total price: Click on the 'ab' button

  • Click on the total price of the newly created order

  • This will ask you to generate an assertion that the element contains the text “$2599.98”. Click the checkmark.

  

5. View newly created order (via UI)

• From your order in the Orders page, Click on “View Details”

• Assert that the customer name in the order is correct:

  • Click on the 'ab' button

  • Click on the customer name for the newly created order

  • This will ask you to generate an assertion that the element contains the text “sahil@skyramp.dev”. Click the checkmark.

  

• Assert that the product s are appearing:

  • Click on the button

  • Click on the product name for the order item

  • This will ask you to generate an assertion that the element contains the text “SkyPad Pro”. Click the checkmark.

  

6. Delete newly created order (via UI)

• From the order details page, click on “Delete Order”

• This should redirect you to the “Orders” page

• Assert that a deleted banner is visible

  • Click on the 'Assert visibility' button on the Playwright toolbar

  • Click on the red banner component

  • This will generate an assertion that the banner is visible.

    

End Trace Collection

Once the scenario is completed, you can end the trace collection in the spawned shell with Ctrl + D. You will see for which endpoints Skyramp was also able to collect backend traces - this might be a very long list as we didn’t apply filtering.

WARNING Closing playwright browsers. If they are not closed, please manually close them  cmd

Skyramp creates two trace files: e2e_test_trace.json and e2e_test_playwright.zip. The first file only contains the backend traces that were collected, while the zip file includes all collected traces from Playwright. Both traces will be used for the generation of the E2E test.

You can find sample traces for the test scenario here:

• Skyramp Trace

• Playwright Trace