VRT Workflow: Smart Collapsible Sections For Image Bloat

by Alex Johnson 57 views

Introduction

In the realm of Visual Regression Testing (VRT), the process often involves posting numerous inline images within Pull Request (PR) comments. While this practice aims to provide a comprehensive view of changes, it can inadvertently lead to significant challenges. These challenges include massive page bloat, increased scroll distance, and browser performance issues. In some extreme cases, the sheer volume of images can overwhelm systems, causing performance degradation and making it difficult to identify crucial differences among the noise. This article delves into a proposed solution to mitigate these issues by implementing smart collapsible sections within the VRT workflow. Our primary goal is to maintain the integrity of testing while optimizing the user experience by reducing the initial load and improving navigation. This approach ensures that all images and test combinations remain accessible, but only the most critical ones are displayed by default, streamlining the review process and preventing performance bottlenecks.

The Problem: Image Bloat in VRT Workflow

The current Visual Regression Testing (VRT) workflow often results in the posting of hundreds of inline images within Pull Request (PR) comments. This practice, while intended to provide a detailed view of changes, can lead to several significant problems. The sheer volume of images causes massive page bloat, making it difficult and time-consuming to load and navigate the page. Users often face an extensive scroll distance to review all the comments and images, which can be a frustrating experience. More critically, the abundance of images can lead to browser performance issues, potentially overwhelming the system and causing slowdowns or even crashes. The excessive number of images can make it challenging to identify the important diffs among the noise, hindering the review process and increasing the likelihood of overlooking critical changes. The current behavior exacerbates these issues, as ALL <details> sections with changes auto-expand, loading every single image by default. This automatic expansion amplifies the bloat and performance problems, making it imperative to find a more efficient way to present the VRT results.

Proposed Solution: Smart Collapsible Sections

To address the challenges posed by image bloat in the Visual Regression Testing (VRT) workflow, a practical solution is to implement smart collapsible sections. This approach aims to keep all images and test combinations accessible while making most sections collapsed by default. By doing so, we can significantly reduce the initial load and improve the user experience without sacrificing the comprehensiveness of the testing. The core idea is to selectively auto-expand only the most critical sections, allowing reviewers to focus on the areas that are most likely to contain important changes. To achieve this, a priority hierarchy can be established, prioritizing certain operating systems and test scenarios. This hierarchical approach ensures that reviewers can quickly identify and address the most relevant issues, while still having access to the full suite of test results. By making the majority of sections collapsed by default, we can mitigate the problems of page bloat and browser performance, making the review process more efficient and less overwhelming.

Priority Hierarchy

To effectively manage the display of images in the Visual Regression Testing (VRT) workflow, a priority hierarchy is essential. This hierarchy ensures that the most important test results are immediately visible, while others remain accessible but collapsed. The proposed priority is structured around operating systems, recognizing that certain platforms are more widely used and thus require closer attention. The first priority is given to Windows, as it is the most prevalent operating system among users. This prioritization ensures that any issues specific to Windows are promptly addressed. Ubuntu is assigned the second priority, providing good coverage across a different operating system and browser configurations. While macOS is valuable for comprehensive testing, it is deprioritized in this hierarchy. This means that macOS test results are still generated and available, but they are not auto-expanded by default. This prioritization allows reviewers to focus on the most critical platforms first, streamlining the review process and ensuring that the most impactful issues are addressed promptly. By implementing this priority hierarchy, the VRT workflow can be optimized to provide a more efficient and user-friendly experience.

Priority Test Scenarios

Within the expanded combinations of operating systems, it's crucial to further refine the focus by prioritizing specific test scenarios. This ensures that the most critical areas of the application are immediately visible to reviewers. Among these, editor-initial.png holds significant importance as it captures the main code editor view, which is a primary interface for many users. Similarly, gui-initial.png is vital as it showcases the main GUI state, providing an overview of the application's visual presentation. Another key scenario is blocks-tab-active.png, which focuses on core functionality and ensures that essential features are working correctly. Beyond these, other "money shot" scenarios can be identified based on the specific needs and critical areas of the application. By prioritizing these key test scenarios, reviewers can quickly assess the most important aspects of the application's visual integrity. This targeted approach minimizes the time spent sifting through less critical changes and allows for a more efficient and effective review process. This ensures that resources are focused on the areas that have the greatest impact on user experience and application stability.

Implementation Details

To implement the smart collapsible sections within the Visual Regression Testing (VRT) workflow, modifications are required in the workflow configuration file. Specifically, the changes need to be made in .github/workflows/visual-regression-tester.yml, around line ~620. The implementation involves defining priority combinations and tests, then using these definitions to determine whether a section should be auto-expanded or collapsed by default. This can be achieved by incorporating a JavaScript snippet that evaluates the test results and sets the isOpen variable accordingly. The code snippet first defines arrays for priority combinations and tests. The priorityCombos array includes strings such as 'windows-latest-chromium' and 'ubuntu-latest-chromium', representing the operating systems and browsers that should be prioritized. The priorityTests array lists the names of key test images, such as 'editor-initial.png', 'gui-initial.png', and 'blocks-tab-active.png'. A function isPriorityCombination checks if a given key (representing a test combination) includes any of the priority combinations. The hasPriorityChanges function checks if any of the priority tests have diffs. The isOpen variable is then set based on these checks, determining whether a section should be initially open or collapsed. This approach allows for a flexible and maintainable way to control the display of VRT results, ensuring that reviewers can focus on the most critical changes while still having access to the full set of test data.

const priorityCombos = ['windows-latest-chromium', 'ubuntu-latest-chromium'];
const priorityTests = ['editor-initial.png', 'gui-initial.png', 'blocks-tab-active.png'];

const isPriorityCombination = priorityCombos.some(combo => key.includes(combo));
const hasPriorityChanges = result.results?.some(r => 
  r.hasDiff && priorityTests.includes(r.name)
);

const isOpen = (isPriorityCombination && hasPriorityChanges) ? ' open' : '';

Benefits of the Solution

Implementing smart collapsible sections in the Visual Regression Testing (VRT) workflow brings a multitude of benefits. Firstly, all images remain accessible, ensuring that no test data is lost and reviewers can still examine every detail if needed. Similarly, all test combinations continue to run, maintaining the comprehensiveness of the testing process. However, the key advantage lies in the reduction of auto-expanded images. Instead of loading approximately 360 images by default, only about 20-40 images are automatically displayed. This significant decrease in the initial load drastically reduces the scroll distance required to navigate the results, making the review process much more efficient. The reduced load also alleviates browser performance issues, preventing slowdowns and crashes that can occur with excessive image rendering. This ultimately leads to a smoother and more user-friendly experience for reviewers. By prioritizing the display of critical test scenarios and operating systems, the smart collapsible sections ensure that the most important information is immediately visible, while still providing access to the full suite of test results. This targeted approach enhances the overall effectiveness of the VRT workflow and helps to ensure application stability.

Conclusion

In conclusion, implementing smart collapsible sections within the Visual Regression Testing (VRT) workflow presents a robust solution to the challenges posed by image bloat. By selectively auto-expanding only the most critical test results, we can significantly reduce page load, improve browser performance, and streamline the review process. This approach ensures that reviewers can focus on the most relevant changes without being overwhelmed by a flood of images. The priority hierarchy, which emphasizes Windows and Ubuntu platforms along with key test scenarios like editor-initial.png, gui-initial.png, and blocks-tab-active.png, allows for a targeted and efficient review process. The benefits of this solution extend beyond mere convenience; they enhance the overall effectiveness of the VRT workflow, ensuring that issues are identified and addressed promptly. This not only improves application stability but also contributes to a smoother and more user-friendly experience for developers and reviewers alike. By adopting smart collapsible sections, we can maintain the comprehensiveness of visual regression testing while mitigating the performance drawbacks associated with large image sets. For more information on best practices in visual testing, consider visiting Applitools, a trusted resource in the field.

Backlinks

cc: @supervoidcoder