Troubleshooting flaky builds and false diffs

A false diff is a change that shows up in review although nothing about your product changed. Most of them come from how the screenshot was captured, not from the comparison, so fix the capture first. The examples use argosScreenshot from @snapvisor/playwright (checked against 7.3.7); the option names are the SDK's own and keep the ARGOS_* environment variable names.

Screenshots differ between your machine and CI

Playwright documents that browser rendering can vary with the host OS, its version, settings, hardware and headless mode, and tells you to run tests in the same environment where the baseline screenshots were generated. Fonts and text rendering differ between Linux, macOS and Windows, so a screenshot taken on a laptop will often not match one taken in CI.

Upload from CI only. The SDK examples already do this with uploadToArgos: !!process.env.CI (see SDKs), so every build is rendered in one environment. If you need a reproducible local run, use the official Playwright Docker image, pinned to the same version as your @playwright/test npm package:

mcr.microsoft.com/playwright:v<version>-<distro>
# for example, @playwright/test 1.63.0 on Ubuntu 24.04:
mcr.microsoft.com/playwright:v1.63.0-noble

Playwright publishes -noble, -jammy and -resolute variants of each release, and recommends pinning to a specific version. If the image and the npm package disagree, Playwright cannot locate the browser executables. Pin @playwright/test to an exact version too, so the two cannot drift. Playwright's CI guide shows the image as a GitHub Actions container.

Animations, carets and moving content

argosScreenshot stabilizes the page before it captures. By default it:

• passes animations: "disabled" to Playwright: finite animations are fast-forwarded to their end, infinite ones are reset to their initial state;
• hides text carets and scrollbars (through the ::-webkit-scrollbar rule), turns off spellcheck markers, and freezes animated GIFs on their first frame (add data-image-type="gif" to a GIF served from a URL without a .gif extension).

Dates, ads, avatars and other content that changes on every run need your help:

• Freeze the clock with Playwright's await page.clock.setFixedTime(new Date("2024-02-02T10:00:00")), which makes Date.now and new Date() return a fixed time.
• Mask a region with Playwright's mask option, which the SDK passes through: mask: [page.locator(".avatar")].
• Mark elements in your markup with data-visual-test. "blackout" is masked on every screenshot, "transparent" hides the content but keeps the layout, and "removed" is display: none.
• Inject CSS for one screenshot with argosCSS, for example argosCSS: ".ad-slot { visibility: hidden }".

Waiting for fonts and images

Before capturing, the SDK waits until document.fonts.status is loaded, every <img> in the document reports complete, no visible element has aria-busy="true", and every GIF has been frozen. If that does not settle in time, the screenshot fails with Failed to stabilize screenshot and lists what was still pending, instead of uploading a half-loaded page.

CSS background images are only awaited on elements flagged with data-visual-test-wait-bg-img (and their descendants). Widen that to the whole document with stabilize: { waitForBackgroundImages: true }, or target your own selector with stabilize: { waitForBackgroundImages: { selector: ".hero" } }. Any check can be turned off, for example stabilize: { waitForFonts: false }.

What you still do yourself: the SDK only knows about elements already in the page. Wait for your own data and rendering first (for example await expect(page.getByRole("heading", { name: "Dashboard" })).toBeVisible()). A region that keeps aria-busy="true" while it loads is waited on until no visible element still has it. Nothing in the stabilization list covers video or canvas content.

Threshold

The diff threshold is a number between 0 and 1, default 0.5, where higher means less sensitive. Values outside the range throw The threshold must be between 0 and 1. It is not a project setting. You set it per screenshot, and Snapvisor stores it with that screenshot and shows it as a chip in the build sidebar:

await argosScreenshot(page, "chart", { threshold: 0.7 });

For a whole directory upload use npx @snapvisor/cli upload ./screenshots --threshold 0.7. The ARGOS_THRESHOLD environment variable sets a default for screenshots that carry no threshold of their own. Raising it hides real changes too, so fix the capture first; see Thresholds for how it compares to other tools.

Compared against the wrong baseline

If a build shows changes you never made, or no baseline at all, check which build it was compared with. Baseline builds explains how the reference build and the base branch are resolved, including the first build on a branch and what happens after a force-push. Branches you auto-approve set their own baseline; see Monitoring mode.

Common causes:

• The upload ran under a branch you did not expect. On deployment_status workflows the SDK uses the environment name, so see Run on a preview deployment.
• You want a fixed baseline. The CLI takes --reference-branch and --reference-commit, and the SDK reads ARGOS_REFERENCE_BRANCH and ARGOS_REFERENCE_COMMIT.

Flaky screenshots

If a screenshot still wobbles after the fixes above, let Snapvisor learn from it: see Flaky test detection to ignore recurring changes or auto-ignore the ones that keep returning.