Skip to content
kumo

Bugs & QA

A bug in kumo is a task of a bug type, so it moves on the same board as the rest of the work. What makes it a bug is that it knows its build: the build that found it and the build that fixed it. The bug, crash and feedback types, their severity and their steps are on every board from the start, and the studio can change them.

Tasks → Bugs groups what comes in the way an error tracker does: each issue with its count of events, the builds and platforms it was seen on, its stack trace and what led up to it. Assign, resolve or ignore an issue, or merge it into another. Where the studio connects Sentry, the page stays in step with it, merges included.

  • Crashes. One bug per crash group, from Sentry, Backtrace or BugSplat, or from kumo’s own receiver for Unreal’s crash reporter if the studio has no crash service. The bug keeps its count, first and last build, platforms and top frames current.
  • Errors from any code, with replays. kumo speaks Sentry’s own protocol, so any Sentry SDK (browser, Node, Python, Unity, Godot, native) can report to the studio’s server instead of sentry.io: point its DSN at the one under Insights → Connect a game. Each error becomes a bug of its group, with its stack, breadcrumbs, release and a hash of the user; a native crash keeps its minidump. A web page’s session replay plays on the event, its text and inputs hidden as the SDK masks them, and is kept 30 days.
  • Inside the game. The in-game reporter lets a tester send a bug with the build and a screenshot attached. It ships in test builds, and in a Shipping build only where the studio turns it on.
  • Players. Steam reviews and messages in chosen Discord channels are read for bugs. kumo proposes a bug; a person confirms it, merges it into an existing one or dismisses it.
  • A public form anyone can send a bug through.
  • Discord. /bug from any channel.
  • Email. A mail to the project’s address becomes a task.

A Shipping build ships without its debug symbols, so the stack a player’s crash reporter can read is module names and addresses. kumo’s receiver keeps the crash’s minidump and reads it again on the studio’s server with the build’s own symbols, as Sentry and Backtrace do: the game’s frames come back with their function, file and line, the engine’s with their function.

  • Builds of the studio’s CI service send their symbols by themselves: the build computer writes them from the build’s PDBs, ELF debug files or dSYMs as it sends the build.

  • Any other build sends them from the computer that has it, with one line in its build script:

    Terminal window
    python cli.py symbols.upload --path <the build's folder or zip> --build <build>

    or the Send the build’s symbols step after an automation’s package step. <build> is what the game names its build (KumoBuild, its project version unless the settings say otherwise). Crashes of that build from the last 30 days that lacked them are read again.

  • Windows’ own modules are read with Microsoft’s public symbols.

A crash is grouped by its first frames of the game’s own code, which stay the same from build to build; a crash filed before its symbols arrived moves to that bug once it is read. Each event says whether it was read, which of the game’s modules lacked symbols and of which build, or why its frames are the ones its reporter sent; Save the minidump gives the dump to open in a debugger. Where the board knows the studio’s source control (Symbols on the Bugs page), each of the game’s frames links to its file at the revision the build records.

To see a crash arrive end to end, launch a build with -KumoTestCrash: the game crashes as it starts, from kumo’s game plugin. What a kept minidump holds is in the privacy policy.

Builds are listed under Tasks → Automations → Builds. There each build is moved into QA, released or withdrawn, the in-game reports are switched on or off, and the bugs a build fixed are listed. See Builds & release.

QA works from the task it is given. A task’s Testing section holds:

  • Test runs: a suite’s checklist run against a build and a platform, step by step: smoke tests, regression and certification alike.
  • Playtests: the build to play, who plays it, the survey, and the feedback grouped and turned into tasks.

Certification checklists are test suites run against a release build. kumo ships their structure, never the platform holders’ text: the console requirements are under NDA, so each studio brings its own.