Providing Reproduction Steps
Updated 209 Days AgoPublic

When you submit a bug report about the Zero Engine or an associated service or product, you MUST include reproduction steps. We can not help you with bugs we can not reproduce, and will not accept reports which omit reproduction steps or have incomplete or insufficient instructions.

Reproduction steps must allow us to reproduce the problem locally on a clean, up-to-date install of Phabricator.
Reproduction should be as simple as possible.
Good reproduction steps can take time to write out clearly, simplify, and verify. As a reporter, we expect you to shoulder as much of this burden as you can, to make make it easy for us to reproduce and resolve bugs. We do not have the resources to pursue reports with limited, inaccurate, or incomplete reproduction steps, and will not accept them. These reports require large amounts of our time and are often fruitless.

Example

==Reproduction Steps:

  • Open the engine
  • Press ++
  • Enter the command: Export Project

    ==Expected Result:

    Windows explorer window should open on top the the zero engine window to allow me to select the export location.

    ==Actual Result:

    Windows explorer window opens behind editor making it so I have to focus another application then refocus the zero engine in order to select my export location.

These steps are complete and self-contained: anyone can reproduce the issue by following these steps. These steps are clear and easy to follow. These steps are also simple and minimal: they don't include any extra unnecessary steps.

Finally, these steps explain what the reporter expected to happen, what they observed, and how those behaviors differ. This isn't as important when the observation is an obvious error like an exception, but can be important if a behavior is merely odd or ambiguous.

Reliable Reproduction

When you file a bug report, the first thing we do to fix it is to try to reproduce the problem locally. We will do this by following the steps you provide. If we can recreate the issue locally, we can almost always resolve the problem. However, many reports do not have enough information, are missing important steps, or rely on data (like other users, other projects, permission settings, etc) that we don't have access to. We either can't follow these steps, or can't reproduce issues by following them.

Reproduction steps must be complete and self-contained, and must allow anyone to reproduce the issue on the same build of the Zero Engine. If the bug you're seeing depends on data or configuration which would not be present by default in the engine, you need to include that information/content in your report. For example, if you're seeing an issue which depends on a particular art asset that asset should be included with your report.

Getting Started

To generate reproduction steps, first find a sequence of actions which reproduce the issue you're seeing reliably.

Next, write down everything you did as clearly as possible. Make sure each step is self-contained: anyone should be able to follow your steps, without access to private or proprietary data.

Now, to verify that your steps provide a complete, self-contained way to reproduce the issue, follow them yourself, if possible on a new and empty project in the same build.

If you can follow your steps and reproduce the issue in a clean project, we'll probably be able to follow them and reproduce the issue ourselves.

If you can't follow your steps because they depend on information which is not available in a clean project(for example, a certain art asset in the game), rewrite them to include instructions to add that information (for example, downloading required asset from bug report and importing it into a clean project).

Once you have working reproduction steps, your steps may have details which aren't actually necessary to reproduce the issue. You may be able to simplify them by removing some steps or describing steps more narrowly. For help, see "Simplifying Steps" below.

Simplifying Steps

In general, you'll simplify reproduction steps by first finding a scenario where the issue reproduces reliably (a "bad" case) and a second, similar situation where it does not reproduce (a "good" case). Once you have a "bad" case and a "good" case, you'll change the scenarios step-by-step to be more similar to each other, until you have two scenarios which differ only a very small amount. This remaining difference usually points clearly at the root cause of the issue.

With a simpler reproduction scenario, you can simplify your steps to be more tailored and mimimal. This will help us pointpoint the issue more quickly and be more certain that we've understood and resolved it.

It is more important that steps be complete than that they be simple, and it's acceptable to submit complex instructions if you have difficulty simplifying them, so long as they are complete, self-contained, and accurately reproduce the issue.

Things to Avoid

Insufficient Information: The most common issue we see is instructions which do not have enough information: they are missing critical details which are necessary in order to reproduce the issue.

Inaccurate Steps: The second most common issue we see is instructions which do not actually reproduce a problem when followed. Verify your steps by testing them.

Private Information: Some users provide reports which hinge on the particulars of commits in proprietary repositories we can not access, or on assets they do not include in their report. This is not useful, because we can not examine the underlying data which may be the cause of the issue.

Screenshots: Screenshots can be helpful to explain a set of steps or show what you're seeing, but they usually aren't sufficient on their own because they don't contain all the information we need to reproduce them.

For example, a screenshot may show a particular object, but not have enough information for us rebuild a similar object locally.

Last Author
arend.danielek
Last Edited
Jun 26 2018, 11:33 AM
Projects
Subscribers
None