DebugContext - error files gets overriden on outline scenarios

[jorgetutor]

When a step fails a screenshot is automatically generated using the feature + step name. The file is overridden on the next outline variable so we only have the last executed scenario (attaching screenshot).

Options:

• Improve naming to include outline variables.
• Add a random string to the file name. It will create several files between different executions, which I don't know if will be an issue or a feature.

[omarlopesino]

I think using outline variables, which should be possible as we already have scenario information, is a good option. because it would help to classify the files. Maybe we should not let the outline variable values to exceed a specific length, in case there are variables with long values (currently it doesn't happen).

I think having different files for different locations may produce a lot of garbage, and I don't know a clear reason to need to preserve files from older executions.

[rsanzante]

I agree with @mistermoper on adding a outline variable to the files.

However, I think it would be interesting to keep different results. Sometimes you are in a debugging session an is interesting to check the evolution. For this I would create a new folder for each execution. But this is a complete new issue (having different folders or each test run).

[jorgetutor]

The file name is calculated in the nuvoleweb library. Should we patch/contribute (nuvoleweb) it or override it as we extend it?

  /**
   * Take a screenshot after failed steps or save the HTML for non js drivers.
   *
   * @AfterStep
   */
  public function takeScreenshotAfterFailedStep(AfterStepScope $event) {
    if ($event->getTestResult()->getResultCode() !== TestResult::FAILED) {
      // Not a failed step.
      return;
    }
    try {
      $step = $event->getStep();
      $file_name = str_replace('/', '-', $event->getFeature()->getFile()) . '-' . $step->getKeyword() . '_' . $step->getText();
      $file_name = 'behat-failed__' . $file_name;
      $message = "Screenshot for failed step created in @file_name";
      $this->createScreenshotsForErrors($file_name, $message, $event->getTestResult());
    }
    catch (DriverException $e) {
    }
  }

[rsanzante]

I would report it in their repo with a patch if possible, if they prefer to leave it as is we can use the patch to override it in our context.

[omarlopesino]

Probably we can propose a PR to nuvole/drupal-behat overriding the filename in the YML configuration.

Something like this:

filename: behat-failed__[feature]-[step-keyword]-[step-text]

That would be the default name.

Then, if we want to override it in other projects we can put

filename: behat-failed__[feature]-[outline-identifier]-[step-keyword]-[step-text]

Then code is not needed to override the filename, only configuration.

What do you think @rsanzante @jorgetutor ?

[rsanzante]

Are those tokens already available or we should add code to support them?

[omarlopesino]

Both the tokens and the filename YAML property do not exist, it must be added code in the ScreenshotContext to support them. I was thinking in doing a basic str_replace

[rsanzante]

For what I'm see in the original code only the outline identifier needs to be added.

The problem is AfterStepScope has no knowledge of the outline. This means we would have create our own outline identifier on each scenario outline (I guess
using BeforeScenarioScope and AfterScenarioScope.

But this seems too complex.

What about checking the filename before writing and adding a number (_1, _2, _3) if it already exists?