Bazel test rule for the Python BDD tool "behave"

Question

Question:

Hello, dear bazel heroes!

For our system tests, I am trying to get the behavior-driven development tool behave to run with bazel as

• bazel test system/acceptance_criteria:nice

What needs be accomplished:

• calling the command line behave <feature_file> (solved by filtering runfiles and using <feature_directory> instead of <feature_file>)
• adding the folder steps to the runfiles (solved by explicitly prefixing a file with the path)
• adding relevant Python step files to the runfiles (solved)
• adding the system under test to the PYTHONPATH (open)

What would you recommend or do you have any other suggestions on how to solve this?

• using Skylark for a new rule (I've tried that but struggling with PYTHONPATH and runfiles. I am getting a ModuleNotFoundError: No module named 'my_system')
• using Skylark for a macro with a genrule
• calling a script within a py_test (a possible but not so flexible workaround since I did not figure out how to pass parameters from bazel to the script)
• some other best practice?

Cheers, Chris

If possible, provide a minimal example to reproduce the problem:

What I've tried was pretty much copying the behavior of a py_test in Skylark and letting it run the command behave <feature_directory> or alternatively python -m behave <feature_directory>:

project/

• WORKSPACE (empty)
• system/
  • BUILD
  • my_system.py
  • acceptance_criteria/
    • BUILD
    • nice.feature
    • steps/
      • reusable_steps.py
• tools/
  • BUILD (empty)
  • behave_rule.bzl
  • bazel.rc

system/BUILD:

py_binary(
    name = "main",
    main = "my_system.py",
    srcs = ["my_system.py"],
    # deps = [], excluded for this example
    imports = ["."], # needed for adding PYTHONPATH in acceptance_criteria
    visibility = ["//system/acceptance_criteria:__pkg__"]
)

system/my_system.py:

class MySystem():
    def yeah(self):
        return "yeah it works"

system/acceptance_criteria/BUILD:

load("//tools:behave_rule.bzl", "py_bdd_test")

py_bdd_test(
    name = "nice",
    feats = [
        "nice.feature",
    ],
    steps = [
        "steps/reusable_steps.py",
    ],
    deps = [
        # TODO figure out how to add the implicit imports to the PYTHONPATH
        "//system:main",
    ],
    size = "small",
)

system/acceptance_criteria/nice.feature:

Feature: Multiprocess software

Scenario: 4 processes
  Given the device is powered on

steps/reusable_steps.py:

from behave import *
from my_system import MySystem

@Given("the device is powered on")
def step_impl(context):
    raise NotImplementedError(MySystem().yeah())

tools/behave_rule.bzl:

# =============================================================================
# Description: Adds a test rule for the BDD tool behave to the bazel rule set.
# Knowledge:
# * https://bazel.build/versions/master/docs/skylark/cookbook.html
# * https://bazel.build/versions/master/docs/skylark/rules.html
# * https://bazel.build/versions/master/docs/skylark/lib/ctx.html
# * http://pythonhosted.org/behave/gherkin.html
# =============================================================================

"""Private implementation of the rule py_bdd_test.
"""
def _rule_implementation(ctx):

  # Store the path of the first feature file
  features_dir = ctx.files.feats[0].dirname

  # We want a test target so make it create an executable output.
  # https://bazel.build/versions/master/docs/skylark/rules.html#test-rules
  ctx.file_action(
      # Access the executable output file using ctx.outputs.executable.
      output=ctx.outputs.executable,
      content="behave %s" % features_dir,
      executable=True
  )
  # The executable output is added automatically to this target.

  # Add the feature and step files for behave to the runfiles.
  # https://bazel.build/versions/master/docs/skylark/rules.html#runfiles
  return [DefaultInfo(
      # Create runfiles from the files specified in the data attribute.
      # The shell executable - the output of this rule - can use them at runtime.
      # It is also possible to define data_runfiles and default_runfiles.
      # However if runfiles is specified it's not possible to define the above
      # ones since runfiles sets them both.
      runfiles = ctx.runfiles(
          files = ctx.files.feats + ctx.files.steps + ctx.files.deps)
  )]

"""An example documentation.

Args:
  name:
    A unique name for this rule.
  feats:
    Feature files used to run this target.
  steps:
    Files containing the mapping of feature steps to actual system API calls.
    Note: Since this rule implicitely uses the BDD tool "behave" they must
be in the "steps" folder (https://pythonhosted.org/behave/gherkin.html).
  deps:
    System to test.
"""
py_bdd_test = rule(
    implementation=_rule_implementation,
    attrs={
      # Do not declare "name": It is added automatically.
      "feats": attr.label_list(allow_files=True),
      "steps": attr.label_list(allow_files=True),
      "deps":
        attr.label_list(
            mandatory=True,
            non_empty=True,)
    },
    test=True,
)

tools/bazel.rc:

test --test_output=errors

Environment info

• Operating System: Windows 10 with docker (see Dockerfile below)
• Bazel version (output of bazel info release): release 0.5.1rc1

Dockerfile:

FROM ubuntu:xenial

# Install essentials
ENV DEBIAN_FRONTEND noninteractive
RUN apt-get update \
  && apt-get install -y gnupg git \
  && apt-get clean

# -----------------------------------------------------------------------------

# Install the awesome build automation tool bazel
# https://bazel.build/
# version > 0.4.5 since it has a bug in the rule extension skylark
RUN echo "deb http://storage.googleapis.com/bazel-apt testing jdk1.8" \
  > /etc/apt/sources.list.d/bazel.list
RUN apt-key adv --keyserver pool.sks-keyservers.net --recv-key 3D5919B448457EE0
RUN apt-get update \
  && apt-get install -y openjdk-8-jdk bazel \
  && apt-get clean \
  && rm -rf /var/lib/apt/lists/* && mkdir /var/lib/apt/lists/partial
ENV DEBIAN_FRONTEND ""

# Setup environment for bazel
ENV JAVA_HOME /usr/lib/jvm/java-8-openjdk-amd64

# Run bazel a first time for it to self-extract
RUN /usr/bin/bazel version

# -----------------------------------------------------------------------------

# Install Python (using the non-default version 3.6)
RUN echo "deb http://ppa.launchpad.net/jonathonf/python-3.6/ubuntu xenial main" \
  > /etc/apt/sources.list.d/jonathonf-ubuntu-python-3_6-xenial.list \
  && apt-key adv --keyserver pool.sks-keyservers.net --recv-key 8CF63AD3F06FC659 \
  && apt-get update \
  && apt-get install -y python3.6

# Create symlink from python3.6 to python
RUN ln -s /usr/bin/python3.6 /usr/local/bin/python

# Install the Python package manager pip (for the non-default version 3.6)
# https://en.wikipedia.org/wiki/Pip_(package_manager)
RUN apt-get update \
  && apt-get install -y wget \
  && wget https://bootstrap.pypa.io/get-pip.py \
  && python3.6 get-pip.py \
  && rm get-pip.py \
  && pip install --upgrade pip

# -----------------------------------------------------------------------------

# Install the Gherkin-based BDD tool "behave" for Python
RUN pip install behave

# -----------------------------------------------------------------------------

# Our build environment is based on bazel. Now run tests with it.
ENTRYPOINT "/usr/bin/bazel"

Have you found anything relevant by searching the web?

Unfortunately not. As mentioned in the comment of the behave_rule.bzl I looked into

• https://bazel.build/versions/master/docs/skylark/cookbook.html
• https://bazel.build/versions/master/docs/skylark/rules.html
• https://bazel.build/versions/master/docs/skylark/lib/ctx.html

and I've also found

• https://github.com/bazelbuild/bazel/issues/702

Anything else, information or logs or outputs that would be helpful?

The command line output:

root@eec1fa791491:/project# bazel test system/acceptance_criteria:nice
INFO: Found 1 test target...
FAIL: //system/acceptance_criteria:nice (see /root/.cache/bazel/_bazel_root/2ca1f4ebdc59348ffdc31d97a51a98d5/execroot/project/bazel-out/local-fastbuild/testlogs/system/acceptance_criteria/nice/test.log).
INFO: From Testing //system/acceptance_criteria:nice:
==================== Test output for //system/acceptance_criteria:nice:
Exception ModuleNotFoundError: No module named 'my_system'
Traceback (most recent call last):
  File "/usr/local/bin/behave", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.6/dist-packages/behave/__main__.py", line 109, in main
    failed = runner.run()
  File "/usr/local/lib/python3.6/dist-packages/behave/runner.py", line 672, in run
return self.run_with_paths()
File "/usr/local/lib/python3.6/dist-packages/behave/runner.py", line 678, in run_with_paths
    self.load_step_definitions()
File "/usr/local/lib/python3.6/dist-packages/behave/runner.py", line 658, in load_step_definitions
    exec_file(os.path.join(path, name), step_module_globals)
File "/usr/local/lib/python3.6/dist-packages/behave/runner.py", line 304, in exec_file
    exec(code, globals, locals)
File "system/acceptance_criteria/steps/reusable_steps.py", line 5, in <module>
    from my_system import MySystem
ModuleNotFoundError: No module named 'my_system'
================================================================================
Target //system/acceptance_criteria:nice up-to-date:
bazel-bin/system/acceptance_criteria/nice
INFO: Elapsed time: 0.538s, Critical Path: 0.22s
//system/acceptance_criteria:nice                                        FAILED in 0.2s
/root/.cache/bazel/_bazel_root/2ca1f4ebdc59348ffdc31d97a51a98d5/execroot/project/bazel-out/local-fastbuild/testlogs/system/acceptance_criteria/nice/test.log

Executed 1 out of 1 test: 1 fails locally.

Originated from:

• https://github.com/bazelbuild/bazel/issues/3194#issuecomment-309236573

Answer 1

After some years have passed, sat down for a while and worked out this

bdd.bzl

"""Custom Bazel Starlark rule for running behave tests.

Idea taken from https://github.com/bazelbuild/examples/
Also read https://docs.bazel.build/versions/main/skylark/concepts.html
"""

def _bdd_test_impl(ctx):
    # Assemble the command to run.
    bdd_tool = 'behave'
    default_args = '--no-capture --no-capture-stderr --no-logcapture'
    # Optional arguments are passed either
    #  explicitly as --test_arg=--foo="bar bar" (no space in between)
    #  or implicitly via ctx.attr.args
    optional_args = '"$@"'

    feature_file = ctx.file.main.basename
    features_path = ctx.label.package

    command = ' '.join([bdd_tool, default_args, features_path, '--include', feature_file])
    command = ' '.join([
        bdd_tool, features_path, '--include', feature_file, default_args, optional_args
    ])

    # Wrap an executable around the command.
    ctx.actions.write(
        output = ctx.outputs.executable,
        content = command,
    )

    # Ensure the files needed by the command are available at runtime
    srcs = ctx.files.main + ctx.files.steps + ctx.files.data
    deps = []
    for dep in ctx.attr.deps:
        # Collecting the files from the dependency tree is a little more hassle
        # since not well-documented and prone to change between bazel versions.
        deps += dep.default_runfiles.files.to_list()

    return [DefaultInfo(
        runfiles = ctx.runfiles(
            files = srcs + deps,
        ),
    )]

bdd_test = rule(
    implementation = _bdd_test_impl,
    test = True,
    attrs = {
        "main": attr.label(
            allow_single_file = [".feature"],
            mandatory = True,
        ),
        "steps": attr.label_list(
            allow_files = True,
        ),
        "deps": attr.label_list(),
        "data": attr.label_list(
            allow_files = True,
        ),
    },
)

BUILD

load("bdd.bzl", "bdd_test")

[bdd_test(
    name = filename[:filename.rfind(".feature")],
    main = filename,
    steps = ["environment.py"] + glob(["steps/*.py"]),
    deps = [
        # ':some_fixture',
    ],
) for filename in glob(["*.feature"])]