详谈pytest中的xfail

原文链接：　https://docs.pytest.org/en/7.2.x/how-to/skipping.html

链接中详细阐述了skip和xfail两种情况

  • xfail 应该译作expected fail，预期失败(我知道这个用例是因为某些原因会失败的)
  • 有哪些原因呢？
    • 暂未实现的功能
    • 实现的功能有bug
  • 如果一个本应失败的用例通过了，那会标记为XPASS，失败是符合预期的，应显示为XFAIL

使用方法

装饰器用法及入门

  • 示例代码

    # demo.py #
import pytest

def add(x,y):
    return x*y # 这个功能有问题

@pytest.mark.xfail  # 你知道这个case会失败，才打了这个标记
def test_add():
    assert add(2,3) == 5 # 你预期是5，哪怕你知道结果会是6，你应该要写最终的

if __name__ == '__main__':
    pytest.main(['-sv',__file__])

  • 示例输出

    test_xfail.py::test_add XFAIL

  • 如果你不带-v得到的结果是

    pytest.main(['-s',__file__])

test_xfail.py x  # 注意，此处是小写的x

  • 如果开发解决了这个bug，但你不知道，你运行你的用例，得到的结果是XPASS

    import pytest

def add(x,y):
    return x+y # 这个功能解决了

@pytest.mark.xfail  # 你以为这个case会失败，实际开发解决了，没告诉你，才打了这个标记
def test_add():
    assert add(2,3) == 5 

if __name__ == '__main__':
    pytest.main(['-sv',__file__])

  • 示例输出

    test_xfail.py::test_add XPASS

  • 同样的，你不带-v运行

    pytest.main(['-s',__file__])

test_xfail.py X  # 此处是大写的X

函数内嵌在用例中

  • 示例代码

    import pytest

def valid_config():
    '''
    做了一些配置
    '''
    return False # 应该返回True

def test_function1():
    if not valid_config():
        pytest.xfail('配置失败了...')
    else:
        print('配置成功,继续测试')
        
if __name__ == '__main__':
    pytest.main(['-sv',__file__])

  • 示例输出

    test_xfail.py::test_function1 XFAIL (配置失败了...)

pytest.mark.xfail中的参数

  • 概览

    pytest.mark.xfail(condition=None, *, reason=None, raises=None, run=True, strict=False)¶

  • 参数说明

    参数 类型 说明
    condition bool or str Condition for marking the test function as xfail (True/False or a condition string). If a bool, you also have to specify reason
    reason str Reason why the test function is marked as xfail.
    raises Type[Exception] Exception subclass (or tuple of subclasses) expected to be raised by the test function; other exceptions will fail the test.
    run bool If the test function should actually be executed. If False, the function will always xfail and will not be executed (useful if a function is segfaulting).
    strict bool If False (the default) the function will be shown in the terminal output as xfailed if it fails and as xpass if it passes. In both cases this will not cause the test suite to fail as a whole. This is particularly useful to mark flaky tests (tests that fail at random) to be tackled later.
    If True, the function will be shown in the terminal output as xfailed if it fails, but if it unexpectedly passes then it will fail the test suite. This is particularly useful to mark functions that are always failing and there should be a clear indication if they unexpectedly start to pass (for example a new release of a library fixes a known bug)

condition参数

  • condition参数可以是字符串或布尔值(更常见)，这是xfail生效的条件

  • 示例代码

    import pytest

age = 24

@pytest.mark.xfail(age>=18,reason='年龄满足大于等于18岁')
def test_function():
    assert False   # 你预期这个用例是失败的才加xfail,所以你这里应该是False


if __name__ == '__main__':
    pytest.main(['-sv', __file__])

  • 注意，当条件满足的时候(age=24,大于18)，这个xfail会加到被测函数上

  • 此时的结果就是

    test_xfail.py::test_function XFAIL (年龄满足大于等于18岁)

  • 如果改为age = 16

  • 运行结果是

    test_xfail.py:6: AssertionError
=========================== short test summary info ===========================
FAILED test_xfail.py::test_function - assert False
============================== 1 failed in 0.07s ==============================

  • 意思就是条件不满足了，你的xfail不会挂到被测函数上

    #@pytest.mark.xfail(age>=18,reason='年龄满足大于等于18岁')  # 相当于去掉，注释掉了
def test_function():
    assert False

  • 有的时候是无条件的(跟运行环境无关)，那你直接写为True即可

    @pytest.mark.xfail(True,reason='这个case目前有BUG')
def test_function():
    assert False 
# test_xfail.py::test_function XFAIL (这个case目前有BUG)

  • 注意，当condition是布尔值的时候，reason参数是必须要写的，否则会报错

    @pytest.mark.xfail(True)
def test_function():
    assert False
# Error evaluating 'xfail': you need to specify reason=STRING when using booleans as conditions.

  • condition的值是str的话，可以不传reason，但也不能随便传

    @pytest.mark.xfail('what')
def test_function():
    assert False

  • 会提示

    name 'what' is not defined

During handling of the above exception, another exception occurred:
Error evaluating 'xfail' condition
    what

  • 你写的str应该是可以用python的eval处理的一个expression，比如这样

    @pytest.mark.xfail('True')

@pytest.mark.xfail('False')

@pytest.mark.xfail('sys.version_info >= (3,11)') # 当前的python版本是否大于等于3.11
def test_function():
    ...

reason参数

  • 为何会标记为xfail的原因，就是个字符串

  • 示例代码

    @pytest.mark.xfail(reason="known  issue")
def test_function():
    assert False

  • 示例结果

    test_xfail.py::test_function XFAIL (known parser issue)

  • 没有-v可看不到

raises参数

  • 你预期这个case会出现某个exception(或者多个)，如果是其他的exception会让这个用例失败

  • 示例代码

    @pytest.mark.xfail(raises=NameError)
def test_function():
    print(a)  # a没有定义
    assert True

  • 结果会是XFAIL：test_xfail.py::test_function XFAIL

  • 如果你改为其他的exception，你raises的只有NameError

    @pytest.mark.xfail(raises=NameError)
def test_function():
    print(3/0)  # ZeroDivisionError
    assert True

  • 此时的结果

    =========================== short test summary info ===========================
FAILED test_xfail.py::test_function - ZeroDivisionError: division by zero
============================== 1 failed in 0.11s ==============================

  • 如果要多个exception，就写到tuple里面即可

    @pytest.mark.xfail(raises=(NameError,ZeroDivisionError))
def test_function():
    print(3/0)
    print(a)
    assert True   # 其实跟此处的True或者False无关，只要报出来exception，都会标记用例为XFAIL

run参数

  • If the test function should actually be executed. If False, the function will always xfail and will not be executed (useful if a function is segfaulting).

  • 说那么多，意思其实是，你如果设置run=False，就不会去运行你的用例，直接就标记为XFAIL了

  • 示例代码

    @pytest.mark.xfail(run=False)
def test_function():
    print('hello')
    assert False

  • 结果是这样的

    test_xfail.py::test_function XFAIL ([NOTRUN] )

  • run=True是默认行为，不写就是之前所有的测试效果咯

strict 参数

  • 如果你了这个参数，并设置为strict=True，那么XPASS就不存在了，会被标记为failed

  • 先来个没有strict的，回忆下

    @pytest.mark.xfail(True,reason='KNOWN BUG')
def test_function():
    assert True

  • 结果是XPASSED

  • 加参数

    @pytest.mark.xfail(True,reason='KNOWN BUG',strict=True)
def test_function():
    assert True

  • 结果是这样的

    ================================== FAILURES ===================================
________________________________ test_function ________________________________
[XPASS(strict)] KNOWN BUG
=========================== short test summary info ===========================
FAILED test_xfail.py::test_function
============================== 1 failed in 0.06s ==============================

  • 如果你case是False的，那无论这个strict是否为True(不写就是为False)，结果都是XFAIL

  • 等价的写法，写到pytest.ini

    [pytest]
xfail_strict=true

抛弃xfail

  • 如果哪一天你在你的代码中写了很多的xfail

  • 一个个去掉略显麻烦

  • 好的处理方法就是加个命令行参数--runxfail

  • 示例代码：下面的代码有--runxfail和没有，你可以执行看看

    import pytest

@pytest.mark.xfail
def test_function1():
    assert False

@pytest.mark.xfail
def test_function2():
    assert False

if __name__ == '__main__':
    pytest.main(['-sv', '--runxfail',__file__])

  • 也等价于

    [pytest]
addopts = --runxfail

pytest的-r参数

  • pytest有个命令行参数-r，跟xfail有一定的关系，来看下解释

    -r chars

show extra test summary info as specified by chars: (f)ailed, (E)rror, (s)kipped, (x)failed,
(X)passed, (p)assed, (P)assed with output, (a)ll except passed (p/P), or (A)ll. (w)arnings are enabled by default (see --disable-warnings), 'N' can be used to reset the list. (default: 'fE').

  • 来看看效果，示例代码

    import pytest

@pytest.mark.xfail(True,reason='known bug1')
def test_function1():
    assert False

@pytest.mark.xfail(True,reason='known bug2')
def test_function2():
    assert True

@pytest.mark.skip(reason="Not Implemented")
def test_function3():
    assert True

if __name__ == '__main__':
    pytest.main(['-r xXs', __file__])

  • 如果加了-r xXs的话，输出结果类似这样

    =========================== short test summary info ===========================
XFAIL test_xfail.py::test_function1
  known bug1
XPASS test_xfail.py::test_function2 known bug2
SKIPPED [1] test_xfail.py:11: Not Implemented
================== 1 skipped, 1 xfailed, 1 xpassed in 0.11s ===================

  • 如果没有-r

    test_xfail.py xXs                                                        [100%]
# 如果带-sv
test_xfail.py::test_function1 bug1
XFAIL (known bug1)
test_xfail.py::test_function2 bug2
XPASS (known bug2)
test_xfail.py::test_function3 SKIPPED (Not Implemented)

================== 1 skipped, 1 xfailed, 1 xpassed in 0.11s ===================

  • 你会发现-sv的效果差不多，只是说多了个short test summary info

pytest.param

  • 当你是用pytest.mark.parametrize的时候，对其中部分用例要做skip或者xfail的话，可以用pytest.param来实现

  • 比如这样

    import sys

import pytest


@pytest.mark.parametrize(
    ("actual_num", "expected"),
    [
        (1, 2),
        pytest.param(1, 0, marks=pytest.mark.xfail),
        pytest.param(1, 3, marks=pytest.mark.xfail(reason="some bug")),
        pytest.param(
            10, 11, marks=pytest.mark.skipif(sys.version_info >= (3, 0), reason="py2k")
        ),
    ],
)
def test_increment(actual_num, expected):
    assert actual_num + 1 == expected

if __name__ == '__main__':
    pytest.main(['-sv',__file__])

  • 输出结果

    test_xfail.py::test_increment[1-2] PASSED
test_xfail.py::test_increment[1-0] XFAIL
test_xfail.py::test_increment[1-3] XFAIL (some bug)
test_xfail.py::test_increment[10-11] SKIPPED (py2k)

=================== 1 passed, 1 skipped, 2 xfailed in 0.11s ===================

进程已结束，退出代码为 0