pytest: Save full error output to file
I recently had a problem with some rare occurring failures in tests, they happened at a rate of about 1 time per 5000 full suite runs. These were property-based tests so input-data was automatically generated.
As to make matters worse the entire test suite froze after the failure before it was unable to print what went wrong, at that point Travis-CI would kill the test-runner before we had a chance to catch it.
The natural next step was to start logging failures to a file while running the test suite continuously on a machine set up for the purpose. I used a snippet from the pytest site. At that point I let the tests run in an infinite loop and the next week I hit my first failure. Turns out I hadn’t looked at the snippet to closely, it only logs which tests failed and no details about which params were used or what went wrong.
So I had to tweak the code some and ended up with the following snippet.
At this point, I started running my tests again and this time I were rather lucky, only about 30 hours later I got my first failure with a proper falsifying, repeatable example.
At this point, we could finally fix the bug. I hope other Python developers using pytest can find this useful to achieve the same results.
