How to Autorun Python Tests in VSCode

Step-by-step guide on how to automatically test your Python code on every change.

Photo by Florian Olivo on Unsplash

Introduction

I’ve been using PyCharm as my main development IDE for a long time, and think its integration is still the best for Python programming. The major drawback, in my opinion, is the fact that Jupyter support is not available in the community edition. As I explained in my previous story “Why Data Scientists Should use Jupyter Notebooks with Moderation”, what works best for me is to develop the main code in an comprehensive IDE and test it “on-the-fly” using Jupyter. This can be achieved using PyCharm and an instance of Jupyter, side by side as shown in Figure 1.

Figure 1: Working with PyCharm and Jupyter side by side. Image by author.

Although it works well, it lacks some integration when we need, for example, to debug pieces of code that are spread across the two environments. VSCode came in handy to solve this with its integrated Jupyter support. It is possible, for example to start debugging in the Jupyter and follow the execution in the code flawlessly.

The Issue

One point that was still annoying for me (at least up to now) was the automation of tests with pytest. In PyCharm everything is automated, so when you change your code base, the tests are repeated, but in VSCode there is no such functionality.

This functionallity has been already demanded in the VSCode project, but due to the limited number of “votes” it has been declined. You can follow the issue here: Support for VS Code Testing — Auto Run.

Thank you for submitting your feature request and everyone who considered it! Unfortunately this issue did not receive enough votes over the allotted time, and so we are closing the issue.

The workaround

The best workaround I found for this is from Dan Ciborowski (from MSFT) in this thread: https://stackoverflow.com/questions/72041831/auto-run-tests-in-visual-studio-code/72042405.

Although it works, it calls pytest from the console and the output is not visually provided in the IDE. Now, I will show how to fully automate the tests for a Python (that will also work for other languages) project.

Step 1

First, create the tests folder within your project to store your tests code. Then, configure the testing environment in the activity bar (on the left), or in the command palette (CTRL+Shift+P — Python: Configure tests) choose pytest and point to the tests folder recently created (Figure 2). Make sure pytest is installed in your environment.

Figure 2: Configuring the testing environment.

Step 2

Once the tests are configured in the folder, create a simple test.py file (inside tests) and write a test function. It can be as simple as the one shown in Figure 3, and then it will start to showing the results in the testing Active Bar.

Figure 3: Test function and the testing tree.

Step 3

Now, to make it automatic when we save a file, we will use the RunItOn extension (here). I know the most common is RunOnSave, but it will not run an internal VSCode command, just shell commands.

Once the extension RunItOn is installed, we can go to settings.json file and copy the following configuration. That tells RunItOn to run the internall command testing.runAll everytime a .py file is saved within the project (Figure 4).

"runItOn": {
        "commands": [
            {
                "match": "\\.py$",
                "isAsync": true,
                "isShellCommand": false,
                "cmd": "testing.runAll"
            },
        ],
    },

Figure 4: RunItOn configuration.

Conclusion

Testing is a best practice I think everyone should follow, even for data science projects … that usually don’t follow best practices (Read more here: 7 Reasons Why Scientific Software are Not Well Designed). O spent almost one afternoon trying to figure out how to do this little trick in VSCode and I hope this simple tutorial can help other developers to do it quicker.

Join Medium with my referral link - Maurício Cordeiro

More content at PlainEnglish.io. Sign up for our free weekly newsletter. Follow us on Twitter, LinkedIn, YouTube, and Discord.

Interested in scaling your software startup? Check out Circuit.