Running UI tests in GitHub Actions

17 Feb 2023 🔖 devops web development integration git azure
💬 EN

Think of UI testing frameworks like Selenium and Playwright as pseudo-programming-languages that you use from inside another programming language, like:

• Arithmetic
• Regular expressions

Once you think of them this way, it’s easier to understand why there’s a bit of overhead to get them running without human intervention, like in a GitHub Action.

Languages inside languages

Arithmetic as a sublanguage

Once you understand the arithmetic concept of exponentiation, you know that 3⁴ is 81.

However, to get a computer to tell you that 3⁴ is 81, you might have to do one of the following:

• Have the computer run a Java or older JavaScript program that includes the expression:

    Math.pow(3, 4)
  

• Have the computer run a Python or newer (ES7) JavaScript program that includes the expression:

    3 ** 4
  

• Have the computer run a BASIC program that includes the expression:

    3 ^ 4
  

• Have the computer run a Windows PowerShell script that includes the expression:

    [Math]::Pow(3, 4)
  

• Have the computer run a Linux Bash shell script that includes the expression:

    $((3**4))
  

Regex as a sublanguage

Similarly, once you understand Regular Expression basics, you will know that you can extract the year “2023” from the phrase “On 2023-02-17, I wrote this” by inspecting the value of capturing group #1 when running that sentence against a regular expression of “^.*(\d{4})\-\d{2}\-\d{2}.*$.

However, to get a computer to turn “On 2023-02-17, I wrote this” into “2023,” you might have to do one of the following:

• Have the computer run a Python program that includes the expression:

    re.search("^.*(\\d{4})\\-\\d{2}\\-\\d{2}.*$", "On 2023-02-17, I wrote this").group(1)
  

• Have the computer run a Java program that includes statements like:

    Matcher m = Pattern.compile("^.*(\\d{4})\\-\\d{2}\\-\\d{2}.*$").matcher("On 2023-02-17, I wrote this");
    m.find();
    System.out.println(m.group(1));
  

• Have the computer run a JavaScript program that includes the expression:

    Array.from(("On 2023-02-17, I wrote this").matchAll(/^.*(\d{4})\-\d{2}\-\d{2}.*$/g), m => m[1])
  

• Have the computer run a Windows PowerShell script that includes the expression:

    'On 2023-02-17, I wrote this' -replace '^.*(\d{4})\-\d{2}\-\d{2}.*$', '$1'
  

• Have the computer run a Linux Bash shell script that includes the expression:

    sed -n "s/^.*\([[:digit:]]\{4\}\)\-[[:digit:]]\{2\}\-[[:digit:]]\{2\}.*$/\1/p" <(echo "On 2023-02-17, I wrote this")
  

Selenium as a sublanguage

UI testing frameworks like Selenium and Playwright have standards similar to arithmetic and regular expressions, but typically you ask a computer to run them from within the context of a full-featured programming language whose available instruction set you have expanded by importing an appropriate library, package, or module.

To get a computer to run Selenium and make Selenium search a web page and set aside the contents of that web page’s its first <H1> tag for further processing, you might have to do one of the following:

• Have the computer run a Python program that includes the expression:

    your_python_variable_representing_selenium.find_element(By.TAG_NAME, 'h1')
  

• Have the computer run a JavaScript (web-browser executed) or Node.js (computer-operating-system-executed JavaScript) program that includes the expression:

    your_javascript_variable_representing_selenium.findElement(By.css('h1'))
  

• Have the computer run a Windows PowerShell script that includes the expression (thanks Abdulkadir Akyurt – note though that the library in question is _not officially maintained by the authors of Selenium, so be careful trusting it)_:

    $your_powershell_variable_representing_selenium.FindElement([OpenQA.Selenium.By]::TagName('h1'))
  

---

Installation prerequisites

Programming language and library

Unlike arithmetic and regex, there’s no officially-maintained library of Selenium commands for operating system command line interpreters (“CLIs”) such as Windows PowerShell or Linux Bash.

Before you can run Selenium from a computer’s operating system, you have to install something onto that computer that can interpret commands from 1 of the 5 officially supported programming languages (if it doesn’t already come with one – modern Linux, for example, typically comes with a Python runtime preinstalled), as well as installing a Selenium-specific library/module/package for that programming language onto the computer:

1. C#
2. JavaScript
3. Java
4. Python
5. Ruby

Same idea for Microsoft’s Selenium competitor Playwright, which also needs you to install something onto the computer that can interpret commands from 1 of the 4.5 officially supported programming languages if the computer doesn’t already come with one, as well as installing a Playwright-specific library/module/package for that programming language onto the computer:

1. .NET (e.g. C#)
2. JavaScript & TypeScript (I’m counting these together as “1.5” languages in my “4.5” above)
3. Java
4. Python

Headless web browser

Furthermore, even if you’ve made that computer able to run your programming language of choice and installed the appropriate library, the first thing Selenium is going to try to do is open Chrome, FireFox, Edge, Internet Explorer, or Safari as a “headless web browser” (a.k.a. Selenium’s “WebDriver“).

So you’ll also need to make sure you’ve installed a “WebDriver” executable for the web browser of your choice onto the same computer.

---

GitHub Actions is rent-a-computer

Once you’ve written a program (in your programming language of choice) that uses a UI testing framework like Selenium or Playwright to look for the first <H1> tag in a given webpage and makes sure its contents say “Hello World,” you’ll probably want to run it somewhere else besides your own personal computer.

• Maybe you want to run it nightly at 2AM.
  • (Note: In this case, your <H1>-detecting codebase can have its own Git repository if you’d like.)
• Maybe you want to run it every time you try to do a “pull request” of the codebase behind the website whose home page should always say “Hello World” from one branch of a Git-tracked codebase into another.
  • (Note: In this case, you should probably store your <H1>-detecting codebase inside the same Git repository as the one you’ve already set up to track the codebase behind your website. Furthermore, it probably makes sense to write your <H1>-detecting automation in the same programming language as the codebase behind your website.)

Just add one more YAML-formatted plaintext file to the repository containing your <H1>-detecting program and GitHub will take care of the rest by running a “GitHub Action” as specified in your .yaml file.

(Note: my use of the word “rent” doesn’t mean GitHub Actions will necessarily cost you money. But I believe you’re limited on just how many of them you can run, and how often, before you have to start paying GitHub money. I don’t think they let you run the entirety of a Fortune 500 company’s IT infrastructure on a free tier.)

---

Example code

Head over to the following GitHub repositories to see examples of automated user interface (“UI”) testing codebases that GitHub Actions runs nightly at 2:43AM to validate that the <title> tag of katiekodes.com still has the phrase “Katie Kodes” in it:

1. Using Playwright, written in JavaScript