PhantomJS Has Many Uses, But Functional Testing Isn't One of Them

February 7th, 2015 Permalink

PhantomJS is a comparatively lightweight headless web browser built on WebKit. When it comes to rendering and interacting with web pages it does everything that any other full-featured browser does with the exception of providing a graphical user interface. You can install it via NPM, meaning that for Node.js ecosystem developers is easy to put PhantomJS in place on build, test, and deployment servers even in more restricted environments. You control PhantomJS either directly via a Javascript API or through a WebDriver interface such as that used by Selenium. So what can you do with a web browser that has no graphical user interface?

Take Screenshots

PhantomJS is fully rendering pages under the hood, so the results can be exported as images. This is very easy to set up, and so is a useful approach for most projects requiring the generation of many browser screenshots.

Generate Data for Performance Analysis

PhantomJS makes it fairly straightforward to automate the generation of data for performance analysis of web applications. Most organizations do far too little of this in any rigorous way, so anything that makes it easier is welcome.

Scrape Javascript Rendered Content

If you want to build automation that extracts a specific value from a website, but that value is put on the page via the operation of Javascript, then you have to run a full-featured browser to get at it. PhantomJS is usually a fair choice for this sort of task. It remains distressingly common to be working with a business partner whose staff offer a full-featured flashy modern website, yet no API for the particular piece of data you really need, and no intention of building that API any time soon.

Automate Interaction with a Web Application

In a similar vein, it is not unusual to find yourself in the situation of having to automate the manual operation of a third party web application. The usual situation is that the API is limited or non-existent, while the nature of the business relationship and available resources make it more cost effective to build the tool than arm-wrestle with the third party and have them add the features you need.

But What About Functional Testing?

Seeing the uses noted above, one might then think that it is helpful to apply PhantomJS to the automated functional testing of a web application. By this I primarily mean end to end testing, such as via Selenium: the automation drives an actual browser through use cases and checks the results. Most of what I'm about to say still applies to unit tests, however. It is tempting to reach for PhantomJS for automated testing when you are stuck in the all too common situation of limited access and control over your test deployment platforms, a sad state of affairs that is all too common in larger organizations. If you are working in the Node.js ecosystem then PhantomJS can become just another package dependency, and now your test scripts can run a browser in addition to a server. You don't even have to involve the IT and devops groups to obtain this functionality.

Tempting as it might be this is not the way to go, however. For any Javascript web application of even moderate complexity, and especially where hover effects are commonplace, there are always significant differences between PhantomJS and browsers employed by end users. In any larger collection of test scripts there will be failures that only occur in PhantomJS, just as there are failures specific to Chromium, Firefox, various iterations of Internet Explorer, and so forth. The problems here are twofold.

Firstly, You Should Be Testing in Those Other Browsers Anyway

Functional testing of a web application should run in all of the browsers you support. That is where the problems are, after all. If you are automating tests, then you should be automating tests in those browsers. If there are issues specific to Chromium then you are not going to discover those problems by testing in PhantomJS. Similarly a clean set of tests conducted in PhantomJS should give you little to no confidence that you have ruled out all meaningful problems in other browsers.

Secondly, PhantomJS Errors are Not Actionable

So there is a test failure in PhantomJS. You fire up Chrome and Firefox and find that the same test case works just fine. This is a lot more common than you might perhaps imagine for any web application that involves a modern Javascript framework, a full team of developers, and hundreds or more automated unit and end to end tests. You have your PhantomJS failure, and now what? Figuring out exactly why a particular test or feature is failing in PhantomJS is considerably harder than the same debugging exercise in Chrome or Firefox, given the available tools, and what exactly does that knowledge give you in the end? Why are you even bothering to fix this since none of your end users make use of PhantomJS?

PhantomJS is Not a Short Cut for Automated Functional Testing

Thus if you are automating your functional end to end testing, then sorry, you are absolutely going to have to set up Selenium servers or some similar tool for all of the browsers you support, or work with third party services such as BrowserStack or SauceLabs, and get involved with the devops and IT groups that manage the infrastructure for your build and deployment process. PhantomJS isn't a shortcut that lets you evade all of this.