Cross browser testing

The internet is really only made useful by the way we interact with it, and that's usually through a web browser of some description. Even some of the apps we use on our phones are nothing more than browsers limited to certain activities.

As with most tools there are many browsers to choose from. There are hundreds of browsers on the market but there are four or five big players that most people design and build sites to work on.

These are:
  • Mozilla Firefox
  • Google Chrome
  • Microsoft Internet Explorer
  • Apple's Safari
  • Opera

These browsers are designed for different operating systems also such as Apple, Windows, Linux and the growing array of mobile platforms too.

Add to this the sheer number of different versions of each browser (as they release updates) and it can quickly become practically impossible to test against every browser, version and platform.

To deal with this I would suggest a few things.

What is supported?

Check what your business says it supports. Most companies are going down the route of only supporting the latest big version number of a browser, for example Internet Explorer 10.

Others don't specify at all which presumably means their customers can use any browser they wish.

Building and designing websites to work on old browsers is really hard and not a path I would suggest unless your customer really do use old browsers. It's expensive to build, design, test and maintain software for old browsers. So start by working out what your company support.

You can then immediately narrow down your matrix of browsers and versions.

For example, you may decide you only support IE10, latest Chrome, latest Firefox and Safari MAC.

What do your customers use?

If you already have a website being used by your customers then install tools such as Google Analytics or NewRelic (other tools are available). These tools will tell you what browsers your customers used to connect to your site.

Compare what is being used with what is being supported and you may be surprised. Just because you don't "support" an old browser doesn't mean that a large portion of your user base aren't using it.

Whatever the outcome you will have more information to use in making a decision. The information will help you to choose what you support, what you build against and what you test against.


How to test across browsers

There are two main approaches to bear in mind when testing on browsers.

Firstly, does the business logic and flow of the site work in the browser? Can the customer do what you want them do?
Secondly, does the site look like what you expected it would look like? Rendering issues are common.

The first approach is something you should consider writing automated tests for. Once you have an automated test you will know if the logic no longer works; you will have a failed test.

The second approach is much harder to write an automated test for. This is the kind of testing that is best done by a human.

Run tests against a mixture of browser environments
One of the simplest ways to test across browsers is to simply use a number of supported browsers as you test.

For example, you may be testing a simple web application where the user can log in, generate some reports, send the reports and then log out. The system also has a simple “management" system where sys admins or managers can view who is changing what.

To get a wider coverage you could test your log on functionality in one browser, test the send report functionality in another browser and then the audit trail functionality using a third browser.

This is an effective way of covering different combinations of browsers at the same time as doing your day-to-day testing. The above example won’t highlight a bug with the audit trail functionality in the first and second browser though. So there are plenty of gaps for bugs to slip through, however the time it saves can give you some confidence that the basics are working.

This approach also assumes that the site is not very big and hence won't take a huge amount of time to test.

Tools such as Browserstack (https://www.browserstack.com/) and Saucelabs (https://saucelabs.com/) provide browsers of all sorts, types and versions for you to use for testing. There is some cost for this service but it's better than trying to virtualise your own browsers.

Let someone else do it
For appearance issues there are many online tools like Browser Shots (http://browsershots.org/), which will load your page in any of the supported browsers (they support a great many versions), take a screen shot and then make these screen shots available to you.

This is great for websites, but applications that require credentials or have a huge number of pages are somewhat more difficult.

Check against standards
Checking against HTML will give you more confidence the site works across many browsers, but there are still some browsers that will render pages differently.

You could validate your site against an HTML standards checker.

The W3 checker is a good standards checker - (http://validator.w3.org/)

Automate
You could automate your checking using something like Selenium (http://seleniumhq.org/) and then use their Selenium Server product with an in-built grid feature.

Using Selenium has infrastructure requirements and is reliant on you having an automation suite.

Automated checks will only check what you have coded them to check. Automated checks are great for checking functionality but will not tell you about layout issues between browsers; unless of course the layout issue causes some functionality not to work.

Manually Test It
You could manually test against each version by installing all of the versions on different machines. To make this less of a chore it would be worthwhile using Virtual Machines, or a service like Browserstack or Saucelabs.

If you are using your own virtual machines then you can clone them and other people can use them. This approach also relies on infrastructure, hardware and lots of time.

Emulate it
You could emulate the different browsers. There are a number of tools on the market, as well as some inbuilt tools within the browsers themselves to emulate older versions.

A few words of caution though, they are not wholly accurate in my experience, but they *may* provide insights that are useful.

I don't have any specific tool to recommend as I've rarely seen them produce the right results, but I include emulators here for completeness.

Outsource Selenium
If you have a suite of Selenium tests but do not have the infrastructure or experience in house to get a grid up and running then you might find services like Sauce Labs (http://saucelabs.com/), Browserstack and Testing Bot (http://testingbot.com/) quite useful.


Conclusion
I’ve not touched on different operating systems or mobile devices but as you can see there are a number of ways to approach the cross browser issues.

Cross browser support may be your biggest pain-point, especially with the rapid rate of release that most of the browser companies are now adopting.

In my experience a nice selection of a number of the above works well.

Useful Hint

Some browsers come with compatibility modes, which allow you to emulate different versions.

Some browsers, like Chrome, have developer tools for rendering pages in different browser and Operating Systems.

Useful Links

Compatibility Mode page on Wikipedia - http://en.wikipedia.org/wiki/Compatibility_mode_(software)

Sauce Labs - http://saucelabs.com/

Testing Bot - http://testingbot.com/

Browserstack - https://www.browserstack.com/

Browsers and Rendering Engines - http://en.wikipedia.org/wiki/Web_browser_engine

W3 HTML checker - http://validator.w3.org/

Selenium - http://seleniumhq.org/



If you’re interested in a career in Software Testing then check out my book Remaining Relevant And Employable (Tester’s Edition) - it’s packed full of ideas about writing good CVs, communicating your value to employers and doing well in an interview.