Performance data collection using BrowserMob Proxy and Selenium
RSSS

BrowserMob Proxy with Selenium

BrowserMob Proxy is a simple utility that makes it easy to capture performance data from browsers, typically written using automation toolkits such as Selenium and Watir.

BrowserMob Proxy can capture performance data for web apps (via the HAR format). Used to collect the performance data from the client side.

What is a HAR format ?

HAR(HTTP Archive) is a online tool visualizing. HAR is produced by the HTTP tracking tools, these files captures the details of the client/server communication and this can be used for analysis like Page Load Performance.

How the HAR files will look ?

proxy is programmatically controlled via a REST interface or by being embedded directly inside Java-based programs and unit tests.It can control HTTP trafic for,

  • blacklisting and whitelisting certain URL patterns
  • simulating various bandwidth and latency
  • remapping DNS lookups
  • flushing DNS caching
  • controlling DNS and request timeouts
  • automatic BASIC authorization

How to Use BrowserMob Proxy?
Step 1: Download the BrowserMob Proxy

Step 2: Start the proxy by running `browsermob-proxy` on Mac/*Nix based machines and `browsermob-proxy.bat`for Windows in the bin directory
On Mac it should be like sh browsermob-proxy -port 9090

Step 3: You can create a New Java Project in Eclipse as documented here

Step 4: You use the below mentioned Java code as an example,

package com.selenium.performancetest;

import java.io.FileOutputStream;

import org.browsermob.core.har.Har;
import org.browsermob.proxy.ProxyServer;
import org.openqa.selenium.By;
import org.openqa.selenium.Proxy;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.firefox.FirefoxDriver;
import org.openqa.selenium.remote.CapabilityType;
import org.openqa.selenium.remote.DesiredCapabilities;

public class PerfTest {

	public static void main(String[] args) throws Exception {

		String strFilePath = "";

		// start the proxy
	    ProxyServer server = new ProxyServer(4444);
	    server.start();
	    //captures the moouse movements and navigations
	    server.setCaptureHeaders(true);
        server.setCaptureContent(true);

	    // get the Selenium proxy object
	    Proxy proxy = server.seleniumProxy();

	    // configure it as a desired capability
	    DesiredCapabilities capabilities = new DesiredCapabilities();
	    capabilities.setCapability(CapabilityType.PROXY, proxy);

	    // start the browser up
	    WebDriver driver = new FirefoxDriver(capabilities);

	    // create a new HAR with the label "apple.com"
	    server.newHar("assertselenium.com");

	    // open yahoo.com
	    driver.get("http://assertselenium.com");

	    driver.get("http://assertselenium.com/2012/10/30/transformation-from-manual-tester-to-a-selenium-webdriver-automation-specialist/");

	    // get the HAR data
        Har har = server.getHar();
        FileOutputStream fos = new FileOutputStream(strFilePath);
        har.writeTo(fos);
        server.stop();
		driver.quit();

	}

}

You will have to add all the jar files that are in lib folder in the downloaded package to this Eclipse Project Build Path

Step 5: Run the script as Right click and Run As Java Application

Step 6: You can view the Eclipse console for run time logs generated

Step 7: Refresh your project, now you can see a folder HAR files is generated inside the HARFiles folder

You can now view the Har file for performance data here

Now we can view the performance data for http://assertselenium.com and it looks like the below screenshot,

Hope this post helps in setting up the BrowserMob Proxy.

Please share and leave your comments if any.

 

The same implementation in .NET C#

using AutomatedTester.BrowserMob.HAR;
using OpenQA.Selenium;
using OpenQA.Selenium.Firefox;

namespace BrowserMob.PerfTest
{
    public class ExampleClass
    {
        public void ExampleUse()
        {
            // Supply the path to the Browsermob Proxy batch file
            Server server = new Server(@"C:BrowserMobProxybinbrowsermob-proxy.bat");
            server.Start();

            Client client = server.CreateProxy();
            client.NewHar("assertselenium");

            var seleniumProxy = new Proxy { HttpProxy = client.SeleniumProxy };
            var profile = new FirefoxProfile();

            profile.SetProxyPreferences(seleniumProxy);
            // Navigate to the page to retrieve performance stats for
            IWebDriver driver = new FirefoxDriver(profile);
            driver.Navigate().GoToUrl("http://assertselenium.com");

            // Get the performance stats
            HarResult harData = client.GetHar();

            // Do whatever you want with the metrics here. Easy to persist
            // out to a data store for ongoing metrics over time.

            driver.Quit();
            client.Close();
            server.Stop();
        }
    }
}

The Python Implementation

from browsermobproxy import Server
                server = Server("path/to/browsermob-proxy")
                server.start()
                proxy = server.create_proxy()

                from selenium import webdriver
                profile  = webdriver.FirefoxProfile()
                profile.set_proxy(proxy.selenium_proxy())
                driver = webdriver.Firefox(firefox_profile=profile)

                proxy.new_har("google")
                driver.get("http://www.google.co.uk")
                proxy.har # returns a HAR JSON blob

                proxy.stop()
                driver.quit()

24 thoughts on “Performance data collection using BrowserMob Proxy and Selenium

  1. Manish
    November 2, 2012 at 12:34 pm

    Nice Post.

    1. Ramesh Jhajharia
      November 3, 2012 at 10:51 am

      Nice Post and nice blog :)

  2. Ramesh Jhajharia
    November 3, 2012 at 10:51 am

    Can you please post for .NET C# also ?

    1. November 3, 2012 at 11:33 am

      Thanks Ramesh for your support. I C# I havent tried yet, but will look into it and post it here. Keep visiting

      1. Ramesh Jhajharia
        November 6, 2012 at 6:38 pm

        I am using this with ASP .Net C#.
        But I am facing an issue that I am not able to remap host or you can say the browser mob proxy is not reading my host file .
        I found that calls to remap a host is available in java but not in Asp .net c#.
        Please help me if you have any idea on this.

        Thanks,
        Ramesh Jhajharia

  3. November 6, 2012 at 6:45 pm

    Hey Ramesh,
    I saw your post in the Google Groups as well. I have just started looking at the .NET bindings of BrowserMob Proxy. I will post the code here for the same If I get to work them properly.

    Havent fully explored all the API’s in Java itself. Let me try to help you out soon.

    Thanks
    Manoj

    1. November 6, 2012 at 11:57 pm
  4. November 23, 2012 at 1:28 pm

    great article. I was able to implement the browsermob proxy using C# however what i’m finding it difficult to understand is how do i go about implementing it for my whole project? I mean i have a login page, and when the login button is being clicked i’m being redirected to the same page instead of having the app proceeding to the next screen. does this mean that browsermob proxy has to be implemented each time i want to monitor traffic?

    1. November 26, 2012 at 9:38 am

      It shouldn’t be like that way. As far as I have seen. I was able to got to a website click on multiple tab and generate the performance data. For e.g: http://www.apple.com in that Clicking on all tabs like Mac, iPhone, Ipad etc…
      You can do.

    1. January 14, 2013 at 4:20 pm

      Thanks for the TracBack’s

  5. vikas
    February 14, 2013 at 5:46 pm

    Hi monoj, when i am trying to run https://www.gmail.com using above code. i am getting some proxy issues. Firefox is opening and giving a message stating: Firefox is configured to use a proxy server that is refusing connections.
    Check the proxy settings to make sure that they are correct.
    Contact your network administrator to make sure the proxy server is
    working.
    Do i need to change any settings to make it work? please let me know

    1. February 14, 2013 at 6:31 pm

      That is completely dependent on the network that you use. I think you are under a firewall protected environment, where you are using your AD credentials to access the webpages ? If yes, you can configure Firefox with proxy or add the webpage under exemption list

  6. vikas
    February 14, 2013 at 5:56 pm

    Forgot to mention my firefox version is 18.0.2 and i am using latest browsermob jars: browsermob-proxy-2.0-beta-6

  7. bianca pirte
    February 27, 2013 at 12:38 am

    using python on windows:
    for
    from browsermobproxy.server import Server
    from selenium import webdriver

    server = Server(“Libbrowsermobproxybin”)
    server.start()
    proxy = server.create_proxy()

    profile = webdriver.FirefoxProfile()
    profile.set_proxy(proxy.selenium_proxy())
    driver = webdriver.Firefox(firefox_profile=profile)

    proxy.new_har(“google”)
    driver.get(“http://www.google.co.uk”)
    proxy.har # returns a HAR JSON blob

    proxy.stop()
    driver.quit()

    i get

    from browsermobproxy.server import Server
    ImportError: No module named server

    even if the import is the one Eclipse suggested for Server

    i have:
    python 2.7
    selenium 2.27
    browsermob 0.20
    and I run the test from eclipse

    any suggestions?

    thanks,
    -Bianca

    1. February 27, 2013 at 2:10 am

      is that the latest browsermob version ?

  8. Melvin Lobo
    April 5, 2013 at 2:49 am

    Hi All,

    My main goal is to create the har files.

    I tried using the above code and works fine but if I test a site which requires login cedentials i,e session id, it fails past the login screen. Below is the code snippet
    driver.get(_sBaseURL + “/Login/Login.aspx”);
    driver.findElement(By.id(“User.UserName”)).clear();
    driver.findElement(By.id(“User.UserName”)).sendKeys(new String[]{“jmathe02″});
    driver.findElement(By.id(“User.Password”)).clear();
    driver.findElement(By.id(“User.Password”)).sendKeys(new String[]{“cat1dog”});
    driver.findElement(By.id(“btnLogin”)).click();

    driver.findElement(By.xpath(“//table[@id='tblAgenciesList']/tbody/tr[13]/td”)).click();
    ===============
    However if I change to proxy to use ProxyType.Direct it works but then does not create the har entires.

    Was wondering if some could help.

    Melvin

  9. Mike
    November 13, 2013 at 9:22 pm

    I am not seeing the HARFiles folder: Althought everything else seems to work perfectly. Anyone else had this issue?

  10. Hemanth
    November 18, 2013 at 3:09 pm

    I have one doubt,
    First of all its a great article and the har file was generated but when i am placing it in Har Viewer it is telling it is not a valid har when I uncheck validate Har it is showing the har file data.
    Am i missing anything in generating the har file

  11. Jessica
    March 18, 2014 at 3:10 pm

    Hi, any chance of the code being supplied in PHP?

  12. kalyani
    March 21, 2014 at 5:17 pm

    I couldn’t get HAR Files generated in the target folder or created in any folder within the project.

    For StrFilePath I had to enter a path where “fos” can be saved as it complains about fileNotFound.
    fos doesn’t give much details neither HAR Files folder is generated after running the script.

    Suggestions please.

    Thanks
    Kalyani.

  13. Madhur
    April 14, 2014 at 10:02 pm

    Hi,

    Thanks for your post. It is really helpful. I am trying to capture the HAR file for various browsers. Firefox and Chrome work. But Safari does not. I get an empty har file. I have the following code:
    capabilities = DesiredCapabilities.SAFARI
    capabilities['proxy'] = proxy.proxy
    capabilities['platform'] = ‘MAC’
    capabilities['version'] = ’7.0′
    proxy.new_har(“SafariApple2″)
    browser2 = webdriver.Safari(desired_capabilities=capabilities)
    browser2.get(‘http://www.apple.com/iphone-5s’)
    with open(“/Users/madhuranand/localhost-dev-overlay/perf/iphone-5s_s.json”, ‘w’) as outfile3:
    json.dump(proxy.har, outfile3)

    Can you please give me some pointers as to what i am missing.

    thanks,
    Madhur

  14. Goldking
    May 21, 2014 at 7:21 am

    Hi when i run this code, it’s create blank HAR file. i use to lunch with IE8.

Leave a Reply

Your email address will not be published. Required fields are marked *