Overview
Recently I've got the request to implement authenticated proxy support for our product test framework. The problem is that recent browsers do not allow the widely popular
http://username:password@proxy.example.com
syntax and still ask you to manually enter credentials.
The next problem is that Selenium does not let you interact with these basic auth dialogs
[1][2]. So how should one go about this?
Chrome allows you to do this with a custom extension that you can insert with selenium/watir.
One additional complication is that we can use a different proxy server each time. Thus extension needs to be packaged on the fly.
Chrome extension
This is the proxy extension as I use it. See it as an example for whatever you'll be trying to do. It consists of only 2 files you can put in an empty directory.
manifest.json
{
"version": "0.0.1",
"manifest_version": 2,
"name": "Authenticated Proxy",
"permissions": [
"<all_urls>",
"proxy",
"unlimitedStorage",
"webRequest",
"webRequestBlocking",
"storage",
"tabs"
],
"background": {
"scripts": ["background.js"]
},
"minimum_chrome_version":"23.0.0"
}
background.js.erb
var config = {
mode: "fixed_servers",
rules: {
singleProxy: {
scheme: "<%= proxy_proto %>",
host: "<%= proxy_host %>",
port: parseInt(<%= proxy_port %>)
},
bypassList: <%= proxy_bypass.split(/[ ,]/).delete_if(&:empty?).to_json %>
}
};
chrome.proxy.settings.set({value: config, scope: "regular"}, function() {});
function callbackFn(details) {
return {
authCredentials: {
username: "<%= proxy_user %>",
password: "<%= proxy_pass %>"
}
};
}
chrome.webRequest.onAuthRequired.addListener(
callbackFn,
{urls: ["<all_urls>"]},
['blocking']
);
As you can see on the web site, Protocol Buffers is a method of serializing structured data. For CRX3 (unlike CRX2) format it is part of the required header for the extension.
I decided to use
ruby-protobuf project instead of the google ruby library because it appeared well maintained and pure ruby. I assume google ruby library will work well too.
The Packager
A CRX v3 file would consist of:
- Cr24 - ASCII 8bit magic string
- 3 - protocol version in unsigned 32bit little endian
- header length in bytes in unsigned 32bit little endian
- header itself - the protobuf serialized object
- crx3.proto - the protobuf descriptor
- as a rule of thumb
- all lengths inside are given as unsigned 32bit little-endian integers
- key files are inserted in PKCS#8 binary encoding (Ruby's
key.to_der
worked fine)
- ZIP archive of the extension files
Generating protobuf stub
We need to install Google protobuf compiler
protoc
. You can save the protocol file in a directory where you want stub to live in. Then generate by
protoc --plugin=protoc-gen-ruby-protobuf=`ls ~/bin/protoc-gen-ruby` --ruby-protobuf_out=./ path/chrome_crx3/crx3.proto
This will create a file
crx3.pb.rb
in the same directory as the protocol file. All you need is to
require 'path/crx3.pb.rb'
wherever you want to use that format.
Actual packager
At this point the packager is straightforward to implement. Pasting the whole logic here.
We have one
::zip
method to generate a ZIP archive in memory. If an ERB binding is provided by caller, any
.erb
files are processed. That's how the above
background.js.erb
works.
The method
::header_v3_extension
generates the signature and constructs the whole file header.
Finally
::pack_extension
just glues the two methods above to generate the final extension.
chrome_extension.rb
require 'erb'
require 'find'
require 'openssl'
require 'zip'
require_relative 'resource/chrome_crx3/crx3.pb.rb'
class ChromeExtension
def self.gen_rsa_key(len=2048)
OpenSSL::PKey::RSA.generate(len)
end
# @note file format spec pointers:
# https://groups.google.com/a/chromium.org/d/msgid/chromium-extensions/977b9b99-2bb9-476b-992f-97a3e37bf20c%40chromium.org
def self.header_v3_extension(data, key: nil)
key ||= gen_rsa_key()
digest = OpenSSL::Digest.new('sha256')
signed_data = Crx_file::SignedData.new
signed_data.crx_id = digest.digest(key.public_key.to_der)[0...16]
signed_data = signed_data.encode
signature_data = String.new(encoding: "ASCII-8BIT")
signature_data << "CRX3 SignedData\00"
signature_data << [ signed_data.size ].pack("V")
signature_data << signed_data
signature_data << data
signature = key.sign(digest, signature_data)
proof = Crx_file::AsymmetricKeyProof.new
proof.public_key = key.public_key.to_der
proof.signature = signature
header_struct = Crx_file::CrxFileHeader.new
header_struct.sha256_with_rsa = [proof]
header_struct.signed_header_data = signed_data
header_struct = header_struct.encode
header = String.new(encoding: "ASCII-8BIT")
header << "Cr24"
header << [ 3 ].pack("V") # version
header << [ header_struct.size ].pack("V")
header << header_struct
return header
end
# @param file [String] to write result to
# @param dir [String] to read extension from
# @param key [OpenSSL::PKey]
# @param crxv [String] version of CRX file to create
# @param erb_binding [Binding] optional if you want to process ERB files
# @return undefined
def self.pack_extension(file:, dir:, key: nil, crxv: "v3", erb_binding: nil)
zip = zip(dir: dir, erb_binding: erb_binding)
File.open(file, 'wb') do |io|
io.write self.send(:"header_#{crxv}_extension", zip, key: key)
io.write zip
end
end
# @param dir [String] to read extension from
# @param erb_binding [Binding] optional if you want to process ERB files
# @return [String] the zip file content
def self.zip(dir:, erb_binding: nil)
dir_prefix_len = dir.end_with?("/") ? dir.length : dir.length + 1
zip = StringIO.new
zip.set_encoding "ASCII-8BIT"
Zip::OutputStream::write_buffer(zip) do |zio|
Find.find(dir) do |file|
if File.file? file
if erb_binding && file.end_with?(".erb")
zio.put_next_entry(file[dir_prefix_len...-4])
erb = ERB.new(File.read file)
erb.location = file
zio.write(erb.result(erb_binding))
Kernel.puts erb.result(erb_binding)
else
zio.put_next_entry(file[dir_prefix_len..-1])
zio.write(File.read(file))
end
end
end
end
return zip.string
end
end
Using the packager
Packing the extension is as simple as:
require 'chrome_extension'
ChromeExtension.pack_extension(file: "/path/of/target/extension.crx", dir: "/path/of/proxy/extension")
Using the extension with Watir
proxy_proto, proxy_user, proxy_pass, proxy_host, proxy_port = <...>
chrome_caps = Selenium::WebDriver::Remote::Capabilities.chrome()
chrome_caps.proxy = Selenium::WebDriver::Proxy.new({http: "#{proxy_proto}://#{proxy_host}:#{proxy_port}", :ssl => "#{proxy_proto}://#{proxy_host}:#{proxy_port}")
# there is a bug in Watir where providing an object here results in an error
# options = Selenium::WebDriver::Chrome::Options.new
# options.add_extension proxy_chrome_ext_file if proxy_chrome_ext_file
options = {}
options[:extensions] = [proxy_chrome_ext_file] if proxy_chrome_ext_file
browser = Watir::Browser.new :chrome, desired_capabilities: chrome_caps, switches: chrome_switches, options: options
Bonus content - CRX2 method
# @note original crx2 format description https://web.archive.org/web/20180114090616/https://developer.chrome.com/extensions/crx
def self.header_v2_extension(data, key: nil)
key ||= gen_rsa_key()
digest = OpenSSL::Digest.new('sha1')
header = String.new(encoding: "ASCII-8BIT")
# it is exactly same signature as `ssh_do_sign(data)` from net/ssh does
signature = key.sign(digest, data)
signature_length = signature.length
pubkey_length = key.public_key.to_der.length
header << "Cr24"
header << [ 2 ].pack("V") # version
header << [ pubkey_length ].pack("V")
header << [ signature_length ].pack("V")
header << key.public_key.to_der
header << signature
return header
end
Credits