There are a number of reasons that tracking referrals to your website from third-party websites can be problematic if you attempt to do so with server-side code.

This is a technical article intended for insurance software developers but the problem described affects all of us who work with affiliates and resellers and attempt to track referrals server-side.

Let me break it down so you see the process, and the flaws.

Before I start, let me just make it clear: REFERRER and REFERER are alternative spellings of the same word, I think probably UK and US respectively. Since I’m in the UK right now I’ll be sticking to ‘referrer’ most of the time but forgive me if I lapse and switch

The tracking process we usually use to capture referrer is as follows:

1. The affiliate (e.g. an aggregator like CompareTheMarket or MoneySupermarket, or an independent broker or authorised representative, or even a shopping affiliate like Froogle, Kelkoo, SiteRunner, Ciao) publishes a link with their ID and our ID on their website.
2. A visitor clicks on the link with the intention to buy from your site.
3. The link takes the visitor back to the site of the affiliate that logs the visit, referrer, etc. and sets a cookie in the visitor’s browser.
4. The visitor is redirected to your site, where you log the new visitor as coming from the affiliate either by reading an ID in the querystring or by reading the *referrer value* (HTTP_REFERER) server side.
5. The visitor proceeds to shop while we track them around the site with an id binding them to the original affiliate.
6. The visitor either leaves or places an order, in which case we record the affiliate details in the order/quote/policy record.

How affiliates usually track us:

Since affiliates don’t trust their merchants to report this information back to them, there are two very similar methods they ask us to use to help them to capture more information:

1. They give us a snippet of code to be embedded in the source code of the ‘Thank you’ page that connects to the affiliate site notifying them the details of the successful transaction.
2. They give us an IMG tag to embed instead, linking to a server-side script on their site.

Note that this is not reliable, since we could easily randomize or fake the reporting of click-through. They are very much at our mercy, just as – in the other direction – we are at theirs, as you will see…

The industry-wide problem with all of this is:

There is absolutely no way to guarantee the referral data, for a number of reasons. In some respects HTTP itself is a flawed protocol in this context: namely the HTTP_REFERER server variable that we all use is not reliable – often empty or containing fraudulent data – and this server variable is the only way to capture the origin of a click event required to credit an affiliate referrer.

It can be empty because:

1. Some affiliates use a meta-refresh to redirect people to the merchant site (a-la Kelkoo), allowing them to deliver an extra banner ad, but often ditching the all important referrer info.
2. Some browsers don’t pass this variable at all.
3. Some browsers discard this variable if the site is opened in a new window, or if you drag a link and drop it on another window, or use a desktop/start menu shortcut, bookmark, etc.
4. Some people disable it manually, for privacy (usually outside of a single domain).
5. Some firewalls block it (e.g. Norton, usually outside of a single domain).
6. Many e-mail programs like Outlook filter it.
7. Most download managers don’t send it.

Furthermore, in order to track the referrer as people move around the site it has to be stored in the session (usually the only solution for storing information about someone who hasn’t yet registered or logged-in, but a very volatile mechanism which can easily expire), or a cookie (which people often disable, firewalls block etc.).

Also, if people enter the domain name of the site manually or set a bookmark once they found us on the affiliate site and then return later, there is nothing to track or detect.

Are there any workarounds?

Unfortunately the entire system is based on trust between affiliate and merchant, and worse still, trust of the browser/software/network/pc of the visitor, all of which is unreliable, and data – however stored – can be faked, since there is not actual authentication between the affiliate and us.

However, there is a dim light at the end of the tunnel, if you squint and look hard enough…

• Javascript, client based tracking like Google Analytics, Mint or BLVDStatus do a good job of telling you where visitors are coming from.  They’re not so good at showing clearly when visitors who originally came from a referrer return at a later date, because of the ‘last click wins behaviour‘.  To get around this when using Google Analytics, you need to override the last click wins behaviour.  You may also like to track all of your outgoing links.
• Alternatively or in addition, affiliates could provide authentication by SSL or a similar mechanism to prove to us that they are who they say they are and visa-versa, making the whole process of delivering customers a secure transaction.  I’ve not seen anything like this in practice but I’m sure it goes on.  If anyone has any examples, let me know!
• Often the only practical short-term solution is to create a different URL for each affiliate, which forces them to visit a different link to their counterparts, (e.g. http://www.yoursite.com/froogle ), but this can lead to resubmissions by fraudulent affiliates, since there is no way for us to stop it.

In my experience, use every method you can and compare the data coming from each approach which will help you highlight any grey areas or holes, and pinpoint fraudulent activity by any web partners you are working with.

As ever, your comments are very welcome!

As always, based on any comments received I will update this article so it remains authoritative.

Do you think that web based insurance software is the best thing since sliced bread? Is it the right fit for the insurance industry?

I would say so, but then I’ve always been a fan of web based, ever since Yahoo Mail (1998?). In any case the advantages are well known…

1. Never install anything
2. Always up-to-date
3. Safe & secure
4. Backed-up
5. Access from anywhere
6. Works on every operating system/cellphone
7. No IT staff are required
8. No servers required

But for many insurance agencies, these advantages are not plain. The reason is a simple one: many insurance broking companies have not had a chance yet to use web based software for work, and some not even at home. Describing the advantages to the uninitiated can be a hurdle.

What do you think? Is web based insurance agency software the panacea to deal with the distributed nature of our insurance operations, or is it all hype?

As the economic crisis begins to lift and people start to buy in to software again, we should decide!

Well? Does it work for you or not?

After many, many months of hard work, love, sweat, pain and great code(!), we’re pleased to unveil the new SchemeServe, version 4.4.

There are over 100 new features and improvements to look forward to in this release, and a list of the main features is here.  The most important new ones include:

Split Risk Rating Matrices now editable by you.

Edit all types of auto-rating rules when cover is first held, at renewal and MTA (e.g. automatically charge more for specific coverage based on policy CLR for specific claim types).

Create granular rules for auto-generation of static documents.

…alongside dynamic ones.  With this mix and match approach you can use dynamic documents when you need special fields, and PDF static docs can be attached at other times, for instance for summary of cover documents.

Use dynamic text in your endorsement library.

…and control rules for automatic endorsement creation (e.g. conditions, terms and memoranda added to documentation generated).

Improved Scheme Editor.

Launch new schemes even more easily in-house, making it a snap to mix direct and via-subbroker schemes, including with differing payment options (or if you prefer, take payment in-house).

Even more graphs and reports.

Take a look through the new reporting and graphs section, split risk groups in particular has meant lots of changes.

Even better system performance and speed!

There are a whole host of optimisations in there again this time, so you should find the product another order of magnitude faster again.

Referral Authority levels.

Optionally only allow referals to be cleared by staff with the correct authority level for that specific type of referral).

Policy Allocation.

Set rules to allocate users to manage only specific schemes and/or accounts, with overrides for staff absence.

More payment options.

If taking payments online with SchemeServe, now use Sage Pay (formerly ProtX) or PayPal.

Improved Direct Scheme Support.

Get Quote process for direct schemes now easier than ever, making it a snap for clients to get quotes and go-oncover etc.

Improved Questions Editor.

More powerful questions editor with more options to trigger new questions and pages, specify more field types, legal ranges, risk groups and modifiers.

New Import/Export for rating matrices.

Export the underlying rating matrices for any scheme to Excel, edit, then re-import.

New look Admin page.

New icons and Admin panel with improved accessibility to core features and navigation shortcuts.

—

As always, existing SchemeServe users need do nothing – you will automatically be given access to the new version of SchemeServe at the same URL, at no charge.

To celebrate the release of the new version we’ll be offering £200 cash back to our first 100 sign-ups here.

As always, we offer a no obligation 30-day free trial on all our plans, so check it out.

Let us know what you think!

Something that has continually amazed me the past few years is the level of Technofear in the insurance industry, or, as I’ve decided to call it right now: “Ristechnophobia”. (Think it will catch on?).

Edit: Thanks to Reinsurance Girl for the example above.

It’s true that most of us who work in the insurance industry have seen countless insurance software programs which simply don’t work; and we’ve seen this again and again and again, with every software release. But despite what some of us may think, in other industries software doesn’t get such a bad wrap. So what’s going on? Are we right: is it really time to string up the techies and go back to paper, or are we missing something?

Given the massive complexity of most underwriting operations it’s no surprise that software developers struggle to quickly put together meaningful – and adaptable – solutions. Couple this with the ever-changing landscape and personality of any individual business and you have a recipe for sure-fire flaky and unfinished software implementations.

In this environment, with little or no solutions that match your distinctive trading style, and with nothing certain in this world(!), you’d be forgiven for having not taken the software plunge; abandoning tried and tested underwriting methods you know and trust for something you know can’t work.

But we know there are some websites out there providing a rich user experience throughout the insurance life-cycle without so much as a hickup. So how do they do it?

Well some of them have spent millions, and many man-years of research and development time on bespoke in-house software – but not all of them. The truth is that the insurance software landscape is changing, and finally finding it’s feet.

There are great providers out there but you have to be really picky in comparing them. Be sure to ask your provider:

1. How quickly and easily can i edit:
   • The questions I ask my customers and the answers allowed
   • Auto referral and endorsement rules
   • Auto rating rules throughout life-cycle
   • The documents generated and sent to clients and/or sub-brokers
   • My website
   • The commissions and fees I pay at each stage of the policy life-cycle?
2. Can I do all this for free without in-house programmers, risk, and extra costs?
3. How quickly can I get graphs of my data or export to Excel or PDF?
4. Is everything real-time, online and seamless?
5. Can I mix Sub-broker and Direct schemes with multiple currencies in the same software without extra upgrades?
6. How secure and backed-up are the servers you use? Do you maintain them for free or do I need my own IT staff?
7. Does the software run on every operating system and Smart Phone? Windows? Mac? Linux? iPhone? Blackberry?
8. How long has the software version I am buying been in production and active use?

And most importantly:

9.   Can I see all of this working 100% with no obligation before I sign-up?

If the answer to all of these answers is positive, you know you’re dealing with the right team. The next generation of insurance software providers are finally here – really! – and they bring with them a customer experience which is hard to rival with ink and quill.

So what are you waiting for, there’s nothing to fear, you just might need to sift for a long time till you get the gold!

I hope this has been useful. Let me know in the comments if you need any advice, I’m always on hand to help.