Wandering around the supermarket

Imagine, if you will, wandering around your local Tesco or Sainsbury’s and every time you stopped to look at something on the shelf a little bit of data was stored that recorded what you had looked at, and referenced that against your Clubcard or your Nectar Card. Perhaps the card has a small chip inside that the supermarket tracks around the store, so they can tell what aisles you have walked down, where you stop, what direction you are facing when you stop and so on. Pretty much every behaviour you have in-store, recorded and logged against your personal Clubcard or Nectar Card account.

Now I’m not suggesting that either supermarket goes this far in their tracking. But how would you feel if they did? Would you change your shopping habits? Would you feel more drawn to supermarkets that don’t have such loyalty cards? Would you stop using your loyalty card, perhaps simply leaving it at home? Or would you be happy with Tesco or Sainsbury’s knowing what products you have looked at but not bought, what magazines you had picked off the shelf to browse through, what medicines you had looked at?

Loyalty cards

These loyalty card schemes already track what you actually buy. Tesco and Sainsbury can target their direct mail advertising to you based on your buying habits. Those money off vouchers that arrive in the post are not random – that is behavioural advertising.

You may be comfortable with that, and that’s fine. In the confines of a supermarket’s stock range, there isn’t a huge amount to be embarrassed about buying. But what if this data were to be shared to other advertisers, or what if other buying data from elsewhere were linked to your Clubcard or Nectar Card account? In fact that is already the case for the Nectar card, as you can use this across different outlets, including online only outlets like Amazon, eBay, and Expedia. So in reality, you can give Sainsbury’s access to your online buying habits at these stores.

Data about your buying and shopping habits is already rife within business and influences marketing that you receive. I’m willing to bet very few Nectar points collectors have ever considered what data Sainsbury’s might hold about them.

Of course, there is a trade-off. Earn Nectar points, spend them on travel, food, electronics or groceries. Sainsbury’s, and the other participating businesses, are buying your data from you.

Wandering around online

Online, websites track users through websites, often (but not always) using cookies. The purpose of the cookies might vary – they are used to ensure that a website remembers that you have logged-in, or that you have added something to your shopping cart, and these functions would not work at all well without cookies. In short, cookies are primarily used so that a website knows that you are the same person that just viewed a different page on the site a few seconds ago.

Cookies can also be used to store data about you, such as a password, but this is so insecure that it generally isn’t done, and certainly not by reputable and security conscious businesses, although they are much more likely to set a cookie to say that you have logged in, and to keep you logged in persistently (i.e. for much longer than just this visit – Facebook is an obvious utiliser of this policy).

Cookies are not loyalty cards

Cookies themselves are not equivalent to Nectar Cards and Clubcards. Cookies generally do not know who YOU are. They don’t know your address, or how many kids you have. They just link up data about you.

However, what cookies do is enable sites to link the bits of data they do have about you together, if you enable this. Log into a shopping site with Facebook, and it is likely you are giving that shop data about what you like, who your favourite music acts are, what TV shows you like and so on.

Some cookies are set and travel with you across different websites enabling tracking services for ad networks such as Double Click and Specific Media to know what kinds of websites you look at. Not just what you buy. Not just what you choose to give data to, but what you simply look at.

But who is being tracked?

Furthermore, they’re not tracking you. They’re tracking the website browser you use. What if you share your computer with someone else? Deleting cookies when you are done doesn’t always help. Some advertising networks have chosen to create cookies that are virtually undeleteable – often called zombie cookies, as they seem to rise from the dead and recreate themselves.

What if you share a PC with your kids, but in the evening, when they have gone to bed, you like to watch horror films or violent thrillers through an online film subscription service. What if you participate in forums that are about, say, tattoos and body art that by definition may contain nudity, even though it isn’t of a sexual nature. Heck, what if you actually do watch porn in your own free time? All the while, you might be being tracked by ad servers that might choose to serve ads based on your browsing history.

Horror, or Messi?

Now clearly, the ad networks that show ads on the Disney Channel website will be filtering out inappropriate ads for you kids, but what about news sites, forums, social media, video sharing sites, games sites and so on? Can you be sure that ad network X that knows you like horror films doesn’t trigger a horror film pre-roll trailer on a video site that your kids is using to watch that great Lionel Messi goal from the other week?

This is the kind of thinking that has driven the EU Cookie Directive, and in those terms it might seem quite sensible.

In fact, I’d go so far as to say that most people are uncomfortable with the idea of their behaviour being tracked, even if it didn’t put their kids at risk of seeing blood or boobs.

It’s like 10,000 spoons, only actually ironic

There was a veritable uproar on Twitter at the beginning of April, when news broke that there were plans to allow GCHQ access to emails and social media in real time. Many of those posting sarcastic “#telldaveeverything” tweets were people in the digital marketing industry (see this, for example). Many of those same people are up in arms about the EU Cookie Directive, or the fact the Google no longer passes all keyword data through to Analytics. I’ll have my irony with chips, thank you :-).

The point is that we are in the midst of the biggest shake up of liberty in Western Europe, perhaps since the French revolution. This unchartered libertarian ground is terrifying those who might control our data. The legal ramifications might be dramatic. That is why even Bing will leap to Google’s defence about the ‘not provided’ issue (see around about the 36 minute mark).

As individuals, we want to be free to do what we want to do, look at what we want to look at, and listen to what we want to listen to, unimpeded by those in power. We are uncomfortable with CCTV, although we see a crime fighting benefit. We are terrified of identity theft. We are concerned about what inappropriate materials our kids might have access to.

But as businesses, we strive for more data about our customers – we want to know what they like, what they don’t and what makes them want to buy.

Cookies enable us to track users and give us that data, but users are understandably uncomfortable with the idea of being tracked.

It is now less than one month until the end of the twelve month period that the ICO has given UK businesses to comply the EU Cookie Directive, which came into force of 25th May 2011.

Realistically, if you comply with the letter of the legislation, how many of your customers do you anticipate happily opting in to being tracked? I don’t expect it will be many.

Perhaps there is a lesson to learn from the Clubcard and Nectar points – as a marketing industry we have to give Internet users something back as a reward for opting in.

What the Internet is really for

And on that note, as a reward for reading this far, here’s Avenue Q explaining why some people might not be so keen on being tracked.

Intended as a site for the Scottish people, DirectScot builds on years of work and learning from governments’ worldwide as they seek to establish new and better ways to deliver content and services to citizens.

Launched in December 2011 as a mobile-optimised ‘experimental prototype’, DirectScot aims to demonstrate world-class search & discovery capabilities – in this phase seamlessly surfacing content and services from both Directgov and a small number of local authorities within Scotland.

DirectScot is built to service the needs the citizen – from information on refuse collection to council tax banding – with the ambition to become a one stop shop for those wanting to locate and access public service information in Scotland. As such, the current version of the site addresses some 700 ‘needs’ and sets about making them as findable as possible to the citizen. And overtime it is expected that the site will become a central resource for accessing Scottish public sector information and transactional services.

In this way, DirectScot is Scotland’s equivalent of GOV.UK – the beta site launched by the UK’s Government Digital Service in early February 2012.

Managed and sponsored by the Scottish Government on behalf of the Scottish public sector as a whole, DirectScot closely follows and builds upon some of the design principles espoused and adopted by the Government Digital Service, while adopting new ideas and localisations specifically for the Scottish context.

As such, following a public consultation, the Scottish Government will now review and publish its findings which will, in turn, inform the next development phase of this exciting and innovative project for the Scottish people.

People want to become more active participants in the healthcare and services they receive. This is not just about making services better for the engaged few, but by delivering services to the unengaged many.

The single largest untapped resource in the NHS is the patient and that patient is more than ready to get involved.

DIY Health
This general trend can be referred to as DIY Health and it relies on a confluence of factors all of which are already in place and gaining traction extremely quickly:

• Mobile technology used to collate data
• Apps to deliver services
• Social networks to share experiences
• Crowd-sourcing to collate and rate services
• An ability to process vast quantities of (realtime) data

From the NHS Scotland eHealth Strategy 2011 – 2017 we can see a recognition of some of these changes:

• Improved broadband coverage across Scotland is allowing greater and faster internet access from home and smartphones
• For young people in particular, the internet and mobile phones are fast becoming the primary means by which they can be communicated with and through which they expect to access services
• Society is increasingly comfortable with self-service models of interaction

Paradigm shift not Channel Shift
On the surface, providing mobile access to an NHS portal which connects me to my patient record and allows me to access information and book services would address the trend identified in the strategy document. But this approach would miss the point entirely.
This channel shift would offer benefits but it is still reactive, relies on one-to-one service delivery and enshrines the sticky issue of accessing a patient record.

People are using devices / apps / smartphones to build their own Health Graph richer and far more complete than their existing patient record. That Health Graph may not (yet) contain the list of medical facts and incidents the NHS patient record contains but it will hold vast swathes of data that provide the means by which I can receive warnings and recommendations, support and advice, nudges and reminders – all of which provides the foundation for DIY Health Services.

What we now need are precisely those services which operate on this data, and a way for apps to consume these services and make them useful to us.

Introducing the GP API
The patient needs a new type of service from the NHS which works with our new patient record. This will incorporate:

• The ability to accept, process and analyse a large amount of personal data
• The ability to recognise salient patterns within this data
• The ability to connect these salient patterns to useful information and advice
• A way to package this service to allow data to be securely transmitted and recommendations to be immediately provided

This is the GP API.

Like a real GP apps using the GP API could be used by to assess your data – your extensive range of vitals – and to form an opinion about your health, offering warnings and advice about changes you might make.
They could make recommendations about specialist service providers best placed to deal with your complaints.
They can perform differential diagnosis based on you making simple answers to questions they pose.
Given a diagnosis they can recommend a treatment plan.

Unlike a real GP apps can go home with you, provide you with a huge resource of additional material to help you understand your condition, nudge you to stay with your prescribed treatment and help monitor your rehabilitation.

Unlike a real GP, if you are unfortunate enough to have a chronic illness, apps will be on standby for the remainder of your life giving you and your family and friends practical advice and support to manage your condition.

Unlike a real GP apps using the GP API would provide a highly available service operating 365x24x7 in every part of the country simultaneously, servicing thousands of concurrent requests – without falling over.

Why is the NHS well placed to create a GP API?

• The NHS has access to the expertise needed to provide such a service
• The NHS has the authority to assure patients that they are getting safe and considered advice
• The NHS has the scale and resources to manage a single central system tasked with collating and analysing vast amounts of data

Why should the NHS want to create a GP API?

• Making good use of existing resources and investment (in line with strategic objectives)
• Improves healthcare services for patients
• Isolates expert services from the distributed delivery of those services
• Provides an architecture that breaks down the healthcare stack into more manageable units
• Having access to this data and developing these services would allow the creation of a world class system capable of delivering global services, at a profit

What would the GP API look like?
This is the sort of thing I do with my GP:

• Get a health check
• Take the advice to avoid getting ill, or discover I may already be ill
• Get a differential diagnosis to narrow down the condition I might have
• Find out more about this condition, read about symptoms, causes and the potential treatment plans
• Look for a specialist to investigate further

Here is an overview of GP API v0.1

//to get a health check – using data from your health graph
Consultation(data){

Potential conditions
Recommended Actions

}
//to diagnose a condition – similar to theengine behind NHS 24 decision tree
DifferentialDiagnosis (data, answers to simple questions){

Possible conditions
More simple questions

}
//to find out more about a condition – similar NHS Choices API
GetDetailsAboutCondition (condition){

Symptoms
Causes
Treatment Plan

}
//to find out where to get specialist services – should leverage patient ratings
GetRecommendedSpecialist (condition){

Specialist practitioner
Centres of excellence

}
Using the GP API
By publishing a GP API we would see an explosion of apps built by vendors across the globe. Storm are involved in building precisely the sort of apps in the private sector that would leverage this type of service.

Apps might use the GP API to treat a range of DIY Health scenarios:

• Preventative health management: ‘wellness’
• Post consultation support (understanding diagnosis, managing prescriptions, rehabilitation programmes)
• Management of chronic conditions (such as diabetes, depression, etc)
• Social care

This fits entirely with the NHS Scotland eHealth Strategy 2011 – 2017, where we find the following strategic aims:

• “support people to communicate with the NHSS, manage their own health and wellbeing, and to become more active participants in the care and services they receive; “
• “contribute to care integration and to support people with long term conditions;”
• “improve the availability of appropriate information for healthcare workers and the tools to use and communicate that information effectively to improve quality;”
• “improve the safety of people taking medicines and their effective use.”

Additionally this approach speaks directly to the heart of the quality ambitions for eHealth, entirely delivering a person-centred approach:

• “support innovative applications of eHealth which enable the delivery of the NHSS’s Three Quality Ambitions
• Person-centred
• Safe
• Effective

Ultimately this approach finally delivers on long held ambitions which precede even the current eHealth strategy:

• “making patient care safer and more effective by making available the right information at the right place and at the right time;”

Our mobile design and development experts used the latest development technology to bring Burn’s work to the palm of your hand. The apps feature a searchable database of 558 of Burns’ poems and love songs, with a glossary of terms that floats over the poem to help you with the Scots’ dialect. There is also an About Burns section with facts about the poet’s life and a Burns supper guide to help you plan your own Burns-themed event.

The apps have been a hit with Burn’s fans the world over and has been used almost 40,000 times since launch in January 2012!

You can get yours here:

http://itunes.apple.com/gb/app/the-works-of-robert-burns/id414486992?mt=8

https://play.google.com/store/apps/details?id=com.stormid.robertburns

1. Overhaul Your Social Media Channels

Write down all of the social media channels that you’re active on. Digital Marketers always get really excited when a new social media channel becomes available, however it’s important to avoid jumping in on the latest ‘fad’ without thinking whether it is applicable to your overall marketing strategy. Streamline your social media channels to those that are relevant to your business goals.

It’s best to control 3 channels that really add value to your business, than manage 8 channels that nobody engages with.

2. Aim For Engagement

Struggling to get people to talk to you? Take a look at what you’re posting. If your messages look like one long sales pitch – then you’re less likely to get any kind of response from your audience. Engagement is key with social media, make sure to encourage a response from your audience by asking open ended questions. “Fill in the blank” posts are an excellent way to get people to comment on your post, just make sure to mix it up from time to time. Video, photos and polls are also great ways to seed discussion. Reach out to your community to encourage a response.

I find _________ is the best way to engage with my social media audience.

3. Personality Injection

Inject some personality into your social media strategy. Each channel provides some sort of cusomisation, whether it’s a Twitter background, Facebook cover photo or a YouTube skin. Your brand is so much more than a logo. Think about its personality, does it have a mission and characteristics? Write down 5 words that describe your brand then look at whether that message is reflected in your social media channels.

Social media amplifies your brand’s message, therefore make sure that each channel uses similar visuals!

4. Increase Efficiency With Tools, Just Don’t Rely On Them!

Tools can really add value to your social media strategy, assisting you to manage each channel from one place. This is especially useful if your job requires you to manage more than social media. However picking the right tool can be tricky, as the market is saturated with both free and paid for buzz-marketing tools.

Here are some of my favourite free tools, that I use regularly:

TweetDeck: This desktop tool is ideal for managing several Twitter accounts

CoTweet: Manage your Twitter and Facebook fan pages in one place

Social Mention: Track what is being said about your brand in the social community and measure sentiment

Conversocial: Great for customer service focused brands, get alerts everytime someone posts, comments and messages you

Avoid using tools to post to your social media channels, people are switched on to what is a genuine post and what’s pre-canned. Keep it personable and optimise the message for each channel. There’s nothing worse than seeing a Facebook post crammed with hashtags.

#HashtagsAreUselessOnFacebook

5. Create Awareness To Build Awareness

Set up a Facebook page and still waiting for an influx of fans? Social media needs a bit more investment in order to start enjoying the perks of viral marketing and a mass market following. You’re going to need to create awareness of your social media awareness, before social media can increase brand awareness.

Take a look at your website, do you have social media icons directing users through to your pages? Users recognise these icons and are much more likely to click on them than a link that says “Facebook”.

Inform your website visitors what added value your social media pages provide. Do you run regular competitions or provide new information? Include this with a call to action to drive traffic to your social media pages.

Include your social media presence in your e-newsletter. Contrary to rumours, email is still an effective method of communication, and a sure way of driving your social following.

In terms of Facebook, use Facebook’s advertising platform to drive fans to your page. Facebook allows specific targeting for best value for money. If you already know your target market, then you can target them on Facebook.

Cross promote between your channels. Post your YouTube content on Facebook and Twitter. Direct Facebook fans to your Twitter competitions. Install a Flickr page to your Facebook. Let your social media channels work in harmony with one another, to drive the amount of people who engage regularly with your page.

Once you have fans, remember to keep them there by providing engaging, relevant and useful content.

The QR code for this article

Here are two important facts about QR codes.

• QR codes are an awesome mobile marketing opportunity. Fact!
• The way some businesses use QR codes is pretty far from awesome. Fact!

My eight year-old nephew has just been learning the difference between opinion and fact at school, so he might want to correct those two statements. If he did, though, it would only be to say “opinion – almost certainly true”, instead of falsehood, but I digress!

If you are new to QR codes, I’ll give you a quick introduction. QR codes are essentially 2 dimensional barcodes that can be scanned by smartphones to cause some kind of action – typically a phone number or web address, although other actions are also possible. When a smartphone user scans the QR code, their phone offers to undertake that action for them.

Of course, this has some outstanding marketing potential – giving marketers the potential to connect inanimate objects to the Internet via smartphones opens up a world of ways for the real world to connect with the Internet.

For example, point of sale displays in shops could use QR codes to link to product reviews and other marketing information, which might be especially useful for high-ticket consideration purchases, such as laptops, bikes, washing machines and so on. According to an On Device Research survey of mobile users, 60% of respondents have checked the mobile Internet in-store, with price comparison, checking store locations and hunting for discounts being the most common reasons.

QR codes enable shop owners to link such point of sale locations to such information online.

Of course, in-store price comparison and suchlike could be construed as a threat by many retailers. Some retailers are alleged to have attempted to reduce mobile data coverage in their stores to prevent this. Clearly, such action might not be considered especially sensible in the long term.

A more sensible approach would be for stores to ensure that they have an effective mobile website, coupled with a sound business value proposition.

QR codes can also be used in other innovative ways.  Conference organisers can consider adding QR codes to lanyard inserts to enable delegates to swap contact details quickly via QR code links to v-cards.

Product packaging could use QR codes as shortcuts to online competitions, or other promotional material, such as Facebook pages or Twitter accounts.

Pubs and clubs could use QR codes on beermats to promote the establishment’s website or online promotions.

Shop window displays for shops that also have an ecommerce presence can use QR codes to link to online products promoted in the window display, or alternatively to downloadable vouchers and suchlike that can be used in-store.

Shop or restaurant receipts could have QR codes to link to promotional material for the business, such as website, Facebook, or vouchers or coupons.

The important thing to remember when looking for how to get the best from QR codes is to consider the environment in which they will be used. Ensure that they are likely to be scanned in an area that has good mobile data or public WiFi access. Ensure that the web page linked to is mobile-ready. Ensure that the users can easily scan the QR code without endangering themselves. Ensure that the user will be keen to scan the code – make sure the benefit of doing so outweighs any risk that the user might feel would be inherent.

Some examples of things I have seen where these things have not been considered include:

• QR codes used inside tube trains – try getting a mobile data signal 200 feet under London.
• QR codes on tube posters on the other side of the tracks to the platform – doesn’t seem a safe place for users to be trying to scan while on a busy platform!
• QR codes linking to large-screen poorly optimised websites – creating issues for the user to interact with the webpage the QR takes them to.
• QR codes on public posters for products that users are likely to feel embarrassed to scan in public, such as for personal hygiene products, and debt management services.
• QR codes on web pages – if the user is already online, why use a QR code when a simple link would do the same job?

QR codes in use in-store

Using QR codes effectively requires putting a little thought into how your users are going to interact with the QR code, including where they are going to interact with it, how they will be encouraged to react with it, what the benefit will be to the user, what the benefit will be to the business, and how will the business will get value from the engagement.

For example, you might choose to add a QR code to a point-of-sale ticket on a washing machine. The QR code might take the user to a web page with more product information, and include a call to action to buy, and pay via an online service such as PayPal or similar. The user gets more product information before they buy, and are able to buy and pay quickly, and if the service were done well, they would be able to arrange delivery through the same interface.

Or a musician’s or merchandise packaging could include a QR code to download exclusive online content – adding further value to traditional promotional material such as tee shirts or beanie hats.

QR codes are like the web links your offline promotional materials always wanted. They connect the offline world to the online world, generating what should be a seamless link between the two – when they are used appropriately.

So, in summary, QR codes ARE a great marketing tool for the right purpose BUT you have to use them appropriately to get a benefit for your business. Awesome! Fact!

Image by http://www.flickr.com/photos/travelinlibrarian/5301073148/sizes/m/

Consequently, the organic search industry is a breeding ground for paranoia and cynicism.

Often cited, but probably overstated opinions include:

1. Google just wants people to click on their paid ads
2. Google upranks site with Adsense advertising so there is more click-through on ads
3. Google hates search engine optimisers [2] [3]
4. SEO is dead. [2] [3]

Eeek! None of that looks good for my career!

BUT…

It’s all nonsense, or at least misrepresentation.

Of course Google would like their users to click on their ads, but what is more important for Google is that it maintains or grows its user base.  It won’t do that by only delivering ads that might not fit what a user has searched for.

It is nonsense to believe that Google upranks sites using Adsense. In fact, all the indications are that sites containing too much advertising (including Adsense) might suffer in search results. Google’s famous Panda updates specifically address this issue, among others.

Google doesn’t hate search engine optimisers. It hates “bad” search engine optimisers.  More specifically it hates those that overcook their SEO –  those that try to ranks sites of little or no value; those that really do try to “game” the algorithms.

But, don’t all optimisers try to “game” the algorithms?

No.

Good search engine optimisers don’t game algorithms – good search engine optimisers work to help websites to look at their strategy to make it line up better with those things that search engines want to see to make sites rank better.

image by Camdiluv

Bad search engine optimisers look to take advantage of loopholes in what search engines look for to gain an advantage over the search engine – looking for ways to get links that only serve to be counted by Google, or to create web pages whose only purpose is to rank for popular searches, all of this with no regard to the value that the website has to their users.

Good search engine optimisers look to add value to websites in ways that search engines appreciate, creating more relevant and useful content for users.

Bad search engine optimisers have little regard for users – their philosophy is to stack the traffic high and hope some converts to sale.

Good search engine optimisers’ philosophy is to gain relevant and engaged traffic that is already interested in what the website has to offer.

So despite the paranoia, which often comes from those that many in the business consider to be the “good” search engine optimisers, there is most certainly a type of search engine optimiser that Google does love.

Google wants to provide its own users with the best experience they can have. Not only is organic search ruled by algorithmic judgement by Google, even the paid search activity needs to adhere to quality standards, albeit it is possible to be more lax on this if you have money to burn on advertising.

To provide the best experience for its users, Google wants to provide links to websites (or other services, like maps, products, reviews etc.) that will best serve the users’ needs. Google tries (relatively well most of the time) to understand the context of a user’s search and give the best results back for what that user wants to do.

A good SEO will try to ensure that a site or service that could fulfil a user’s needs in their industry does so. It’s not simply a matter of having more links, more keywords on the page, or more social mentions. It’s about meeting the needs of users searching in Google.  If you don’t do that, Google really isn’t that interested in your website.  If you do, you’re in, and then it becomes a contest of relevance and link building.

When Matt Cutts announced that Google wanted to “level the playing field” between webmasters, his assertion was that Google is out to negate the effects of overly aggressive “bad” optimisers.

However, Google takes the view that if you are useful, you’ll get links on the strength of that usefulness, and your content will be relevant on the strength of that usefulness too.  Being useful does not give you free rein to spam.

Content and link building strategies need to focus on content and service promotion, not on simply gaining quantity of links, improving keyword density, or stuffing Meta data.

Good search engine optimisers work to meet Google’s objectives as well as those of their website.

Google even encourages good search engine optimisation techniques, creating services that rely on the SEO industry to implement, such as rich snippets, rel=author, canonical tags and so on. It’s just that Google expects you to use these for their benefit (even if they don’t phrase it that way!).

There will always be a part of the industry that is dedicated to leveraging the loopholes and weaknesses in Google approach, and the most skilled of these “bad” optimisers will always do well – but all of the techniques are under threat constantly, and this part if the industry will continuously need to reinvent itself.

Good search engine optimisation is a toolkit for life that occasionally gets some new tools to add to its arsenal. Good search engine optimisation is an effective long term strategy for business, not just for ranking.

A good search engine optimiser works to make Google a better place for users to visit, because Google’s users find their website useful when they click through from a search result. Their methods are to help websites to make themselves more attractive to Google to rank higher. The user is happy. Your business is happy. Google is happy.

So, really…

Google does love SEO <3

(image by Camdiluv)

Back at the beginning of October, Google introduced SSL encryption to search for users searching through Google.com while logged into their Google account.

Google describes their reasoning in this blog post, describing how SSL encryption protects privacy, particularly for users of unsecured Internet connections such as WiFi hotspots.

Many users are logged into their Google accounts frequently – this includes users who are logged into Gmail, Google Plus, Analytics, Adwords, Google Docs, and other services.

There is a downside for Analytics users

However, while striving for privacy is admirable, there is a downside for websites that use Google Analytics.

One of the metrics gathered by Analytics is “organic keywords” – that is the phrases people have searched in order to arrive at the website. This data is anonymous, in that it cannot be traced back to individual users. It is, however, of great benefit to websites in helping to shape content strategy and to understand how to make their sites more relevant and useful to the people that use them.

For example, if a number of users have arrived at the site by searching for ‘red widgets’, and the conversion rate for buying red widgets is high, this is a good situation, but if an equally high number of users arrived at the site searching for ‘blue widgets’  and conversion is low, this could suggest a number of things might be wrong, such as pricing, availability, product information given on the site, calls-to-action, whether the users are landing on the right page, and so on.

It is not simply a case of looking at the sales figures, as it is important to know the demand – i.e. the relative levels of searches for each product from people arriving at your site.

The problem is that Google no longer sends the keyword data in the HTTP-Referer header for people using SSL search, which means that a portion of the organic keyword search data is absent.  These searches are instead recorder as “(not provided)” in the keyword data in Analytics.

It is important to note that this does not affect paid search, or Bing and Yahoo (at least not yet).

Google is now extending the SSL-search beyond the USA

Earlier this week, Google announced that it had begun to extend the SSL-encryption to users in some territories outside of the United States.  In reality, some users outside the United States were already using SSL-search anyway, as some user search Google.com directly from other locations than the States.

Consequently, on some Analytics accounts, we are beginning to see the ratio of searches recorded with “(not provided)” as the keyword increase from usually around 2% to somewhere around 15%, already.

It’s more important than ever to consider paid search

The effects of this on process mean that it is now more important than ever to consider paid search as a sandbox for conversion process interrogation using the process described above.  There are other advantages to using paid search for this of course – it is possible to control many more variables directly, such as ad copy and landing page – such things are not directly controllable through organic search, although it is possible to influence what Google might show.

Additionally, with paid search it is possible to ensure that things like search position between two products are more equivalent to give a more even testing ground for this process.  Paid search is also more immediately responsive to changes you make.

But remember organic search is still vital

That said, organic search is still a vital ingredient for success.  Best practice for organic search reaps rewards long into the future of a website, and will continue to create excellent ROI.

Order Interests by Categories

Want specific news? This new Facebook feature allows you to subscribe to certain categories that are relevant to you, such as fashion, recipes and sports. Interests will also allow you to create your own categories, so you only see what you’re interested in. An example would be if I set up a Facebook interest for fashion, I can put all my favourite brands in there – such as Burberry, Ralph Lauren and Gucci. When I log in, I can simply click on that category and it will filter my newsfeed to show me all of the “news” from these pages.

It will be like reading your own personalised newspaper, news from the pages that you subscribe to.

Cleaning up the News Feed

This doesn’t mean that the current news feed will become extinct. The top stories from each interest (your likes) will still populate the news feed, based on Facebook’s EdgeRank algorithm.

Sound a bit like Google Reader? Twitter lists? Pinterest? You’re right. Facebook’s latest feature will allow its user base to syphon out the “noise” in their news feed.

Trouble for Brands?

What does this mean for brands? There will definitely be more pressure to provide interesting, captivating and relevant content, to engage your Facebook fan base. A major benefit is that is could even lead to more interaction, as your page may sit within extremely popular interests.

Facebook interests will be rolled out within the commencing few weeks. What do you think of Facebook Interests?

Master your robot overlords with robots.txt

At Storm ID we pride ourselves on providing cutting-edge solutions and modern and exciting advice and consultancy.

However, while the industry evolves, grows and constantly invents new and energising technologies and opportunities, some of the basic issues never seem to go away. There seem to be a number of issues that many in the industry never seem to grasp properly (thankfully, I’m not including our lovely development team here at Storm in that group – but we frequently see 3^rd party managed sites coming through to our marketing team with such issues).

One of the most common issues we see that should really be part of web development 101 is the use of robots exclusion protocol – robots.txt.

Why should websites use robots.txt?

Since robots exclusion was first considered (in 1994!) there have been several valid reasons for wanting to exclude robots from indexing some or all of your website content.

The reasons might historically have included:

1. privacy – you might not have wanted some of your website content to appear in search engines, such as customer data
2. security – you might have had concerns that a robot might index secure content such as financial data
3. duplicate content – you might have wanted to have different versions of your site for different users – such as having a large text version, or a printer friendly version – which you wanted to avoid a search engine indexing and counting as duplicate
4. You might have been cloaking (naughty, naughty – slaps your wrist) – you could have created a text heavy version that used some kind of JavaScript redirect to push real users to a robots excluded image or Flash rich version of the site (welcome to 1998!) *disclaimer – I would heavily recommend NOT doing this, even in 1998!

In 2012, there are even more reasons to consider using robots exclusion, though. Search engines, and Google in particular, are concerned with the overall quality of your website. Google wants to index useful content.  It’s not so concerned with all the other detritus of the web.  You might not think your 2 million internal search result pages are detritus – but Google might. If Google sees a few really good pages on your site, but many more useless ones, there is a real risk that your good pages will suffer in rankings, pulled down by the negative weight of the bad pages. Google ranks the content of web pages, but the reputation and importance of web sites.

Why robots.txt needs care

If you don’t correctly set up your robots exclusion file, there are a number of bad side-effects that might happen.

Clearly, one risk is that it is ignored completely, and search engines end up indexing everything on your website, including all the stuff you actually wanted to exclude.

Worse still, you might accidentally end up blocking important content from being indexed, may be even all of your content.

Getting the syntax right

Perhaps the most common mistake I see in robots.txt files is incorrect syntax. Specifically, the single most common mistake is use of “Allow” as an instruction.

There are only two valid instructions that robots will parse in a robots.txt file (actually there is a third one that some will parse but I will come to that later on).

Those two instructions are: “User-agent:” and “Disallow:”

As the names imply, “User-agent:” is used to instruct a specific robot based-on its user agent, and “Disallow:” is used to inform the robot what URLs it must not index.

There is no “Allow” instruction to make any exception to the syntax.

However, it is acceptable to comment within the robots.txt file using the “#” character at the start of each comment line. The comment is ended by a newline character.

Furthermore, there is no use of wildcards or regular expressions in the robots.txt protocol, with the exception that the wildcard “*” can be used to instruct all robots in the directive “User-agent: *”.

The Disallow field strictly describes partial URLs (not directories, as is commonly described, although a directory is one kind of partial URL).

For example:
Disallow: /help # disallows /help/index.htm and /help.htm
Disallow: /help/ # disallows /help/index.htm but allows /help.htm

You must place your robots.txt file in the root of your website, i.e. at http://www.yourwebsite.com/robots.txt, and it must be called robots.txt.

The third instruction – Sitemap

There is a third instruction that is obeyed by some robots.  That is the instruction “Sitemap:” followed by a URL.  This instruction is used by some robots to indicate the location of an XML file with a list of URLs that should be indexed.

As not all robots obey “Sitemap:”, it should be placed at the end of the robots.txt file to enable other robots to parse the other instructions in the file. Multiple lines should be used to list multiple sitemap files, and only exact URLs can be specified for each sitemap.

For example:
Sitemap: /sitemap.xml
Sitemap: /videositemap.xml

Obviously, this creates the possibility that a URL indicated in a sitemap might otherwise be implicitly or explicitly excluded in the robots “Disallow:” fields. It isn’t always clear what action a search engine will take when it receives contradictory instructions such as this.  Furthermore, it is an indicator that not enough care is being taken in the preparation of the robots.txt and sitemap files, and as a result the search engines may reduce the trust they have in the site, which may impact ranking.

Alternative tactics

If you can’t access your robots.txt file to edit it, then there are alternative tactics that can be used (although if your robots.txt file disallows something, you can’t use these to override that).

Noindex

The Meta tag “robots” is correctly used to indicate to a robot whether to index a page, and whether to follow links on the page. The permission to index or follow is implied by absence, and so there is no need to instruct a robot to index or to follow.  However, you may have reasons to cite noindex, or nofollow.  Noindex is equivalent to a web page being disallowed in robots.txt.

The syntax is:
<meta name="robots" content="noindex, nofollow">

Edit: there are also some other values you can apply to the robots Meta tag, although they aren’t specifically equivalent to robots.txt behaviour, like ‘noindex’ is. Thanks @John_Meffen for reminding me to mention that ;-)

x-robots-tag

Another alternative, which is also useful for binary files such as images and downloads, is a header response sent by the server that instructs a robot that a file is not to be indexed.

Different server software and server-side languages have different ways of creating the x-robots-tag. In PHP, for example, you would include the following line at the top of your PHP file:
header("X-Robots-Tag: noindex", true);

The difference between indexing and ranking

It is important to understand that blocking a page from being indexed by a search engine does not mean that it is prevented from appearing in Google and other search engines.  Indexing means the process a search engine uses to crawl and understand the web page’s content – this is what robots.txt prevents.

Google et al will still potentially include URLs in their index that they find via other publically available data, such as links from third party sites (even links that are “nofollowed”), social sharing such as Facebook and Google Plus shares.

Summary

To round up, I will emphasise that it is important to make sure that your good content is not blocked by robots.txt, but your poor content perhaps should be so that the good content benefits (perhaps you would be better with a proper content strategy, though!).

You should block content you don’t want to be indexed, such as private content or secure content, but really you should ensure that they also live only behind a log in to avoid the URLs being listed in Google also – remember to use the correct response headers for users that are not logged-in – use a 401 response, not a 200 or 301/302.

Remember if you have a staging server or test server, block it all with both robots.txt and hide it behind a log-in.  Make sure, though, that when you push your code to the live server that you don’t push the robots.txt or the log-in live too.

References:

http://support.google.com/webmasters/bin/answer.py?hl=en&answer=156449

http://www.robotstxt.org/orig.html and http://www.robotstxt.org/robotstxt.html

http://en.wikipedia.org/wiki/Robots.txt

http://www.w3.org/TR/html4/appendix/notes.html#h-B.4.1.1

http://yoast.com/x-robots-tag-play/

‘Robot Overlord’ photo by Tom Hilton