Consequently, the organic search industry is a breeding ground for paranoia and cynicism.

Often cited, but probably overstated opinions include:

1. Google just wants people to click on their paid ads
2. Google upranks site with Adsense advertising so there is more click-through on ads
3. Google hates search engine optimisers [2] [3]
4. SEO is dead. [2] [3]

Eeek! None of that looks good for my career!

BUT…

It’s all nonsense, or at least misrepresentation.

Of course Google would like their users to click on their ads, but what is more important for Google is that it maintains or grows its user base.  It won’t do that by only delivering ads that might not fit what a user has searched for.

It is nonsense to believe that Google upranks sites using Adsense. In fact, all the indications are that sites containing too much advertising (including Adsense) might suffer in search results. Google’s famous Panda updates specifically address this issue, among others.

Google doesn’t hate search engine optimisers. It hates “bad” search engine optimisers.  More specifically it hates those that overcook their SEO –  those that try to ranks sites of little or no value; those that really do try to “game” the algorithms.

But, don’t all optimisers try to “game” the algorithms?

No.

Good search engine optimisers don’t game algorithms – good search engine optimisers work to help websites to look at their strategy to make it line up better with those things that search engines want to see to make sites rank better.

image by Camdiluv

Bad search engine optimisers look to take advantage of loopholes in what search engines look for to gain an advantage over the search engine – looking for ways to get links that only serve to be counted by Google, or to create web pages whose only purpose is to rank for popular searches, all of this with no regard to the value that the website has to their users.

Good search engine optimisers look to add value to websites in ways that search engines appreciate, creating more relevant and useful content for users.

Bad search engine optimisers have little regard for users – their philosophy is to stack the traffic high and hope some converts to sale.

Good search engine optimisers’ philosophy is to gain relevant and engaged traffic that is already interested in what the website has to offer.

So despite the paranoia, which often comes from those that many in the business consider to be the “good” search engine optimisers, there is most certainly a type of search engine optimiser that Google does love.

Google wants to provide its own users with the best experience they can have. Not only is organic search ruled by algorithmic judgement by Google, even the paid search activity needs to adhere to quality standards, albeit it is possible to be more lax on this if you have money to burn on advertising.

To provide the best experience for its users, Google wants to provide links to websites (or other services, like maps, products, reviews etc.) that will best serve the users’ needs. Google tries (relatively well most of the time) to understand the context of a user’s search and give the best results back for what that user wants to do.

A good SEO will try to ensure that a site or service that could fulfil a user’s needs in their industry does so. It’s not simply a matter of having more links, more keywords on the page, or more social mentions. It’s about meeting the needs of users searching in Google.  If you don’t do that, Google really isn’t that interested in your website.  If you do, you’re in, and then it becomes a contest of relevance and link building.

When Matt Cutts announced that Google wanted to “level the playing field” between webmasters, his assertion was that Google is out to negate the effects of overly aggressive “bad” optimisers.

However, Google takes the view that if you are useful, you’ll get links on the strength of that usefulness, and your content will be relevant on the strength of that usefulness too.  Being useful does not give you free rein to spam.

Content and link building strategies need to focus on content and service promotion, not on simply gaining quantity of links, improving keyword density, or stuffing Meta data.

Good search engine optimisers work to meet Google’s objectives as well as those of their website.

Google even encourages good search engine optimisation techniques, creating services that rely on the SEO industry to implement, such as rich snippets, rel=author, canonical tags and so on. It’s just that Google expects you to use these for their benefit (even if they don’t phrase it that way!).

There will always be a part of the industry that is dedicated to leveraging the loopholes and weaknesses in Google approach, and the most skilled of these “bad” optimisers will always do well – but all of the techniques are under threat constantly, and this part if the industry will continuously need to reinvent itself.

Good search engine optimisation is a toolkit for life that occasionally gets some new tools to add to its arsenal. Good search engine optimisation is an effective long term strategy for business, not just for ranking.

A good search engine optimiser works to make Google a better place for users to visit, because Google’s users find their website useful when they click through from a search result. Their methods are to help websites to make themselves more attractive to Google to rank higher. The user is happy. Your business is happy. Google is happy.

So, really…

Google does love SEO <3

(image by Camdiluv)

Back at the beginning of October, Google introduced SSL encryption to search for users searching through Google.com while logged into their Google account.

Google describes their reasoning in this blog post, describing how SSL encryption protects privacy, particularly for users of unsecured Internet connections such as WiFi hotspots.

Many users are logged into their Google accounts frequently – this includes users who are logged into Gmail, Google Plus, Analytics, Adwords, Google Docs, and other services.

There is a downside for Analytics users

However, while striving for privacy is admirable, there is a downside for websites that use Google Analytics.

One of the metrics gathered by Analytics is “organic keywords” – that is the phrases people have searched in order to arrive at the website. This data is anonymous, in that it cannot be traced back to individual users. It is, however, of great benefit to websites in helping to shape content strategy and to understand how to make their sites more relevant and useful to the people that use them.

For example, if a number of users have arrived at the site by searching for ‘red widgets’, and the conversion rate for buying red widgets is high, this is a good situation, but if an equally high number of users arrived at the site searching for ‘blue widgets’  and conversion is low, this could suggest a number of things might be wrong, such as pricing, availability, product information given on the site, calls-to-action, whether the users are landing on the right page, and so on.

It is not simply a case of looking at the sales figures, as it is important to know the demand – i.e. the relative levels of searches for each product from people arriving at your site.

The problem is that Google no longer sends the keyword data in the HTTP-Referer header for people using SSL search, which means that a portion of the organic keyword search data is absent.  These searches are instead recorder as “(not provided)” in the keyword data in Analytics.

It is important to note that this does not affect paid search, or Bing and Yahoo (at least not yet).

Google is now extending the SSL-search beyond the USA

Earlier this week, Google announced that it had begun to extend the SSL-encryption to users in some territories outside of the United States.  In reality, some users outside the United States were already using SSL-search anyway, as some user search Google.com directly from other locations than the States.

Consequently, on some Analytics accounts, we are beginning to see the ratio of searches recorded with “(not provided)” as the keyword increase from usually around 2% to somewhere around 15%, already.

It’s more important than ever to consider paid search

The effects of this on process mean that it is now more important than ever to consider paid search as a sandbox for conversion process interrogation using the process described above.  There are other advantages to using paid search for this of course – it is possible to control many more variables directly, such as ad copy and landing page – such things are not directly controllable through organic search, although it is possible to influence what Google might show.

Additionally, with paid search it is possible to ensure that things like search position between two products are more equivalent to give a more even testing ground for this process.  Paid search is also more immediately responsive to changes you make.

But remember organic search is still vital

That said, organic search is still a vital ingredient for success.  Best practice for organic search reaps rewards long into the future of a website, and will continue to create excellent ROI.

Master your robot overlords with robots.txt

At Storm ID we pride ourselves on providing cutting-edge solutions and modern and exciting advice and consultancy.

However, while the industry evolves, grows and constantly invents new and energising technologies and opportunities, some of the basic issues never seem to go away. There seem to be a number of issues that many in the industry never seem to grasp properly (thankfully, I’m not including our lovely development team here at Storm in that group – but we frequently see 3^rd party managed sites coming through to our marketing team with such issues).

One of the most common issues we see that should really be part of web development 101 is the use of robots exclusion protocol – robots.txt.

Why should websites use robots.txt?

Since robots exclusion was first considered (in 1994!) there have been several valid reasons for wanting to exclude robots from indexing some or all of your website content.

The reasons might historically have included:

1. privacy – you might not have wanted some of your website content to appear in search engines, such as customer data
2. security – you might have had concerns that a robot might index secure content such as financial data
3. duplicate content – you might have wanted to have different versions of your site for different users – such as having a large text version, or a printer friendly version – which you wanted to avoid a search engine indexing and counting as duplicate
4. You might have been cloaking (naughty, naughty – slaps your wrist) – you could have created a text heavy version that used some kind of JavaScript redirect to push real users to a robots excluded image or Flash rich version of the site (welcome to 1998!) *disclaimer – I would heavily recommend NOT doing this, even in 1998!

In 2012, there are even more reasons to consider using robots exclusion, though. Search engines, and Google in particular, are concerned with the overall quality of your website. Google wants to index useful content.  It’s not so concerned with all the other detritus of the web.  You might not think your 2 million internal search result pages are detritus – but Google might. If Google sees a few really good pages on your site, but many more useless ones, there is a real risk that your good pages will suffer in rankings, pulled down by the negative weight of the bad pages. Google ranks the content of web pages, but the reputation and importance of web sites.

Why robots.txt needs care

If you don’t correctly set up your robots exclusion file, there are a number of bad side-effects that might happen.

Clearly, one risk is that it is ignored completely, and search engines end up indexing everything on your website, including all the stuff you actually wanted to exclude.

Worse still, you might accidentally end up blocking important content from being indexed, may be even all of your content.

Getting the syntax right

Perhaps the most common mistake I see in robots.txt files is incorrect syntax. Specifically, the single most common mistake is use of “Allow” as an instruction.

There are only two valid instructions that robots will parse in a robots.txt file (actually there is a third one that some will parse but I will come to that later on).

Those two instructions are: “User-agent:” and “Disallow:”

As the names imply, “User-agent:” is used to instruct a specific robot based-on its user agent, and “Disallow:” is used to inform the robot what URLs it must not index.

There is no “Allow” instruction to make any exception to the syntax.

However, it is acceptable to comment within the robots.txt file using the “#” character at the start of each comment line. The comment is ended by a newline character.

Furthermore, there is no use of wildcards or regular expressions in the robots.txt protocol, with the exception that the wildcard “*” can be used to instruct all robots in the directive “User-agent: *”.

The Disallow field strictly describes partial URLs (not directories, as is commonly described, although a directory is one kind of partial URL).

For example:
Disallow: /help # disallows /help/index.htm and /help.htm
Disallow: /help/ # disallows /help/index.htm but allows /help.htm

You must place your robots.txt file in the root of your website, i.e. at http://www.yourwebsite.com/robots.txt, and it must be called robots.txt.

The third instruction – Sitemap

There is a third instruction that is obeyed by some robots.  That is the instruction “Sitemap:” followed by a URL.  This instruction is used by some robots to indicate the location of an XML file with a list of URLs that should be indexed.

As not all robots obey “Sitemap:”, it should be placed at the end of the robots.txt file to enable other robots to parse the other instructions in the file. Multiple lines should be used to list multiple sitemap files, and only exact URLs can be specified for each sitemap.

For example:
Sitemap: /sitemap.xml
Sitemap: /videositemap.xml

Obviously, this creates the possibility that a URL indicated in a sitemap might otherwise be implicitly or explicitly excluded in the robots “Disallow:” fields. It isn’t always clear what action a search engine will take when it receives contradictory instructions such as this.  Furthermore, it is an indicator that not enough care is being taken in the preparation of the robots.txt and sitemap files, and as a result the search engines may reduce the trust they have in the site, which may impact ranking.

Alternative tactics

If you can’t access your robots.txt file to edit it, then there are alternative tactics that can be used (although if your robots.txt file disallows something, you can’t use these to override that).

Noindex

The Meta tag “robots” is correctly used to indicate to a robot whether to index a page, and whether to follow links on the page. The permission to index or follow is implied by absence, and so there is no need to instruct a robot to index or to follow.  However, you may have reasons to cite noindex, or nofollow.  Noindex is equivalent to a web page being disallowed in robots.txt.

The syntax is:
<meta name="robots" content="noindex, nofollow">

Edit: there are also some other values you can apply to the robots Meta tag, although they aren’t specifically equivalent to robots.txt behaviour, like ‘noindex’ is. Thanks @John_Meffen for reminding me to mention that ;-)

x-robots-tag

Another alternative, which is also useful for binary files such as images and downloads, is a header response sent by the server that instructs a robot that a file is not to be indexed.

Different server software and server-side languages have different ways of creating the x-robots-tag. In PHP, for example, you would include the following line at the top of your PHP file:
header("X-Robots-Tag: noindex", true);

The difference between indexing and ranking

It is important to understand that blocking a page from being indexed by a search engine does not mean that it is prevented from appearing in Google and other search engines.  Indexing means the process a search engine uses to crawl and understand the web page’s content – this is what robots.txt prevents.

Google et al will still potentially include URLs in their index that they find via other publically available data, such as links from third party sites (even links that are “nofollowed”), social sharing such as Facebook and Google Plus shares.

Summary

To round up, I will emphasise that it is important to make sure that your good content is not blocked by robots.txt, but your poor content perhaps should be so that the good content benefits (perhaps you would be better with a proper content strategy, though!).

You should block content you don’t want to be indexed, such as private content or secure content, but really you should ensure that they also live only behind a log in to avoid the URLs being listed in Google also – remember to use the correct response headers for users that are not logged-in – use a 401 response, not a 200 or 301/302.

Remember if you have a staging server or test server, block it all with both robots.txt and hide it behind a log-in.  Make sure, though, that when you push your code to the live server that you don’t push the robots.txt or the log-in live too.

References:

http://support.google.com/webmasters/bin/answer.py?hl=en&answer=156449

http://www.robotstxt.org/orig.html and http://www.robotstxt.org/robotstxt.html

http://en.wikipedia.org/wiki/Robots.txt

http://www.w3.org/TR/html4/appendix/notes.html#h-B.4.1.1

http://yoast.com/x-robots-tag-play/

‘Robot Overlord’ photo by Tom Hilton

Storm’s Digital Marketing team have just found out that through their top class SEO and PPC skills, they have more than doubled the site visits and more importantly more than doubled the number of room bookings for the Someston Hotel’s Holiday Inn Express portfolio. The results for March 2011 over march 2010 saw a 127% uplift in site visits and 118% uplift in room nights across the 32 hotels over the same period last year. The team took only took over the portfolio in late December 2010: what an excellent example of what a difference strong SEO and PPC can make to your business’ bottom line!

Well done team.

Holiday Inn Express Bath on the first Google page using the term “bath hotel”

For more information on our Digital Marketing Team take a look at http://www.stormid.com/, contact us by email or give us a call: 0131 561 1250.

First off – Microsoft and Google take a bite out of Apple..

Apple VS Microsoft- New Guinness World Record – Microsoft’s Kinect toppled the iPhone as the quickest selling electronic device EVER!
Microsoft have sold 10 Million of them!
Sales per day:

• Kinect: 133,000 (released 4th November, 2010)
• iPhone 4: 120,000 (released 24th June, 2010)
• Wii: 85,000 (released 19th November, 2006)
• iPad: 50,000 (released 3rd April, 2010)

Apple VS Google Android - Android tops the charts in US smartphone platform market share.

Next up – Some UK web stats..
There sure are a lot of people online..

• 38.3 million UK adults used the internet last year <- 30.1 million adults in the UK went online every day or almost every day!
• 31 million bought something online <- 62% of all UK adults
• 19.2 Million households have an internet connection <-73% of all households.

And a lot of people are building new websites…

• 1,000,000 new domain names registered in the UK in 2010..

But.. Here’s a shocker of a fact…

• 40% of UK businesses receive zero traffic to their websites via PPC or Search Engines!

Storm Fact: After building Glasgow Life’s new website and adding in our top class SEO and Digital Marketing – Storm ID have increased Glasgow Life’s monthly page hits over 450% – FACT!!

People often ask us: “Storm – How do you keep on finding these exciting and informative facts?”
We’ve said it before and we say it again: “We live and breathe digital. Everyday we are reading articles, crunching stats, talking to people, going to conferences, speaking at events and generally enjoying our passion for all things web, social and mobile. How can we not come up with exciting and informative facts from all that activity!”

Like these facts? Follow our tweets @stormid and check back here for more facts every time you see #StormFF

Hungry for facts about Storm? See our website: http://www.stormid.com/or get in touch on 0131 561 1250.

Except that such doom-mongery is nonsense. Quite the opposite is true, in fact, as I wrote on my own blog yesterday. It is now much more important to ensure that your website ranks well in search engines.

Organic search, when a website is optimised properly (ignoring PPC for now) can account for over 85% of all the traffic to a website. If a website also uses PPC (the doom-mongery also insisted that PPC might be dead too, in some quarters!), then the combined effect might increase traffic to almost double that, meaning less than 10% of your website traffic is from any other sources than search engines. In my best sarcastic voice – yes, SEO is dead!

The other significant talking point in search this week is the release by SEOMoz of their findings in an investigation into the correlation between Google’s search results and an information retrieval method called latent Dirichlet allocation (LDA).

If you are of a nervous disposition, you may want to ignore the maths, and assume we all know what we’re talking about. I wrote about LDA in my blog here.

In summary, LDA helps an information retrieval system, like a search engine for example, to determine the relevancy of a text document to a set of predefined topics. It helps combat ambiguity where a word or phrase might have multiple potential meanings. For example, the word “space” might relate to the gap between words, NASA, Star Trek, room, and many more different contexts. If I’m searching for how to organise my work space, I don’t want to see search results about Star Trek and NASA, clearly. LDA helps a search engine overcome that.

The latest skirmish in the escalating search engine space race has seen Google follow in Bing’s footsteps by retrieving real-time content from social networks (including Twitter, Myspace and Facebook), blog posts and news feeds, and surfacing all this in Google searches. Most users can currently see these feeds in the middle of the page, amidst organic retrievals. This really contributes to the sense that the real-time web is becoming increasingly and unavoidably built in to the wider online experience.

From Storm ID’s perspective, we can help our clients develop appropriate responses to embrace this shift, while acknowledging that, essentially, the rules of the game remain the same. While it’s important to engage in social media channels to contribute to your communications output, content that follows our recommendations in terms of keyword density, URL, and meta-data, will still index web-pages consistently and impressively. The core tenets of good SEO for websites have not changed; the changes we are embracing extend beyond the website to how we support our clients in syndicating their content to other channels, especially those with real-time updates. Credibility of the source, as well as quality, location and timing will be key.

Fortunately we’re able to say we’re ahead of the game. We often have clients asking how they can have invigorate their presence on microblogging platforms without increasing overheads and dedicating more staff and time. Our solution has been to build social media feeds into your Content Management System, ensuring that their website content, and user-generated forum and comments content, is automatically posted to social networking accounts. These tools will help our clients coordinate these multiple channels: they can dictate their own careful balance beteen automation – to save time – and manual control – to sculpt the message effectively.

How the real-time feeds will evolve is uncertain, and their positional dominance and density on search pages could well change in the long-term. However, they have already taken steps to ensure that their championing of relevant, authoritative content is not undermined. Amit Singhal of Google stressed that search giant has plenty of new technology in place to filter out lightweight or spam content – clearly that filtering process is actually taking a bit of time as tweets are lagging a couple of minutes behind their ‘live’ output.

Spam is a problem on all of these networks. We tend to use ‘followers’, ‘friends’ or ‘groups’ to protect ourselves from this, but Google will need to do the same on our behalf and can leverage a whoe manner of measures of clout/credibility when deciding to surface these results. Timing is a factor and so is content, but Google effectively needs to invent a concept of rank to apply to these new sources of content.

To distinguish good real-time content, live search will alrgely be connected to the Google Trends feature, which identifies spikes in web search traffic. We should see an increasing resemblance here to Twitter’s popular Trending Topics section, as users turn to Google to search for breaking stories and news.

Undoubtedly it’s an exciting and changing time in web culture, but try not to lose sight of the basic elements of SEO, which ultimately remain the best way of reaching the holy grail of organic first page indexing

Google’s remit is to organise the world’s information. This week, that world just got a bit bigger and quicker.