Ross Mayfield's Weblog

No Free Links and Why nofollow Doesn't Work for Wikis

There is no such thing as a free link.  No matter what the person who is selling you it tells you.  And the web will always adapt to make it so.  Even if its hard work.

Barry Schwartz posted Get A Free Link From Wired today on an SEO blog, noting that:

Some SEOs were saddened when Wikipedia added nofollows to external links. Perhaps they'll perk up to discover that Wired's semi-Wikipedia challenger has no such blocking.

The post proceeded to give advice and instructions for how to spam the Wired How To Wiki.  And it has been by the SEO community.  As a member of the community and working at the provider of its most excellent wiki software, I edited out some of the spam as it didn't meet the goals of the community.  But its not that simple.

I believe the intent was to point out an exploit, and not necessarily for bad.  The problem rests with nofollow being a good tool for some purposes (blog comment spam) and not for others (wikis in general).  I had a quick email exchange with Danny Sullivan and he also talked with Wired and edited the post:

NOTE FROM DANNY: We've talked with Wired about the situation, and they are putting a robots.txt block on links coming out of the wiki so that links won't pass credit. Also, our apologies to Wired in that we've ended up causing a run on the wiki with new pages being created. That was definitely not our intent -- the headline of getting a free link, and the article itself, was more tongue-in-cheek about how the system was and might further get abused, rather than advice for people to really misuse the wiki for promotional purposes. I don't agree with that type of abuse in general, and as someone who has had to deal with it in comments or submissions to our forums, it's no fun. In hindsight, we probably should have just dropped a note pointing out the vulnerability. We've also asked that our test page be completely removed -- it has served its purpose now.

If every website was a wiki I might have edited something similar on their post while disabling the link.  Later over a drink with Paolo Valdermin I was joking that we should invent the unfollow link, or make it so any traffic coming from the site doing damage was rejected.  But I'm joking and need to digress into a topic most people don't understand:

unfollow was a byproduct of Vote Links, created by Kevin Marks, which I believe I had a hand in at least inspiring.  It was designed as a tool for making blog comments not count in Google's PageRank, while letting the blog post's links count.  Blog comments are relatively good at dealing with comment spam because of what Clay Shirky described as encapsulation -- every blog has an owner who can determine how to moderate their comments.  nofollow is just one way. 

Now comments with nofollow enabled get spam anyway, maybe because the fact that the blog host is communicating behind the scenes with search engines is ignored, or that SEO isn't the only goal of the spammer.  If you own a blog, you deal with manually sorting through vandalism all the time. 

Still, nofollow is a good thing for some Social Software.  But it doesn't belong or work everywhere.  Can you imagine a web that works as good as today's if every link was tagged with nofollow?

Wikipedia enabled nofollow, much to the chagrin of the SEO community, for some good reasons that also need additional context:

• Wikipedia is an exception to any other wiki community on the web
• Wikipedia has no feature to detect and delete spam, instead it is a feature made of people, who are better at such decisions when one of the goals of the system is openness
• Wikipedia's core editors are extremely burdened with this task because of how valuable its attention is
• The result is Wikipedia gets lots of link love, but doesn't give any to anyone

IMHO, Wikipedia made a mistake implementing nofollow.  While blogs have one or a couple authors, a few commenters, and many readers -- wikis have many authors and many readers.  In an a wide open wiki, just one possible configuration, you can't tell who is trustable or accountable given the missing identity and reputation layers of the web.  nofollow doesn't work for wikis because:

• Wikis are part of the web of links
• I haven't seen a partial and prudent implementation of nofollow in a wiki.  And its hard to picture one that doesn't discriminate upon users automatically.  You could make nofollow apply to links from "trusted users" but different communities should develop different rules for trust.  And such rules could kill communities, especially nascent ones by hard coding rules too early.
• There are other wiki spam countermeasures
• If you enable nofollow, every link gives no value, regardless of its value, and that value is best defined by the community using it

Part of how wikis deal with spam is the phantom authority -- there is a higher transaction cost for damaging a wiki than fixing it.  And people are generally good.

That said, even though I may not want to be in business with a web like this, I care about what's been created on this wiki.  And so I'll be spending some of my time this weekend reverting vandalism.  Better that than giving in by shutting it down, or uninformed suggestions to bluntly implement nofollow.  Besides, the irony is that some of the SEO posts actually have some good How To content that actually needs a little human editing.

Some links to SEO Posts about all this (maybe they will pitch in if I link to them): WebProNews, Wired News and Mashable!

Wikipedia Shouldn't Follow Nofollow

The SEO Blogs are all abuzz about Wikipedia applying the "nofollow" tag to all outbound links so Google doesn't index potential wikispam.

At Jimbo Wales' directive, all external links within the English language Wikipedia are now coded "nofollow" -- this should help cut spamming immensely once word gets out in the SEO community.

This was mentioned in the discussion Wikipedia:Administrators' noticeboard#Globalwarming awareness2007/SEO world championship -- expect a spam onslaught.

I haven't spoken with Jimmy about this, but I think this is the wrong decision.  As blogged previously, nofollow is a partial fit for blog comments, but not wikis:

I should mention that the wiki world isn't wild about nofollow for at least one simple reason. On a blog you have an author and the audience (commentators?).  Within a wiki, everyone is an author.  We are still evaluating where we will use nofollow, I personally see it as  great industry cooperation creating a tool to use.

Unfortunately this is throwing the baby out with the spamwater.  We need an alternative to this broad stroke.  Wikipedia is perhaps the best source of outbound links on the web, with a no-spam policy that while expensive to implement, largely works.  I'm concerned about how this effects the health of the web, search result and what precedent it sets for indexing public wikis as a whole.

Searching Wiki Feeds

Tim Oren picks up the RSS deficit in wiki land, via a Google translation of a German post and Dave Johnson post where Scott Rafer comments:

“Much of the work to be done is on the wiki side, unfortunately. Feedster, et al, would be thrilled to make wiki changes as easy to search as everything else, but (…) the Wiki vendors need to make RSS output a standard option”

Much of this thread was started by Jeremy Zawodny's valid complaints about RSS feeds that are barely-human-readable Recent Changes statistics.  He picks on the Channel9 feed, but its a common feature for wikis.

Socialtext was one of the first to provide RSS feeds for Recent Changes (partially because Steve Gillmor was bugging me for them).  We chose RSS 2.0 full text feeds as the first implementation in recognition of how news aggregators were adding track changes, which complements the diff of History when logged into the Workspace.  You can find the same approach with Kwiki, Purple Wiki JSP Wiki and other open source wikis by now.

The problem is in high volume wikis, getting a copy of every changed page is too burdensome, a problem noted by Jeff Nolan (btw, go read his 10 questions to ask a VC).  This part of the reason we offered tightly integrated group weblogs within Socialtext.  Any wiki page can be added to a weblog which has its own RSS feed.  One of our users created a convention called a Track Blog, where instead of flagging or bookmarking things of interest, they add it to their own blog (like a Watchlist) which pings them when there is an update.

The Pull Model of attention management puts the user back in control of what consumes their time.  Email notifications at the interval of their choosing, RSS the subscribe to, and more imporantly, unsubscribe from on their own accord.  To state it once again, RSS is pull, not push.  The model only works when a user can leverage:

• Transparency -- when everything is on a need-to-know and C.Y.A. basis, occupational spam proliferates and social discovery suffers.  When people work openly you can browse the periphery of your attention when its less scarce.
• Amplification -- when other people find something of interest they can edit it or link to it to bring back to top of group mind.  In other words, when you miss something in a first scan, there is a greater chance people will bring it to your attention. First order merits of attention are usually personal, covered by email and IM. Second order merits of attention are more difficult to judge at first pass and are best offloaded to a group.
• Search -- when you have confidence in your ability to recall the past, you can focus on the critical path of the present.

Which brings me back to Scott's comment.  I believe we helped start a general trend for RSS in wikis and this conversation may help raise the bar again.  Even though the vast majority of Socialtext wikis are private (providing private syndication), our handful of public spaces will ping cooperatively (we ping Technorati today). 

Meanwhile, Jimmy Wales and others are working on Wikia, a wiki search engine, and Wikipedia produces a nice diff feed.  Adapting to MediaWiki covers 1/4 of public wikis.  There are well over 100 open source wikis, a wonderful diversity to respect, and search engines would do well to adapt to them over time just as they have with less standard blog implementations.

Tim's basic point was Wikis do not supply contentful RSS feeds.  I'd suggest that blog search engines have had the ethic of just ping us and feed us, we'll do the rest -- which should apply not only to blogs, but wikis and whatever else we dream up.

As almost a side-note, I should mention that the wiki world isn't wild about nofollow for at least one simple reason. On a blog you have an author and the audience (commentators?).  Within a wiki, everyone is an author.  We are still evaluating where we will use nofollow, I personally see it as  great industry cooperation creating a tool to use.

Nofollow is Leadership

Wow.  Now here is a case of cooperation over competition. Six Apart, Google, Yahoo, MSN, WordPress, Userland and others worked together to support the nofollow tag to reduce the value of comment spamming.

Reminds me of Anti-links and VoteLinks, an interesting affordance.  You could use Nofollow to link to someone you disagree with and not have it contribute to their PageRank. 

UPDATE: Scoble makes a similar point to anti-links.  Sunir, who is actively involved in fighting wiki spam, raises some really significant issues.  Pete suggests VoteLinks to negatively rate spammers may be more effective. 

Best thing about cooperation is that it can be built upon.  Good thing too, as curbing link spam will take many iterations in practice and this is a change in the structure of the web itself.