There’s a new digital divide, a fissure opening wider and wider as the social web makes encroachment into most forms of information. You may have heard of the ‘darknets’ — unseen networks of computers for filesharing — networks you’re only allowed onto if you’re trusted not to give the game away. What I think I’m seeing the emergence of is almost the exact opposite, but increasingly disconnected.

There are hundreds of websites, lovingly researched and maintained by enthusiastic and knowledgeable people, that it’s becoming almost impossible to find. The sites are built on old technology, and that contributes to their decreasing visibility but it’s not the only reason. The lack of RSS feeds, pinging servers, dynamically generated sitemaps and up-to-date robots.txt files makes it more difficult for other sites to keep in touch with them. That they are often built in HTML by hand makes them more difficult to update, and fresh content is prized by search engines.

The lack of RSS and knowing when updates occur also decreases people’s awareness of the sites, you either have to remember they’re there and how you found them — or bookmark — and check for updates on a regular schedule. Less reminders, less nudges, so less incoming links.

They are often in very niche areas, like local history, local news, and so generate a limited number of hyperlinks from other sites. They often only get links from each other, which is great in a community sense but means the incoming links are low in pagerank (which would push them higher up Google searches).

As more and more sites get better and better at search engine optimisation, as blogs and other social websites link and link again and expand into more areas, and as Google relies on the same sources more and more the sites are getting less and less visible.

And that’s bad because they have a wealth of important content that we need to be able to find.

I’m calling it the ‘hinternet‘.

(From hinterland, in German the part of a country where only few people live and where the infrastructure is underdeveloped.)

Solving the problem is a tricky one: Google’s mission to index the entire World’s information doesn’t always mean that we can find what we’re searching for, the semantic web will only work if the correct metadata is stored with the hinternet sites (and they’re already often “behind” technology-wise).

Search needs to get better, but us on the social-web also need to help. We need not only to link to these sites, but — where we can — help nudge the guardians of the hinternet towards greater visability by becoming “social”.

October 15th, is Blog Action Day. The concept being to encourage blogs around the world to all write about one issue on the same day - hoping that through the various niches (social media types, big shoes, ZX Spectrum nostalgia) the message will reach as many people as possible. This year the theme is poverty.

Blog Action Day is a fine example of “organising with out organisations” as Clay Shirky puts it, how social media can facilitate a co-ordinated effort without the need for huge hierarchies. Started only last year and by only a couple of committed people (who all have other jobs) and now in its second year there are 8,240 Sites with a total of regular readers 9,203,161. Those nine million plus are subscriber numbers by RSS — demographics-wise I would say that puts those 9 million in the most digitally savvy groups of people that there are.

The web allows ideas to spread quickly, social media helps people to connect quickly, to collaborate on actions. It could usher a new era of awareness, or protesting could become so easy that it ceases to mean anything.

When talking of the internet and poverty the ‘digital divide’ is what is often focused upon. There really a couple of different things that this covers: the first, inescapably, is poverty. Some people are to poor (or too isolated in undeserved areas) to be able to get access to computers. This is where governmental effort or philanthropy is needed — moves like the OLPC (One Laptop Per Child) might do it, eventually, but my best guess is that the technology will change faster than any initiative.

UK Twitters awoke to a email this morning telling them that updates by text were no more (follow the reaction on Twitter itself). Unlike in the US where the standard has been to pay to receive texts, Twitter has been stumping up the cost of updating UK subscribers. That, according to Twitter has become a cost too much to bear:

It pains us to take this measure. However, we need to avoid placing undue burden on our company and our service. Even with a limit of 250 messages received per week, it could cost Twitter about $1,000 per user, per year to send SMS outside of Canada, India, or the US. It makes more sense for us to establish fair billing arrangements with mobile operators than it does to pass these high fees on to our users.

Whatever you think of their calculations, you can purchase SMSs much cheaper in bulk (for fractions of a penny) and I’m not convinced that it’s a huge number of people with text updates on at all times, if they can’t afford it then that’s fair enough. Negotiating with the carriers for a cheaper rate doesn’t seem the best solution to me, even if they reduce costs per user growth in Twitter usage will eat into their funding.

Better to take this opportunity to introduce advertising in some form.

If a third party service was to set up and use the Twitter API to send out free text updates, but send one advert for every ten (at the same time as a real update so you didn’t get a false alarm) I’m guessing it would have both a revenue model and a lot of sign-ups. If it introduced additional functionality (perhaps a contacts directory/speeddial thing so you didn’t have to keep typing out @conmpl1cated_names-withpuntuat|on), then it would be adding value to the SMS-based service.

So, perhaps Twitter could do it too — they at least have unfettered access to the API and with the Summize technology they could deliver context sensitive ads. If it worked in the UK, then maybe it could even expand to countries where — at the moment — they can still afford to text you.

Sir Tim Berners-Lee has a Mac, he doesn’t use Skitch though, or Adium, in fact his menu bar was free of clutter when I saw him talk at NESTA on “The Future of the Web” (it was webcast [edit now up here], this isn’t it but the content of TBLs talk was similar). I suspect his mind is fairly clutter free too, you don’t invent the web (an idea his then boss called “vague but interesting”) without a talent for seeing through the mess to a simple system.

Despite that the web is complex, in fact TBL likened what the web has become to (and I roughly quote here and later, from my notes):

“that sort of gunky-hairy thing that you get out when you unblock the sink. You don’t know how it got that tangled, and you couldn’t have planned it, but there are lots of ways for bugs to get from one point to another.”

It’s also huge. He showed us a slide of the exponential growth in the first couple of year and there’s no sign of that slowing.

TBL (and others) have come to the conclusion that the web needs studying:

“Because it’s big, tangled and complex, and we have a duty because we created it. We have to ask ‘is it going to lead to a society that we want?’ ‘How can we engineer it so it does?’”

To do this properly, they argue (they being the Web Science Research Initiative) a new field of science “Web Science” needs to be developed, pulling together the social studies, the networking studies and the data studies. There a few interesting points on this in The Guardian’s interview.

TBL’s ‘Future of the Web’ - Semantics

TBL doesn’t like to predict with “Future of the Web”, which is why he wants to be able to study it in greater detail, his thing at the moment (and as Bill Thompson pointed out, with a question from the floor, for a long while now) is the Semantic Web. He attempts to explain it here on the Today programme (via NBSE blog and others), but it is a difficult concept to grasp. More in depth discussion here as part of a podcast.

In short (now I may be getting it wrong here) the Semantic Web is a bunch of standards that describe the data available on the Web rather than documents (pages, graphics, video and the like) and how the data may be accessed. It’s confusing and scary in a way as the distinction in our minds between the data and the documents isn’t clear. TBL calls it “the interesting bits in the documents”, he means dates, times, facts, relationships. I don’t think interesting is the right word here, for we all enjoy the flowery stuff around the facts, the soft context — but the ability to get at the facts and relationships in a standard way opens up a real progress in the ways we can communicate.

It’s starting to happen a little already, with APIs being a must have for all sorts of services, but “semantic startups are still guesswork”. Like the Web itself, the applications that will build on the Semantic Web are pretty much unknown — TBL says (and what a sentence to be able to come up with:

“When I built the web, I didn’t know what would happen. Soon – as long as we give people a place to play – there will be new innovation, something we haven’t thought of.”

There are already standards being agreed on, by the W3C (an organisation TBL formed, he says “like jumping into a bobsled to try to steer, after pushing it off”), but the data standards are designed so they can be expanded by individuals within the agreed framework.

To get the Semantic Web up and running there is a huge amount of work to do to. It’s daunting because we don’t separate data from our words and pictures naturally. Folksonomies like tagging help but, at least in the short term, if individuals are to contibute to the “web of data” it’s going to be through services designed to take it from us and process it to meet the standards.

The ‘other’ Future of the Web - the battle for control

Alongside TBL on the panel at the event were Charles Leadbeater, author of ‘We Think‘, and Andy Duncan, CEO of Channel 4. You can see why they were invited, to help provide a bridge between the intended audience (not sure that was us geeks) and the super-brain of TBL, but as others have said time with TBL was limited and precious.

Not scientists, these guys were more comfortable with talking about what they thought the “future” would bring, although in Andy Duncan’s case he didn’t seem to have many ideas (he talked of “innovation”, but didn’t manage to get any opinions across).

Charles is a very astute thinker, however — he sees the main question, the main challenge of the web to not only keep it open (echoing TBLs “ISPs - give me fast non-discriminatory access, and don’t sell my click-streams”) and to actually prove the social worth of our connections. “Can we make these open communities last, prevent creeping re-regulation… share skills and values… are we capable of self regulating?”.

I may have missed the point of Andy’s contribution, the two main threads boiled down to - paraphrasing horribly (Simpson’s-style voice in my head) “won’t somebody please think of the children” and “Google’s YouTube is pinching our content”. He was worried about the lack of control over the internet, where as I think that precise “lack of control” is what made our tangled gloopy bunch of hair what it is now.

Does anyone want to help make a quick website that could answer this eternal question, and perhaps spread a bit of environmental advice as it goes?

Along the lines of isitChristmas, but obviously localised by Post Code, the site would offer RSS and iCal feeds of whether it’s bin day for you — with reminders the day before, and telling you what week it is for recycling purposes (green or paper/plastic) for those that have differences. Along with this simple, but useful service it could impart environmental advice and info slipped into the RSS as well as somewhere on the site. It could even do calculations of stuff like “my council doesn’t collect X, is it better to just bin or drive it to the recycling centre?”.

It could get a few ads from electricity suppliers etc to pay its way perhaps.

I’ve just come back from The Big Debate (part of the New Generation Arts Festival), a cosy couple of hours listening to talk on the subject ‘Digital Uptopia — more power or powerless?’. As is the way of these things the proposition was skirted round by most. The lack of a digital naysayer on the panel might have warned the organisers that it wasn’t to be a heated discussion, I think that the twitter/liveblog backchannel (albeit composed by the local digerati, who ought to be on the ‘power’ side really — the digerati in the room were hampered by failing wifi tho’)  tried hard to counteract that but still…

Two of the panel, Anthony from the BBC and Doug from BT, would have had be commenting on the live blog like mad during their opening addresses. Anthony, who works on the iPlayer, seemed to confuse “choice” with “power”,  I was waiting for the payoff where he reconciled the two concepts but it never came. How the choice to watch the same stuff on TV (by other means) at different times equates to ‘power’ I couldn’t grasp — the day they let people chose what is made rather than transmitted would be when iPlayer effected power at all.

Where as Anthony seemed not to have got hold of the right end of the stick, I’m not sure Doug was even in the same building as the stick. He talked (again, again it seems to those of us that follow discussions of this nature — something picked up on the live blog) of the history of media and how people hadn’t looked at the problems of what he wants to call “shape shifting media” yet. Shape shifting media seems to be an IPTV version of “chose your own adventure books”, and there was much online grumbling that the 30 odd years of video gaming has been addressing exactly that. “Not quite a game, not quite a film. Somewhere inbetween.” was one of his phrases. I’m not sure that this has anything to do with power (power to be entertained in a slightly different semi-interactive way?), and I’m not sure this is anything like a laudable aim (anyone remember the pretty but boring Don Bulth games?).

There are sort of two threads to the discussion that work for me, one is whether the ‘democratising effect’ of social media does mean more power in the hands of the individual, Jo had a few good points on that from the standpoint of local ‘traditional media’, but apart from that it wasn’t overly discussed. Again I think because the panel were all of the mind that there was more power, (but think about privacy for example) — oh for a member of the No2ID lot on the panel.

The second topic, and a secondary thread to the first, is whether (accepting that internet access is empowering) there really is a “digital divide” and if so how is it best dealt with. There were interesting points from the audience on this, “was the divide one of motivation, or economics?” and if economic who should pay? A great discussion, but not one there was enough time for here.

Really, for me at least,  the true digital empowerment of the digital age for me has come at the expense of events like this. Apart from Joanna Geary, whose opinions I have come to trust though her writings and actions, the panel had to work very hard to make their points to me. In the pre-internet age, the opinions of panellists, debaters, those “selected” where the only ones heard and would be automatically given credence, but now unless the reputation of the speaker precedes them I can think of twenty people I regularly communicate online with who would tear the discussion apart with wit and experience.

It’s those voices that I want to hear and online is the only real way to get them all together.

I’ve been thinking some more about the whole, information overload, autogenerated echo, crossposting thing.

I’ve come to the conclusion that I don’t want RSS feeds aggregated for me on yet another web service, I don’t want every feed from every person and have to filter them out (and for duplicates). In short I want all my information in one place, custom search feeds and the like as well as people’s RSS, news as well as flickr tag feeds.

I like the Google Reader experience, I like that it’s in sync across my laptop, my phone, other computers. Google Reader could blow FriendFeed and others away if it implemented a few new features.

Here’s my new feature wishlist for Google Reader:

• The ability to filter feeds as the come in (by location would be great, I have a lot of searches for “Birmingham” and only want the UK versions).
• The ablitity to remove duplicate items from different feeds (and chose which “original” version remains). Two examples: blog/news results in my search feeds when I already subscribe to the originating feed. Also removing auto generated posts: twitters in friends’ Facebook statuses or “daliy links” posts in blogs when I already subscribe to the del.icio.us feed.
• Filters to “mark as read” posts (similar to GMail). By tag would be fine — Google Reader’s search feature is brilliant (allowing you to seach within everything that’s come through), there are things I’d like to be able to search (obsure news feeds, heavy feeds like Digg content) but I don’t want to have them as outstanding posts to be read. You’d be building up your own subset of the web.
• See other people’s notes (if shared of course) — the new notes feature is great, you could have a conversation with the notes if you could see other peoples’. A little like the “comment on anything” stuff that people are hot for on FriendFeed.

I’ve been quietly impressed with Get Satisfaction, which is sort of best described as a “social customer service” site. Twitter and some other big-name players on the internet use it for their official support channels - the idea of the site being that employees of the companies join in with discussion of “problems” that people are having. Some employees just join to help, others are granted “official” status and can speak on behalf of the organisation.

Of course lots of problems that we have with products or services aren’t really problems (or are well know and documented) - in these instances other users are happy to help (very much like unofficial forums for software). ‘Users’ are also welcome to point out possible solutions to anything - and of course they do.

So, I thought, could this work for a local council? Imagine time saved by council officials if knowledgeable citizens helped answer questions, imagine the resources available (once someone had explained how to apply for a licence, the information would be there for everyone), imagine a monolithic body “joining the conversation”.

Rather than deciding to attempt to persuade my local council (Birmingham City Council - one of the largest in the UK) that this would be a good idea, I discovered that - as the site is “a space for an open conversation between you and other people with interests and passions in this organization.” - anyone can set a company page up. So I have.

I don’t have anything to ask at the moment, but I’m hoping that it might get used.

“Sometimes representatives from the company or organization may take part in the conversation too.” says the blurb — wouldn’t that be great?

Now (I know it’s been doing it a while) that Facebook sends the text of messages to your email, it really needs to do something whereby a read-receipt marks it as read in FB itself. Having to do that manually to messages I’ve read is just one more annoying thing that is meaning I use it less and less.

This and other mapping at CR Blog.