Sift’d Blog

Sometimes getting the big picture means you have to head to Google after reading an article so you can find out more, and that’s just wrong. Context is invaluable, especially in this day and age when we’re flooded with news from so many sources.  It’s also sorely missing from a lot of websites these days - many popular news websites and blogs just don’t want to link to anyone but themselves.

Right from the start we’ve had a mechanism in place for users to add related links to a story. You can also vote them up or down, and the top 3 are displayed with the story. Up until now though it’s been something you had to do manually after a story is submitted.  That’s a little bit more complicated than it needs to be.

So we’ve made it even easier. When you submit stories now in the last step (previewing your submission) you’re presented with some related links you can automatically add to the story you’re submitting. We’re pulling these links in from Google News, but in the future we might also check Technorati so you can get the blogosphere’s perspective on a story too.

It looks great too! Here’s a screenshot:

Discuss this on Sift’d

We’ve been quiet the last week because we’ve been working overtime on a significant improvement to Sift’d. The new layout really focuses on making it more intuitive and easier to get the latest news and interesting links that have surfaced to the top. It’s presented more like a traditional news website rather than a social news site but still powered completely by the community.

There are a few bugs that we’re aware of, mostly to do with the dropdown menus in IE 6 and 7 and a couple of other CSS issues. Tomorrow we’ll start going through the site with a fine tooth comb and sort out any issues we come across. The good news is the code base seems to be pretty solid now, we’re only a few weeks in to the beta and aren’t finding that many bugs.

Feedback is always wanted, please let us know how the layout can be improved if you’ve got some ideas.

Discuss this on Sift’d

What is OpenID?
OpenID is a system that lets you have one login name you use on any website that supports OpenID. The easiest way to think about it is like Microsoft’s Passport service that lets you sign in to any Microsoft web property using the one account.

There’s a couple of cool things about OpenID that we really like. The first is you don’t have to remember all those different username and passwords, you just have the one username and password with your OpenID account. The other thing we love is that it reduces the information you have to give websites - because of the switch we don’t need to maintain passwords, email addresses or secret questions and answers. That’s less information for you to enter when you register and it’s less information for you to manage.

The other thing we love about OpenID is that you’ve almost definitely got one already. Some really huge names are behind the push to get OpenID out there - anyone using Yahoo, AOL, Facebook, WordPress, Blogger or Smugmug already has an OpenID! For the 27 people in the world who aren’t using those services you can create one at myopenid.com very easily, which you can then use on the websites that supports OpenID. Like we do now!

Yahoo has probaby the best plain English description of OpenID I’ve seen. So here it is verbatim:

Are you tired of creating a new account on every website you use? Do you avoid new websites because they come with yet another username and password? Do you paste stickies with password hints all over your computer monitor?

OpenID is an open technology standard that solves all of these problems. The OpenID technology will allow you to use your Yahoo! account to sign in to hundreds of websites! And this list is growing every day…

Once you enable your Yahoo! account for OpenID access, you can simply tell any OpenID enabled website that you are a Yahoo! user. You will be sent to Yahoo! to verify your Yahoo! ID and password and then signed in to the website. Its that easy!

What does this mean for our awesome users?
The first thing you’ll notice is the sign in page is very different now. There’s a link to the old sign in page that will let you continue logging in with your username and password. When you sign in you’ll notice a message saying you’re going to have to associate an OpenID account with your Sift’d account at some stage. There’s really no hurry on that, we said by the end of July but we’ll keep providing the old authentication route until it’s no longer required.

When you associate your OpenID account we remove the old information we don’t need anymore. That means we stop holding that password you use for everything from internet banking to eBay. For the record that information was encrypted anyway.

What does this mean for new users?
It means registering just got easier. You enter your OpenID, when you get back from authenticating at your provider all we ask for is a username, your date of birth and country. Just three little pieces of information which you can control access to (except your username) via your privacy settings in Account > Privacy.

What does this mean for Sift’d?
It means one day we can delete THREE fields from our database and SIX stored procedures. Basically we all win!

We think OpenID is just a fantastic idea and with the giants of the internet stepping in as identity providers it’s up to the little guys to take the plunge and become relying parties (that means you use your identity with us). The more websites that support OpenID - and there are tens of thousands already - the easier our lives as users will be.

I for one look forward to the day where I don’t have to remember usernames and passwords for the sites and services I use. Especially when I have to have numbers and letters tacked on to my username because someone beat me to it.

One of our key features is the ability to create filters that remove content you’re sick of seeing or that just doesn’t interest you. We provide users with the ability to create filters that block users, websites, categories, media types, and comprehensive keyword filtering. Users can also create advanced filters that use any combination to filter out very specific content, like videos submitted by a certain user in The Lounge but not anything else that user submits.

Our newest addition to the site combines our powerful filtering options with a unified RSS aggregator as a power tool for social news junkies but also a useful tool for anyone being overwhelmed by too many websites publishing too much stuff that doesn’t interest them. This allows users to add all of the feeds they subscribe to, have them merged into one feed and processed against their text filters to remove anything they don’t like. Feed items have direct submission links appended to them for all the popular social bookmarking and news websites so you can be the first to submit the stories at the social websites you use.

New feeds can be added just by pasting in the feed url, and users can unsubscribe from any feeds they’re tired of with one click.

A bunch of bugs have surfaced and been fixed during our first week of live testing, and additionally some new features, tweaks and code refactoring (particularly with the JavaScript) have been implemented.

There were several bugs during the submission process which were resolved yesterday. Most of them had to do with the JavaScript that powers that process, and it underwent a complete overhaul yesterday.

Some URL rewriting has been implemented to make the profile pages more friendly looking, profiles are now /users/{username}, with the sub-pages being /users/{username}/{subpage} and the RSS feeds /users/{username}/{subpage}/rss.

The submission process was updated to include one of those ‘loading’ animated gifs for thumbnails and during the URL validation process. The URL validation process can take a few seconds because while it checks the URL exists it also examines the page to find images and YouTube videos. For the thumbnails this was a bad oversight - they’re generated on the fly until the submission is saved and that can mean the servers are downloading and resizing any number of large images from any number of servers, which so far is showing to take a few seconds.

The widgets shouldn’t let text be selectable as they’re dragged around the screen anymore. There is still a bug with the Activity Spy widget which we haven’t gotten around to fixing yet but it’s on our to-do list.

The way stories are presented underwent a change to further distance us from the look and feel of Digg. We’re playing with a few ideas on how to really add some panache there and to spice it up with some live updating in the background.

We added a “Share” menu to each story so you can easily get the original URL or the discussion page on Sift’d over to the social bookmarking and news services you’re using.

Other minor bugs were found and fixed along the way, submitting stories originally logged a score of 0 when it should have been 1 for instance, I don’t actually recall the others but I know there were a few.

There are a few more areas we’re looking at improving, and a few secret features we’re working out that’ll make Sift’d a valuable asset in your online experience. Searching hasn’t been implemented yet, at the moment it’s not vital and we’re hanging out to see what happens with Google launching hosted site searching.

Overall we’re on track for making Sift’d a great site, and our fledgling community is doing a wonderful job submitting interesting links and building the foundation of a site and an experience we can all enjoy.

We’ve implemented a bunch of features which we hope will make Sift’d unattractive to spammers. Our main goal with spam is to remove the incentives that attract spammers in the first place, which primarily is the benefits they get with search engines when they post their links on websites.

Upcoming and failed submissions have the rel=”nofollow” attribute appended to their source link so there is no SEO benefit to submitting stories unless they are made popular. By removing the “free backlink” incentive from stories that don’t deserve it we remove the reward dodgy search engine optimizers reap from sites that allow them to submit links.

Each story can have related links added to them that users are also able to vote on. Any related link with a negative score is replaced with a url shortening service that redirects to the original url, but which search engines won’t attach any weight to. Links in comments are also piped through a url shortening service to make comment spam less attractive.

Additionally, each user can create their own set of filters to block any users or websites they feel are spam.

SiftKarma is also a powerful tool to prevent spam. Each time people bury a spammer’s submissions and comments the spammer has less and less weight on the site, making it harder for their submissions to reach the comment place and not displaying their comments once they reach a certain negative threshold.

None of these features actually prevent spam or hinder site usage in any way, but they do give us a foundation to work from that we can develop further as needed to combat spam. One idea we’re toying with is to create a system that detects frequently blocked users and websites and blocks them from using the site until they can be human-reviewed, after which they can be reinstated or permanently blocked.

We welcome all discussion and ideas on how spam can be beaten without compromising usability and honest users’ experience.

The front paging algorithm is very simple.  Every 15 minutes the top upcoming stories in each section are examined.  We automatically exclude the category, user and source that was last made popular in that section.

Each story has an invisible score that is the sum of each voter, burier and commenter’s SiftKarma at the time that they committed the action on that story.  From that score and the age of the story we calculate the momentum each story has and if it is greater than or equal to the threshold for that section then it is made popular.

The threshold for each section is a sliding scale.  It is 20% less than the score that was previously made popular in that section.  If no submissions meet that criteria it is reduced by 20% to lower the barrier for the next time the software runs 15 minutes later.

For making stories featured we follow the same process but by examining stories that have been made popular in the last 2 hours.

Discuss this on Sift’d

Every user at Sift’d has a value between 1 and 100 percent that is their SiftKarma. Every time they vote or comment they ’spend’ some of their SiftKarma, and every time another someone votes on their submissions and comments they ‘earn’ more.

We think this is going to be a great way to combat spammers because people who submit junk that gets buried will end up with less and less weight each time.

It also effectively deals with anyone who makes multiple accounts. If a person makes 40 accounts and tries to get their submission front paged it might work - once or twice - but eventually their 40 accounts will have a combined weight of one user with average SiftKarma. Their 40 accounts, even if they’re used to +vote each other, result in a net loss for each account.

The only way a person could maintain their deception would be by having each of their accounts participating on the site and receiving positive votes by real users which might be feasible for a couple of accounts but maintaining multiple personas that attract positive attention is likely to be impossible for any lengthy period of time.

People using services like usersubmitter.com are also going to find themselves out of luck unless they can devote time into maintaining good SiftKarma to offset what they use selling their votes.

Normal users shouldn’t suffer much, if any, penalty because of SiftKarma. Anyone using the site submitting stories, voting and commenting is going to attract positive and negative votes from other users that should keep them well balanced - unless they’re habitual trolls anyway.

Discuss this on Sift’d

Discuss this on Sift’d

A couple of sites have tossed this idea around while reviewing us and we figured from the start it was going to happen - the always cool CNET blog WebWare and Killer Startups - as well as some submissions across the tubes on other social sites. We don’t think we are. Digg was definitely a part of the inspiration that led to Sift’d, but we have created a very unique interpretation of the social news, user-powered concept.

To call Sift’d a clone gives us too little credit and Digg far too much. The real inspiration came from Digg’s users, of which we are a part. We’ve all seen the site gradually transform into something new, and some of us aren’t fond of the direction it’s moving in.

There is room for improvement on many features on Digg (and of course on our site) but above all we don’t think they’re providing for all the different demographics that make up their community. Various categories and groups dominate the site and the only recourse users have - confirmed during their Townhall meeting the other night - is to turn off entire categories?

The single most defining difference between Digg and Sift’d is we’re in it for the end users. We wanted to create an environment where the lolcat, Obama, Ron Paul, Apple fans and every other flavour of fanboy can coexist without sacrificing anyone else’s experience because they’re part of a smaller demographic. We’re also committed to being and remaining very open about how the site works, and the data we’re collecting, Above all, we’re committed to working with our community to develop the site in the directions the users want.

To achieve that, we’ve created a collaboration section which will be a permanent and constant two-way communication channel between our users and our developers. We will also be launching a live development version of the site to sit in parallel, and provide users a chance to explore, test and criticize as we develop and enhance new features. An API allowing unfettered access to almost our entire database is in the works, and will be launching some time in the not too distant future.

Sift’d is packing a considerable arsenal of features that are all designed to enrich your experience on the site. Digg users will recognize some of them - we’ve been begging for them for a long time. You can get a good overview of our features here. To cater to a diverse audience we made a bunch of decisions that ended up being pretty cool and implemented some unique behind the scenes logic and programming to make everything much better for our users.

Users can create filters that allow them to block websites, other users, keywords in the title / description / both, categories and media types. They can also create advanced filters that use any combination of those criteria.

We also designed the site from the ground up so that each section has it’s own independent front page. Popular stories move from the upcoming queue to the front page of each section, and from the front page of each section to the home page. The sections don’t compete with each other. In other words, a flood of political news doesn’t mean every other section gets less fresh content.

We also have prevented consecutive categories from being made popular. That leaves room for popular categories to play a larger role in their section, but not to overwhelm it. Between the filters and the independent sections we think we can produce an experience that satisfies a large group of people. As trends appear and new sections become necessary we’ll add them too.

We want to give our users direct access to our developers. We think it is important that the users have the ability to bypass our marketing and operations people. Although those people are very important to us, they usually are the stumbling blocks in most companies that slow down the process of giving users exactly what they want.

Discuss this on Sift’d

Sift’d is one of those social buzzword sites you hear about on the internet. I guess technically you could call it a “social news site” but it’s a lot more diverse than that.

We only started our public beta test yesterday - not even 24 hours ago yet - so please be patient while we drum up some users and get the content flowing.

If you’re new to this type of site the goal is to have lots of people submitting lots of news and interesting links, and voting on the stuff they find the most interesting. The most popular stuff gets featured on the section front pages, and the really popular stuff gets featured on the home page.