User21

Following my previous post, my FriendFeed crawler is ready. Well, at least a version 0.1 of it. Actually it didn’t take too long to develop and it was a nice exercise.

In any case I have sent it to crawl the FriendFeed main feed at regular intervals and I should be getting some initial results soon. I will share them of course but first of all we need to understand the methodology and limitations of my analysis.

How does the crawler work?

The crawler starts with the current public feed. For each entry it extracts (discovers) the poster as well as usernames of people who liked or commented on that entry. For each user discovered the crawler reads his or her subscriptions list and keeps on going from there.

So generally speaking the process is: 1) read the feed 2) discover users 3) extend discovery through subscriptions 4) repeat.

What kind of data is collected?

The crawler generates a long list of pairs where each pair represents a single subscription, a relation between a subscriber and the user he is subscribed to. For example, the relation “Robert Scoble -> Michael Arrington” means Robert Scoble is subscribed to Michael Arrington’s feed. Given enough data, I should be able to tell you who else is subscribed to Arrington, at least among the relatively active users.

What are the imitations?

1. Only active users are covered: if you open a new FF account, you have no other user subscribed to you and you do not publish/like/comment on anything then there’s no way my crawler can discover you. In my view this is a good limitation since it narrows the analysis down to the interesting users, excluding the inactive ones.
2. Only public users are covered: if your feed is private I cannot read your subscription list or do anything of value for that matter.

Any interesting numbers to share?

The crawler is running. I will test it, run it once or twice, and then share the results when I decide I’ve reached critical mass.

While working on a social media trends project (I’ll probably write more about it in the future but for now there’s not much to say or show), I had this idea that I think I’m going to investigate.

FriendFeed lets you see a user’s subscriptions but it doesn’t let you see which users are subscribed to that user. For example, you can see who Robert Scoble follows, but you can’t see Scoble’s followers (the list of users subscribed to his feed).

So, I’m thinking maybe I can crawl the public feed, discover users and read their subscriptions. Each subscription will point at other users to discover and so on. Once I get enough data, I can just look at the list of subscriptions from the opposite direction (from the subscribers point of view), meaning I would be able to see which users and how many are subscribed to a user.

This will effectively give an estimate as to the “social weight” of that user, at least among the active and discoverable users of FriendFeed.

Interesting. I’ll get to work on it as soon as I can and share the results.

Lately I’ve been playing with the FriendFeed API. It is generally a well written and responsive API. There are a few bugs and limitations (I even found a little bug in the C# library).

The 2 most annoying limitations for me are:

1. You cannot retrieve entries by a date range, you can only specify a start position and how many entries you want to retrieve. So if you’re trying to scan backwards you need to jump through hoops to decide when to stop.
2. The “page 11″ limitation - try browsing FriendFeed and navigate through previous pages. Page 9, page 10, page 11, page 12 … What’s that? Clicking anything past page 11 just gives you the same results over and over again. With a limitation of 30 entries per API request this means you can get a maximum of 300 entries.

No other major rants at this point.

Let’s just skip to the second post.

BTW, welcome to this blog.