Analyzing Twitter with Neo4j and Rails

Having recently become interested in making it easy to pull data from Twitter with neo4apis-twitter I also decided that I wanted to be able to visualize and browse the data that I’ve downloaded. Therefore I created the twitter_analytics rails app!

What cool stuff does this do?

Dashboard

My first goal was to create a dashboard. I had run across Keen IO’s bootstrap dashboard templates (on twitter, of course) and decided this would be a great chance to use one. I liked the split-centered template the best and, after converting it to slim, I built a nice DRY way to dump whatever chartkick charts I wanted to in each panel (see the source)

Tweet cards

I used the twitter API to dump tweet card HTML into my views which make tweets visually interesting. I cache the results in the database because the API is relatively slow and it helps avoid hitting API query limits. Also as a bonus I noticed that tweets with included images seem to always go to the “Top Tweets” metrics, so that give me some insight as well.

Hashtag graph visualization

On the hashtag detail page you can also see a graph visualization (using sigma.js) of the local network of related hashtags up to two hops away. In this case “related” means that the hashtags are both used in at least one tweet. The current hashtag is shown as a large white circle, the directly adjacent hashtags are medium sized green circles, and the hastags two hops away are small red circles. Also the thickness of the line between two hashtags represents how many times those hashtags were used together.

User summary

When you browse to the user detail page you can see all of the user’s properties from the API. This overview also has graphs to show top hashtags and original/retweet ratios for the user’s tweets.

How to use it

No doubt this gotten you excited to analyze some tweets! All you need to do is:

Of course you can always browse the data using neo4j’s excellent web console (by default at http://localhost:7474):

Want to help out?

The app is still a work in progress so I welcome any contributions! There are some ideas and issues on the github issues page for a start.

There are also some really cool high level things which I’d really like to see such as:

  • Centrality analysis to find important users, hashtags, and tweets
  • Cluster analysis to identify groups of users
  • Natural language processing to expand on hashtag analysis

Happy analyzing!

2021

How Far Can I Push a GenServer?

I’ve been using Elixir for a while and I’ve implemented a number of GenServers. But while I think I mostly understand the purpose of them, I’ve not gotten t...

Why I Love Lodash

I love Lodash, but I’m not here to tell you to use Lodash. It’s up to you to decide if a tool is useful for you or your project. It will come down to the n...

Back to Top ↑

2020

Structuring an Elixir+Phoenix App

I’ve mix phx.new ed many applications and when doing so I often start with wondering how to organize my code. I love how Phoenix pushes you to think about th...

Back to Top ↑

2015

Analyzing Ruby’s ObjectSpace with Neo4j

Recently the continuous builds for the neo4j Ruby gem failed for JRuby because the memory limit had been reached. I wanted to see if I could use my favorite...

Master Data Management Scoring Examples

A while ago my colleague Michael suggested to me that I draw out some examples of how my record linkage algorithm did it’s thing. In order to do that, I’ve ...

Loading SQL to Neo4j Like Magic

When using neo4j for the first time, most people want to import data from another database to start playing around. There are a lot of options including LOA...

Back to Top ↑

2014

Analyzing Twitter with Neo4j and Rails

Having recently become interested in making it easy to pull data from Twitter with neo4apis-twitter I also decided that I wanted to be able to visualize an...

neo4apis

I’ve been reading a few interesting analyses of Twitter data recently such as this #gamergate analysis by Andy Baio. I thought it would be nice to have a ...

Normalizing Religion in Ireland

When I told the people of Northern Ireland that I was an atheist, a woman in the audience stood up and said, ‘Yes, but is it the God of the Catholics or t...

Back to Top ↑