I’ve mix phx.new ed many applications and when doing so I often start with wondering how to organize my code. I love how Phoenix pushes you to think about th...
Analyzing Twitter with Neo4j and Rails
Having recently become interested in making it easy to pull data from Twitter with neo4apis-twitter I also decided that I wanted to be able to visualize and browse the data that I’ve downloaded. Therefore I created the twitter_analytics rails app!
What cool stuff does this do?
My first goal was to create a dashboard. I had run across Keen IO’s bootstrap dashboard templates (on twitter, of course) and decided this would be a great chance to use one. I liked the split-centered template the best and, after converting it to slim, I built a nice DRY way to dump whatever chartkick charts I wanted to in each panel (see the source)
I used the twitter API to dump tweet card HTML into my views which make tweets visually interesting. I cache the results in the database because the API is relatively slow and it helps avoid hitting API query limits. Also as a bonus I noticed that tweets with included images seem to always go to the “Top Tweets” metrics, so that give me some insight as well.
Hashtag graph visualization
On the hashtag detail page you can also see a graph visualization (using sigma.js) of the local network of related hashtags up to two hops away. In this case “related” means that the hashtags are both used in at least one tweet. The current hashtag is shown as a large white circle, the directly adjacent hashtags are medium sized green circles, and the hastags two hops away are small red circles. Also the thickness of the line between two hashtags represents how many times those hashtags were used together.
When you browse to the user detail page you can see all of the user’s properties from the API. This overview also has graphs to show top hashtags and original/retweet ratios for the user’s tweets.
How to use it
No doubt this gotten you excited to analyze some tweets! All you need to do is:
- Clone the github repository
- Download and start the neo4j server (see the README)
- Create a
config/twitter.ymlfile (see config/twitter.yml.example)
- Import data from twitter (again, see the README)
- Start up the app and view the results (by default at
Of course you can always browse the data using neo4j’s excellent web console (by default at
Want to help out?
The app is still a work in progress so I welcome any contributions! There are some ideas and issues on the github issues page for a start.
There are also some really cool high level things which I’d really like to see such as:
- Centrality analysis to find important users, hashtags, and tweets
- Cluster analysis to identify groups of users
- Natural language processing to expand on hashtag analysis
2020Back to Top ↑
For just over a year I’ve been obsessed on-and-off with a project ever since I stayed in the town of Skibbereen, Ireland. Taking data from the 1901 and 1911...
A while ago my colleague Michael suggested to me that I draw out some examples of how my record linkage algorithm did it’s thing. In order to do that, I’ve ...
Last night I ran a very successful workshop at the Friends of Neo4j Stockholm meetup group. The format was based on a workshop that I attended in San Franci...
In my last two posts I covered the process of importing data from StackOverflow and GitHub for the purpose of creating a combined MDM database. Now we get t...
In my last post I said I would “bring in another data source, show how I linked the data together, and demonstrate the sort of bigger picture that one can ge...
Joining multiple disparate data-sources, commonly dubbed Master Data Management (MDM), is usually not a fun exercise. I would like to show you how to use a g...
When using neo4j for the first time, most people want to import data from another database to start playing around. There are a lot of options including LOA...
I am he as you are he as you are me and we are all together – The Beatles
When I told the people of Northern Ireland that I was an atheist, a woman in the audience stood up and said, ‘Yes, but is it the God of the Catholics or t...
“Wilkins! Yes! I’ve considered decorating these walls with some graffiti of my own, and writing it in the Universal Character.. but it is too depressing...