neo4apis

I’ve been reading a few interesting analyses of Twitter data recently such as this #gamergate analysis by Andy Baio. I thought it would be nice to have a mechanism for people to quickly and easily import data from Twitter to Neo4j for research purposes. Like a good programmer I had to go up at least one level of abstraction. Thus was born the ruby gems neo4apis and neo4apis-twitter (and, incidentally, neo4apis-github just to prove it was repeatable).

Using the neo4apis-twitter gem is easy and can be used either in your ruby code or from the command line. neo4apis takes care of loading your data efficiently as well as creating database indexes so that you can query it effectively.

Using ruby

First you need to make an instance of Neo4Apis::Twitter. Pass in a Neo4j::Session from the neo4j-core gem specify the database to import to:

neo4japis_twitter = Neo4Apis::Twitter.new(Neo4j::Session.open,
                                          import_retweets: true,
                                          import_hashtags: true)

From there we can initialize a client from the twitter gem and import some data:

twitter_client = Twitter::REST::Client.new ...

neo4japis_twitter.batch do 
  twitter_client.search('ebola', result_type: 'recent').take(100).each do |tweet|
    neo4apis_client.import :Tweet, tweet
  end
end

This will perform a search using the twitter gem and for each tweet it imports:

  • The tweet
  • The tweeter

Because we set the import_retweets and import_hashtags options to true above, it will also import:

  • The original tweet if a retweet
  • The tweeter for the original tweet
  • The hash tags from the tweet

These are all loaded as nodes into the neo4j database and connected with relationships as appropriate.

Using the command line

If you don’t want to bother with writing the ruby code, some commands are provided for you! After installing the gem and setting up the twitter authentication config file (see the README), you just need to perform your search:

neo4apis twitter search "hugs and puppies" 50 --import-retweets --import-hashtags

That will import 50 tweets (and associated users, and retweets, and hashtags, etc…) for the “hugs and puppies” search into neo4j! Note that you can see the documentation for the search command by typing:

neo4apis twitter help search

There is also a command to stream tweets continuously:

neo4apis twitter filter "ballooning etiquette"

Data will be continuously imported until you hit Ctrl-C. Again, for help with this command you can type:

neo4apis twitter help filter

That’s it! You want data? You’ve got data! From there you can just jump right into the awesome Neo4j web console or use the also awesome neo4j and neo4j-core gems to start playing with the data!

2020

Structuring an Elixir+Phoenix App

I’ve mix phx.new ed many applications and when doing so I often start with wondering how to organize my code. I love how Phoenix pushes you to think about th...

Back to Top ↑

2015

Master Data Management Scoring Examples

A while ago my colleague Michael suggested to me that I draw out some examples of how my record linkage algorithm did it’s thing. In order to do that, I’ve ...

Loading SQL to Neo4j Like Magic

When using neo4j for the first time, most people want to import data from another database to start playing around. There are a lot of options including LOA...

Back to Top ↑

2014

Normalizing Religion in Ireland

When I told the people of Northern Ireland that I was an atheist, a woman in the audience stood up and said, ‘Yes, but is it the God of the Catholics or t...

Back to Top ↑