I’ve mix phx.new ed many applications and when doing so I often start with wondering how to organize my code. I love how Phoenix pushes you to think about th...
I’ve been reading a few interesting analyses of Twitter data recently such as this #gamergate analysis by Andy Baio. I thought it would be nice to have a mechanism for people to quickly and easily import data from Twitter to Neo4j for research purposes. Like a good programmer I had to go up at least one level of abstraction. Thus was born the ruby gems neo4apis and neo4apis-twitter (and, incidentally, neo4apis-github just to prove it was repeatable).
neo4apis-twitter gem is easy and can be used either in your ruby code or from the command line.
neo4apis takes care of loading your data efficiently as well as creating database indexes so that you can query it effectively.
First you need to make an instance of
Neo4Apis::Twitter. Pass in a
Neo4j::Session from the
neo4j-core gem specify the database to import to:
neo4japis_twitter = Neo4Apis::Twitter.new(Neo4j::Session.open, import_retweets: true, import_hashtags: true)
From there we can initialize a client from the
twitter_client = Twitter::REST::Client.new ... neo4japis_twitter.batch do twitter_client.search('ebola', result_type: 'recent').take(100).each do |tweet| neo4apis_client.import :Tweet, tweet end end
This will perform a search using the
- The tweet
- The tweeter
Because we set the
import_hashtags options to
true above, it will also import:
- The original tweet if a retweet
- The tweeter for the original tweet
- The hash tags from the tweet
These are all loaded as nodes into the neo4j database and connected with relationships as appropriate.
Using the command line
If you don’t want to bother with writing the ruby code, some commands are provided for you! After installing the gem and setting up the twitter authentication config file (see the README), you just need to perform your search:
neo4apis twitter search "hugs and puppies" 50 --import-retweets --import-hashtags
That will import 50 tweets (and associated users, and retweets, and hashtags, etc…) for the “hugs and puppies” search into neo4j! Note that you can see the documentation for the
search command by typing:
neo4apis twitter help search
There is also a command to stream tweets continuously:
neo4apis twitter filter "ballooning etiquette"
Data will be continuously imported until you hit
Ctrl-C. Again, for help with this command you can type:
neo4apis twitter help filter
2020Back to Top ↑
For just over a year I’ve been obsessed on-and-off with a project ever since I stayed in the town of Skibbereen, Ireland. Taking data from the 1901 and 1911...
A while ago my colleague Michael suggested to me that I draw out some examples of how my record linkage algorithm did it’s thing. In order to do that, I’ve ...
Last night I ran a very successful workshop at the Friends of Neo4j Stockholm meetup group. The format was based on a workshop that I attended in San Franci...
In my last two posts I covered the process of importing data from StackOverflow and GitHub for the purpose of creating a combined MDM database. Now we get t...
In my last post I said I would “bring in another data source, show how I linked the data together, and demonstrate the sort of bigger picture that one can ge...
Joining multiple disparate data-sources, commonly dubbed Master Data Management (MDM), is usually not a fun exercise. I would like to show you how to use a g...
When using neo4j for the first time, most people want to import data from another database to start playing around. There are a lot of options including LOA...
I am he as you are he as you are me and we are all together – The Beatles
When I told the people of Northern Ireland that I was an atheist, a woman in the audience stood up and said, ‘Yes, but is it the God of the Catholics or t...
“Wilkins! Yes! I’ve considered decorating these walls with some graffiti of my own, and writing it in the Universal Character.. but it is too depressing...