Analyzing Ruby's ObjectSpace with Neo4j

Recently the continuous builds for the neo4j Ruby gem failed for JRuby because the memory limit had been reached. I wanted to see if I could use my favorite tool (Neo4j) to analyize the memory usage. So I threw together a bit of Ruby code to use Ruby’s ObjectSpace.each_object functionality to dump every object in memory after the test suite of the gem finished (garbage collecting first, of course). First let’s take a look at the model:

A quick Ruby primer:

• Objects are always instantiations of one and only one Class
• Classes can include zero or more Modules which extend their APIs
• Everything in Ruby is an Object, including Classes and Modules

So with that in mind I set out to output some CSVs. This was my first opportunity to output some CSV files for the neo4j-import tool and it was quite a straightforward standard to implement. Here is an example of each of the files that I generated:

objects.csv

object_id:ID	inspect	:LABEL
70099268550680	<RubyVM::InstructionSequence:block in contains_requirable_file?@/Users/brian/.rvm/rubies/ruby-2.2.0/lib/ruby/2.2.0/rubygems/basic_specification.rb>	Object
70099273105500	RubyVM::InstructionSequence	Object;Class
70099273176540	Class	Object;Class
70099270029100	RSpec::Core::SharedExampleGroup::TopLevelDSL	Object;Module
70099273176580	Module	Object;Class
70099277099900	RSpec::Core::DSL	Object;Module
70099275863180	PP::ObjectMixin	Object;Module
70099278769700	JSON::Ext::Generator::GeneratorMethods::Object	Object;Module
70099273176460	Kernel	Object;Module

instance_variables.csv

:START_ID	:END_ID	variable
70099268552320	70099268552120	@version
70099268552320	70099277643340	@segments
70099268552320	0	@prerelease
70099268552320	-4656870404935510835	@hash
70099268553460	70099268553260	@version
70099268553460	70099277643720	@segments
70099268553460	0	@prerelease
70099268553460	2810968934318335001	@hash
70099268558740	70099272991680	@_declared_property_manager

object_classes.csv

:START_ID	:END_ID
70099273176540	70099273176540
70099273176580	70099273176540
70099277099900	70099273176580
70099275863180	70099273176580
70099278769700	70099273176580
70099273176460	70099273176580
70099270029100	70099273176580
70099273105500	70099273176540
70099268550680	70099273105500

class_superclasses.csv

:START_ID	:END_ID
70271986243040	70271986243080
70272011737340	70271986239340
70271986207920	70271986243120
70271986179720	70271986243120
70271986207000	70271986243120
70271986517200	70271986243120
70271986241140	70271986243120
70271986225760	70271986225920
70271986206080	70271986243120

class_modules.csv

:START_ID	:END_ID
70099273176580	70099270029100
70099273176580	70099277099900
70099273176580	70099275863180
70099273176580	70099278769700
70099273176580	70099273176460
70099273176540	70099270029100
70099273176540	70099277099900
70099273176540	70099275863180
70099273176540	70099278769700

Again, you can see the code for generating these files here.

As you can see I chose to make my files tab-separated. I did this because the inspect column was going to have lots of double quotes in it and it would create a simpler file. I also used the pipe (|) as the quoting character because of some conflicts with the way Ruby and Neo4j interpret escaping of quoted characters. Here is the command that I used to import the CSV files into a Neo4j database:

./db/bin/neo4j-import \
        --delimiter TAB \
        --quote "|" \
        --into ./db/data/graph.db \
        --nodes ./objects.csv \
        --relationships:INSTANCE_VARIABLE ./instance_variables.csv \
        --relationships:HAS_CLASS ./object_classes.csv \
        --relationships:INCLUDES_MODULE ./class_modules.csv

Great, now we’re cooking with graphs! What does this look like in the Neo4j web console?

MATCH (o:Object) WHERE NOT(o:Class) AND NOT(o:Module) WITH o LIMIT 20
MATCH o-[:HAS_CLASS]->(c:Class)
OPTIONAL MATCH c-[:INCLUDES_MODULE]->(m:Module)
RETURN *

Nice! As another sanity check, let’s see what our top allocated objects are:

MATCH (o:Object)-[:HAS_CLASS]->(c:Class)
WHERE NOT(o:Class) AND NOT(o:Module)
RETURN c.inspect, count(o) ORDER BY count(o) DESC LIMIT 10

That seems right! Any experienced Rubyists will know that Ruby will allocate a lot of strings.

So let’s really get into it and use the power of graphs. Let’s find out which objects have the most references via instance variables. This query takes every object in the database as the root of a tree of instance variable references and calculates the total number of descendent objects in that tree. This should give us an idea for what objects have a lot of other objects depending on them which cannot be garbage collected.

// Count of tree
MATCH (o:Object) WHERE NOT(o:Class) AND NOT(o:Module) WITH o
MATCH o-[:HAS_CLASS]->(c:Class)
OPTIONAL MATCH path=o-[:INSTANCE_VARIABLE*]->(other)
RETURN c.inspect, o.inspect, count(other) ORDER BY count(other) DESC LIMIT 10

Not bad for a query which runs in 6.5 seconds (with 335,446 total objects on my MacBook Air)!

As part of the import the objects I also create a graph structure to define classes, modules, and their relationships. Let’s see what are the most included modules in our object space:

Looking at this I learned a couple of things I didn’t know about Ruby:

• The Kernel module is included everywhere (which makes sense)
• When loading the pretty-print and JSON libraries a corresponding module is included everywhere (which is probably why they aren’t loaded by default)

While we’re here let’s grab a random sample of classes and modules (excluding some ones that I hand-picked which had too many connections to make for a useful graph visualization):

MATCH (o:Object)
WHERE (o:Class OR o:Module) AND
      NOT(o.inspect IN ['Class', 'Module', 'Kernel', 'PP::ObjectMixin']) AND
      NOT o.inspect =~ "JSON::Ext.*"
RETURN o LIMIT 80

Here’s the graph (click to zoom):

That’s all for now, but please let me know if you have other thoughts for how to use this approach!

Along a similar line I’ve been thinking about graphing dependency trees like in RubyGems or NPM. Other project ideas are also welcome!

• 2024 1
• 2023 2
• 2021 2
• 2020 2
• 2015 9
• 2014 5