Jekyll2023-06-16T17:37:29+02:00http://www.brian-underwood.codes//feed.xmlbrian-underwood.blogsWrite an awesome description for your new site here. You can edit this line in _config.yml. It will appear in your document head meta (for Google search results) and in your feed.xml site description.Brian UnderwoodLifting Your Loads for Maintainable Elixir Applications2023-06-15T00:00:00+02:002023-06-15T00:00:00+02:00http://www.brian-underwood.codes//elixir/2023/06/15/lifting-your-loads-for-maintainable-elixir-applications<p>(This post was originally created for the Erlang Solutions blog. The original can be found <a href="https://www.erlang-solutions.com/blog/lifting-your-loads-for-maintainable-elixir-applications/">here</a>)</p>
<p>This post will discuss one particular aspect of designing Elixir applications using the Ecto library: separating data loading from using the data which is loaded. I will lay out the situations and present some solutions, including a new library called <a href="https://github.com/cheerfulstoic/ecto_require_associations">ecto_require_associations</a>.</p>
<p>Applications will differ, but let’s look at <a href="https://github.com/plausible/analytics/blob/7d935b79bf516deaa0175ffe1b07784a8c72f3c2/lib/plausible/billing/plans.ex#LL39C1-L71C1">this example</a> from the Plausible Analytics repo[1]:</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="n">plans_for</span><span class="p">(</span><span class="n">user</span><span class="p">)</span> <span class="k">do</span>
<span class="n">user</span> <span class="o">=</span> <span class="no">Repo</span><span class="o">.</span><span class="n">preload</span><span class="p">(</span><span class="n">user</span><span class="p">,</span> <span class="ss">:subscription</span><span class="p">)</span>
<span class="c1"># … other code …</span>
<span class="n">raw_plans</span> <span class="o">=</span>
<span class="k">cond</span> <span class="k">do</span>
<span class="n">contains?</span><span class="p">(</span><span class="n">v1_plans</span><span class="p">,</span> <span class="n">user</span><span class="o">.</span><span class="n">subscription</span><span class="p">)</span> <span class="o">-></span>
<span class="n">v1_plans</span>
<span class="n">contains?</span><span class="p">(</span><span class="n">v2_plans</span><span class="p">,</span> <span class="n">user</span><span class="o">.</span><span class="n">subscription</span><span class="p">)</span> <span class="o">-></span>
<span class="n">v2_plans</span>
<span class="n">contains?</span><span class="p">(</span><span class="n">v3_plans</span><span class="p">,</span> <span class="n">user</span><span class="o">.</span><span class="n">subscription</span><span class="p">)</span> <span class="o">-></span>
<span class="n">v3_plans</span>
<span class="n">contains?</span><span class="p">(</span><span class="n">sandbox_plans</span><span class="p">,</span> <span class="n">user</span><span class="o">.</span><span class="n">subscription</span><span class="p">)</span> <span class="o">-></span>
<span class="n">sandbox_plans</span>
<span class="no">true</span> <span class="o">-></span>
<span class="c1"># … other code …</span>
<span class="k">end</span>
<span class="c1"># … other code …</span>
<span class="k">end</span>
</code></pre></div></div>
<p>Here the <code class="language-plaintext highlighter-rouge">subscription</code> association is preloaded for the given user, which is then immediately used as part of the logic of <code class="language-plaintext highlighter-rouge">plans_for/1</code>. I’d like to try to convince you that in cases like this, it would be better to “lift” that loading of data out of the function.</p>
<p>Let’s start with a brief background on Ecto:</p>
<h1 id="the-ecto-library">The Ecto Library</h1>
<p>Ecto was uses the “repository pattern”. With Elixir being a language drawing a lot of people from the Ruby community, this was a departure from the “active record pattern” used in the “ActiveRecord” library made popular by Ruby on Rails. This pattern uses classes to fetch data (e.g. <code class="language-plaintext highlighter-rouge">User.find(id)</code>) and object methods to update data (e.g. <code class="language-plaintext highlighter-rouge">user.save</code>). Since “encapsulation” is a fundamental aspect of object-oriented programming, it seems logical at first to hide away the database access in this way. Many found, however, that while encapsulation of business logic makes sense, also encapsulating the database logic often led to complexity when trying to control the lifecycle of a record.</p>
<p>The repository pattern – as implemented by Ecto – requires explicit use of a “Repo” module for all queries to and from the database. With this separation of database access, <code class="language-plaintext highlighter-rouge">Ecto.Schema</code> modules can focus on defining application data and <code class="language-plaintext highlighter-rouge">Ecto.Changeset</code> module focus on validating and transforming application data.</p>
<p>This separation is made very clear in Ecto’s <code class="language-plaintext highlighter-rouge">README</code> which states: “Ecto is also commonly used to map data from any source into Elixir structs, whether they are backed by a database or not.”</p>
<p>Even if you’re fairly familiar with Ecto, I highly recommend watching Darin Wilson’s <a href="https://www.youtube.com/watch?app=desktop&v=YQxopjai0CU">“Thinking in Ecto”</a> presentation for a really good overview of the hows and whys of Ecto.</p>
<p>Ecto’s separation of a repository layer makes even more sense in the context of functional programming. For context in discussing solutions to the problem presented above, it’s important to understand a bit about functional programming.</p>
<p>One big idea in functional programming (or specifically <a href="https://en.wikipedia.org/wiki/Purely_functional_programming">“purely functional programming”</a>) is the notion of minimising and isolating side-effects. A function with side-effects is one which interacts with some global state such as memory, a file, or a database. Side-effects are commonly thought of as changing global state, but <em>reading</em> global state means that the output of the function could change depending on when it’s run.</p>
<p>What benefits does avoiding side-effects give us?</p>
<ul>
<li>Separating database access from operations on data is a great separation of concerns which can lead to more modular and therefore more maintainable code.</li>
<li>Defining a function where the output depends completely on the input makes the function easier to understand and therefore easier to use and maintain.</li>
<li>Automated tests for functions without side-effects can be much simpler because you don’t need to setup external state.</li>
</ul>
<h1 id="a-solution">A Solution</h1>
<p>A first approach to lifting the <code class="language-plaintext highlighter-rouge">Repo.preload</code> in the example above would be to do just that. That would suddenly make the function “pure” and dependent only on the caller passing in a value for the user’s <code class="language-plaintext highlighter-rouge">subscription</code> field. The problem comes when the person writing code which calls <code class="language-plaintext highlighter-rouge">plans_for/1</code> forgets to preload. Since Ecto defaults associations on schema structs to have an <code class="language-plaintext highlighter-rouge">%Ecto.Association.NotLoaded{}</code> value, this approach would lead to a confusing error message like:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(KeyError) key :paddle_plan_id not found in:
#Ecto.Association. NotLoaded association :subscription is not loaded>
</code></pre></div></div>
<p>This is because the <a href="https://github.com/plausible/analytics/blob/7d935b79bf516deaa0175ffe1b07784a8c72f3c2/lib/plausible/billing/plans.ex#L139-L146"><code class="language-plaintext highlighter-rouge">contains/2</code></a> function accesses <code class="language-plaintext highlighter-rouge">subscription.paddle_plan_id</code>.</p>
<p>So, it would probably be better to explicitly look to see if the subscription is loaded. We could do this with pattern matching in an additional function definition:</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="n">plans_for</span><span class="p">(%{</span><span class="ss">subscription:</span> <span class="p">%</span><span class="no">Ecto</span><span class="o">.</span><span class="no">Association</span><span class="o">.</span><span class="no">NotLoaded</span><span class="p">{}}),</span> <span class="k">do</span><span class="p">:</span> <span class="k">raise</span> <span class="s2">"Expected subscription to be preloaded"</span>
</code></pre></div></div>
<p>Or, if we want to avoid referencing the <code class="language-plaintext highlighter-rouge">Ecto.Association.NotLoaded</code> module in your application’s code, there’s even a function Ecto provides to allow you to check at runtime:</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="n">plans_for</span><span class="p">(</span><span class="n">user</span><span class="p">)</span> <span class="k">do</span>
<span class="k">if</span><span class="p">(</span><span class="n">!Ecto</span><span class="o">.</span><span class="n">assoc_loaded?</span><span class="p">(</span><span class="n">user</span><span class="o">.</span><span class="n">subscription</span><span class="p">)</span> <span class="k">do</span>
<span class="k">raise</span> <span class="s2">"Expected subscription to be preloaded"</span><span class="p">)</span>
<span class="k">end</span>
<span class="c1"># …</span>
</code></pre></div></div>
<p>This can get repetitive and potentially error-prone if you have a larger set of associations that you would like your function to depend on. I’ve created a small library called <a href="https://github.com/cheerfulstoic/require_associations_test"><code class="language-plaintext highlighter-rouge">ecto_require_associations</code></a> to take care of the details for you. If you’d like to load multiple associations you can use the same syntax used by Ecto’s <code class="language-plaintext highlighter-rouge">preload</code> function:</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="n">plans_for</span><span class="p">(</span><span class="n">user</span><span class="p">)</span> <span class="k">do</span>
<span class="no">EctoRequireAssociations</span><span class="o">.</span><span class="n">ensure!</span><span class="p">(</span><span class="n">user</span><span class="p">,</span> <span class="p">[</span><span class="ss">:subscriptions</span><span class="p">,</span> <span class="ss">site_memberships:</span> <span class="ss">:site</span><span class="p">])</span>
<span class="c1"># …</span>
</code></pre></div></div>
<p>The above call would check:</p>
<ul>
<li>If the <code class="language-plaintext highlighter-rouge">subscriptions</code> association has been loaded for the user</li>
<li>If the <code class="language-plaintext highlighter-rouge">site_memberships</code> association has been loaded for the user</li>
<li>If the <code class="language-plaintext highlighter-rouge">site</code> association has been loaded on each site membership</li>
</ul>
<p>If, for example, one or more of the <code class="language-plaintext highlighter-rouge">site</code> memberships hasn’t been loaded then an exception is raised like:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(ArgumentError) Expected association to be set: `site_memberships.site`
</code></pre></div></div>
<p>It can even work on a list of records given to it, just like <code class="language-plaintext highlighter-rouge">Repo.preload</code>.</p>
<h1 id="going-too-far-the-other-way">Going Too Far The Other Way</h1>
<p>Hopefully I’ve convinced you that the above approach can be helpful for creating more maintainable code. At the same time, I want to caution against another potential problem on the “other side”. Let’s say we have a function to get a user like this:</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defmodule</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">Users</span> <span class="k">do</span>
<span class="k">def</span> <span class="n">get_user</span><span class="p">(</span><span class="n">id</span><span class="p">)</span> <span class="k">do</span>
<span class="no">MyApp</span><span class="o">.</span><span class="no">Repo</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">id</span><span class="p">)</span>
<span class="o">|></span> <span class="no">MyApp</span><span class="o">.</span><span class="no">Repo</span><span class="o">.</span><span class="n">preload</span><span class="p">([</span><span class="ss">:api_keys</span><span class="p">,</span> <span class="ss">:subscriptions</span><span class="p">,</span> <span class="ss">site_memberships:</span> <span class="ss">:site</span><span class="p">])</span>
<span class="k">end</span>
<span class="k">def</span> <span class="n">get_admins</span> <span class="k">do</span>
<span class="n">from</span><span class="p">(</span><span class="n">user</span> <span class="ow">in</span> <span class="no">User</span><span class="p">,</span> <span class="ss">where:</span> <span class="n">user</span><span class="o">.</span><span class="n">is_admin</span><span class="p">)</span>
<span class="o">|></span> <span class="no">MyApp</span><span class="o">.</span><span class="no">Repo</span><span class="o">.</span><span class="n">all</span><span class="p">()</span>
<span class="o">|></span> <span class="no">MyApp</span><span class="o">.</span><span class="no">Repo</span><span class="o">.</span><span class="n">preload</span><span class="p">([</span><span class="ss">:api_keys</span><span class="p">,</span> <span class="ss">:subscriptions</span><span class="p">,</span> <span class="ss">site_memberships:</span> <span class="ss">:site</span><span class="p">])</span>
<span class="k">end</span>
</code></pre></div></div>
<p>When loading data to support pure functions, it could be tempting to load everything that might be needed by all functions which have a user argument. The risk then becomes one of loading too much data. Functions like <code class="language-plaintext highlighter-rouge">get_user</code> and <code class="language-plaintext highlighter-rouge">get_admins</code> are likely to be used all over your application, and generally you won’t need all of the associations loaded. This is a scaling problem that isn’t a problem until your application gets popular.One common pattern to solve this is to simply have a <code class="language-plaintext highlighter-rouge">preloads</code> argument:</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defmodule</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">Users</span> <span class="k">do</span>
<span class="k">def</span> <span class="n">get_user</span><span class="p">(</span><span class="n">id</span><span class="p">,</span> <span class="n">preloads</span> <span class="p">\\</span> <span class="p">[])</span> <span class="k">do</span>
<span class="no">MyApp</span><span class="o">.</span><span class="no">Repo</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">id</span><span class="p">)</span>
<span class="o">|></span> <span class="no">MyApp</span><span class="o">.</span><span class="no">Repo</span><span class="o">.</span><span class="n">preload</span><span class="p">(</span><span class="n">preloads</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">def</span> <span class="n">get_admins</span><span class="p">(</span><span class="n">preloads</span> <span class="p">\\</span> <span class="p">[])</span> <span class="k">do</span>
<span class="n">from</span><span class="p">(</span><span class="n">user</span> <span class="ow">in</span> <span class="no">User</span><span class="p">,</span> <span class="ss">where:</span> <span class="n">user</span><span class="o">.</span><span class="n">is_admin</span><span class="p">)</span>
<span class="o">|></span> <span class="no">MyApp</span><span class="o">.</span><span class="no">Repo</span><span class="o">.</span><span class="n">all</span><span class="p">()</span>
<span class="o">|></span> <span class="no">MyApp</span><span class="o">.</span><span class="no">Repo</span><span class="o">.</span><span class="n">preload</span><span class="p">(</span><span class="n">preloads</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="c1"># Usage</span>
<span class="no">MyApp</span><span class="o">.</span><span class="no">Users</span><span class="o">.</span><span class="n">get_admins</span><span class="p">([</span><span class="ss">:api_keys</span><span class="p">])</span>
</code></pre></div></div>
<p>This does solve the problem and allows you to load associations only where you need them. I would say, however, that this code falls into the same trap as the Active Record library by intertwining database and application logic.</p>
<p>The Ecto library, your schemas, and your associations aren’t secrets. You absolutely should encapsulate things like your complex query logic, the details for how you calculate numbers, or the decisions you make based on data. But it’s fine to ask Ecto to preload the associations and let your query functions just do querying. This can give you a clean separation of concerns:</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defmodule</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">Users</span> <span class="k">do</span>
<span class="k">def</span> <span class="n">get_user</span><span class="p">(</span><span class="n">id</span><span class="p">),</span> <span class="k">do</span><span class="p">:</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">Repo</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">id</span><span class="p">)</span>
<span class="k">def</span> <span class="n">get_admins</span> <span class="k">do</span>
<span class="n">from</span><span class="p">(</span><span class="n">user</span> <span class="ow">in</span> <span class="no">User</span><span class="p">,</span> <span class="ss">where:</span> <span class="n">user</span><span class="o">.</span><span class="n">is_admin</span><span class="p">)</span>
<span class="o">|></span> <span class="no">MyApp</span><span class="o">.</span><span class="no">Repo</span><span class="o">.</span><span class="n">all</span><span class="p">()</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="c1"># Usage:</span>
<span class="no">MyApp</span><span class="o">.</span><span class="no">Users</span><span class="o">.</span><span class="n">get_admins</span><span class="p">()</span>
<span class="o">|></span> <span class="no">MyApp</span><span class="o">.</span><span class="no">Repo</span><span class="o">.</span><span class="n">preload</span><span class="p">(</span><span class="ss">:api_keys</span><span class="p">)</span>
</code></pre></div></div>
<p>That said, if you have associations you need for specific functions, you may want to create functions which can preload without the caller knowing the details. This saves repetition and helps clarify overlapping dependencies:</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">defmodule</span> <span class="no">MyApp</span><span class="o">.</span><span class="no">Users</span> <span class="k">do</span>
<span class="c1"># You can call `preload_for_access_check(user)` to load the required data</span>
<span class="k">def</span> <span class="n">can_access?</span><span class="p">(</span><span class="n">user</span><span class="p">,</span> <span class="n">resource</span><span class="p">)</span> <span class="k">do</span>
<span class="no">EctoRequireAssociations</span><span class="o">.</span><span class="n">ensure!</span><span class="p">(</span><span class="n">user</span><span class="p">,</span> <span class="p">[</span><span class="ss">:roles</span><span class="p">,</span> <span class="ss">:permissions</span><span class="p">])</span>
<span class="c1"># …</span>
<span class="k">end</span>
<span class="k">def</span> <span class="n">preload_for_access_check</span><span class="p">(</span><span class="n">user</span><span class="p">)</span> <span class="k">do</span>
<span class="no">MyApp</span><span class="o">.</span><span class="no">Repo</span><span class="o">.</span><span class="n">preload</span><span class="p">(</span><span class="n">user</span><span class="p">,</span> <span class="p">[</span><span class="ss">:roles</span><span class="p">,</span> <span class="ss">:permissions</span><span class="p">])</span>
<span class="k">end</span>
<span class="k">def</span> <span class="n">preload_for_something_else</span><span class="p">(</span><span class="n">user</span><span class="p">)</span> <span class="k">do</span>
<span class="no">MyApp</span><span class="o">.</span><span class="no">Repo</span><span class="o">.</span><span class="n">preload</span><span class="p">(</span><span class="n">user</span><span class="p">,</span> <span class="p">[</span><span class="ss">:roles</span><span class="p">,</span> <span class="ss">:regions</span><span class="p">])</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>
<p>Remember that <code class="language-plaintext highlighter-rouge">Repo.preload</code> won’t load data if it’s already been set on the struct unless you specify <code class="language-plaintext highlighter-rouge">force: true</code>!</p>
<p>So if we shouldn’t put our data loading along with our business logic or with our queries, where should it go? The answer to this is fuzzier, but it makes sense to think about what parts of your code should exist as “coordination” points. Let’s talk about some good options:</p>
<h2 id="phoenix-controllers-channels-liveviews-and-email-handlers">Phoenix Controllers, Channels, LiveViews, and Email handlers</h2>
<p>Controllers, LiveViews, and email handlers are all places where we render a template, and generally we are loading some sort of data to be able to do that. Channels and LiveViews are dealing with events which are also often a place where we’ll need to load data to provide some sort of update to a client. In both cases, this is the place where we know what data will be needed, so it makes sense to keep the responsibility of choosing what data to load in this code.</p>
<h2 id="absinthe-resolvers">Absinthe Resolvers</h2>
<p>Absinthe is a library for implementing a GraphQL API. Not only will you need to preload data, sometimes you may use the <a href="https://github.com/absinthe-graphql/dataloader">Dataloader</a> to efficiently load data outside of manual preloading. This highlights how loading of associated data is a separate concern from evaluating it.</p>
<h2 id="scripts-and-tasks">Scripts and Tasks</h2>
<p>Scripts and <code class="language-plaintext highlighter-rouge">mix</code> tasks are another place for coordinating loading and logic. This code might even be one-off and/or short-lived, so it may not even make sense to define functions inside of context modules. Depending on the importance and lifecycle of a script/task it could be that none of the above discussion is applicable.</p>
<h2 id="high-level-context-functions">High Level Context Functions</h2>
<p>The discussion above suggests pushing loading logic up and out of <a href="https://hexdocs.pm/phoenix/contexts.html">context</a>-type modules. However, if you have a high-level function which is an entrypoint into some complex code, then it may make sense to coordinate your loading and logic there. This is especially true if the function is used from multiple places in your application.</p>
<h1 id="conclusion">Conclusion</h1>
<p>It seems like a small detail, but making more functions purely functional and isolating your database access can have compounding effects on the maintainability of your code. This can be especially true when the codebase is maintained by more than one person, making it easier for everybody to change the code without worrying about side-effects. Try it out and see!</p>
<p><code class="language-plaintext highlighter-rouge">[1]</code>: Please don’t see this as me picking on the Plausible Analytics folks in any way. I think that their project is great and the fact that they open-sourced it makes it a great resource for real-world examples like this one!</p>Brian Underwood(This post was originally created for the Erlang Solutions blog. The original can be found here)Can’t Live `with` It, Can’t Live `with`out It2023-02-23T00:00:00+01:002023-02-23T00:00:00+01:00http://www.brian-underwood.codes//elixir/2023/02/23/cant-live-with-it-cant-live-without-it<p>(This post was originally created for the Erlang Solutions blog. The original can be found <a href="https://www.erlang-solutions.com/blog/cant-live-with-it-cant-live-without-it">here</a>)</p>
<p>I’d like to share some thoughts about Elixir’s <code class="language-plaintext highlighter-rouge">with</code> keyword. <code class="language-plaintext highlighter-rouge">with</code> is a wonderful tool, but in my experience it is a bit overused. To use it best, we must understand how it behaves in all cases. So, let’s briefly cover the basics, starting with pipes in Elixir.</p>
<h1 id="pipes-are-a-wonderful-abstraction">Pipes are a wonderful abstraction</h1>
<p>But like all tools, you should think about when it is best used…</p>
<p>Pipes are <strong>at their best</strong> when you expect your functions to accept and return basic values. But often we don’t have only simple values because we need to deal with <strong>error cases</strong>. For example:</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">region</span>
<span class="o">|></span> <span class="no">Module</span><span class="o">.</span><span class="n">fetch_companies</span><span class="p">()</span>
<span class="o">|></span> <span class="no">Module</span><span class="o">.</span><span class="n">fetch_departments</span><span class="p">()</span>
<span class="o">|></span> <span class="no">Enum</span><span class="o">.</span><span class="n">map</span><span class="p">(</span><span class="o">&</span> <span class="nv">&1</span><span class="o">.</span><span class="n">employee_count</span><span class="p">)</span>
<span class="o">|></span> <span class="n">calculate_average</span><span class="p">()</span>
</code></pre></div></div>
<p>If our <code class="language-plaintext highlighter-rouge">fetch_*</code> methods return list values there isn’t a problem. But often we fetch data from an external source, which means we introduce the possibility of an <code class="language-plaintext highlighter-rouge">error</code>. Generally in Elixir this means <code class="language-plaintext highlighter-rouge">{:ok, _}</code> tuples for success and <code class="language-plaintext highlighter-rouge">{:error, _}</code> tuples for failure. Using pipes that might become:</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">region</span>
<span class="o">|></span> <span class="no">Module</span><span class="o">.</span><span class="n">fetch_companies</span><span class="p">()</span>
<span class="o">|></span> <span class="k">case</span> <span class="k">do</span>
<span class="err"> </span><span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">companies</span><span class="p">}</span> <span class="o">-></span> <span class="no">Module</span><span class="o">.</span><span class="n">fetch_departments</span><span class="p">(</span><span class="n">companies</span><span class="p">)</span>
<span class="err"> </span><span class="p">{</span><span class="ss">:error</span><span class="p">,</span> <span class="n">_</span><span class="p">}</span> <span class="o">=</span> <span class="n">error</span> <span class="o">-></span> <span class="n">error</span>
<span class="k">end</span>
<span class="o">|></span> <span class="k">case</span> <span class="k">do</span>
<span class="err"> </span><span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">departments</span><span class="p">}</span> <span class="o">-></span>
<span class="err"> </span><span class="n">departments</span>
<span class="err"> </span><span class="o">|></span> <span class="no">Enum</span><span class="o">.</span><span class="n">map</span><span class="p">(</span><span class="o">&</span> <span class="nv">&1</span><span class="o">.</span><span class="n">employee_count</span><span class="p">)</span>
<span class="err"> </span><span class="o">|></span> <span class="n">calculate_average</span><span class="p">()</span>
<span class="err"> </span><span class="p">{</span><span class="ss">:error</span><span class="p">,</span> <span class="n">_</span><span class="p">}</span> <span class="o">=</span> <span class="n">error</span> <span class="o">-></span> <span class="n">error</span>
<span class="k">end</span>
</code></pre></div></div>
<p>Not horrible, but certainly not beautiful. Fortunately, Elixir has <code class="language-plaintext highlighter-rouge">with</code>!</p>
<h1 id="with-is-a-wonderful-abstraction"><code class="language-plaintext highlighter-rouge">with</code> is a wonderful abstraction</h1>
<p>But like all tools, you should think about when it’s best used…</p>
<p><code class="language-plaintext highlighter-rouge">with</code> is <strong>at it’s best</strong> when dealing with the <strong>happy paths</strong> of a set of calls which <strong>all return similar things</strong>. What do I mean by that? Let’s look at what this code might look like using <code class="language-plaintext highlighter-rouge">with</code>?</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">with</span> <span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">companies</span><span class="p">}</span> <span class="o"><-</span> <span class="no">Module</span><span class="o">.</span><span class="n">fetch_companies</span><span class="p">(</span><span class="n">region</span><span class="p">),</span>
<span class="err"> </span><span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">departments</span><span class="p">}</span> <span class="o"><-</span> <span class="no">Module</span><span class="o">.</span><span class="n">fetch_departments</span><span class="p">(</span><span class="n">companies</span><span class="p">)</span> <span class="k">do</span>
<span class="err"> </span><span class="n">departments</span>
<span class="err"> </span><span class="o">|></span> <span class="no">Enum</span><span class="o">.</span><span class="n">map</span><span class="p">(</span><span class="o">&</span> <span class="nv">&1</span><span class="o">.</span><span class="n">employee_count</span><span class="p">)</span>
<span class="err"> </span><span class="o">|></span> <span class="n">calculate_average</span><span class="p">()</span>
<span class="k">end</span>
</code></pre></div></div>
<p>That’s definitely better!</p>
<ul>
<li>We separated out the parts of our code which might fail (remember that failure is a sign of a side-effect and in functional programming we want to isolate side-effects).</li>
<li>The body is only the things that we don’t expect to fail.</li>
<li>We don’t need to explicitly deal with the <code class="language-plaintext highlighter-rouge">{:error, _}</code> cases (in this case <code class="language-plaintext highlighter-rouge">with</code> will return any clause values which don’t match the pattern before <code class="language-plaintext highlighter-rouge"><-</code>).</li>
</ul>
<p>But this is a great example of a <strong>happy path</strong> where the set of calls <strong>all return similar things</strong>. But where are some examples of where we might go wrong with <code class="language-plaintext highlighter-rouge">with</code>?</p>
<h2 id="non-standard-failure">Non-standard failure</h2>
<p>What if <code class="language-plaintext highlighter-rouge">Module.fetch_companies</code> returns <code class="language-plaintext highlighter-rouge">{:error, _}</code> but <code class="language-plaintext highlighter-rouge">Module.fetch_departments</code> returns just <code class="language-plaintext highlighter-rouge">:error</code>? That means your <code class="language-plaintext highlighter-rouge">with</code> is going to return two different error results. If your <code class="language-plaintext highlighter-rouge">with</code> is the end of your function call then that complexity is now the caller’s responsibility. You might not think that’s a big deal because we can do this:</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">else</span>
<span class="err"> </span><span class="ss">:error</span> <span class="o">-></span> <span class="p">{</span><span class="ss">:error</span><span class="p">,</span> <span class="s2">"Error fetching departments"</span><span class="p">}</span>
</code></pre></div></div>
<p>But this breaks to more-or-less important degrees because:</p>
<ul>
<li>… once you add an <code class="language-plaintext highlighter-rouge">else</code> clause, you need to take care of <em>every</em> non-happy path case (e.g. above we should match the <code class="language-plaintext highlighter-rouge">{:error, _}</code> returned by <code class="language-plaintext highlighter-rouge">Module.fetch_companies</code> which we didn’t need to explicitly match before) 😤</li>
<li>… if either function is later refactored to return another pattern (e.g. <code class="language-plaintext highlighter-rouge">{:error, _, _}</code>) – there will be a <code class="language-plaintext highlighter-rouge">WithClauseError</code> exception (again, because once you add an <code class="language-plaintext highlighter-rouge">else</code> the fallback behavior of non-matching <code class="language-plaintext highlighter-rouge"><-</code> patterns doesn’t work) 🤷♂️</li>
<li>… if <code class="language-plaintext highlighter-rouge">Module.fetch_departments</code> is later refactored to return <code class="language-plaintext highlighter-rouge">{:error, _}</code> – we’ll then have an unused handler 🤷♂️</li>
<li>… if another clause is added which also returns <code class="language-plaintext highlighter-rouge">:error</code> the message <code class="language-plaintext highlighter-rouge">Error fetching departments</code> probably won’t be the right error 🙈</li>
<li>… if you want to refactor this code later, you need to understand <em>everything</em> that the called functions might potentially return, leading to code which is hard to refactor. If there are just two clauses and we’re just calling simple functions, that’s not as big of a deal. But with many <code class="language-plaintext highlighter-rouge">with</code> clauses which call complex functions, it can become a nightmare 🙀</li>
</ul>
<p>So the first major thing to know when using <code class="language-plaintext highlighter-rouge">with</code> is what happens <strong>when a clause doesn’t match it’s pattern</strong>:</p>
<ul>
<li>If <code class="language-plaintext highlighter-rouge">else</code> <strong>is not</strong> specified then the non-matching clause is returned.</li>
<li>If <code class="language-plaintext highlighter-rouge">else</code> <strong>is</strong> specified then the code for the first matching <code class="language-plaintext highlighter-rouge">else</code> pattern is evaluated. If no <code class="language-plaintext highlighter-rouge">else</code> pattern matches , a <code class="language-plaintext highlighter-rouge">WithClauseError</code> is raised.</li>
</ul>
<p>As <a href="http://stratus3d.com/blog/2022/06/01/the-problem-with-elixirs-with/">Stratus3D</a> excellently put it: “<code class="language-plaintext highlighter-rouge">with</code> blocks are the only Elixir construct that implicitly uses the same else clauses to handle return values from different expressions. The lack of a one-to-one correspondence between an expression in the head of the <code class="language-plaintext highlighter-rouge">with</code> block and the clauses that handle its return values makes it impossible to know when each <code class="language-plaintext highlighter-rouge">else</code> clause will be used”. There are a couple of well known solutions to address this. One is using “tagged tuples”:</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">with</span> <span class="p">{</span><span class="ss">:fetch_companies</span><span class="p">,</span> <span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">companies</span><span class="p">}</span> <span class="o"><-</span> <span class="p">{</span><span class="ss">:fetch_companies</span><span class="p">,</span> <span class="no">Module</span><span class="o">.</span><span class="n">fetch_companies</span><span class="p">(</span><span class="n">region</span><span class="p">)},</span>
<span class="err"> </span><span class="p">{</span><span class="ss">:fetch_departments</span><span class="p">,</span> <span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">departments</span><span class="p">}</span> <span class="o"><-</span> <span class="p">{</span><span class="ss">:fetch_departments</span><span class="p">,</span> <span class="no">Module</span><span class="o">.</span><span class="n">fetch_departments</span><span class="p">(</span><span class="n">companies</span><span class="p">)},</span>
<span class="err"> </span><span class="n">departments</span>
<span class="err"> </span><span class="o">|></span> <span class="no">Enum</span><span class="o">.</span><span class="n">map</span><span class="p">(</span><span class="o">&</span> <span class="nv">&1</span><span class="o">.</span><span class="n">employee_count</span><span class="p">)</span>
<span class="err"> </span><span class="o">|></span> <span class="n">calculate_average</span><span class="p">()</span>
<span class="k">else</span>
<span class="err"> </span><span class="p">{</span><span class="ss">:fetch_companies</span><span class="p">,</span> <span class="p">{</span><span class="ss">:error</span><span class="p">,</span> <span class="n">reason</span><span class="p">}}</span> <span class="o">-></span> <span class="o">...</span>
<span class="err"> </span><span class="p">{</span><span class="ss">:fetch_departments</span><span class="p">,</span> <span class="ss">:error</span><span class="p">}</span> <span class="o">-></span> <span class="o">...</span>
<span class="k">end</span>
</code></pre></div></div>
<p>Though tagged tuples should be avoided for various reasons:</p>
<ul>
<li>They make the code a lot more verbose</li>
<li><code class="language-plaintext highlighter-rouge">else</code> is now being used, so we need to match all patterns that might occur</li>
<li>We need to keep the clauses and <code class="language-plaintext highlighter-rouge">else</code> in sync when adding/removing/modifying clauses, leaving room for bugs.</li>
<li><strong>Most importantly</strong>: the value in an abstraction like <code class="language-plaintext highlighter-rouge">{:ok, _}</code> / <code class="language-plaintext highlighter-rouge">{:error, _}</code> tuples is that you can handle things generically without needing to worry about the source</li>
</ul>
<p>A generally better solution is to create functions which normalize the values matched in the patterns. This is covered well in <a href="https://hexdocs.pm/elixir/Kernel.SpecialForms.html#with/1-beware">a note in the docs for <code class="language-plaintext highlighter-rouge">with</code></a> and I recommend checking it out. One addition I would make: in the above case you could leave the <code class="language-plaintext highlighter-rouge">Module.fetch_companies</code> alone and just surround the <code class="language-plaintext highlighter-rouge">Module.fetch_departments</code> with a local <code class="language-plaintext highlighter-rouge">fetch_departments</code> to turn the <code class="language-plaintext highlighter-rouge">:error</code> into an <code class="language-plaintext highlighter-rouge">{:error, reason}</code>.</p>
<h2 id="non-standard-success">Non-standard <em>success</em></h2>
<p>We can even get unexpected results when <code class="language-plaintext highlighter-rouge">with</code> succeeds! To start let’s look at the <code class="language-plaintext highlighter-rouge">parse/1</code> function from the excellent <code class="language-plaintext highlighter-rouge">decimal</code> library. It’s typespec tells us that it can return <code class="language-plaintext highlighter-rouge">{Decimal.t(), binary()}</code> or <code class="language-plaintext highlighter-rouge">:error</code>. If we want to match a decimal value without extra characters, we could have a <code class="language-plaintext highlighter-rouge">with</code> clause like this:</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">with</span> <span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">value</span><span class="p">}</span> <span class="o"><-</span> <span class="n">fetch_value</span><span class="p">(),</span>
<span class="err"> </span><span class="p">{</span><span class="n">decimal</span><span class="p">,</span> <span class="s2">""</span><span class="p">}</span> <span class="o"><-</span> <span class="no">Decimal</span><span class="o">.</span><span class="n">parse</span><span class="p">(</span><span class="n">value</span><span class="p">)</span> <span class="k">do</span>
<span class="err"> </span><span class="p">{</span><span class="ss">:ok</span><span class="p">,</span> <span class="n">decimal</span><span class="p">}</span>
</code></pre></div></div>
<p>But if <code class="language-plaintext highlighter-rouge">value</code> is given as <code class="language-plaintext highlighter-rouge">"1.23 "</code> (with a space at the end), then <code class="language-plaintext highlighter-rouge">Decimal.parse/1</code> will return <code class="language-plaintext highlighter-rouge">{#Decimal<1.23>, " "}</code>. Since that doesn’t match our pattern (string with a space vs. an empty string), the body of the <code class="language-plaintext highlighter-rouge">with</code> will be skipped. If we don’t have an <code class="language-plaintext highlighter-rouge">else</code> then instead of returning a <code class="language-plaintext highlighter-rouge">{:ok, _}</code> value, we return <code class="language-plaintext highlighter-rouge">{#Decimal<1.23>, " "}</code>.</p>
<p>The solution may seem simple: match on <code class="language-plaintext highlighter-rouge">{decimal, _}</code>! But then we match strings like <code class="language-plaintext highlighter-rouge">"1.23a"</code> which is what we were trying to avoid. Again, we’re likely better off defining a local <code class="language-plaintext highlighter-rouge">parse_decimal</code> function which returns <code class="language-plaintext highlighter-rouge">{:ok, _}</code> or <code class="language-plaintext highlighter-rouge">{:error, _}</code>.</p>
<p>There are other, similar, situations:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">{:ok, %{"key" => value}} <- fetch_data(...)</code> – the value inside of the <code class="language-plaintext highlighter-rouge">{:ok, _}</code> tuple may not have a <code class="language-plaintext highlighter-rouge">"key"</code> key.</li>
<li><code class="language-plaintext highlighter-rouge">[%{id: value}] <- fetch_data(...)</code> – the list returned may have more or less than one item, or if it does only have one item it may not have the <code class="language-plaintext highlighter-rouge">:id</code> key</li>
<li><code class="language-plaintext highlighter-rouge">value when length(value) > 2 <- fetch_data(...)</code> – the <code class="language-plaintext highlighter-rouge">when</code> might not match. There are two cases where this might surprise you:
<ul>
<li>If <code class="language-plaintext highlighter-rouge">value</code> is a list, the length of the list being 2 or below will return the list.</li>
<li>If <code class="language-plaintext highlighter-rouge">value</code> is a string, <code class="language-plaintext highlighter-rouge">length</code> isn’t a valid function (you’d probably want <code class="language-plaintext highlighter-rouge">byte_size</code>). Instead of an exception, the guard simply fails and the pattern doesn’t match.</li>
</ul>
</li>
</ul>
<p>The problem in all of these cases is that the intermediate value from <code class="language-plaintext highlighter-rouge">fetch_data</code> will be returned, not what the body of the <code class="language-plaintext highlighter-rouge">with</code> would return. This means that our <code class="language-plaintext highlighter-rouge">with</code> returns “uneven” results. We can handle these cases in the <code class="language-plaintext highlighter-rouge">else</code>, but again, once we introduce <code class="language-plaintext highlighter-rouge">else</code> we need to take care of all potential cases.</p>
<p>I might even go to the extent of recommending that you don’t define <code class="language-plaintext highlighter-rouge">with</code> clause patterns which are at all deep in their pattern matching unless you are very sure the <strong>success case</strong> will be able to match the <strong>whole pattern</strong>. One example where you might take a risk is when matching <code class="language-plaintext highlighter-rouge">%MyStruct{key: value} <- …</code> where you <strong>know</strong> that a <code class="language-plaintext highlighter-rouge">MyStruct</code> value is going to be returned and you know that <code class="language-plaintext highlighter-rouge">key</code> is one of the keys defined for the struct. No matter the case, dialyzer is one tool to gain confidence that you will be able to match on the pattern (at least for your own code or libraries which also use dialyzer).</p>
<p>One of the simplest and most standard ways to avoid these issues is to make sure the functions that you are calling return <code class="language-plaintext highlighter-rouge">{:ok, variable}</code> or <code class="language-plaintext highlighter-rouge">{:error, reason}</code> tuples. Then <code class="language-plaintext highlighter-rouge">with</code> can fall through cleanly (<em>definitely</em> check out Chris Keathley’s discussion of “Avoid else in with blocks” in his post <a href="https://keathley.io/blog/good-and-bad-elixir.html">“Good and Bad Elixir”</a>).</p>
<p>With all that said, I recommend using <code class="language-plaintext highlighter-rouge">with</code> statements whenever you can! Just make sure that you think about fallback cases that might happen. Even better: write tests to cover all of your potential cases! If you can strike a balance and use <code class="language-plaintext highlighter-rouge">with</code> carefully, your code can be both cleaner <strong>and</strong> more reliable.</p>Brian Underwood(This post was originally created for the Erlang Solutions blog. The original can be found here)How Far Can I Push a GenServer?2021-07-23T00:00:00+02:002021-07-23T00:00:00+02:00http://www.brian-underwood.codes//elixir/2021/07/23/How-Far-Can-I-Push-a-GenServer<p>I’ve been using Elixir for a while and I’ve implemented a number of GenServers. But while I think I mostly understand the purpose of them, I’ve not gotten the chance to push one it’s limits, scale it up, and find ways to address it’s bottlenecks. I thought that it would be fun to create something to which I could give the URL as part of a presentation and have some confidence that it would be able to handle all the users who connected to it. So recently I implemented a simple game grid using Phoenix LiveView and emojis as indicators of player and objects. If you would like to learn about my journey, read on! But note that you’ll probably want to have at least a basic understanding of GenServers first. You might start by <a href="https://elixir-lang.org/getting-started/mix-otp/genserver.html">reading this</a> and/or <a href="https://elixircasts.io/intro-to-genserver">watching this</a>.</p>
<p>If you would like to look at the source code for this project you can see it <a href="https://github.com/cheerfulstoic/emoji_game">on GitHub</a>.</p>
<p>TL;DR: This post isn’t very TL;DR-able 😁 I don’t have any conclusions or recommendations to share, just a journey. I invite you to join me to learn what I’ve learned!</p>
<h1 id="overview-of-the-project">Overview of the Project</h1>
<p>
<div>
<a href="/assets/images/how-far-can-i-push-a-genserver/game-grid.png">
<img src="/assets/images/how-far-can-i-push-a-genserver/game-grid.png" style="max-width: 500px" />
</a>
</div>
<div><em>The Game Grid</em></div>
</p>
<p>It is an Elixir Phoenix app which starts up a <code class="language-plaintext highlighter-rouge">GenServer</code> process as the game server which:</p>
<ul>
<li>Keeps track of a 1000x1000 square game grid, populated with random objects (trees, to start)</li>
<li>Allows clients (other Elixir processes, players + actors) to make moves</li>
<li>Keeps track of the current position of clients, indexed by their process ID (PID)</li>
</ul>
<p>There will then also be two kinds of “client”s of the game server:</p>
<ul>
<li>A LiveView UI which handle the player’s session and key presses</li>
<li>A set of <code class="language-plaintext highlighter-rouge">GenServer</code>s (the “actors”) which will move randomly every second.</li>
</ul>
<h1 id="the-move-message">The <code class="language-plaintext highlighter-rouge">move</code> Message</h1>
<p>To implement moves I started out with a simple <code class="language-plaintext highlighter-rouge">move</code> message. Clients can send this message to the game server with their new position. An LiveView process sends this once at startup to establish a starting position and then again each time the player presses an arrow key. An actor processes simply sends a <code class="language-plaintext highlighter-rouge">move</code> message once every second to move randomly.</p>
<p>The <code class="language-plaintext highlighter-rouge">move</code> message itself worked fine, but a player wouldn’t see any movement from actor processes until they (the player) moved. I dealt with this problem later, so see the “Sending Map Updates Asyncronously” section below for my solution.</p>
<h1 id="linking-clients">Linking Clients</h1>
<p>One thing that I discovered early on was that the game would sometimes crash and the player would end up with a extra “dead” copy of themself on the board. It turns out that this would happen whenever the player pressed any non-arrow key (including modifier keys like alt/option). This is because I was handling key presses with a <code class="language-plaintext highlighter-rouge">case</code> statement without accounting for anything other than the four arrow keys. When another key was pressed a <code class="language-plaintext highlighter-rouge">CaseClauseError</code> error would be raised and crash the LiveView process. LiveView handled this great by automagically creating a new process for the session, but my game server would still think that the old process was around and thus the old avatar was still there. My fix: the game server calls <code class="language-plaintext highlighter-rouge">Process.link</code> to link itself to each client and then traps exits to clean up the information it has about that client’s PID.</p>
<details class="nitty-gritty">
<summary>Nitty-gritty detail</summary>
<p>I initially called <code class="language-plaintext highlighter-rouge">Process.link</code> whenever a client sent a <code class="language-plaintext highlighter-rouge">move</code> message. The documentation says “If such a link exists already, this function does nothing”, so it wasn’t a problem. But the more elegant way would be to just do it once.</p>
<p>Also: I started with <code class="language-plaintext highlighter-rouge">handle_cast</code>, but it doesn’t give you a PID. So I couldn’t just have clients send a move without getting a response because the game server needs to track clients by their PID. So I used <code class="language-plaintext highlighter-rouge">handle_call</code>, but that will probably be needed in the long run anyway since clients will eventually need feedback to know if their moves are invalid (i.e. something is there which can’t occupy the same space at the same time)</p>
</details>
<h1 id="registering">Registering</h1>
<p>I was beginning to have various reasons for thinking I should create a new kind of message that clients send to the game server to register themselves:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">Process.link</code> could be called only once once</li>
<li>It would simplify the code to handle <code class="language-plaintext highlighter-rouge">move</code> messages (no need to check if the PID already exsits)</li>
<li>It would allow for the game server to send the map section right away</li>
</ul>
<p>It will also later turn out to be a useful opportunity for the client to send configuration options to the server (keep reading for more!)</p>
<h1 id="measuring">Measuring</h1>
<p>With that done I wanted to optimize the process so that I could handle more clients. But, before optimizing my code, I wanted to add some metrics tracking because you can’t optimize what you can’t see! Thanks to <code class="language-plaintext highlighter-rouge">telemetry</code> being included with Phoenix I was able to quickly add metrics for the game server’s queue size and the response time of the <code class="language-plaintext highlighter-rouge">move</code> message (as seen from the actors). From there on I was able to see those metrics in the Phoneix LiveDashboard. Since LiveDashboard only shows recent values, I also used the <code class="language-plaintext highlighter-rouge">telemetry_metrics_statsd</code> library to send the metrics to a local statsd server with graphite for visualization (using the <code class="language-plaintext highlighter-rouge">graphiteapp/graphite-statsd</code> docker image).</p>
<details class="tabbed-git-gist">
<summary>See the code</summary>
<div x-data="{data : {files: []}, current_tab: 'benchmark.exs'}" x-init="data = await (await fetch('https://api.github.com/gists/9498a7f262ffc5a12b5ec87be8db2590')).json(); current_tab = Object.keys(data.files)[0]">
<nav>
<template x-for="filename in Object.keys(data.files)" x-bind:key="filename">
<div class="tab" x-bind:class="{ 'active': current_tab === filename }" x-on:click.prevent="current_tab = filename; window.location.hash = filename" x-text="filename" href="#"></div>
</template>
</nav>
<div class="tabs-body">
<template x-for="filename in Object.keys(data.files)" x-bind:key="filename">
<iframe x-show="current_tab === filename" x-bind:srcdoc="`<script src='https://gist.github.com/9498a7f262ffc5a12b5ec87be8db2590.js?file=${filename}'></script>`">
</iframe>
</template>
</div>
</div>
<style>
iframe {
border: 0;
width: 100%;
height: 307px;
}
.tab {
display: inline-block;
border: 1px solid black;
border-bottom: 0;
background-color: white;
padding: 0.4em;
}
.tab.active {
position: relative;
top: 1px;
background-color: #ddd;
}
.tabs-body {
background-color: #ddd;
border: 1px solid black;
}
</style>
</details>
<p>
<div>
<a href="/assets/images/how-far-can-i-push-a-genserver/livedashboard.png">
<img src="/assets/images/how-far-can-i-push-a-genserver/livedashboard.png" style="max-width: 100%" />
</a>
</div>
<div><em>LiveDashboard</em></div>
</p>
<p>
<div>
<a href="/assets/images/how-far-can-i-push-a-genserver/graphite.png">
<img src="/assets/images/how-far-can-i-push-a-genserver/graphite.png" style="max-width: 100%" />
</a>
</div>
<div><em>Graphite (via statsd)</em></div>
</p>
<p>I started out with 20 actors. This worked fine, and I was seeing response times between 5 and 65ms. I then raised the count by 20 each time. Getting up to 60 went fine overall with occasional increases in queue size / response time. But, when I got up to 80, it took some time before after startup before the queue went down to zero. Going up a bit to 90 led to the queue rarely going down to zero and response times of between 250 and 450ms, which I considered to be around the breaking point.</p>
<h1 id="reducing-the-move-message-response-size">Reducing The <code class="language-plaintext highlighter-rouge">move</code> Message Response Size</h1>
<p>My first idea for allowing me to handle more simultaneous clients was to change my default map response from being a List-of-Lists to a Map.</p>
<details class="show-the-code">
<summary>Show me the code!</summary>
<p>What this means is that, instead of the server returning this to clients:</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">[[</span><span class="no">nil</span><span class="p">,</span> <span class="no">nil</span><span class="p">,</span> <span class="ss">:tree</span><span class="p">],</span>
<span class="p">[</span><span class="ss">:actor</span><span class="p">,</span> <span class="no">nil</span><span class="p">,</span> <span class="no">nil</span><span class="p">],</span>
<span class="p">[</span><span class="ss">:actor</span><span class="p">,</span> <span class="ss">:tree</span><span class="p">,</span> <span class="no">nil</span><span class="p">]]</span>
</code></pre></div></div>
<p>It would return this:</p>
<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">%{</span>
<span class="p">{</span><span class="mi">2</span><span class="p">,</span> <span class="mi">0</span><span class="p">}</span> <span class="o">=></span> <span class="ss">:tree</span><span class="p">,</span>
<span class="p">{</span><span class="mi">0</span><span class="p">,</span> <span class="mi">1</span><span class="p">}</span> <span class="o">=></span> <span class="ss">:actor</span><span class="p">,</span>
<span class="p">{</span><span class="mi">0</span><span class="p">,</span> <span class="mi">2</span><span class="p">}</span> <span class="o">=></span> <span class="ss">:actor</span><span class="p">,</span>
<span class="p">{</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">}</span> <span class="o">=></span> <span class="ss">:tree</span>
<span class="p">}</span>
</code></pre></div></div>
<p>While that isn’t much sparser in those examples, my hope was that with a larger map section with trees and actors spread out that this would be less to send.</p>
</details>
<p>Unfortunately, it didn’t seem to actually help much. Thinking about this later, I realized one reason could be that I was starting all of the actors in the same place on the map, meaning that the “sparse” format wasn’t making much of a difference in the case that I was testing.</p>
<p>While this didn’t help, I still kept the change as I felt that it was a better solution and because I didn’t have any reason to think it would perform worse 😊</p>
<h1 id="sending-map-updates-asyncronously">Sending Map Updates Asyncronously</h1>
<p>The next thing that I tried was simply returning <code class="language-plaintext highlighter-rouge">true</code> when handling <code class="language-plaintext highlighter-rouge">move</code> messages but sending a map update to clients asyncronously. While this didn’t help with scaling, I was able to fix the above problem where players only see updates when they move. I also felt that it was a good way to setup the messages as it is the same way that LiveView works (incoming messages update the state and LiveView sends outgoing websocket messages to all clients whenever there is a need for them to know about state changes).</p>
<h1 id="reducing-the-map-section-size-returned">Reducing the Map Section Size Returned</h1>
<p>My next experiment was to reduce the size of the map section which is returned to clients. Originally I returned a 27x27 section of the map which I reduced to a 17x17 section. This means that the number of potential squares returned is 289 instead of 729. This helped me have up to 130 actors at once, which was definitely a good jump! This was an easy change to make because I didn’t have a specific purpose for the game yet, but if this is an actual game that I wanted to release I would want to think as the “game designer” what map size is important. Overall, though, this would be much less important given the next thing that I tried…</p>
<h1 id="allow-clients-to-ask-for-map-updates">Allow Clients to Ask For Map Updates</h1>
<p>If we have, for example, 100 clients which are moving once every second and each client gets an update whenever another client moves, the game server will need to send around 10,000 messages every second! While the LiveView clients which connect to the game need to these map updates from the game server, the actor clients which are just moving randomly didn’t use that information. This means that we can avoid sending many of these messages! So my next idea: allow a client to specify if they want updates (or not) when they register.</p>
<p>Since there was already a <code class="language-plaintext highlighter-rouge">register</code> message that clients send on startup, I was easily able to add an <code class="language-plaintext highlighter-rouge">options</code> argument which can have a <code class="language-plaintext highlighter-rouge">return_view_update</code> key. By having actors specify that they didn’t want updates I was able to spin up 1,000 actors. Big progress!</p>
<h1 id="scale-back-the-rate-of-updates">Scale back the rate of updates</h1>
<p>Increasing the number of actors by 770% is great, but even though our <code class="language-plaintext highlighter-rouge">LiveView</code> process is the only one receiving updates it is still getting 1,000 updates every second! A human eye only sees updates on the order of 60 frames per second. Fortunately it’s easy to use <code class="language-plaintext highlighter-rouge">Process.send_after</code> to have our game server send itself a message on a regular basis. Using this, I started with having the game server update clients around once every 200ms. This meant five frames per second.</p>
<p>This was a bit sluggish, but I realized that the most important thing is for a player to see their <em>own</em> updates quickly. If there is a 200-300ms delay between an actor moving and the player seeing it then probably players won’t notice. So, I simply send a message back to just the client every time they move (if they’ve set <code class="language-plaintext highlighter-rouge">return_view_update</code>). This means that they get an update as fast as possible without worrying about updating other clients immediately.</p>
<p>Overall this worked quite well, allowing me to have around 6000 actors at once! Since making updates to players now happens asyncronously this change was super easy!</p>
<h1 id="scaling-too-high">Scaling Too High</h1>
<p>I wrote above that I had scaled to around 6000 actors, but for a little while I was scaling up and up and up until I got to around 50,000 actors thinking that I had stumbled onto something amazing! It was then that my skepticism kicked in, and I realized I was just changing the number of trees generated on the board. When I thought I was able to handle 50,000 clients, it was really 50,000 trees. So, while not as useful, I guess at least I know I can have at least 50,000 trees in my fake game.</p>
<p>(Also, to be completely open: I end up making the above mistake one more time later on. 🤦♂️)</p>
<h1 id="time-out">Time Out!</h1>
<p>It was about this point that I was starting to see timeouts from the actors during startup. There are a couple of twists here, so I’ll try to lead you carefully along this path:</p>
<p>Since early in the development of the game server the actors have been started up in a <code class="language-plaintext highlighter-rouge">handle_continue</code>. The <code class="language-plaintext highlighter-rouge">handle_continue</code> callback exists to continue the work of the <code class="language-plaintext highlighter-rouge">init</code> callback, both of which are for running code when the process first starts up. During <code class="language-plaintext highlighter-rouge">init</code>, though, messages can’t arrive in the processes’ mailbox while they can during <code class="language-plaintext highlighter-rouge">handle_continue</code>. Since starting up actors isn’t critical to getting a game server running, I put the login inside of <code class="language-plaintext highlighter-rouge">handle_continue</code></p>
<p>My worry was that the actor startup was taking too long. The default timeout for <code class="language-plaintext highlighter-rouge">GenServer</code> messages is 5000ms, so even if the mailbox can receive messages during the running of <code class="language-plaintext highlighter-rouge">handle_continue</code>, if <code class="language-plaintext highlighter-rouge">handle_continue</code> takes too long some messages may time out while waiting. If the message times out, then the actor crashes. Perhaps starting up more and more actors was keeping the game server from getting down to it’s job quickly enough?</p>
<p>What’s one thing to try in Elixir when you’ve got a problem? Create another process! <a href="##" title="just kidding... mostly"><sup>1</sup></a> I thought to use a <code class="language-plaintext highlighter-rouge">Task</code> (specifically <code class="language-plaintext highlighter-rouge">Task.Supervisor.async_nolink</code>) to startup a temporary process which just has the job of starting up all of the actors.</p>
<p>That was maybe a good idea, in a way, but it didn’t actually help with the timeouts! After some tracing and debugging I found that it actually started up the actors quite quickly! It seemed that the <code class="language-plaintext highlighter-rouge">register</code> and <code class="language-plaintext highlighter-rouge">move</code> messages were just coming at the server so quickly at first that it struggled to keep up. After a certain number of clients, it couldn’t respond to messages within the 5000ms timeout.</p>
<p>So, ok, the timeout is just a default, so increase the timeout, right? That certainly works! But:</p>
<ul>
<li>Since it’s the default, I didn’t want to change it without a good reason</li>
<li>If I change the timeout, I couldn’t compare my previous results to my future results</li>
</ul>
<p>So, with increasing timeouts as a tool in my pocket, I moved on!</p>
<h1 id="trying-mapupdate">Trying <code class="language-plaintext highlighter-rouge">Map.update!</code></h1>
<p>Another small improvement I tried: Using <code class="language-plaintext highlighter-rouge">Map.update</code> instead of <code class="language-plaintext highlighter-rouge">Map.update!</code>. My thought was that <code class="language-plaintext highlighter-rouge">Map.update</code> was probably doing some sort of <code class="language-plaintext highlighter-rouge">if</code> check for the key. Perhaps without that and depending on an exception be raised/caught would improve things? That turned out to be very <em>not</em> true, and with the help of the <code class="language-plaintext highlighter-rouge">benchee</code> library I was able to see how they compared in cases where the key in question existed or not.</p>
<p>You can see the results of that benchmark in the snippets below. But, regardless of which version, I tried it didn’t make a difference to how many actors I could create. It was great to learn something about Elixir, but this was not a bottleneck.</p>
<details class="tabbed-git-gist">
<summary>See the code</summary>
<div x-data="{data : {files: []}, current_tab: 'benchmark.exs'}" x-init="data = await (await fetch('https://api.github.com/gists/e00bb6f451f4eed32e96fd481880a920')).json(); current_tab = Object.keys(data.files)[0]">
<nav>
<template x-for="filename in Object.keys(data.files)" x-bind:key="filename">
<div class="tab" x-bind:class="{ 'active': current_tab === filename }" x-on:click.prevent="current_tab = filename; window.location.hash = filename" x-text="filename" href="#"></div>
</template>
</nav>
<div class="tabs-body">
<template x-for="filename in Object.keys(data.files)" x-bind:key="filename">
<iframe x-show="current_tab === filename" x-bind:srcdoc="`<script src='https://gist.github.com/e00bb6f451f4eed32e96fd481880a920.js?file=${filename}'></script>`">
</iframe>
</template>
</div>
</div>
<style>
iframe {
border: 0;
width: 100%;
height: 307px;
}
.tab {
display: inline-block;
border: 1px solid black;
border-bottom: 0;
background-color: white;
padding: 0.4em;
}
.tab.active {
position: relative;
top: 1px;
background-color: #ddd;
}
.tabs-body {
background-color: #ddd;
border: 1px solid black;
}
</style>
</details>
<h1 id="move_and_update-a-very-specific-solution"><code class="language-plaintext highlighter-rouge">move_and_update</code>: A Very Specific Solution</h1>
<p>I thought perhaps that the <code class="language-plaintext highlighter-rouge">if</code> check for <code class="language-plaintext highlighter-rouge">return_view_update</code> could be a place to improve. Perhaps instead of registering to request updates during registration, clients could send a <code class="language-plaintext highlighter-rouge">move_and_update</code> message instead of a <code class="language-plaintext highlighter-rouge">move</code> when they want an update afterward. It did seem to help a bit (allowing me to go from around 8,000 actors to around 9,000), but I didn’t really like the solution, so I decided the improvement wasn’t worth it. I like the simplicity of just having the <code class="language-plaintext highlighter-rouge">move</code> message overall. If using <code class="language-plaintext highlighter-rouge">move_and_update</code> made a big difference or if it helped me to meet the requirements of the application I’m building I would revisit this.</p>
<h1 id="having-actors-wait-on-startup">Having Actors Wait on Startup</h1>
<p>I thought about how I could independently control:</p>
<ul>
<li>how long actors wait before sending their first message (via process startup)</li>
<li>how long an actor waits between moves (via the repeating <code class="language-plaintext highlighter-rouge">handle_info</code>)</li>
</ul>
<p>Since I was still getting timesouts, I decided to try waiting 3000ms at startup while leaving the time between moves as 1000ms. It helped a bit: again allowing me to go from around 8,000 actors to around 9,000. Since I’m OK with the actors taking a bit more time to move, I left this one in.</p>
<h1 id="a-discovery">A Discovery!</h1>
<p>Then I came to a big discovery. During client registration the server was still sending back the map section as a response, even if the player hadn’t asked for updates! Since registration happens just once, while the <code class="language-plaintext highlighter-rouge">move</code> message happens a lot, this might not seem like a big deal. However, when the game tried to start up many, many actors, it is just the sort of thing that could cause it get behind in processing it’s queue and…. create timeouts!</p>
<p>So I simply had the response from <code class="language-plaintext highlighter-rouge">register</code> be a <code class="language-plaintext highlighter-rouge">true</code>. Then I could just call the same function that I already had to queue up a map update message for the client (if they’ve set <code class="language-plaintext highlighter-rouge">return_view_update</code>). With that change, I was then able to start up around 13,000 actors, making for a 44% increase!</p>
<h1 id="a-step-backward">A Step Backward</h1>
<p>I was pretty sure that this wouldn’t make things better, but it turned out to make things much worse:</p>
<p>I tried putting the logic of sending map updates during <code class="language-plaintext highlighter-rouge">register</code> or <code class="language-plaintext highlighter-rouge">move</code> in a <code class="language-plaintext highlighter-rouge">handle_continue</code>. This is for code which executes after a message is handled, just like after <code class="language-plaintext highlighter-rouge">init</code>. This didn’t allow me to scale any more, but it also had the effect that responsiveness to player moves were slower, especially when the game server had high queue lengths.</p>
<p>Maybe using “continue” meant the server sending a message to itself and that the message goes to back of the queue? That would mean that the map update message had to wait to be sent instead of being sent right away. I’m not 100% sure about this though. When “continue” is used during init of a GenServer it is guarunteed to be processed before other messages, but when it’s used in a <code class="language-plaintext highlighter-rouge">handle_call</code> where other messages have already been queued up that may not be the same</p>
<p>Anyway, I removed that change :)</p>
<h1 id="sleep-to-scale">Sleep to Scale</h1>
<p>Even with the reduction in work when handling <code class="language-plaintext highlighter-rouge">register</code> messages, I would still, at some point, get timeouts. I thought perhaps that if I put a delay (say 10ms) between startup of each of my actors I could avoid that or at least push it back further. This stretches out the time it would take to start up all of the actors to at least a couple of minutes. If I were still doing the work directly in the <code class="language-plaintext highlighter-rouge">handle_continue</code> this would be a problem as the game server wouldn’t be able to respond to messages during that time. Fortunately I was still using <code class="language-plaintext highlighter-rouge">Task.Supervisor.async_nolink</code> so the the game server gets down to handling messages just as soon as it’s handed off the task of starting up actors to the <code class="language-plaintext highlighter-rouge">Task</code>.</p>
<p>Overall it helped some! I was able to startup around 15,000 actors and the server was pretty stable and responsive (though at that point the queue generally wasn’t getting worked down and <code class="language-plaintext highlighter-rouge">move</code> response time can be between 1000 and 2000ms). But as an unexpected benefit: by starting up the actors slowly I was able to more easily see the point at which the metrics show things becoming unstable.</p>
<h1 id="other-notes--learnings">Other Notes / Learnings</h1>
<h2 id="visualizing-possibilites">Visualizing Possibilites</h2>
<p>I found it fascinating to watch the actors spread out from a single point. Since the actors move randomly one space at a time, the most likely thing for them to do would be to stay in one spot. If I spawned 1,000 actors then I could see some making it out of the central mass, giving a viseral sense of how far actors could go in the rare cases. It’s was a sort-of live 2D histogram of potential end positions after X number of moves.</p>
<h2 id="livedashboard">LiveDashboard</h2>
<p>LiveDashboard and <code class="language-plaintext highlighter-rouge">telemetry</code> were amazing for getting up-and-running very quickly, but it was sometimes frustrating to only see the recent values. Also, sometimes when I would refresh the metrics would change dramatically, giving me the feeling that something wasn’t updating correctly. Sending the metrics to statsd/graphite was slower and had less resolution, but felt more reliable. I’m super glad that <code class="language-plaintext highlighter-rouge">telemetry</code> allows for both options (and more) to exist!</p>
<h1 id="other-potential-improvements">Other Potential Improvements</h1>
<p>At this point I had made a lot of progress and learned a lot of things. If I would actually want to use this for a real project, there are some other things that I could investigate improving:</p>
<h2 id="using-an-ets-table">Using an ETS Table</h2>
<p>If I used an ETS table that only the game server writes to, it could allow client processes to retrieve a map state by querying the table. This could reduce the load of the game server and maybe allowing for better scaling.</p>
<h2 id="reducing-actor-movements">Reducing Actor Movements</h2>
<p>If the actors moved less often then the game server would have fewer messages / second, allowing it to support more clients at once. Having actors wait a random amount of time between movements could preserve a feeling of realism for the player.</p>
<h1 id="potential-challenges-for-building-a-game">Potential Challenges for Building a Game</h1>
<p>I also have some ideas for what I might do to actually turn this into a real game (at least something which people might want to play for a bit as a demo)</p>
<h2 id="testing-real-player-load">Testing Real Player Load</h2>
<p>I’ve pushed the limit for how many actors which can be run at once, but they don’t need the map updates. Adding (or simulating) many real players would be different and might require a strategy like having an ETS table.</p>
<h2 id="enforcing-game-rules">Enforcing Game Rules</h2>
<p>There will probably be a need to implement some game rules. For a start I might implement a rule like “players/actors cannot be in the same spot as a tree”.</p>
<h2 id="other-kinds-of-actors">Other Kinds of Actors</h2>
<p>While having just one kind of actor made benchmarking straightforward, it would be great to make the world more interesting by adding other kinds of actors. One example I can imagine would be bacteria emoji which grow and die with rules similar to <a href="https://en.wikipedia.org/wiki/Conway%27s_Game_of_Life">Conway’s Game of Life</a>.</p>Brian UnderwoodI’ve been using Elixir for a while and I’ve implemented a number of GenServers. But while I think I mostly understand the purpose of them, I’ve not gotten the chance to push one it’s limits, scale it up, and find ways to address it’s bottlenecks. I thought that it would be fun to create something to which I could give the URL as part of a presentation and have some confidence that it would be able to handle all the users who connected to it. So recently I implemented a simple game grid using Phoenix LiveView and emojis as indicators of player and objects. If you would like to learn about my journey, read on! But note that you’ll probably want to have at least a basic understanding of GenServers first. You might start by reading this and/or watching this.Why I Love Lodash2021-05-13T22:26:00+02:002021-05-13T22:26:00+02:00http://www.brian-underwood.codes//javascript/2021/05/13/why-I-love-lodash<p>I love <a href="https://lodash.com/">Lodash</a>, but I’m not here to tell you to use Lodash. It’s up to you to decide if a tool is useful for you or your project. It will come down to the needs of the project (file size, browser/Node.js, how much you use it, etc…). But my new team was surprised by my passion for it and so I wanted to share my thoughts. Specifically: I want to focus in this post on those things which I like, including things which I think people often miss.</p>
<h2 id="checking-types">Checking types</h2>
<p>To start with something simple, let’s look at identifying variable types. I’ll take the examples from <a href="https://dev.to/jmitchell38488/it-s-time-to-let-go-of-lodash-221f">this post</a>:</p>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">Array</span><span class="p">.</span><span class="nf">isArray</span><span class="p">(</span><span class="nx">foo</span><span class="p">);</span>
<span class="nx">foo</span> <span class="o">===</span> <span class="kc">null</span><span class="p">;</span>
<span class="nx">foo</span> <span class="o">!==</span> <span class="kc">null</span> <span class="o">&&</span> <span class="k">typeof</span> <span class="nx">foo</span> <span class="o">===</span> <span class="dl">"</span><span class="s2">object</span><span class="dl">"</span> <span class="o">&&</span> <span class="nc">Object</span><span class="p">(</span><span class="nx">foo</span><span class="p">)</span> <span class="o">===</span> <span class="p">{};</span>
<span class="k">typeof</span> <span class="nx">foo</span> <span class="o">===</span> <span class="dl">'</span><span class="s1">boolean</span><span class="dl">'</span>
</code></pre></div></div>
<p>All of that works, technically, but it’s so inconsistent. This is especially true of the third example checking if something is an object. Such a check requires a deeper understanding of Javascript objects that not everybody has and it might not always be implemented correctly.</p>
<p>Also, regarding the last example for checking boolean, I should admit that I lied a bit: it isn’t from the post I linked to! The example they gave was: <code class="language-plaintext highlighter-rouge">Boolean(foo)</code>, but in the comments somebody pointed out that truthy values like <code class="language-plaintext highlighter-rouge">Boolean(1)</code> would return <code class="language-plaintext highlighter-rouge">true</code>. For me, all of this is more complex and error prone than just doing:</p>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nx">_</span><span class="p">.</span><span class="nf">isArray</span><span class="p">(</span><span class="nx">foo</span><span class="p">);</span>
<span class="nx">_</span><span class="p">.</span><span class="nf">isNull</span><span class="p">(</span><span class="nx">foo</span><span class="p">);</span>
<span class="nx">_</span><span class="p">.</span><span class="nf">isObject</span><span class="p">(</span><span class="nx">foo</span><span class="p">);</span>
<span class="nx">_</span><span class="p">.</span><span class="nf">isBoolean</span><span class="p">(</span><span class="nx">foo</span><span class="p">);</span>
</code></pre></div></div>
<p>With those, I’m a lot less likely to come back to my code later because of a bug due to my understanding of type checking.</p>
<h2 id="chaining">Chaining</h2>
<p>I very often have a need to transform a data structure in multiple steps:</p>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// Find adults, group by age (ten-year spans), and find if any in each group own a pet</span>
<span class="kd">let</span> <span class="nx">people</span> <span class="o">=</span> <span class="p">[{</span><span class="na">id</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span> <span class="na">name</span><span class="p">:</span> <span class="dl">'</span><span class="s1">Jane Doe</span><span class="dl">'</span><span class="p">,</span> <span class="na">age</span><span class="p">:</span> <span class="mi">33</span><span class="p">,</span> <span class="na">ownsPet</span><span class="p">:</span> <span class="kc">true</span><span class="p">},</span> <span class="p">...]</span>
<span class="kd">let</span> <span class="nx">adults</span> <span class="o">=</span> <span class="nx">people</span><span class="p">.</span><span class="nf">filter</span><span class="p">((</span><span class="nx">person</span><span class="p">)</span> <span class="o">=></span> <span class="nx">person</span><span class="p">.</span><span class="nx">age</span> <span class="o">>=</span> <span class="mi">18</span><span class="p">)</span>
<span class="kd">let</span> <span class="nx">groupedAdults</span> <span class="o">=</span> <span class="nx">adults</span><span class="p">.</span><span class="nf">reduce</span><span class="p">((</span><span class="nx">result</span><span class="p">,</span> <span class="nx">adult</span><span class="p">)</span> <span class="o">=></span> <span class="p">{</span>
<span class="kd">let</span> <span class="nx">groupingNumber</span> <span class="o">=</span> <span class="nb">Math</span><span class="p">.</span><span class="nf">floor</span><span class="p">(</span><span class="nx">adult</span><span class="p">.</span><span class="nx">age</span> <span class="o">/</span> <span class="mi">10</span><span class="p">)</span> <span class="o">*</span> <span class="mi">10</span><span class="p">;</span>
<span class="nf">if </span><span class="p">(</span><span class="nx">result</span><span class="p">[</span><span class="nx">groupingNumber</span><span class="p">]</span> <span class="o">==</span> <span class="kc">null</span><span class="p">)</span> <span class="p">{</span> <span class="nx">result</span><span class="p">[</span><span class="nx">groupingNumber</span><span class="p">]</span> <span class="o">=</span> <span class="p">[]</span> <span class="p">}</span>
<span class="nx">result</span><span class="p">[</span><span class="nx">groupingNumber</span><span class="p">].</span><span class="nf">push</span><span class="p">(</span><span class="nx">adult</span><span class="p">);</span>
<span class="nf">return</span><span class="p">(</span><span class="nx">result</span><span class="p">);</span>
<span class="p">},</span> <span class="p">{})</span>
<span class="kd">let</span> <span class="nx">petExistsForGroup</span> <span class="o">=</span> <span class="p">{}</span>
<span class="nb">Object</span><span class="p">.</span><span class="nf">keys</span><span class="p">(</span><span class="nx">groupedAdults</span><span class="p">).</span><span class="nf">forEach</span><span class="p">((</span><span class="nx">key</span><span class="p">)</span> <span class="o">=></span> <span class="p">{</span>
<span class="kd">let</span> <span class="nx">petExists</span> <span class="o">=</span> <span class="kc">false</span><span class="p">;</span>
<span class="nx">groupedAdults</span><span class="p">[</span><span class="nx">key</span><span class="p">].</span><span class="nf">forEach</span><span class="p">((</span><span class="nx">adult</span><span class="p">)</span> <span class="o">=></span> <span class="p">{</span>
<span class="nx">petExists</span> <span class="o">=</span> <span class="nx">petExists</span> <span class="o">||</span> <span class="nx">adult</span><span class="p">.</span><span class="nx">ownsPet</span><span class="p">;</span>
<span class="p">})</span>
<span class="nx">petExistsForGroup</span><span class="p">[</span><span class="nx">key</span><span class="p">]</span> <span class="o">=</span> <span class="nx">petExists</span><span class="p">;</span>
<span class="p">})</span>
</code></pre></div></div>
<p>Seem about what you expect from Javascript? Maybe not the nicest bit of code in the world, but it does the job. Good enough, right? Let’s try using Lodash chaining, along with other helpers:</p>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">let</span> <span class="nx">people</span> <span class="o">=</span> <span class="p">[{</span><span class="na">id</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span> <span class="na">name</span><span class="p">:</span> <span class="dl">'</span><span class="s1">Jane Doe</span><span class="dl">'</span><span class="p">,</span> <span class="na">age</span><span class="p">:</span> <span class="mi">33</span><span class="p">,</span> <span class="na">ownsPet</span><span class="p">:</span> <span class="kc">true</span><span class="p">},</span> <span class="p">...]</span>
<span class="kd">let</span> <span class="nx">petExistsForGroup</span> <span class="o">=</span>
<span class="nf">_</span><span class="p">(</span><span class="nx">people</span><span class="p">)</span>
<span class="p">.</span><span class="nf">filter</span><span class="p">((</span><span class="nx">person</span><span class="p">)</span> <span class="o">=></span> <span class="nx">person</span><span class="p">.</span><span class="nx">age</span> <span class="o">>=</span> <span class="mi">18</span><span class="p">)</span>
<span class="p">.</span><span class="nf">groupBy</span><span class="p">((</span><span class="nx">adult</span><span class="p">)</span> <span class="o">=></span> <span class="nb">Math</span><span class="p">.</span><span class="nf">floor</span><span class="p">(</span><span class="nx">adult</span><span class="p">.</span><span class="nx">age</span> <span class="o">/</span> <span class="mi">10</span><span class="p">)</span> <span class="o">*</span> <span class="mi">10</span><span class="p">)</span>
<span class="p">.</span><span class="nf">mapValues</span><span class="p">((</span><span class="nx">adults</span><span class="p">)</span> <span class="o">=></span> <span class="nx">_</span><span class="p">.</span><span class="nf">some</span><span class="p">(</span><span class="nx">adults</span><span class="p">,</span> <span class="dl">'</span><span class="s1">ownsPet</span><span class="dl">'</span><span class="p">))</span>
<span class="p">.</span><span class="nf">value</span><span class="p">()</span>
</code></pre></div></div>
<p>It’s another way of thinking, right? Shorter for sure, but also higher-level and more declarative. The mechanics drop away and we’re left with our code business logic.</p>
<p>This is one of the big things that I love about Lodash. If you take the time to learn it, it allows you to work at a higher level, not worrying about small details. For that reason I enjoy just browsing through the documentation sometimes, like browsing an IKEA catalog and thinking to myself “oooh, that could be nice…”. It’s good to know what’s there, even if you’re not sure why you would use it. I’ve found <code class="language-plaintext highlighter-rouge">_.partition</code> surprisingly useful!</p>
<p>Did I choose an example that plays to Lodash’s strengths? Absolutely! But I challenge anybody to make the first example shorter while still being readable using pure Javascript.</p>
<p>It’s also worth mentioning that Lodash has methods which basically just duplicate the functionality of their javascript counterparts (e.g. <code class="language-plaintext highlighter-rouge">_.concat</code> or <code class="language-plaintext highlighter-rouge">_.fill</code>). This maybe seems unnecessary to people who are skeptical of Lodash, but remember that because Lodash offers chaining it needs to provide all of the potential methods that you might use.</p>
<h2 id="iteratee-shorthand">Iteratee shorthand</h2>
<p>You may have noted the <code class="language-plaintext highlighter-rouge">_.some(adults, 'ownsPet')</code> bit above. By passing in a string instead of a function, Lodash automatically uses an identity function. This would be the equivilent of: <code class="language-plaintext highlighter-rouge">_.some(adults, (adult) => adult.ownPet)</code>. True the second version isn’t much longer, but the first is more at-a-glance readable.</p>
<p>But it doesn’t stop there! You can use the string syntax for nested paths:</p>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">let</span> <span class="nx">things</span> <span class="o">=</span> <span class="p">[{</span><span class="na">name</span><span class="p">:</span> <span class="dl">'</span><span class="s1">Car</span><span class="dl">'</span><span class="p">,</span> <span class="na">owner</span><span class="p">:</span> <span class="p">{</span><span class="na">contactDetails</span><span class="p">:</span> <span class="p">{</span><span class="na">phoneNumber</span><span class="p">:</span> <span class="dl">'</span><span class="s1">123-456-7890</span><span class="dl">'</span><span class="p">,</span> <span class="na">email</span><span class="p">:</span> <span class="dl">'</span><span class="s1">me@thingowner.com</span><span class="dl">'</span><span class="p">}}},</span> <span class="p">...]</span>
<span class="nx">_</span><span class="p">.</span><span class="nf">map</span><span class="p">(</span><span class="nx">things</span><span class="p">,</span> <span class="dl">'</span><span class="s1">owner.contactDetails.phoneNumber</span><span class="dl">'</span><span class="p">)</span>
<span class="nx">_</span><span class="p">.</span><span class="nf">find</span><span class="p">(</span><span class="nx">things</span><span class="p">,</span> <span class="p">[</span><span class="dl">'</span><span class="s1">owner.contactDetails.email</span><span class="dl">'</span><span class="p">,</span> <span class="dl">'</span><span class="s1">me@thingowner.com</span><span class="dl">'</span><span class="p">])</span>
<span class="c1">// ... more?</span>
</code></pre></div></div>
<p>Not only does it provide a simple syntax for going deep, but if any step along the way doesn’t exist (say, the <code class="language-plaintext highlighter-rouge">contactDetails</code> field is <code class="language-plaintext highlighter-rouge">null</code> or <code class="language-plaintext highlighter-rouge">undefined</code>), it will simply return <code class="language-plaintext highlighter-rouge">undefined</code>, similar to Javascript’s <code class="language-plaintext highlighter-rouge">?.</code> operator.</p>
<h2 id="javascript-as-the-write-it-yourself-language">Javascript as the “write it yourself” language</h2>
<p>Having been a Ruby programmer for a long time, I will often search for a way to manipulate some data in Ruby and very often there will by either a simple function to call or a discussion on how to write something concise but readable. Whenever I’m working with Javascript, however, the answer very often seems to be either “write it yourself” or “copy a solution”. Here is a small selection that I gathered relatively quickly:</p>
<ul>
<li><a href="https://stackoverflow.com/questions/45342155/how-to-subtract-one-array-from-another-element-wise-in-javascript">How to subtract one array from another, element-wise, in javascript</a> (<code class="language-plaintext highlighter-rouge">_.difference</code>)</li>
<li><a href="https://stackoverflow.com/questions/4550505/getting-a-random-value-from-a-javascript-array">Getting a random value from a JavaScript array</a> (<code class="language-plaintext highlighter-rouge">_.sample</code>)</li>
<li><a href="https://stackoverflow.com/questions/979256/sorting-an-array-of-objects-by-property-values">Sorting an array of objects by property values</a>(<code class="language-plaintext highlighter-rouge">_.sortBy</code>)</li>
<li><a href="https://stackoverflow.com/questions/22015684/how-do-i-zip-two-arrays-in-javascript">How do I zip two arrays in JavaScript?</a> (<code class="language-plaintext highlighter-rouge">_.zip</code>)</li>
</ul>
<p>Sometimes you’ll even get answers, like with <a href="https://stackoverflow.com/questions/2450954/how-to-randomize-shuffle-a-javascript-array">How to randomize (shuffle) a JavaScript array?</a> (<code class="language-plaintext highlighter-rouge">_.shuffle</code>) which talks about the ideal algorithm (the “Fisher-Yates (aka Knuth) Shuffle” in this case). Academically I enjoy learning about different algorithms, but when I’m trying to get higher-level done I just want something that works well. I’m not sure if the performance of my sorting / shuffling algorithm has ever been a practical concern.</p>
<p>Some of these solutions are very complex, but some are pretty simple, even if they aren’t as simple as just calling a Lodash function. So what’s so bad about using pure Javascript to zip two arrays?</p>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nx">a</span><span class="p">.</span><span class="nf">map</span><span class="p">(</span><span class="kd">function</span><span class="p">(</span><span class="nx">e</span><span class="p">,</span> <span class="nx">i</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="p">[</span><span class="nx">e</span><span class="p">,</span> <span class="nx">b</span><span class="p">[</span><span class="nx">i</span><span class="p">]];</span>
<span class="p">});</span>
</code></pre></div></div>
<p>The problem, as I see it, is that it:</p>
<ul>
<li>is not immediately obvious</li>
<li>is a distraction from the task that you’re trying to accomplish</li>
<li>leaves room for custom functions to be implemented in broken, odd, or inconsistent ways</li>
</ul>
<p>The more time that I’ve spent programming, the less I trust myself and the more I trust open source solutions which have been developed and vetted by a community. Also, most of my experience in programming comes from working in teams. When working in teams the goal is for code to be readable and maintainable. If you scatter the codebase with versions of functions that have been implemented over and over again elsewhere (almost certainly in a better way), you’re generally just spending extra time and slowing others down.</p>
<p>Of course there’s nothing wrong with enjoying the challenge of writing your own algorithms! I just think it’s much better done in https://exercism.io rather than your codebase.</p>
<h2 id="being-fair-to-javascript">Being fair to Javascript</h2>
<p>All of the above said, Javascript has come a long way in recent years. Some examples:</p>
<ul>
<li>I love the fact that I can use <code class="language-plaintext highlighter-rouge">{a: 'b', ...obj}</code> instead of <code class="language-plaintext highlighter-rouge">_.merge({a: 'b'}, obj)</code> (as long as it’s not part of a larger flow which works better as a Lodash chain)</li>
<li>I often want to return the unique items from an array, and so <code class="language-plaintext highlighter-rouge">[...new Set(array)]</code> provides a reasonably readable solution (if only slightly more verbose and less obvious than <code class="language-plaintext highlighter-rouge">_.uniq(array)</code>)</li>
<li>In ES2019 arrays now have <code class="language-plaintext highlighter-rouge">flat()</code> and <code class="language-plaintext highlighter-rouge">flatMap()</code> functions</li>
</ul>
<h2 id="summing-up">Summing up</h2>
<p>Lodash is self-consistent and holistically designed. “The principle of least astonishment” is a way to not get bogged down in details, be less prone to errors, and increase overall developer happiness. When bringing in a new developer it’s easier for them to come up-to-speed quickly.</p>
<p>And my favorite response to “You Don’t Need Lodash”:</p>
<blockquote class="twitter-tweet"><p lang="en" dir="ltr">I guess not, but I want it. "You don't need Lodash/Underscore" <a href="https://t.co/keLJ43U0pa">https://t.co/keLJ43U0pa</a></p>— Tero Parviainen (@teropa) <a href="https://twitter.com/teropa/status/692280179666898944?ref_src=twsrc%5Etfw">January 27, 2016</a></blockquote>
<script async="" src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>Brian UnderwoodI love Lodash, but I’m not here to tell you to use Lodash. It’s up to you to decide if a tool is useful for you or your project. It will come down to the needs of the project (file size, browser/Node.js, how much you use it, etc…). But my new team was surprised by my passion for it and so I wanted to share my thoughts. Specifically: I want to focus in this post on those things which I like, including things which I think people often miss.Structuring an Elixir+Phoenix App2020-07-11T00:00:00+02:002020-07-11T00:00:00+02:00http://www.brian-underwood.codes//elixir/2020/07/11/structuring-an-elixir+phoenix-app<p>I’ve <code class="language-plaintext highlighter-rouge">mix phx.new</code> ed many applications and when doing so I often start with wondering how to organize my code. I love how Phoenix pushes you to think about the different domains in your app via generators while at the same time I have the freedom to organize modules on my own. Ecto schemas make for a nice abstraction, but where should I put code related just to that table? It could be in the context, but I don’t want the context to become a “grab bag” of unorganized function calls.</p>
<p>In the past, I’ve searched for someone writing on the subject but haven’t come up with much. I’ve even done some cursory glancing into repositories to get a feeling for what they do, but I’ve never looked thoroughly at different options. In this post, I share what I have found from four different open source Phoenix+Ecto applications. And as the old joke goes, I’ll be asking four developers for their opinions and getting four different answers. In the end, I’ll summarize how I plan to move forward.</p>
<p>Notes:</p>
<p>Phoenix has evolved in how modules are organized, most notably splitting into <a href="https://hexdocs.pm/phoenix/directory_structure.html#content"><code class="language-plaintext highlighter-rouge">my_app</code> and <code class="language-plaintext highlighter-rouge">my_app_web</code></a> folders and with the concept of <a href="https://hexdocs.pm/phoenix/contexts.html">contexts</a>. Some of these applications were created with early versions of Phoenix which could explain some of the differences.</p>
<p>When I say “typical Ecto schema logic” below, I’m referring to examples in the <a href="https://hexdocs.pm/ecto/Ecto.Schema.html">Ecto documentation</a> and the community on the things to put into schema files (field definitions, schema attributes (such as <code class="language-plaintext highlighter-rouge">@primary_key</code>, <code class="language-plaintext highlighter-rouge">@schema_prefix</code>, etc…), and changeset logic)</p>
<h2 id="avia">Avia</h2>
<p><a href="https://github.com/aviacommerce/avia/">GitHub</a></p>
<p><strong>Repository description:</strong> “open source e-commerce framework”</p>
<p>A lot of the business logic can be found under <code class="language-plaintext highlighter-rouge">apps/snitch_core/lib/core</code>. There is a <code class="language-plaintext highlighter-rouge">domain</code> folder containing what appears to be the front-end API modules (what Phoenix might call “contexts”). Next to the <code class="language-plaintext highlighter-rouge">domain</code> folder is a <code class="language-plaintext highlighter-rouge">data</code> folder containing <code class="language-plaintext highlighter-rouge">schema</code> and <code class="language-plaintext highlighter-rouge">model</code> directories.</p>
<p>The <code class="language-plaintext highlighter-rouge">schema</code> directory contains typical Ecto schema files. The <code class="language-plaintext highlighter-rouge">model</code> directory contains correspondingly named modules with CRUD functions (like <code class="language-plaintext highlighter-rouge">create</code>, <code class="language-plaintext highlighter-rouge">update</code>, <code class="language-plaintext highlighter-rouge">delete</code>, <code class="language-plaintext highlighter-rouge">get</code>) but also occasionally some helper functions related to those domain objects (functions like <code class="language-plaintext highlighter-rouge">formatted_list</code> or <code class="language-plaintext highlighter-rouge">get_all_by_shipping_category</code>)</p>
<p>Each type of module also has a <code class="language-plaintext highlighter-rouge">use</code> statement at the top (i.e. <code class="language-plaintext highlighter-rouge">use Snitch.Data.Model</code>) referring to a module containing shared logic. It’s worth looking at what that shared logic is:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> # apps/snitch_core/lib/core/domain/domain.ex
alias Ecto.Multi
alias Snitch.Data.{Model, Schema}
alias Snitch.Domain
alias Snitch.Core.Tools.MultiTenancy.Repo
# apps/snitch_core/lib/core/data/model/model.ex
import Ecto.Query
alias Snitch.Core.Tools.MultiTenancy.Repo
alias Snitch.Tools
alias Tools.Helper.Query, as: QH
# apps/snitch_core/lib/core/data/schema/schema.ex
use Ecto.Schema
import Ecto.Changeset
import Snitch.Tools.Validations
alias Snitch.Core.Tools.MultiTenancy.Repo
</code></pre></div></div>
<p>The domain modules alias the model modules and the model modules alias the schema modules, indicating the usage pattern of going deeper (Domain -> Model -> Schema):</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> # apps/snitch_core/lib/core/domain/stock/stock_location.ex
alias Model.StockLocation, as: StockLocationModel
# apps/snitch_core/lib/core/data/model/stock/stock_location.ex
alias Snitch.Data.Schema.StockLocation, as: StockLocationSchema
</code></pre></div></div>
<h2 id="changelog">Changelog</h2>
<p><a href="https://github.com/thechangelog/changelog.com">GitHub</a></p>
<p><strong>Repository description:</strong> This is the CMS behind <a href="https://changelog.com/">changelog.com</a>.</p>
<p>The business logic is under <code class="language-plaintext highlighter-rouge">lib/changelog</code>. This directory seems to contain various modules as well as directories containing grouped functionality. All of the Ecto logic looks to be under the <code class="language-plaintext highlighter-rouge">schema</code> directory which contains some base schema modules as well as directories containing grouped schema functionality.</p>
<p>Schemas have the typical Ecto schema logic but also sometimes many helpers like <code class="language-plaintext highlighter-rouge">admins</code>, <code class="language-plaintext highlighter-rouge">with_email</code>, <code class="language-plaintext highlighter-rouge">get_by_website</code> which are scoping/querying as well as defining changeset functions like <code class="language-plaintext highlighter-rouge">auth_changeset</code>, <code class="language-plaintext highlighter-rouge">admin_insert_changeset</code>, <code class="language-plaintext highlighter-rouge">admin_update_changeset</code>, <code class="language-plaintext highlighter-rouge">file_changeset</code>, etc…</p>
<p>The schemas use the <code class="language-plaintext highlighter-rouge">Changelog.Schema</code> module which, in addition to adding many helper functions like <code class="language-plaintext highlighter-rouge">any?</code>, <code class="language-plaintext highlighter-rouge">by_position</code>, <code class="language-plaintext highlighter-rouge">limit</code>, <code class="language-plaintext highlighter-rouge">newest_first</code>, <code class="language-plaintext highlighter-rouge">newest_last</code>, etc…, does this:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> use Ecto.Schema
use Arc.Ecto.Schema
import Ecto
import Ecto.Changeset
import Ecto.Query, only: [from: 1, from: 2]
import EctoEnum, only: [defenum: 2]
alias Changelog.{Hashid, Repo}
</code></pre></div></div>
<h2 id="hexpm">Hexpm</h2>
<p><a href="https://github.com/hexpm/hexpm">GitHub</a></p>
<p><strong>Repository description:</strong> API server and website for Hex <a href="https://hex.pm/">https://hex.pm</a></p>
<p>The <code class="language-plaintext highlighter-rouge">lib/hexpm</code> directory contains some modules with basic logic, but the schemas and contexts exist inside of grouping folders. For example, the <code class="language-plaintext highlighter-rouge">lib/hexpm/accounts</code> folder has the <code class="language-plaintext highlighter-rouge">User</code> schema and the <code class="language-plaintext highlighter-rouge">Users</code> context as well as the <code class="language-plaintext highlighter-rouge">Organization</code> schema and the <code class="language-plaintext highlighter-rouge">Organizations</code> context. The singular modules (i.e. <code class="language-plaintext highlighter-rouge">User</code> and <code class="language-plaintext highlighter-rouge">Organization</code>) have the typical Ecto schema logic.</p>
<p>The two types of module <code class="language-plaintext highlighter-rouge">use</code> the <code class="language-plaintext highlighter-rouge">Hexpm.Schema</code> and <code class="language-plaintext highlighter-rouge">Hexpm.Context</code> modules:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> # lib/hexpm/schema.ex
import Ecto
import Ecto.Changeset
import Ecto.Query, only: [from: 1, from: 2]
import Hexpm.Changeset
alias Ecto.Multi
use Hexpm.Shared
# lib/hexpm/context.ex
import Ecto
import Ecto.Changeset
import Ecto.Query, only: [from: 1, from: 2]
import Hexpm.Accounts.AuditLog,
only: [audit: 3, audit: 4, audit_many: 4, audit_with_user: 4]
alias Ecto.Multi
alias Hexpm.Repo
use Hexpm.Shared
</code></pre></div></div>
<p>You might have noticed that both <code class="language-plaintext highlighter-rouge">use</code> the <code class="language-plaintext highlighter-rouge">Hexpm.Shared</code> module. This just does a lot of aliases which means that modules like <code class="language-plaintext highlighter-rouge">Hexpm.Accounts.AuditLog</code> and <code class="language-plaintext highlighter-rouge">Hexpm.Repository.Download</code> become just <code class="language-plaintext highlighter-rouge">AuditLog</code> and <code class="language-plaintext highlighter-rouge">Download</code>…</p>
<p>While that pattern seems common, it’s not always the case. There is an <code class="language-plaintext highlighter-rouge">Auth</code> module which is just a plain module as well as <code class="language-plaintext highlighter-rouge">UserHandles</code> and <code class="language-plaintext highlighter-rouge">Email</code> schema modules without corresponding context modules. <code class="language-plaintext highlighter-rouge">Hexpm.Accounts.Email</code> actually seems to be used in the emails folder in <code class="language-plaintext highlighter-rouge">Hexpm.Emails</code> and <code class="language-plaintext highlighter-rouge">Hexpm.Emails.Bamboo</code>, which seems to be a case of one context reaching into another.</p>
<h2 id="elixirstatus-web">elixirstatus-web</h2>
<p><a href="https://github.com/rrrene/elixirstatus-web">GitHub</a></p>
<p><strong>Repository description:</strong> Community site for Elixir project/blog post/version updates</p>
<p>At the root of this project, there are <code class="language-plaintext highlighter-rouge">lib</code> and <code class="language-plaintext highlighter-rouge">web</code> directories. The schemas are located under <code class="language-plaintext highlighter-rouge">web/models</code>. This appears to be a pretty old app (the <code class="language-plaintext highlighter-rouge">LICENSE</code> file is five years old), which is probably why it’s not using the recent pattern of putting business logic outside of the “web” part of the app.</p>
<p>The <code class="language-plaintext highlighter-rouge">models</code> directory contains four schemas (<code class="language-plaintext highlighter-rouge">Impression</code>, <code class="language-plaintext highlighter-rouge">Posting</code>, <code class="language-plaintext highlighter-rouge">ShortLink</code>, and <code class="language-plaintext highlighter-rouge">User</code>) which all define typical Ecto schema logic. These all <code class="language-plaintext highlighter-rouge">use ElixirStatus.Web, :model</code> which does:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> use Ecto.Schema
import Ecto
import Ecto.Changeset
</code></pre></div></div>
<p>Another module under <code class="language-plaintext highlighter-rouge">web/models</code> is <code class="language-plaintext highlighter-rouge">Avatar</code> which doesn’t seem to be a schema but rather a grouping of helper functions.</p>
<p>As an example of an context-like module, the <code class="language-plaintext highlighter-rouge">Impressionist</code> module (stored at <code class="language-plaintext highlighter-rouge">lib/elixir_status/impressionist.ex</code>) defines various querying methods for the <code class="language-plaintext highlighter-rouge">Impression</code> schema along with some other helpers.</p>
<h2 id="my-thoughts">My thoughts:</h2>
<p>I already like Phoenix conventions like:</p>
<ul>
<li>Separating business logic from the web application logic</li>
<li>Separating business logic into contexts with well-established APIs</li>
<li>Ecto schema modules which are focused on mapping and validation of the data source
Things I like about these projects:</li>
<li>It’s very nice to have modules headed with something like <code class="language-plaintext highlighter-rouge">use MyApp.Schema</code> or <code class="language-plaintext highlighter-rouge">use MyApp.Context</code> as the Hexpm project does. Even if the used module doesn’t do much, it provides an at-a-glance label when browsing files.</li>
<li>I like that Hexpm has established a bit of a convention around schemas (singular <code class="language-plaintext highlighter-rouge">User</code>) vs contexts (plural <code class="language-plaintext highlighter-rouge">Users</code>).</li>
<li>I like how the Avia project separates “domain”, “model”, and “schema”. In particular as a fan of <a href="https://en.wikipedia.org/wiki/Domain-driven_design">Domain Driven Design</a> using the word “domain” is nice and I think it’s used in the same way.</li>
</ul>
<p>Things I don’t like from these projects:</p>
<ul>
<li>Aliasing the right-most module in a path (as the Avia project does) drops it’s context. If <code class="language-plaintext highlighter-rouge">Hexpm.Accounts.AuditLog</code> is aliased as <code class="language-plaintext highlighter-rouge">AuditLog</code>, that might not be so bad because <code class="language-plaintext highlighter-rouge">AuditLog</code> is potentially a unique concept. But aliasing <code class="language-plaintext highlighter-rouge">Hexpm.Repository.Download</code> as <code class="language-plaintext highlighter-rouge">Download</code> could confuse. If you alias <code class="language-plaintext highlighter-rouge">Hexpm.Accounts</code> or <code class="language-plaintext highlighter-rouge">Hexpm.Repository</code> you can refer to <code class="language-plaintext highlighter-rouge">Accounts.AuditLog</code> or <code class="language-plaintext highlighter-rouge">Repository.Download</code> which I find clearer.</li>
<li>In the Avia project sometimes there are aliases like <code class="language-plaintext highlighter-rouge">Model.StockLocation</code> aliased as <code class="language-plaintext highlighter-rouge">StockLocationModel</code>. I would find it simpler to just refer to <code class="language-plaintext highlighter-rouge">Model.StockLocation</code> which is one character longer but makes the source clearer.</li>
<li>In hexpm the schema vs context convention doesn’t help when browsing a directory to distinguish schemas from plain module files.</li>
</ul>
<p>As a long-time Rails developer, one thing that makes Rails nice is being able to go between apps easily because there is always a place for everything. But as an app grows large, grouping files by type means that directories like <code class="language-plaintext highlighter-rouge">controllers</code> and <code class="language-plaintext highlighter-rouge">models</code> get very full. The Phoenix project, I think trying to learn from Rails, encourages using contexts with well-defined APIs. Since each context often needs to solve different problems (such as wrapping a database, creating an API client, or just doing calculations), these can be structured however you like. But when it makes sense I think that we could create directories according to conventions to organize our code. For a long time, many projects have established loose conventions with directories like <code class="language-plaintext highlighter-rouge">lib</code>, <code class="language-plaintext highlighter-rouge">docs</code>, <code class="language-plaintext highlighter-rouge">log</code>, and <code class="language-plaintext highlighter-rouge">test</code>. In the web part of a Phoenix application, we have <code class="language-plaintext highlighter-rouge">controller</code>, <code class="language-plaintext highlighter-rouge">channel</code>, <code class="language-plaintext highlighter-rouge">view</code>, etc…</p>
<p>We could do the same in the very common case where our contexts contain Ecto database logic. We are given the “schema” idea from Ecto itself as a way to separate transformation and validation logic. This helps us trim the fat from our “fat model” problem. But we’re left to put other query logic either into our schema or to have it mixed it with all of our context’s business logic.</p>
<p>So after my investigation, the way that I plan to move forward:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> # The context's public API, headed with `use MyApp.Context`
my_app/<context>.ex
# Headed with `use MyApp.Schema`
my_app/<context>/schema/user.ex
# Headed with `use MyApp.Query`
my_app/<context>/query/user.ex
# For non-DB business logic
my_app/<context>/<some_module>.ex
my_app/<context>/<some_module>/<sub_module>.ex
</code></pre></div></div>
<p>These things might certainly change, but having looked through some other codebases and reflecting on what I like and don’t like, I think that this will be a good start.</p>Brian UnderwoodI’ve mix phx.new ed many applications and when doing so I often start with wondering how to organize my code. I love how Phoenix pushes you to think about the different domains in your app via generators while at the same time I have the freedom to organize modules on my own. Ecto schemas make for a nice abstraction, but where should I put code related just to that table? It could be in the context, but I don’t want the context to become a “grab bag” of unorganized function calls.Expecting the Unexpected in Elixir2020-01-31T00:00:00+01:002020-01-31T00:00:00+01:00http://www.brian-underwood.codes//elixir/2020/01/31/expecting-the-unexpected-in-elixir<p><strong>What can a 50 year old cryptic error message teach us about the software we write today?</strong></p>
<p><img style="width: 800px; margin: 0 auto;" src="/assets/expecting_the_unexpected/rocket.jpg" /></p>
<p style="text-align: center;">
<em>Apollo 11's Saturn V rocket on the launchpad at the Kennedy Space Center. 1 July 1969. Photo: <a href="https://commons.wikimedia.org/wiki/File:Apollo_11_Saturn_V_on_the_pad_at_the_Kennedy_Space_Center.jpg">NASA</a></em>
</p>
<p><strong>Cross-posted from <a href="https://medium.com/fishbrain/expecting-the-unexpected-in-elixir-a24deb06b5a6">Medium</a></strong></p>
<p>On July 24, 1969 Neil Armstrong and Buzz Aldrin were attempting to land on the moon for the first time in human history. As you might expect it was a tense situation. Of course it was just then that they got what every computer user hates to get… a cryptic error code:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Neil Armstrong: PROGRAM ALARM.
Mission Control: It’s looking good to us. Over.
Neil Armstrong: It’s a 1202.
Buzz Aldrin: 1202.
(14 seconds later…)
Neil Armstrong: Give us a reading on the 1202 PROGRAM ALARM.
Mission Control: Roger. We got — We’re GO on that alarm
</code></pre></div></div>
<p>Source: <a href="https://www.hq.nasa.gov/alsj/a11/a11transcript_tec.html">Apollo 11 — Technical Air-To-Ground Voice Transcription</a></p>
<p>Unhelpfully, the error (along with a related 1201 error) continued to pop up in the minutes before landing as Armstrong focused on piloting. As they finally touched down the error was forgotten and they prepared to take their giant leap for mankind. But what was the mysterious error?</p>
<h2 id="expecting-the-unexpected">Expecting the Unexpected</h2>
<p>Some months back I wanted to give a brief presentation at our local Stockholm Elixir meetup group. As Scenic (a application user interface library) had been recently released it was a perfect opportunity to learn something new while hanging out with other Alchemists. When thinking of what to do with Scenic, I decided that I wanted to integrate with one of my favorite games: Kerbal Space Program. After all, why have only one challenge when you can have two instead?</p>
<p>Kerbal Space Program is a game that gives you a small space center and the opportunity to try rocket science for yourself (be ready for lots of explosions). From the community of its fans we also get the kRPC plugin which allows other programs to interface with the game’s internal APIs using Protocol Buffers. This was just what I needed to implement a Kerbal Space Program dashboard demo.</p>
<p><img style="width: 800px; margin: 0 auto;" src="/assets/expecting_the_unexpected/dashboard.png" /></p>
<p style="text-align: center;">
<em>The Demo Dashboard Interface</em>
</p>
<p>As part of the Apollo program, the MIT Instrumentation Laboratory was tasked with designing the Apollo Guidance Computer. For that the team exhaustively tested the computer and the mission programs to make sure there would always be capacity. Fortunately the computer was also designed so that if it did exceed capacity it would restart and pick back up where it left off. The 1202 / 1201 errors were indications of this happening.</p>
<p>While you may never build something as important as the Apollo Guidance Computer, users of your software would prefer that it Just Worked. Fortunately with Elixir we are given the tools which allow us to give our users just as seamless of an experience.</p>
<p>In creating my demo I realized that Scenic and Kerbal Space Program would make a great demonstration on how to design for failure in (mock) life-critical situations. Since Scenic uses supervisors for its components it’s designed to fail and recover seamlessly. Perfect to visually demonstrate designing for failure.</p>
<h2 id="integrating-with-scenic-and-krpc">Integrating with Scenic and kRPC</h2>
<p>When I researched Elixir solutions to work with kRPC I found the <a href="https://github.com/FiniteMonkeys/jooce">jooce</a> project. This project had some good example code which I looked through, though when I tried running the code it didn’t work. After quite a lot of struggling I found that it seemed to have been designed for an older version of the kRPC plugin. Because of this I ended up building my own (very simple) interface using <a href="https://github.com/bitwalker/exprotobuf">exprotobuf</a>. This worked, more or less, though I needed to use a specific branch of the Erlang gpb (Google Protobuf) library from CraigCottingham’s fork which fixed a bug I was experiencing.</p>
<p>All of this worked fine and you can see the code <a href="https://gitlab.com/cheerfulstoic/ksp_scenic_dashboard/blob/master/lib/krpc.ex">here</a>, but more recently I found the <a href="https://github.com/wisq/space_ex">space_ex</a> Elixir library. This probably would have been an ideal solution if I had found it when I was putting together my demo project.</p>
<p>Once I was able to integrate kRPC, I needed to be able to display the dashboard. In Scenic you create a hierarchy of components representing more and more focused parts of your UI. The components are Elixir processes and can receive messages which may change the state of the components and cause the display to be refreshed. I was able to use the <a href="https://github.com/boydm/scenic_sensor">scenic_sensor</a> library to create another supervised hierarchy of to hold my “sensors”. These sensors would simply send themselves messages to loop every 200 milliseconds at which point they could query the game and then send the latest readings to the Scenic components. The supervision tree looked like this:</p>
<p><img style="width: 800px; margin: 0 auto;" src="/assets/expecting_the_unexpected/supervision_tree.png" /></p>
<p>When I was first testing this out, my components would flash back and forth. After debugging for a while it I found that while each component would set its own value just fine, the Scenic graph that was being used was always the initial one. Therefore each change reflected only the most recent sensor. Fortunately the fix in my project was easy and a simple PR to the <a href="https://github.com/boydm/scenic_new">scenic_new</a> project would help prevent others from running into it.</p>
<p>With the UI updating smoothly, I wanted to be able to simulate failure in the system. I decide to start with each component having a 1% chance of failing on every sensor check. With a check every 200 milliseconds and two sensors refreshing, that means that there should be a failure approximately every 10 seconds! That worked well, though when I would try to scale it up I would get a mess of Elixir backtraces that I had a hard time figuring out. After more struggling I realized that I was hitting the retry limits of the supervisors and they were simply giving up. After a simple update of <code class="language-plaintext highlighter-rouge">max_restarts</code>, I could monitor my sensors without even noticing flickering as things constantly crashed. I was even able to demo it at the meetup!</p>
<h2 id="bonus-notes">Bonus Notes</h2>
<ul>
<li>I found this <a href="https://www.youtube.com/watch?v=Qj2IETkScWA">video</a> of somebody simulating the Apollo Guidance Computer to use in my presentation. In includes audio from the astronauts asking about the 1202 alarm</li>
<li>For those interested in more information, Vintage Space has a good <a href="https://www.youtube.com/watch?v=kGD0zEbiDPQ">video</a> and <a href="https://www.discovermagazine.com/the-sciences/apollo-11s-1202-alarm-explained#.W__d35NKh24">writeup</a> about the 1201 / 1202 alarms.</li>
<li>Aviation Week had an <a href="https://web.archive.org/web/20180505110204/http://aviationweek.com/blog/dragons-radiation-tolerant-design">interview</a> (Wayback Machine link) with John Muratore at SpaceX discussing how they make radiation-tolerant hardware (which double checks it’s work and restarts as needed)</li>
</ul>Brian UnderwoodWhat can a 50 year old cryptic error message teach us about the software we write today?Analyzing Ruby Code with Neo4j2016-01-11T18:09:00+01:002016-01-11T18:09:00+01:00http://www.brian-underwood.codes//ruby/rails/neo4j/2016/01/11/analyzing_ruby_code_with_Neo4j<p>For a long time I’ve been wanting to use Neo4j as a tool to analyze Ruby code. Using Ruby for almost a decade, I have a lot of experience with effectively finding my way around Ruby code. Still Ruby programs can by very dynamic and thus tools are always welcome. As I’ve begun to use Neo4j I’ve been inspired by code analysis tools such as <a href="http://mlsec.org/joern/">joern</a> and <a href="http://jqassistant.org/">jQAssistant</a> as well as Aaron Patterson’s <a href="http://tenderlove.github.io/heap-analyzer/">heap-analyzer</a>. In that spirit I’d like to announce a new project called <a href="https://github.com/neo4jrb/neolytics">Neolytics</a>.</p>
<p>Neolytics is a Ruby gem which records the execution of a block of Ruby code in a Neo4j database. There are three major things which it records:</p>
<h3 id="each-step-of-the-execution">Each step of the execution</h3>
<p>This is thanks to Ruby’s excellent <a href="http://ruby-doc.org/core-2.0.0/TracePoint.html">TracePoint</a> class. TracePoint gives a hook into each execution step of Ruby code so that we can record line executions, method calls/returns, and raising of executions.</p>
<p><img src="https://raw.githubusercontent.com/neo4jrb/neolytics/master/examples/model/trace_point_flow.png" alt="TracePoint flow model" /></p>
<h3 id="objects-which-are-found-along-the-way">Objects which are found along the way</h3>
<p>For each trace point Neolytics finds:</p>
<ul>
<li>the object which is the context of the TracePoint</li>
<li>objects which are passed as arguments (if a call)</li>
<li>objects which are returned (if a return)</li>
<li>objects which are referenced via variables</li>
</ul>
<p>For each object the class and module ancestry information is also recursively imported.</p>
<p><img src="https://raw.githubusercontent.com/neo4jrb/neolytics/master/examples/model/trace_point_objects.png" alt="TracePoint objects" style="width: 49%; float: left" /></p>
<p><img src="https://raw.githubusercontent.com/neo4jrb/neolytics/master/examples/model/object_relationships.png" alt="Object space model" style="width: 49%" /></p>
<h3 id="abstract-syntax-trees-of-all-ruby-code-files-involved-in-the-execution">Abstract syntax trees of all Ruby code files involved in the execution</h3>
<p>With thanks to the <a href="https://github.com/whitequark/parser">parser</a> gem! Additionally method definitions in the AST are linked to their corresponding TracePoint.</p>
<p><img src="https://raw.githubusercontent.com/neo4jrb/neolytics/master/examples/model/ast.png" alt="Abstract Syntax Tree model" style="width: 49%" />
<img src="https://raw.githubusercontent.com/neo4jrb/neolytics/master/examples/model/trace_point_ast_nodes.png" alt="TracePoint AST Link" style="width: 49%; vertical-align: top" /></p>
<p>To record the data, I used my <a href="https://github.com/neo4jrb/neo4apis">neo4apis</a> gem to send data to Neo4j efficiently without a lot of round trips to the database.</p>
<p>Lastly, I’ve built a <a href="https://github.com/neo4j-examples/ruby_code_analytics">ruby_code_analytics</a> Rails application to give a UI to examine the results of the dump. You can see an example running <a href="https://ruby-neo4j-code-analysis.herokuapp.com">on Heroku</a> which is a record of the following simple Ruby code:</p>
<pre><code class="language-ruby">
neo4j_session = Neo4j::Session.open(:server_db, neo4j_url)
Neolytics.record_execution(neo4j_session) do
doc = Nokogiri::HTML(open('https://www.google.com').read)
doc.xpath('//form/input').map(&:name)
end
</code></pre>
<h2 id="querying-the-data">Querying the data</h2>
<p>Let’s look at some examples of things that you can do with the data from neolytics!</p>
<h3 id="examining-the-execution">Examining the execution</h3>
<p>Let’s say that we want to know what goes on during the execution of a method. With the following query we grab the first TracePoint which was a <code class="language-plaintext highlighter-rouge">call</code> event and find the entire series of TracePoints until the method is returned from:</p>
<pre><code class="language-cypher">
MATCH (call_tp:TracePoint {event: 'call'})
WITH call_tp LIMIT 1
MATCH path=shortestPath((call_tp)-[:NEXT*]->(return_tp:TracePoint {event: 'return'}))
RETURN path
</code></pre>
<p>Visually we can see the query like this:</p>
<div style="text-align: center">
<a href="/assets/neolytics/method_call_chain.png"><img src="/assets/neolytics/method_call_chain.png" style="width: 800px" /></a>
</div>
<p>In this case the method is the <code class="language-plaintext highlighter-rouge">#plural</code> method from the <code class="language-plaintext highlighter-rouge">active_support</code> gem which lets you pluralize strings. You can compare the graph database model with the actual source code <a href="https://github.com/rails/rails/blob/v4.2.5/activesupport/lib/active_support/inflector/inflections.rb#L105">here on GitHub</a></p>
<p>In the screenshot above, each node is displaying a TracePoint event (i.e. <code class="language-plaintext highlighter-rouge">call</code>, <code class="language-plaintext highlighter-rouge">line</code>, etc…), the class and method in question, and the line number. Starting from the node on the left you can follow the <code class="language-plaintext highlighter-rouge">NEXT</code> relationships to see each point in the execution. Note that every node has a <code class="language-plaintext highlighter-rouge">HAS_PARENT</code> relationship back to the <code class="language-plaintext highlighter-rouge">call</code> TracePoint, so we can see that this method didn’t make any further method calls (at least no Ruby calls. We see some <code class="language-plaintext highlighter-rouge">c_call</code> events which TracePoint doesn’t expose). Lastly note the <code class="language-plaintext highlighter-rouge">STARTED_AT</code> relationship from the <code class="language-plaintext highlighter-rouge">return</code> event to the <code class="language-plaintext highlighter-rouge">call</code> event which is there as a convenience for any query that need to get both nodes at once (e.g. if you need to get both the arguments and the return value of a method).</p>
<p>For more details you can see <a href="https://github.com/neo4jrb/neolytics/blob/master/examples/output/plural_trace_points.csv">this example tabular output</a> of the tracepoints.</p>
<h3 id="running-code-metrics">Running code metrics</h3>
<p>Let’s run a query to evaluate the <a href="https://en.wikipedia.org/wiki/Cyclomatic_complexity">cyclomatic complexity</a> of our methods:</p>
<pre><code class="language-cypher">
MATCH (def:ASTNode {type: 'def'})
OPTIONAL MATCH (def)<-[:HAS_PARENT*]-(condition:ASTNode)
WHERE condition.type IN ['begin', 'if', 'while', 'until', 'for', 'rescue', 'when', 'and', 'or']
RETURN def.name, def.file_path, def.first_line, count(condition)
ORDER BY count(condition) DESC
LIMIT 10
</code></pre>
<p>This gives us a straightforward listing of methods, where to find them, and their complexity score, all ordered with the most complex methods at the top:</p>
<table>
<thead>
<tr>
<th>method</th>
<th>Path and line</th>
<th>complexity</th>
</tr>
</thead>
<tbody>
<tr>
<td>initialize_pattern</td>
<td>…/rubies/ruby-2.2.3/lib/ruby/2.2.0/uri/rfc2396_parser.rb:342</td>
<td>85</td>
</tr>
<tr>
<td>validate</td>
<td>…/rubies/ruby-2.2.3/lib/ruby/2.2.0/rubygems/specification.rb:2453</td>
<td>65</td>
</tr>
<tr>
<td>perform_arguments</td>
<td>…/gems/ruby-2.2.3/gems/sass-3.4.20/lib/sass/tree/visitors/perform.rb:14</td>
<td>57</td>
</tr>
<tr>
<td>_next_token</td>
<td>…/gems/ruby-2.2.3/gems/nokogiri-1.6.7/lib/nokogiri/css/tokenizer.rb:55</td>
<td>56</td>
</tr>
<tr>
<td>parse</td>
<td>…/gems/ruby-2.2.3/gems/tzinfo-1.2.2/lib/tzinfo/zoneinfo_timezone_info.rb:95</td>
<td>55</td>
</tr>
<tr>
<td>pretty_print</td>
<td>…/rubies/ruby-2.2.3/lib/ruby/2.2.0/pp.rb:421</td>
<td>53</td>
</tr>
<tr>
<td>merge_final_ops</td>
<td>…/gems/ruby-2.2.3/gems/sass-3.4.20/lib/sass/selector/sequence.rb:342</td>
<td>52</td>
</tr>
<tr>
<td>call</td>
<td>…/gems/ruby-2.2.3/gems/sprockets-3.5.2/lib/sprockets/server.rb:22</td>
<td>48</td>
</tr>
<tr>
<td>convert_input</td>
<td>…/gems/ruby-2.2.3/gems/erubis-2.7.0/lib/erubis/converter.rb:127</td>
<td>43</td>
</tr>
<tr>
<td>visit_rule</td>
<td>…/gems/ruby-2.2.3/gems/sass-3.4.20/lib/sass/tree/visitors/to_css.rb:280</td>
<td>43</td>
</tr>
</tbody>
</table>
<p>Now let’s extend this complexity metric by using the TracePoint data to see how long each method actually took:</p>
<pre><code class="language-cypher">
MATCH (tp:TracePoint)
WITH sum(tp.execution_time) AS total_execution_time
MATCH (node:ASTNode {type: 'def'})
OPTIONAL MATCH (node)<-[:HAS_PARENT*]-(condition:ASTNode)
WHERE condition.type IN ['begin', 'if', 'while', 'until', 'for', 'rescue', 'when', 'and', 'or']
WITH node, count(condition) AS complexity, total_execution_time
MATCH (node)<-[:HAS_AST_NODE]-(tp:TracePoint)<-[:STARTED_AT]-(return_tp:TracePoint)
WITH
complexity,
total_execution_time,
tp.path + ':' + tp.lineno + ' (' + return_tp.defined_class + '#' + return_tp.method_id + ')' AS method,
count(tp) AS executions,
sum(return_tp.execution_time) AS total_method_execution_time
RETURN
method,
complexity AS cc,
executions AS execs,
total_method_execution_time AS total_time,
100.0 * (total_method_execution_time / total_execution_time) AS percentage_of_total_time,
total_method_execution_time / executions AS avg_exec
ORDER BY total_method_execution_time DESC
LIMIT 10
</code></pre>
<p>With this we get a nice table of methods which take the most time, along with the cyclomatic complexity. We can sort by either metric or create a combined metric of our own in order to determine which methods might be ripe for refactoring.</p>
<table>
<thead>
<tr>
<th>method</th>
<th>cc</th>
<th>execs</th>
<th>total_time</th>
<th>% total time</th>
<th>avg_exec</th>
</tr>
</thead>
<tbody>
<tr>
<td>…/rubies/ruby-2.2.3/lib/ruby/2.2.0/set.rb:289 <br /><strong>Set#add</strong></td>
<td>1</td>
<td>2343</td>
<td>11457</td>
<td>2.31</td>
<td>4.89</td>
</tr>
<tr>
<td>…/gems/ruby-2.2.3/gems/activesupport-4.2.5/lib/active_support/inflector/methods.rb:91 <br /><strong>ActiveSupport::Inflector#underscore</strong></td>
<td>6</td>
<td>237</td>
<td>5057</td>
<td>1.02</td>
<td>21.34</td>
</tr>
<tr>
<td>…/gems/ruby-2.2.3/gems/pry-0.10.3/lib/pry/command.rb:27 <br /><strong>#<Class:Pry::Command>#match</strong></td>
<td>4</td>
<td>810</td>
<td>4917</td>
<td>0.99</td>
<td>6.07</td>
</tr>
<tr>
<td>…/gems/ruby-2.2.3/gems/pry-0.10.3/lib/pry/command.rb:43 <br /><strong>#<Class:Pry::Command>#command_options</strong></td>
<td>2</td>
<td>641</td>
<td>3790</td>
<td>0.76</td>
<td>5.91</td>
</tr>
<tr>
<td>…/gems/ruby-2.2.3/gems/pry-0.10.3/lib/pry/command.rb:177 <br /><strong>#<Class:Pry::Command>#command_regex</strong></td>
<td>6</td>
<td>510</td>
<td>3722</td>
<td>0.75</td>
<td>7.30</td>
</tr>
<tr>
<td>…/gems/ruby-2.2.3/gems/pry-0.10.3/lib/pry/command.rb:185 <br /><strong>#<Class:Pry::Command>#convert_to_regex</strong></td>
<td>1</td>
<td>1020</td>
<td>3537</td>
<td>0.71</td>
<td>3.47</td>
</tr>
<tr>
<td>…/gems/ruby-2.2.3/gems/activesupport-4.2.5/lib/active_support/dependencies/autoload.rb:35 <br /><strong>ActiveSupport::Autoload#autoload</strong></td>
<td>4</td>
<td>233</td>
<td>3186</td>
<td>0.64</td>
<td>13.67</td>
</tr>
<tr>
<td>…/gems/ruby-2.2.3/gems/sprockets-3.5.2/lib/sprockets/utils.rb:33 <br /><strong>Sprockets::Utils#hash_reassoc1</strong></td>
<td>4</td>
<td>115</td>
<td>2366</td>
<td>0.48</td>
<td>20.57</td>
</tr>
<tr>
<td>…/gems/ruby-2.2.3/gems/activesupport-4.2.5/lib/active_support/core_ext/module/delegation.rb:151 <br /><strong>Module#delegate</strong></td>
<td>36</td>
<td>43</td>
<td>1164</td>
<td>0.23</td>
<td>27.07</td>
</tr>
<tr>
<td>…/gems/ruby-2.2.3/gems/pry-0.10.3/lib/pry/command.rb:37 <br /><strong>#<Class:Pry::Command>#description</strong></td>
<td>2</td>
<td>152</td>
<td>1158</td>
<td>0.23</td>
<td>7.62</td>
</tr>
</tbody>
</table>
<h2 id="more-options">More options</h2>
<p>There are many ways to use this data model to debug and otherwise analyze our code. I’ve provided some more sample queries in <a href="https://github.com/neo4jrb/neolytics">the README</a> for the neolytics gem. I’ve also started <a href="https://github.com/neo4j-examples/ruby_code_analytics/labels/idea%20for%20query">a list of ideas</a> for new queries which could be made. These could be used directly or perhaps built into a reporting feature of the <code class="language-plaintext highlighter-rouge">ruby_code_analytics</code> Rails application. I’m excited to see where things could go!</p>
<p>Analysis app running on Heroku: <a href="https://ruby-neo4j-code-analysis.herokuapp.com">https://ruby-neo4j-code-analysis.herokuapp.com</a></p>
<style>
table {
border-collapse: collapse;
}
table, th, td {
border: 1px solid #CCC;
padding: 0.3em;
}
</style>Brian UnderwoodFor a long time I’ve been wanting to use Neo4j as a tool to analyze Ruby code. Using Ruby for almost a decade, I have a lot of experience with effectively finding my way around Ruby code. Still Ruby programs can by very dynamic and thus tools are always welcome. As I’ve begun to use Neo4j I’ve been inspired by code analysis tools such as joern and jQAssistant as well as Aaron Patterson’s heap-analyzer. In that spirit I’d like to announce a new project called Neolytics.GraphStarter: Getting a Neo4j Rails app up and running quickly2015-10-27T10:55:00+01:002015-10-27T10:55:00+01:00http://www.brian-underwood.codes//2015/10/27/graph_starter<p>For a while now I’ve been building various <a href="http://neo4jrb.io/">Neo4j.rb</a> educational resources using the example of an <a href="http://github.com/neo4j-examples/asset_portal">asset portal</a>. There has been:</p>
<ul>
<li>a screencast series (the <a href="http://neo4j.com/blog/create-a-ruby-on-rails-app-with-neo4j-screencast-series/">first half</a> and the <a href="http://neo4j.com/blog/advanced-ruby-on-rails-with-neo4j/">second half</a>)</li>
<li>a <a href="http://www.sitepoint.com/why-you-should-use-neo4j-in-your-next-ruby-app/">SitePoint article</a> on building recommendations and access control</li>
<li>a <a href="https://www.youtube.com/watch?v=dlRL-3XZvHs">webinar</a> on advanced access control</li>
<li>a <a href="http://neo4j.com/developer/ruby-course/">introduction course</a> for Neo4j using Ruby</li>
</ul>
<p>As part of this process I’ve wanted to use what I’ve been building and allow anybody to easily create a UI for their own assets in Rails. I’m pleased to say that I’ve got a good start with the <a href="https://github.com/neo4j-examples/graph_starter">graph_starter</a> gem.</p>
<p>The <code class="language-plaintext highlighter-rouge">graph_starter</code> gem is a Rails engine, which means that it can be placed within a Rails application. The goal is to be able to quickly set up a basic UI for your entities, but to also be able to override it when you want to provide custom logic.</p>
<p>Setting up a <code class="language-plaintext highlighter-rouge">graph_starter</code> application is as simple as the following steps:</p>
<h2 id="installation">Installation</h2>
<p>Using <code class="language-plaintext highlighter-rouge">graph_starter</code> is easy!</p>
<p>First create a Rails application if you don’t already have one:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>rails new application_name
</code></pre></div></div>
<p>Include the <code class="language-plaintext highlighter-rouge">graph_starter</code> gem (it will include the Neo4j.rb gems for you):</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code># Gemfile
gem 'graph_starter'
</code></pre></div></div>
<p>Mount the engine:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code># config/routes.rb
mount GraphStarter::Engine => '/'
</code></pre></div></div>
<p>Create some asset models:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code># app/models/product.rb
class Product < GraphStarter::Asset
# `title` property is added automatically
property :name
property :description
property :price, type: Integer
has_images
has_one :in, :vendor, type: :SELLS_PRODUCT
end
# app/models/vendor.rb
class Vendor < GraphStarter::Asset
property :brand_name
property :code
name_property :brand_name
has_many :out, :products, origin: :vendor
end
</code></pre></div></div>
<p>These models are simply Neo4j.rb <code class="language-plaintext highlighter-rouge">ActiveNode</code> models so you can refer to the <a href="http://neo4jrb.readthedocs.org/">Neo4j.rb documentation</a> to define them. They do have some special methods, however, which let you control how GraphStarter works. In the above <code class="language-plaintext highlighter-rouge">Product</code> model, for example, <code class="language-plaintext highlighter-rouge">has_images</code> has been called to indicate that products have images which defines a separate <code class="language-plaintext highlighter-rouge">Image</code> model along with the neccessary association. See the <a href="https://github.com/neo4j-examples/graph_starter#models">graph_starter README</a> for documentation on how to configure aspects of your models.</p>
<p>Once that framework is in place you can define a way to import data, if desired. For this I would suggest making a rake task:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code># lib/tasks/store.rake
namespace :store do
task :import do
CSV.open(File.read('vendors.csv')).each do |row|
Vendor.create(name: row['brand_name'],
code: row['code'])
end
CSV.open(File.read('products.csv')).each do |row|
product = Product.create(name: row['name'],
description: row['description'],
price: row['price'].to_i)
product.vendor = Vendor.find_by(code: row['vendor_code'])
end
end
end
</code></pre></div></div>
<p>And that’s all!</p>
<p>When everything is in place you can simply start up your Rails server (by running <code class="language-plaintext highlighter-rouge">rails server</code>) and you get a UI which looks like this example site I made using data from the <a href="http://www.nhm.ac.uk/">Natural History Museum</a> in London:</p>
<p><img style="width: 100%; display: block; margin: 0 auto" src="/assets/graph_starter/assets_index.png" /></p>
<p><img style="width: 100%; display: block; margin: 0 auto" src="/assets/graph_starter/assets_show.png" /></p>
<p>You can <a href="http://nhm-portal.herokuapp.com/">browse the app</a> on Heroku and <a href="https://github.com/neo4j-examples/nhm_asset_portal">checkout the repository</a> on Github</p>
<p>I’ll be working on a new project to create a <a href="http://graphgist.neo4j.com/">GraphGist portal</a> based on the <code class="language-plaintext highlighter-rouge">graph_starter</code> gem so I plan to continue improving it!</p>Brian UnderwoodFor a while now I’ve been building various Neo4j.rb educational resources using the example of an asset portal. There has been:Using Graph Structure Record Linkage on Irish Census Data with Neo4j2015-08-20T09:55:00+02:002015-08-20T09:55:00+02:00http://www.brian-underwood.codes//2015/08/20/using_graph_structure_record_linkage_on_irish_census_data_with_neo4j<p>For just over a year I’ve been obsessed on-and-off with a project ever since I stayed in the town of Skibbereen, Ireland. Taking data from the <a href="http://www.census.nationalarchives.ie/">1901 and 1911 Irish censuses</a> I hoped I would be able to find a way to reliably link resident records from the two together to identify the same residents. Since then I’ve learned a bit about <a href="/tag/master-data-management">master data management and record linkage</a> and so I thought I would give it another stab. Here I’d like to talk about how I’ve been matching records based on the local data space around objects to improve my record linkage scoring.</p>
<p>The data model of the imported data is very linear:</p>
<p><img style="width: 100%; display: block; margin: 0 auto" src="/assets/neo4j-mdm/irish_census_wrah.png" /></p>
<p>In this post, however, I’m going to be focusing on Houses and Residents and creating relationships between them based on their properties.</p>
<h2 id="relations-to-the-head">Relations to the Head</h2>
<p>To view an example of what a census record from 1911 Ireland looks like you can have a look at the McCarthys of <a href="http://www.census.nationalarchives.ie/pages/1901/Cork/Cloghdowell/Barnagowlane/1154382/">1901</a> and <a href="http://www.census.nationalarchives.ie/pages/1911/Cork/Cloghdonnell/Barnagowlane/440559/">1911</a>. Charles is the head of the family with his wife Hannah, mother Ellen, children (two in 1901 and seven in 1911), and a servant (Timothy Walsh in 1901 and William Regan in 1911).</p>
<style>
table {
border-collapse: collapse;
margin: 0 auto !important;
}
table, th, td {
border: 1px solid #CCC;
padding: 0.3em;
}
</style>
<table>
<tr>
<td style="font-weight: 0.8em; font-weight: bold; text-align: center">
<a href="http://www.census.nationalarchives.ie/pages/1901/Cork/Cloghdowell/Barnagowlane/1154382/">
<img style="width: 350px; display: block; margin: 0 auto" src="/assets/neo4j-mdm/mccarthy_census_1901.png" />
The McCarthy family of Barnagowlane, Cloghdowell, Cork, 1901
</a>
</td>
<td style="font-weight: 0.8em; font-weight: bold; text-align: center">
<a href="http://www.census.nationalarchives.ie/pages/1911/Cork/Cloghdonnell/Barnagowlane/440559/">
<img style="width: 350px; display: block; margin: 0 auto" src="/assets/neo4j-mdm/mccarthy_census_1911.png" />
The McCarthy family of Barnagowlane, Cloghdonnell, Cork, 1911
</a>
</td>
</tr>
</table>
<table>
<thead>
<tr>
<th>Surname</th>
<th>Forename</th>
<th>Age</th>
<th>Sex</th>
<th>Relation to Head</th>
<th>Religion</th>
<th> </th>
<th>Surname</th>
<th>Forename</th>
<th>Age</th>
<th>Sex</th>
<th>Relation to Head</th>
<th>Religion</th>
</tr>
</thead>
<tbody>
<tr>
<td>McCarthy</td>
<td>Charles</td>
<td>37</td>
<td>Male</td>
<td>Head of Family</td>
<td>Roman Catholic</td>
<td> </td>
<td>McCarthy</td>
<td>Charles</td>
<td>47</td>
<td>Male</td>
<td>Head of Family</td>
<td>Roman Catholic</td>
</tr>
<tr>
<td>McCarthy</td>
<td>Hannah</td>
<td>25</td>
<td>Female</td>
<td>Wife</td>
<td>Roman Catholic</td>
<td> </td>
<td>McCarthy</td>
<td>Hannah</td>
<td>35</td>
<td>Female</td>
<td>Wife</td>
<td>Roman Catholic</td>
</tr>
<tr>
<td>McCarthy</td>
<td>William</td>
<td>1</td>
<td>Male</td>
<td>Son</td>
<td>Roman Catholic</td>
<td> </td>
<td>McCarthy</td>
<td>William</td>
<td>11</td>
<td>Male</td>
<td>Son</td>
<td>Roman Catholic</td>
</tr>
<tr>
<td>McCarthy</td>
<td>Bridget</td>
<td> </td>
<td>Female</td>
<td>Daughter</td>
<td>Roman Catholic</td>
<td> </td>
<td>McCarthy</td>
<td>Bridget</td>
<td>10</td>
<td>Female</td>
<td>Daughter</td>
<td>Roman Catholic</td>
</tr>
<tr>
<td> </td>
<td> </td>
<td> </td>
<td> </td>
<td> </td>
<td> </td>
<td> </td>
<td>McCarthy</td>
<td>Ellen</td>
<td>8</td>
<td>Female</td>
<td>Daughter</td>
<td>Roman Catholic</td>
</tr>
<tr>
<td> </td>
<td> </td>
<td> </td>
<td> </td>
<td> </td>
<td> </td>
<td> </td>
<td>McCarthy</td>
<td>Kate</td>
<td>6</td>
<td>Female</td>
<td>Daughter</td>
<td>Roman Catholic</td>
</tr>
<tr>
<td> </td>
<td> </td>
<td> </td>
<td> </td>
<td> </td>
<td> </td>
<td> </td>
<td>McCarthy</td>
<td>Florence</td>
<td>4</td>
<td>Male</td>
<td>Son</td>
<td>Roman Catholic</td>
</tr>
<tr>
<td> </td>
<td> </td>
<td> </td>
<td> </td>
<td> </td>
<td> </td>
<td> </td>
<td>McCarthy</td>
<td>Charles Peter</td>
<td>2</td>
<td>Male</td>
<td>Son</td>
<td>Roman Catholic</td>
</tr>
<tr>
<td> </td>
<td> </td>
<td> </td>
<td> </td>
<td> </td>
<td> </td>
<td> </td>
<td>McCarthy</td>
<td>Annie</td>
<td> </td>
<td>Female</td>
<td>Daughter</td>
<td>Roman Catholic</td>
</tr>
<tr>
<td>McCarthy</td>
<td>Ellen</td>
<td>65</td>
<td>Female</td>
<td>Mother</td>
<td>Roman Catholic</td>
<td> </td>
<td>McCarthy</td>
<td>? Ellen</td>
<td>75</td>
<td>Female</td>
<td>Mother</td>
<td>Roman Catholic</td>
</tr>
<tr>
<td>Walsh</td>
<td>Timothy</td>
<td>25</td>
<td>Male</td>
<td>Servant</td>
<td>Roman Catholic</td>
<td> </td>
<td> </td>
<td> </td>
<td> </td>
<td> </td>
<td> </td>
<td> </td>
</tr>
<tr>
<td> </td>
<td> </td>
<td> </td>
<td> </td>
<td> </td>
<td> </td>
<td> </td>
<td>Regan</td>
<td>William</td>
<td>24</td>
<td>Male</td>
<td>Servant</td>
<td>Roman Catholic</td>
</tr>
</tbody>
</table>
<p>The McCarthys are an almost exact match between two census records between 1901 and 1911. The names, ages, occupations, and relationships all match perfectly. Unfortunately the story for other records is not so simple. Many times houses, which to the human eye seem to be the same house, can have wildly varying details. For example Hannah might go be listed as Hana or Anne in a different census. Likewise ages vary a lot more than you might think. In examining the records I regularly found ages varying by a year or two and have even found a few houses with ages off by as much as 10-15 years.</p>
<p>In both censuses there is a field for residents to fill out called “Relation to Head”. This gives us information about how each resident is related to the head of the house. In the case of the McCarthys, Charles is listed as “Head of Fa<D-i>mily" in both years. The rest of the family has a nice representation of things that we often see in the data: "Wife", "Son", "Daughter", and "Servant".</D-i></p>
<p>We might be tempted to say “This person was the head in 1901, so they must be the same person who was the head in 1911”. Often, however, the head of the family can die or retire leaving the roll of head of the family to their wife or child. Can the “Relation to Head” values still be useful to us to match any given resident from 1901 to another resident in 1911?</p>
<p>First let’s cover the general the process of record linkage I have been using. To find a match for a resident I start by using an <code class="language-plaintext highlighter-rouge">elasticsearch</code> server (which contains a duplicate of my Neo4j census data) to quickly find a list of other residents with a match on very rough criteria:</p>
<ul>
<li>Is the resident in the other census?</li>
<li>Does the sex match (or it it <code class="language-plaintext highlighter-rouge">NULL</code>)?</li>
<li>Is the resident’s age within 15 years of what it would be expected to be in the other census?</li>
<li>Does the name match roughly (within an edit distance of 4)</li>
</ul>
<p>This comes back with anywhere from zero to hundreds of results. I call these “similarity candidates” and for each I create a relationship between the original record and the candidate.</p>
<p>With this list I can compare the attributes of the two records (using the <a href="https://rubygems.org/gems/record_linkage">record_linkage</a> gem I created) to see how closely they match. The closer their name, sex, age, etc.. matches, the higher score they get. Ideally the real match should have the highest score, but that isn’t always true and can take <a href="http://127.0.0.1:4000/2015/05/14/master_data_management_scoring_examples/">some tuning</a>.</p>
<p>In addition to this simple comparison of attributes, I have now added a process to take advantage of the similarity candidate relationships to compare family relationships. Let’s start with this example of a sub-graph pattern:</p>
<p><img style="width: 850px; margin: 0.6em;" src="/assets/neo4j-mdm/mccarthy_charles_comparison.png" /></p>
<p>The relationship <code class="language-plaintext highlighter-rouge">CHILD_OF</code> is created whenever there is a “Son” or “Daughter” in the “Realation to Head” field. Likewise we can create other gender-neutral relationships like <code class="language-plaintext highlighter-rouge">MARRIED_TO</code>, <code class="language-plaintext highlighter-rouge">SIBLING_OF</code>, <code class="language-plaintext highlighter-rouge">NIECE_NEPHEW_OF</code>, etc…</p>
<p>In this case the resident in question is the 1901 record for William. When we are evaluating the 1911 record of William as a potential match we can explore other residents in the same house as evidence of similarity. The diagram above shows that both records have a <code class="language-plaintext highlighter-rouge">CHILD_OF</code> relationship to the two “Charles” records which furthermore are linked via a <code class="language-plaintext highlighter-rouge">SIMILARITY_CANDIDATE</code> relationship. Because of this we can say that there is a greater chance that the two “William” records represent the same person.</p>
<p>This only gives us the ability to find these relationships between the head of the family and other residents. What about generically matching based on the relationship of any two residents of a house? Let’s say that Charles died sometime between 1901 and 1911. If his wife Hannah takes over as the head of the family we would have a sub-graph which looks like this:</p>
<p><img style="width: 850px; margin: 0.6em;" src="/assets/neo4j-mdm/mccarthy_hannah_comparison.png" /></p>
<p>We could say that when we have the paths <code class="language-plaintext highlighter-rouge">-CHILD_OF-><-MARIED_TO-</code> and <code class="language-plaintext highlighter-rouge">-CHILD_OF-></code> on either side that we can build our case for a match a bit more. This kind of matching can be used on all of the other residents of the house with <code class="language-plaintext highlighter-rouge">SIMILARITY_CANDIDATE</code> relationships. For example, <code class="language-plaintext highlighter-rouge">-CHILD_OF-><-CHILD_OF-</code> could be matched to <code class="language-plaintext highlighter-rouge">-CHILD_OF-><-CHILD_OF-</code> even in this case where the wife becomes the head of the house. Or if a child becomes the head then it could be compared to a <code class="language-plaintext highlighter-rouge">-SIBLING_OF-</code> relationship.</p>
<h2 id="the-code">The Code</h2>
<p>So how do we actually do this? First let’s take our sub-graph and turn our nodes into variables:</p>
<p><img style="width: 850px; margin: 0.6em;" src="/assets/neo4j-mdm/irish_census_relationship_mapping.png" /></p>
<p>In this example let’s take resident <code class="language-plaintext highlighter-rouge">h1 r1</code> (house 1, resident 1) as the resident in question and <code class="language-plaintext highlighter-rouge">h2 r1</code> as the candidate that we want to compare it to. This is the sort of query that Neo4j is wonderful at both performing quickly and making easy to formulate. Let’s look at part of the Ruby code:</p>
<pre><code class="language-ruby">
def get_similarity_candidate_relationship_paths
self.query_as(:h1_r1)
.match('(h1:House), (h2:House)')
.match('h1<-[:LIVES_IN]-h1_r1-[sc_1:similarity_candidate]-(h2_r1)-[:LIVES_IN]->h2')
.match('h1<-[:LIVES_IN]-h1_r2-[sc_2:similarity_candidate]-(h2_r2)-[:LIVES_IN]->h2')
.match('path1=h1_r1-[:born_to|married_to|grandchild_of|niece_nephew_of|sibling_of|cousin_of|child_in_law_of|step_child_of*1..2]-h1_r2')
.match('path2=h2_r1-[:born_to|married_to|grandchild_of|niece_nephew_of|sibling_of|cousin_of|child_in_law_of|step_child_of*1..2]-h2_r2')
.pluck(
:h2_r1,
'collect([path1, rels(path1), path2, rels(path2)])'
).each_with_object({}) do |(r2, data), result|
result[r2] = data.inject(0) do |total, (path1, rels1, path2, rels2)|
relations1 = relation_string_from_path_and_rels(path1, rels1)
relations2 = relation_string_from_path_and_rels(path2, rels2)
if relations1 == relations2
1.0
elsif score = (RELATION_EQUIVILENCE_SCORES[relations1] || {})[relations2]
score
else
-2.0
end + total
end
end
end
</code></pre>
<p>Here we start with a Cypher query using the <code class="language-plaintext highlighter-rouge">Query</code> API from neo4j.rb. The object upon which we’ve called <code class="language-plaintext highlighter-rouge">get_similarity_candidate_relationship_paths</code> is our <code class="language-plaintext highlighter-rouge">h1_r1</code> anchor. Note here that we match paths with a length of either one or two relationships long from between two residents of the same house. Then we return all residents found via the <code class="language-plaintext highlighter-rouge">SIMILARITY_CANDIDATE</code> relationship from our anchor and the family relationship paths aggregated into an Array.</p>
<p>Once the Cypher query returns data we call <code class="language-plaintext highlighter-rouge">relation_string_from_path_and_rels</code> which is a way of transforming the path into a string like <code class="language-plaintext highlighter-rouge">-BORN_TO-><-BORN_TO</code>. This string gives us a simple way to express the path between the two residents as a string.</p>
<p>We then can give a score based on the two paths. If the paths are the same then we say that the score is 1.0. If the pair of paths is something like <code class="language-plaintext highlighter-rouge">-BORN_TO-><-BORN_TO</code> and <code class="language-plaintext highlighter-rouge">-SIBLING_OF-></code> then we can give a score based on a lookup. We add these scores up to give us a total score comparing our anchor resident and each of it’s similarity candidates. All with just one query to the database.</p>
<h3 id="challenges">Challenges</h3>
<p>There are a couple of things that I needed to do to make this work:</p>
<p>Previously I was simply grabbing one resident at a time, finding all of the similarity candidates, and then creating a set of relationships to link the resident with the candidates and to store the record linkage scores (both the individual scores for fields and the total score). However this approach requires all of the candidates in the house to have <code class="language-plaintext highlighter-rouge">SIMILARITY_CANDIDATE</code> relationships in order to compare family relationships. So I first process all residents for a house to create the similarity candidate relationships and store the record linkage scores and then go through them again with the graph-based comparisons and store that score and update the total.</p>
<p>Beyond that there is the conceptual problem of determining the scoring when comparing paths. For example, if somebody was <code class="language-plaintext highlighter-rouge">BORN_TO</code> the head one year but their spouse takes over as the head, could we say that they’re <code class="language-plaintext highlighter-rouge">BORN_TO</code> the spouse if they are are a step-child? Family relationships are complicated and don’t always fit neatly into our properties and algorithms.</p>
<h3 id="conclusion">Conclusion</h3>
<p>Most record linkage focuses on the properties of an object, but we need to remember that relationships are data about our entities too. With Neo4j we have a powerful tool for analyzing those relationships natuarally and quickly. Additionally I have found that the ability to create relationships on the fly to aggregate calculations like the ones discussed above is a wonderful way to find the best solution quickly.</p>Brian UnderwoodFor just over a year I’ve been obsessed on-and-off with a project ever since I stayed in the town of Skibbereen, Ireland. Taking data from the 1901 and 1911 Irish censuses I hoped I would be able to find a way to reliably link resident records from the two together to identify the same residents. Since then I’ve learned a bit about master data management and record linkage and so I thought I would give it another stab. Here I’d like to talk about how I’ve been matching records based on the local data space around objects to improve my record linkage scoring.Analyzing Ruby’s ObjectSpace with Neo4j2015-06-03T00:47:00+02:002015-06-03T00:47:00+02:00http://www.brian-underwood.codes//2015/06/03/analyzing_rubys_objectspace_with_neo4j<p>Recently the continuous builds for the <a href="https://github.com/neo4jrb/neo4j">neo4j Ruby gem</a> failed for JRuby because the memory limit had been reached. I wanted to see if I could use my favorite tool (Neo4j) to analyize the memory usage. So I threw together <a href="https://github.com/neo4jrb/neo4j_ruby_object_space">a bit of Ruby code</a> to use Ruby’s <code class="language-plaintext highlighter-rouge">ObjectSpace.each_object</code> functionality to dump every object in memory after the test suite of the gem finished (garbage collecting first, of course). First let’s take a look at the model:</p>
<div style="text-align: center">
<img src="/assets/neo4j-ruby-memory/model.png" style="width: 600px" />
</div>
<p>A quick Ruby primer:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">Object</code>s are always instantiations of one and only one <code class="language-plaintext highlighter-rouge">Class</code></li>
<li><code class="language-plaintext highlighter-rouge">Class</code>es can <code class="language-plaintext highlighter-rouge">include</code> zero or more <code class="language-plaintext highlighter-rouge">Module</code>s which extend their APIs</li>
<li>Everything in Ruby is an <code class="language-plaintext highlighter-rouge">Object</code>, including <code class="language-plaintext highlighter-rouge">Class</code>es and <code class="language-plaintext highlighter-rouge">Module</code>s</li>
</ul>
<p>So with that in mind I set out to output some CSVs. This was my first opportunity to output some CSV files for the <a href="http://neo4j.com/docs/stable/import-tool.html"><code class="language-plaintext highlighter-rouge">neo4j-import</code> tool</a> and it was quite a straightforward standard to implement. Here is an example of each of the files that I generated:</p>
<p><code class="language-plaintext highlighter-rouge">objects.csv</code></p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>object_id:ID inspect :LABEL
70099268550680 <RubyVM::InstructionSequence:block in contains_requirable_file?@/Users/brian/.rvm/rubies/ruby-2.2.0/lib/ruby/2.2.0/rubygems/basic_specification.rb> Object
70099273105500 RubyVM::InstructionSequence Object;Class
70099273176540 Class Object;Class
70099270029100 RSpec::Core::SharedExampleGroup::TopLevelDSL Object;Module
70099273176580 Module Object;Class
70099277099900 RSpec::Core::DSL Object;Module
70099275863180 PP::ObjectMixin Object;Module
70099278769700 JSON::Ext::Generator::GeneratorMethods::Object Object;Module
70099273176460 Kernel Object;Module
</code></pre></div></div>
<p><code class="language-plaintext highlighter-rouge">instance_variables.csv</code></p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>:START_ID :END_ID variable
70099268552320 70099268552120 @version
70099268552320 70099277643340 @segments
70099268552320 0 @prerelease
70099268552320 -4656870404935510835 @hash
70099268553460 70099268553260 @version
70099268553460 70099277643720 @segments
70099268553460 0 @prerelease
70099268553460 2810968934318335001 @hash
70099268558740 70099272991680 @_declared_property_manager
</code></pre></div></div>
<p><code class="language-plaintext highlighter-rouge">object_classes.csv</code></p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>:START_ID :END_ID
70099273176540 70099273176540
70099273176580 70099273176540
70099277099900 70099273176580
70099275863180 70099273176580
70099278769700 70099273176580
70099273176460 70099273176580
70099270029100 70099273176580
70099273105500 70099273176540
70099268550680 70099273105500
</code></pre></div></div>
<p><code class="language-plaintext highlighter-rouge">class_superclasses.csv</code></p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>:START_ID :END_ID
70271986243040 70271986243080
70272011737340 70271986239340
70271986207920 70271986243120
70271986179720 70271986243120
70271986207000 70271986243120
70271986517200 70271986243120
70271986241140 70271986243120
70271986225760 70271986225920
70271986206080 70271986243120
</code></pre></div></div>
<p><code class="language-plaintext highlighter-rouge">class_modules.csv</code></p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>:START_ID :END_ID
70099273176580 70099270029100
70099273176580 70099277099900
70099273176580 70099275863180
70099273176580 70099278769700
70099273176580 70099273176460
70099273176540 70099270029100
70099273176540 70099277099900
70099273176540 70099275863180
70099273176540 70099278769700
</code></pre></div></div>
<p>Again, you can see the code for generating these files <a href="https://github.com/neo4jrb/neo4j_ruby_object_space">here</a>.</p>
<p>As you can see I chose to make my files tab-separated. I did this because the <code class="language-plaintext highlighter-rouge">inspect</code> column was going to have lots of double quotes in it and it would create a simpler file. I also used the pipe (<code class="language-plaintext highlighter-rouge">|</code>) as the quoting character because of some conflicts with the way Ruby and Neo4j interpret escaping of quoted characters. Here is the command that I used to import the CSV files into a Neo4j database:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>./db/bin/neo4j-import \
--delimiter TAB \
--quote "|" \
--into ./db/data/graph.db \
--nodes ./objects.csv \
--relationships:INSTANCE_VARIABLE ./instance_variables.csv \
--relationships:HAS_CLASS ./object_classes.csv \
--relationships:INCLUDES_MODULE ./class_modules.csv
</code></pre></div></div>
<p>Great, now we’re cooking with graphs! What does this look like in the Neo4j web console?</p>
<pre><code class="language-cypher">
MATCH (o:Object) WHERE NOT(o:Class) AND NOT(o:Module) WITH o LIMIT 20
MATCH o-[:HAS_CLASS]->(c:Class)
OPTIONAL MATCH c-[:INCLUDES_MODULE]->(m:Module)
RETURN *
</code></pre>
<div style="text-align: center">
<img src="/assets/neo4j-ruby-memory/console.png" style="width: 800px" />
</div>
<p>Nice! As another sanity check, let’s see what our top allocated objects are:</p>
<pre><code class="language-cypher">
MATCH (o:Object)-[:HAS_CLASS]->(c:Class)
WHERE NOT(o:Class) AND NOT(o:Module)
RETURN c.inspect, count(o) ORDER BY count(o) DESC LIMIT 10
</code></pre>
<table>
<thead>
<tr>
<th>c.inspect</th>
<th>count(o)</th>
<th>c.inspect</th>
<th>count(o)</th>
</tr>
</thead>
<tbody>
<tr>
<td>String</td>
<td>171,082</td>
<td>RubyVM::Env</td>
<td>17,731</td>
</tr>
<tr>
<td>Array</td>
<td>53,532</td>
<td>Gem::Requirement</td>
<td>4,227</td>
</tr>
<tr>
<td>Proc</td>
<td>21,894</td>
<td>RSpec::Core::Hooks::HookCollection</td>
<td>3,252</td>
</tr>
<tr>
<td>RubyVM::InstructionSequence</td>
<td>19,416</td>
<td>Gem::Dependency</td>
<td>2,973</td>
</tr>
<tr>
<td>Hash</td>
<td>18,081</td>
<td>Regexp</td>
<td>2,971</td>
</tr>
</tbody>
</table>
<p>That seems right! Any experienced Rubyists will know that Ruby will allocate a lot of strings.</p>
<p>So let’s really get into it and use the power of graphs. Let’s find out which objects have the most references via instance variables. This query takes every object in the database as the root of a tree of instance variable references and calculates the total number of descendent objects in that tree. This should give us an idea for what objects have a lot of other objects depending on them which cannot be garbage collected.</p>
<pre><code class="language-cypher">
// Count of tree
MATCH (o:Object) WHERE NOT(o:Class) AND NOT(o:Module) WITH o
MATCH o-[:HAS_CLASS]->(c:Class)
OPTIONAL MATCH path=o-[:INSTANCE_VARIABLE*]->(other)
RETURN c.inspect, o.inspect, count(other) ORDER BY count(other) DESC LIMIT 10
</code></pre>
<table>
<thead>
<tr>
<th>c.inspect</th>
<th>o.inspect</th>
<th>count(other)</th>
</tr>
</thead>
<tbody>
<tr>
<td>RSpec::Core::ExampleGroup::Nested_31::Nested_5</td>
<td>#<RSpec::Core::ExampleGroup::Nested_31:: Nested_5:0x007f8285c325d0 @__memoized=nil, @prev_wrapped_classes=[Student(name: Object), Teacher(name: Object) …</td>
<td>284</td>
</tr>
<tr>
<td>Neo4j::ActiveNode::Reflection::AssociationReflection</td>
<td>#<Neo4j::ActiveNode::Reflection:: AssociationReflection:0x007f828a2fcfb0 @macro=:has_many, @name=:teachers, @association=#<Neo4j::ActiveNode::HasN::Ass …</td>
<td>193</td>
</tr>
<tr>
<td>Neo4j::ActiveNode::HasN::Association</td>
<td>#<Neo4j::ActiveNode::HasN:: Association:0x007f828a2fd5f0 @type=:has_many, @name=:teachers, @direction=:out, @target_class_name_from_name=”Teacher”, @ta …</td>
<td>190</td>
</tr>
<tr>
<td>RSpec::Core::Runner</td>
<td>#<RSpec::Core::Runner:0x007f82859f89e0 @options=#<RSpec::Core::ConfigurationOptions:0x007f8284241dd8 @args=[], @command_line_options={:files_or_direct …</td>
<td>127</td>
</tr>
<tr>
<td>Bundler::LazySpecification</td>
<td>#<Bundler::LazySpecification:0x007f82853c47b8 @name=”neo4j”, @version=#<Gem::Version “5.0.0.rc.2”>, @dependencies=[<Gem::Dependency type=:runtime name …</td>
<td>110</td>
</tr>
<tr>
<td>Bundler::LazySpecification</td>
<td>#<Bundler::LazySpecification:0x007f82853bdc38 @name=”neo4j-core”, @version=#<Gem::Version “5.0.0.rc.1”>, @dependencies=[<Gem::Dependency type=:runtime …</td>
<td>110</td>
</tr>
<tr>
<td>Bundler::LazySpecification</td>
<td>#<Bundler::LazySpecification:0x007f82853778c8 @name=”parser”, @version=#<Gem::Version “2.2.0.3”>, @dependencies=[<Gem::Dependency type=:runtime name=” …</td>
<td>103</td>
</tr>
<tr>
<td>Bundler::LazySpecification</td>
<td>#<Bundler::LazySpecification:0x007f8285367bd0 @name=”rainbow”, @version=#<Gem::Version “2.0.0”>, @dependencies=[], @platform=”ruby”, @source=#<Bundler …</td>
<td>103</td>
</tr>
<tr>
<td>Bundler::LazySpecification</td>
<td>#<Bundler::LazySpecification:0x007f8285395850 @name=”faraday_middleware-multi_json”, @version=#<Gem::Version “0.0.6”>, @dependencies=[<Gem::Dependency …</td>
<td>103</td>
</tr>
<tr>
<td>Bundler::LazySpecification</td>
<td>#<Bundler::LazySpecification:0x007f8285396d68 @name=”erubis”, @version=#<Gem::Version “2.7.0”>, @dependencies=[], @platform=”ruby”, @source=#<Bundler: …</td>
<td>103</td>
</tr>
</tbody>
</table>
<p>Not bad for a query which runs in 6.5 seconds (with 335,446 total objects on my MacBook Air)!</p>
<p>As part of the import the objects I also create a graph structure to define classes, modules, and their relationships. Let’s see what are the most <code class="language-plaintext highlighter-rouge">include</code>d modules in our object space:</p>
<table>
<thead>
<tr>
<th>m.inspect</th>
<th>count(c)</th>
<th>m.inspect</th>
<th>count(c)</th>
</tr>
</thead>
<tbody>
<tr>
<td>Kernel</td>
<td>136</td>
<td>RSpec::Core::MockFrameworkAdapter</td>
<td>8</td>
</tr>
<tr>
<td>PP::ObjectMixin</td>
<td>136</td>
<td>RSpec::Core::Extensions::InstanceEvalWithArgs</td>
<td>8</td>
</tr>
<tr>
<td>JSON::Ext::Generator::GeneratorMethods::Object</td>
<td>136</td>
<td>RSpec::Core::Pending</td>
<td>8</td>
</tr>
<tr>
<td>Enumerable</td>
<td>21</td>
<td>ActiveAttr::Matchers</td>
<td>8</td>
</tr>
<tr>
<td>Comparable</td>
<td>12</td>
<td>SimpleCov::HashMergeHelper</td>
<td>6</td>
</tr>
<tr>
<td>RSpec::Matchers</td>
<td>9</td>
<td>JSON::Ext::Generator::GeneratorMethods::Hash</td>
<td>6</td>
</tr>
<tr>
<td>RSpec::Core::SharedExampleGroup</td>
<td>8</td>
<td>RSpec::Core::ExampleGroup::Nested_1::NamedSubjectPreventSuper</td>
<td>3</td>
</tr>
<tr>
<td>ActiveNodeRelStubHelpers</td>
<td>8</td>
<td>RSpec::Core::ExampleGroup::Nested_1::LetDefinitions</td>
<td>3</td>
</tr>
<tr>
<td>Neo4jSpecHelpers</td>
<td>8</td>
<td>File::Constants</td>
<td>3</td>
</tr>
<tr>
<td>RSpec::Core::MemoizedHelpers</td>
<td>8</td>
<td>SimpleCov::ArrayMergeHelper</td>
<td>3</td>
</tr>
</tbody>
</table>
<p>Looking at this I learned a couple of things I didn’t know about Ruby:</p>
<ul>
<li>The <code class="language-plaintext highlighter-rouge">Kernel</code> module is included everywhere (which makes sense)</li>
<li>When loading the pretty-print and JSON libraries a corresponding module is included everywhere (which is probably why they aren’t loaded by default)</li>
</ul>
<p>While we’re here let’s grab a random sample of classes and modules (excluding some ones that I hand-picked which had too many connections to make for a useful graph visualization):</p>
<pre><code class="language-cypher">
MATCH (o:Object)
WHERE (o:Class OR o:Module) AND
NOT(o.inspect IN ['Class', 'Module', 'Kernel', 'PP::ObjectMixin']) AND
NOT o.inspect =~ "JSON::Ext.*"
RETURN o LIMIT 80
</code></pre>
<p>Here’s the graph (click to zoom):</p>
<div style="text-align: center">
<a href="/assets/neo4j-ruby-memory/class_module_graph.png"><img src="/assets/neo4j-ruby-memory/class_module_graph.png" style="width: 800px" /></a>
</div>
<p>That’s all for now, but please <a href="https://twitter.com/cheerfulstoic">let me know</a> if you have other thoughts for how to use this approach!</p>
<p>Along a similar line I’ve been thinking about graphing dependency trees like in RubyGems or NPM. Other project ideas are also welcome!</p>
<style>
table {
border-collapse: collapse;
width: 600px;
}
table, th, td {
border: 1px solid black;
}
td {
padding: 0.5em;
}
</style>Brian UnderwoodRecently the continuous builds for the neo4j Ruby gem failed for JRuby because the memory limit had been reached. I wanted to see if I could use my favorite tool (Neo4j) to analyize the memory usage. So I threw together a bit of Ruby code to use Ruby’s ObjectSpace.each_object functionality to dump every object in memory after the test suite of the gem finished (garbage collecting first, of course). First let’s take a look at the model: