Moving to a Rails Engine

At work we have a primary Rails application (ye ole monolith) and several microservices that handle our business, which is an email newsletter provider and delivery system. As you can probably imagine, email newsletters are sent to, well, emails! When a client adds new emails (read: people) to the system, we define them as members.

Simple Models, Complex Actions

The basic structure of our models, as it relates to new members, is as follows (pretty easy and straight forward):

class Member < ApplicationRecord
  belongs_to :client
  # there are other relationships but they're not really relevant here
end

class Client < ApplicationRecord
  has_many :members
end

The Problem

Despite the simplicity of the models, the rules surrounding adding new members varies based on what app your are in, how the member is being added, and so on. In fact, there are 5 distinct ways in which a new member can be added:

  1. Via the admin interface (i.e., a typical form) in our app by an administrator
  2. Via the admin interface in our app by a client (this form is different than the administrator form)
  3. Via a spreadsheet with many (or just one) emails uploaded by the client
  4. Via a client’s public newsletter signup form
  5. Via a client’s newsletter, which has an option to “refer a friend”

Each of these ways of adding a new member has some slight variations in what they need to do to make sure a new member is being legitimately added. I don’t think it’s a big step for readers to imagine the sh$t-storm of conditional logic that would get thrown around five different controllers across three or four apps just to make sure all the right rules are followed, etc.

We had a lot of little issues over the years as our client and member base grew and we had more and more people adding members to their DB in all the different ways.

The Solution, Part I

I went through quite a few options and approaches to creating a consistent interface for adding new members to our system. The most important step in that process was stepping back and asking the following question: “what actions are common across all instances of adding a new member?”

What I found was that the following things were common and could be coded in the base Member model:

class Member < ApplicationRecord
  include Verifiable
  include EmailValidatable

  validates :email, presence: true, email: true, if: proc { |object| object.new_record? || object.email_changed? }, uniqueness: { scope: :client_id, message: 'already subscribed' }
  validates_with ExclusiveValidator, if: proc { |object| object.new_record? || object.email_changed? }
  validates_with PermittedValidator, if: proc { |object| object.new_record? || object.email_changed? }
  validates_with SpamTrapValidator, if: proc { |object| object.new_record? || object.email_changed? }
end  

When a new member is added, we have to first verify it’s a legitimately formatted email address (the validates :email line). Next, we have to make sure that the email is actually a real, live, usable email and not some spam trap, etc. (this is performed via both the EmailValidtable module, which is really part of the validates :email method, and Verfiiable concern).

Once we know that the email is legit-legit, we can proceed to our next common steps and ensure it’s a permitted email (we do not allow role accounts, such as info@domainname.extension or sales@domainname.extension, for example), not an already found Spam Trap (in case FreshAddress missed it, we double-check our records) and, finally, we make sure that the email is not exclusive to another client in our system.

If all those rules are met, we can proceed with adding the new member to the system … assuming the specific rules for the 5 spots to add a new member, noted above, are also met. Note: I did not determine the business logic and, while some of it is just the result of the growth of a large, enterprise application over 10 years but some of it is just the result of business people trying to dictate coding rules … pretty sure most of us have been there LOL!

Plug: We use FreshAddress as our third-party email verification system and it is pretty dang impressive how many crap emails they can weed out for us. I’m just offering in case anyone else finds themselves in need of such a service and is not sure who is good and who isn’t.

The Solution, Part II

Now that I knew what commonality exists in the action of adding a new member, I had to figure out how in the eff I was going to pull it off. After a good bit of reading, searching, browsing, reading and then some more browsing and reading, I opted for a mountable Rails Engine.

After all that reading, I ended up being pretty uncreative and just used the Rails Guides - https://guides.rubyonrails.org/engines.html - to get me started. And it was plenty. I have to note that we in the Ruby and Ruby on Rails world are so fortunate to have such fantastically detailed documentation.

I chose a mountable engine because I needed to easily include it in several of our Rails apps and use it without having to refactor a ton of code. Basically, I needed as close to a plug and play solution as I could get.

The Engine, named the member_validator, has a base set of the models related to adding a new member to be used in each and every app in which it’s used. In total, the engine ended up with a set of 14 models, 3 concerns (2 for models and 1 for a controller in the including app) and a bevy of service objects.

One concern we had with mounting the engine was namespacing. Lots of engines will namespace their models in the including app with a format such as EngineName::ModelName.

Because this engine was being extracted from an existing, nearly-decade-old application (as well as going into three three-to-four-year-old microservices), we had to make sure that the models could be used in those apps without namespacing them (because there were 100s or 1000s of calls across all the apps that did not use any namespacing in the pre-engine world).

By mounting it and not using an engine-specific namespace, we were able to use the models in the existing apps without having to refactor a ton of code. Any call to Member.where(...) would still work in the existing apps and would not need to be changed to MemberValidator::Member.where(...).

Long and short, coding in the engine was really easy, especially since Rails gives you a “dummy” app in the Engine you can use to test, etc. But, in the end, I just “coded” the Engine like I would any Rails app (ok, just about any Rails app).

The API

The Engine exposes a five methods to the including app, which is all handled through a Controller Concern that is/will be included in the apps. The two that are used quite regularly are certify and pre_flight. The others are helpful but rarely used across the apps.

module Validatable
  extend ActiveSupport::Concern

  # To use these methods in the containing app:
  # include Validatable
  # returns an EmailAddress object
  # verifies the email is legit across the board
  def certify(member_params:)
    member = member_params.is_a?(Member) ? member_params : Member.new(member_params)
    Members::Certify.new(member_to_certify: member).call
  end

  # sometimes an email goes bad and we need to check it ... that's what this does
  def recertify(member:, date_check: true)
    Members::Recertify.new(member: member).call if date_check
  end

  # no validation requried (admin only feature!)
  def manual_override(member:)
    Members::ManualOverride.new(member: member).call
  end

  # just seeing if things are going to go well here
  def spot_check(email)
    Emails::VerificationResults.new(member: Member.new(email: email)).call
  end

  # this basically does a quickie check on the email address
  def pre_flight(email, client_id)
    Emails::PreFlight.new(email: email, client_id: client_id).call
  end
end

The Solution, Part III

Now, we have the fun of integrating the Engine. Step one is pretty darn easy. Add your Engine, which is really just a gem, to your Gemfile:

gem 'member_validator'

Next, in a controller where you’d like to use the Engine, you would do the following:

class EmailProcessorsController < ApplicationController
  include Validatable # this is the main thing! 
  helper EmailProcessorsHelper

  def create
    # here we are calling the pre_flight method from the Engine
    @results = pre_flight(params[:email].strip.downcase, params[:client_id])
    respond_to do |format|
      format.turbo_stream {}
    end
  end
end

Here is the same in another controller but where we certify the email:

class ClientsController < ApplicationController
  def create
    # the certify method from the engine
    @member = certify(member_params: member_params) 
    # do other stuff here :) 
  end
end

The Results

We’ve found that this new approach has reduced both the complexities in our codebase(s) as well as making the process of adding a new member far more consistent and reliable. It was pretty easy to build out and include the Engine in our apps once we figured out that the Engine was the way for us to go in terms of solving this particular problem.

One piece of general advice that I had to tell myself over and over: DO NOT ENGINE ALL THE THINGS! You, like me, may want to as you get started but, really, it’s the the solution for every problem like this. It just happened to be the right one for our app.

Gotchas/Issues

One thing we did run into with the move to an Engine was the models themselves. Using just our base Member model as an example, it had a different set of methods in it in one app versus another. Part of this was the model being used differently in each app but it was also a legacy issue from our early days of development when we followed the “Skinny Controller, Fat Model” approach.

Over time, this became, “ALL THE THINGS ARE SKINNY”. Now our models are merely wrappers for the table they represent (we do have a few, more robust, tableless models) and maybe a method or two to easily manipulate/present the data (e.g., the Member model has a full_name method that returns the first and last name of the member). However, for these older models, we could not just remove all the methods and logic because they were called from all over each app. We simply have not had enough time for such a refactoring.

Anyway, there was/is a nice solution: Overrides. In each app, we have a folder called app/overrides that includes just the models from the Engine that we need to override in the particular app:

# it feels backwards but the `instance_eval` is your class methods (i.e., self.method_name)
Member.instance_eval do
  # we don't use all relationships in all apps so we can customize them here for this particular app's override of the model
  has_many :other_model, dependent: :destroy
  has_many :other_model_2, dependent: :destroy
  
  # great example where we do not keep an audit log for all things in all apps and did not want to 
  # add PaperTrail to the Engine's gem dependencies since it's only used in a couple of the apps, we let the apps handle that.
  has_paper_trail ignore: %i[created_at updated_at]

  # this scope is only used in one app s... 
  scope :at_risk_members, lambda {
    # fancy scope things here
  }
end

# it feels backwards but the `class_eval` is your instance methods
Member.class_eval do
  def phone_number
    # a nifty way to get a phone number
  end
end

To load your overrides, you only need to add the following code block to config/application.rb:

# overrides
overrides = Rails.root.join('app/overrides')
Rails.autoloaders.main.ignore(overrides)
config.to_prepare do
  Dir.glob("#{overrides}/**/*_override.rb").each do |override|
    load override
  end
end

In some apps, having to add overrides could be a pain in the butt. Fortunately, it was pretty easy across our apps and, in most cases, we only override a couple models here and there from the Engine.

Screenshot of Engine File Structure

On its own, the Engine is really nothing more than an incredibly simple Rails app. I’ve included a screenshot of the file structure below just to illustrate how “normal” the Engine will look to a Rails developer. For me, seeing and understanding that it was, really, just another Rails app made it more approachable and less intimidating.

Engine Screenshot