Send Rails console commands to Slack

At Drivy, our main repository is a Ruby on Rails application that we run on Heroku. Sometimes, things don’t go as planned and we need to run one-off commands to fix a particular piece of data or to investigate a bit further about an issue. To do this, we use the rails console command in the production environment.

Let’s be clear: we reach for the console only if we have no other choice. We always deploy new code to fix bugs or use migrations if we need to update a large number of database rows. But as we still need it sometimes, we need to be sure this is a tool we can trust and control. Our rule is that if a command has been run more than twice, it needs to be automated in our back-office. We also have processes to limit the access to this feature to a group of people and rules in place to comply with our data privacy policy.

Reporting on console commands

We want commands typed by authorised developers or system administrators to be made public and available in real-time. This serves multiple purposes:

• be aware when this happen: commands should be executed manually only on special conditions. If we need to run commands to fix events often, we need to build something that can handle automatically this kind of events to avoid at all cost the need to run console commands.
• have an history of executed commands: if at some point we encounter an issue we had weeks ago, we can see how we fixed the issue by looking at the commands’ log.
• let developers discover commands: because commands are made public, developers often discover interesting ways to fix an issue. This leads to discussions and code improvements later.

Hooking into the Rails console

We use pry locally, but the Rails console uses irb in staging or production. We needed a way to hook into irb and we used the fact that irb interacts with the standard output to override the behaviour of the STDOUT class. To know if we need to change the behaviour of the standard output, we check if we are running in a one-off Heroku dyno thanks to the environment variable DYNO set by Heroku.

We added an initializer which looks like this:

is_staging_or_prod = Rails.env.production? || Rails.env.staging?
dyno_in_run_mode = ENV.fetch('DYNO', 'nope').starts_with?('run')
dev_name = ENV.fetch('DEV_NAME', '')

if is_staging_or_prod && dyno_in_run_mode
  raise Drivy::Errors::DevNameNotSetError if dev_name.blank?

  # Override how printing to sdtout works by sending
  # the output of stdout to a Slack webhook also.
  # When writing commands in irb, irb prints to stdout
  class << STDOUT
    include Drivy::Console::ReportCommand
    alias :usual_write :write

    def write(string)
      usual_write(string)
      send_command_to_slack(dev_name, string)
    end
  end
end

And the ReportCommand class actually does the work of reading from the standard output history using Readline::HISTORY and sending the data to an external service (Slack for us). The code below gives the main logic, the complete code is available in a gist.

module Drivy::Console::ReportCommand

  def send_command_to_slack(developer_name, command_output)
    return unless has_command? && has_output?(command_output)

    # Documentation is at https://api.slack.com/docs/message-attachments
    fields = [
      {
        title: "Command",
        value: wrap_command(read_command),
        short: true,
      },
      {
        title: "Output",
        value: wrap_command(parse_output(command_output)),
        short: false,
      },
      {
        title: "Developer",
        value: developer_name,
        short: true,
      }
    ]

    env_color, env_title = ["#e74c3c", "production"]

    params = {
      attachments: [
        {
          fields: fields,
          color: env_color,
          footer: "Console #{env_title} spy",
          footer_icon: "https://drivy-prod-static.s3.amazonaws.com/slack/spy-small.png",
          ts: Time.zone.now.to_i,
          mrkdwn_in: ["fields"],
        }
      ]
    }

    response = slack_client.post('', params)

    raise "Failed to notify Slack of console command, status: #{response.status}" unless response.success?
  end

  private

  def has_command?
    Readline::HISTORY.length >= 1
  end

  def has_output?(command_output)
    return false unless command_output.instance_of? String
    command_output.strip.start_with? "=>"
  end

  def read_command
    Readline::HISTORY[Readline::HISTORY.length-1]
  end
end

We use the console thanks to our homemade Drivy CLI and not directly through the Heroku CLI. We will likely talk about our CLI in upcoming posts, it is a tool we use to manage our day-to-day operations (running commands, releasing, handling database migrations, managing content…). After configuring the Slack webhook integration, the final result looks like this:

We’re pretty happy about this new tool because we gained a lot in visibility and confidence in our operations. We are always looking forward to improving our developers’ tooling.