At Drivy, our main repository is a Ruby on Rails application that we run on Heroku. Sometimes, things don’t go as planned and we need to run one-off commands to fix a particular piece of data or to investigate a bit further about an issue. To do this, we use the rails console
command in the production environment.
Let’s be clear: we reach for the console only if we have no other choice. We always deploy new code to fix bugs or use migrations if we need to update a large number of database rows. But as we still need it sometimes, we need to be sure this is a tool we can trust and control. Our rule is that if a command has been run more than twice, it needs to be automated in our back-office. We also have processes to limit the access to this feature to a group of people and rules in place to comply with our data privacy policy.
We want commands typed by authorised developers or system administrators to be made public and available in real-time. This serves multiple purposes:
We use pry locally, but the Rails console uses irb
in staging or production. We needed a way to hook into irb
and we used the fact that irb
interacts with the standard output to override the behaviour of the STDOUT
class. To know if we need to change the behaviour of the standard output, we check if we are running in a one-off Heroku dyno thanks to the environment variable DYNO
set by Heroku.
We added an initializer which looks like this:
is_staging_or_prod = Rails.env.production? || Rails.env.staging?
dyno_in_run_mode = ENV.fetch('DYNO', 'nope').starts_with?('run')
dev_name = ENV.fetch('DEV_NAME', '')
if is_staging_or_prod && dyno_in_run_mode
raise Drivy::Errors::DevNameNotSetError if dev_name.blank?
# Override how printing to sdtout works by sending
# the output of stdout to a Slack webhook also.
# When writing commands in irb, irb prints to stdout
class << STDOUT
include Drivy::Console::ReportCommand
alias :usual_write :write
def write(string)
usual_write(string)
send_command_to_slack(dev_name, string)
end
end
end
And the ReportCommand
class actually does the work of reading from the standard output history using Readline::HISTORY
and sending the data to an external service (Slack for us). The code below gives the main logic, the complete code is available in a gist.
module Drivy::Console::ReportCommand
def send_command_to_slack(developer_name, command_output)
return unless has_command? && has_output?(command_output)
# Documentation is at https://api.slack.com/docs/message-attachments
fields = [
{
title: "Command",
value: wrap_command(read_command),
short: true,
},
{
title: "Output",
value: wrap_command(parse_output(command_output)),
short: false,
},
{
title: "Developer",
value: developer_name,
short: true,
}
]
env_color, env_title = ["#e74c3c", "production"]
params = {
attachments: [
{
fields: fields,
color: env_color,
footer: "Console #{env_title} spy",
footer_icon: "https://drivy-prod-static.s3.amazonaws.com/slack/spy-small.png",
ts: Time.zone.now.to_i,
mrkdwn_in: ["fields"],
}
]
}
response = slack_client.post('', params)
raise "Failed to notify Slack of console command, status: #{response.status}" unless response.success?
end
private
def has_command?
Readline::HISTORY.length >= 1
end
def has_output?(command_output)
return false unless command_output.instance_of? String
command_output.strip.start_with? "=>"
end
def read_command
Readline::HISTORY[Readline::HISTORY.length-1]
end
end
We use the console thanks to our homemade Drivy CLI and not directly through the Heroku CLI. We will likely talk about our CLI in upcoming posts, it is a tool we use to manage our day-to-day operations (running commands, releasing, handling database migrations, managing content…). After configuring the Slack webhook integration, the final result looks like this:
We’re pretty happy about this new tool because we gained a lot in visibility and confidence in our operations. We are always looking forward to improving our developers’ tooling.