Integrating Paperclip with Heroku and AWS S3

06/13/2015

Introduction

This post and tutorial gives background into how to build a basic create/read/update/delete (CRUD) application using Ruby/Rails. The focus of this post is how to set up a file attachment system and put it into production, using the fantastic resources contributed to the the rails and open source community from thoughtbot, Heroku, and Amazon. This post explores the specifics of thoughtbot’s paperclip gem, Heroku’s application hosting platform, and Amazon Web Services’ S3 for attachment/image hosting.

The Challenge and End Result

This post walks through the build out of a basic rails CRUD app and specifically addresses the following:

  • Integrating paperclip, heroku, aws s3, and friendly_id into a basic rails CRUD application that allows for image uploads.
  • Integration of a css library, jasny-bootstrap, that provides a clean UI for file uploads.
  • Basic setup of javascript for image previews, client side form validation, and Turbolinks functionality.
  • Deployment, hosting, and basic maintenance operations for paperclip in a production environment on Heroku.
  • Basic unit and integration testing with paperclip using spork, rspec, capybara, factory_girl, and selenium.

The end result of this tutorial is an application that can be seen HERE (please note this app is hosted using Heroku's free dynos and it may take 5-10 seconds to fire up).
All of the source code for this application is accessible on GitHub HERE.


Setting Up the Application - Front End UI, Back End, and Paperclip

We start by firing up a new rails app with the following:

rails new paperclip_heroku_aws --database=postgresql
rake db:create db:migrate

For this application we will call our main resource ‘posts’, and we can run the rails scaffold to give us a view directory, controller, and model.

rails g scaffold Post name:string description:text

Each post will have an optional image attachment. For this functionality, paperclip will act as the file attachment system. First, add the ‘paperclip’ gem to the gemfile:

#gemfile
gem "paperclip", "~> 4.2"

Now that we have the gem installed, we can use the paperclip generator to add the attribute to our posts table, which we will call ‘post_image’.

rails g paperclip post post_image

We also update the post model file to reflect the changes, validating that the attachments are of an image type and are less than 5 megabytes. If you are following with the source code of the final, you will see an additional style for the attachment. This is an agile addition covered later in this post.

#post.rb

has_attached_file :post_image

  validates_attachment :post_image,
                       :content_type => { :content_type => ["image/jpeg", "image/jpg", "image/gif", "image/png"] },
                       :size => { :less_than => 5.megabytes }

Lastly, update the post controller to white list the new field.

#posts_controller.rb

def post_params
    params.require(:post).permit(:name, :description, :post_image)
end

Now that we have the basic structure for our application, we can begin filling in the front end code to clean up the UI. I encourage you to view the source code of this project to see the HTML and CSS/SCSS. For file uploads, jasny-bootstrap provides a great library and interface as outlined here: http://jasny.github.io/bootstrap/2.3.1/javascript.html#fileupload. We have included this css and js library in our application (in the vendor directory), along with bootstrap.

//application.js

//= require jquery
//= require jquery.turbolinks
//= require jquery_ujs
//= require turbolinks
//= require bootstrap
//= require jasny-bootstrap
//= require posts
/*application.css*/

 *= require_self
 *= require scaffolds
 *= require bootstrap
 *= require bootstrap-reset
 *= require jasny-bootstrap
 *= require navigation
 *= require footer
 *= require posts
 *= require responsive
 *= require font-awesome

With the jasny and bootstrap asset libraries, we can develop a clean interface for uploading and previewing image attachments. The html and erb for the file upload preview displays the file upload button for new posts or posts with no image. For any posts with existing images, the change and remove buttons are shown. Similarly, the image preview area displays the existing post image as applicable, and is blank otherwise.

#posts/_form.html.erb

<div class="resource_form_section js-resource_form_section">
        <div class="row">
        <div class="col-sm-6 form_block_left">
          <div class="resource_form_section_title">
            <div class="title_left">Post Image</div>
            <div class="title_right"><i class="fa fa-check js-success_display"></i></div>
            <div class="title_right js-title_right js-title_optional">(optional)</div>
          </div>

          <% if @post.post_image.blank? %>
              <div class="fileupload fileupload-new" data-provides="fileupload">
                <div class="input-append">
                  <div class="new_resource_field">
                    <span class="fileupload-preview js-resource_field"></span>
                  </div>
                    <span class="btn btn-white btn-file">
                      <span class="fileupload-new">
                        Select file
                      </span>
                      <span class="fileupload-exists">
                        Change
                      </span>
                      <%= f.file_field :post_image, id: "post_image_input", class: "default" %>
                    </span>
                  <span class="btn btn-white fileupload-exists" data-dismiss="fileupload" id="post_image_remove_button">Remove</span>
                </div>
              </div>
          <% else %>
              <div class="fileupload fileupload-exists" data-provides="fileupload">
                <div class="input-append">
                  <div class="new_resource_field">
                  <span class="fileupload-preview js-resource_field">
                    <%= @post.post_image_file_name %>
                  </span>
                  </div>
                    <span class="btn btn-white btn-file">
                      <span class="fileupload-new">
                        Select file
                      </span>
                      <span class="fileupload-exists">
                        Change
                      </span>
                      <%= f.file_field :post_image, id: "post_image_input", class: "default" %>
                    </span>
                  <span class="btn btn-white fileupload-exists" data-dismiss="fileupload" id="post_image_remove_button">Remove</span>
                </div>
              </div>
          <% end %>
          <%= f.hidden_field 'hidden_post_image', value: @post.post_image_file_name, id: "hidden_post_image" %>
        </div>

There are a few javascript components of this image upload and preview to call out specifically.

First is the javascript responsible for changing the “Select file” button to two buttons of “Change file” and “Remove File” when a file is selected. This is handled by the jasny-bootstrap.js file. This file also places the file name text in the preview box.

The next piece of javascript is the image preview. The function for the file upload preview can be seen here.

//posts.js

$(document).on("change", "#post_image_input", function(event) {
        var files = event.target.files;
        var image = files[0];
        var reader = new FileReader();
        reader.onload = function(file) {
            var img = new Image();
            img.src = file.target.result;
            $("#post_image_target").css('background-image', 'url(' + img.src + ')'); //render the preview
            $('#hidden_post_image').val("file_present"); //set the hidden value with default text
        }
        reader.readAsDataURL(image);

        $(this).closest(".js-resource_form_section").find(".js-title_right").hide();
        $(this).closest(".js-resource_form_section").find(".js-success_display").show();
 });

The final javascript component to call out is the code that removes an image from an existing post when it is being edited. The documentation for paperclip outlines this scenario, and to accomplish this we set the image to nil in the posts_controller. In order to make the controller aware of an image removal, we use a hidden value field. This hidden value stores a string, and if this string is blank when passed to the controller, the image is set to nil.

#posts_controller.rb

def update

    #if the hidden_post_image field is blank, the user removed the image - set to nil before saving
    if params[:post]['hidden_post_image'] == ''
      @post.post_image = nil
    end

    #...

end

Error handling is a requirement in this basic application. Post names and post descriptions should not be blank, and likewise should not exceed 100 and 200 characters respectively. For server side validation, we can add the validations to the model. To render alerts on the client side we add in this erb to display the error values.

#posts/_form.html.erb

<% flash.each do |key, value| %>
         <div class="alert alert-error alert-block">
            <i class="fa fa-times fa-1x close_error_alert" data-dismiss="alert"></i>
            <%= value unless value.blank? %>
         </div>
<% end %>

Customizing the error alert displayed on the client can be accomplished with the following controller code. Note that flash.now prevents the error message from persisting between page loads on the client.

#posts_controller.rb

flash.now[:error] = "There was an error creating your Post:  " + @post.errors.full_messages.to_sentence

Rails also adds a ‘.fields_with_errors’ class to any fields with server side validation errors. We can customize the look by modifying the default ‘scaffolds.css.scss’ file to change the border color of any fields displaying errors.

//scaffolds.css.scss

.field_with_errors {
  .new_resource_field {
    border-color: $error_red;
  }
}

We must also include server side validation for the image attachment component of our application. Paperclip provides code to validate the content type and content size. We will restrict file attachments to those of images (.jpg, .jpeg, .gif, and .png extensions). In our application an image attachment will be optional, but if this were a requirement, we could add the additional presence validation test in the model.

At this point, we have the basic UI, error handling, and functionality for our application and we can begin writing tests. For an overview of the testing of this application, refer to the testing section below.


Client Side Validation and Turbolinks

There is one limitation of paperclip, however. Suppose we create a new post, enter a post name, and attach an image. We forget to fill in the description field, however, and leave it blank as we press the button to create the new post. Because of the server side validation requiring the presence of a post description, the new post will not be created and the “new” template is again rendered. This is the expected behavior with one caveat - the attached image is not preserved and will not carry over within the request. More information on this issue can be seen here and here.

While this is not a pressing issue for this application, it could be frustrating for users if there were multiple file attachments. For this reason, we will add in client side validation to our application. This validation prevents posts from being created or updated unless the post name and description fields are completed correctly. We will leave the server side validation in place for extra security. The code for client side validation can be seen in the posts.js file. Note that jQuery selectors are prefixed with “js-” to help remove dependencies with css class naming.

One additional important note to make regarding javascript in this application is the use of Turbolinks. As outlined in the documentation, Turbolinks makes web applications faster as it replaces only the body of each page. CSS and Javascript libraries are only loaded once, making links between pages faster. This is great for speed but also presents new issues, specifically with structuring javascript. For example, in posts.js we have two functions that run after the DOM has loaded. This works fine when the application is loaded the first time, but fails when linking with another page. To fix this issue, we add the ‘jquery-turbolinks’ gem and follow the documentation as outlined here: https://github.com/kossnocorp/jquery.turbolinks.

#gemfile

gem 'jquery-turbolinks'

Turbolinks also requires binding functions to the document rather than individual elements. This example application does not require the need for complex javascript structure. Should our trivial application here grow in size and complexity, Brandon Hilkert has put together great resources on how to structure page specific javascript in a rails app using Turbolinks.


Paperclip Integration with Heroku and AWS

At this point our application has its basic UI, and client side and server side error handling. We can now focus on the specifics required for putting an application using paperclip into production on Heroku.

First, setup and create your AWS S3 account. This will give you three environment variables, which will be added to Heroku. Next we add the aws-sdk gem to our gemfile. Note that a version less than 2.0 is used due to the following issue.

#gemfile

gem 'aws-sdk', '< 2.0'

Lastly we must update our production.rb file to configure paperclip with our AWS environment variables as outlined below.

#production.rb

config.paperclip_defaults = {
      :storage => :s3,
      :s3_host_name => 's3-us-west-2.amazonaws.com',
      :s3_protocol => :https,
      :s3_credentials => {
          :bucket => ENV['S3_BUCKET_NAME'],
          :access_key_id => ENV['AWS_ACCESS_KEY_ID'],
          :secret_access_key => ENV['AWS_SECRET_ACCESS_KEY']
      }
}

There are a few things to note with the above code. The s3_host_name corresponds with the region of the AWS bucket as outlined in this SO post. The s3_protocol of https ensures that the application calls the images using SSL. Without the https protocol, the following error will be present in your production environment:

Mixed Content: The page at 'https://guarded-plateau-4115.herokuapp.com/' was loaded over HTTPS, but requested an insecure image 'http://s3-us-west-2.amazonaws.com/paperclipherokuaws/posts/post_images/000/000/003/original/5k_prep.png?1434035423'. This content should also be served over HTTPS.

The last step before deploying to Heroku is to ensure the environment variables specified in the production.rb file are properly configured and added to our Heroku production environment. We will add in the three variables required for paperclip and AWS S3 (this can be done using the command line or Heroku's UI).

heroku config:set S3_BUCKET_NAME=your_aws_s3_bucket_name

And now we are all set. We can follow the general procedures for loading our application into production on Heroku.


Application Maintenance and Updates

Now that our application is in production, we realize there are two new requirements.

The first requirement is that we would like to have “friendly urls” for each post. Rather than '/posts/1' we would like to have 'posts/name-of-post'. This can be quickly accomplished using the friendly_id gem. Following the friendly_id documentation, we add the gem to our gemfile and run the generator.

gem 'friendly_id', '~> 5.1.0'
rails g friendly_id

This adds a table to our application called ‘friendly_id_slugs’ and stores the history of each slugged resource. Friendly_id will generate a unique “slug” that becomes the url displayed with each post. As such, we must add a slug column to our posts table.

#add_slug_to_posts.rb - migration file

class AddSlugToPosts < ActiveRecord::Migration
  def change
    add_column :posts, :slug, :string
    add_index :posts, :slug, unique: true
  end
end

Now that we have the new table column to store post slugs, we must add in the associated model code to specify that url slugs should be made from a given posts name. We also override the should_generate_new_friendly_id? method provided by friendly_id so that a new post slug is created whenever the name of the post is edited. The use: :history option ensures that even if a post’s name is changed, any old names associated with that post can still be routed correctly to the post in its current state.

#post.rb

include FriendlyId
friendly_id :slug_candidates, use: :history

def slug_candidates
  [
        :name,
        [:name, :id]
  ]
end

#post slug should change every time the post's name is updated
#the old name will still be accessible in a url due to the friendly_id history feature
def should_generate_new_friendly_id?
  name_changed? || super
end

Lastly, we must update our posts controller so that it can correctly find the posts in question, using the "friendly" finder provided by friendly_id.

#posts_controller.rb

def set_post
    @post = Post.friendly.find(params[:id])
end

Now that friendly_id has been correctly installed, we must update all of the previous records we created in order to give them their unique url slugs. We could do this running the rails console, but we will also have to do this in our production environment as well so we will create a rake task for it.

#/lib/tasks/update_post_slugs.rake

task :update_post_slugs => :environment do
  Post.find_each(&:save)
end

One more requirement we have deals with paperclip. On the front end of our application we take advantage of percentages to allow for responsive resizing of images. On the posts index page, however, the size of the image will remain fixed. There is no need to load the image at its native size just to resize the container div for the image down to a smaller size. To increase speed of the application when loading the posts index page, we can make use of paperclips post processing to create a copy of the uploaded image at a smaller resolution. We know that the size of images on the index page will be a width of 289px and a height of 130px. As such we add a style (called ‘index’) to the post model.

#post.rb

has_attached_file :post_image, styles: {index: "289x130#"}

Now in the index view we can make use of the smaller image to enhance page processing.

#posts/index.html.erb

<div class="aspect_wrapper_main_image">
    <div class="content_main_image" style="background-image: url('<%= post.post_image.url(:index) %>');">
    </div>
</div>

Just as before with friendly_id we must update all existing post images to make use of the new style we created. Fortunately, thoughtbot has already created a rake task in paperclip that accomplishes this.

rake paperclip:refresh:missing_styles

Now we can push our application back into production, migrate the database and run the rake tasks on Heroku.

heroku run rake update_post_slugs
heroku run rake paperclip:refresh:missing_styles

And there we have it! A trivial, but fully functioning app that makes use of several great gems.


Testing

Of course this application would not be complete without testing. For this application we will make use of unit testing and integration testing. Scenarios we must test for include proper handling of incomplete fields and invalid attachment types. In order to complete this testing we will make use of several gems including factory_girl, rspec, and capybara.

In our gemfile we add the following:

#gemfile

group :development, :test do
  gem 'factory_girl_rails'
  gem 'rspec-rails', '2.13.1' #http://stackoverflow.com/questions/22962048/rails-4-could-not-find-generator-rspecinstall
  gem 'selenium-webdriver', "~> 2.43"
  gem 'capybara'
  gem 'spork-rails'
  gem 'childprocess'
  gem "chromedriver-helper" #use chrome with selenium
  gem "database_cleaner"
  gem 'minitest' #needed for rspec support
end

First we run the generator for rspec as follows:

rails generate rspec:install

This creates the /spec folder where all tests will reside. Next we run the spork generator.

spork rspec --bootstrap

Spork creates a copy of our rails application and sets up a test suite. Spork allows us to make changes to tests and re run the test suite all without reloading the entire application. Take a look at the /spec/spec_helper.rb file to see the custom configuration of including the rspec configuration within the spork prefork block. We also add the capybara gem, which is used for integration testing.

In addition to paperclip, thoughtbot has also contributed the factory_girl_rails gem to the open source community. Factory girl lets us quickly define test fixtures for each model. In our example application, there is only one model, but factory girl allows for quick setup of test objects and any model associations. Our factories.rb file defines our Post factory:

#spec/factories/factories.rb

FactoryGirl.define do
  factory :post do
    sequence(:name)  { |n| "Post#{n}" }
    sequence(:description)  { |n| "Description#{n}" }
    post_image { File.new(Rails.root.join('app', 'assets', 'images', 'test_image.png')) }
    #http://stackoverflow.com/questions/3294824/how-do-i-use-factory-girl-to-generate-a-paperclip-attachment
  end
end

There is another necessary configuration before moving into the actual tests. In our test suite we will need to use javascript in a browser for one of the tests. This requires the use of selenium, which lets us load and automate a browser for testing. Using selenium also requires us to install the Database cleaner gem. Avdi Grimm provides great resources in setting up a rails test suite for use with rspec, capybara, selenium, and database_cleaner here. Because we are using javascript and selenium in a test we must follow the guidelines he sets forth so that we use a truncation rather than a transaction strategy for this type of test. As a result, we add the following to our spec_helper file within the RSpec configuration block.

#spec/spec_helper.rb

#http://devblog.avdi.org/2012/08/31/configuring-database_cleaner-with-rails-rspec-capybara-and-selenium/ --------------
      config.use_transactional_fixtures = false

      config.before(:suite) do
        DatabaseCleaner.clean_with(:truncation)
      end

      config.before(:each) do
        DatabaseCleaner.strategy = :transaction
      end

      config.before(:each, :js => true) do
        DatabaseCleaner.strategy = :truncation
      end

      config.before(:each) do
        DatabaseCleaner.start
      end

      config.after(:each) do
        DatabaseCleaner.clean
      end
#---------------------------------------

One last configuration is also necessary for testing paperclip file attachments. All of the test images we create need a test directory. Without a test directory specified (I originally neglected this step), these images will end up in the public/system/posts/post_images directory, and you will run into issues deploying your application to Heroku and your compiled Heroku slug will exceed the maximum size allowed. So we will add the following code to our spec_helper file, which removes the temporary file attachment directory after each test suite run.

#spec/spec_helper.rb

config.after(:suite) do
      FileUtils.rm_rf(Dir["#{Rails.root}/spec/test_files/"]) #removes the temporary directory to hold paperclip file attachments
    end

And now we are finally to build and run some tests. The test files can be seen in the /spec folder and can be run as follows (be sure to load the spork server in a new terminal window):

spork
rspec spec/models
rspec spec/features
rspec spec

One thing to call out is the use of let and let! in our initialization of factory_girl test objects. Post_1, Post_2, and Post_3 are all initialized using let!. meaning that they are available in all tests and have been initialized. However, the Post object will not be initialized until it is explicitly called within a given test. Each time it is called, it is unique to that specific test.

#spec/features/posts_spec.rb

let(:post) { FactoryGirl.build(:post) }

let!(:post_1) {FactoryGirl.create(:post, created_at: Time.now - 4.days)} #oldest
let!(:post_2) {FactoryGirl.create(:post, created_at: Time.now - 3.days)} #middle
let!(:post_3) {FactoryGirl.create(:post, created_at: Time.now - 2.days)} #newest

And that wraps it up, we have our tests in place. One thing to note is that these tests are still largely the reflection of server side validations. With client side validation (and javascript enabled in the browser), many of the tests will not reach server side validation.


Acknowledgements and Thank You's

As you saw throughout this post, there are several people who deserve the credit and thanks for the information presented here. First, is a thank you to thoughtbot for their great paperclip gem and fantastic documentation. Also a thank you to heroku and Amazon for their great platforms that make putting rails applications into production very straightforward.

Thank you to Avdi Grimm for his helpful resources on testing, and Arnold Daniels for his jasny-bootstrap contributions.

And of course thanks to all of the gem contributors and authors of dozens of StackOverflow Posts, all of which were used in the making of this example application:
paperclip
friendly_id
jquery-turbolinks