How to Use Postgres to Calculate and Save Rankings in Rails or Anything

Standard

When you need to calculate rank and save it to a field in Postgres.

A naive way would be to build an iterative method with an index.

Product.order(price: :desc).each_with_index do |p, i|
  p.price_rank = i + 1
  p.save
end

This might work great if you have a limit(10) or have a small data sample. But what if you 100,000 records or even worse 1 million records.

For 100 items the above code took 0.181s.
For 1,000 items .49s.
For 100,000 items 155.04s.

Now using Postgres I know it’ll be faster but how much?

query = "UPDATE products SET price_rank = r.rnk FROM (SELECT id, RANK() OVER (ORDER BY price DESC) as rnk FROM products) r WHERE products.id = r.id"

ActiveRecord::Base.connection.execute(query)

I am getting run times of .0005 because I am dumping the work onto the database.

Running EXPLAIN ANALYZE on the query.

 Update on products  (cost=16813.40..25200.70 rows=101002 width=112) (actual time=15247.855..15247.855 rows=0 loops=1)
   ->  Hash Join  (cost=16813.40..25200.70 rows=101002 width=112) (actual time=8693.946..12299.080 rows=101002 loops=1)
         Hash Cond: (products.id = r.id)
         ->  Seq Scan on products  (cost=0.00..3391.02 rows=101002 width=68) (actual time=0.584..990.371 rows=101002 loops=1)
         ->  Hash  (cost=14563.87..14563.87 rows=101002 width=56) (actual time=8693.010..8693.010 rows=101002 loops=1)
               Buckets: 65536  Batches: 4  Memory Usage: 2685kB
               ->  Subquery Scan on r  (cost=11786.32..14563.87 rows=101002 width=56) (actual time=2278.120..7232.235 rows=101002 loops=1)
                     ->  WindowAgg  (cost=11786.32..13553.85 rows=101002 width=24) (actual time=2278.083..5283.472 rows=101002 loops=1)
                           ->  Sort  (cost=11786.32..12038.82 rows=101002 width=16) (actual time=2278.053..3262.086 rows=101002 loops=1)
                                 Sort Key: products_1.price DESC
                                 Sort Method: external merge  Disk: 2576kB
                                 ->  Seq Scan on products products_1  (cost=0.00..3391.02 rows=101002 width=16) (actual time=0.639..992.346 rows=101002 loops=1)
 Planning time: 3.271 ms
 Execution time: 15252.513 ms

Over a 100,000 records ranked and updated with values in 15s. Not bad.

You also get the added logic of a true ranking when there is a tie like #1, #2, #2, #2, #5, #6, etc. The calculated version was simple and did not account for it. If you do want to, you can implement it but with Ruby, you’ll add more bloat and complexity.

Is Rails Scalable?

Standard

Being an optimist I would say, Yes! Having worked on a large scale high visibility site I’ve personally seen it happen, so definitely Yes! When the budget was not a limiting factor, I saw vertical and horizontal scaling at it’s best. Get the biggest badest server, then multiply that by 20. CDN cache everything possible. Scaling solved. Millions of views, thousands of transactions per second no problem.

But what if you can’t. Here are my experiences.

Vertical Scaling

You add more RAM, CPUs, faster pipes (fiber optics directly to another server or network), SSDs. This is all possible, as CPUs reach their limits this gets to be a limiting factor. This has a threshold. With more users, the server can still meltdown. This is not for an exponent growth more of a linear one.

This also holds true for the database server, as most issues are database related. Slow queries can run faster by beefing up the hardware it runs on.

I consider this a quick fix for linear or predictable growth.

Horizontal Scaling

Load Balancing! Get more servers to do the job. Two or twenty is better than one. Share the load between servers. This can and will get costly. Be it cloud or metal or a combination of the two. Horizontal scaling if done correctly can be a really easy solution to scaling issues.

Continue reading

Part 1: An API using Phoenix, Absinthe (GraphQL), Guardian, React, Apollo Client

Standard

I love my Rails, but this is something I am really interested in.

It all started with a little bookkeeping app written using Phoenix as an API and React as a view layer. Has now turned into an extremely deep rabbit hole. After completing the first features of the app, I had the question “These states in these components are getting out of hand, I wonder if there’s something to help me?” This lead to Redux, one object to hold the state of the entire app. Then lead to the question, “Is there a better way to handle the data request to the API?” This lead to GraphQL which lead to Absinthe which made sense since I am using Phoenix as my API.

I think the best way to learn is to build something simple but not trivial. So let us begin with an e-commerce store that sales widgets.

Users >> Carts >> Widgets

Users can have carts to place widgets in them, etc.

I am going to break this down to 3 to 4 articles:
Part 1: Introduction to Phoenix, Absinthe, and Guardian – explaining the pieces.

Part 2: Implementation of the Phoenix/Absinthe API.
This is how you go about creating the API.

Part 3: Implementation of React/Redux with Apollo client.
We will create the frontend while still inside Phoenix, no additional steps Webpack will handle it.

Part 4: Additional nice things to have like: Elasticsearch, a Pub/Sub feature, etc. This one is a maybe.

Code can be found here.

https://github.com/mfung/widgets_r_us

What is Phoenix?

Phoenix is a web development framework similar in vein to Ruby on Rails. Phoenix is written in Elixir. Elixir is a functional, concurrent, and general-purpose programming language. Phoenix follows the MVC pattern and was developed to create highly performant and scalable web applications.

Laymen terms: You use it to build websites fast and efficiently.

What is GraphQL?

GraphQL was developed by Facebook in 2012 and released to the public in 2015. It is a data query and manipulation language for APIs.

Simply: An efficient and flexible way to get data, by eliminating unwanted data. I define what data I need from the endpoint.

What is Absinthe GraphQL?

Absinthe is “The GraphQL toolkit for Elixir.” Since Phoenix is written in Elixir and most APIs are web APIs. This just makes sense.

Guardian

If you are familiar with Devise in the Ruby world Guardian is similar. It is an authenication library for Elixir applications.

React

React is a JavaScript library used to build user interfaces for single-page applications and is optimal for fetching rapidly changing data.

Apollo GraphQL (Client)

Apollo comes in server and client. We will be dealing with the client. Apollo Client is a state management library for JavaScript GraphQL apps.

Next

docker-credential-pass Ubuntu setup for Bamboo CI/CD

Standard

Ubuntu: 18.04
Docker: 18.09.5
docker-credential-pass: 0.6.2
pass: 1.7.1
gpg2: 2.2.4

Since my setup is for CI/CD I have a bamboo user to deploy my docker containers to Docker Hub and deploy to my production servers. Your use case may vary, but if you want your credentials to persist longer than a couple of hours before your system starts asking you to re-enter your passphrase this might work for you.

0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
 
# First install apps we'll need
sudo apt upgrade && apt install pass gnupg2 tmux
 
# Download docker-credential-pass
wget https://github.com/docker/docker-credential-helpers/releases/download/v0.6.2/docker-credential-pass-v0.6.2-amd64.tar.gz
 
# Unpack
tar xvf docker-credential-pass-v0.6.2-amd64.tar.gz
 
# Move into /usr/bin
sudo mv docker-credential-pass /usr/bin
 
# Check to see if it's working should return {}
docker-credential-pass list
 
# Here is where my setup will diverge
# I login to my bamboo user
su - [username]
 
# I run tmux
tmux
 
vi ~/.gnupg/gpg-agent.conf
 

My gpg-agent.conf file looks like this:

0
1
2
3
 
default-cache-ttl 315569520
max-cache-ttl 315569520
 
0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
 
# Restart gpg-agent
echo RELOADAGENT | gpg-connect-agent
 
# Generate key
gpg2 --full-generate-key
 
# You'll get an output with a key, copy it to use
pass init DXXXXXXXXXXXXXX6E1AFXXXXXXXXXXB9XXXX497
 
# Enter a password
pass insert docker-credential-helpers/docker-pass-initialized-check
 
# You should get a return of {} instead of 'pass store is uninitialized'
docker-credential-pass list
 
# login to docker
docker login
 
# Check to see if it is stored
docker-credential-pass list
 
# should return something like
{"https://index.docker.io/v1/":"[username]"}
 

This worked for me. Also note the default-cache-ttl value is in seconds, the values I used is 10 years in seconds.

Getting SimpleCov-summary (0.0.5) to work with SimpleCov (0.16.1) and Rails 5.2

Standard

If you run into issues with installing simplecov-summary using:

Ruby
0
1
2
3
4
5
6
7
8
9
10
11
12
13
 
# Put this in for RSpec, Minitest will plug in automatically
  RSpec.configure do |config|
    config.after(:suite) do
      if SimpleCov.running
        silence_stream(STDOUT) do
          SimpleCov::Formatter::HTMLFormatter.new.format(SimpleCov.result)
        end
 
        SimpleCov::Formatter::SummaryFormatter.new.format(SimpleCov.result)
      end
    end
  end
 

Here’s how to fix it:

Ruby
0
1
2
3
4
5
6
7
8
9
10
 
# Put this in for RSpec, Minitest will plug in automatically
  RSpec.configure do |config|
    config.after(:suite) do
      SimpleCov.at_exit do
        SimpleCov::Formatter::SummaryFormatter.new.format(SimpleCov.result)
        SimpleCov::Formatter::HTMLFormatter.new.format(SimpleCov.result)
      end
    end
  end
 

Since Rails 5 removed silence_stream because it is not thread-safe. I found implementations of silence_stream but I got it to work by just removing it completely and I prefer to see these outputs at the end of my test run.

Rails 5.2 Has And Belongs To Many with UUID

Standard

When using UUIDs and you are creating a join table like so:

Shell
0
1
2
 
rails g migration CreateJoinTableUserCompany users companies
 

You will get a migration file like so:

Ruby
0
1
2
3
4
5
6
7
8
9
 
class CreateJoinTableUserCompany < ActiveRecord::Migration[5.2]
  def change
    create_join_table :users, :companies do |t|
      #t.index [:user_id, :company_id]
      #t.index [:company_id, :user_id]
    end
  end
end
 

You’ll need to modify it accordingly:

Ruby
0
1
2
3
4
5
6
7
8
9
 
class CreateJoinTableUserCompany < ActiveRecord::Migration[5.2]
  def change
    create_join_table :users, :companies, column_options: { type: :uuid } do |t|
      t.index [:user_id, :company_id]
      t.index [:company_id, :user_id]
    end
  end
end
 

Then run:

Shell
0
1
2
 
rails db:migrate
 

And you should be good. Just remember if you are using UUIDs you might want to make sure all models you generate use UUIDs.

In config/application.rb:

Ruby
0
1
2
3
4
 
config.generators do |g|
  g.orm :active_record, primary_key_type: :uuid
end
 

And make sure your generated migrations have:

Ruby
0
1
2
3
4
5
6
7
8
9
 
class CreateCompanies < ActiveRecord::Migration[5.2]
  def change
    create_table :companies, id: :uuid do |t|
      t.string :name, limit: 80
      ...
    end
  end
end
 

Enjoy!

Installing Jira Software Server

Standard

Assuming you have downloaded from Atlassian. If not read how to prepare for installs.

Shell
0
1
2
3
4
 
chmod +x atlassian-jira-software-7.12.1-x64.bin
 
sudo ./atlassian-jira-software-7.12.1-x64.bin
 

Follow prompts and make sure you enable the service option if you want Jira Software to autostart on bootup.

If you installed using sudo it’ll place the app in dir: /opt/atlassian/jira, which is good.

Shell
0
1
2
3
4
5
6
7
8
9
10
11
 
# Install directory
/opt/atlassian/jira
 
# Data directory
/var/atlassian/application-data/jira
 
# Use database settings
# Use the created username password we created earlier
jirasoftware_user
supersecretpassword
 

Note these files in case you don’t want it to be installed as a service or if something happens.

Shell
0
1
2
3
 
/opt/atlassian/jira/bin/start-jira.sh
/opt/atlassian/jira/bin/stop-jira.sh
 

Preparing a Server for Atlassian Software

Standard

I’ve been a user of Jira before the kanban “agile” interface and it was still one suite with an agile plugin. I use both Jira Core (“task”) and Jira Software (“agile”). I use Core for task based projects and Software for well software based projects.

I actually use Core for home projects like: my daughters 3rd birthday, remodeling the garage, gardening task like cloning my chili plant or gardenias. I have a project with multiple task and sub-task.

Given: I have an over powered media server running Ubuntu 18.04 and I have a need to track my personal task and projects. And I thought it would be a fun weekend project.

When: I install Atlassian’s Jira Software Server, Jira Core Server, and Confluence Server (I will be also installing Bamboo Server and BitBucket Server at a later time)

Then: I will have access to Jira for any task or project and I can track progress. I will have also learned how to manage and use Jira from another point of view. Not just moving features across and completing them.

Continue reading

Updating Ubuntu 17.10 to 18.04

Standard

If you need to upgrade your Ubuntu to a new release.

Shell
0
1
2
3
4
5
6
 
sudo apt update
sudo apt install update-manager-core # do what the prompt wants you to do like
# sudo apt autoremove
 
sudo do-release-upgrade
 

Wait a few minutes/hours, you should have upgraded your server.

New Rails Project On Clean System

Standard

Creating a new Rails project on a clean system.

Install iTerm2. *My terminal of choice.

Install Xcode and command line tools.

Install Homebrew.

Install OhMyZSH. *My preferred shell.

Install rvm or rbenv or docker.

Shell
0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
 
# Installing the ruby you want or need to use.
rvm list known
 
# Installing 2.5.1
rvm install 2.5.1 --default # sets 2.5.1 as your default ruby
 
# install rails
gem install rails bundler
 
## optional
# figure out how many cores are on your system
sysctl -n hw.ncpu
 
# number of cores and ram can effect workers
# mine returned 8 so i'll use 7 workers.
 
# global config for bundler to use 7 workers
bundle config --global jobs 7
 

I prefer to use rvm on my development system and/or docker, I moved over from vagrant and vmware fusion. I prefer rvm on my dev system over rbenv. But I prefer rbenv over rvm on production servers if i’m not using docker.

Using MyNewProject as the project name.

1. Create the working directory.

Shell
0
1
2
3
4
5
 
mkdir my_new_project && cd $_
 
# If using OhMyZSH
take my_new_project
 

2. Create the Rails app.

Shell
0
1
2
 
rails new .
 

3. Done

Use your favorite text editor. Mine is Sublime and Vi.