Going Serverless: AWS Lambda, and S3

Standard
Reading Time: < 1 minute

Setting up a local development environment for a serverless AWS system. I will be setting up a local development environment for development.

Things needed:
NVM
Node
AWS CLI tool Version 1
Serverless Framework

Install AWS CLI Version 1:

The guide to install can be found here.

Install NVM:

curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.35.1/install.sh | bash

Install Node:

nvm install 10

nvm use 10

Install Serverless Framework:

npm install -g serverless

Create Project:

serverless create --template aws-nodejs --path projectName
cd projectName

Initialize NPM:

npm init

Node packages we’ll be using:

npm install aws-sdk --save-dev
npm install serverless-offline --save-dev
npm install serverless-http -save-dev
npm install serverless-s3-local --save-dev

AWS Profile (~/.aws/credentials):

[s3local]
aws_access_key_id = S3RVER
aws_secret_access_key = S3RVER

Using the profile:

export AWS_PROFILE=s3local

Testing and Using S3

Configure: (serverless.yml)

plugins:
  - serverless-s3-local
  - serverless-offline

custom:
  s3:
    port: 8081
    directory: ./tmp

resources:
  Resources:
    NewResources:
      Type: AWS::S3::Bucket
      Properties:
        BucketName: local-bucket

...

functions:
  s3hook:
    handler: handler.s3hook
    events:
      - s3: local-bucket

Edit handler.js:

...
module.exports.s3hook = async (event, context) => {
  console.log(JSON.stringify(event));
};

Start Serverless Offline: (try to use separate terminals)

sls offline start

Pushing up file to Local S3:

aws --endpoint http://localhost:8081 s3api put-object --bucket local-bucket --key handler.js --body ./handler.js

Retrieving a file from Local S3:

aws --endpoint http://localhost:8081 s3api get-object --bucket local-bucket --key handler.js ./tmp/handler.js

Results:

On your second terminal, the one you ran sls offline start you should have log outputs stating you have successful PUT and GET responses.

For the GET request, you should have a file in your tmp/ directory.

Upgrading Rails 5.2.2.1 to 6.0

Standard
Reading Time: < 1 minute

Update Gemfile

# update gem 'rails', '~> 5.2.2.1' to
gem 'rails', '~> 6.0.0'

# add
gem 'webpacker', '~> 4.0'

Grab updates & install webpack

bundle update

bin/rails webpacker:install

Run your test and fix any issues. I mostly got issues revolving around update_attributes and expanding where.not() queries.

I had to update a bunch of
@resource.update_attributes(resource_params)
to
@resource.update(resource_params).

I also had scopes that looked like:

scope :complete, -> { where.not(resume_id: nil, cover_letter_id: nil, company_id: nil) }

They had to be updated to:

scope :complete, -> { where.not(resume_id: nil).where.not(cover_letter_id: nil).where.not(company_id: nil) }

Also, I had an issue with one of my render partials.

Failure/Error: <%= render partial: :company_datum, collection: @company_data, locals: { offset: @company_data.offset }, cached: true %>

Had to update it to:

<%= render partial: 'company_datum', collection: @company_data, locals: { offset: @company_data.offset }, cached: true %>

Updating application.rb

# config/application.rb
# change config.load_defaults 5.2
config.load_defaults 6.0

# I removed
config.i18n.fallbacks = [I18n.default_locale]

If using rubocop: add this to your .rubocop.yml

AllCops:
  Exclude:
    - 'node_modules/**/*'
...

Fin.

How to Use Postgres to Calculate and Save Rankings

Standard
Reading Time: 2 minutes

When you need to calculate rank and save it to a field in Postgres. This example uses Rails ActiveRecord but if you can execute the query you are good.

A naive way would be to build an iterative method with an index. An example in Ruby:

Product.order(price: :desc).each_with_index do |product, index|
  product.price_rank = index + 1
  product.save
end

This might work great if you have a limit(10) or have a small data sample. But what if you have 100,000 records or even worse 1 million records.

For 100 items the above code took 0.181s.
For 1,000 items .49s.
For 100,000 items 155.04s that’s roughly 2 minutes 36 seconds.

Now using Postgres I know it’ll be faster but how much?

query = "UPDATE products SET price_rank = r.rnk FROM (SELECT id, RANK() OVER (ORDER BY price DESC) as rnk FROM products) r WHERE products.id = r.id"

ActiveRecord::Base.connection.execute(query)

I am getting run times of .0005 because I am dumping the work onto the database.

Running EXPLAIN ANALYZE on the query.

 Update on products  (cost=16813.40..25200.70 rows=101002 width=112) (actual time=15247.855..15247.855 rows=0 loops=1)
   ->  Hash Join  (cost=16813.40..25200.70 rows=101002 width=112) (actual time=8693.946..12299.080 rows=101002 loops=1)
         Hash Cond: (products.id = r.id)
         ->  Seq Scan on products  (cost=0.00..3391.02 rows=101002 width=68) (actual time=0.584..990.371 rows=101002 loops=1)
         ->  Hash  (cost=14563.87..14563.87 rows=101002 width=56) (actual time=8693.010..8693.010 rows=101002 loops=1)
               Buckets: 65536  Batches: 4  Memory Usage: 2685kB
               ->  Subquery Scan on r  (cost=11786.32..14563.87 rows=101002 width=56) (actual time=2278.120..7232.235 rows=101002 loops=1)
                     ->  WindowAgg  (cost=11786.32..13553.85 rows=101002 width=24) (actual time=2278.083..5283.472 rows=101002 loops=1)
                           ->  Sort  (cost=11786.32..12038.82 rows=101002 width=16) (actual time=2278.053..3262.086 rows=101002 loops=1)
                                 Sort Key: products_1.price DESC
                                 Sort Method: external merge  Disk: 2576kB
                                 ->  Seq Scan on products products_1  (cost=0.00..3391.02 rows=101002 width=16) (actual time=0.639..992.346 rows=101002 loops=1)
 Planning time: 3.271 ms
 Execution time: 15252.513 ms

Over 100,000 records ranked and updated with values in 15s. Not bad.

You also get the added logic of a true ranking when there is a tie like #1, #2, #2, #2, #5, #6, etc. The calculated version was simple and did not account for it. If you do want to, you can implement it but with Ruby, you’ll add more bloat and complexity.

Is Rails Scalable?

Standard
Reading Time: 7 minutes

Being an optimist I would say, Yes! Having worked on a large scale high visibility site I’ve personally seen it happen, so definitely Yes! When the budget was not a limiting factor, I saw vertical and horizontal scaling at it’s best. Get the biggest badest app server, then multiply that by 20 with multiple load balancers. CDN cache everything possible. Scaling solved. Millions of views, thousands of transactions per second no problem. The databases were actually starting to buckle from all the connections. Enter pgBouncer connection pooling and more load balancers, but that is the point. You can scale to the point where it isn’t a problem with code but infrastructure.

But what if you can’t. Here are my experiences.

Vertical Scaling

You add more RAM, CPUs, faster pipes (fiber optics directly to another server or network), SSDs. This is all possible, as CPUs reach their limits this gets to be a limiting factor. This has a threshold. With more users, the server can still meltdown. This is not for an exponent growth more of a linear one.

This also holds true for the database server, as most issues are database related. Slow queries can run faster by beefing up the hardware it runs on.

I consider this a quick fix for linear or predictable growth.

Horizontal Scaling

Load Balancing! Get more servers to do the job. Two or twenty is better than one. Share the load between servers. This can and will get costly. Be it cloud or metal or a combination of the two. Horizontal scaling if done correctly can be a really easy solution to scaling issues.

Continue reading

Part 1: An API using Phoenix, Absinthe (GraphQL), Guardian, React, Apollo Client

Standard
Reading Time: 2 minutes

I love my Rails, but this is something I am really interested in.

It all started with a little bookkeeping app written using Phoenix as an API and React as a view layer. Has now turned into an extremely deep rabbit hole. After completing the first features of the app, I had the question “These states in these components are getting out of hand, I wonder if there’s something to help me?” This lead to Redux, one object to hold the state of the entire app. Then lead to the question, “Is there a better way to handle the data request to the API?” This lead to GraphQL which lead to Absinthe which made sense since I am using Phoenix as my API.

I think the best way to learn is to build something simple but not trivial. So let us begin with an e-commerce store that sales widgets.

Users >> Carts >> Widgets

Users can have carts to place widgets in them, etc.

I am going to break this down to 3 to 4 articles:
Part 1: Introduction to Phoenix, Absinthe, and Guardian – explaining the pieces.

Part 2: Implementation of the Phoenix/Absinthe API.
This is how you go about creating the API.

Part 3: Implementation of React/Redux with Apollo client.
We will create the frontend while still inside Phoenix, no additional steps Webpack will handle it.

Part 4: Additional nice things to have like: Elasticsearch, a Pub/Sub feature, etc. This one is a maybe.

Code can be found here.

https://github.com/mfung/widgets_r_us

What is Phoenix?

Phoenix is a web development framework similar in vein to Ruby on Rails. Phoenix is written in Elixir. Elixir is a functional, concurrent, and general-purpose programming language. Phoenix follows the MVC pattern and was developed to create highly performant and scalable web applications.

Laymen terms: You use it to build websites fast and efficiently.

What is GraphQL?

GraphQL was developed by Facebook in 2012 and released to the public in 2015. It is a data query and manipulation language for APIs.

Simply: An efficient and flexible way to get data, by eliminating unwanted data. I define what data I need from the endpoint.

What is Absinthe GraphQL?

Absinthe is “The GraphQL toolkit for Elixir.” Since Phoenix is written in Elixir and most APIs are web APIs. This just makes sense.

Guardian

If you are familiar with Devise in the Ruby world Guardian is similar. It is an authenication library for Elixir applications.

React

React is a JavaScript library used to build user interfaces for single-page applications and is optimal for fetching rapidly changing data.

Apollo GraphQL (Client)

Apollo comes in server and client. We will be dealing with the client. Apollo Client is a state management library for JavaScript GraphQL apps.

Next

docker-credential-pass Ubuntu setup for Bamboo CI/CD

Standard
Reading Time: < 1 minute

Ubuntu: 18.04
Docker: 18.09.5
docker-credential-pass: 0.6.2
pass: 1.7.1
gpg2: 2.2.4

Since my setup is for CI/CD I have a bamboo user to deploy my docker containers to Docker Hub and deploy to my production servers. Your use case may vary, but if you want your credentials to persist longer than a couple of hours before your system starts asking you to re-enter your passphrase this might work for you.

0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
 
# First install apps we'll need
sudo apt upgrade && apt install pass gnupg2 tmux
 
# Download docker-credential-pass
wget https://github.com/docker/docker-credential-helpers/releases/download/v0.6.2/docker-credential-pass-v0.6.2-amd64.tar.gz
 
# Unpack
tar xvf docker-credential-pass-v0.6.2-amd64.tar.gz
 
# Move into /usr/bin
sudo mv docker-credential-pass /usr/bin
 
# Check to see if it's working should return {}
docker-credential-pass list
 
# Here is where my setup will diverge
# I login to my bamboo user
su - [username]
 
# I run tmux
tmux
 
vi ~/.gnupg/gpg-agent.conf
 

My gpg-agent.conf file looks like this:

0
1
2
3
 
default-cache-ttl 315569520
max-cache-ttl 315569520
 
0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
 
# Restart gpg-agent
echo RELOADAGENT | gpg-connect-agent
 
# Generate key
gpg2 --full-generate-key
 
# You'll get an output with a key, copy it to use
pass init DXXXXXXXXXXXXXX6E1AFXXXXXXXXXXB9XXXX497
 
# Enter a password
pass insert docker-credential-helpers/docker-pass-initialized-check
 
# You should get a return of {} instead of 'pass store is uninitialized'
docker-credential-pass list
 
# login to docker
docker login
 
# Check to see if it is stored
docker-credential-pass list
 
# should return something like
{"https://index.docker.io/v1/":"[username]"}
 

This worked for me. Also note the default-cache-ttl value is in seconds, the values I used is 10 years in seconds.

Getting SimpleCov-summary (0.0.5) to work with SimpleCov (0.16.1) and Rails 5.2

Standard

Reading Time: < 1 minuteIf you run into issues with installing simplecov-summary using:

Ruby
0
1
2
3
4
5
6
7
8
9
10
11
12
13
 
# Put this in for RSpec, Minitest will plug in automatically
  RSpec.configure do |config|
    config.after(:suite) do
      if SimpleCov.running
        silence_stream(STDOUT) do
          SimpleCov::Formatter::HTMLFormatter.new.format(SimpleCov.result)
        end
 
        SimpleCov::Formatter::SummaryFormatter.new.format(SimpleCov.result)
      end
    end
  end
 

Here’s how to fix it:

Ruby
0
1
2
3
4
5
6
7
8
9
10
 
# Put this in for RSpec, Minitest will plug in automatically
  RSpec.configure do |config|
    config.after(:suite) do
      SimpleCov.at_exit do
        SimpleCov::Formatter::SummaryFormatter.new.format(SimpleCov.result)
        SimpleCov::Formatter::HTMLFormatter.new.format(SimpleCov.result)
      end
    end
  end
 

Since Rails 5 removed silence_stream because it is not thread-safe. I found implementations of silence_stream but I got it to work by just removing it completely and I prefer to see these outputs at the end of my test run.

Rails 5.2 Has And Belongs To Many with UUID

Standard

Reading Time: < 1 minuteWhen using UUIDs and you are creating a join table like so:

Shell
0
1
2
 
rails g migration CreateJoinTableUserCompany users companies
 

You will get a migration file like so:

Ruby
0
1
2
3
4
5
6
7
8
9
 
class CreateJoinTableUserCompany < ActiveRecord::Migration[5.2]
  def change
    create_join_table :users, :companies do |t|
      #t.index [:user_id, :company_id]
      #t.index [:company_id, :user_id]
    end
  end
end
 

You’ll need to modify it accordingly:

Ruby
0
1
2
3
4
5
6
7
8
9
 
class CreateJoinTableUserCompany < ActiveRecord::Migration[5.2]
  def change
    create_join_table :users, :companies, column_options: { type: :uuid } do |t|
      t.index [:user_id, :company_id]
      t.index [:company_id, :user_id]
    end
  end
end
 

Then run:

Shell
0
1
2
 
rails db:migrate
 

And you should be good. Just remember if you are using UUIDs you might want to make sure all models you generate use UUIDs.

In config/application.rb:

Ruby
0
1
2
3
4
 
config.generators do |g|
  g.orm :active_record, primary_key_type: :uuid
end
 

And make sure your generated migrations have:

Ruby
0
1
2
3
4
5
6
7
8
9
 
class CreateCompanies < ActiveRecord::Migration[5.2]
  def change
    create_table :companies, id: :uuid do |t|
      t.string :name, limit: 80
      ...
    end
  end
end
 

Enjoy!

Installing Jira Software Server

Standard

Reading Time: < 1 minuteAssuming you have downloaded from Atlassian. If not read how to prepare for installs.

Shell
0
1
2
3
4
 
chmod +x atlassian-jira-software-7.12.1-x64.bin
 
sudo ./atlassian-jira-software-7.12.1-x64.bin
 

Follow prompts and make sure you enable the service option if you want Jira Software to autostart on bootup.

If you installed using sudo it’ll place the app in dir: /opt/atlassian/jira, which is good.

Shell
0
1
2
3
4
5
6
7
8
9
10
11
 
# Install directory
/opt/atlassian/jira
 
# Data directory
/var/atlassian/application-data/jira
 
# Use database settings
# Use the created username password we created earlier
jirasoftware_user
supersecretpassword
 

Note these files in case you don’t want it to be installed as a service or if something happens.

Shell
0
1
2
3
 
/opt/atlassian/jira/bin/start-jira.sh
/opt/atlassian/jira/bin/stop-jira.sh
 

Preparing a Server for Atlassian Software

Standard

Reading Time: 2 minutes

I’ve been a user of Jira before the kanban “agile” interface and it was still one suite with an agile plugin. I use both Jira Core (“task”) and Jira Software (“agile”). I use Core for task based projects and Software for well software based projects.

I actually use Core for home projects like: my daughters 3rd birthday, remodeling the garage, gardening task like cloning my chili plant or gardenias. I have a project with multiple task and sub-task.

Given: I have an over powered media server running Ubuntu 18.04 and I have a need to track my personal task and projects. And I thought it would be a fun weekend project.

When: I install Atlassian’s Jira Software Server, Jira Core Server, and Confluence Server (I will be also installing Bamboo Server and BitBucket Server at a later time)

Then: I will have access to Jira for any task or project and I can track progress. I will have also learned how to manage and use Jira from another point of view. Not just moving features across and completing them.

Continue reading