Uncategorized

Going Serverless: AWS Lambda, and S3

Standard
Reading Time: < 1 minute

Setting up a local development environment for a serverless AWS system. I will be setting up a local development environment for development.

Things needed:
NVM
Node
AWS CLI tool Version 1
Serverless Framework

Install AWS CLI Version 1:

The guide to install can be found here.

Install NVM:

curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.35.1/install.sh | bash

Install Node:

nvm install 10

nvm use 10

Install Serverless Framework:

npm install -g serverless

Create Project:

serverless create --template aws-nodejs --path projectName
cd projectName

Initialize NPM:

npm init

Node packages we’ll be using:

npm install aws-sdk --save-dev
npm install serverless-offline --save-dev
npm install serverless-http -save-dev
npm install serverless-s3-local --save-dev

AWS Profile (~/.aws/credentials):

[s3local]
aws_access_key_id = S3RVER
aws_secret_access_key = S3RVER

Using the profile:

export AWS_PROFILE=s3local

Testing and Using S3

Configure: (serverless.yml)

plugins:
  - serverless-s3-local
  - serverless-offline

custom:
  s3:
    port: 8081
    directory: ./tmp

resources:
  Resources:
    NewResources:
      Type: AWS::S3::Bucket
      Properties:
        BucketName: local-bucket

...

functions:
  s3hook:
    handler: handler.s3hook
    events:
      - s3: local-bucket

Edit handler.js:

...
module.exports.s3hook = async (event, context) => {
  console.log(JSON.stringify(event));
};

Start Serverless Offline: (try to use separate terminals)

sls offline start

Pushing up file to Local S3:

aws --endpoint http://localhost:8081 s3api put-object --bucket local-bucket --key handler.js --body ./handler.js

Retrieving a file from Local S3:

aws --endpoint http://localhost:8081 s3api get-object --bucket local-bucket --key handler.js ./tmp/handler.js

Results:

On your second terminal, the one you ran sls offline start you should have log outputs stating you have successful PUT and GET responses.

For the GET request, you should have a file in your tmp/ directory.

Upgrading Rails 5.2.2.1 to 6.0

Standard
Reading Time: < 1 minute

Update Gemfile

# update gem 'rails', '~> 5.2.2.1' to
gem 'rails', '~> 6.0.0'

# add
gem 'webpacker', '~> 4.0'

Grab updates & install webpack

bundle update

bin/rails webpacker:install

Run your test and fix any issues. I mostly got issues revolving around update_attributes and expanding where.not() queries.

I had to update a bunch of
@resource.update_attributes(resource_params)
to
@resource.update(resource_params).

I also had scopes that looked like:

scope :complete, -> { where.not(resume_id: nil, cover_letter_id: nil, company_id: nil) }

They had to be updated to:

scope :complete, -> { where.not(resume_id: nil).where.not(cover_letter_id: nil).where.not(company_id: nil) }

Also, I had an issue with one of my render partials.

Failure/Error: <%= render partial: :company_datum, collection: @company_data, locals: { offset: @company_data.offset }, cached: true %>

Had to update it to:

<%= render partial: 'company_datum', collection: @company_data, locals: { offset: @company_data.offset }, cached: true %>

Updating application.rb

# config/application.rb
# change config.load_defaults 5.2
config.load_defaults 6.0

# I removed
config.i18n.fallbacks = [I18n.default_locale]

If using rubocop: add this to your .rubocop.yml

AllCops:
  Exclude:
    - 'node_modules/**/*'
...

Fin.

Part 1: An API using Phoenix, Absinthe (GraphQL), Guardian, React, Apollo Client

Standard
Reading Time: 2 minutes

I love my Rails, but this is something I am really interested in.

It all started with a little bookkeeping app written using Phoenix as an API and React as a view layer. Has now turned into an extremely deep rabbit hole. After completing the first features of the app, I had the question “These states in these components are getting out of hand, I wonder if there’s something to help me?” This lead to Redux, one object to hold the state of the entire app. Then lead to the question, “Is there a better way to handle the data request to the API?” This lead to GraphQL which lead to Absinthe which made sense since I am using Phoenix as my API.

I think the best way to learn is to build something simple but not trivial. So let us begin with an e-commerce store that sales widgets.

Users >> Carts >> Widgets

Users can have carts to place widgets in them, etc.

I am going to break this down to 3 to 4 articles:
Part 1: Introduction to Phoenix, Absinthe, and Guardian – explaining the pieces.

Part 2: Implementation of the Phoenix/Absinthe API.
This is how you go about creating the API.

Part 3: Implementation of React/Redux with Apollo client.
We will create the frontend while still inside Phoenix, no additional steps Webpack will handle it.

Part 4: Additional nice things to have like: Elasticsearch, a Pub/Sub feature, etc. This one is a maybe.

Code can be found here.

https://github.com/mfung/widgets_r_us

What is Phoenix?

Phoenix is a web development framework similar in vein to Ruby on Rails. Phoenix is written in Elixir. Elixir is a functional, concurrent, and general-purpose programming language. Phoenix follows the MVC pattern and was developed to create highly performant and scalable web applications.

Laymen terms: You use it to build websites fast and efficiently.

What is GraphQL?

GraphQL was developed by Facebook in 2012 and released to the public in 2015. It is a data query and manipulation language for APIs.

Simply: An efficient and flexible way to get data, by eliminating unwanted data. I define what data I need from the endpoint.

What is Absinthe GraphQL?

Absinthe is “The GraphQL toolkit for Elixir.” Since Phoenix is written in Elixir and most APIs are web APIs. This just makes sense.

Guardian

If you are familiar with Devise in the Ruby world Guardian is similar. It is an authenication library for Elixir applications.

React

React is a JavaScript library used to build user interfaces for single-page applications and is optimal for fetching rapidly changing data.

Apollo GraphQL (Client)

Apollo comes in server and client. We will be dealing with the client. Apollo Client is a state management library for JavaScript GraphQL apps.

Next

docker-credential-pass Ubuntu setup for Bamboo CI/CD

Standard
Reading Time: < 1 minute

Ubuntu: 18.04
Docker: 18.09.5
docker-credential-pass: 0.6.2
pass: 1.7.1
gpg2: 2.2.4

Since my setup is for CI/CD I have a bamboo user to deploy my docker containers to Docker Hub and deploy to my production servers. Your use case may vary, but if you want your credentials to persist longer than a couple of hours before your system starts asking you to re-enter your passphrase this might work for you.

0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
 
# First install apps we'll need
sudo apt upgrade && apt install pass gnupg2 tmux
 
# Download docker-credential-pass
wget https://github.com/docker/docker-credential-helpers/releases/download/v0.6.2/docker-credential-pass-v0.6.2-amd64.tar.gz
 
# Unpack
tar xvf docker-credential-pass-v0.6.2-amd64.tar.gz
 
# Move into /usr/bin
sudo mv docker-credential-pass /usr/bin
 
# Check to see if it's working should return {}
docker-credential-pass list
 
# Here is where my setup will diverge
# I login to my bamboo user
su - [username]
 
# I run tmux
tmux
 
vi ~/.gnupg/gpg-agent.conf
 

My gpg-agent.conf file looks like this:

0
1
2
3
 
default-cache-ttl 315569520
max-cache-ttl 315569520
 
0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
 
# Restart gpg-agent
echo RELOADAGENT | gpg-connect-agent
 
# Generate key
gpg2 --full-generate-key
 
# You'll get an output with a key, copy it to use
pass init DXXXXXXXXXXXXXX6E1AFXXXXXXXXXXB9XXXX497
 
# Enter a password
pass insert docker-credential-helpers/docker-pass-initialized-check
 
# You should get a return of {} instead of 'pass store is uninitialized'
docker-credential-pass list
 
# login to docker
docker login
 
# Check to see if it is stored
docker-credential-pass list
 
# should return something like
{"https://index.docker.io/v1/":"[username]"}
 

This worked for me. Also note the default-cache-ttl value is in seconds, the values I used is 10 years in seconds.

Getting SimpleCov-summary (0.0.5) to work with SimpleCov (0.16.1) and Rails 5.2

Standard

Reading Time: < 1 minuteIf you run into issues with installing simplecov-summary using:

Ruby
0
1
2
3
4
5
6
7
8
9
10
11
12
13
 
# Put this in for RSpec, Minitest will plug in automatically
  RSpec.configure do |config|
    config.after(:suite) do
      if SimpleCov.running
        silence_stream(STDOUT) do
          SimpleCov::Formatter::HTMLFormatter.new.format(SimpleCov.result)
        end
 
        SimpleCov::Formatter::SummaryFormatter.new.format(SimpleCov.result)
      end
    end
  end
 

Here’s how to fix it:

Ruby
0
1
2
3
4
5
6
7
8
9
10
 
# Put this in for RSpec, Minitest will plug in automatically
  RSpec.configure do |config|
    config.after(:suite) do
      SimpleCov.at_exit do
        SimpleCov::Formatter::SummaryFormatter.new.format(SimpleCov.result)
        SimpleCov::Formatter::HTMLFormatter.new.format(SimpleCov.result)
      end
    end
  end
 

Since Rails 5 removed silence_stream because it is not thread-safe. I found implementations of silence_stream but I got it to work by just removing it completely and I prefer to see these outputs at the end of my test run.

ADD NEW USER CENTOS, ADD TO SUDOER, ADD RSA KEYS

Standard

Reading Time: < 1 minute

Shell
0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
 
#as root
sudo su -
 
adduser username
 
visudo
# search for root
/root
# esc keypress n
## Allow root to run any commands anywhere
root ALL=(ALL) ALL
username ALL=(ALL) NOPASSWD:ALL
 
# OR
visudo username ALL=(ALL) NOPASSWD:ALL
 
#exit
 
# su – to setup right user environment
su – username
 
ssh-keygen -t rsa -b 4096
# enter enter enter
 
cd ~/.ssh
touch authorized_keys
chmod 0700 authorized_keys
vi authorized_keys
 
# get the pub keys from your development computer
# less ~/.ssh/id_rsa.pub
# paste into authorized_keys
 
# from your dev computer
# ssh username@url
 

 

Rsync Remote

Standard

Reading Time: < 1 minute

Shell
0
1
2
3
4
 
rsync -a –progress -e ssh root@url:~/filename.sh .
 
rsync -a database.yml -e ssh qrsmart_user@url:~/
 

 

POSTGRESQL

Standard

Reading Time: < 1 minute

Shell
0
1
2
3
4
5
6
7
8
9
10
11
12
 
# shows system tables
postgres=# dS
 
# list databases
postgres=# l
 
# list tables
postgres=# d
 
# list users
postgres=# SELECT * FROM pg_user;
 

 

PALINDROME

Standard

Reading Time: < 1 minute

Ruby:

Ruby
0
1
2
3
4
5
6
7
8
9
 
class String
  def palindrome?
    self.downcase == self.downcase.reverse
  end
end
p "Bob".palindrome?
p "andy".palindrome?
 

Python:

Python
0
1
2
3
4
5
6
7
8
9
10
11
12
13
 
class myString(str):
def __init__ (self, value):
self.value = value
def is_palindrome(self):
return self.lower() == self[::-1].lower()
bob = myString("Bob")
andy = myString("andy")
print bob.is_palindrome()
print andy.is_palindrome()