Diamonds For Nothing a parody for Minecrafters

Now look at them diggy-diggies that's the way you do it
You play the pick-axe on the MC-YT
That ain't diggin' that's the way you do it
Diamonds for nothin' and chickens for free
Now that ain't diggin' that's the way you do it
Lemme tell ya them miners ain't dumb
Maybe get a arrow on your right arm
Maybe get a arrow on the back of your head

We gotta install these coal-powered ovens
Custom jukebox deliveries
We gotta move these iron bars
We gotta move these chestie's

See the little noob with the skin and the hat
Yeah buddy that's his own hat
That little miner got his own jet pack
That little miner he's a millionaire

We gotta install these coal-powered ovens
Custom jukebox deliveries
We gotta move these iron bars
We gotta move these chestie's

I shoulda learned to play the game
I shoulda learned to play them picks
Look at that miner, she got it stickin' in the camera
Man we could have some fun
And he's up there, what's that? Indie noises?
Bangin' on the creepers like a simoneeze
That ain't workin' that's the way you do it
Diamonds for nothin' and chickens for free

We gotta install these coal-powered ovens
Custom jukebox deliveries
We gotta move these iron bars
We gotta move these chestie's

Now that ain't diggin' that's the way you do it
You play the pick on the MC-YT
That ain't diggin' that's the way you do it
Diamonds for nothin' and chickens for free
Diamonds for nothin' and chickens for free

I want my
I want my
I want my MC-YT!

Docker: Creating ruby 2.2.0, mysql and rails images

For the purposes of this post, I'm assuming you have a private repository with the address 10.0.0.1:5000.

In this example, you will notice a distinct similarity in the build scripts.
Hmm.
Funny that.
I created a tool sleet which is like fleet but for single server installs.
So that is the essence of those build scripts.
If I have time somewhere in my aging schedule and downright ludicrous deadlines, I'll open source it.

This creates 4 images in your private repo.

- debian which is an instance of jessie with various compilers etc installed.
- ruby-2.2.0 compiled from source
- mysql-ruby-2.2.0 with the build dependencies
- rails-2.2.0 from all of the above

The purpose was to get an image that can be used for `rails-api` micro services that builds and deploys ultra-fast.
This is done by pre-installing the most commonly used gems into the `rails-2.2.0` image.

Then when a build is done that inherits from the `rails-2.2.0` image, the `bundle install` simply uses the local gems and doesn't have to go off to any external gem sources.
The bundle output shows `Using` instead of `Installing` the latter of which involves snarfing gems off the web and purhaps compiling things.


First up, here's the tree you'll be creating:

.
├── debian
│   └── jessie
│       ├── Dockerfile
│       └── build.sh
├── mysql
│   └── 2.2.0
│       ├── Dockerfile
│       └── build.sh
├── rails
│   └── 2.2.0
│       ├── Dockerfile
│       ├── build
│       │   └── Gemfile
│       └── build.sh
└── ruby
    └── 2.2.0
        ├── Dockerfile
        └── build.sh

Here are the files:

debian/jessie/Dockerfile

FROM debian:jessie

MAINTAINER Your Name 

ENV REFRESHED_AT 2015-01-08

RUN apt-get update \
  && apt-get install -y --no-install-recommends \
    git curl procps \
    autoconf build-essential \
    libbz2-dev libcurl4-openssl-dev libffi-dev libssl-dev libreadline-dev libyaml-dev \
    zlib1g-dev --no-install-recommends \
  && rm -rf /var/lib/apt/lists/*

debian/jessie/build.sh

#!/bin/sh

IMAGE=debian

export DOCKER_HOST=tcp://192.168.59.103:2376
export DOCKER_TLS_VERIFY=1
export DOCKER_CERT_PATH=${HOME}/.boot2docker/certs/boot2docker-vm
IMAGE_REPO=10.0.0.1:5000

echo "------------------------------------------------------------------------"
echo " ENVIRONMENT"
echo "      DOCKER_HOST     ${DOCKER_HOST}"
echo "      IMAGE           ${IMAGE}"
echo "========================================================================"
echo ""

echo "------------------------------------------------------------------------"
echo " ENVIRONMENT"
echo "      DOCKER_HOST     ${DOCKER_HOST}"
echo "      IMAGE           ${IMAGE}"
echo "========================================================================"
echo ""

echo "------------------------------------------------------------------------"
echo " DELETING OLD IMAGES"
docker rmi ${IMAGE_REPO}/images/${IMAGE} ${IMAGE}
echo "========================================================================"
echo ""

set -e

echo "------------------------------------------------------------------------"
echo "BUILD"
docker build -t="${IMAGE}" .
echo "========================================================================"
echo ""

echo "------------------------------------------------------------------------"
echo "IMAGES"
docker images | egrep "^${IMAGE}"
echo "========================================================================"
echo ""

echo "------------------------------------------------------------------------"
echo "HISTORY"
docker history ${IMAGE}
echo "========================================================================"
echo ""

echo "------------------------------------------------------------------------"
echo " TAG AND PUSH"
docker images | grep "^${IMAGE}"
sha=`docker images | grep "^${IMAGE}" | awk '{ print $3; }' | cut -f1`
echo "sha=${sha}"
docker tag ${sha} ${IMAGE_REPO}/images/${IMAGE}
docker push       ${IMAGE_REPO}/images/${IMAGE}
echo "========================================================================"
echo ""

ruby/2.2.0/Dockerfile

FROM 10.0.0.1:5000/images/debian

MAINTAINER Your Name 

ENV REFRESHED_AT 2015-01-08

ENV RUBY_MAJOR 2.2
ENV RUBY_VERSION 2.2.0

RUN apt-get update \
  && apt-get install -y bison ruby \
  && rm -rf /var/lib/apt/lists/* \
  && mkdir -p /usr/src/ruby \
  && curl -SL "http://cache.ruby-lang.org/pub/ruby/$RUBY_MAJOR/ruby-$RUBY_VERSION.tar.bz2" | tar -xjC /usr/src/ruby --strip-components=1 \
  && cd /usr/src/ruby \
  && autoconf \
  && ./configure --disable-install-doc \
  && make -j"$(nproc)" \
  && apt-get purge -y --auto-remove bison ruby \
  && make install \
  && rm -r /usr/src/ruby

# skip installing gem documentation
RUN echo 'gem: --no-rdoc --no-ri' >> "$HOME/.gemrc"

# install things globally, for great justice
ENV GEM_HOME /usr/local/bundle
ENV PATH $GEM_HOME/bin:$PATH
RUN gem install bundler \
  && bundle config --global path "$GEM_HOME" \
  && bundle config --global bin "$GEM_HOME/bin"

# don't create ".bundle" in all our apps
ENV BUNDLE_APP_CONFIG $GEM_HOME

CMD [ "irb" ]

ruby/2.2.0/build.sh

#!/bin/bash

IMAGE=ruby-2.2.0

export DOCKER_HOST=tcp://192.168.59.103:2376
export DOCKER_TLS_VERIFY=1
export DOCKER_CERT_PATH=${HOME}/.boot2docker/certs/boot2docker-vm
IMAGE_REPO=10.0.0.1:5000

echo "------------------------------------------------------------------------"
echo " ENVIRONMENT"
echo "      DOCKER_HOST     ${DOCKER_HOST}"
echo "      IMAGE           ${IMAGE}"
echo "========================================================================"
echo ""

echo "------------------------------------------------------------------------"
echo " DELETING OLD IMAGES"
docker rmi ${IMAGE_REPO}/images/${IMAGE} ${IMAGE}
echo "========================================================================"
echo ""

set -e

echo "------------------------------------------------------------------------"
echo " BUILD"
docker build -t="${IMAGE}" .
echo "========================================================================"
echo ""

echo "------------------------------------------------------------------------"
echo "IMAGES"
docker images | egrep "^${IMAGE}"
echo "========================================================================"
echo ""

echo "------------------------------------------------------------------------"
echo "HISTORY"
docker history ${IMAGE}
echo "========================================================================"
echo ""

echo "------------------------------------------------------------------------"
echo " TAG AND PUSH"
docker images | grep "^${IMAGE}"
sha=`docker images | grep "^${IMAGE}" | awk '{ print $3; }' | cut -f1`
echo "sha=${sha}"
docker tag ${sha} ${IMAGE_REPO}/images/${IMAGE}
docker push       ${IMAGE_REPO}/images/${IMAGE}
echo "========================================================================"
echo ""

mysql/2.2.0/Dockerfile

FROM 10.0.0.1:5000/images/ruby-2.2.0

MAINTAINER Your Name 

ENV REFRESHED_AT 2015-01-08

ENV RUBY_MAJOR 2.2
ENV RUBY_VERSION 2.2.0

RUN buildDeps='libmysqlclient-dev'; \
      set -x \
      && apt-get update && apt-get install -y $buildDeps --no-install-recommends

CMD [ "irb" ]

mysql/2.2.0/build.sh

#!/bin/bash

IMAGE=ruby-2.2.0

export DOCKER_HOST=tcp://192.168.59.103:2376
export DOCKER_TLS_VERIFY=1
export DOCKER_CERT_PATH=${HOME}/.boot2docker/certs/boot2docker-vm
IMAGE_REPO=10.0.0.1:5000

echo "------------------------------------------------------------------------"
echo " ENVIRONMENT"
echo "      DOCKER_HOST     ${DOCKER_HOST}"
echo "      IMAGE           ${IMAGE}"
echo "========================================================================"
echo ""

echo "------------------------------------------------------------------------"
echo " DELETING OLD IMAGES"
docker rmi ${IMAGE_REPO}/images/${IMAGE} ${IMAGE}
echo "========================================================================"
echo ""

set -e

echo "------------------------------------------------------------------------"
echo " BUILD"
docker build -t="${IMAGE}" .
echo "========================================================================"
echo ""

echo "------------------------------------------------------------------------"
echo "IMAGES"
docker images | egrep "^${IMAGE}"
echo "========================================================================"
echo ""

echo "------------------------------------------------------------------------"
echo "HISTORY"
docker history ${IMAGE}
echo "========================================================================"
echo ""

echo "------------------------------------------------------------------------"
echo " TAG AND PUSH"
docker images | grep "^${IMAGE}"
sha=`docker images | grep "^${IMAGE}" | awk '{ print $3; }' | cut -f1`
echo "sha=${sha}"
docker tag ${sha} ${IMAGE_REPO}/images/${IMAGE}
docker push       ${IMAGE_REPO}/images/${IMAGE}
echo "========================================================================"
echo ""

rails/2.2.0/Dockerfile

FROM 10.0.0.1:5000/images/debian-mysql-ruby-2.2.0

MAINTAINER Your Name 

ENV REFRESHED_AT 2015-01-08

ENV HOME /build
ENV RUBY_MAJOR 2.2
ENV RUBY_VERSION 2.2.0
ENV GEM_HOME /usr/local/bundle
ENV PATH $GEM_HOME/bin:$PATH

ADD ./build /build

WORKDIR /build

RUN bundle install --jobs 8

CMD [ "irb" ]
rails/2.2.0/build/Gemfile

source 'http://your.private.gem.server:9900'

# ---------------------------------------------------------------------------
# Always these
# ---------------------------------------------------------------------------
gem 'rails', '4.2.0'
gem 'mysql2'

gem 'thin'
gem 'foreman'

gem 'jsonapi-resources' # and/or roar

gem 'db_populate', git: 'https://github.com/ffmike/db-populate.git'

gem 'rack-cors'
gem 'activeresource'

gem 'jbuilder', '~> 2.0'
gem 'typhoeus'
gem 'etcd'
gem 'hutch'
gem 'elasticsearch'
gem 'searchkick'

gem 'therubyracer'
gem 'oj'

# ---------------------------------------------------------------------------
# Unused
# ---------------------------------------------------------------------------
# gem 'jquery-rails'
# gem 'turbolinks'
# gem 'bcrypt', '~> 3.1.7' # Use ActiveModel has_secure_password
# gem 'sass-rails', '~> 4.0.3'
# gem 'uglifier', '>= 1.3.0'
# gem 'coffee-rails', '~> 4.0.0'

rails/2.2.0/build.sh

#!/bin/bash

IMAGE=rails-2.2.0

export DOCKER_HOST=tcp://192.168.59.103:2376
export DOCKER_TLS_VERIFY=1
export DOCKER_CERT_PATH=${HOME}/.boot2docker/certs/boot2docker-vm
IMAGE_REPO=10.0.0.1:5000

echo "------------------------------------------------------------------------"
echo " ENVIRONMENT"
echo "      DOCKER_HOST     ${DOCKER_HOST}"
echo "      IMAGE           ${IMAGE}"
echo "========================================================================"
echo ""

echo "------------------------------------------------------------------------"
echo " DELETING OLD IMAGES"
docker rmi ${IMAGE_REPO}/images/${IMAGE} ${IMAGE}
echo "========================================================================"
echo ""

set -e

echo "------------------------------------------------------------------------"
echo " BUILD"
docker build -t="${IMAGE}" .
echo "========================================================================"
echo ""

echo "------------------------------------------------------------------------"
echo "IMAGES"
docker images | egrep "^${IMAGE}"
echo "========================================================================"
echo ""

echo "------------------------------------------------------------------------"
echo "HISTORY"
docker history ${IMAGE}
echo "========================================================================"
echo ""

echo "------------------------------------------------------------------------"
echo " TAG AND PUSH"
docker images | grep "^${IMAGE}"
sha=`docker images | grep "^${IMAGE}" | awk '{ print $3; }' | cut -f1`
echo "sha=${sha}"
docker tag ${sha} ${IMAGE_REPO}/images/${IMAGE}
docker push       ${IMAGE_REPO}/images/${IMAGE}
echo "========================================================================"
echo ""

Enjoy.


YMMV.

Docker: Run/Administer an instance of elasticsearch on your boot2docker vm

First ssh into your boot2docker vm, then run these commands:

docker pull dockerfile/elasticsearch
mkdir elastic_search
cat > elastic_search/elasticsearch.yml <<-EOF
path:
  logs: /data/log
  data: /data/data

http.cors.enabled: true
EOF
docker run -d -p 9200:9200 -p 9300:9300 \
  -v ${HOME}/elastic_search:/data \
  --name ElasticSearch \
  dockerfile/elasticsearch \
  /elasticsearch/bin/elasticsearch \
  -Des.config=/data/elasticsearch.yml
docker@boot2docker:~$ docker ps -a
CONTAINER ID  IMAGE COMMAND  CREATED  STATUS  PORTS  NAMES
[whatever]    dockerfile/elasticsearch:latest "/elasticsearch/bin/ 24 seconds ago Up 23 seconds 0.0.0.0:9200->9200/tcp,0.0.0.0:9300->9300/tcp ElasticSearch

Now from your Mac browse to http://192.168.59.103:9200 and you should get a status result.

Now run elasticsearch-head on your Mac (not the vm):

cd /some/tools/folder/of/your/choice
git clone git://github.com/mobz/elasticsearch-head.git
cd elasticsearch-head
npm install
grunt server

Now browse to http://localhost:9100 and:
1) set the connect box to http://192.168.59.103:9200 and
2) click connect

You now have ES running on your boot2docker instance and can access it from your Mac.

Docker: Private Registry push yields "Error: Invalid registry endpoint" and "insecure-registry"

For some time I've been using boot2docker on my mac.
This was version 1.3.1.
I also have a private registry which for the purposes of this post I've called 10.0.0.1:5000.

Some time ago I accidently brew updated and got docker 1.4.1.
So I couldn't access the running boot2docker daemon.
For some reason that seemed quite reasonable at the time, I didn't restart the boot2docker vm.
Go figure.

So I had to do this kind of thing:

(
export DOCKER_HOST=tcp://192.168.59.103:2376
export DOCKER_CERT_PATH=${HOME}/.boot2docker/certs/boot2docker-vm
export DOCKER_TLS_VERIFY=1
/usr/local/Cellar/docker/1.3.2/bin/docker push 10.0.0.1:5000/some_image
)

Then today I accidently started Kitematic.
Which upgraded the boot2docker vm and restarted it.
And blammo I could no longer push to the private repository.

FATAL[0002] Error: Invalid registry endpoint https://10.0.0.1:5000/v1/: \
  Get https://10.0.0.1:5000/v1/_ping: EOF. \
  If this private registry supports only HTTP or HTTPS with an unknown CA certificate, \
  please add `--insecure-registry 10.0.0.1:5000` to the daemon's arguments. \
  In the case of HTTPS, if you have access to the registry's CA certificate, \
  no need for the flag; simply place the CA certificate at \
  /etc/docker/certs.d/10.0.0.1:5000/ca.crt

Bugger.
Dope slap.
Noob.

So the fix is:

1) ssh into your boot2docker vm and
2) sudo vi /var/lib/boot2docker/profile and
3) Add 'EXTRA_ARGS="--insecure-registry 10.0.0.1:5000"' to it, :wq and
4) sudo /etc/init.d/docker restart
5) And just use the /usr/local/bin/docker and not the Cellar version.

YMMV.

Docker: Delete "dangling" images

After using docker for a while one notices a distinct slow down in performance of your host.

Some digging revealed that "dangling" images are the culprit.

While being a little simplistic, these are images that are no longer associated with any running container.

I dare you to go onto your docker host (even boot2docker) and do this:

docker images -q --filter "dangling=true"

My guess is that you'll see hundreds (or thousands if you've been busy) of those.

Here's the trick to getting rid of those large incontinent beasts that are leaving great piles of steaming images in your host:

docker images -q --filter "dangling=true" | xargs docker rmi

You may see a few false positives, but "hey!" at least the trash has been taken out.

Might be worth cronning it...

YMMV.

Get next free port on Linux with Ruby

Get next free port:

require 'socket'
(1..10).each do |n|
socket = Socket.new(:INET, :STREAM, 0)
socket.bind(Addrinfo.tcp("127.0.0.1", 0))
port = socket.local_address.ip_port
puts "port -> #{port}"
socket.close
end

command line:

port=$(ruby -e "require 'socket'; \
s = Socket.new(:INET, :STREAM, 0); \
s.bind(Addrinfo.tcp('127.0.0.1', 0)); \
port = s.local_address.ip_port; \
s.close; \
p port")