Upgrading the docker service from 1.3 to 1.5 on Ubuntu 14.04

Recently I had to upgrade a production server to ensure it was running the latest version of docker (1.5.0 at time of writing).

The current install was 1.3.1 and we wanted all docker servers to be identical.

First up, assume you have done a sudo -i to ensure all commands are run as root.

Also, some commands are prefixed with $ . This is to identify the command from it's output.

And further, I have used bogus IPv4 addresses for certain URLs.

For reference, I used 10.0.0.3 as the docker server IP.

# First some useful commands
# Useful to find files in packages or use packages.ubuntu.com

apt-file search nslookup

# -------------------------------------------------------------------
# Install tree and dnsutils (nslookup and friends)
# -------------------------------------------------------------------

apt-get install tree dnsutils

# -------------------------------------------------------------------
# Kill all containers
# -------------------------------------------------------------------
docker ps -a | egrep 'ls-api' | awk '{ print $NF; }' | xargs docker kill
docker ps -a | egrep 'ls-api' | awk '{ print $NF; }' | xargs docker rm

# -------------------------------------------------------------------
# Kill all images
# -------------------------------------------------------------------
docker images -q --filter "dangling=true" | xargs docker rmi
docker images | grep -v REPOSITORY | awk '{ print $3; }' | \
  sort | uniq | xargs docker rmi

# -------------------------------------------------------------------
# Stop docker
# -------------------------------------------------------------------
service docker stop

# -------------------------------------------------------------------
# Uninstall docker.io (You may need to do 'aptitude search docker'
# to see if it is installed. If you do not have 'docker' installed,
# but 'lxc-docker', then remove that instead
# -------------------------------------------------------------------
apt-get remove docker.io

# -------------------------------------------------------------------
# Cleanup any remaining files you want from:
# /var/lib/docker 
# /etc/init.d 
# /etc/default/docker
# /var/log/docker* 
# Your choice.
# -------------------------------------------------------------------

# -------------------------------------------------------------------
# Install AUFS support
# -------------------------------------------------------------------
apt-get update
apt-get install linux-image-extra-`uname -r`

# -------------------------------------------------------------------
# Fix keys
# -------------------------------------------------------------------
apt-get update
apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 \
  --recv-keys 36A1D7869245C8950F966E92D8576A8BA88D21E9

# -------------------------------------------------------------------
# Update your apt sources
# Note: First check your /etc/apt/sources.list.d/docker.list to see 
#       if it already has this
# -------------------------------------------------------------------
sh -c "echo deb http://get.docker.com/ubuntu docker main > /etc/apt/sources.list.d/docker.list"

# -------------------------------------------------------------------
# Install latest docker 
# -------------------------------------------------------------------
apt-get update
apt-get install lxc-docker

# Note: During install, yo may get a message about 
#       /etc/init/docker.conf being present.
#       If so, choose Y to overwrite your old one with the new one.

# -------------------------------------------------------------------
# Change /etc/default/docker:
# -------------------------------------------------------------------

# Use DOCKER_OPTS to modify the daemon startup options.
#DOCKER_OPTS="--dns 8.8.8.8 --dns 8.8.4.4"
DOCKER_OPTS="-H tcp://0.0.0.0:2376 \
  -H unix:///var/run/docker.sock \
  --dns 10.0.0.1 \                      <<- Your local DNS
  --dns 10.0.0.2 \                      <<- services
  --insecure-registry 10.0.0.4:5000"    <<- If you have a private repo

# -------------------------------------------------------------------
# Look at the upstart jobs:
# -------------------------------------------------------------------
service --status-all
 [ + ]  apparmor
 [ ? ]  console-setup
 [ + ]  cron
 [ - ]  docker
 [ - ]  grub-common
 [ ? ]  killprocs
 [ ? ]  kmod
 [ ? ]  networking
 [ + ]  ntp
 [ ? ]  ondemand
 [ ? ]  open-vm-tools
 [ + ]  postfix
 [ - ]  procps
 [ + ]  puppet
 [ + ]  rabbitmq-server
 [ ? ]  rc.local
 [ + ]  resolvconf
 [ - ]  rsync
 [ + ]  rsyslog
 [ ? ]  sendsigs
 [ + ]  snmpd
 [ - ]  ssh
 [ - ]  sudo
 [ + ]  udev
 [ ? ]  umountfs
 [ ? ]  umountnfs.sh
 [ ? ]  umountroot
 [ - ]  unattended-upgrades
 [ - ]  urandom

# -------------------------------------------------------------------
# Restart the service to enable AUFS support
# -------------------------------------------------------------------
service docker restart

# -------------------------------------------------------------------
# Get info
# -------------------------------------------------------------------
$ docker info
Containers: 0
Images: 0
Storage Driver: aufs        <-- AUFS
 Root Dir: /mnt/docker/aufs <-- AUFS
 Backing Filesystem: extfs
 Dirs: 0
Execution Driver: native-0.2
Kernel Version: 3.13.0-24-generic
Operating System: Ubuntu 14.04.1 LTS
CPUs: 16
Total Memory: 15.67 GiB
Name: dock-prod-001
ID: AAAA:BBBB:CCCC:DDDD:EEEE:FFFF:GGGG:HHHH:0000:1111:2222:3333

# Your ID will be different of course 

# -------------------------------------------------------------------
# Start Seagull to have a pretty interface to containers
# See https://registry.hub.docker.com/u/tobegit3hub/seagull/
# -------------------------------------------------------------------
docker run -d \
  -p 10086:10086 \
  -v /var/run/docker.sock:/var/run/docker.sock \
  --name=Seagull \
  tobegit3hub/seagull

# Browse via your desktop to http://10.0.0.3:10086/containers

# -------------------------------------------------------------------
# Install Elastic
# See https://registry.hub.docker.com/_/elasticsearch/
# The '_' in the URL means it's an official image
# -------------------------------------------------------------------
docker run -d \
  -p 9200:9200 -p 9300:9300 \
  -v /some/path/to/elastic_search_data:/data \
  --name=Elastic \
  elasticsearch \
  elasticsearch -Des.config=/data/elasticsearch.yml
# Note: The first 'elasticsearch' is the image, and
#       the second 'elasticsearch' is the command plus options

# Browse to http://10.0.0.3:9200 and http://10.0.0.3:9200/_search

# -------------------------------------------------------------------
# Install Redis
# See https://registry.hub.docker.com/u/library/redis/
# -------------------------------------------------------------------
docker run -d \
  -p 6379:6379 \
  -v /some/path/to/redis_data:/data \
  --name=Redis \
  redis redis-server \
  --appendonly yes
# Note: The 'redis' is the image, and
#       the 'redis-server' is the command plus any options

# -------------------------------------------------------------------
# Install redis command line tools
# So you can interact with the Redis container
# -------------------------------------------------------------------
apt-get update
apt-get install redis-tools

# -------------------------------------------------------------------
# Test
# -------------------------------------------------------------------
# First go to the redis_data folder which was defined via the start
$ cd redis_data
# Let's look at the data store
$ ls -l appendonly.aof
total 0
-rw-r--r-- 1 deploy docker 0 Mar 18 09:24 appendonly.aof
# Empty, so let's add a key
$ redis-cli
127.0.0.1:6379> keys *
(empty list or set)
127.0.0.1:6379> set hello world
OK
127.0.0.1:6379> keys *
1) "hello"
127.0.0.1:6379> get hello
"world"
127.0.0.1:6379> quit
# Now let's look at it
$ ls -l appendonly.aof
total 4
-rw-r--r-- 1 deploy docker 58 Mar 18 09:33 appendonly.aof
# Ooo. Changed, so let's see if we can view that file
$ file appendonly.aof
appendonly.aof: ASCII text, with CRLF line terminators
# Yup. So let's look at it:
$ cat appendonly.aof
*2
$6
SELECT
$1
0
*3
$3
set
$5
hello
$5
world

# -------------------------------------------------------------------
# Look at the containers:
# -------------------------------------------------------------------
$ docker ps -a
CONTAINER ID IMAGE                      COMMAND              CREATED        STATUS        PORTS                                          NAMES
c758b7dc90e5 redis:latest               "/entrypoint.sh redi 4 minutes ago  Up 4 minutes  0.0.0.0:6379->6379/tcp                         Redis
310ccb182746 elasticsearch:latest       "elasticsearch -Des. 8 minutes ago  Up 8 minutes  0.0.0.0:9200->9200/tcp, 0.0.0.0:9300->9300/tcp Elastic
e78470ddebca tobegit3hub/seagull:latest "./seagull"          30 minutes ago Up 30 minutes 0.0.0.0:10086->10086/tcp                       Seagull

# -------------------------------------------------------------------
# Start doing your own deployments!
# -------------------------------------------------------------------

Thank you Dr Mark Courtney and Dr Paul Murphy and all the staff at John Flynn Hospital

Ok. I survived my latest surgery. Many thanks to the staff of John Flynn Private Hospital and Dr Mark Courtney and Dr Mark Murphy for making it memorable.

I say memorable in that my handbag and overnight bag got lost. I'm not complaining mind you.

Seriously.

6 hours of searching by security staff eventually found them. Keys, cards, prescriptions, etc basically 2/3rds of my life recovered. Thanks guys!

In the mean time I had to wear paper clothing and be restricted to my room. Which was awesome I have to say. Fantastic views. But no underwear. Sucks to be me. Dr Courtney came round to see how I was faring at 7:30pm. Way, way, seriously way, after his visiting times.

Now that's dedication to doing the right thing.

Awesome!

He was an angel and sent me home rather than stay overnight because:

1) The cyst (huge bugger as it was) was easy to remove and
2) I didn't bleed like a stuck pig and
3) He understood I had to pay for the room out of my own pocket and
4) ALIENS! No. Not really. Just people. Humans. Good humans.
5) Now where did I put number 6?
6) Oh! Here it is!

I shook his hand warmly and gave him a hug. He positively beamed happiness.

And a big shout-out to Dr Paul Murphy.
We had met before and I kept calling him 'Paul'.
Kinda odd in a professional kind of way.
I kept expected him to say

I didn't spend 3 years in "evil Anaesthesia College" to be called "Paul" thank you very much. It's Dr Murphy if you please.

Sorry about that.

Oh. I have to mention. While in pre-op I overheard the birth of two babies. AWESOME. Made me smile. And many of the staff I have to say. Cool.

I now have a HUGE plaster on my neck and my neck hurts like... Like... Buggery... But I have antibiotics, pan forte and FINALLY have this damn thing out of my neck. Two years of coughing myself to distraction every morning for 2 hours. Gone. Clicking when I swallow. Gone. Glands the size of golf balls. Gone. Finally. Gone.

Thank you Dr Mark Courtney and Dr Mark Murphy and all the staff at John Flynn Hospital for making it go away. Thank you.

And A massive shout out to security for ransacking every ward and every locker for my bags.

Thank you.

Dr Courtney.
Dr Murphy.
ALL THE STAFF AT JOHN FLYNN PRIVATE HOSPITAL!
ALL OF YOU.
SPECIAL SHOUT-OUT TO SECURITY - AWESOME JOB DUDES.

Docker: Find what the container port is from inside the container!

Ok. Service registration and discovery inside a docker container can be fiddly sometimes.
And I wanted to have dynamic port numbers when starting a container so I could 'register' the service in Redis.

So how do you do it?

I first fiddled with using socat inside the startup for the service which worked, but was ugly.

FYI: I ran up a vagrant ubuntu VM and installed docker 1.5 to test this.

So here's the way to do it in ruby.

# ---------------------------------------------------------------------------
# Find out what our container port is
# ---------------------------------------------------------------------------

SVC_NAME = 'api-dummy_1'

require 'socket'
require 'net/http'

# Create the socket to the docker host
sock = Net::BufferedIO.new(UNIXSocket.new('/var/run/docker.sock'))

# Go grab all the containers details
request = Net::HTTP::Get.new('/containers/json')
request.exec(sock, '1.1', '/containers/json')
begin
  response = Net::HTTPResponse.read_new(sock)
end while response.kind_of?(Net::HTTPContinue)
response.reading_body(sock, request.response_body_permitted?) { }

# Parse and loop over it trying to find our name
data = JSON.parse(response.body)
puts "Data received: #{data}"
data.each do |container|
  puts "Looking at: #{container}"
  if container['Names'].include? "/#{SVC_NAME}"
    container_port = container['Ports'][0]['PublicPort']
    puts "CONTAINER_PORT: #{container_port}"
    ENV['SVC_PORT'] = container_port.to_s
    break
  end
end

Obviously you'd have to do something to handle it if you can't find the name...
And should really check the Ports array better.

The socket call returns an array something like this:

[
{
  "Command":"/bin/sh -c 'bundle exec foreman start'",
  "Created":1425423198,
  "Id":"5b11471046a04b64fffc2866d4eb67568221fb8c3445a326557182208559e460",
  "Image":"my_repo:5000/something/api-dummy_1:latest",
  "Names":["/api-dummy_1"],
  "Ports":[{"IP":"0.0.0.0","PrivatePort":5000,"PublicPort":49172,"Type":"tcp"}],
  "Status":"Up 1 seconds"
},
{
  ...another one...
}
]

You have to map the /var/run/docker.sock on running the container of course.
Something like this:

#!/bin/bash
export SVC_NAME=api-dummy_1
export DOCKER_HOST=tcp://127.0.0.1:2378
export REPO=10.0.0.1 # Whatever
docker pull ${REPO}:5000/something/${SVC_NAME}
docker kill ${SVC_NAME}
docker rm ${SVC_NAME}
docker run -d --env RAILS_ENV=production \
              --env HOST_IP=10.0.0.2 \
              --env SVC_NAME=api-dummy_1 \
              --env REDIS=10.0.0.3 \
              --name ${SVC_NAME} \
              -p :5000 \
              -v /var/run/docker.sock:/var/run/docker.sock \
              ${REPO}:5000/something/${SVC_NAME}

Names and IPs to be changed of course.

Still fiddly, but it works.

YMMV.

Update:

I just realised that docker provides a HOSTNAME environment variable which is essentially the container id which would allow you to call /containers/#{ENV['HOSTNAME']}/json instead of doing the loop to get the configuration for that specific id.

The configuration returned is slightly different. See https://docs.docker.com/reference/api/docker_remote_api_v1.15/#inspect-a-container for details

Enjoy.