Introduction

The Swift account manager can supply a Docker image in which the account’s storlets are to be executed. When planning the Docker image, the account manager needs to consider the following:

  1. The image is built over an existing image that contains the storlets run time.
  2. The image would be executed with no network devices, with limited memory and cpu.
  3. A Docker image is not brought up as a general purpose Linux machine in terms of the init process. Specifically, do not install daemons that require special initializations on ‘OS bring up’.

The idea is that a user supplied docker image would contain dependencies required by storlets in the form of libraries.

Logical Flow

The flow of deploying an account manager tailored Docker image to Swift involves both the account manager (customer side) and the Swift Storlet manager (provider side) Below are the steps of this flow:

  1. A prerequisite for the account manager to deploy a Docker image to Swift is having an account that is enabled for Storlets. This is an operation that is done by the Swift Storlet manager and is explained in <https://github.com/openstack/storlets/blob/master/doc/source/storlets_management.rst>.
  2. Once the account is enabled for Storlets, a container named docker_images is created, with access to both the account manager as well as the storlet manager. That container will include a basic Docker image consisting of some Storlet engine code that allows the Swift Storlet engine to work with that image.
  3. The account manager can download this image, adjust it to his needs in terms of the installed software stack, and upload it back to the docker_images container.
  4. Once uploaded, the account manager must notify the Swift Storlet engine manager of the update. The storlets manager would take care of testing and deploying it to all Swift nodes. Again, <https://github.com/openstack/storlets/blob/master/doc/source/storlets_management.rst> describes the provided tool the Storlet manager can use for the actual deployment.

The sections below describe in detail the steps taken by the account manager.

Downloading the Docker Image

Downloading the Docker image involves a simple retrieval of a Swift object. To get the exact name of the object just list the docker_images container in the account. The name will carry the base OS system and engine language binding run time. An example might be: ubuntu_14.04_jre8_storlets reflecting the following facts:

  1. The base OS is Ubuntu 14.04. Currently this is the only base OS we support.
  2. The Storlets run time is jre8. Currently storlets can be written only in Java.
  3. The storlet engine code is installed.

The image will come in a .tar format.

Below is an example of downloading the image from the tenant’s docker_images container using the swift CLI. As with all examples using the Swift CLI, we are using environment variables defining the tenant, user credentials and auth URI. All these are required for the operation of any Swift CLI. Please change them accordingly.

export OS_USERNAME=swift
export OS_PASSWORD=passw0rd
export OS_TENANT_NAME=service
export OS_AUTH_URL=http://127.0.0.1:35357/v2.0

In the below we show:

  1. Listing the docker_images container.
  2. Downloading the image object
  3. Getting the image object’s metadata. Pay attention to the image_name metadata field of the object. It is required for the next steps.
 eranr@lnx-ccs8:~$ swift list docker_images
 ubuntu_14.04_jre8_storlets.tar
 eranr@lnx-ccs8:~$ swift download docker_images ubuntu_14.04_jre8_storlets.tar
 ubuntu_14.04_jre8_storlets.tar [headers 0.311s, total 8.550s, 68.008 MB/s]
 eranr@lnx-ccs8:~$ swift stat docker_images ubuntu_14.04_jre8_storlets.tar
        Account: AUTH_305f5f3d12834be187238e080b8643e4
      Container: docker_images
         Object: ubuntu_14.04_jre8_storlets.tar
   Content Type: application/x-tar
 Content Length: 581439488
  Last Modified: Sat, 25 Oct 2014 19:47:13 GMT
           ETag: ac014db984be37faf7307801baa11ab0
Meta Image-Name: ubuntu_14.04_jre8_storlets
     Meta Mtime: 1414266426.880534
  Accept-Ranges: bytes
    X-Timestamp: 1414266432.09929
     X-Trans-Id: tx794e21cd40b544e6a377b-00544bfed3

Tuning the Docker Image

To tune the Docker image, Docker must be used. To install please refer to http://docs.docker.com/installation/

The below steps illustrate the tuning process:

  1. Use docker load to load the .tar image. Each Docker client maintains a local repository of the images from which containers can be executed. The load operation simply loads the .tar file to that local repository. Note that once the .tar is loaded, the docker images command shows the image, whose name has a suffix identical to the image object Swift metadata.
root@lnx-ccs8:/home/eranr# docker load -i ubuntu_14.04_jre8_storlets.tar
root@lnx-ccs8:/home/eranr# docker images
REPOSITORY                                  TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
localhost:5001/ubuntu_14.04_jre8_storlets   latest              f6929e6abc60        3 days ago          563.6 MB

  1. Use a Docker file that is based on the loaded image to make the necessary changes to the image. Below is a Dockerfile for installing ‘ffmpeg’. Few notes are in place:

    1. The first line “FROM” must carry the image name we have downloaded.

    2. The maintainer needs to be a user that is allowed to do the actual actions within the container. Please leave it as is.

    3. The below example shows ffmpeg installation. For more options and

      information on Dockerfiles, please refer to: http://docs.docker.com/reference/builder/

    4. One MUST refrain from using the Dockerfile ENTRYPOINT and CMD. Using those will cause the image from being unusable by the Storlet engine.

root@lnx-ccs8:/home/eranr/dockerfile_example# cat Dockerfile
FROM 127.0.0.1:5001/ubuntu_14.04_jre8_storlets

MAINTAINER root

RUN ["apt-get", "update"]
RUN ["apt-get", "install","-y", "software-properties-common"]
RUN ["add-apt-repository","deb http://ppa.launchpad.net/jon-severinsson/ffmpeg/ubuntu trusty main"]
RUN ["apt-key", "adv", "--recv-keys", "--keyserver", "keyserver.ubuntu.com", "1DB8ADC1CFCA9579"]
RUN ["apt-key", "update"]
RUN ["apt-get", "update"]
RUN ["apt-get", "install", "-y", "ffmpeg"]
  1. We now use the Docker fie to create a new image from it. Note the -t directive for the new image name to be created. The name of the image would be required for the Storlet manager to deploy the Storlet. Also, note that the command ends with a dot ”.” specifying in which directory the build is taking place. when building an image that copies stuff into the image, all that stuff must reside in that building directory.
root@lnx-ccs8:/home/eranr/dockerfile_example# docker build -t service_tenant_image .
Sending build context to Docker daemon  2.56 kB
Sending build context to Docker daemon
Step 0 : FROM 127.0.0.1:5001/ubuntu_14.04_jre8_storlets
---> f6929e6abc60
......
Processing triggers for libc-bin (2.19-0ubuntu6.3) ...
---> 11975468ecf8
Removing intermediate container 226d2510b925
Successfully built 11975468ecf8
  1. At this point listing the images, shows the newly created image.
root@lnx-ccs8:/home/eranr/dockerfile_example# docker images
REPOSITORY                                  TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
service_tenant_image                        latest              11975468ecf8        7 minutes ago       660.1 MB
localhost:5001/ubuntu_14.04_jre8_storlets   latest              f6929e6abc60        4 days ago          563.6 MB

Currently, we have no testing tool that can actually test a storlet inside the created image. The best one can do is run a Docker container based on the image, and run within it code that simulates how the Storlet would use the image. Below we run /bin/bash inside a container based on the newly created image. We then invoke ffmpeg showing that the installation was indeed successful. Note that the ‘debug’ parameter tells our entry point not to execute the storlet engine but rather the /bin/bash from which we can run ffmpeg

root@lnx-ccs8:/home/eranr/dockerfile_example# docker run -i -t service_tenant_image debug /bin/bash
root@b129c3e6e76b:/# ffmpeg
ffmpeg version 1.2.6-7:1.2.6-1~trusty1 Copyright (c) 2000-2014 the FFmpeg developers
  built on Apr 26 2014 18:52:58 with gcc 4.8 (Ubuntu 4.8.2-19ubuntu1)
  configuration: --arch=amd64 --disable-stripping --enable-avresample --enable-pthreads --enable-runtime-cpudetect --extra-version='7:1.2.6-1~trusty1' --libdir=/usr/lib/x86_64-linux-gnu --prefix=/usr --enable-bzlib --enable-libdc1394 --enable-libfreetype --enable-frei0r --enable-gnutls --enable-libgsm --enable-libmp3lame --enable-librtmp --enable-libopencv --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-libschroedinger --enable-libspeex --enable-libtheora --enable-vaapi --enable-vdpau --enable-libvorbis --enable-libvpx --enable-zlib --enable-gpl --enable-postproc --enable-libcdio --enable-x11grab --enable-libx264 --shlibdir=/usr/lib/x86_64-linux-gnu --enable-shared --disable-static
  libavutil      52. 18.100 / 52. 18.100
  libavcodec     54. 92.100 / 54. 92.100
  libavformat    54. 63.104 / 54. 63.104
  libavdevice    53.  5.103 / 53.  5.103
  libavfilter     3. 42.103 /  3. 42.103
  libswscale      2.  2.100 /  2.  2.100
  libswresample   0. 17.102 /  0. 17.102
  libpostproc    52.  2.100 / 52.  2.100
Hyper fast Audio and Video encoder
usage: ffmpeg [options] [[infile options] -i infile]... {[outfile options] outfile}...

Use -h to get full help or, even better, run 'man ffmpeg'

Uploading the Docker Image

  1. Use docker save to save the image as a tar file:
root@lnx-ccs8:/home/eranr/dockerfile_example# docker save -o service_tenant_image.tar service_tenant_image
  1. Again, we use the Swift CLI to upload the image. We assume the appropriate environment variables are in place.
root@lnx-ccs8:/home/eranr/dockerfile_example# swift upload docker_images service_tenant_image.tar
service_tenant_image.tar