for work

  • geovisual analytics
    • computer vision/machine learning scientist
  • us doc—institute for telecommunication sciences
    • "expert"—use ml to perform audio and video quality estimation

for fun:

what even is docker

docker is a way to build infrastructure using code

okay thanks, but what

imagine setting up your dev/prod environment with one command.

imagine a shell script to set up your dev/prod environment but without the pain.

imagine not worrying about dependencies.

then take about one step back from there.

why are you here, you are not devops

there are lots of good python/docker tutorials out there! definitely check them out.

but here's a crash course that covers some of the subtleties that aren't always covered in the tutorials.

speak up if you see something that doesn't make sense!

let's deploy our python code!

using docker!

why

  1. why not
  2. because
  3. predictable production environments are rad
$> docker run -t your_app
----------------------
generating profit...
done!!!1
----------------------

start from the basics: build a simple linux image

baby's first dockerfile

# Dockerfile---image name: dd
FROM alpine:3.10

RUN apk add sl

CMD /bin/sh

a devops poem in three layers

(it's a haiku if you say it right!)

sweet, wat do

layer 1: import a "base image"—in this case a tiny linux

FROM alpine:3.10

layer 2: install a system-level package in our little container

RUN apk add sl

layer 3: run a command after everything else is set up, a.k.a. entrypoint

CMD /bin/sh

sweet, how to

build the docker "image" and tag it with the name dd

docker build  -t dd .

run the docker "image" by specifying the tag name—launches a shell!

docker run -it dd

specifying -i tells docker to let you interact via the terminal

check it out.

now let's spice it up with python

baby's second dockerfile, hello cruel world

# Dockerfile---image name: py_hello_world
FROM python:3.7.3-alpine3.10

CMD python -c "print('Hello World')"

sweet, wat do

import a "base image"

FROM python:3.7.3-alpine3.10

run a command after everything else is set up

CMD python -c "print('Hello World')"

sweet, how to

build the docker "image" and tag it with the name py_hello_world

docker build  -t py_hello_world .

run the docker "image" by specifying the tag name. python prints Hello World to the terminal

docker run -t py_hello_world

check it out.

okay, but my program has more than one line

i thought you'd never ask! here's a dockerfile:

# Dockerfile---image name: py_ezpz
FROM python:3-alpine3.10

WORKDIR /app

COPY app /app

RUN pip install -r requirements.txt

COPY content /content

CMD ["python", "/app/analyze.py"]

sweet, wat do

you can pick from many different images (debian, alpine, windows) with python already installed

FROM python:3-alpine3.10

specify a directory to start in and then copy our code into the image

WORKDIR /app

COPY app /app

oh snap

RUN pip install -r requirements.txt

copy some more things into the image

COPY content /content

set our entrypoint.

CMD ["python", "/app/analyze.py"]

/app/requirements.txt:

parse
In [ ]:
# /app/analyze.py
from parse import parse

def is_vogon(poem):
    """determines whether or not a poem is vogon"""
    for line in poem:
        parse_result = parse('On a {} bee,', line)
        if parse_result:
            if parse_result[0] == 'lurgid':
                return True

    return False

if __name__ == "__main__":
    # print('VERSION TWO POINT OH!!!!!!')
    print('analyzing poem readability')
    # open the poem
    with open('/content/poem.txt') as fp:
        poem = fp.readlines()
    # should we read it?????????
    if is_vogon(poem):
        print('third worst poetry in the galaxy, pls avoid')
    else:
        print('definitely worth reading')

/content/poem.txt:

Oh freddled gruntbuggly,
Thy micturations are to me, (with big yawning)
As plurdled gabbleblotchits, in midsummer morning
On a lurgid bee,
That mordiously hath blurted out,
Its earted jurtles, grumbling
Into a rancid festering confectious organ squealer. [drowned out by moaning and screaming]
Now the jurpling slayjid agrocrustles,
Are slurping hagrilly up the axlegrurts,
And living glupules frart and stipulate,
Like jowling meated liverslime,
Groop, I implore thee, my foonting turlingdromes,
And hooptiously drangle me,
With crinkly bindlewurdles,mashurbitries.
Or else I shall rend thee in the gobberwarts with my blurglecruncheon,
See if I don't!

try it out

you know the drill:

docker build  -t py_ezpz .

and run it

docker run -t py_ezpz

check it out.

pretty cool, but could be better

  1. reinstalling your environment when you make a code change is not ideal
  2. not really great that our content can't update short of rebuilding the container

we can fix this.

  1. order matters
  2. mount a volume instead of copying the content.

order matters

# Dockerfile---image name: py_ezpz
FROM python:3-alpine3.10

WORKDIR /app

COPY app /app       <-------- CACHE BUSTER

RUN pip install -r requirements.txt

COPY content /content

CMD ["python", "/app/analyze.py"]

order matters

# Dockerfile---image name: py_ezpz
FROM python:3-alpine3.10

copy requirements.txt requirements.txt

RUN pip install -r requirements.txt

WORKDIR /app

COPY app /app

COPY content /content

CMD ["python", "/app/analyze.py"]

mount a volume

docker run -v ~/gitrepos/pydocker_talk/py_ezpz/content/:/content -t py_ezpz

check it out.

can we make this even more reusable

we can make this even more reusable.

imagine the case where everybody on a team uses the same tools (dependencies) but has different tasks.

let's build our system in stages

why?

  1. why not
  2. because
  3. the fewer times you have to rebuild your environment the happier you are
  4. it is actually really easy to distribute a docker image (upload to dockerhub!)

first write our own base image

# Dockerfile---image name: analyze_base
FROM python:3-alpine3.10

COPY requirements.txt requirements.txt

RUN pip install -r requirements.txt

and build it:

docker build -t analyze_base -f Dockerfile.base .

next construct the rest of our app

# Dockerfile---image name: analyze
FROM analyze_base

WORKDIR /app

COPY app /app

CMD ["python", "/app/analyze.py"]

and build it:

docker build --no-cache -t analyze .

try it out

docker run -v ~/gitrepos/pydocker_talk/py_inherit/content/:/content -t analyze

check it out.

what if you need conda???

shudder

no but really, conda can be very useful because in many cases it will handle system level dependencies as well.

it is definitely quirky though.

you can inherit from a miniconda3 container and start building your environment from there.

pro tips:

  1. use the conda-forge channel as much as possible and remove defaults from your .condarc.
  2. try not to pin versions unless absolutely necessary.
  3. use a multi-stage build! you really don't want to be rebuilding your conda environment all the time.

what if you have lots of internal packages

this starts to get a little more tricky.

  1. part of your dockerfile can be RUN commands that install your packages
  2. you could host your own pypi server/conda channel

then you could build the image locally and distribute using dockerhub

are there any other suggestions for how to securely/privately do this over the internet?

In [ ]: