In a previous post, I discussed moving my websites into Docker containers with their own separate httpd servers from their previous setup as virtual hosts on a single httpd server. This post will discuss integrating HAProxy and Fig into my installation. This will allow for load balancing, proper routing, and easy deployments.
To add HAProxy, I simply used the library haproxy image. You will just need to create your own Dockerfile and copy in your haproxy.cfg as the instructions state from the link. The first part of the config file is fairly standard:
1: global 2: log 127.0.0.1 local0 3: log 127.0.0.1 local1 notice 4: chroot /var/lib/haproxy 5: user haproxy 6: group haproxy 7: 8: 9: defaults 10: log global 11: mode http 12: option httplog 13: option dontlognull 14: option forwardfor 15: option http-server-close 16: timeout connect 5000 17: timeout client 50000 18: timeout server 50000 19: errorfile 400 /etc/haproxy/errors/400.http 20: errorfile 403 /etc/haproxy/errors/403.http 21: errorfile 408 /etc/haproxy/errors/408.http 22: errorfile 500 /etc/haproxy/errors/500.http 23: errorfile 502 /etc/haproxy/errors/502.http 24: errorfile 503 /etc/haproxy/errors/503.http 25: errorfile 504 /etc/haproxy/errors/504.http 26: stats enable 27: stats uri /haproxy?stats
It is the rest of the file that does the work for us. I have multiple websites served from the same server, and each website is also accessed by multiple domains (i.e. cafezvous.com is also cafezvous.net). There may be a less verbose way of doing what I've done, but I haven't found that method. We need to declare our frontend and bind it to port 80 on any incoming IP address. We then define ACL's (Access Control Lists) for each domain and associate them with the appropriate generic host. We then associate each host with the backend cluster that will serve the calls to that website. We finally use cafezvous as the default cluster if no ACL is matched.
29: frontend http-in 30: bind *:80 31: 32: # Define hosts 33: acl host_cafezvous hdr(host) -i cafezvous.com 34: acl host_cafezvous hdr(host) -i cafezvous.co 35: acl host_cafezvous hdr(host) -i cafezvous.info 36: acl host_cafezvous hdr(host) -i cafezvous.org 37: acl host_cafezvous hdr(host) -i cafezvous.net 38: acl host_cafezvous hdr(host) -i www.cafezvous.com 39: acl host_cafezvous hdr(host) -i www.cafezvous.co 40: acl host_cafezvous hdr(host) -i www.cafezvous.info 41: acl host_cafezvous hdr(host) -i www.cafezvous.org 42: acl host_cafezvous hdr(host) -i www.cafezvous.net 43: acl host_dbdevs hdr(host) -i dbdevs.com 44: acl host_dbdevs hdr(host) -i dbdevs.co 45: acl host_dbdevs hdr(host) -i dbdevs.info 46: acl host_dbdevs hdr(host) -i dbdevs.org 47: acl host_dbdevs hdr(host) -i dbdevs.net 48: acl host_dbdevs hdr(host) -i www.dbdevs.com 49: acl host_dbdevs hdr(host) -i www.dbdevs.co 50: acl host_dbdevs hdr(host) -i www.dbdevs.info 51: acl host_dbdevs hdr(host) -i www.dbdevs.org 52: acl host_dbdevs hdr(host) -i www.dbdevs.net 53: acl host_danpluslaura hdr(host) -i danpluslaura.com 54: acl host_danpluslaura hdr(host) -i danpluslaura.co 55: acl host_danpluslaura hdr(host) -i danpluslaura.info 56: acl host_danpluslaura hdr(host) -i danpluslaura.org 57: acl host_danpluslaura hdr(host) -i danpluslaura.net 58: acl host_danpluslaura hdr(host) -i www.danpluslaura.com 59: acl host_danpluslaura hdr(host) -i www.danpluslaura.co 60: acl host_danpluslaura hdr(host) -i www.danpluslaura.info 61: acl host_danpluslaura hdr(host) -i www.danpluslaura.org 62: acl host_danpluslaura hdr(host) -i www.danpluslaura.net 63: 64: ## figure out which one to use 65: use_backend cafezvous_cluster if host_cafezvous 66: use_backend dbdevs_cluster if host_dbdevs 67: use_backend danpluslaura_cluster if host_danpluslaura 68: 69: default_backend cafezvous_cluster
Normally our backends would be associated to some IP address, but we don't know what the IP address of a container will be until it's created. We could wait until all of the servers are started to then add each IP address and port individually, but that doesn't seem reasonable or easy. Also, we'd be creating a new container for each restart. We could also map each website to a particular port on the host and then update this with the host IP address/localhost and port. The problem isn't as bad with this solution, but this won't scale well. Plus, the smart guys at Docker have thought of a clever way to associate an IP with a container. They create an entry in /etc/hosts when the --link command is used to link containers together. So, the cafezvous container can be referenced by cafezvous with a trailing port number. And dbdevs is referenced the same way. All of them can continue to use the common port of 80 and this can be deployed anywhere without checking to make sure a port isn't already mapped to the host.
71: backend cafezvous_cluster 72: balance leastconn 73: option httpclose 74: option forwardfor 75: cookie JSESSIONID prefix 76: server node1 cafezvous:80 cookie A check 77: 78: backend dbdevs_cluster 79: balance leastconn 80: option httpclose 81: option forwardfor 82: cookie JSESSIONID prefix 83: server node1 dbdevs:80 cookie A check 84: 85: backend danpluslaura_cluster 86: balance leastconn 87: option httpclose 88: option forwardfor 89: cookie JSESSIONID prefix 90: server node1 danpluslaura:80 cookie A check
Now we can start our containers:
$ docker run -dtP --name dbdevs barkerd427/dbdevs ed2ac1134324a7dda48d20567efb52e051d57f844c89b2a98a220a1c8b297a74 $ docker run -dtP --name cafezvous barkerd427/cafezvous ec7bfaf06985ace3f617ffc466c93eabfe4cfca916ff8c71c4a125e5d0a7dfae $ docker run -dtP --name danpluslaura barkerd427/danpluslaura 8ea8b1261d2ec1e5268e55f7504e4fab308cdaea1a5301e382ccb5de0e0718ee $ docker run -dtP --name haproxy --link cafezvous:cafezvous --link dbdevs:dbdevs --link danpluslaura:danpluslaura barkerd427/haproxy 7971754d55d10972370b401c8061218bf53b495aafdc82d366f9232aaa192a03 $ docker ps -a -s CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES SIZE 7971754d55d1 barkerd427/haproxy:0.1 "bash /haproxy-start 18 seconds ago Up 6 seconds 0.0.0.0:49161->443/tcp, 0.0.0.0:49162->80/tcp haproxy 0 B 8ea8b1261d2e barkerd427/danpluslaura:0.20 "httpd -DFOREGROUND" 2 minutes ago Up 2 minutes 0.0.0.0:49158->80/tcp danpluslaura 2 B ec7bfaf06985 barkerd427/cafezvous:0.4 "httpd -DFOREGROUND" 2 minutes ago Up 2 minutes 0.0.0.0:49157->80/tcp cafezvous 2 B ed2ac1134324 barkerd427/dbdevs:0.4 "httpd -DFOREGROUND" 3 minutes ago Up 2 minutes 0.0.0.0:49156->80/tcp dbdevs 2 B
This is a bit cumbersome to have to add all of these links individually, especially as the infrastructure becomes more complex. So, we'll go ahead and add a simple Fig configuration file to the mix and get all of these commands condensed into one. Here's the configuration fig.yml file:
haproxy: image: barkerd427/haproxy ports: - "80:80" links: - cafezvous - dbdevs - danpluslaura cafezvous: image: barkerd427/cafezvous ports: - "80" dbdevs: image: barkerd427/dbdevs ports: - "80" danpluslaura: image: barkerd427/danpluslaura ports: - "80"
We explicitly call out the port mapping for HAProxy to the host, but we leave the others to be assigned by Docker. We don't really care what the ports are on the host for this purpose, but we do need to know that the container port 80 is mapped generically. We also explicitly call out the links that HAProxy needs to the other container names. Now when I run fig up -d, everything will start in the correct order and run in the background. If you don't add the -d, then everything will shutdown when that session ends.
To run fig up, you need to be in the directory with the fig.yml file or reference that file with the -f or --file option. The project name defaults to the directory name, but this can be overridden with -p or --project-name. You can also attach to fig to see the output of the containers by using fig logs.
$ fig up -d Creating websitefig_danpluslaura_1... Creating websitefig_dbdevs_1... Creating websitefig_cafezvous_1... Creating websitefig_haproxy_1... $ fig ps Name Command State Ports ------------------------------------------------------------------------------------- websitefig_cafezvous_1 httpd -DFOREGROUND Up 0.0.0.0:49171->80/tcp websitefig_danpluslaura_1 httpd -DFOREGROUND Up 0.0.0.0:49169->80/tcp websitefig_dbdevs_1 httpd -DFOREGROUND Up 0.0.0.0:49170->80/tcp websitefig_haproxy_1 bash /haproxy-start Up 443/tcp, 0.0.0.0:80->80/tcp $ docker ps -a -s CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES SIZE 81ee5adf1aa3 barkerd427/haproxy:0.1 "bash /haproxy-start About a minute ago Up 59 seconds 443/tcp, 0.0.0.0:80->80/tcp websitefig_haproxy_1 0 B 7df01566911e barkerd427/cafezvous:0.4 "httpd -DFOREGROUND" About a minute ago Up 59 seconds 0.0.0.0:49171->80/tcp websitefig_cafezvous_1 2 B 0f7c971a41aa barkerd427/dbdevs:0.4 "httpd -DFOREGROUND" About a minute ago Up About a minute 0.0.0.0:49170->80/tcp websitefig_dbdevs_1 2 B e02f0cdc37b1 barkerd427/danpluslaura:0.20 "httpd -DFOREGROUND" About a minute ago Up About a minute 0.0.0.0:49169->80/tcp websitefig_danpluslaura_1 2 B $ fig logs Attaching to websitefig_cafezvous_1, websitefig_dbdevs_1, websitefig_danpluslaura_1 danpluslaura_1 | [Mon Dec 01 14:30:59.367252 2014] [mpm_event:notice] [pid 1:tid 140438754260864] AH00489: Apache/2.4.10 (Unix) configured -- resuming normal operations danpluslaura_1 | [Mon Dec 01 14:30:59.367462 2014] [core:notice] [pid 1:tid 140438754260864] AH00094: Command line: 'httpd -D FOREGROUND' dbdevs_1 | [Mon Dec 01 14:30:59.708416 2014] [mpm_event:notice] [pid 1:tid 140529524463488] AH00489: Apache/2.4.10 (Unix) configured -- resuming normal operations dbdevs_1 | [Mon Dec 01 14:30:59.708595 2014] [core:notice] [pid 1:tid 140529524463488] AH00094: Command line: 'httpd -D FOREGROUND' cafezvous_1 | [Mon Dec 01 14:31:00.034189 2014] [mpm_event:notice] [pid 1:tid 140501628454784] AH00489: Apache/2.4.10 (Unix) configured -- resuming normal operations cafezvous_1 | [Mon Dec 01 14:31:00.034919 2014] [core:notice] [pid 1:tid 140501628454784] AH00094: Command line: 'httpd -D FOREGROUND' cafezvous_1 | 172.17.0.25 - - [01/Dec/2014:14:31:15 +0000] "GET / HTTP/1.1" 200 18045 danpluslaura_1 | 172.17.0.25 - - [01/Dec/2014:14:31:19 +0000] "GET / HTTP/1.1" 200 24280 dbdevs_1 | 172.17.0.25 - - [01/Dec/2014:14:31:26 +0000] "GET / HTTP/1.1" 200 11154
We can then bring everything down by using the command fig stop. We can restart them all by using fig up again, which will actually reuse the containers from the previous run unless there has been an update. In fact, you don't even have to call fig stop to make an update. Simply run fig up and containers with an update will be updated.
Currently this is a good solution for my often unvisited websites, but if I made a serious push to make a business out of these sites, then this solution would not suffice. Currently, I cannot add another website container for cafezvous to scale its ability to handle additional workloads. This is something that is definitely needed, and I will post my experiences in the future with Serf and hopefully Kubernetes. These systems will allow for dynamically growing my services if needed.