Banner 9 and Docker
19 Oct 2022
Important Banner-ecosystem servers such as Jobsub, the Banner database server, the coordination server for "Docker containers" all have an "ESM Jenkins agent" installed onto them that lets our server hosting "Ellucian Solution Manager" ("ESM") manage files and run code on those servers.
ESM facilitates installing upgrades from Ellucian onto these servers.
Unlike Jobsub and the Banner database server, the coordination server for "Docker containers" doesn't actually do anything meaningful in between Banner upgrades/maintenance. It's just a server we log into and do work on when it's time for upgrades or maintenance. However, the way we use this server when we use it is complex and interesting.
The world wide web exists -- this much we know is true
Let's work backwards from a running Banner-ecosystem website like "BannerAdmin" or "Banner 9 Employee" or "Banner 9 Student" or "Ethos."
Obviously, it exists on the internet -- you can visit it with your web browser!
ECS runs web servers from ECR
So ... How did this website get there?
Well, using an AWS (Amazon Web Services) configuration management tool called Terraform, we told AWS's Elastic Container Service ("ECS") that we'd love it if it would just have a look at all of the "Docker images" (think of them kind of like computer backups, for webservers) that we've stored inside of AWS's "Elastic Container Registry" ("ECR") and, as needed, constantly create and kill webservers from these "Docker images".
e.g.
- We might give AWS ECS our blessing to create extra "Banner 9 Student" web servers during the heavy load of registration, and then auto-scale down.
- We might give AWS ECS our blessing to spin up a fresh webserver and kill off an old webserver if a webserver starts to look unhealthy -- like running out of memory.
The nifty thing about a well-designed piece of web server software it doesn't store data -- that's the job of the database that it talks to -- so it's okay just ... vanishing clean off the internet.
- (That is, as long as there's an identical web server doing the work in its place. Amazon comes with some pretty smart "load balancers" to manage the details of how to ensure websites never entirely disappear off the internet during this churn of server creation and destruction.)
Cool.
How we get Docker images into ECR
So, how do we get our Docker images into ECR? Is it Git or something?
Nope, not quite.
For each of our "Banner 9" websites, Sidi wrote a helper codebase that includes a file called ".../docker/push.sh
". This script's "true home" is in an AWS CodeCommit Git repository that you might or might not be allowed to see.
Sidi also makes sure to keep a copy of that Git repository synced to the Docker coordination server so that, for example, all we have to do to put a given website's freshly-built Docker image into ECR is log into the Docker coordination server and execute the "/software/docker/b9_(some_website_name_here)/docker/push.sh
" script corresponding to the website.
push.sh
" for a website by using Terraform to explicitly tell it to destroy all running servers and start over afresh.)Sidi's script is not Git. However, it might kind of remind you of Git a little bit in that you take something "you have" and give it to "the cloud" by running a command with the word "push" in it.
How we obtain Docker images
So ... how exactly did we get "our copy" of this Docker image that we just pushed into ECR? Did Ellucian give it to us?
No, we're not all the way back to what Ellucian gives us just yet. We're heading there, but not there.
For each of our "Banner 9" websites, Sidi wrote a helper codebase that includes a file called ".../docker/build.sh
". This script's "true home" is in an AWS CodeCommit Git repository that you might or might not be allowed to see.
Sidi also makes sure to keep a copy of that Git repository synced to the Docker coordination server so that, for example, all we have to do to "build" a Docker image (out of parts Ellucian gave or told us) is log into the Docker coordination server and execute the "/software/docker/b9_(some_website_name_here)/docker/build.sh
" script corresponding to the website.
When run, "build.sh
" looks through "/software/docker/(some_website_name_here)/app/
" for files of the filename pattern "(some_website_name_here)-(some_version_number_here).war
". If it finds several, it picks the one with the largest version number and ignores the rest.
When run, "build.sh
" combines the contents of:
- The "
/software/docker/b9_(some_website_name_here)/app/(some_website_name_here)-(some_version_number_here).war
" file.- This is a copy of the file Ellucian "gave" us with ESM, but ESM isn't how it got here. Stay tuned!
- Some other files in "
/software/docker/b9_(some_website_name_here)/
" (like "Dockerfile
" and "entrypoint.sh
"). These files do things like:- Specify that the "Docker image" being built is going to need to include software called Apache Tomcat that specializes in making computers behave like web servers.
- How did we know what to put there?
- Sidi read a lot of directions from Docker's website.
- Sidi read a lot of directions and release notes for each of Ellucian's products, one by one, to figure out how exactly it likes to be configured.
- Some sensitive files of the filename pattern "
/esmshare/(some_banner_environment_here)/ban9/shared_configuration/(some_website_name_here)_(some_purpose_here).(some_file_extension_here)
".- How did we know what to put here?
- Ellucian usually gives us copies of sample files through ESM (though ESM isn't how this file got here -- Sidi handmade these files based off of samples).
- Sidi read a lot of directions and release notes for each of Ellucian's products.
- Sidi read a lot of directions and release notes for some of the 3rd-party software Ellucian rolled into their "
.war
" files (things like "Grails" that make it easier for vendors like Ellucian to write code intended to run on a web server), since some of these sensitive files have a lot more to do with that 3rd-party software than with anything Ellucian-specific.
- How did we know what to put here?
Notes:
- Files found in "
/software/docker/
" are all "clean" (nothing sensitive stored in them) and their "true home" is up in an AWS CodeCommit Git repository. - Files found in "
/esmshare/(some_banner_environment_here)/ban9/shared_configuration/
" potentially contain sensitive data and do not exist in AWS CodeCommit. Their "true home" is right here in this folder (don't worry -- there's some backup for the folder).
How we prep a WAR file for Docker Build
Whoever is in charge of installing Ellucian's upgrades onto UST's servers will be the one to actually click buttons in ESM (Ellucian Solution Manager).
- When certain ESM buttons get clicked, ESM performs work on our Jobsub and Banner database servers.
- (Sometimes there is also followup work to be done on those servers that is written in release notes authored by Ellucian.)
- Note: particularly on our 2nd, 3rd, or 4th round of installing a given set up updates via ESM (e.g. our production go-live after multiple test environments), doing such "release-notes" followup work and validating that ESM did the work correctly can usually be done without worrying about whether Banner-ecosystem websites are up and running again.
- (This is handy for letting ODS work proceed against a freshly-updated Banner database even before website updates go into effect -- in fact, technically, there are probably reasonably safe ways to validate ESM's work in a Banner database alone, and to give that go-ahead even before having ESM do work in Jobsub.)
- Also occurring when those certain ESM buttons get clicked is that ESM dumps files into "
/esmshare/(some_banner_environment_here)/ban9/war/(some_website_longer_name_here)_(some_version_number_here)/current/dist/(some_website_name_here).war
" on the Docker coordination server.- This file-dump by ESM into "
/esmshare/*.war
," is the bottom of our journey backwards from "we know internet exists ... how did it get there?"
Let's bridge the final gap from where we last left off to "/esmshare/*.war
":
- This file-dump by ESM into "
Before we run "build.sh
", we just run a Linux "cp
" (copy) command. We copy the ESM-dropped file into the folder that "build.sh
" reads:
cp /esmshare/(some_banner_environment_here)/ban9/war/(some_website_longer_name_here)_(some_version_number_here)/current/dist/(some_website_name_here).war /software/docker/(some_website_name_here)/app/(some_website_name_here)-(some_version_number_here).war
And, ta-da!
As promised, we've arrived at the part of our "journey working backwards from visiting a website" that explains how a website update makes its way from "where Ellucian drops it" to "where we need it."
/software/docker/b9_(some_website_name_here)/app/
" and Git-track all changes in that folder up to AWS CodeCommit before running "build.sh
".
How we prep a configuration or sensitive file for Docker Build
Honestly? "The hard way."
We tend to presume that, from version to version that Ellucian releases, nothing meaningful has changed about what Elluican's ".war
" file expects of its corresponding "Dockerfile
," "/esmshare/(some_banner_environment_here)/ban9/shared_configuration/*
," etc.
Most of the time, we're right.
For each "Banner 9" website, when there's an upgrade, the following process usually works nicely to bring the website up to date:
- Hand-copy the "
.war
" file that ESM put onto one part of the Docker coordination server's filesystem into another part of the Docker coordination server's filesystem. - (Skipped for now -- this is where the "oops" part goes if we discover we did need to edit other files, too.)
- Commit that new "
.war
" file to the cloud with Git. - Hand-run "
build.sh
" to stitch together a "Docker image" out of the ".war
" file and its friends. - Hand-run "
deploy.sh
" to push that "Docker image" up to AWS ECR.
(Hmmmm. Did I make a typo here? Should this say "push.sh
" or something? Did I make the typo earlier? Did I forget to document a shell script?) - Go over into Terraform and tell AWS ECS to destroy all existing servers and start anew from the latest "Docker image" in AWS ECR.
- Visit the website we expect AWS ECS to be responsible for keeping online with a web browser and validate that it looks normal.
- Party.
- Repeat for all "Banner 9" websites for which we know Ellucian delivered us updated "
.war
" files when Rich ran ESM.
We don't really worry about whether we need to edit the "and friends" files like "Dockerfile
" or "/esmshare/(some_banner_environment_here)/ban9/shared_configuration/*
," unless step 7 -- visiting the website -- fails.
This is not as sketchy of a process as it might sound -- remember that we do this several times in nonproduction environments before we do it in a production environment.
By the time we're going into production, we won't be guessing anymore what edits we might need to make to the "and friends" files, and we won't skip that step, because we'll know it was an issue in the nonproduction environments.