Git and SourceTree setup with AWS federation on Windows

20 Feb 2019 🔖 git python aws windows
💬 EN

Do you have “single sign-on” for your Amazon Web Services accounts, but need to set up your computer to talk to their CodeCommit git repository hosting?

If you have Windows, Python, and Git installed on your computer, here’s how you can do so without messing up any other plain-old-username-and-password repositories you connect to!

I didn’t figure any of this out – I’m just transcribing it “for dummies”
(i.e. my future self when my computer gets replaced).

Summary

Here’s the guts of what we’re going to do to tell Git how to talk to Amazon:

1. We’ll set up an easy way of putting fresh contents into c:\users\MY_USERNAME\.aws\credentials once an hour
   (that is, we’ll make it easy to run a 300-line Python script dedicated to giving us a nice little “username” & “password” prompt)
2. We’ll tell our Git installation that any repositories hosted at AWS CloudCommit should use the AWS CLI to facilitate login, not a username and password, but to go ahead and use username and password for everything else
3. We’ll install the “AWS CLI” onto our computers and tell Git how to find it when executing the command “aws”

Level: intermediate+

Not everything is covered here. I’m going to presume you have the experience necessary to do the following things on your own and that you have admin access to your computer to facilitate software installation:

1. Use command lines comfortably
2. Set up Git / SourceTree and establish a working relationship between a repository on your hard drive and an ordinary “remote” repository on another machine (when connecting to the “remote” is as simple as entering a username and password as prompted by Git or SourceTree, anyway)
3. Edit your Windows “PATH” environment variables
4. Set up Python on your computer and demonstrate that your computer is capable of executing print('Hello World') (even if just through an IDE like Spyder)
5. Install Python modules you don’t yet have as part of your Python installation with the pip command in whatever command-line environment you installed / configured to be able to run pip (e.g. “Anaconda Prompt”)

Getting the URL for your repository

1. There’s just “N. Virginia” (“East 1”) where all of CodeCommit lives now. So the URL to see your repositories is:
   https://console.aws.amazon.com/codecommit/home?region=us-east-1
2. Although first, to log in, you may have to go to:
   https://SOME-SORT-OF-WEB-SITE-YOUR-COMPANY-GIVES-YOU.com
3. In the left nav, expand “Source” and click on “Repositories”
   https://console.aws.amazon.com/codesuite/codecommit/repositories?region=us-east-1
4. Find a repository you have access to and click it.
   https://console.aws.amazon.com/codesuite/codecommit/repositories/OUR_AWESOME_REPOSITORY/browse?region=us-east-1
5. Far right, near the top of the page but not in the top nav, there’s a button with a drop-down that says “Clone URL.”
   Drop it down and pick “Connection Steps.”
   https://console.aws.amazon.com/codesuite/codecommit/repositories/OUR_AWESOME_REPOSITORY/connect?region=us-east-1
6. If you see a warning at the top of the instructions with the following text, this blog post is for you!

   You are signed in using federated access or temporary credentials. The only supported connection method for these sign-in types is to use the credential manager included with the AWS CLI, as documented below. To configure a connection using SSH or Git credentials over HTTPS, sign in as an IAM user.

7. There should still be a “Clone URL” drop-down in about the same area of the page as it was on the last page.
   Drop it down and pick “Clone HTTPS.”
8. A green alert bar should appear at the top of the page under the top-nav saying:

   Copied

   https://git-codecommit.us-east-1.amazonaws.com/v1/repos/OUR_AWESOME_REPOSITORY

9. Put that URL for your repository somewhere … you’ll need it later.

Install Git & SourceTree on your computer, if you haven’t

Test that your Git setup is working with some “ordinary” remote repository you’ve already had working in the past.

• Use “git fetch --all” with the Git Bash command-line interface, make sure you get prompted for a password, and save
• Edit a post in a readme or a blog, pushing the changes to SourceTree, and make sure they go live

Get all that sorted out if even that doesn’t work.

Edit your Git installation’s “global” GitConfig file

You’re going to need to edit your “global” GitConfig file to include these lines:

[credential]
	helper = 

[credential "https://github.com"]
	helper = 

[credential "https://bitbucket.org"]
	helper = 
	
[credential "https://git-codecommit.us-east-1.amazonaws.com"]
	helper = "!aws codecommit credential-helper "
	UseHttpPath = true

Mine was stored in the C:\users\MY_USERNAME\ folder, and was a file called .gitconfig

Test your git setup with “ordinary” remotes again to make sure you didn’t break anything.

(Close & re-open SourceTree after saving your GitConfig contents and before testing.)

Instead of “helper = ,” you can say “helper = wincred” for GitHub and BitBucket if you don’t want the Git Bash to prompt you for a password every time you do a fetch, push, pull, clone, etc.

If you ever realize you regret storing your password for a given account in the Windows Credential Manager, just type the Windows Key, start searching “manage windo…,” and click the Control Panel app “Manage Windows Credentials.” You can go through the list and delete your credentials against a given repository there.

Getting an hourly token from AWS

Tim Sullivan built a handy Python script that lets you fetch a fresh credential from AWS and dump it onto the part of your hard drive that AWS’s “command line interface” tool looks for such things, called “aws_saml_auth.py.”

(You’ll need to edit the value of idpentryurl on line 34, as the script on GitHub is tailored to the employer he was working for when he wrote the script.)

I don’t have “Python” added to my path, so running a given Python script from my command line looks a bit absurd right now:

c:\users\MY_USERNAME\AppData\Local\Continuum\anaconda3\python.exe c:\users\MY_USERNAME\Desktop\auth_saml\aws_saml_auth.py

If I put Python into my Windows “PATH” environment variable and executed the command from within the same folder that I’d put “aws_saml_auth.py,” I could have simply executed the following command:

python aws_saml_auth.py

Presuming my Python installed has all the modules that “aws_saml_auth.py” refers to installed (look up pip install), running this script on a command-line interface should ask me for my username & password (it’s my single-sign-on credentials for checking my work e-mail … with the “@CompanyName.com” at the end of the username).

If I type the correct values, I’ll see the following output, telling me that there’s a new folder on my hard drive under C:\Users\MY_USERNAME\ called .aws with a file called credentials (no file extension) in it whose contents will be good for signing into Amazon for the next hour (after which, I’ll have to run this Python script again to generate fresh contents for “credentials“):

----------------------------------------------------------------
Your new access key pair has been stored in the AWS configuration file C:\Users\MY_USERNAME\.aws\credentials under the default profile.
Note that it will expire at 2019-02-20 19:35:01+00:00.
After this time, you may safely rerun this script to refresh your access key pair.
If you specified a credential other than default, to use this credential, call the AWS CLI with the --profile option (e.g. aws --profile default ec2 describe-instances).
----------------------------------------------------------------

Note that it says the word “default” in lines 1 and 4 because that’s what “aws_saml_auth.py” says to do in line 87: ????

profile: ("The name of the profile we want to put it the credential file", 'option', 'p')='default',

You’ll need to to run this Python script once an hour or so.

Therefore, you might find it handy to add a command to your command-line environment called “awslogin” as a shortcut to pull up the username & password prompt any time you notice “git fetch,” etc. isn’t working against an AWS-hosted repository that uses “federated” login.

My colleague set it up as a command called “login” that only worked within the “Git Bash” command-line interface by going to C:\Program Files\Git\usr\bin\ and creating a new file in it called “login” (no file extension) whose contents are:

python /usr/bin/aws_saml_auth.py

(In his case, he had saved “aws_saml_auth.py” into C:\Program Files\Git\usr\bin\, and “python” was part of his PATH.)

Installing the AWS CLI so that Git’s attempt to execute the command “aws codecommit credential-helper” actually works

Now we need to install the AWS CLI so that Git’s attempt to execute the command “aws codecommit credential-helper” actually works.

Otherwise, Git Bash / SourceTree will dutifully follow our instructions to “use the command ‘aws’ instead of username and password for AWS CloudCommit remotes,” then “fall back” to prompting for a username & password (which won’t work, because Amazon doesn’t allow that for “federated” logins).

Let’s download the AWS CLI installer, install it, and add “aws” to our PATH environment variable.

In my case, the AWS CLI installer took care of the PATH environment variable for me: it added C:\Program Files\Amazon\AWSCLI\bin\ to the system-wide PATH.

If I open up a standard Windows command prompt (Windows key -> type “cmd” and hit Enter), and if I type “aws --version,” I get an answer about what version of the AWS CLI is installed on my computer.

However, Git still doesn’t quite understand what “aws” is supposed to mean.

I can see that I have this problem because if I open up the “Git Bash” CLI and type “aws --version,” I got an “aws: command not found” complaint.

To fix that, I added a file called “aws” (with no file extension) to C:\Program Files\Git\bin whose contents are:

C:\Program Files\Amazon\AWSCLI\bin\aws.cmd

(Thanks, Stack Overflow.)

I think that’s what made it suddenly start working, anyway.

I’m a little nervous that isn’t it, because now that I’ve gotten it working, if I erase the contents of this file, aws --version still works.

Anyway, as soon as “aws --version” works from the Git Bash CLI, I should be good to go!

1. I have a way of putting fresh contents into c:\users\MY_USERNAME\.aws\credentials once an hour
2. I’ve told Git that any repositories hosted at AWS CloudCommit use the AWS CLI to facilitate login, not a username and password, but to go ahead and use username and password for everything else
3. I’ve installed the AWS CLI onto my computer and told Git how to find it

Detecting and fixing expired credentials

Unfortunately, getting all this set up and blogging about it took so much troubleshooting to figure out, my 1-hour credentials from Amazon expired, so I may need to run “aws_saml_auth.py” from a command line again and update the contents of c:\users\MY_USERNAME\.aws\credentials with a fresh token.

Expired credentials looked like this when doing a git fetch --all in the Git Bash CLI against a repo I’d gotten everything working for earlier:

MY_USERNAME@MY_COMPUTER_ID MINGW64 ~/Desktop/aws_test3/test_repo_3 (master)
$ git fetch --all
Fetching origin
fatal: unable to access 'https://git-codecommit.us-east-1.amazonaws.com/v1/repos/OUR_AWESOME_REPOSITORY/': The requested URL returned error: 403
error: Could not fetch origin

In SourceTree, right-clicking on “origin” beneath “Remotes” in the left-nav and clicking “Fetch from origin” gave me the following error:

git -c diff.mnemonicprefix=false -c core.quotepath=false fetch origin
fatal: unable to access 'https://git-codecommit.us-east-1.amazonaws.com/v1/repos/OUR_AWESOME_REPOSITORY/': The requested URL returned error: 403
Completed with errors, see above.

A successful fetch in the Git Bash looks like this:

MY_USERNAME@MY_COMPUTER_ID MINGW64 ~/Desktop/aws_test3/test_repo_3 (master)
$ git fetch --all
Fetching origin

MY_USERNAME@MY_COMPUTER_ID MINGW64 ~/Desktop/aws_test3/test_repo_3 (master)

And in SourceTree, the progress bar just goes away, unless you have “Show Full Output” checked, in which case the output looks like this:

git -c diff.mnemonicprefix=false -c core.quotepath=false fetch origin
Completed successfully.

Running your first working git command against AWS

Of course, it was a bit presumptuous of me to presume you’d be doing a “git fetch” – that implies you had a remote-local pair set up in the first place. :-)

Remember that URL we got from Amazon waaaaay back at the beginning?

https://git-codecommit.us-east-1.amazonaws.com/v1/repos/OUR_AWESOME_REPOSITORY

Let’s set up 2 local repositories on our computer, each using it as a “remote” – one with the “Git Bash” CLI (or with our Windows CLI if we set it up to be able to execute “git” commands when we installed Git), and the other with SourceTree.

Clone with Git Bash

1. Create a new folder on your hard drive, c:\example\
2. Do cd c:\example in the Git Bash command prompt
3. Do git clone https://git-codecommit.us-east-1.amazonaws.com/v1/repos/OUR_AWESOME_REPOSITORY aws_test_1

The output should look something like this, and you should have a new c:\example\aws_test_1\ folder on your computer. There should be files in it, if there were files in the remote.

MY_USERNAME@MY_COMPUTER_ID MINGW64 /c/example
$ git clone https://git-codecommit.us-east-1.amazonaws.com/v1/repos/OUR_AWESOME_REPOSITORY aws_test_1
Cloning into 'aws_test_1'...
remote: Counting objects: 3, done.
Unpacking objects: 100% (3/3), done.

Clone with SourceTree

1. Click the “+” by the tabs of your open repositories, near the top, and in the big buttons near the top of the “Local Repositories” screen, click the “Clone” button with an icon that looks like an arrow pointing into an open box.
2. In “Source Path / URL,” type: https://git-codecommit.us-east-1.amazonaws.com/v1/repos/OUR_AWESOME_REPOSITORY
3. In “Destination Path,” type: C:\example\aws_test_2
4. In “Name,” type: Best AWS Test 2
5. Click the “Clone” button

If you don’t check the “Show Full Output” checkbox of the progress bar, it might just disappear when done. But if you do, its log will look something like this:

git -c filter.lfs.smudge= -c filter.lfs.required=false -c diff.mnemonicprefix=false -c core.quotepath=false clone --branch master https://git-codecommit.us-east-1.amazonaws.com/v1/repos/OUR_AWESOME_REPOSITORY C:\example\aws_test_2
Cloning into 'C:\example\aws_test_2'...
warning: templates not found in C:\Program Files\Git\share\git-core\templates

remote: 
remote: Counting objects: 0
remote: Counting objects: 3
remote: Counting objects: 3
remote: Counting objects: 3, done.        
Completed successfully.

In the left nav, if you click “Branches” and click “master” (or whatever your branch is called), you should be able to see the full commit-log history from the remote copy of the repository (that is, AWS), presuming there is any history.

You should have a new c:\example\aws_test_2\ folder on your computer. There should be files in it, if there were files in the remote.

CONGRATULATIONS, YOU’RE IN!

Happy git-ting!

Further Resources

• Using CodeCommit and GitHub Credential Helpers by James Wing