Authors: Prasad Mistary & Anusree KJ
Introduction
This blog describes how to create repositories and add code in AWS codecommit for code deployment. It also provides a detailed explanation of how to integrate codecommit with Jenkins, along with a brief description of how to create and add objects or files to the S3 bucket in AWS. Let’s start building the pipeline step by step.
Contents
AWS CodeCommit repositories for Versioning.
AWS S3 Bucket for Versioning of ML Model.
Integrating AWS CodeCommit and S3 with Jenkins.
AWS CodeCommit repositories for versioning.
Repository creation in a code commit
Open the CodeCommit console at AWS CodeCommit from AWS Services > Developer Tools > CodeCommit
Use the region selector to choose the AWS Region where you want to create the repository.
On the Repositories page, choose to Create repository.
On the Create repository page, in Repository name, enter a name for your repository (for example, Treble_Configuration).
Choose Create
Once created, the created repositories can be viewed in the CodeCommit:
Adding code in repository
Select the repository to which code has to be added or uploaded.
In the navigation bar for the repository, opt for Code.
Choose Add file, then opt for whether or not to make a file or transfer a file from your pc.
To create a file, do the following:
In the drop-down list of branches, opt for the branch wherever you would like to feature the file. The default branch is chosen mechanically for you. If you would like to feature the file to a special branch, opt for a special branch.
In the File name, enter a reputation for the file. Within the code editor, enter the code for the file.
To upload a file, take the following steps:
Opt for add file and click Choose file.
Then choose the file you want to upload.
Add Email address and Author name before committing (for both create or upload code)
In the Author name, enter the name you would like presented to different repository users.
In the Email address, enter an email address.
(Optional) within the Commit message, enter a quick message. Although this is often nonmandatory, we have a tendency to advocate that you simply add a commit message to assist your team members in perceiving why you are adding this file. If you are not entering a commit message, a default message is employed.
Now Click Commit changes.
To browse the repository
From Repositories, choose the repository by clicking on the repository name.
The page displays the contents within the default branch of your repository.
To view another branch or to look at the code at a selected tag, opt for the branch or tag you wish to look at from the list.
In the following screenshot, the read is about the most branch.
S3 BUCKET for adding KNN MODEL PICKLE file
AWS S3 bucket is used for object storage. The KNN model for choosing the best players is dumped into a pickle file, and the same is stored in AWS S3 treblepickle bucket.
S3 Bucket creation
Following are the steps to create an S3 bucket in AWS:
1. log in to AWS console
2. Choose S3 from AWS Services
3. Choose Create Bucket
4. In the General Configuration, the bucket name and AWS region are required to be given.
5. On Object possession, opt for either ACLs enabled or ACLs disabled as per demand.
All public access is blocked by selecting Block all public access from Block Public Access settings for this bucket.
6. Choose change or disable from Bucket Versioning to either change or disable versioning of the bucket. If disabled, the bucket versioning is enabled when creation.
Choose to change or disable Server-side Encryption from Default Encryption.
7. In Advanced settings, Enable Object Locks. Enabling Object Lock automatically enables the bucket versioning as we want to version the ML Models.
Adding files or objects to S3 bucket
Following are the steps to add objects to the bucket:
1. Log in to the AWS console and select S3 from AWS Services. Select the bucket name to which the object has to be added. (In the figure, treblepickle bucket is selected)
2. Opt for Create Folder or Upload to Whether to create or upload a file to the bucket.
3. To create a folder, choose Create Folder and give a name to the folder.
4. Keep or Change the Server-side encryption depending on the requirement and choose Create Folder.
5. For uploading a folder, Click Upload and choose to add a file or add a folder to upload a file or folder, respectively.
6. Opt for the folder or file to be added and click upload
7. Once the upload is complete, the uploaded file or folder can be seen in the bucket.
INTEGRATING AWS CODECOMMIT WITH JENKINS
After Installing Jenkins and Setting up Repository, we need to configure AWS CLI:
In cmd, enter the Jenkins folder:
cd ~jenkins
sudo -u jenkins aws configure
Go with the defaults for the AWS access key and AWS secret access key;
For the region and output, the format is mentioned below.
AWS Access Key ID [None]: Press Enter
AWS Secret Access Key [None]: Press Enter
Default region name [None]: sort us-east-1 here, and so press Enter
Default output format [None]: sort json here, and so press Enter
After Configuring Git to use IAM credentials and an HTTP path to access the repositories situated in AWS CodeCommit.
sudo -u jenkins git config --global credential.helper '!aws codecommit credential-helper $@'
sudo -u jenkins git config --global credential.useHttpPath true
sudo -u jenkins git config --global user.email "me@mycompany.com"
sudo -u jenkins git config --global user.name "MyJenkinsServer"
Start Jenkins.
sudo service jenkins start
sudo chkconfig jenkins on
Configure global security
Goto Jenkins home Website (http://<public DNS name of EC2 instance>:8080) in your browser.
Select Manage Jenkins and Configure Global Security.
Select the Enable Security check box.
Under Security Realm, choose the Jenkins’ own user information radio button.
Uncheck the checkbox and Allow signing up to the users.
Under Authorization, choose the Logged-in users will do something with the radio button.
Configuring the Git plugin
Opt for Manage Jenkins and Manage Plugins
On the Available tab, check with the Filter box to get the Git Plugin
Choose the Install check box beside to Git Plugin
Select Download now and restart after installation.
Now that the Dev environment is ready and the Jenkins server is set up, change the source in the local repository and push the change to the main repository situated in AWS CodeCommit.
On your workstation, modify the directory to a local repository and form a branch where the changes will be made.
cd DemoRepo
Choose any editor to modify the code, and then save the file in the DemoRepo directory.
Initialize the following git commands to commit and push your change.
git add HelloWorld.java
git commit -m "Modified HelloWorld.java for scenario 1"
git push origin
After a few minutes, go to the Jenkins home page. A build option can be seen.
In the Last Success section, choose the build (shown here as #1). This will take you to the build phase of output. Select Console Output to see the build details.
After the files have been stored on AWS CodeCommit and Configured with Jenkins. Open Jenkins on your public IP mentioned in the EC2 Instance.
Click on the project name and then click on the Build now button.
Click on Console Output to view your linked files running.
Conclusion
Hence, the AWS CodeCommit used for versioning of the source code is orchestrated with AWS S3, which is used for versioning the Machine Learning Model(Artifacts) using Jenkins Pipeline.