Author: Ramandeep Bhasin
Introduction
This blog will help you to learn about the Project Creation and Integration of AWS and Snowflake with Matillion. You can create your own project according to your project needs, as explained in the blog. Before integrating AWS and Snowflake with Matillion, let's briefly learn about Snowflake, AWS, and Matillion.
What is Snowflake?
Snowflake is a great tool that enables data storage, data processing, and data analytic solutions, which are easier to use and more flexible than traditional offerings. It is a cloud-based technology that provides data warehousing solutions. It is a truly Software-as-a-Service (SaaS) platform that can be used as a Data Engineering tool, Data Lake, and Data Warehouse. It can also be used for building data-related applications, as we know that sharing of data is necessary in the data world. So Snowflake has data-sharing capabilities. We can share the data with other snowflake accounts or by creating reader accounts. It has a multi-cluster and shared data architecture which is highly scalable. For loading data into Snowflake, one can use various ELT tools which are available in the market. One of the ELT tools is Matillion which is also available in the Snowflake Partner Connect.
What is Matillion?
Matillion is an ELT tool that can be used to load data into Snowflake or any other data warehouse. It is used for extracting data from data sources and loading it into Snowflake. It can also be used for transforming data so that one can generate meaningful insights from the transformed data. It is available in two versions. One is Matillion Data Loader, and another is Matillion ETL. Matillion Data Loader is available free of cost, and Matillion ETL is not a free offering. To learn more about Matillion visit: www.matillion.com
What is AWS?
AWS stands for Amazon Web Services. It is a cloud platform that offers many cloud-based services. There are a total of 84 availability zones within 26 regions. It offers a free tier for up to 400 credits then AWS will charge you based on Pay-as-you-go Model. For more information about pricing visit: www.aws.amazon.com/pricing/
Create a New Project by connecting AWS and Snowflake
Now we need to create a new project. We need to integrate our AWS and Snowflake account with a new project in our Matillion. We also need to create a new Environment. Follow the below steps to integrate your AWS and Snowflake accounts with the new project.
Step 1: After Sign In to the Matillion account. Click on the Project Menu available on the top left corner of the User Interface.
Step 2: Then Click on Switch Project. After Clicking on the Switch Project, a new dialog box will appear. Now click on the Create Project button available on the button left of the dialog box.
Step 3: After clicking on the Create Project button, a new dialog box will appear. Fill in the Project Group, Project Name, and Project Description according to your project requirements. Then Click on the Next button available on the right button as shown below.
Step 4: After clicking on the Next button, a dialog box will appear. Now click on the Manage button, and a new dialog box will appear.
Step 5: Now Click on the + (Plus) button, and a new dialog box named Create AWS Credential will appear. Fill in the Credential Name, Access Key ID, and Secret Key of your AWS user. Choose the Encryption type according to your need and then click Ok.
Step 6: Now choose the AWS credential name that you have just created, and then Click Ok.
Step 7: After choosing AWS credential, now fill in the Environment Name that you want to create and then click on the Next Button, which is available on the right button of the Create Project dialog box.
Step 8: After clicking on the Next button, fill in the Snowflake Connection details. In Account, fill the Snowflake account URL from Account Locator up to Cloud Provider Name. Fill in the Username and Password of your Snowflake user that you want to connect to Matillon. Then Click on the Next button available on the bottom right of the dialog box.
Step 9: After Clicking on the Next button, choose the Snowflake Defaults by clicking on the drop-down of Default Role, Default Warehouse, Default Database, and Default Schema. You will see all the available roles, warehouses, databases, and schemas for the user that you have filled in the previous step. If you were not able to see those defaults by clicking on the drop-down, then check your credentials once again. If the credentials are right, then check the role (attached to the user) privileges. After that, click on the Finish button available on the right button as shown below.
Conclusion
The Integration of AWS and Snowflake with Matillion is completed. Now you are able to access the objects of the snowflake for which we have privileges. You are also able to access different AWS services such as SQS, SNS, CloudWatch, and many more.
Reference
https://documentation.matillion.com/docs/2819274