Authors: Nikhil Pai J, Krushik Reddy, and Nikhil Ranade
Introduction
Snowflake’s data-sharing ability is a one-of-a-kind feature that lets you share data between Snowflake or non-Snowflake customers in real time. The Data-sharing ability is mainly leveraged by business owners or Data administrators to seamlessly share the data with their consumers. There is a certain amount of SQL and Snowflake knowledge that is required in order to set up a Snowflake Direct data share and it becomes more complex as the use case widens. This is where we bring in an In-house developed Streamlit-based application that overcomes this complexity of having to write SQL queries and configure Shares using the Snowflake’s UI into a single UI-based no-code application for all things Direct Data Sharing.
Introduction to ShareMon
ShareMon is an Interactive web application used to automate and ease the processes of Snowflake's Direct data sharing and monitoring. Its UI allows you to create and maintain shares without having to write a single line of code. This no-code Application is developed to ease the process of writing long & repeated SQL codes on Snowflake to share Data with your consumers. It works on Streamlit - a Python-based UI framework and Snowflake connectors at the back end to interpret the user's choices.
Pictorial representation of ShareMon
Personas
ShareMon offers various functionalities that can help various people and teams across the organization. Some of the key personas who could be ShareMon’s primary users are:
Administrators - All kinds of administrators (general admins, IT admins, DB admins, etc) would like to create and configure the data shares themselves rather than giving control of the data to the development teams to ensure there are no loopholes.
Data Owners/ Business Stakeholders - Data owners and Business leaders who are impacted by the budgets or IT spending, would like to understand the utilization of the data that has been shared across multiple reader accounts. They would also like to monitor the computing and storage costs if they are looking to monetize their data.
How does ShareMon work
As discussed earlier, ShareMon is a Streamlit-based web application that lets you create and monitor your data shares. Streamlit is an open-source Python-based library that makes it easy to create and share beautiful, custom web apps that can be seamlessly integrated with Snowflake. We used a Snowflake connector for connecting Streamlit with Snowflake. The Snowflake connector dynamically passes the credentials to validate and connect to the Snowflake account.
We have used the below snippet as a connector:
snowflake.connector.connect(user=f"{username}",password=f"{password}",account=f"{url}")
What can we do with Sharemon?
Creating a new share
This page will allow the user to choose an existing warehouse or create a new warehouse as per the user’s choice. It will allow the user to create an empty share that includes the share name and comment. This share can be used in further steps to add shareable objects to it and share it with a consumer of your choice.
Adding Snowflake-supported shareable Objects to the Share
This page allows the user to add any shareable objects to the selected share of his choice. The list of shareable objects includes
Tables
External tables
Secure views
Secure materialized views
Secure UDFs
The user will be able to select one Database per share as per Snowflake’s data-sharing rules. Multiple Schemas can be chosen from the selected Database and other snowflake objects from within the selected Schemas.
Similarly, we can drop the unwanted objects from the share or even we can drop the entire share by selecting the drop objects radio button.
Adding Consumers to the Share
This page will allow the user to add full accounts and reader accounts as consumers to the shares. To add a full account as the consumer, you either need to validate the URL or account locator, cloud, and region.
Reader Account Initial Setup
The user can complete the initial set-up of a reader account on this page. Select a reader account from the drop-down menu to log in, then provide the username and password for the same account. Once you have successfully logged in, you can set up a new role, user, and warehouse and be able to create resource monitors on the account.
Monitoring the shares
The user can choose an Outbound share of his choice and obtain details about it on the Share Monitor page, including
The database associated with the share.
Consumer accounts associated with the share.
If the consumer account is a Full/Reader account.
Date the share was created.
Deployment
Since ShareMon is a Python code at the end of the day, you can deploy this on any cloud or web hosting platform like AWS (EC2) or Heroku, enter your Snowflake credentials, and start using the application. The only thing you have to keep in mind is that there is a set of pre-requisite scripts that you need to run in your Snowflake account before you can start using the application.
Future
We will be looking to add more KPIs on the monitoring side so that data owners can get a clearer picture of the consumption of their data
We will be looking to add some KPIs from the consumer’s perspective as well
We will analyze the historical trend of data usage and see if we can suggest some recommendations and actions based on that
