Authors: Nikhil Pai J, Krushik Reddy, and Nikhil Ranade
Introduction
Secure Data Sharing is a way of sharing selected objects in a database from one Snowflake account to other accounts. We can share tables, external tables, secure views, secure materialized views, and secure UDFs.
There are three types of Data sharing:
Direct Data Sharing: The easiest way to share data between Snowflake accounts using secure shares.
Data Marketplace: Useful for sharing or consuming data from third-party vendors.
Data Exchange: Mainly used for data collaboration between a selected group of members.
Business Importance of data sharing:
Data sharing encourages more connection and collaboration between researchers, which can result in important new findings within the field. In a time of reduced monetary investment in science and research, data sharing is more efficient because it allows researchers to share resources.
Data sharing helps:
Ideate different market strategies based on the insights, such as audience targeting to monetize this data.
Generate multiple revenue streams.
Develop multi-party partnerships and collaboration between different organizations.
Get deeper insights into your data by finding intersections with other organizations.
Current challenges in direct data sharing with Snowflake:
Snowflake/SQL knowledge to create and configure the shares.
One has to have some level of understanding of Snowflake or basic scripting knowledge to create and configure the shares. In this case, it becomes really hard for the data owners to create and configure the shares themselves. This makes the creation and configuration of shares go through an IT life cycle where data owners would create different tickets, and the data engineers would perform all the action on the ground. This process is lengthy, and data owners have little control over their data.
Snowflake UI and its limitations.
Snowflake provides a UI with which you can create and configure a share in a no-code manner, but that also has limitations. With the help of the Snowflake UI, we are not able to share all the sharable objects like secure user-defined functions or external tables.
Monitoring of shares.
Monitoring and governance are important aspects of data sharing. With data exchange and data marketplace, Snowflake gives some level of analytics over the data being shared (like Provider Studio to monitor all your listings in the marketplace). But with direct data sharing, Snowflake has no control over the data being shared, and hence it is the provider's responsibility to monitor the shares. Since there is no dashboard or a UI that provides the information about the data being shared, we again have to rely on firing SQL queries to query Snowflake's metadata to get the required information. There is no single place where you will get all the required information about your shared data.
Reader account chargebacks.
When data is shared to a reader account, the provider's Snowflake account bears all costs associated with that particular reader account with respect to the compute consumption or storage consumption. Hence if a provider wants to charge a particular customer for the usage of the reader account, he or she needs the consumption of that reader account for the same. Again this information is not readily available, and we have to dig Snowflake metadata for the same.
To overcome all these challenges, we have developed an application called ShareMon which makes your data-sharing journey smoother.
Introduction to ShareMon:
ShareMon is an Interactive web application used to automate and ease the processes of Snowflake's Direct data sharing and monitoring. Its UI allows you to create & maintain shares without having to write a single line of code. This no-code application is developed to ease the process of writing long and repeated SQL codes on Snowflake to share data with your consumers. It works on Streamlit - a Python-based UI framework, and Snowflake connectors at the back end to interpret the user's choices.
Features & Functionalities of ShareMon:
Create a new Snowflake direct data share
Create a warehouse for data-sharing activities
Add any shareable objects to the share
Create a secure view to add to the share
Add consumers to existing or newly created shares (Full/Reader account)
Provide temporary access on share to consumer (Full/Reader account)
Set up a newly created Reader account for the DDS.
Drop objects from the share
Monitor the consumers attached to each share
Monitor the shares that were attached to the consumer
Monitor the shares by compute cost & storage cost
Benefits of using ShareMon over the Snowflake UI
Configuration of the Snowflake Share in a no-code manner.
The configuration and monitoring of shares which includes creating, updating, and dropping shares and monitoring of shares, are all done without writing a single line of code. The ShareMon UI consists of clickable and selecting options to bring out the desired outcome compared to the Snowflake UI, which would require SQL code to be written and executed by the user. Data owners can now have complete control over their data and can make the necessary changes whenever required.
Providing Efficient chargeback for Reader accounts.
Reader accounts are an extension of the Snowflake account they are created from, which means all the costs incurred on the Reader account is borne by the main Snowflake account. The storage cost associated with sharing data with them, the compute cost incurred when querying the shared data, and the type of query being used on the shared data become critical when it comes to monitoring. ShareMon allows users to monitor all these Reader account usage aspects in easily interpretable charts and tables.
Governance on Shared Data
ShareMon provides a 360-degree view of your Snowflake account's entire Direct Data sharing ecosystem. All Sharing activities, including monitoring, are centralized in a single application, from creating and sharing secure views to consumer account validation for sharing.
No more Swivel chair.
Direct data sharing using Snowflake is a blend between using the Snowflakes worksheet for writing SQL queries and moving back to the UI for configuring the shares. For a non-Snowflake user, this could be quite inconvenient. ShareMon overcomes this by providing a single UI with no code, specially meant for admins or business users who can rely on ShareMon's UI to configure and monitor the shares without having to move to and from writing SQL statements and the UI.
Conclusion
In conclusion, Sharemon bridges the gap between using Snowflake's worksheets/UI and the requirement to have SQL and Snowflake knowledge to create, manage and monitor the shares. It is a one-stop solution to direct data-sharing needs with an intuitive UI. The application can be easily navigated through and provides additional features that a direct data share might require without having to write complex stored procedures, tasks, or queries.