Author: Rupesh Neve
Introduction:
A few months ago, Snowflake released support for Snowpark in a public preview. Now DBT has come up with native support of python with the beta 1.3 version of the DBT core in which you can write a python model similar to your SQL models.
DBT is a tool that helps to perform transformations with the help of SQL statements. In the recent release of DBT core, DBT has extended the functionality of python, which makes use of Snowpark in the backend. We can directly make .py files in the model's directory similar to .sql files and can perform the transformations using python support. One of today's most well-liked tools for data transformation is DBT. And up to this point, DBT has only supported SQL-based transformations. However, since DBT version 1.3 was released, it is now feasible to build models in DBT that are based on both Python and SQL.
Why DBT core:
DBT core is open source, and one can use it freely.
We can write more flexible and customizable models and can easily deploy them.
Along with SQL, it works with python (Snowpark).
When we write models using python, we can easily manipulate data as we get it in a data frame.
Why DBT core with Snowpark:
You can address the problems using DBT Python models (Snowpark) that SQL cannot.
You can perform analyses using tools available in the open-source Python ecosystem, including the packages for data science and statistics.
Python-based DBT models are easily possible by Snowflake’s new Python support and Snowpark API for Python (Snowpark Python, in short).
When users work with Snowpark python (with DBT python models), one can leverage the following functionalities:
1) Python (DataFrame) ie, numpy, pandas
2) Python Scalar User-Defined Functions (UDFs)
3) Python UDF Batch API (Vectorized UDFs)
4) Python Table Functions (UDTFs)
5) Python SPs
6) Anaconda repository packages.
With Snowflake's Snowpark Python capabilities, you no longer need to maintain, secure and pay for separate infrastructure to run and maintain Python code, as it can now be run directly within Snowflake’s platform.
Conclusion:
In this blog, we discussed some points why one should use DBT core with Snowpark python.