Hi! Thanks for creating this project!
I am wondering, have you considered supporting ADBC as a connecting interface towards the different databases? ADBC is relatively new, but read for production imo. I think introducing ADBC would:
- simplify the codebase, and ease the maintenance of the several engine adapters, as they share the similar dbapi interface and have less non-shared dependencies.
- improve the preformance of
fetchdf when moving large amounts of data.
- allow a BYOD (bring your own driver) approach.
Some pointers:
https://arrow.apache.org/adbc/current/python/recipe/postgresql.html#using-pandas-and-adbc
https://docs.adbc-drivers.org/drivers/bigquery/index.html
The difference between for example BigQuery, Databricks and DuckDB Quack and Trino then becomes:
from adbc_driver_manager import dbapi
dbapi.connect(
driver="bigquery" # driver="databricks" or driver="quack" or driver="trino"
db_kwargs={
# these can be passable via the config and vary per database and user requirements
# "uri": "quack://localhost:9494/?token=quack-secret"
},
)
I think it would simplify managing models in multiple databases/datalakes, and enhances/leverage the existing power of SQLGlot even more.
I'd have to dive deeper into the code to figure out the overall impact on the codebase though.
Hi! Thanks for creating this project!
I am wondering, have you considered supporting ADBC as a connecting interface towards the different databases? ADBC is relatively new, but read for production imo. I think introducing ADBC would:
fetchdfwhen moving large amounts of data.Some pointers:
https://arrow.apache.org/adbc/current/python/recipe/postgresql.html#using-pandas-and-adbc
https://docs.adbc-drivers.org/drivers/bigquery/index.html
The difference between for example BigQuery, Databricks and DuckDB Quack and Trino then becomes:
I think it would simplify managing models in multiple databases/datalakes, and enhances/leverage the existing power of SQLGlot even more.
I'd have to dive deeper into the code to figure out the overall impact on the codebase though.