Sep-30-2022, 12:14 PM
Hi Team,
how to download sql table data with header from using SQL Alchemy.
using pyodbc library I have downloaded 60gb data. Using read in chunk
pandas is slow I think,
how to download sql table data with header from using SQL Alchemy.
using pyodbc library I have downloaded 60gb data. Using read in chunk
pandas is slow I think,
from sqlalchemy import create_engine
import pandas as pd
#value to connect server
DRIVER= "ODBC Driver 17 for SQL Server"
server="DESKTOP-GQK64O6"
DATABASE="Customer"
#establish connection
str_conn = f'mssql://{server}/{DATABASE}?Driver={DRIVER}'
connection = create_engine(str_conn).connect()
mylist = []
#Read in Chunk
for chunk in pd.read_sql_table('employee',connection,chunksize=1000):
mylist.append(chunk)
df = pd.concat(mylist)
ndf = pd.DataFrame(df)
ndf.to_parquet("C:\\C_Programming\\output.csv")
