Python Forum
Python script to extract data from API to database
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Python script to extract data from API to database
#1
I'm making a Python script to extract data from some APIs, such as TCGA API, in order to extract it and import it into my SQL database, which tracks types of cancers along with the cancer patients' data (i.e. age, gender, etnicity, type of cancer ecc.) .

The problem relies within the extraction of the data from all of the pages of the API. For example, I've managed to extract data from only one page. Does anyone have some piece of advice to optimize my code? It would be so much helpful.

Here's a piece of my code:

 
import requests
import json
import re

file_id = "b658d635-258a-4f6f-8377-767a43771fe4"

data_endpt = "https://api.gdc.cancer.gov/data/{}".format(file_id)

response = requests.get(data_endpt, headers = {"Content-Type": "application/json"})

response_head_cd = response.headers["Content-Disposition"]

file_name = re.findall("filename=(.+)", response_head_cd)[0]

with open(file_name, "wb") as output_file:
    output_file.write(response.content) 
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Why can't it extract the data from .txt well? Melcu54 4 2,728 Dec-12-2024, 07:36 PM
Last Post: Melcu54
  JSON File - extract only the data in a nested array for CSV file shwfgd 2 2,250 Aug-26-2024, 10:14 PM
Last Post: shwfgd
  Is it possible to extract 1 or 2 bits of data from MS project files? cubangt 8 6,463 Feb-16-2024, 12:02 AM
Last Post: deanhystad
  python script is hanging while calling a procedure in database prasanthi417 4 3,220 Jan-17-2024, 02:33 PM
Last Post: deanhystad
  How to detect abnormal data in big database python vanphuht91 5 3,523 Jun-27-2023, 11:22 PM
Last Post: Skaperen
Question Need help for a python script to extract information from a list of files lephunghien 6 3,659 Jun-12-2023, 05:40 PM
Last Post: snippsat
  Database that can compress a column, or all data, automatically? Calab 3 3,001 May-22-2023, 03:25 AM
Last Post: Calab
  Basic SQL query using Py: Inserting or querying sqlite3 database not returning data marlonbown 3 4,213 Nov-08-2022, 07:16 PM
Last Post: marlonbown
  python Extract sql data by combining below code. mg24 1 2,011 Oct-03-2022, 10:25 AM
Last Post: mg24
  SQL Alchemy help to extract sql data into csv files mg24 1 4,175 Sep-30-2022, 04:43 PM
Last Post: Larz60+

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020