Pigz inside python - Reading compressed .gz file much faster

[jsmith7279]

Hello Pythoners-

I am a linux admin. And one of our users were wondering on how to make the below script faster using pigz or any other multi-threading methods. I have no idea regarding python. Can someone please share on how to make the below part a little bit faster? She said it currently takes around 45minutes to parse on compressed .gz file that is 1GB in size.

if infile.endswith(".gz"):
data = gzip.open(infile, 'rb')
else:
data = open(infile, "r")
outfile = infile.split(".txt")[0] +"_step1.gz"
outdata = gzip.open(outfile, "wb")

## take line by line
for line in data:
line1 = line.rstrip()
if line.startswith("@"):
....
....
....
....
....
outdata.close()
data.close()
print ">Output file: "+ outfile # end of run

Thank you. This is not a homework task. This is a biology lab's problem.

[Larz60+]

you can use python magic
Although this module is in PyPi,  the name conflicts with other packages of the same name, so you have to download and install the wheel
To do this:

• Get the wheel from PyPi as follows
  
• go to:  https://pypi.python.org/pypi/python-magic/
  
• Download the wheel file (Current version): python_magic-0.4.15-py2.py3-none-any.whl
  
• change directory to one containing wheel
  
• from command line, install with:

  pip install python_magic-0.4.15-py2.py3-none-any.whl

Once you have that package installed, use the following code to find file type:

def check_filetype(filename):
    f = magic.Magic(mime=True, uncompress=True, filename)
    return f.from_file(filename)

This will avoid having to load entire zip file.
It will return a string of type:

Output:
'text/plain'

See the documentation here: https://github.com/ahupp/python-magic

[jsmith7279]

Hey Larzo60+

Thanks friend. We are okay with memory. The bottle neck is the read and write speeds which is where the time is being wasted. Do you still know if python-magic helps in those areas?

[Larz60+]

No, let me give you a sample for reading the files ... Be back soon

Please answer this. What is the goal of reading a zip file in this way.
There may already be a package that does what you're trying to do  in record time.

Example (built into python) see: https://docs.python.org/3.6/library/gzip.html

[jsmith7279]

I honestly don't know. I was asked for help to make it faster. Decided to ask someone who knows the left and right of python. I have no clue @Larzo60+.

Thanks

[Larz60+]

Hard to write something without knowing what the goal is.

[nilamo]

http://aripollak.com/pythongzipbenchmarks/

Looks like the speed depends pretty heavily on which version of python you're running.  You might also gain some improvement by wrapping the gzip object in io.BufferedReader.

I wouldn't mind seeing more of your code, though, as 45minutes for 1gb sounds excessive.  Depending on what you're doing (and the power of the computer it's running on), maybe we can create a process queue and take advantage of multiple cores/processors.