-n parameter fails to cleanly close samtools (v1.9) pipe

[jfy133]

When using the following command:

samtools calmd -b <mybam>.bam hs37d5.fa | samtools view -h - | pmdtools --deamination --range 10 --UDGhalf  -n 10000 > <mybam>.cpg.range.txt 

The following error occurs

samtools view: writing to standard output failed: Broken pipe
samtools view: error closing standard output: -1
[E::bgzf_flush] File write failed (wrong size)
samtools calmd: failed to write to output file: Broken pipe
[E::bgzf_close] File write failed

However the txt file is still created.

When removing -n parameter, the command completes correctly.

Versions: samtools 1.9
PMDtools version 0.60 (installed via bioconda)

[Note than pmdtools --version for this install still reports 0.5]

[pontussk]

Thank you James. This is because -n 10000 will exit PMDTOOLS when it has processed 10000 reads, but SAMTOOLS in the step before in the pipe doesn't know that all is OK. It may exist, but I am not aware of a way to around this. All should be fine in terms of your workflow though. I also recommend trying the --platypus option that separates CpG sites if you are using UDG-treated data, as the CpG dinucleotides retain almost the entire deamination signal. Best, Pontus

…

On Wed, Dec 18, 2019 at 1:53 PM James A. Fellows Yates < ***@***.***> wrote: When using the following command: samtools calmd -b <mybam>.bam hs37d5.fa | samtools view -h - | pmdtools --deamination --range 10 --UDGhalf -n 10000 > <mybam>.cpg.range.txt The following error occurs samtools view: writing to standard output failed: Broken pipe samtools view: error closing standard output: -1 [E::bgzf_flush] File write failed (wrong size) samtools calmd: failed to write to output file: Broken pipe [E::bgzf_close] File write failed However the txt file is still created. When removing -n parameter, the command completes correctly. Versions: samtools 1.9 PMDtools version 0.60 (installed via bioconda) [Note than pmdtools --version for this install still reports 0.5] — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#7?email_source=notifications&email_token=ABO76I4IYK3XDN44LFIPHALQZITMJA5CNFSM4J4LKU52YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4IBLCZSA>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABO76I52QBSJUOEEJKRV7FLQZITMJANCNFSM4J4LKU5Q> .

[jfy133]

Hi Pontus,

Thanks for the explanation. I actually found the error when running at a (naïve) test of nf-core/eager. The error caused the process to recieve a non 0 exit status, thus causing the entire pipeline to fail.

If there isn't a easy workaround for you, I'll modify nf-core/eager to ignore 143 errors for the pmdtools process(es).

Noted for --platypus! I'll look into it.

Cheers,
James

[pontussk]

Thanks James. Let me know if I can provide any more information. I am very positive about the nf-core/eager initiative, and would be happy to help in any way. I also have an option to estimate the deamination rate just on the first base with a standard error (--first) and which can also be on CpG sites to make it work on any UDG treated data (--CpG). Best, Pontus

…

On Thu, Dec 19, 2019 at 8:17 AM James A. Fellows Yates < ***@***.***> wrote: Hi Pontus, Thanks for the explanation. I actually found the error when running at a (naïve) test of nf-core/eager. The error caused the process to recieve a non 0 exit status, thus causing the entire pipeline to fail. If there isn't a easy workaround for you, I'll modify nf-core/eager to ignore 143 errors for the pmdtools process(es). Noted for --platypus! I'll look into it. Cheers, James — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#7?email_source=notifications&email_token=ABO76I437KJN6EESRYT4MSLQZMUZTA5CNFSM4J4LKU52YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEHIZX4I#issuecomment-567385073>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABO76IYJFBS6RH7KBUJJXR3QZMUZTANCNFSM4J4LKU5Q> .

[jfy133]

Thanks Pontus!

Really what we need now is people actually testing the pipeline to try and break it, and maybe given guidance of current 'best practise' settings for the various tools (like pmdtools), as I'm not familiar with the more-pop gen focused tools and Alex has been out of the field for a while.

In fact I'm encountering another error which I'm not sure about: nf-core/eager#324 . If you could have a look that would be great.