The problem
writeSamples() can remove samples from a channel. It can also join the wrong samples together. It gives no error and no warning.
This happens when the length of a signal does not agree with its sample frequency. The new warning from #320 does not catch this case.
How to see the problem
import numpy as np
import pyedflib
from pyedflib import highlevel
# two channels at 100 Hz, but one has 150 samples and the other has 350
sigs = [np.arange(150, dtype=float), np.arange(1000, 1350, dtype=float)]
sheads = [highlevel.make_signal_header(f'ch{i}', sample_frequency=100,
physical_min=-1, physical_max=2000)
for i in range(2)]
f = pyedflib.EdfWriter('ragged.edf', 2, file_type=pyedflib.FILETYPE_EDFPLUS)
f.setSignalHeaders(sheads)
f.writeSamples(sigs)
f.close()
r = pyedflib.EdfReader('ragged.edf')
print(len(r.readSignal(1)))
print(np.round(r.readSignal(1)[95:106]))
r.close()This is the output:
200
[1095. 1096. 1097. 1098. 1099. 1250. 1251. 1252. 1253. 1254. 1255.]
I wrote 350 samples to channel 1, but the file has only 200. The samples 1100 to 1249 are not in the file. The signal goes directly from 1099 to 1250, so there is a gap of 150 samples in the middle of the data.
The cause
Two errors work together here.
First, the record loop stops as soon as the shortest channel has less data than one full record (edfwriter.py:1241). The other channels can still have more than one record of data at this time.
Second, the code for the last record does two incorrect things (edfwriter.py:1253-1261):
lastSampleInd = int(np.max(data_list[i].shape) - ind[i])
lastSampleInd = int(np.min((lastSampleInd, smp_per_record[i])))
...
lastSamples[:lastSampleInd] = data_list[i][-lastSampleInd:]
The second line keeps a maximum of one record and throws away all the data after it. The last line then reads from the end of the array. It must continue at position ind[i]. This is why the file gets the last 100 samples and not the correct ones.
buffered=True has the same cause at _flush_sample_buffer() (edfwriter.py:1375). It keeps the samples in the correct sequence, but it loses the same 150 samples.
More information
If the lengths agree with the sample frequencies, the result is correct. I tested 100 Hz for 3 s together with 200 Hz for 3 s. The maximum error was 0.023, which is only the quantization step.
highlevel.write_edf() has the same problem. It makes sure that there is one signal header for each signal, but it does not look at the lengths.
Suggestion
writeSamples() must compare the lengths before it writes anything, and it must raise an error if they do not agree. An error is much better than a file with wrong data in it.
The problem
writeSamples()can remove samples from a channel. It can also join the wrong samples together. It gives no error and no warning.This happens when the length of a signal does not agree with its sample frequency. The new warning from #320 does not catch this case.
How to see the problem
This is the output:
I wrote 350 samples to channel 1, but the file has only 200. The samples 1100 to 1249 are not in the file. The signal goes directly from 1099 to 1250, so there is a gap of 150 samples in the middle of the data.
The cause
Two errors work together here.
First, the record loop stops as soon as the shortest channel has less data than one full record (
edfwriter.py:1241). The other channels can still have more than one record of data at this time.Second, the code for the last record does two incorrect things (
edfwriter.py:1253-1261):The second line keeps a maximum of one record and throws away all the data after it. The last line then reads from the end of the array. It must continue at position
ind[i]. This is why the file gets the last 100 samples and not the correct ones.buffered=Truehas the same cause at_flush_sample_buffer()(edfwriter.py:1375). It keeps the samples in the correct sequence, but it loses the same 150 samples.More information
If the lengths agree with the sample frequencies, the result is correct. I tested 100 Hz for 3 s together with 200 Hz for 3 s. The maximum error was 0.023, which is only the quantization step.
highlevel.write_edf()has the same problem. It makes sure that there is one signal header for each signal, but it does not look at the lengths.Suggestion
writeSamples()must compare the lengths before it writes anything, and it must raise an error if they do not agree. An error is much better than a file with wrong data in it.