Bug in EPW data collections

[tg359]

I'm using Pollination v1.70.1.0 (based on the number associated with the uninstaller), with ladybug-core 0.44.49 and have found a few bugs.

They're probably related, but the two I've stumbled across are:

When saving an EPW, I get this error:

from ladybug.epw import EPW

epw = EPW(r"C:\Users\...\source.epw")
epw.save(r"C:\Users\...\temp.epw")

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[23], line 14
     13 epw = EPW(epw_file)
---> 14 epw.save(r"C:\Users\...\temp.epw")

File .\.venv\lib\site-packages\ladybug\epw.py:1931, in EPW.save(self, file_path)
   1925 def save(self, file_path):
   1926     """Write EPW object as a file.
   1927 
   1928     Args:
   1929         file_path: Text for the full path to where the file will be written.
   1930     """
-> 1931     file_data = self.to_file_string()
   1932     write_to_file(file_path, file_data, True)
   1933     return file_data

File .\.venv\lib\site-packages\ladybug\epw.py:1876, in EPW.to_file_string(self)
   1874 # load data if it's  not loaded convert to SI if it is in IP
   1875 if not self.is_data_loaded:
-> 1876     self._import_data()
   1877 originally_ip = False
   1878 if self.is_ip:

File .\.venv\lib\site-packages\ladybug\epw.py:379, in EPW._import_data(self, import_header_only)
    377     self._import_location(line)
    378     header_lines = [line] + [epwin.readline() for i in xrange(7)]
--> 379     self._import_header(header_lines)
    380 if import_header_only:
    381     return

File .\.venv\lib\site-packages\ladybug\epw.py:513, in EPW._import_header(self, header_lines)
    509     grnd_header = Header(temperature.GroundTemperature(), 'C',
    510                          AnalysisPeriod(), header_meta)
    511     grnd_vals = [float(x) for x in grnd_data[st_ind + 4: st_ind + 16]]
    512     self._monthly_ground_temps[float(grnd_data[st_ind])] = \
--> 513         MonthlyCollection(grnd_header, grnd_vals, list(xrange(12)))
    514     st_ind += 16
    516 # parse leap year, daylight savings and comments.

File .\.venv\lib\site-packages\ladybug\datacollection.py:1359, in __init__(self, header, values, datetimes)
   1357 self._datetimes = tuple(datetimes)
   1358 self.values = values
-> 1359 self._validated_a_period = False

TypeError: descriptor '_header' for 'BaseCollection' objects doesn't apply to a 'MonthlyCollection' object

And the other is when accessing data collections from an SQLiteResult:

from ladybug.sql import SQLiteResult

sql_obj = SQLiteResult(r"C:\Users\...\model\openstudio\run\eplusout.sql")
for outp in sql_obj.available_outputs:
    sql_obj.data_collections_by_output_name(outp)

---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
Cell In[1], line 5
      3 sql_obj = SQLiteResult(r"C:\Users\...\model\openstudio\run\eplusout.sql")
      4 for outp in sql_obj.available_outputs:
----> 5     sql_obj.data_collections_by_output_name(outp)

File c:\Users\...\.venv\lib\site-packages\ladybug\sql.py:371, in SQLiteResult.data_collections_by_output_name(self, output_name)
    369 if report_frequency == 'Hourly' or isinstance(report_frequency, int):
    370     for head, values in zip(headers, all_values):
--> 371         data_colls.append(HourlyContinuousCollection(head, values))
    372 elif report_frequency == 'Daily':
    373     for head, values in zip(headers, all_values):

File c:\Users\...\.venv\lib\site-packages\ladybug\datacollection.py:614, in HourlyContinuousCollection.__init__(self, header, values)
    609 assert header.analysis_period.end_hour == 23, \
    610     'analysis_period end hour of {} must be 23. Got {}'.format(
    611         self.__class__.__name__, header.analysis_period.end_hour)
    613 self._header = header
--> 614 self.values = values
    615 self._datetimes = None
    616 self._validated_a_period = True

File c:\Users\...\.venv\lib\site-packages\ladybug\_datacollectionbase.py:96, in BaseCollection.values(self, values)
     94 @values.setter
     95 def values(self, values):
---> 96     self._check_values(values)
     97     self._values = list(values)

File c:\Users\...\.venv\lib\site-packages\ladybug\datacollection.py:1024, in HourlyContinuousCollection._check_values(self, values)
   1020 """Check values whenever they come through the values setter."""
   1021 assert isinstance(values, Iterable) and not isinstance(
   1022     values, (str, dict, bytes, bytearray)), \
   1023     'values should be a list or tuple. Got {}'.format(type(values))
-> 1024 assert len(values) == len(self.header.analysis_period), 'Length of ' \
   1025     'values does not match that expected by the header analysis_period.'\
   1026     ' {} != {}'.format(len(values), len(self.header.analysis_period))

AssertionError: Length of values does not match that expected by the header analysis_period. 8760 != 8784

[tg359]

I think I've found the culprit of the SQL bug. The IDF used to generate the SQL is using 2024 for its RunPeriod, which was a leap year. So no issue with the LB code it seems. I'm manually setting that value to a non-leap year (or whatever corresponds with the same start day of week and leaped-ness of the input EPW) for now.

Perhaps this is one for honeybee_energy instead, the RunPeriod.from_analysis_period method?

I also downgraded to Pollination 1.68.1.0 and ladybug-core 0.44.46 the EPW().save method works in there still, so hopefully that narrows down where this could have been introduced.

[chriswmackey]

Hey @tg359 ,

I cannot recreate the bug that you're experiencing with epw.save(). It all runs fine with the EPWs in the tests of this repo:

Can you share the EPW that you are using to see if I can recreate the issue?

If it's working with that version of ladybug, I assume that the change which broke your workflows is somewhere in this commit:
736ae83

[chriswmackey]

Also, the SQLite module is supposed to be able to handle SQLs for leap years as you see here in the code:

https://github.com/ladybug-tools/ladybug/blob/master/ladybug/sql.py#L611-L613

I am not sure why the module could not tell that your E+ run period had been for a leap year. If you could give a link to this SQL, I'll be able to figure it out. Otherwise, I'll try to recreate the issue on my end.

[tg359]

Thanks @chriswmackey, Here is the SQL file I used (and the IDF that was created en-route):

• eplusout.sql
• in.idf.txt

These were generated using the hb_model_to_osm method, with run_period from

RunPeriod.from_analysis_period(
    AnalysisPeriod(), start_day_of_week="Monday"
)

... and then run using run_idf.

And the EPW file I was using is this one:

SAU_RI_Riyadh.AB.404380_TMYx.2009-2023.epw.txt

(file extensions changed to .txt to allow upload here)

[chriswmackey]

Thanks, @tg359 . It is odd that I still cannot recreate the bug with the EPW that you supplied:

What version of Python are you using for your scripting? And what Python IDE are you using? I wonder if something has changed in the newer versions of Python in terms of how class attributes are handled.

[tg359]

I'm using a UV managed .venv, with the same target Python version as found in C:\Program Files\ladybug_tools\python\python.exe, which is 3.10.10150.1013. I'm currently on the Pollination 1.68.1.0 installer. The EPW.save bug didn't happen on this one - only with the 1.70 installer, but the SQL bug happened on both for me.

It might be a version mismatch for the EPW.save issue I was having - I can see you're on 3.12. I may have not updated my Python version properly when moving to the 1.70 installer, so that may have caused the EPW.save issue. I'll test that on my side.

[chriswmackey]

Thanks, @tg359 .

The clear description for how you created the SQL issue allowed me to fix the bug here:
ladybug-tools/honeybee-openstudio#73

It happened because E+ changed the RunPeriod object (again). In April, I had to come up with an elaborate workaround to preserve the existing honeybee-energy behavior. Given that the workaround was a lot more complex, there was a bug in it. You can read more here on the forum about why this happened here:

https://discourse.ladybug.tools/t/hb-bug-incorrect-start-day-when-run-period-is-specified/39651/4

I don't know exactly what is going on with your EPW.save() issue but I cannot recreate it in Python 3.10, 3.12 or IronPython 2.7. I feel like it most likely has to do with the virtual environment that you set up. Maybe something about it does not like the fact that __init__ methods are not being called through the inheritance tree between the Monthly/Daily/Hourly collections and the BaseCollection that they all inherit from. I have come to recognize that it is good practice to call the parent __init__ method in the __init__ of children so I am happy to make this change.

But this does not explain why going to an old version of ladybug-core fixes things for you.

[tg359]

Brilliant!

The EPW one may remain a mystery, as I checked my .venv set-up for the newer installer and ended up with the same Python version being used there (3.10 again) - but this time it worked, so I'll blame stray radiation for flipping some bits on my machine for that one (or more likely user-error of some kind).

Thanks for fixing the RunPeriod bug 👍

[chriswmackey]

Ok, I came up with another theory. It seems that I didn't fully understand how __slots__ worked when I set up data collections since the way that I had implemented them was not actually working. The only real consequence of this should have been that we don't get the performance and memory improvement of __slots__ whenever we use data collections. But maybe with the way that some python interpreters implemented __slots__ they throw exceptions when they are not correctly used up the inheritance tree.

If I am right, this PR will fix it:
#657

If I am wrong, well at least the PR still gets __slots__ to work correctly so that we receive the performance benefit from it.