From 38f8e2e0fc0137970d79af30f0685fd6b5a4fd49 Mon Sep 17 00:00:00 2001 From: Marten Henric van Kerkwijk Date: Wed, 28 Feb 2024 17:26:12 -0500 Subject: [PATCH 1/4] Cycle 4 funding: uncertainties, masks, quantities, other types of arrays --- ...vk-uncertainty-masked-units-development.md | 91 +++++++++++++++++++ 1 file changed, 91 insertions(+) create mode 100644 finance/proposal-calls/cycle4/mhvk-uncertainty-masked-units-development.md diff --git a/finance/proposal-calls/cycle4/mhvk-uncertainty-masked-units-development.md b/finance/proposal-calls/cycle4/mhvk-uncertainty-masked-units-development.md new file mode 100644 index 00000000..7e62ca2b --- /dev/null +++ b/finance/proposal-calls/cycle4/mhvk-uncertainty-masked-units-development.md @@ -0,0 +1,91 @@ +### Title + +Support for Uncertainties, Masks, Improved Quantities, Other Types of Arrays + +### Project Team + +Marten van Kerkwijk + +### Project Description + +I request partial buy-out from my professorship at UofT to be able to work one +day a week on projects that are too large for the time I can currently commit +for astropy. Specifically, I propose, + +- To enhance Masked and Distribution such that they can be used in all the + main astropy classes (Time, Representation, Frame, and SkyCoord). +- Use the new numpy dtype machinery to deal with units, thus speeding up units + conversions and facilitating Quantity becoming a container class that can + handle not just ndarray but any type of array, i.e., also dask, jax, etc. +- Extend the same machinery to Masked and Distribution so that all main astropy + classes can use arbitrary array classes. +- Introduce a new Variable class that tracks uncertainties and their correlations + analytically (based on ideas from the uncertainties package). + +### Project / Work + +Currently, I spent about a day per week on astropy core, in reviews, bug +fixes, and development. While I have managed to use extra time for fairly +large developments (Quantity historically and Masked and Uncertainty more +recently, with also fairly major contributions to Time, Table, Representation +and numpy), it has been difficult to find enough time to actually wrap up +larger projects (at least outside sabbaticals). + +In particular, while I found time to enable the use of Masked with Quantity +and Time, the logical extension to coordinates is still missing. Similarly, +Distribution now works well with Quantity, but not with Time and the various +classes underlying coordinates. Solving this will allow masks and (implicit) +error propagation on all astropy core classes. Furthermore, an attempt to +introduce a Variable class that tracks uncertainties and covariances has been +stalled for almost a decade. All these require focussed time. + +An exciting development at numpy has been the new dtype machinery, which +allows much easier design of user data types. So far, the main application +has been a new StringDType, which allows having an array of variable-length +unicode strings (I have been a major reviewer of this, partially to get +familiar with the new machinery; it may be useful for astropy too). + +One of the explicit use cases for the dtype redesign was to support units +(which can be seen as descriptions of how to interpret the data, with +converting to another unit similar to casting to another data type). This +seems one of the easiest ways to reduce the intricate dependencies on ndarray +and remove many of the overrides of its methods. Our unit system itself is +nicely separated out, which should facilitate using it for a new unit-carrying +dtype. Benefits of using this include speed-up (as much more will be done in +C), and removal of quite tricky code to deal with, e.g., structured data +types. I also have some hope of separating out the units/quantity code from +astropy, so that it can be used more generally. + +Using unit-carrying data types should also make it easier (though is not +required) for Quantity to support other array classes (dask, jax, etc.), as +suggested in APE 25. This will help deal with larger data sets (dask) and gain +us GPU acceleration (jax). The nice things is that if Quantity is able to use +other array types than ndarray, then this will nearly automatically extend to +coordinates (since those use quantities almost exclusively). A bit more of an +obstacle will be Time, though there a user dtype to hold the two parts of the +JD (or indeed a proper quad-precision float) may help similarly: make the +implementation a lot cleaner, and allow other array types than ndarray. Also, +once Quantity is done, it will be easy to extend it to Masked and Distribution +(and possibly Variable), as those are basically container classes already. + +I should perhaps add that the different projects can be separated relatively +easily, and do not have a very obvious order. Hence, I can give priority to +whatever is deemed most important. + +### Approximate Budget + +I request funding to replace salary equivalent to one day a week, reducing my +regular employment at the University of Toronto correspondingly. At a +standard rate of USD 150/hour (which happens to be roughly my current salary) +for 8 hours per week and 45 weeks, this corresponds to USD $54000 per year. + +Note: so far I have done nothing beyond asking my Chair whether a reduction, +including of teaching, might be possible in principle. I will ask for details +if this proposal is deemed interesting. + +### Period of Performance + +Ideally, this would be for three years (if funding allows and my university +agrees), but the projects are sufficiently separable that a shorter term or +one split in, say, half-year parts is useful too (but not fewer hours per +week, since the goal is to have full days for astropy development only). From c7868fce35203b1abc2ee86ef098718d7acb8e71 Mon Sep 17 00:00:00 2001 From: Marten van Kerkwijk Date: Tue, 12 Mar 2024 09:30:05 -0400 Subject: [PATCH 2/4] Update work description, stressing it is implementation of existing plans --- ...vk-uncertainty-masked-units-development.md | 54 +++++++++++-------- 1 file changed, 33 insertions(+), 21 deletions(-) diff --git a/finance/proposal-calls/cycle4/mhvk-uncertainty-masked-units-development.md b/finance/proposal-calls/cycle4/mhvk-uncertainty-masked-units-development.md index 7e62ca2b..478804e6 100644 --- a/finance/proposal-calls/cycle4/mhvk-uncertainty-masked-units-development.md +++ b/finance/proposal-calls/cycle4/mhvk-uncertainty-masked-units-development.md @@ -12,15 +12,16 @@ I request partial buy-out from my professorship at UofT to be able to work one day a week on projects that are too large for the time I can currently commit for astropy. Specifically, I propose, -- To enhance Masked and Distribution such that they can be used in all the - main astropy classes (Time, Representation, Frame, and SkyCoord). +- To ensure Masked and Distribution can be used in all the main astropy classes + (Time, Representation, Frame, and SkyCoord). - Use the new numpy dtype machinery to deal with units, thus speeding up units - conversions and facilitating Quantity becoming a container class that can - handle not just ndarray but any type of array, i.e., also dask, jax, etc. + conversions. +- Facilitate Quantity becoming a container class that can handle not just + ndarray but any type of array, i.e., also dask, jax, etc. - Extend the same machinery to Masked and Distribution so that all main astropy classes can use arbitrary array classes. -- Introduce a new Variable class that tracks uncertainties and their correlations - analytically (based on ideas from the uncertainties package). +- Finish my implementation of a Variable class that tracks uncertainties and + their correlations analytically (based on the uncertainties package). ### Project / Work @@ -35,9 +36,13 @@ In particular, while I found time to enable the use of Masked with Quantity and Time, the logical extension to coordinates is still missing. Similarly, Distribution now works well with Quantity, but not with Time and the various classes underlying coordinates. Solving this will allow masks and (implicit) -error propagation on all astropy core classes. Furthermore, an attempt to -introduce a Variable class that tracks uncertainties and covariances has been -stalled for almost a decade. All these require focussed time. +error propagation on all astropy core classes. Furthermore, an +[PR](https://github.com/astropy/astropy/pull/3715) to introduce a Variable +class that tracks uncertainties and covariances (based on the [uncertainties +package](https://pythonhosted.org/uncertainties/), but extended it to deal +natively with arrays), has been stalled for almost a decade. All these +require focussed time to finish the implementation, writing tests, documenting +proper usage, etc. An exciting development at numpy has been the new dtype machinery, which allows much easier design of user data types. So far, the main application @@ -58,15 +63,22 @@ astropy, so that it can be used more generally. Using unit-carrying data types should also make it easier (though is not required) for Quantity to support other array classes (dask, jax, etc.), as -suggested in APE 25. This will help deal with larger data sets (dask) and gain -us GPU acceleration (jax). The nice things is that if Quantity is able to use -other array types than ndarray, then this will nearly automatically extend to -coordinates (since those use quantities almost exclusively). A bit more of an -obstacle will be Time, though there a user dtype to hold the two parts of the -JD (or indeed a proper quad-precision float) may help similarly: make the -implementation a lot cleaner, and allow other array types than ndarray. Also, -once Quantity is done, it will be easy to extend it to Masked and Distribution -(and possibly Variable), as those are basically container classes already. +proposed in [APE 25](https://github.com/astropy/astropy-APEs/pull/91). This +will help deal with larger data sets (dask) or use GPU acceleration(jax). +The [APE 25 report](https://github.com/nstarman/astropy-APEs/blob/units-quantity-2.0/APE25/report.pdf) +lays out in detail how this could work. My proposal here is to implement it, +write proper tests, ensure there are no performance regressions, and of course +document it all. A nice benefit of the approach laid out in APE 25 is that it +will be very easy to extend it to Masked and Distribution (and possibly +Variable), as those basically are already the type of container classes that +APE 25 envisions. + +A nice things of Quantity being able to use other array types than ndarray is +that this will nearly automatically extend to coordinates (since those use +quantities almost exclusively; I foresee little more work than adjusting +tests!). Time will be slightly more work, as it works directly with ndarray, +but also here the path is straightforward: I can just follow my earlier work +on ensuring Time can work with Masked. I should perhaps add that the different projects can be separated relatively easily, and do not have a very obvious order. Hence, I can give priority to @@ -79,9 +91,9 @@ regular employment at the University of Toronto correspondingly. At a standard rate of USD 150/hour (which happens to be roughly my current salary) for 8 hours per week and 45 weeks, this corresponds to USD $54000 per year. -Note: so far I have done nothing beyond asking my Chair whether a reduction, -including of teaching, might be possible in principle. I will ask for details -if this proposal is deemed interesting. +Note: While I have asked my Chair whether a reduction, including of teaching, +might be possible in principle. I will ask for details if this proposal is +deemed interesting. ### Period of Performance From db9924e6a17f18865de1fa53175581dfa5522c6b Mon Sep 17 00:00:00 2001 From: Marten van Kerkwijk Date: Tue, 12 Mar 2024 09:30:53 -0400 Subject: [PATCH 3/4] Update statement about practical aspects of buy-out --- .../cycle4/mhvk-uncertainty-masked-units-development.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/finance/proposal-calls/cycle4/mhvk-uncertainty-masked-units-development.md b/finance/proposal-calls/cycle4/mhvk-uncertainty-masked-units-development.md index 478804e6..737af9c1 100644 --- a/finance/proposal-calls/cycle4/mhvk-uncertainty-masked-units-development.md +++ b/finance/proposal-calls/cycle4/mhvk-uncertainty-masked-units-development.md @@ -91,9 +91,9 @@ regular employment at the University of Toronto correspondingly. At a standard rate of USD 150/hour (which happens to be roughly my current salary) for 8 hours per week and 45 weeks, this corresponds to USD $54000 per year. -Note: While I have asked my Chair whether a reduction, including of teaching, -might be possible in principle. I will ask for details if this proposal is -deemed interesting. +Note: I have confirmed with my Chair that a reduction, including of teaching, +is possible in principle, but am still in the process of finding out how this +would work in practice. ### Period of Performance From 3e40ef7143de3be5b8dc6a4dad89a198af245eac Mon Sep 17 00:00:00 2001 From: Marten van Kerkwijk Date: Tue, 26 Mar 2024 09:42:52 -0400 Subject: [PATCH 4/4] Remove half-year option (would not give teaching relief) --- .../cycle4/mhvk-uncertainty-masked-units-development.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/finance/proposal-calls/cycle4/mhvk-uncertainty-masked-units-development.md b/finance/proposal-calls/cycle4/mhvk-uncertainty-masked-units-development.md index 737af9c1..0323a5e5 100644 --- a/finance/proposal-calls/cycle4/mhvk-uncertainty-masked-units-development.md +++ b/finance/proposal-calls/cycle4/mhvk-uncertainty-masked-units-development.md @@ -98,6 +98,6 @@ would work in practice. ### Period of Performance Ideally, this would be for three years (if funding allows and my university -agrees), but the projects are sufficiently separable that a shorter term or -one split in, say, half-year parts is useful too (but not fewer hours per -week, since the goal is to have full days for astropy development only). +agrees), but the projects are sufficiently separable that a shorter term is +useful too (but not fewer hours per week, since the goal is to have full days +for astropy development only).