Dashboard API: Add assessment layer calculations

[Bjwebb]

Are these going to take place in dashboard API, or in the frontend?

@markbrough If this is in the dashboard API, I think I'm waiting for a methodology from you.

[markbrough]

Yes, I think this should go in the dashboard API!

[markbrough]

{
  "publisher": {
    "iati_identifier": "US-GOV-1",
    "short_name": "usaid",
    "name": "United States Agency for International Development (USAID)",
    "organisation_type": "10"
  },
  "assessed_at": "2026-09-01T00:00:00Z",
  "activity_count": 12480,
  "overall": {
    "category": "very_good"
  },
  "dimensions": {
    "availability": {
      "category": "excellent",
      "components": {
        "downloadable": {
          "value": true,
          "category": "excellent",
          "detail": {
            "reachable_files": 118,
            "total_files": 118
          }
        },
        "readable": {
          "value": true,
          "category": "excellent",
          "detail": {
            "readable_files": 118,
            "total_files": 118
          }
        },
        "no_errors": {
          "value": true,
          "category": "excellent",
          "detail": {
            "critical_errors": 0
          }
        }
      }
    },
    "timeliness": {
      "category": "excellent",
      "components": {
        "timeliness": {
          "value": 3,
          "category": "excellent",
          "detail": {
            "time_lag": "one_month"
          }
        },
        "frequency": {
          "value": 3,
          "category": "excellent",
          "detail": {
            "frequency": "monthly"
          }
        }
      }
    },
    "coverage": {
      "category": "excellent",
      "components": {
        "coverage": {
          "value": 95,
          "category": "excellent",
          "detail": {
            "iati_spend_usd": 66600000000,
            "reference_spend_usd": 70100000000,
            "reference_source": "oecd_dac_crs"
          }
        },
        "disaggregation": {
          "value": 4,
          "category": "excellent"
        }
      }
    },
    "comprehensiveness": {
      "category": "very_good",
      "components": {
        "organisation_fields": {
          "value": 88,
          "category": "very_good",
          "detail": {
            "numerator": 10982,
            "denominator": 12480
          }
        },
        "basic_fields": {
          "value": 99,
          "category": "excellent",
          "detail": {
            "numerator": 12355,
            "denominator": 12480
          }
        },
        "financial_fields": {
          "value": 79,
          "category": "very_good",
          "detail": {
            "numerator": 9859,
            "denominator": 12480
          }
        },
        "advanced_fields": {
          "value": 58,
          "category": "good",
          "detail": {
            "numerator": 7238,
            "denominator": 12480
          }
        }
      }
    }
  }
}

[markbrough]

This is some sample JSON of what we would want the assessment layer to provide as JSON. NB not sure if the detail section is needed for now, but it could provide useful context so I included it for now.

[markbrough]

Availability:

• downloadable: no files listed here: https://dev2.dashboard.iatistandard.org/errors/download-errors/
• readable: no files with "critical" validation errors on the IATI Validator
• no_errors: no files with "errors" validation errors on the IATI Validator

If all of these criteria are met, the category is: "excellent"

Otherwise, the category is: "needs_improvement"

[markbrough]

Timeliness:

• frequency: https://dev2.dashboard.iatistandard.org/publishing-statistics/timeliness-frequency/
• timelag: https://dev2.dashboard.iatistandard.org/publishing-statistics/timeliness-timelag/

If frequency and timeliness are both monthly / one month, the category is: "excellent"
If frequency and timeliness are both quarterly / one quarter (or if one of them is monthly / one month), the category is: "good"

Otherwise, the category is: "needs_improvement"

[markbrough]

Coverage:

• source for both: https://iati.github.io/dashboard-coverage/

If coverage is over 80% and disaggregation is over 3 (quarters per year), the category is: "excellent"
If coverage is over 60% and disaggregation is over 3 (quarters per year), the category is: "very_good"
If coverage is over 40% and disaggregation is over 2 (quarters per year), the category is: "good"

Otherwise, the category is: "needs_improvement"

[markbrough]

Comprehensiveness:

• source for all: new gherkin tests

If organisation > 75% AND basic > 99% AND financials > 90% AND advanced > 80%, the category is: "excellent"
If organisation > 60% AND basic > 95% AND financials > 70% AND advanced > 60%, the category is: "very_good"
If organisation > 40% AND basic > 90% AND financials > 60%, the category is: "good"

Otherwise, the category is: "needs_improvement"

[markbrough]

Overall assessment:

If all dimensions are excellent, the category is: "excellent"
If all dimensions are very_good or excellent, the category is: "very_good"
If all dimensions are good, very good, or excellent, the category is: "good"

Otherwise, the category is: "needs_improvement"

[markbrough]

TODO:

• instead of detail, we could instead add in the mapping against the gherkin tests keys

[Bjwebb]

Here is my work in progress on the assessment layer (availability is currently missing):

curl -s 'https://dev2.dashboard.iatistandard.org/api/reporting-orgs/fcdo.json/?show_stats=1' | jq .stats.assessment_new

{
  "overall": {
    "category": "needs_improvement",
    "category_number": 0
  },
  "dimensions": {
    "coverage": {
      "category": "very_good",
      "components": {
        "coverage": {
          "value": 0.7057809960962793,
          "category": "very_good",
          "category_number": 2
        },
        "disaggregation": {
          "value": 4.0,
          "category": "excellent",
          "category_number": 3
        }
      },
      "category_number": 2
    },
    "timeliness": {
      "category": "excellent",
      "components": {
        "timelag": {
          "value": 4,
          "category": "excellent",
          "category_number": 3
        },
        "frequency": {
          "value": 4,
          "category": "excellent",
          "category_number": 3
        }
      },
      "category_number": 3
    },
    "comprehensiveness": {
      "category": "needs_improvement",
      "components": {
        "basic": {
          "value": 0.8878342989283866,
          "category": "needs_improvement",
          "category_number": 0
        },
        "advanced": {
          "value": 0.12901694573959907,
          "category": "good",
          "category_number": 1
        },
        "financials": {
          "value": 0.6338050858612424,
          "category": "good",
          "category_number": 1
        },
        "organisation": {
          "value": 1.0,
          "category": "excellent",
          "category_number": 3
        }
      },
      "category_number": 0
    }
  }
}