Improve usage statistics handling in the background/code

[bozana]

Final PRs:
pkp-lib: #8109
ui-library: pkp/ui-library#213
ojs: pkp/ojs#3465
omp: pkp/omp#1161
ops: pkp/ops#313
plugins/generic/lensGalley: asmecher/lensGalley#60

fixes PRs:
s. #8123
s. #8125

do not use installation migrations in upgrade,
fix CompileMonthlyMetrics job and move it to the pkp-lib:
pkp-lib: #8184
ojs: pkp/ojs#3501
omp: pkp/omp#1180
ops: pkp/ops#331

TO-DOs:

• Define the new log format (s. Improve usage statistics handling in the background/code #6782 (comment))
• Define new data model
  (OJS and OPS: Improve usage statistics handling in the background/code #6782 (comment), and
  OMP: Improve usage statistics handling in the background/code #6782 (comment))
• Implement the Geo data solution (s. issue IP location and institution service #6895)
• Implement the solution for Institution (s. issue IP location and institution service #6895)
• Implement usage event + logging using Laravel event and listener
• Implement new statistics settings (site and journal level)
• Migrate old to new setting
• Migrate data from the old DB table metrics that contain ojs::counter metric type to the new tables, keep other metric types in the table metrics_old
• Solve the migration for the old Region codes
• Implement COUNTER R5 log data processing (s. issue COUNTER Release 5 #6781)
• Implement new DB DAOs
• Implement Issue and Context stats API and services, adapt Publication services
• Adapt all current reports
• Implement Geo stats API and services
• Implement CSV API responses, s. Integrate statistics Custom Report Generator with article statistics UI #7318
• Integrate CSV reports in the UI, s. Integrate statistics Custom Report Generator with article statistics UI #7318
• Consider supp file stats (s. Improve usage statistics handling in the background/code #6782)
• Remove custom report generator
• Remove View Report plugin
• Change ViewReport plugin (s. ViewReport: provide PDF, HTML, Other stats instead of stats for each artilce galley #7384)
• Migrate privacy policy and the opportunity to opt-out logging to a new place in the code (was a block plugin within the usageStats plugin) -- not needed
• Migrate usage stats display and its setting to the default theme (was within the usageStats plugin)
• Copy usage stats display and its setting to all themes (This will be done in a separated issue)
• In the upgrade script: remove usageStats and views report plugin, remove usageStats's scheduled task (in Acron plugin settings)
• Add usageStats folder during installation
• Implement SUSHI and COUNTER R5 reports (s. COUNTER Release 5 #6781)
• Implement a tool for the migration of the old into the new log format (s. issue Tool to convert old and apache log files into new format  #6849)
• Use jobs for log file processing
• Ensure correct monthly log files reprocessing
• Adapt for OMP
• Adapt for OPS
• Check all with PostgreSQL
• Check, add, move, delete locale keys
• release notebook (removed global constants), s. Document usage stats changes for release notebook #7953
• documentation, s. Document new usage stats implementation #8014

Some improvements are wished.
S. also #4904 (comment).

1. Different log format:
   a) Instead or additionally to the URL it would be good to have something like:
   contextID: submissionID: representationID: publicationID: fileID
   b) Consider everything needed for institutional subscription statistics.

2. Additional data model that would at least support aggregation of the old usage stats data, so that they can be removed from the DB table metrics.

[bozana]

s. also this: #4904 (comment)

[bozana]

The thoughts/decisions on the new usage stats log format:

• The entries in JSON format (instead of space separated): this will probably need more processing (insignificantly?), but will allow us to easier add new elements if needed.
• The following elements are minimally needed:

issueId (only in OJS)
chapterId (only in OMP)
time
ip (anonymized)
canonicalURL
contextId
submissionId
representationId
assocType
assocId
fileType
userAgent 
country
region
city
institutionIds
version

time, ip and canonicalURL will be used for double click filtering (COUNTER R5).
time, ip, submissionId and assocType + assocId will be used for unique items (COUNTER R5).
userAgent will be used for robots filtering (COUNTER R5).
contextId, submissionId and representationId are logged so that they do not have to be re-calculated later, when processing the log file, because we already have them, so we can just write/overtake them to the DB table. Later they will be used for statistics aggregation (by journal and submission). representationId is needed for the association with the file (currently having only the file id we cannot determine the representation id in OJS).
assocType and assocId are the actual object that is being accessed/visited/used.

[bozana]

@NateWr, @ctgraham and @asmecher, I have a question regarding the following columns in the DB table metrics:
assoc_object_type and assoc_object_id -- those are assoc_type_issue and issue id, only used in OJS, when visiting article, article files, issue toc and issue galley.
pkp_section_id - this is the section id in OJS and OPS, and series id in OMP.
Those are currently only used in reports and not to generate the statistics for those objects.
I am not sure if I should keep them:

• It would be maybe good to keep them, to be on the safe side when doing all the other changes. It would reduce the changes needed for current reports.
• It is maybe better to remove them and keep the DB table metrics small as possible, because we actually do not use them for the statistics aggregation (by them). This way we would not need to care when e.g. an article changes the section or the issue. This would mean more changes/adaptations for the current reports.

Depending what we decide, I would eventually consider them in the log format (see above)...

[asmecher]

JSON I/O in PHP is pretty lightning-fast; I don't foresee any issues there.

Do we want to leave room here for an "institution" field when that's available -- ideally (but probably rarely) a ROR?

[NateWr]

I agree with this proposal. The current statistics service class that delivers the visual statistics uses assoc_type and assoc_id to filter by sections and I think this should be sufficient.

Do we want to leave room here for an "institution" field when that's available -- ideally (but probably rarely) a ROR?

Should we pack this into the metrics table? Or would it make sense to have a separate metrics_institutions table that tracked stats by journal/institution? I don't think we'll need/want all of the same level of detail for institutional metrics, like hits-by-issue or by hits-by-submission, will we?

[ctgraham]

COUNTER reporting would totally be interested in downloads by published item by institution (subscriber).

[bozana]

Yes, the institution will definitely come, this is my next step, will then come back to you regarding this again :-)

So, would you agree that we remove assoc_object_type, assoc_object_id and pkp_section_id from DB table metrics?

[asmecher]

those are assoc_type_issue and issue id, only used in OJS, when visiting article, article files, issue toc and issue galley.

Hmm, maybe it would be clearer to just call it issue_id in OJS and log it accordingly? We don't have a use for this in the other apps, and when the assoc_id/assoc_type pairings don't have extra utility, best to avoid them, I think.

[NateWr]

COUNTER reporting would totally be interested in downloads by published item by institution (subscriber).

Wouldn't this be a privacy concern? I thought that libraries were big on not tracking the specific material that someone is reading. It seems to me like individual resource tracking at the institutional level would frequently result in very low counts, which would make them prone to deanonymisation. In many cases, maybe only one person has visited a specific article from an institution.

Also, it's my understanding that institutional reporting exists to assess ROI. But what relationship does ROI have with article-level stats?

[ctgraham]

The "why track at the article level?" is interesting. I was coming at it from the protocol requirements perspective: COUNTER promises a certain set of reports, and institutions are expected to be interested in any of those reports, so the protocol provides for any of the reports to be filtered at the Customer / Consortia Members level. But practically, a customer is probably more interested in the Platform Report or Database Report or Title Report than they are in an Item Report. And, a metrics service is probably more interested in Items, irrespective of customers.

A counter example (no pun intended) where a library would be interested in item level access would be something like EZPAARSE, where proxy logs are post-processed to gather additional metadata for consultation events represented within the proxy log. This isn't related COUNTER reports, but illustrates that sometimes we library folks are interested in (anonymized) usage of very specific things (even digital things).

In terms of the risk of deanonymization, this would be the case if we had a record of who accessed a particular OJS instance, but not what they accessed, but pulled down the COUNTER report and, because of low usage, now could guess at what they accessed. This seems like an unlikely scenario. What data source would I be using to get a record of a specific user accessing a specific OJS site, which isn't also capturing requested URIs?

[NateWr]

This seems like an unlikely scenario.

I think you're right that I am deep into unlikely but worst-case scenarios. I think my concern is not so much on the institution's side -- the institution can probably track all activity on its network. I'm more concerned about OJS as a store of information on time, place and content.

A worst-case scenario I can imagine is in a country where it is dangerous to access certain information. An academic in that country wants to access such articles safely, and uses a proxy set up by a colleague on the Pitt network to read them. As part of an investigation, the country learns about the proxy and acquires the OJS institutional stats (by hacking on the OJS or Pitt side). By comparing records of the academic's requests to the proxy server (which the state has) with records of institutional stats, the state can infer that the academic is accessing those records.

That's probably pretty unlikely, and storing the stats at journal-level instead of article-level may not help much in such cases. But it illustrates how it can be very hard to prevent deanonymization whenever data about time, place and content is joined together.

We can probably weave our way through this in some sense by offering options at the level of institutional configuration. A journal could say "for this institution, track at the [instance/journal/article] level", and the institution could be required to request finer-grained stats if desired.

[ctgraham]

Interestingly, COUNTER is currently soliciting feedback from publishers and institutional users regarding reporting on the level of geographic subdivisions and institutional attribution of OA access.
https://www.projectcounter.org/counter-consultation/