Metadata and test data for new ERT tables at TEMPEST

[bpbond]

This PR adds test data, design table entries, and qaqc table entries for the new ERT data at TEMPEST.

Here is how the tables are processed:

Logger	Table	Action
ERT-84	Teros12Table	Processed
ERT-84	Teros21Table	Processed
ERT-84	Redox5	Dropped
ERT-84	Redox15	Dropped
ERT-84	RedoxTEST	Processed

@roylrich @stephpenn1 Does this look correct to you?

Also, see discussion below regarding how we assign the Instrument_ID and Sensor_ID information.

Closes #474

[bpbond]

Right now the ERT data outputs in this PR look like this:

• Instrument_ID = "ERT"
• "A" or "B" sensors are identified via Sensor_ID

However the existing redox data have outputs that look like this:

• Instrument_ID is used to identify "A" or "B"

Which form do we prefer?

@stephpenn1 @roylrich (anyone else to tag in on this?)

[stephpenn1]

My vote is for option A (Instrument_ID = "ERT"; "A" or "B" sensors are identified via Sensor_ID).

I'm seeing that the existing design table example looks like the below, I'm confused how the existing redox ends up as your second photo?

Logger	Table	loggernet_variable	Site	Plot	Instrument	Instrument_ID	Sensor_ID	Location	valid_through	research_name	note
COMPASS_GCW_TR_432	Redox5	RedoxRa({1:12})	GCW	TR	Redox		{1:12}a			redox-5cm
COMPASS_GCW_TR_432	Redox5	RedoxRb({1:12})	GCW	TR	Redox		{1:12}b			redox-5cm

[bpbond]

Thanks @stephpenn1

I'm confused how the existing redox ends up as your second photo

It varies between sites -- some use the Instrument_ID, but others don't.

[bpbond]

Sorry for the delay; coming back to wrap this up (hopefully).

Here's how we treat the Instrument_ID and Sensor_ID columns across all sites and plots:

   Site  Plot  Instrument Instrument_ID Sensor_ID research_name
   <chr> <chr> <chr>      <chr>         <chr>     <chr>        
 1 GCW   TR    Redox      ""            10a       redox-5cm    
 2 GCW   UP    Redox      "A"           1         redox-10cm   
 3 GCW   W     Redox      ""            10a       redox-5cm    
 4 GWI   TR    Redox      ""            10a       redox-5cm    
 5 GWI   UP    Redox      ""            10a       redox-5cm    
 6 GWI   W     Redox      ""            10a       redox-5cm    
 7 MSM   TR    Redox      ""            10a       redox-5cm    
 8 MSM   UP    Redox      ""            10a       redox-5cm    
 9 MSM   W     Redox      ""            10a       redox-5cm    
10 SWH   SWAMP Redox      ""            10a       redox-5cm    
11 SWH   TR    Redox      ""            10a       redox-5cm    
12 SWH   UP    Redox      ""            10a       redox-5cm    
13 SWH   W     Redox      ""            10a       redox-5cm    
14 TMP   C     Redox      "A"           1         redox-10cm   
15 TMP   F     Redox      "A"           1         redox-10cm   
16 TMP   S     Redox      "A"           1         redox-10cm   

So, things are inconsistent!

Created using

x <- read_L2_variable("redox-5cm", path = "~/sensor_data/Level2/v2-1/")
y <- read_L2_variable("redox-10cm", path = "~/sensor_data/Level2/v2-1/")
bind_rows(x,y) |> group_by(Site, Plot, research_name) |> slice_head(n = 1)

[stephpenn1]

Oof, so if I'm reading this correctly, we're even treating within site sensors differently 🥴 I'm about to jump on a meeting, will come back at 2pm with my standardization suggestions!

[stephpenn1]

Options I'm seeing:

Instrument_ID: ERT
Sensor_ID: 10a, 10b, etc

OR

Instrument_ID: ERT-1 or 1 (for non ERT sensors)
Sensor_ID: a, b, etc

Any thoughts on which would be easier to parse in code?

[bpbond]

I think they're equivalent in terms of parsing via code; but your second option is more intuitive to me. It seems clearer and cleaner.

[stephpenn1]

Agree - let's proceed with option 2 for redox at all sites!

[stephpenn1]

Hey @bpbond, I've reviewed your ERT design table additions.

I think there's a typo in the Instrument column for redox A/B, and I've moved the research names in the Redox5 and Redox15 tables to the notes section, and moved the research names in the RedoxTEST table to the research_name column. Both are fixed in 2fb4477. Please review!

[bpbond]

@stephpenn1

1. Your design table edits had a mistake starting in line 3246 (the research names were pasted into the valid_through column). Fixed in 04d3653
2. "I've moved the research names in the Redox5 and Redox15 tables to the notes section" -- what's the point of this? I'm confused. If the intention is to drop those tables, why not simply remove their research names? If you can clarify that would be helpful.

Thank you!

[stephpenn1]

what's the point of this?

This was how the tables we're already set up (with the research names in the notes for the RedoxTEST tables) so I continued that format for the other Redox tables in case you made that format for a reason

[bpbond]

Oh okay, sorry I missed that. Let me take another look with fresh eyes.

[roylrich]

Yes from me Sent from my T-Mobile 5G Device Get Outlook for Android<https://aka.ms/AAb9ysg>

…

________________________________ From: Ben Bond-Lamberty ***@***.***> Sent: Wednesday, 26 August 2026 10:40:17 To: COMPASS-DOE/sensor-data-pipeline ***@***.***> Cc: Rich, Roy ***@***.***>; Mention ***@***.***> Subject: [COMPASS-DOE/sensor-data-pipeline] Metadata and test data for new ERT tables at TEMPEST (PR #483) External Email - Exercise Caution This PR adds test data, design table entries, and qaqc table entries for the new ERT data at TEMPEST. Here is how the tables are processed: Logger Table Action ERT-84 Teros12Table Processed ERT-84 Teros21Table Processed ERT-84 Redox5 Dropped ERT-84 Redox15 Dropped ERT-84 RedoxTEST Processed @roylrich<https://github.com/roylrich> @stephpenn1<https://github.com/stephpenn1> Does this look correct to you? Also, see discussion below regarding how we assign the Instrument_ID and Sensor_ID information. Closes #474<#474>

________________________________ You can view, comment on, or merge this pull request online at: #483 Commit Summary * e2ad433<e2ad433> Design table entries and metadata for ERT * 9d46c24<9d46c24> Test darta * 1ba52c6<1ba52c6> Update qaqc_table.csv * 2fb4477<2fb4477> Fix Instrument typos, only use RedoxTEST table * 04d3653<04d3653> Fix a column-paste error File Changes (9 files<https://github.com/COMPASS-DOE/sensor-data-pipeline/pull/483/files>) * A pipeline/data_TEST/Raw/Raw_original/ERT-84_CheckTable_short.dat<https://github.com/COMPASS-DOE/sensor-data-pipeline/pull/483/files#diff-984cba0a2b31508af24e7acc429e521c493e3e35df508b4b069d5448721deb56> (14) * A pipeline/data_TEST/Raw/Raw_original/ERT-84_Redox15_short.dat<https://github.com/COMPASS-DOE/sensor-data-pipeline/pull/483/files#diff-5bba1ccaf1ec49cb01c4eb089f2606ea18a4e31febf75952122d7e7f9318d4b9> (15) * A pipeline/data_TEST/Raw/Raw_original/ERT-84_Redox5_short.dat<https://github.com/COMPASS-DOE/sensor-data-pipeline/pull/483/files#diff-5f6f650e857278093a23c14004ed2c3e257fef2c5d6add31358fcd84194233a5> (15) * A pipeline/data_TEST/Raw/Raw_original/ERT-84_RedoxTEST_short.dat<https://github.com/COMPASS-DOE/sensor-data-pipeline/pull/483/files#diff-0596f5cc746a3de1036e549e8260b2a3356b1c8f4918954c50cba31b0969bba7> (13) * A pipeline/data_TEST/Raw/Raw_original/ERT-84_Teros12Table_short.dat<https://github.com/COMPASS-DOE/sensor-data-pipeline/pull/483/files#diff-cdf19cda7fe89e76795acd35d8f163b49519309e987e0aef5005a2f91f883122> (12) * A pipeline/data_TEST/Raw/Raw_original/ERT-84_Teros21Table_short.dat<https://github.com/COMPASS-DOE/sensor-data-pipeline/pull/483/files#diff-7213f7d45ee84ffee5737338ae5c02107963ca623a7b755f281189cb54157d4e> (16) * M pipeline/metadata/design_table.csv<https://github.com/COMPASS-DOE/sensor-data-pipeline/pull/483/files#diff-1de924fbdaa255290778561b717b9350e0412b4321d413d0d3905e39db0f9286> (211) * M pipeline/metadata/qaqc_table.csv<https://github.com/COMPASS-DOE/sensor-data-pipeline/pull/483/files#diff-db8322c55520a82caef003814497007562636b53cb3f3a13e066d6ac82fb5f34> (7) * M pipeline/metadata/variables_metadata.csv<https://github.com/COMPASS-DOE/sensor-data-pipeline/pull/483/files#diff-1b55afad59381a799d0ff479c8e6d53fe088396c66ad119196a4ff46021511f1> (7) Patch Links: * https://github.com/COMPASS-DOE/sensor-data-pipeline/pull/483.patch * https://github.com/COMPASS-DOE/sensor-data-pipeline/pull/483.diff — Reply to this email directly, view it on GitHub<#483?email_source=notifications&email_token=AFIDRBXZWMVJ264NMQ2M2XD5L3ZFDA5CNFSNUABEM5UWIORPF5TWS5BNNB2WEL2QOVWGYUTFOF2WK43UF4ZTQNZZGMYTENJTGOTHEZLBONXW5J3NMVXHI2LPN2SWK5TFNZ2KYZTPN52GK4S7MNWGSY3L>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AFIDRBWV537KFGC2ZZ7OU235L3ZFDAVCNFSNUABFKJSXA33TNF2G64TZHM2DIMRVGIYDQMRXHNEXG43VMU5TINRXHA2DKNRUGUZ2C5QC>. Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS<https://github.com/notifications/mobile/ios/AFIDRBU4WTRUZ2FQO5IMINL5L3ZFDA5CNFSNUABEM5UWIORPF5TWS5BNNB2WEL2QOVWGYUTFOF2WK43UF4ZTQNZZGMYTENJTGOTHEZLBONXW5J3NMVXHI2LPN2SWK5TFNZ2KUZTPN52GK4S7NFXXG> and Android<https://github.com/notifications/mobile/android/AFIDRBT4RK5BHZOIJ3Y3JFT5L3ZFDA5CNFSNUABEM5UWIORPF5TWS5BNNB2WEL2QOVWGYUTFOF2WK43UF4ZTQNZZGMYTENJTGOTHEZLBONXW5J3NMVXHI2LPN2SWK5TFNZ2K4ZTPN52GK4S7MFXGI4TPNFSA>. Download it today! You are receiving this because you were mentioned.Message ID: ***@***.***>

[stephpenn1]

Options I'm seeing:

Instrument_ID: ERT Sensor_ID: 10a, 10b, etc

OR

Instrument_ID: ERT-1 or 1 (for non ERT sensors) Sensor_ID: a, b, etc

Any thoughts on which would be easier to parse in code?

Agree - let's proceed with option 2 for redox at all sites!

Hi @bpbond, a few things I noticed re: the above change, questions are in bold:

1. As you know, the redox IDs are currently in the Sensor_ID column and have the bracket nomenclature (e.g. {1:12}a). With the proposed option 2, we'd be moving the bracket portion to Instrument_ID a la...

Old Nomenclature:

Instrument_ID	Sensor_ID
	{1:12}a

New Nomenclature:

Instrument_ID	Sensor_ID
{1:12}	a
ERT-{1:12}	a

This, I assume, will need a code tweak to parse these in a different column and with the possible "ERT-" prefix. Correct? I wanted to confirm before I made edits to all redox rows

2. There's a rogue row in the design table:

I assume he belongs up with the other ERT tables but I wanted to confirm before I move it.

[bpbond]

I'd like to separate this PR's changes (new TEMPEST ERT tables) from making redox ID columns consistent across all sites and plots, which will be a big change. So I've moved that discussion to #496 and we'll make a separate PR.

OK @stephpenn1 ? If you agree let's approve and merge this and then proceed in #496

[roylrich]

Can you confirm for ERT that we are pulling RedoxC and RedoxD variables?

[bpbond]

@roylrich Table processing is as above #483 (comment)

[stephpenn1]

Can you confirm for ERT that we are pulling RedoxC and RedoxD variables?

Yes @roylrich we are only pulling the RedoxTEST table for now which has all 4

[stephpenn1]

I'd like to separate this PR's changes (new TEMPEST ERT tables) from making redox ID columns consistent across all sites and plots, which will be a big change. So I've moved that discussion to #496 and we'll make a separate PR.

OK @stephpenn1 ? If you agree let's approve and merge this and then proceed in #496

Fine with me @bpbond