-
Notifications
You must be signed in to change notification settings - Fork 32
feat: add decoding as a multi-threaded CLARA engine #1229
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
baltzell
wants to merge
37
commits into
development
Choose a base branch
from
decoder-engine
base: development
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
37 commits
Select commit
Hold shift + click to select a range
9b0c279
add convenience method
baltzell 4b1b28a
remove unused stuff
baltzell 2c68f8b
add decoding engine
baltzell 8d3d469
add it to the clara yaml
baltzell f87b747
use a pool
baltzell 90581ae
hmm
baltzell a33c1bb
allow decoder instances to share ConstantsManagers
baltzell 64de21e
share ConstantsManagers
baltzell 1c6c280
kludge test
baltzell b94fbe1
cleanup
baltzell 67d4b55
try this
baltzell 588b65d
inherit ConstantsManagers
baltzell 9b79dad
cleanup
baltzell 283e1cf
only check tables if not shared
baltzell 83095a4
higher ports on macos, cleanup process dpe process
baltzell 6dc2216
remove ineffective pid trap
baltzell 3ee7e84
Revert "higher ports on macos, cleanup process dpe process"
baltzell 71b4055
Revert "fix job name"
baltzell 8be927f
Revert "decouple ubuntu/macos builds to reduce wait"
baltzell 11e6b6f
Revert "remove example engine"
baltzell 412040a
restore reported data type
baltzell 9f22d1f
rename class
baltzell eabd5f0
undo
baltzell 4c54cd6
fix rebase oops
baltzell 1fab062
rename class
baltzell 8b1dc56
fix rebase oops
baltzell 842bf16
stf
baltzell 90a186f
just events, not per thread
baltzell fcdd018
reduce constants sharing
baltzell c87118a
bugfix
baltzell a566963
share one RCDBManager
baltzell 855a4da
bugfix
baltzell f9bbfda
restore
baltzell 6373bbc
DecoderEngine: just pass along HIPO events
baltzell b4fdd59
cleanup, avoid class variable
baltzell e613971
remove debugging leftover
baltzell 396cf92
revert to hard-coded byte order
baltzell File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Some comments aren't visible on the classic Files Changed page.
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
113 changes: 113 additions & 0 deletions
113
common-tools/clas-reco/src/main/java/org/jlab/clas/reco/DecoderEngine.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,113 @@ | ||
| package org.jlab.clas.reco; | ||
|
|
||
| import java.util.Set; | ||
| import java.util.HashSet; | ||
| import java.nio.ByteBuffer; | ||
| import java.nio.ByteOrder; | ||
| import java.util.concurrent.ArrayBlockingQueue; | ||
| import java.util.concurrent.BlockingQueue; | ||
| import org.jlab.clara.base.ClaraUtil; | ||
| import org.jlab.clara.engine.Engine; | ||
| import org.jlab.clara.engine.EngineData; | ||
| import org.jlab.clara.engine.EngineDataType; | ||
| import org.jlab.clara.engine.EngineStatus; | ||
| import org.jlab.detector.decode.CLASDecoder; | ||
| import org.jlab.io.evio.EvioDataEvent; | ||
| import org.jlab.io.hipo.HipoDataEvent; | ||
| import org.jlab.jnp.hipo4.data.SchemaFactory; | ||
| import org.json.JSONObject; | ||
|
|
||
| /** | ||
| * | ||
| * @author baltzell | ||
| */ | ||
| public class DecoderEngine implements Engine { | ||
|
|
||
| static final int POOL_SIZE = 64; | ||
| static final Set<EngineDataType> ED_TYPES = ClaraUtil.buildDataTypes( | ||
| Clas12Types.EVIO,Clas12Types.HIPO,EngineDataType.JSON,EngineDataType.STRING); | ||
|
|
||
| SchemaFactory schema; | ||
| BlockingQueue<CLASDecoder> pool; | ||
| int constantsShared = 16; | ||
|
|
||
| public DecoderEngine() { | ||
| schema = new SchemaFactory(); | ||
| schema.initFromDirectory(System.getenv("CLAS12DIR") + "/etc/bankdefs/hipo4"); | ||
| } | ||
|
|
||
| @Override | ||
| public Set<EngineDataType> getInputDataTypes() { return ED_TYPES; } | ||
| @Override | ||
| public Set<EngineDataType> getOutputDataTypes() { return ED_TYPES; } | ||
| @Override | ||
| public EngineData executeGroup(Set<EngineData> set) { return null; } | ||
| @Override | ||
| public Set<String> getStates() { return new HashSet<>(); } | ||
| @Override | ||
| public String getDescription() { return "decoder engine"; } | ||
| @Override | ||
| public String getVersion() { return "1.0"; } | ||
| @Override | ||
| public String getAuthor() { return "baltzell"; } | ||
| @Override | ||
| public void reset() {} | ||
| @Override | ||
| public void destroy() {} | ||
|
|
||
| @Override | ||
| public EngineData configure(EngineData ed) { | ||
| JSONObject json = new JSONObject(ed.getData()); | ||
| pool = new ArrayBlockingQueue<>(POOL_SIZE); | ||
| CLASDecoder d0 = null; | ||
| for (int i=0; i<POOL_SIZE; i++) { | ||
| CLASDecoder d; | ||
| if (i % constantsShared == 0) { | ||
| d0 = new CLASDecoder(); | ||
| if (json.has("variation")) d0.setVariation(json.getString("variation")); | ||
| if (json.has("timestamp")) d0.setVariation(json.getString("timestamp")); | ||
| d = d0; | ||
| } | ||
| else { | ||
| d = new CLASDecoder(d0); | ||
| } | ||
| pool.add(d); | ||
| } | ||
| return ed; | ||
| } | ||
|
|
||
| @Override | ||
| public EngineData execute(EngineData input) { | ||
|
|
||
| EngineData output = input; | ||
|
|
||
| // if it's EVIO, decode it, otherwise just pass it along | ||
| if (input.getMimeType().equals("binary/data-evio")) { | ||
| EvioDataEvent evio; | ||
| try { | ||
| ByteBuffer bb = (ByteBuffer) input.getData(); | ||
| //evio = new EvioDataEvent(bb.array(), bb.order()); | ||
| evio = new EvioDataEvent(bb.array(), ByteOrder.LITTLE_ENDIAN); | ||
| } catch (Exception e) { | ||
| String msg = String.format("Error reading input event%n%n%s", ClaraUtil.reportException(e)); | ||
| output.setStatus(EngineStatus.ERROR); | ||
| output.setDescription(msg); | ||
| return output; | ||
| } | ||
| HipoDataEvent hipo; | ||
| try { | ||
| CLASDecoder d = pool.take(); | ||
| hipo = new HipoDataEvent(d.getDecodedEvent(evio),schema); | ||
| pool.put(d); | ||
| output.setData("binary/data-hipo", hipo.getHipoEvent()); | ||
| } catch (Exception e) { | ||
| String msg = String.format("Error processing input event%n%n%s", ClaraUtil.reportException(e)); | ||
| output.setStatus(EngineStatus.ERROR); | ||
| output.setDescription(msg); | ||
| return output; | ||
| } | ||
| } | ||
|
|
||
| return output; | ||
| } | ||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.