Skip to content

Home

Jump to bottom
Marc Hanheide edited this page May 18, 2022 · 14 revisions

Welcome to the topic_store wiki!

Using convert.py to extract rosbags

Here we are looking at the usecase of extracting data from the LAR mongodb server, on the FirstFleet database (called ff_rasberry), focusing on the topic_store-generated 2021_riseholme_framos_cameras collection. See Example 2021 FirstFleet Document for an example of the document structure.

Installation of topic_store

On supported platforms (like ROS melodic), topic_store is available from the L-CAS ROS distribution, e.g. simply running apt install ros-melodic-topic-store should work. As there are some problems potentially with older shipped version of pymongo that is underpinning topic_store, you are advised to run pip install -U pymongo --user as well.

On other platforms, install topic_store from source.

Make sure to read https://github.com/LCAS/topic_store#installation for further details

Also note, that the convert.py script used below requires a roscore running when generating ros bags, even if it doesn't really use it.

Setting the scene

setup some environment variables first we use in the examples:

# some environment variables, you need a valid user that has read permission on the database
MONGOUSER=reader

# not putting the actual secret here, for obvious reasons... ask for it, if you need it
MONGOPASS=getyoursecretpasswordandputithere

# selecting the database
MONGODB=ff_rasberry

# selecting the collection to go through
COLLECTION=2021_riseholme_framos_cameras

extract ROS bag of a session

Here, for a query on a specific session:

# doing the export with a specific query (-q). Here we chose a dedicated recording session. 
# Note that the server requires SSL/TLS connection. We accept an insecure certificate here.

rosrun topic_store convert.py -i "mongodb://$MONGOUSER:$MONGOPASS@files.lar.lincoln.ac.uk:8098/?authSource=$MONGODB&tls=true&tlsInsecure=true" \
    -c$COLLECTION -q '{"_ts_meta.session":"ObjectId(616800e44e5e826889b3f5dd)"}' \
    -o session_616800e44e5e826889b3f5dd.bag

use "projection" to only extract some data

One can also add a projection (-p), i.e., select which fields should be extracted, here only the laser data:

rosrun topic_store convert.py -i "mongodb://$MONGOUSER:$MONGOPASS@files.lar.lincoln.ac.uk:8098/?authSource=$MONGODB&tls=true&tlsInsecure=true" \
    -c$COLLECTION -q '{"_ts_meta.session":"ObjectId(616800e44e5e826889b3f5dd)"}' \
    -p '{"robot.laser": 1}' \
    -o session_616800e44e5e826889b3f5dd_laser_only.bag

query by date/time (epoch) range

To extract data only from a specific date/time range, convert the start and end time into epochs first using something like https://www.epochconverter.com/. E.g. extracting all data from 1/8/2021 until 8/8/2021, would be

  • 1/8/2021: epoch 1627776000
  • 8/8/2021: epoch 1628380800

So, now we query all documents in that range as follows (has to be greater than ($gt) start epoch and less than ($lt) end epoch. Here, we use projection again, but this time extracting all "robot" data as an example.

rosrun topic_store convert.py -i "mongodb://$MONGOUSER:$MONGOPASS@files.lar.lincoln.ac.uk:8098/?authSource=$MONGODB&tls=true&tlsInsecure=true" \
     -c$COLLECTION -q '{"_ts_meta.sys_time": {"$gt": 1627776000, "$lt": 1628380800}}' \
     -p '{"robot": 1}'     -o robot.bag

export mongodb to local topic_store

Rather than into a rosbag, you can also export form the DB server to a local topic store that can then be access with topic_store methods. Simply replace the .bag in the output (-o) filenames above with .topic_store.

Note that subsequently you can still convert a .topic_store file into a rosbag like this:

rosrun topic_store convert.py -i robot.topic_store -o robot.bag

interact with the mongodb server using mongoshell (e.g. to get distinct values)

Get all the distinct session IDs (can be useful to make the query above

echo "db.getCollection('$COLLECTION').distinct('_ts_meta.session')"| mongo --ssl --sslAllowInvalidCertificates "mongodb://$MONGOUSER:$MONGOPASS@files.lar.lincoln.ac.uk:8098/$MONGODB?authSource=$MONGODB"

Simiarly, we could get all nodes the robot has ever been at:

echo "db.getCollection('$COLLECTION').distinct('robot.current_node.data')"| mongo --ssl --sslAllowInvalidCertificates "mongodb://$MONGOUSER:$MONGOPASS@files.lar.lincoln.ac.uk:8098/$MONGODB?authSource=$MONGODB"

And here is another one, that queries all epoch timestamps for which a recording exists:

echo "db.getCollection('$COLLECTION').distinct('_ts_meta.sys_time')"| mongo --ssl --sslAllowInvalidCertificates "mongodb://$MONGOUSER:$MONGOPASS@files.lar.lincoln.ac.uk:8098/$MONGODB?authSource=$MONGODB"

(convert epoch timestamps into human readable format at e.g. https://www.epochconverter.com/)

Notes

One should really only ever use queries (-q in topic_store), or distinct or find in the mongo shell when there is in index available on the exact field queried. Otherwise, it took take hours for a query to succeed. See Example-2021-FirstFleet-Document#existing-keys-on-collection-2021_riseholme_framos_cameras-as-of-1852022

Clone this wiki locally