MapleStoryDetectionSampleGenerator

Generate Machine Learning Samples Object Detection In MapleStory

Performance

This generator can generate arbitrarily many annotated samples. All bounding boxes are precisely annotated based on rendering coordinates.

With RTMDet and ~10000 samples, it can achieve 97.3%mAP in test set.

Requirement

• .NET 10.0 SDK (10.0.0 or above)

Build

1. Clone this repository with submodules by
   git clone --recursive git@github.com:charlescao460/MapleStoryDetectionSampleGenerator.git.
   Note that --recursive is necessary.
2. Build MapleStoryDetectionSampleGenerator.sln
3. Run MapleStory.MachineLearningSampleGenerator\bin\Release\net10.0-windows7.0\WzComparerR2.exe and open MapRender once. Running WzComparerR2.exe will generate Setting.config, which is required for our MapRender invoker.

Run

(Assuming assemblies are built with Release configuration. Debug configuration is similar)

1. Use WzComparerR2.exe to find the desired map you want to sample. Assuming 993134200.img is the map you want in Limina.
2. From the solution root, prepare a YAML config file. A checked-in example is available at Examples\sample-generator.yml.
3. If you want synthetic players, configure avatar WZ part IDs in the player post-processor. The generator renders player frames on the fly through MapleStory.Avatar.
4. Run dotnet run --project .\MapleStory.MachineLearningSampleGenerator -- --config ".\Examples\sample-generator.yml"
   You can run .\MapleStory.MachineLearningSampleGenerator.exe --help for usage hint, or execute the built binary directly with --config <path>. The config file is the single source of truth for maps, rendering, output, and post-processors.

Example YAML:

mode: character
concurrency: 2

output:
  format: coco
  path: .
  name: sample-generator

render:
  width: 1366
  height: 768

sampling:
  count: 1000
  intervalMs: 0

postProcessors:
  - type: player
    count: 3
    actions: [stand1, walk1, jump]
    emotions: [default]
    avatars:
      - parts: [2000, 12003, 20000, 30000, 1040036, 1060026]
      - parts: [2000, 12003, 20000, 30000, 1040036, 1060026, 1703598]

maps:
  - id: 993134200
  - id: 450007010
    sampling:
      count: 2000
    postProcessors: []

Notes about the YAML format:

• mode is required. Use character for normal map/object samples and rune for rune-arrow keypoint samples.
• concurrency is optional and controls how many map renders run at once. It defaults to 1.
• maps is required. The legacy sequence form lists explicit map IDs, and each id should be the numeric map id without .img.
• Root sampling and postProcessors act as defaults for every map.
• sampling.count is required and controls how many uniformly random camera positions are sampled from each map.
• A map-level sampling block overrides only the fields it sets.
• A map-level postProcessors block replaces the root processor list. postProcessors: [] disables inherited processors for that map.
• player.count is the number of generated player instances added to each sampled screenshot. It defaults to 3 when omitted.
• player.avatars[].parts is an ordered list of WZ part IDs. Later IDs replace earlier slot conflicts, matching the avatar generator behavior.
• Relative paths are resolved from the YAML file location.

Rune mode uses the same render and sampling sections, but requires output.format: coco and does not support postProcessors. Each output image is a center-square crop with four generated rune_arrow annotations and two COCO keypoints per arrow.

Rune mode can also randomly choose maps from all numeric *.img map nodes in the MapleStory data. Explicit entries are always included first; random.count adds that many additional maps and excludes duplicate explicit IDs. Add seed when you need repeatable selection.

Use maps.allMaps: true to sample every numeric *.img map node. Explicit entries are still included first and are not duplicated. allMaps cannot be combined with maps.random.

mode: rune
output:
  format: coco
  path: ./rune-output
  name: rune-sample
render:
  width: 1366
  height: 768
sampling:
  count: 1000
  intervalMs: 0
maps:
  entries:
    - id: 410013660
  random:
    count: 50
    seed: 12345

Note

• Since NPCs look like players, including them without annotation could result a negative effect on our model. Therefore, by default, we changed WzComparerR2.MapRender/MapData.cs to prevent any NPC data loaded into map render when invoking from MapleStory.MachineLearningSampleGenerator.exe,

Output Formats

Tensorflow TFRecord

According to Tensorflow official document, the output .tfrecord contains multiple tf.train.Example in single file. With each example store in the following formats:

uint64 length
uint32 masked_crc32_of_length
byte   data[length]
uint32 masked_crc32_of_data

And

masked_crc = ((crc >> 15) | (crc << 17)) + 0xa282ead8ul

Each tf.train.Example is generated by protobuf-net according to Tensorflow example.proto

Darknet

Output directory structure:

data/
|---obj/
|   |---1.jpg
|   |---1.txt
|   |---......
|---obj.data
|---obj.names
|---test.txt
|---train.txt

obj.data contains

classes=2
train=data/train.txt
valid=data/test.txt
names=data/obj.names
backup = backup/

And obj.names contains the class name for object. test.txt and train.txt contains samples for testing/training with ratio of 5:95 (5% of images in obj/ are used for testing).

COCO

Output directory structure:

coco/
|---train2017/
|   |---1.jpg
|   |---2.jpg
|   |---......
|---val2017/
|   |---1000.jpg
|   |---1001.jpg
|   |---......
|---annotations/
|   |---instances_train2017.json
|   |---instances_val2017.json

The COCO json is defined as following:

{
  "info": {
    "description": "MapleStory 993134100.img Object Detection Samples - Training",
    "url": "https://github.com/charlescao460/MapleStoryDetectionSampleGenerator",
    "version": "1.0",
    "year": 2021,
    "contributor": "CSR"
  },
  "licenses": [
    {
      "url": "https://github.com/charlescao460/MapleStoryDetectionSampleGenerator/blob/main/LICENSE",
      "id": 1,
      "name": "MIT License"
    }
  ],
  "images": [
    {
      "license": 1,
      "file_name": "30a892e1-7f3d-4c65-bdd1-9d28f1ae5187.jpg",
      "coco_url": "",
      "height": 768,
      "width": 1366,
      "flickr_url": "",
      "id": 1
    },
    ...],
  "categories": [
    {
      "supercategory": "element",
      "id": 1,
      "name": "Mob"
    },
    {
      "supercategory": "element",
      "id": 2,
      "name": "Player"
    }
  ],
  "annotations": [
    {
      "segmentation": [
        [
          524,
          429,
          664,
          429,
          664,
          578,
          524,
          578
        ]
      ],
      "area": 20860,
      "iscrowd": 0,
      "image_id": 1,
      "bbox": [
        524,
        429,
        140,
        149
      ],
      "category_id": 1,
      "id": 1
    },
    ...]

Note that segmentation covers the area as the same as bbox does. No segmentation or masked implemented .