Skip to content

Add classes for better dummies encoding, flags for missing data, and missing data replacement - #3

Open
brettasmi wants to merge 14 commits into
madrury:masterfrom
brettasmi:master
Open

Add classes for better dummies encoding, flags for missing data, and missing data replacement#3
brettasmi wants to merge 14 commits into
madrury:masterfrom
brettasmi:master

Conversation

@brettasmi

Copy link
Copy Markdown
Contributor

I have added the DummiesEncoder, MissingIndicator, and NanReplacer classes to dftransformers.py.

They seem to work well at this point and be robust to np.nan of different dtypes (which I spent yesterday tearing my hair out over). This code was a lot cleaner until I had to add lots of special nan handling. There are still some issues to work out and features to add including:

  • code handlers for np.nan in custom DummiesEncoder levels dict
  • deal with np.inf and any other special values
  • add ability to handle custom missing data values to MissingIndicator and NanReplacer, e.g. -9
  • rename NanReplacer to something more appropriate once the above bullet is completed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant