v1.0.0

Description

1. LMFlow now defaults to using Accelerate (i.e, run scripts using accelerate launch ... finetune.py ...). If you prefer to use deepspeed (deepspeed ... finetune.py ...) or accelerate + deepspeed backend, please install using pip install -e '.[deepspeed]'
2. Removed/archived some less frequently used docs/scripts/modules.
   • docker
   • scripts/data_preprocess, scripts/speculative_decoding, scripts/tools, scripts/vocab_extension
   • service
   • utils

What's Changed

• [Feature] vllm inferencer and memory safe vllm inferencer by @wheresmyhair in #860
• [Feature] Add vllm inference example by @wheresmyhair in #863
• Add customized optimizer support by @research4pan in #865
• Expanding Optimization Options by @tianshijing in #871
• [Feature] reward model inferencer and dpov2 aligner by @wheresmyhair in #867
• Added more custom optimizers by @tianshijing in #876
• [ADD] T2I finetuning with SD1 and SD2 to contrib by @Aziily in #877
• update naacl best paper award by @shizhediao in #881
• [usability] make dpo eval dataset optional by @wheresmyhair in #889
• function-call-finetune by @HALIS-sh in #884
• Modify the formatter of function and observation by @HALIS-sh in #892
• [Feature] Iterative DPO by @wheresmyhair in #883
• [usability] remove numpy version requirement by @wheresmyhair in #901
• Fix load from LoRA weight & empty tools and system by @HALIS-sh in #902
• [fix] merge lora fix by @wheresmyhair in #909
• [usability] temporarily change default version to 0.0.8 by @wheresmyhair in #911
• [doc] Add wandb setup guide by @wheresmyhair in #912
• [temp] temporarily restrict transformers version by @wheresmyhair in #914
• [usability] deps streamlining by @wheresmyhair in #905
• [doc] update default branch by @wheresmyhair in #915
• Add Hymba and DoRA support by @Dominic789654 in #919
• announce support of hymba by @shizhediao in #921
• Hymba support announcement by @shizhediao in #922
• [usability] Add hymba lora target by @wheresmyhair in #924
• [usability] Change dataset check method by @wheresmyhair in #930
• [usability] support qwen2.5 and deepseek by @wheresmyhair in #931
• [fix] fix jinja template issue caused by empty system prompt by @wheresmyhair in #935
• [fix] fix get dataset type function by @wheresmyhair in #937
• [doc] readme update by @wheresmyhair in #938
• Added Muon Optimizer by @tianshijing in #939

New Contributors

• @tianshijing made their first contribution in #871
• @Aziily made their first contribution in #877

Full Changelog: v0.0.8...v1.0.0

[2024-06-19] LMFlow v0.0.8

Major new features since v0.0.4

• Support conversation templates
• Support new optimization algorithms, e.g. LISA
• Update requirements to support latest models
• Fix bugs in qlora/lora scripts
• Fix tokenization parallelism bug
• Improve script interfaces

What's Changed

• README refactor by @shizhediao in #607
• Improve interface of finetuning scripts by @research4pan in #611
• resize banner by @shizhediao in #612
• Doc Reformat by @shizhediao in #614
• Dev update transformers by @yaoguany in #616
• Added QLoRA support for Decoder transformers with tune_strategy "Normal" by @TensorBlast in #613
• announce long context support by @shizhediao in #621
• fix deepspeed zero3 config bugs by @yaoguany in #622
• Update version.py by @hendrydong in #624
• update qr code by @shizhediao in #625
• FIX BUG: trust_remote_code flag didn't take effect by @conght in #633
• Add explanations about supported CUDA versions by @research4pan in #634
• Added citation for RAFT by @shizhediao in #635
• Update qrcode by @shizhediao in #636
• [Features] Support multi_modal training by @lianqing11 in #628
• Add scripts to convert raw file to text-only json by @research4pan in #638
• speculative decoding by @wheresmyhair in #630
• [Feature] Speculative Inference by @wheresmyhair in #640
• add readme for speculative decoding by @wheresmyhair in #641
• update news about speculative decoding by @shizhediao in #642
• update llama flash attention by @yaoguany in #646
• update qrcode by @shizhediao in #647
• [FIX] Fix multi-modal training by @lianqing11 in #648
• Fix: --disable_group_texts 1 keep short samples by @research4pan in #649
• Support all types with --disable_group_texts 1 by @research4pan in #650
• Fix model downloading for CPU-only servers by @research4pan in #651
• add block size to fingerprint by @RolandMinrui in #653
• update qrcode by @shizhediao in #654
• Update QR code for wechat by @research4pan in #656
• update qrcode by @shizhediao in #657
• update qr code for wechat by @research4pan in #667
• Update version of datasets dependency by @research4pan in #668
• Add flash attention install for A6000 by @research4pan in #669
• Update hf_decoder_model.py by @yaoguany in #670
• fix bugs in llama flash attention by @yaoguany in #681
• code exection class and test cases by @Bob17293729 in #674
• Update README to reflect changes in v0.0.6 by @research4pan in #696
• fix bugs in requirements.txt since previous one can cause errors by @xu1868 in #697
• fix merge lora bug by @Dominic789654 in #698
• Upgrade transformers deps to support mistral by @research4pan in #700
• add lisa code and lisa args by @Dominic789654 in #701
• add GPU memory check script by @Dominic789654 in #702
• Support multi-gpu inference by @research4pan in #699
• src/lmflow/args.py typo fix by @wheresmyhair in #703
• add more info when fail to import flash attn by @wheresmyhair in #704
• Add script to finetune llama-2 with lisa by @research4pan in #705
• add LISA args introduction by @Dominic789654 in #706
• add lisa bibtex by @Dominic789654 in #708
• Rpan add news by @research4pan in #709
• Fix typo in README.md by @research4pan in #710
• Update lisa code by @Dominic789654 in #711
• Fix recent bad commits, which lead to OOM in 7B by @research4pan in #712
• Add more memory saving options for LISA script by @research4pan in #716
• Add stable version 0.0.7 by @research4pan in #717
• Export --lisa_interval_steps by @research4pan in #718
• Update documentation.yaml to fix github check failure when submitting PRs by @wheresmyhair in #723
• support more model by @Dominic789654 in #724
• Update README.md to temporary allow latest fix by @research4pan in #725
• Rpan dev lisa by @research4pan in #715
• ShareGPT format support by @wheresmyhair in #720
• Add latest support for flash-attention from hf by @research4pan in #727
• Fix LISA init OOM caused by deepspeed by @research4pan in #728
• Improve installation documentation in README.md by @research4pan in #733
• Add temporary dataset document by @research4pan in #734
• Improve README and requirement.txt by @research4pan in #735
• documentation update and action fix by @wheresmyhair in #736
• documentation page style update and readme dataset update by @wheresmyhair in #737
• Add explanation for LISA with model-parallelism by @research4pan in #738
• typo fix and disable html logo by @wheresmyhair in #739
• Update dependency for transformers by @research4pan in #740
• Rpan lisa more models by @research4pan in #742
• Fix: typo in --lisa_layers_attribute by @research4pan in #743
• Add inferencer by @research4pan in #744
• Improve LISA layers_attribute mapping logics by @research4pan in #746
• Change --disable_group_texts default to True by @research4pan in #747
• readme (zh) update by @wheresmyhair in #748
• readme (zh) typo fix by @wheresmyhair in #749
• Change version to 0.0.7 in version.py by @research4pan in #750
• readme (ko) update by @wheresmyhair in #751
• Change lmflow.org to a temporary ip by @research4pan in #753
• Add more clarification for data server problem by @research4pan in #755
• Remove flash-attn in requirements.txt by @research4pan in #757
• ko and zh readme update by @wheresmyhair in #758
• ko readme update by @wheresmyhair in #761
• [New feature] Integrate DPO by @gzliyu in #762
• readme jp update by @wheresmyhair in #763
• Recover data server by @research4pan in #764
• remove data server address notice by @wheresmyhair in #765
• readme update by @wheresmyhair in #766
• hindi readme update by @wheresmyhair in #769
• conversation dataset (sharegpt format) custom template dev by @wheresmyhair in #767
• custom template document update by @wheresmyhair in #770
• conversation template doc typo fix by @wheresmyhair in #771
• template info update by @wheresmyhair in #772
• add support for llama-3 template by @Wheres...

Aug 8: LMFlow 0.0.4

Merge pull request #603 from OptimalScale/doc-edit

Edit doc and readme

Jul 21: LMFlow 0.0.3

Merge pull request #560 from yaoguany/main

allow exceeding model maximum length when train&inference

initial release

initial release