Skip to content

terminate called after throwing an instance of 'c10::Error' what(): [enforce fail at inline_container.cc:319] . unexpected pos 2302963008 vs 2302962896 frame #0: c10::ThrowEnforceNotMet(char const*, int, char const*, std::string const&, void const*) + 0x55 (0x15158b1320a5 in /usr/local/lib/python3.8/dist-packages/torch/lib/libc10.so) #63

Description

@lldd1001

frame #1: + 0x34396bc (0x1514c3a796bc in /usr/local/lib/python3.8/dist-packages/torch/lib/libtorch_cpu.so)
frame #2: mz_zip_writer_add_mem_ex_v2 + 0x5b1 (0x1514c3a73a91 in /usr/local/lib/python3.8/dist-packages/torch/lib/libtorch_cpu.so)
frame #3: caffe2::serialize::PyTorchStreamWriter::writeRecord(std::string const&, void const*, unsigned long, bool) + 0xb9 (0x1514c3a7ac79 in /usr/local/lib/python3.8/dist-packages/torch/lib/libtorch_cpu.so)
frame #4: caffe2::serialize::PyTorchStreamWriter::writeEndOfFile() + 0x2c3 (0x1514c3a7b143 in /usr/local/lib/python3.8/dist-packages/torch/lib/libtorch_cpu.so)
frame #5: caffe2::serialize::PyTorchStreamWriter::~PyTorchStreamWriter() + 0x125 (0x1514c3a7b3b5 in /usr/local/lib/python3.8/dist-packages/torch/lib/libtorch_cpu.so)
frame #6: + 0x6d90a5 (0x15155c0700a5 in /usr/local/lib/python3.8/dist-packages/torch/lib/libtorch_python.so)
frame #7: + 0x36658f (0x15155bcfd58f in /usr/local/lib/python3.8/dist-packages/torch/lib/libtorch_python.so)
frame #8: + 0x36747f (0x15155bcfe47f in /usr/local/lib/python3.8/dist-packages/torch/lib/libtorch_python.so)
frame #9: /usr/bin/python3.8() [0x5ec780]
frame #10: /usr/bin/python3.8() [0x5441f8]
frame #11: /usr/bin/python3.8() [0x54424a]
frame #12: _PyEval_EvalFrameDefault + 0x286b (0x56d1eb in /usr/bin/python3.8)
frame #13: _PyFunction_Vectorcall + 0x1b6 (0x5f6836 in /usr/bin/python3.8)
frame #14: _PyEval_EvalFrameDefault + 0x72e (0x56b0ae in /usr/bin/python3.8)
frame #15: _PyEval_EvalCodeWithName + 0x26a (0x56939a in /usr/bin/python3.8)
frame #16: _PyFunction_Vectorcall + 0x393 (0x5f6a13 in /usr/bin/python3.8)
frame #17: _PyEval_EvalFrameDefault + 0x190c (0x56c28c in /usr/bin/python3.8)
frame #18: _PyFunction_Vectorcall + 0x1b6 (0x5f6836 in /usr/bin/python3.8)
frame #19: _PyEval_EvalFrameDefault + 0x85a (0x56b1da in /usr/bin/python3.8)
frame #20: _PyEval_EvalCodeWithName + 0x26a (0x56939a in /usr/bin/python3.8)
frame #21: _PyFunction_Vectorcall + 0x393 (0x5f6a13 in /usr/bin/python3.8)
frame #22: _PyEval_EvalFrameDefault + 0x56b5 (0x570035 in /usr/bin/python3.8)
frame #23: _PyFunction_Vectorcall + 0x1b6 (0x5f6836 in /usr/bin/python3.8)
frame #24: _PyEval_EvalFrameDefault + 0x72e (0x56b0ae in /usr/bin/python3.8)
frame #25: _PyFunction_Vectorcall + 0x1b6 (0x5f6836 in /usr/bin/python3.8)
frame #26: PyObject_Call + 0x1f7 (0x5f3547 in /usr/bin/python3.8)
frame #27: _PyEval_EvalFrameDefault + 0x1f4d (0x56c8cd in /usr/bin/python3.8)
frame #28: _PyEval_EvalCodeWithName + 0x26a (0x56939a in /usr/bin/python3.8)
frame #29: _PyFunction_Vectorcall + 0x393 (0x5f6a13 in /usr/bin/python3.8)
frame #30: _PyEval_EvalFrameDefault + 0x72e (0x56b0ae in /usr/bin/python3.8)
frame #31: _PyFunction_Vectorcall + 0x1b6 (0x5f6836 in /usr/bin/python3.8)
frame #32: PyObject_Call + 0x1f7 (0x5f3547 in /usr/bin/python3.8)
frame #33: _PyEval_EvalFrameDefault + 0x1f4d (0x56c8cd in /usr/bin/python3.8)
frame #34: _PyEval_EvalCodeWithName + 0x26a (0x56939a in /usr/bin/python3.8)
frame #35: _PyFunction_Vectorcall + 0x393 (0x5f6a13 in /usr/bin/python3.8)
frame #36: _PyEval_EvalFrameDefault + 0x56b5 (0x570035 in /usr/bin/python3.8)
frame #37: /usr/bin/python3.8() [0x6b2b5c]
frame #38: _PyEval_EvalFrameDefault + 0x72e (0x56b0ae in /usr/bin/python3.8)
frame #39: _PyEval_EvalCodeWithName + 0x26a (0x56939a in /usr/bin/python3.8)
frame #40: PyEval_EvalCode + 0x27 (0x68d047 in /usr/bin/python3.8)
frame #41: /usr/bin/python3.8() [0x67e351]
frame #42: /usr/bin/python3.8() [0x67e3cf]
frame #43: /usr/bin/python3.8() [0x67e471]
frame #44: PyRun_SimpleFileExFlags + 0x197 (0x67e817 in /usr/bin/python3.8)
frame #45: Py_RunMain + 0x212 (0x6b6fe2 in /usr/bin/python3.8)
frame #46: Py_BytesMain + 0x2d (0x6b736d in /usr/bin/python3.8)
frame #47: __libc_start_main + 0xf3 (0x15158ed860b3 in /usr/lib/x86_64-linux-gnu/libc.so.6)
frame #48: _start + 0x2e (0x5fa5ce in /usr/bin/python3.8)

train.sh: line 21: 7442 Aborted (core dumped) fairseq-train ../../data/DDI/relis-bin --save-dir ${SAVE_DIR} --user-dir ../../src --finetune-from-model ../../checkpoints/Pre-trained-BioGPT/checkpoint.pt --task language_modeling_prompt --arch transformer_lm_prompt_biogpt --share-decoder-input-output-embed --decoder-learned-pos --optimizer adam --adam-betas '(0.9, 0.98)' --weight-decay 0.01 --clip-norm 0.0 --lr 1e-4 --lr-scheduler inverse_sqrt --warmup-updates 500 --warmup-init-lr 1e-07 --tokens-per-sample 1024 --max-source-positions 640 --max-target-positions 1024 --max-tokens 1024 --update-freq 32 --skip-invalid-size-inputs-valid-test --max-epoch 100 --keep-last-epochs 5 --learned-prompt 9

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions