You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Please refer to the Slapo-artifact repo for more details. Notice that since you only have limited GPU resources, you cannot exactly reproduce the results we presented in the paper, but you can still modify the number of GPUs following the guide in this README file to run the experiments.
thanks,I understand, but before that, I had some issues running this project and suspected that my environment was not properly configured. I have configured according to the following command. but I found a benchmark folder. Should I follow the instructions here to install the environment
Hello author, I have installed Docker on my server and pulled the image you provided. However, you mentioned that I am using an environment compatible with V100, while mine is A10. Will this cause some errors? In the Kick the Tires section, I encountered the following problem while using the Docker command
=========================================================================== test session starts ===========================================================================
platform linux -- Python 3.8.10, pytest-7.4.4, pluggy-1.3.0
rootdir: /home/deepspeed/slapo
plugins: cov-4.1.0
collected 57 items
Thread 0xThread 0x00007ed86929d74000007507ee85d740 (most recent call first):
(most recent call first):
File File ""//uussrr//llooccaall//lliibb//ppyytthhoonn33..88//ddiisstt--ppaacckkaaggeess//ttoorrcchh//ddiissttrriibbuutteedd//ddiissttrriibbuutteedd__cc1100dd..ppyy"", line , line 15701570 in in bbrrooaaddccaasstt
File File ""//uussrr//llooccaall//lliibb//ppyytthhoonn33..88//ddiisstt--ppaacckkaaggeess//ttoorrcchh//ddiissttrriibbuutteedd//ddiissttrriibbuutteedd__cc1100dd..ppyy"", line , line 14511451 in in wwrraappppeerr
File File ""//hhoommee//ddeeeeppssppeeeedd//ssllaappoo//ssllaappoo//bbuuiilldd..ppyy"", line , line 191191 in in __ccoonnssoolliiddaattee__aanndd__bbrrooaaddccaasstt
File File ""//hhoommee//ddeeeeppssppeeeedd//ssllaappoo//ssllaappoo//bbuuiilldd..ppyy"", line , line 213213 in in __ccoonnssoolliiddaattee__aanndd__bbrrooaaddccaasstt
File File ""//hhoommee//ddeeeeppssppeeeedd//ssllaappoo//ssllaappoo//bbuuiilldd..ppyy"", line , line 220220 in in ccoonnssoolliiddaattee__mmooddeell
File File ""//hhoommee//ddeeeeppssppeeeedd//ssllaappoo//ssllaappoo//bbuuiilldd..ppyy"", line , line 313313 in in bbuuiilldd
File File ""//hhoommee//ddeeeeppssppeeeedd//ssllaappoo//tteessttss//tteesstt__aannnnoottaattee..ppyy"", line , line 6868 in in tteesstt__aarrbbiittrraarryy__aannnnoottaattee
File File ""//uussrr//llooccaall//lliibb//ppyytthhoonn33..88//ddiisstt--ppaacckkaaggeess//__ppyytteesstt//ppyytthhoonn..ppyy"", line , line 194194 in in ppyytteesstt__ppyyffuunncc__ccaallll
File File ""//uussrr//llooccaall//lliibb//ppyytthhoonn33..88//ddiisstt--ppaacckkaaggeess//pplluuggggyy//__ccaalllleerrss..ppyy"", line , line 7777 in in m_umlutlitciaclall
l File
" File /"u/surs/rl/olcoacla/ll/ilbi/bp/yptyhtohno3n.38./8d/idsits-tp-apcakcakgaegse/sp/lpulguggyg/y/m_amnaangaegre.rp.yp", line y115" in , line _115h in o_ohkoeoxkeecx
e File c"
/ File u"s/ru/slro/claolc/alli/bl/ipby/tphyotnh3o.n83/.d8i/sdti-spta-cpkaacgkeasg/epsl/upglguyg/g_yh/o_ohkoso.kpsy."p, line y493" in , line 493 in c_a_lcla_l_l
_ File _"
/ File u"s/ru/slro/claolc/alli/bl/ipby/tphyotnh3o.n83/.d8i/sdti-spta-cpkaacgkeasg/e_sp/y_tpeystte/spty/tphyotnh.opny."p, line y1792" in , line r1792u in nrtuenstte
s File t"
/ File u"s/ru/slro/claolc/alli/bl/ipby/tphyotnh3o.n83/.d8i/sdti-spta-cpkaacgkeasg/e_sp/y_tpeystte/srtu/nrnuenrn.epry."p, line y169" in , line p169y in tpeystte_srtu_nrtuenstte_scta_lcla
l File l"
/ File u"s/ru/slro/claolc/alli/bl/ipby/tphyotnh3o.n83/.d8i/sdt-packages/pluggy/calilsetr-sp.apcyk"a, line g77e in s/mpullutgigcya/l_lc
a File l"l/eurssr./plyo"c, line a77l in /_lmiubl/tpiyctahloln
3 File ."8//udsirs/tl-opcaaclk/algiebs//ppyltuhgogny3/._8m/adniasgter.py", line 115 in -_phaocokkaegxeesc/
p File l"u/gugsyr//_lmoacnaalg/elri.bp/yp"y, line t115h in o_nh3o.o8k/edxiesct
File p"a/cuksarg/elso/cpallu/glgiyb//phyotohkosn.3p.y8"/, line d493i in s_t-cpaalclk_a_g
e File s"//pulsurg/glyo/c_ahlo/olkisb./ppyy"t, line h493o in n_3_.c8a/ldli_s_t
File p"a/cuksarg/elso/c_aply/tleisbt//pryutnhnoenr3..p8y/"d, line i262s in t<-lpaamcbkdaag>e
s File /"_/puystre/slto/crauln/nleirb./ppyy"t, line h262o in n<3l.a8m/bddias>t
File p"a/cuksarg/elso/c_aply/tleisbt//pryutnhnoenr3..p8y/"d, line i341s in tf-rpoamc_kcaaglels
/ File "p/yutsers/tl/orcuanln/elri.bp/yp"y, line t341h in ofnr3o.m_c8a/ldli
s File t"-/puascrk/algoecsa/l/plyitbe/spty/trhuonnn3e.r8./pdyi"s, line t261- in pcaaclkla_greusn/t_epsytt_ehsoto/kr
u File n"n/eurs.rp/yl"o, line c261a in lc/allilb_/rpuynttheosnt3_.h8o/odki
s File t"-/puascrk/algoecsa/l_/plyitbe/spty/trhuonnn3e.r8./pdyi"s, line t222- in pcaaclkla_gaensd/rpeyptoerstt
/ File r"u/nunserr/.lpoyc"a, line l222/ in lciabl/lp_yatnhdo_nr3e.p8o/rdti
s File t"-/puascrk/algoecsa/l/plyitbe/spty/trhuonnn3e.r8./pdyi"s, line t133- in praucnktaegsetsp/r_optyotceoslt
/ File r"u/nunserr/.lpoyc"a, line l133/ in lriubn/tpeystthporno3t.o8c/odli
s File t"-/puascrk/algoecsa/l/plyitbe/spty/trhuonnn3e.r8./pdyi"s, line t114- in ppayctkeasgte_sr/u_nptyetsets_tp/rroutnonceorl.
p File y""/, line u114s in rp/yltoecsatl_/rluinbt/epsytt_hporno3t.o8c/odli
s File t"-/puascrk/algoecsa/lp/lluigbg/yp/y_tchaolnl3e.r8s/.dpiys"t, line -77p in a_cmkualgteisc/aplllu
g File g"y//u_scra/llloecrasl./plyi"b, line /77p in y_tmhuolnt3i.c8a/ldli
s File t"-/puascrk/algoecsa/lp/lluigbg/yp/y_tmhaonna3g.e8r/.dpiys"t, line -115p in a_chkoaogkeesx/epcl
u File g"g/yu/s_rm/alnoacgaelr/.lpiyb"/, line p115y in t_hhoono3k.e8x/edci
s File t"-/puascrk/algoecsa/lp/lluigbg/yp/y_thhooonk3s..8p/y", line 493 in _call__d
i File s"t/-upsarc/klaogceasl//plliubg/gpyy/t_hhoono3k.s8./pdyi"s, line t493- in p_a_cckaalgle_s/
_ File p"y/tuessrt//lmoacianl./plyi"b, line /350p in yptyhtoens3t._8r/udnitsets-tplaocokpa
ges/ File "p/yutsers/tl/omcaailn/.lpiyb"/, line p350y in tphyotne3s.t8/rduinstte-sptalcokoapg
e File s"//pulsurg/glyo/c_acla/llliebr/sp.yptyh"o, line n773 in ._8m/udlitsitc-aplalc
k File a"g/euss/rp/lluogcgayl//_lciabl/lpeyrtsh.opny3"., line 877/ in d_imsutl-tpiaccaklalg
e File s"//pulsurg/glyo/c_amla/nlaigbe/rp.yptyh"o, line n1153 in .8h/odoiksetx-epca
c File k"a/guessr//plloucgagly//l_imba/npaygtehro.np3y."8, line /115d in i_shto-opkaecxkeacg
e File s"//pulsurg/glyo/c_ahlo/olkisb./ppyy"t, line h493o in n_3.c8a/ldli_s_t
File p"a/cuksarg/elso/cpallu/glgiyb//phyotohkosn.3p.y8"/, line d493i in s_t-cpaalclk_a_g
e File s"//upsytre/slto/cmaali/nl.ipby/"p, line y325t in h_omna3i.n8
/ File d"i/suts-rp/alcokcaagle/sl/i_bp/yptyetshto/nm3a.i8n/.dpiys"t, line -325p in a_cmkaaigne
s/pytest File /"m/auisnr./plyo"c, line a271l in /wlriabp/psyetshsoino3n.
8 File /"d/iusstr-/plaocckaalg/elsi/b/ppyyttehsotn/3m.a8i/nd.ipsyt"-, line p271a in cwkraagpe_ss/e_spsyitoens
t File /"m/auisnr./plyo"c, line a318l in /plyitbe/spty_tchmodnl3i.n8e/dmiasitn-
pa File c"k/augsers//l_opcyatl/lib/pytehsotn/3m.a8i/nd.ipsyt"-, line p318a in cpkyatgeesst/pclmudglgiyn/e__cmaalilne
rs.p File y""/, line u77s in r/mluolctailc/allilb
/ File p"y/tuhsorn/3l.o8c/adli/slti-bp/apcyktahgoens3/.p8l/udgigsyt/-pcaaclklaegress./ppyl"u, line g77g in y/m_umlatniacgaelrl.
p File y""/, line u115s in r/hlooockaelx/elci
b File /"p/yutshro/nl3o.c8a/ld/ilsitb-/ppayctkhaogne3s./8p/lduigsgty-/p_amcaknaaggeesr/.pplyu"g, line g115y in /hhooookkesx.epcy
" File , line "493/ in u_s_rc/allolc_a_l/lib/p
y File t"h/ouns3r./8l/odciaslt/-lpiabc/kpaygtehso/np3l.u8g/gdyi/s_th-opoakcsk.apgye"s, line /493 in p_y_tceasltl/c_o
n File f"i/gu/s_r/ilnoicta_l/.lpiyb"/, line p169y in tmhaoinn3
. File 8"//duissrt/-lpoaccakla/gleisb//ppyyttheosnt3/.c8o/ndfiisgt/-p_aicnkiatg_e_s./p_yp"y, line t169e in smta/icno
n File f"i/gu/s_r/ilnoicta_l/.lpiyb"/, line p192y in tchoonns3o.l8e/dmiasitn-
p File a"c/kuasgre/sl/o_cpaylt/elsitb//cpoyntfhiogn/3.8i/ndiits_t-.ppayc"ka, line g192e in sc/opnystoelset/m_a_imna
i File n"/u.spry/"l, line o5c in a<lm/oldiubl/ep>y
th File o"n/3u.s8r//dliisbt/-ppyatchkoang3e.s8//pryutnepsyt./p_y"m, line a87i in n__r_u.np_yc"o, line d5e in
< File m"o/duuslre/>l
ib/py File t"h/ouns3r./8l/irbu/nppyyt.hpoyn"3, line .1948 in /rruunnp_ym.opdyu"l, line e87 in a_sr_umna_icno
de
File "/usr/lib/python3.8/runpy.py", line 194 in _run_module_as_main
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: -7) local_rank: 0 (pid: 99) of binary: /usr/bin/python3
Traceback (most recent call last):
File "/usr/local/bin/torchrun", line 8, in
sys.exit(main())
File "/usr/local/lib/python3.8/dist-packages/torch/distributed/elastic/multiprocessing/errors/init.py", line 346, in wrapper
return f(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/torch/distributed/run.py", line 794, in main
run(args)
File "/usr/local/lib/python3.8/dist-packages/torch/distributed/run.py", line 785, in run
elastic_launch(
File "/usr/local/lib/python3.8/dist-packages/torch/distributed/launcher/api.py", line 134, in call
return launch_agent(self._config, self._entrypoint, list(args))
File "/usr/local/lib/python3.8/dist-packages/torch/distributed/launcher/api.py", line 250, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
===================================================
pytest FAILED
Failures:
[1]:
time : 2025-04-16_00:06:43
host : 494f2fcc8a8d
rank : 1 (local_rank: 1)
exitcode : -7 (pid: 100)
error_file: <N/A>
traceback : Signal 7 (SIGBUS) received by PID 100
Root Cause (first observed failure):
[0]:
time : 2025-04-16_00:06:43
host : 494f2fcc8a8d
rank : 0 (local_rank: 0)
exitcode : -7 (pid: 99)
error_file: <N/A>
traceback : Signal 7 (SIGBUS) received by PID 99
It does not seem to be an error on the Slapo side. Could you first check if torch.distributed is correctly configured? You can run basic tests from PyTorch. Our prebuilt docker probably has some device-dependent libraries installed. You can try building the docker image from source and see if it resolves the issue.
However, every time I execute the megatron roberta-xlarge section, an error message will appear:
Traceback (most recent call last):
File "bench_single_node.py", line 510, in
main()
File "bench_single_node.py", line 483, in main
runner(
File "/home/deepspeed/slapo/benchmark/megatron_utils.py", line 243, in run_megatron
script_file, data_args = gen(exp)
File "/home/deepspeed/slapo/benchmark/megatron_utils.py", line 83, in megatron_roberta_cmd
raise NotImplementedError
and the subsequent code will not execute properly：
slapo-megatron na
slapo-megatron na
slapo-megatron na
slapo-deepspeed na
slapo-deepspeed na
slapo-deepspeed na
BERT-1B {'megatron': [11.25, 0.0, 0.0], 'slapo-megatron': [26.5, 0.0, 0.0], 'deepspeed': [17.27, 0.0, 0.0], 'slapo-deepspeed': [22.0, 0.0, 0.0]}
slapo-megatron na
slapo-megatron na
slapo-megatron na
slapo-deepspeed na
slapo-deepspeed na
slapo-deepspeed na
RoBERTa-1.3B {'megatron': [0.0, 0.0, 0.0], 'slapo-megatron': [0.0, 0.0, 0.0], 'deepspeed': [0.0, 0.0, 0.0], 'slapo-deepspeed': [0.0, 0.0, 0.0]}
slapo-megatron na
slapo-megatron na
slapo-megatron na
slapo-deepspeed na
slapo-deepspeed na
slapo-deepspeed na
GPT-2.9B {'megatron': [0.0, 0.0, 0.0], 'slapo-megatron': [0.0, 0.0, 0.0], 'deepspeed': [0.0, 0.0, 0.0], 'slapo-deepspeed': [0.0, 0.0, 0.0]}
slapo-megatron na
slapo-megatron na
slapo-megatron na
slapo-deepspeed na
slapo-deepspeed na
slapo-deepspeed na
OPT-2.7B {'megatron': [0.0, 0.0, 0.0], 'slapo-megatron': [0.0, 0.0, 0.0], 'deepspeed': [0.0, 0.0, 0.0], 'slapo-deepspeed': [0.0, 0.0, 0.0]}
slapo-megatron na
slapo-megatron na
slapo-megatron na
slapo-deepspeed na
slapo-deepspeed na
slapo-deepspeed na
T5-2.9B {'megatron': [0.0, 0.0, 0.0], 'slapo-megatron': [0.0, 0.0, 0.0], 'deepspeed': [0.0, 0.0, 0.0], 'slapo-deepspeed': [0.0, 0.0, 0.0]}
slapo-megatron na
slapo-megatron na
slapo-megatron na
slapo-deepspeed na
slapo-deepspeed na
slapo-deepspeed na
WideResNet-2.4B {'megatron': [0.0, 0.0, 0.0], 'slapo-megatron': [0.0, 0.0, 0.0], 'deepspeed': [0.0, 0.0, 0.0], 'slapo-deepspeed': [0.0, 0.0, 0.0]}
Activity
chhzh123 commentedon Apr 3, 2025
Please refer to the Slapo-artifact repo for more details. Notice that since you only have limited GPU resources, you cannot exactly reproduce the results we presented in the paper, but you can still modify the number of GPUs following the guide in this README file to run the experiments.
old0110 commentedon Apr 14, 2025
wow，thank you,But I see that the link you provided is deployed through Docker. Can i set the environment through other methods
chhzh123 commentedon Apr 14, 2025
You don't need to use docker. You can always modify the
.cfgfile and pass it into the execution script to achieve the same purposeold0110 commentedon Apr 14, 2025
thanks,I understand, but before that, I had some issues running this project and suspected that my environment was not properly configured. I have configured according to the following command. but I found a benchmark folder. Should I follow the instructions here to install the environment
old0110 commentedon Apr 16, 2025
Hello author, I have installed Docker on my server and pulled the image you provided. However, you mentioned that I am using an environment compatible with V100, while mine is A10. Will this cause some errors? In the Kick the Tires section, I encountered the following problem while using the Docker command
=========================================================================== test session starts ===========================================================================
platform linux -- Python 3.8.10, pytest-7.4.4, pluggy-1.3.0
rootdir: /home/deepspeed/slapo
plugins: cov-4.1.0
collected 57 items
tests/test_checkpoint.py . [ 1%]
tests/test_shard_sync_op.py . [ 3%]
tests/test_annotate.py Fatal Python error: Fatal Python error: Bus errorBus error
Thread 0xThread 0x00007ed78a55f7000000750712dd4700 (most recent call first):
(most recent call first):
Thread 0xThread 0x00007ed78ad6070000007507135d5700 (most recent call first):
(most recent call first):
Thread 0xThread 0x00007ed7d0e267000000750752057700 (most recent call first):
(most recent call first):
Thread 0xThread 0x00007ed86929d74000007507ee85d740 (most recent call first):
(most recent call first):
File File ""//uussrr//llooccaall//lliibb//ppyytthhoonn33..88//ddiisstt--ppaacckkaaggeess//ttoorrcchh//ddiissttrriibbuutteedd//ddiissttrriibbuutteedd__cc1100dd..ppyy"", line , line 15701570 in in bbrrooaaddccaasstt
File File ""//uussrr//llooccaall//lliibb//ppyytthhoonn33..88//ddiisstt--ppaacckkaaggeess//ttoorrcchh//ddiissttrriibbuutteedd//ddiissttrriibbuutteedd__cc1100dd..ppyy"", line , line 14511451 in in wwrraappppeerr
File File ""//hhoommee//ddeeeeppssppeeeedd//ssllaappoo//ssllaappoo//bbuuiilldd..ppyy"", line , line 191191 in in __ccoonnssoolliiddaattee__aanndd__bbrrooaaddccaasstt
File File ""//hhoommee//ddeeeeppssppeeeedd//ssllaappoo//ssllaappoo//bbuuiilldd..ppyy"", line , line 213213 in in __ccoonnssoolliiddaattee__aanndd__bbrrooaaddccaasstt
File File ""//hhoommee//ddeeeeppssppeeeedd//ssllaappoo//ssllaappoo//bbuuiilldd..ppyy"", line , line 220220 in in ccoonnssoolliiddaattee__mmooddeell
File File ""//hhoommee//ddeeeeppssppeeeedd//ssllaappoo//ssllaappoo//bbuuiilldd..ppyy"", line , line 313313 in in bbuuiilldd
File File ""//hhoommee//ddeeeeppssppeeeedd//ssllaappoo//tteessttss//tteesstt__aannnnoottaattee..ppyy"", line , line 6868 in in tteesstt__aarrbbiittrraarryy__aannnnoottaattee
File File ""//uussrr//llooccaall//lliibb//ppyytthhoonn33..88//ddiisstt--ppaacckkaaggeess//__ppyytteesstt//ppyytthhoonn..ppyy"", line , line 194194 in in ppyytteesstt__ppyyffuunncc__ccaallll
File File ""//uussrr//llooccaall//lliibb//ppyytthhoonn33..88//ddiisstt--ppaacckkaaggeess//pplluuggggyy//__ccaalllleerrss..ppyy"", line , line 7777 in in m_umlutlitciaclall
l File
" File /"u/surs/rl/olcoacla/ll/ilbi/bp/yptyhtohno3n.38./8d/idsits-tp-apcakcakgaegse/sp/lpulguggyg/y/m_amnaangaegre.rp.yp", line y115" in , line _115h in o_ohkoeoxkeecx
e File c"
/ File u"s/ru/slro/claolc/alli/bl/ipby/tphyotnh3o.n83/.d8i/sdti-spta-cpkaacgkeasg/epsl/upglguyg/g_yh/o_ohkoso.kpsy."p, line y493" in , line 493 in c_a_lcla_l_l
_ File _"
/ File u"s/ru/slro/claolc/alli/bl/ipby/tphyotnh3o.n83/.d8i/sdti-spta-cpkaacgkeasg/e_sp/y_tpeystte/spty/tphyotnh.opny."p, line y1792" in , line r1792u in nrtuenstte
s File t"
/ File u"s/ru/slro/claolc/alli/bl/ipby/tphyotnh3o.n83/.d8i/sdti-spta-cpkaacgkeasg/e_sp/y_tpeystte/srtu/nrnuenrn.epry."p, line y169" in , line p169y in tpeystte_srtu_nrtuenstte_scta_lcla
l File l"
/ File u"s/ru/slro/claolc/alli/bl/ipby/tphyotnh3o.n83/.d8i/sdt-packages/pluggy/calilsetr-sp.apcyk"a, line g77e in s/mpullutgigcya/l_lc
a File l"l/eurssr./plyo"c, line a77l in /_lmiubl/tpiyctahloln
3 File ."8//udsirs/tl-opcaaclk/algiebs//ppyltuhgogny3/._8m/adniasgter.py", line 115 in -_phaocokkaegxeesc/
p File l"u/gugsyr//_lmoacnaalg/elri.bp/yp"y, line t115h in o_nh3o.o8k/edxiesct
e File s"//pulsurg/glyo/c_ahlo/olkisb./ppyy"t, line h493o in n_3_.c8a/ldli_s_t
s File /"_/puystre/slto/crauln/nleirb./ppyy"t, line h262o in n<3l.a8m/bddias>t
/ File "p/yutsers/tl/orcuanln/elri.bp/yp"y, line t341h in ofnr3o.m_c8a/ldli
s File t"-/puascrk/algoecsa/l/plyitbe/spty/trhuonnn3e.r8./pdyi"s, line t261- in pcaaclkla_greusn/t_epsytt_ehsoto/kr
u File n"n/eurs.rp/yl"o, line c261a in lc/allilb_/rpuynttheosnt3_.h8o/odki
s File t"-/puascrk/algoecsa/l_/plyitbe/spty/trhuonnn3e.r8./pdyi"s, line t222- in pcaaclkla_gaensd/rpeyptoerstt
/ File r"u/nunserr/.lpoyc"a, line l222/ in lciabl/lp_yatnhdo_nr3e.p8o/rdti
s File t"-/puascrk/algoecsa/l/plyitbe/spty/trhuonnn3e.r8./pdyi"s, line t133- in praucnktaegsetsp/r_optyotceoslt
/ File r"u/nunserr/.lpoyc"a, line l133/ in lriubn/tpeystthporno3t.o8c/odli
s File t"-/puascrk/algoecsa/l/plyitbe/spty/trhuonnn3e.r8./pdyi"s, line t114- in ppayctkeasgte_sr/u_nptyetsets_tp/rroutnonceorl.
p File y""/, line u114s in rp/yltoecsatl_/rluinbt/epsytt_hporno3t.o8c/odli
s File t"-/puascrk/algoecsa/lp/lluigbg/yp/y_tchaolnl3e.r8s/.dpiys"t, line -77p in a_cmkualgteisc/aplllu
g File g"y//u_scra/llloecrasl./plyi"b, line /77p in y_tmhuolnt3i.c8a/ldli
s File t"-/puascrk/algoecsa/lp/lluigbg/yp/y_tmhaonna3g.e8r/.dpiys"t, line -115p in a_chkoaogkeesx/epcl
u File g"g/yu/s_rm/alnoacgaelr/.lpiyb"/, line p115y in t_hhoono3k.e8x/edci
s File t"-/puascrk/algoecsa/lp/lluigbg/yp/y_thhooonk3s..8p/y", line 493 in _call__d
i File s"t/-upsarc/klaogceasl//plliubg/gpyy/t_hhoono3k.s8./pdyi"s, line t493- in p_a_cckaalgle_s/
_ File p"y/tuessrt//lmoacianl./plyi"b, line /350p in yptyhtoens3t._8r/udnitsets-tplaocokpa
ges/ File "p/yutsers/tl/omcaailn/.lpiyb"/, line p350y in tphyotne3s.t8/rduinstte-sptalcokoapg
e File s"//pulsurg/glyo/c_acla/llliebr/sp.yptyh"o, line n773 in ._8m/udlitsitc-aplalc
k File a"g/euss/rp/lluogcgayl//_lciabl/lpeyrtsh.opny3"., line 877/ in d_imsutl-tpiaccaklalg
e File s"//pulsurg/glyo/c_amla/nlaigbe/rp.yptyh"o, line n1153 in .8h/odoiksetx-epca
c File k"a/guessr//plloucgagly//l_imba/npaygtehro.np3y."8, line /115d in i_shto-opkaecxkeacg
e File s"//pulsurg/glyo/c_ahlo/olkisb./ppyy"t, line h493o in n_3.c8a/ldli_s_t
e File s"//upsytre/slto/cmaali/nl.ipby/"p, line y325t in h_omna3i.n8
/ File d"i/suts-rp/alcokcaagle/sl/i_bp/yptyetshto/nm3a.i8n/.dpiys"t, line -325p in a_cmkaaigne
s/pytest File /"m/auisnr./plyo"c, line a271l in /wlriabp/psyetshsoino3n.
8 File /"d/iusstr-/plaocckaalg/elsi/b/ppyyttehsotn/3m.a8i/nd.ipsyt"-, line p271a in cwkraagpe_ss/e_spsyitoens
t File /"m/auisnr./plyo"c, line a318l in /plyitbe/spty_tchmodnl3i.n8e/dmiasitn-
pa File c"k/augsers//l_opcyatl/lib/pytehsotn/3m.a8i/nd.ipsyt"-, line p318a in cpkyatgeesst/pclmudglgiyn/e__cmaalilne
rs.p File y""/, line u77s in r/mluolctailc/allilb
/ File p"y/tuhsorn/3l.o8c/adli/slti-bp/apcyktahgoens3/.p8l/udgigsyt/-pcaaclklaegress./ppyl"u, line g77g in y/m_umlatniacgaelrl.
p File y""/, line u115s in r/hlooockaelx/elci
b File /"p/yutshro/nl3o.c8a/ld/ilsitb-/ppayctkhaogne3s./8p/lduigsgty-/p_amcaknaaggeesr/.pplyu"g, line g115y in /hhooookkesx.epcy
" File , line "493/ in u_s_rc/allolc_a_l/lib/p
y File t"h/ouns3r./8l/odciaslt/-lpiabc/kpaygtehso/np3l.u8g/gdyi/s_th-opoakcsk.apgye"s, line /493 in p_y_tceasltl/c_o
n File f"i/gu/s_r/ilnoicta_l/.lpiyb"/, line p169y in tmhaoinn3
. File 8"//duissrt/-lpoaccakla/gleisb//ppyyttheosnt3/.c8o/ndfiisgt/-p_aicnkiatg_e_s./p_yp"y, line t169e in smta/icno
n File f"i/gu/s_r/ilnoicta_l/.lpiyb"/, line p192y in tchoonns3o.l8e/dmiasitn-
p File a"c/kuasgre/sl/o_cpaylt/elsitb//cpoyntfhiogn/3.8i/ndiits_t-.ppayc"ka, line g192e in sc/opnystoelset/m_a_imna
i File n"/u.spry/"l, line o5c in a<lm/oldiubl/ep>y
th File o"n/3u.s8r//dliisbt/-ppyatchkoang3e.s8//pryutnepsyt./p_y"m, line a87i in n__r_u.np_yc"o, line d5e in
< File m"o/duuslre/>l
ib/py File t"h/ouns3r./8l/irbu/nppyyt.hpoyn"3, line .1948 in /rruunnp_ym.opdyu"l, line e87 in a_sr_umna_icno
de
File "/usr/lib/python3.8/runpy.py", line 194 in _run_module_as_main
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: -7) local_rank: 0 (pid: 99) of binary: /usr/bin/python3
Traceback (most recent call last):
File "/usr/local/bin/torchrun", line 8, in
sys.exit(main())
File "/usr/local/lib/python3.8/dist-packages/torch/distributed/elastic/multiprocessing/errors/init.py", line 346, in wrapper
return f(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/torch/distributed/run.py", line 794, in main
run(args)
File "/usr/local/lib/python3.8/dist-packages/torch/distributed/run.py", line 785, in run
elastic_launch(
File "/usr/local/lib/python3.8/dist-packages/torch/distributed/launcher/api.py", line 134, in call
return launch_agent(self._config, self._entrypoint, list(args))
File "/usr/local/lib/python3.8/dist-packages/torch/distributed/launcher/api.py", line 250, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
===================================================
pytest FAILED
Failures:
[1]:
time : 2025-04-16_00:06:43
host : 494f2fcc8a8d
rank : 1 (local_rank: 1)
exitcode : -7 (pid: 100)
error_file: <N/A>
traceback : Signal 7 (SIGBUS) received by PID 100
Root Cause (first observed failure):
[0]:
time : 2025-04-16_00:06:43
host : 494f2fcc8a8d
rank : 0 (local_rank: 0)
exitcode : -7 (pid: 99)
error_file: <N/A>
traceback : Signal 7 (SIGBUS) received by PID 99
chhzh123 commentedon Apr 16, 2025
It does not seem to be an error on the Slapo side. Could you first check if
torch.distributedis correctly configured? You can run basic tests from PyTorch. Our prebuilt docker probably has some device-dependent libraries installed. You can try building the docker image from source and see if it resolves the issue.old0110 commentedon Apr 17, 2025
I have successfully run this section, but I encountered some issues in the experimental section of Figure 7. The content of my single_node_asplos24.cfg is as follows:
megatron bert-xlarge 1 256 0 2 0
slapo-megatron bert-xlarge 1 256 0 12 0
deepspeed bert-xlarge 1 256 0 10 0
slapo-deepspeed bert-xlarge 1 256 0 12 0
megatron bert-xlarge 2 512 0 4 0
slapo-megatron bert-xlarge 2 512 0 20 0
deepspeed bert-xlarge 2 512 0 12 0
slapo-deepspeed bert-xlarge 2 512 0 20 0
megatron bert-xlarge 3 512 0 30 0
slapo-megatron bert-xlarge 3 512 0 84 0
deepspeed bert-xlarge 3 512 0 24 0
slapo-deepspeed bert-xlarge 3 512 0 24 0
megatron roberta-xlarge 1 512 0 13 0
slapo-megatron roberta-xlarge 1 512 0 13 0
deepspeed roberta-xlarge 1 512 0 12 0
slapo-deepspeed roberta-xlarge 1 512 0 18 0
megatron roberta-xlarge 2 512 0 10 0
slapo-megatron roberta-xlarge 2 512 0 20 0
deepspeed roberta-xlarge 2 512 0 12 0
slapo-deepspeed roberta-xlarge 2 512 0 22 0
megatron roberta-xlarge 3 512 0 39 0
slapo-megatron roberta-xlarge 3 512 0 39 0
deepspeed roberta-xlarge 3 512 0 22 0
slapo-deepspeed roberta-xlarge 3 512 0 24 0
megatron EleutherAI/gpt-neo-2.7B 1 512 0 3 1
slapo-megatron EleutherAI/gpt-neo-2.7B 1 512 0 3.5 1
deepspeed EleutherAI/gpt-neo-2.7B 1 512 0 3 1
slapo-deepspeed EleutherAI/gpt-neo-2.7B 1 512 0 3 0.6
megatron EleutherAI/gpt-neo-2.7B 2 512 0 6 1
slapo-megatron EleutherAI/gpt-neo-2.7B 2 512 0 7 1
deepspeed EleutherAI/gpt-neo-2.7B 2 512 0 6 1
slapo-deepspeed EleutherAI/gpt-neo-2.7B 2 512 0 6 0.6
megatron EleutherAI/gpt-neo-2.7B 3 512 0 9 1
slapo-megatron EleutherAI/gpt-neo-2.7B 3 512 0 10.5 1
deepspeed EleutherAI/gpt-neo-2.7B 3 512 0 6 1
slapo-deepspeed EleutherAI/gpt-neo-2.7B 3 512 0 6 0.6
megatron t5-3b 1 1024 512 2 full
slapo-megatron t5-3b 1 1024 512 2 1
deepspeed t5-3b 1 1024 512 1 1
slapo-deepspeed t5-3b 1 1024 512 1 1
megatron t5-3b 2 1024 512 2 full
slapo-megatron t5-3b 2 1024 512 2 1
deepspeed t5-3b 2 1024 512 2 1
slapo-deepspeed t5-3b 2 1024 512 2 1
megatron t5-3b 3 1024 512 6 full
slapo-megatron t5-3b 3 1024 512 6 1
deepspeed t5-3b 3 1024 512 3 1
slapo-deepspeed t5-3b 3 1024 512 3 1
megatron wideresnet-2.4B 1 1 0 8 0
slapo-megatron wideresnet-2.4B 1 1 0 8 0
deepspeed wideresnet-2.4B 1 1 0 10 1
slapo-deepspeed wideresnet-2.4B 1 1 0 12 0.34
megatron wideresnet-2.4B 2 1 0 10 0
slapo-megatron wideresnet-2.4B 2 1 0 10 0
deepspeed wideresnet-2.4B 2 1 0 16 1
slapo-deepspeed wideresnet-2.4B 2 1 0 22 0.34
megatron wideresnet-2.4B 3 1 0 24 0
slapo-megatron wideresnet-2.4B 3 1 0 24 0
deepspeed wideresnet-2.4B 3 1 0 16 1
slapo-deepspeed wideresnet-2.4B 3 1 0 24 0.34
However, every time I execute the megatron roberta-xlarge section, an error message will appear:
Traceback (most recent call last):
File "bench_single_node.py", line 510, in
main()
File "bench_single_node.py", line 483, in main
runner(
File "/home/deepspeed/slapo/benchmark/megatron_utils.py", line 243, in run_megatron
script_file, data_args = gen(exp)
File "/home/deepspeed/slapo/benchmark/megatron_utils.py", line 83, in megatron_roberta_cmd
raise NotImplementedError
and the subsequent code will not execute properly：
slapo-megatron na
slapo-megatron na
slapo-megatron na
slapo-deepspeed na
slapo-deepspeed na
slapo-deepspeed na
BERT-1B {'megatron': [11.25, 0.0, 0.0], 'slapo-megatron': [26.5, 0.0, 0.0], 'deepspeed': [17.27, 0.0, 0.0], 'slapo-deepspeed': [22.0, 0.0, 0.0]}
slapo-megatron na
slapo-megatron na
slapo-megatron na
slapo-deepspeed na
slapo-deepspeed na
slapo-deepspeed na
RoBERTa-1.3B {'megatron': [0.0, 0.0, 0.0], 'slapo-megatron': [0.0, 0.0, 0.0], 'deepspeed': [0.0, 0.0, 0.0], 'slapo-deepspeed': [0.0, 0.0, 0.0]}
slapo-megatron na
slapo-megatron na
slapo-megatron na
slapo-deepspeed na
slapo-deepspeed na
slapo-deepspeed na
GPT-2.9B {'megatron': [0.0, 0.0, 0.0], 'slapo-megatron': [0.0, 0.0, 0.0], 'deepspeed': [0.0, 0.0, 0.0], 'slapo-deepspeed': [0.0, 0.0, 0.0]}
slapo-megatron na
slapo-megatron na
slapo-megatron na
slapo-deepspeed na
slapo-deepspeed na
slapo-deepspeed na
OPT-2.7B {'megatron': [0.0, 0.0, 0.0], 'slapo-megatron': [0.0, 0.0, 0.0], 'deepspeed': [0.0, 0.0, 0.0], 'slapo-deepspeed': [0.0, 0.0, 0.0]}
slapo-megatron na
slapo-megatron na
slapo-megatron na
slapo-deepspeed na
slapo-deepspeed na
slapo-deepspeed na
T5-2.9B {'megatron': [0.0, 0.0, 0.0], 'slapo-megatron': [0.0, 0.0, 0.0], 'deepspeed': [0.0, 0.0, 0.0], 'slapo-deepspeed': [0.0, 0.0, 0.0]}
slapo-megatron na
slapo-megatron na
slapo-megatron na
slapo-deepspeed na
slapo-deepspeed na
slapo-deepspeed na
WideResNet-2.4B {'megatron': [0.0, 0.0, 0.0], 'slapo-megatron': [0.0, 0.0, 0.0], 'deepspeed': [0.0, 0.0, 0.0], 'slapo-deepspeed': [0.0, 0.0, 0.0]}
old0110 commentedon Apr 17, 2025
When running the script that generates Figure 7, the following issue was also encountered
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/filelock/_api.py", line 255, in acquire
self._acquire()
File "/usr/local/lib/python3.8/dist-packages/filelock/_unix.py", line 39, in _acquire
fd = os.open(self.lock_file, open_flags, self._context.mode)
PermissionError: [Errno 13] Permission denied: '/home/deepspeed/.triton/autotune/Fp16Matmul_2d_kernel.pickle.lock'