M_replace_aux Error with DragonFly 1.34.2

Dragonfly Technical
1

ct 27 19:54:58 ip-172-31-0-22 dragonfly[78125]: F20251027 19:54:57.888782 78133 init.cc:140] Uncaught exception: basic_string::_
M_replace_aux
Oct 27 19:55:00 ip-172-31-0-22 dragonfly[78125]: *** Check failure stack trace: ***
Oct 27 19:55:00 ip-172-31-0-22 dragonfly[78125]: @ 0xaaaab1f1f0f4 google::LogMessage::SendToLog()
Oct 27 19:55:00 ip-172-31-0-22 dragonfly[78125]: @ 0xaaaab1f17fd8 google::LogMessage::Flush()
Oct 27 19:55:00 ip-172-31-0-22 dragonfly[78125]: @ 0xaaaab1f19914 google::LogMessageFatal::~LogMessageFatal()
Oct 27 19:55:00 ip-172-31-0-22 dragonfly[78125]: @ 0xaaaab1335f5c _ZZN13MainInitGuardC4EPiPPPcjENKUlvE_clEv.isra.0
Oct 27 19:55:00 ip-172-31-0-22 dragonfly[78125]: @ 0xaaaab133604c _ZZN13MainInitGuardC4EPiPPPcjENUlvE_4_FUNEv
Oct 27 19:55:00 ip-172-31-0-22 dragonfly[78125]: @ 0xaaaab2026e94 __cxxabiv1::__terminate()
Oct 27 19:55:00 ip-172-31-0-22 dragonfly[78125]: @ 0xaaaab2026ee8 std::terminate()
Oct 27 19:55:00 ip-172-31-0-22 dragonfly[78125]: @ 0xaaaab202706c __cxa_throw
Oct 27 19:55:00 ip-172-31-0-22 dragonfly[78125]: @ 0xaaaab12f90d0 std::__throw_length_error()
Oct 27 19:55:00 ip-172-31-0-22 dragonfly[78125]: @ 0xaaaab20a2918 std::__cxx11::basic_string<>::_M_replace_aux()
Oct 27 19:55:00 ip-172-31-0-22 dragonfly[78125]: @ 0xaaaab1a01d80 dfly::CompactObj::GetString()
Oct 27 19:55:00 ip-172-31-0-22 dragonfly[78125]: @ 0xaaaab1a01e48 dfly::CompactObj::GetSlice()
Oct 27 19:55:00 ip-172-31-0-22 dragonfly[78125]: @ 0xaaaab13be138 _ZN4dfly13RdbSerializer9SaveValueERKNS_10CompactObjE.l
ocalalias
Oct 27 19:55:00 ip-172-31-0-22 dragonfly[78125]: @ 0xaaaab13be2e4 dfly::RdbSerializer::SaveEntry()
Oct 27 19:55:00 ip-172-31-0-22 dragonfly[78125]: @ 0xaaaab13d2514 ZN4dfly13SliceSnapshot14SerializeEntryEtRKNS_10Compac
tObjES3.localalias
Oct 27 19:55:00 ip-172-31-0-22 dragonfly[78125]: @ 0xaaaab13d29d8 dfly::SliceSnapshot::SerializeBucket()
Oct 27 19:55:00 ip-172-31-0-22 dragonfly[78125]: @ 0xaaaab13d2cbc dfly::SliceSnapshot::BucketSaveCb()
Oct 27 19:55:00 ip-172-31-0-22 dragonfly[78125]: @ 0xaaaab13d30f8 dfly::SliceSnapshot::IterateBucketsFb()
Oct 27 19:55:00 ip-172-31-0-22 dragonfly[78125]: @ 0xaaaab13d33c0 _ZN5boost7context6detail11fiber_entryINS1_12fiber_reco
rdINS0_5fiberEN4util3fb219FixedStackAllocatorEZNS6_6detail15WorkerFiberImplIZN4dfly13SliceSnapshot5StartEbNSB_13SnapshotFlushEEUlvE2_JEEC4IS7_EESt17basic_str
ing_viewIcSt11char_traitsIcEENS6_13FiberPriorityERKNS0_12preallocatedEOT_OSD_EUlOS4_E_EEEEvNS1_10transfer_tE
Oct 27 19:55:00 ip-172-31-0-22 dragonfly[78125]: @ 0xaaaab1c43794 make_fcontext
Oct 27 19:55:00 ip-172-31-0-22 dragonfly[78125]: *** SIGABRT received at time=1761594900 on cpu 7 ***
Oct 27 19:55:00 ip-172-31-0-22 dragonfly[78125]: PC: @ 0xffffa09779b4 (unknown) __pthread_kill_implementation
Oct 27 19:55:00 ip-172-31-0-22 dragonfly[78125]: @ 0xaaaab1f77a4c 464 absl::lts_20250512::AbslFailureSignalHandle
r()
Oct 27 19:55:00 ip-172-31-0-22 dragonfly[78125]: @ 0xffffa0c79830 5104 (unknown)
Oct 27 19:55:00 ip-172-31-0-22 dragonfly[78125]: @ 0xffffa092e3a0 64 gsignal
Oct 27 19:55:00 ip-172-31-0-22 dragonfly[78125]: @ 0xffffa091a264 320 abort
Oct 27 19:55:00 ip-172-31-0-22 dragonfly[78125]: @ 0xaaaab1f24bb4 32 google::DumpStackTraceAndExit()
Oct 27 19:55:00 ip-172-31-0-22 dragonfly[78125]: @ 0xaaaab1f185d4 192 google::LogMessage::Fail()
Oct 27 19:55:01 ip-172-31-0-22 audit[78125]: ANOM_ABEND auid=4294967295 uid=0 gid=0 ses=4294967295 subj=system_u:system_r:unconfi
ned_service_t:s0 pid=78125 comm=“Proactor7” exe=“/usr/local/bin/dragonfly” sig=6 res=1
Oct 27 19:55:02 ip-172-31-0-22 dragonfly[78125]: @ 0xaaaab1f1f0f4 16 google::LogMessage::SendToLog()
Oct 27 19:55:02 ip-172-31-0-22 dragonfly[78125]: @ 0xaaaab1f17fd8 208 google::LogMessage::Flush()
Oct 27 19:55:02 ip-172-31-0-22 dragonfly[78125]: @ 0xaaaab1f19914 80 google::LogMessageFatal::~LogMessageFatal()
Oct 27 19:55:02 ip-172-31-0-22 dragonfly[78125]: @ 0xaaaab1335f5c 16 MainInitGuard::MainInitGuard()::{lambda()#1
}::operator()()
Oct 27 19:55:02 ip-172-31-0-22 dragonfly[78125]: @ 0xaaaab133604c 352 MainInitGuard::MainInitGuard()::{lambda()#1
}::_FUN()
Oct 27 19:55:02 ip-172-31-0-22 dragonfly[78125]: @ 0xaaaab2026e94 16 __cxxabiv1::__terminate()
Oct 27 19:55:02 ip-172-31-0-22 dragonfly[78125]: @ 0xaaaab2026ee8 16 std::terminate()
Oct 27 19:55:02 ip-172-31-0-22 dragonfly[78125]: @ 0xaaaab202706c 16 __cxa_throw
Oct 27 19:55:02 ip-172-31-0-22 dragonfly[78125]: @ 0xaaaab12f90d0 48 std::__throw_length_error()
Oct 27 19:55:02 ip-172-31-0-22 dragonfly[78125]: @ 0xaaaab20a2918 32 std::__cxx11::basic_string<>::M_replace_au
x()
Oct 27 19:55:02 ip-172-31-0-22 dragonfly[78125]: @ 0xaaaab1a01d80 64 dfly::CompactObj::GetString()
Oct 27 19:55:02 ip-172-31-0-22 dragonfly[78125]: @ 0xaaaab1a01e48 32 dfly::CompactObj::GetSlice()
Oct 27 19:55:02 ip-172-31-0-22 dragonfly[78125]: @ 0xaaaab13be138 224 dfly::RdbSerializer::SaveValue()
Oct 27 19:55:02 ip-172-31-0-22 dragonfly[78125]: F20251027 19:55:01.475198 78131 init.cc:140] Uncaught exception: basic_string::
M_replace_aux
Oct 27 19:55:02 ip-172-31-0-22 dragonfly[78125]: *** Check failure stack trace: ***
Oct 27 19:55:02 ip-172-31-0-22 dragonfly[78125]: @ 0xaaaab1f1f0f4 google::LogMessage::SendToLog()
Oct 27 19:55:02 ip-172-31-0-22 dragonfly[78125]: @ 0xaaaab1f17fd8 google::LogMessage::Flush()
Oct 27 19:55:02 ip-172-31-0-22 dragonfly[78125]: @ 0xaaaab1f19914 google::LogMessageFatal::~LogMessageFatal()
Oct 27 19:55:02 ip-172-31-0-22 dragonfly[78125]: @ 0xaaaab1335f5c _ZZN13MainInitGuardC4EPiPPPcjENKUlvE_clEv.isra.0
Oct 27 19:55:02 ip-172-31-0-22 dragonfly[78125]: @ 0xaaaab133604c _ZZN13MainInitGuardC4EPiPPPcjENUlvE_4_FUNEv
Oct 27 19:55:02 ip-172-31-0-22 dragonfly[78125]: @ 0xaaaab2026e94 __cxxabiv1::__terminate()
Oct 27 19:55:02 ip-172-31-0-22 dragonfly[78125]: @ 0xaaaab2026ee8 std::terminate()
Oct 27 19:55:02 ip-172-31-0-22 dragonfly[78125]: @ 0xaaaab202706c __cxa_throw
Oct 27 19:55:02 ip-172-31-0-22 dragonfly[78125]: @ 0xaaaab12f90d0 std::__throw_length_error()
Oct 27 19:55:02 ip-172-31-0-22 dragonfly[78125]: @ 0xaaaab20a2918 std::__cxx11::basic_string<>::_M_replace_aux()
Oct 27 19:55:02 ip-172-31-0-22 dragonfly[78125]: @ 0xaaaab1a01d80 dfly::CompactObj::GetString()
Oct 27 19:55:02 ip-172-31-0-22 dragonfly[78125]: @ 0xaaaab1a01e48 dfly::CompactObj::GetSlice()
Oct 27 19:55:02 ip-172-31-0-22 dragonfly[78125]: @ 0xaaaab13be138 _ZN4dfly13RdbSerializer9SaveValueERKNS_10CompactObjE.l
ocalalias
Oct 27 19:55:02 ip-172-31-0-22 dragonfly[78125]: @ 0xaaaab13be2e4 dfly::RdbSerializer::SaveEntry()
Oct 27 19:55:02 ip-172-31-0-22 dragonfly[78125]: @ 0xaaaab13d2514 ZN4dfly13SliceSnapshot14SerializeEntryEtRKNS_10Compac
tObjES3.localalias
Oct 27 19:55:02 ip-172-31-0-22 dragonfly[78125]: @ 0xaaaab13d29d8 dfly::SliceSnapshot::SerializeBucket()
Oct 27 19:55:02 ip-172-31-0-22 dragonfly[78125]: @ 0xaaaab13d2cbc dfly::SliceSnapshot::BucketSaveCb()
Oct 27 19:55:02 ip-172-31-0-22 dragonfly[78125]: @ 0xaaaab13d30f8 dfly::SliceSnapshot::IterateBucketsFb()
Oct 27 19:55:02 ip-172-31-0-22 dragonfly[78125]: @ 0xaaaab13d33c0 _ZN5boost7context6detail11fiber_entryINS1_12fiber_reco
rdINS0_5fiberEN4util3fb219FixedStackAllocatorEZNS6_6detail15WorkerFiberImplIZN4dfly13SliceSnapshot5StartEbNSB_13SnapshotFlushEEUlvE2_JEEC4IS7_EESt17basic_str
ing_viewIcSt11char_traitsIcEENS6_13FiberPriorityERKNS0_12preallocatedEOT_OSD_EUlOS4_E_EEEEvNS1_10transfer_tE
Oct 27 19:55:02 ip-172-31-0-22 dragonfly[78125]: @ 0xaaaab1c43794 make_fcontext
Oct 27 19:55:02 ip-172-31-0-22 dragonfly[78125]: [failure_signal_handler.cc : 377] RAW: Signal 6 raised at PC=0xffffa09779b4 whil
e already in AbslFailureSignalHandler()
Oct 27 19:55:02 ip-172-31-0-22 dragonfly[78125]: @ 0xaaaab13be2e4 176 dfly::RdbSerializer::SaveEntry()
Oct 27 19:55:02 ip-172-31-0-22 dragonfly[78125]: @ 0xaaaab13d2514 224 dfly::SliceSnapshot::SerializeEntry()
Oct 27 19:55:02 ip-172-31-0-22 dragonfly[78125]: @ 0xaaaab13d29d8 272 dfly::SliceSnapshot::SerializeBucket()
Oct 27 19:55:02 ip-172-31-0-22 dragonfly[78125]: @ 0xaaaab13d2cbc 80 dfly::SliceSnapshot::BucketSaveCb()
Oct 27 19:55:02 ip-172-31-0-22 dragonfly[78125]: @ 0xaaaab13d30f8 320 dfly::SliceSnapshot::IterateBucketsFb()
Oct 27 19:55:02 ip-172-31-0-22 dragonfly[78125]: @ 0xaaaab13d33c0 240 boost::context::detail::fiber_entry<>()

2

Do you know how this error happened? (i.e., what command you used to trigger this).

I suppose you are using the same setup as mentioned in the other thread?

  • OS - 6.1.148-173.267.amzn2023.aarch64
  • Dragonfly version : 1.34.2

Can you provide these as well?

  • Kernel, by using command: uname -a
  • Containerized?: [Bare Metal, Docker, Docker Compose, Docker Swarm, Kubernetes, Other]
3

Hi @joezhou_df

Thanks for your response.

I was performing an experiment with 20M keys with 1KB payload with Set & expire TTL value for 1 week while SSD Tiering was enabled with 0.7 threshold.

[root@ip-172-31-0-22 dragonfly]# uname -a
Linux ip-172-31-0-22.ap-south-1.compute.internal 6.1.148-173.267.amzn2023.aarch64 #1 SMP Sun Aug 24 03:50:23 UTC 2025 aarch64 aarch64 aarch64 GNU/Linux