招标网站排行榜,代运网站,phpstudy怎么做网站,免费下载ppt模板网站有哪些问题现象
对象存储存储桶无法删除#xff0c;检查发现生命周期过期后存储桶中有文件残留#xff0c;未完全删除#xff0c;但实际访问文件时为404#xff0c;通过s3cmd无法删除对象#xff0c;且无报错。
问题定位
检查bucket当前状态#xff0c;发现桶内有大量object…问题现象
对象存储存储桶无法删除检查发现生命周期过期后存储桶中有文件残留未完全删除但实际访问文件时为404通过s3cmd无法删除对象且无报错。
问题定位
检查bucket当前状态发现桶内有大量object残留
# radosgw-admin bucket stats --bucketxxx
{bucket: xxx,zonegroup: 97cde842-9840-46d2-98f5-2f2ff2b356b9,placement_rule: default-placement,explicit_placement: {data_pool: ,data_extra_pool: ,index_pool: },id: a3c7aedc-fcd5-47f3-8540-887817ea45b0.53713368.242,marker: a3c7aedc-fcd5-47f3-8540-887817ea45b0.53713368.242,index_type: Normal,owner: e638db8fbfbd4eeba23f4e7f386e2234,ver: 0#115872937,1#124262992,2#124096868,3#113933319,4#113752484,5#115978810,6#119804274,7#122088764,8#117238420,9#114292015,10#122281621,11#112069481,12#117514195,13#113249706,14#114992906,15#119836492,16#109344437,17#115584159,18#113904518,19#118475333,20#118442717,21#119161849,22#119636967,23#115195682,24#114825666,25#116687890,26#110362647,27#114200698,28#122991031,29#111118759,30#122337033,31#114870449,32#115295855,33#117904918,34#121221126,35#116571732,36#124130503,37#118485487,38#110489698,39#113517117,40#119301165,41#115423515,42#118595461,43#115347045,44#120019831,45#113194194,46#114868275,47#123410209,48#119010861,49#114866634,50#111604486,51#117146989,52#113854552,53#111169375,54#120663269,55#116393793,56#114043425,57#117807012,58#122349615,59#113390481,60#124332341,61#114732340,62#114933373,63#114692587,64#118841860,65#118823413,66#115203887,67#122762813,68#109180753,69#115918036,70#116907631,71#115166297,72#126285010,73#117308131,74#118488176,75#117960296,76#115163648,77#116434667,78#114904720,79#128437168,80#119323730,81#114563504,82#117850906,83#119937587,84#121641776,85#111226459,86#115483044,87#119067683,88#115415015,89#112365103,90#119033043,91#111300726,92#117803502,93#116570158,94#110132992,95#117143542,96#118852347,97#121028481,98#111041704,99#109977398,100#116800374,101#121455617,102#119020462,103#116583479,104#117083862,105#119048007,106#118643510,107#115722285,108#114712589,109#119526889,110#122204433,111#118918806,112#119785343,113#123881328,114#114785575,115#116062597,116#120837772,117#118009592,118#112875590,119#119865516,120#111902759,121#112887532,122#112571750,123#119188027,124#115919787,125#118978645,126#114970199,127#116637842,master_ver: 0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0,11#0,12#0,13#0,14#0,15#0,16#0,17#0,18#0,19#0,20#0,21#0,22#0,23#0,24#0,25#0,26#0,27#0,28#0,29#0,30#0,31#0,32#0,33#0,34#0,35#0,36#0,37#0,38#0,39#0,40#0,41#0,42#0,43#0,44#0,45#0,46#0,47#0,48#0,49#0,50#0,51#0,52#0,53#0,54#0,55#0,56#0,57#0,58#0,59#0,60#0,61#0,62#0,63#0,64#0,65#0,66#0,67#0,68#0,69#0,70#0,71#0,72#0,73#0,74#0,75#0,76#0,77#0,78#0,79#0,80#0,81#0,82#0,83#0,84#0,85#0,86#0,87#0,88#0,89#0,90#0,91#0,92#0,93#0,94#0,95#0,96#0,97#0,98#0,99#0,100#0,101#0,102#0,103#0,104#0,105#0,106#0,107#0,108#0,109#0,110#0,111#0,112#0,113#0,114#0,115#0,116#0,117#0,118#0,119#0,120#0,121#0,122#0,123#0,124#0,125#0,126#0,127#0,mtime: 2022-09-19 23:05:51.793737,max_marker: 0#,1#,2#,3#,4#,5#,6#,7#,8#,9#,10#,11#,12#,13#,14#,15#,16#,17#,18#,19#,20#,21#,22#,23#,24#,25#,26#,27#,28#,29#,30#,31#,32#,33#,34#,35#,36#,37#,38#,39#,40#,41#,42#,43#,44#,45#,46#,47#,48#,49#,50#,51#,52#,53#,54#,55#,56#,57#,58#,59#,60#,61#,62#,63#,64#,65#,66#,67#,68#,69#,70#,71#,72#,73#,74#,75#,76#,77#,78#,79#,80#,81#,82#,83#,84#,85#,86#,87#,88#,89#,90#,91#,92#,93#,94#,95#,96#,97#,98#,99#,100#,101#,102#,103#,104#,105#,106#,107#,108#,109#,110#,111#,112#,113#,114#,115#,116#,117#,118#,119#,120#,121#,122#,123#,124#,125#,126#,127#,usage: {rgw.main: {size: 57217593456,size_actual: 57233543168,size_utilized: 57217593456,size_kb: 55876557,size_kb_actual: 55892132,size_kb_utilized: 55876557,num_objects: 7896,: {size: 515899392,size_actual: 515899392,size_utilized: 515899392,size_kb: 503808,size_kb_actual: 503808,size_kb_utilized: 503808,num_objects: 123},STANDARD: {size: 56701694064,size_actual: 56717643776,size_utilized: 56701694064,size_kb: 55372749,size_kb_actual: 55388324,size_kb_utilized: 55372749,num_objects: 7773}},rgw.multimeta: {size: 0,size_actual: 0,size_utilized: 216,size_kb: 0,size_kb_actual: 0,size_kb_utilized: 1,num_objects: 6,STANDARD: {size: 0,size_actual: 0,size_utilized: 216,size_kb: 0,size_kb_actual: 0,size_kb_utilized: 1,num_objects: 6}}},bucket_quota: {enabled: false,check_on_raw: false,max_size: -1,max_size_kb: -1,max_objects: -1}
}拉取残余文件列表首先获取pool、bucket id和num_shards
# 获取poolningbo1-zone1.rgw.buckets.index
rados lspools | grep rgw.buckets.index
# 获取bucket ida3c7aedc-fcd5-47f3-8540-887817ea45b0.53713368.242
radosgw-admin bucket stats --bucket {bucket name} | egrep id
# 获取shard数量128
radosgw-admin metadata get bucket.instance:{bucket name}:{bucket id} | grep num_shards然后拉取文件列表
for i in {0..127}
dorados -p ningbo1-zone1.rgw.buckets.index listomapkeys .dir.a3c7aedc-fcd5-47f3-8540-887817ea45b0.53713368.242.$i objects.txt
done检查object状态发现相应数据底层均已删除
for object in cat objects.txt
doradosgw-admin object stat --bucketxxx --object$object
done均报错ERROR: failed to stat object, returned error: (2) No such file or directory。 列出bucket索引发现所有索引exists: true判断为索引残留问题及索引存在但数据已删除。
问题处置
可通过rados rmomapkey删除无效索引
for i in {0..127}
dofor object in rados -p ningbo1-zone1.rgw.buckets.index listomapkeys .dir.a3c7aedc-fcd5-47f3-8540-887817ea45b0.53713368.242.$idorados -c /etc/ceph/ceph.conf -p ningbo1-zone1.rgw.buckets.index rmomapkey .dir.a3c7aedc-fcd5-47f3-8540-887817ea45b0.53713368.242.$i $objectdone
done重新扫描修复桶状态可多次执行
# radosgw-admin bucket check --fix --check-objects --bucketxxx
[]
{}
{existing_header: {usage: {}},calculated_header: {usage: {}}
}再次检查发现桶已正常清空
# radosgw-admin bucket stats --bucketxxx
{bucket: xxx,zonegroup: 97cde842-9840-46d2-98f5-2f2ff2b356b9,placement_rule: default-placement,explicit_placement: {data_pool: ,data_extra_pool: ,index_pool: },id: a3c7aedc-fcd5-47f3-8540-887817ea45b0.53713368.242,marker: a3c7aedc-fcd5-47f3-8540-887817ea45b0.53713368.242,index_type: Normal,owner: e638db8fbfbd4eeba23f4e7f386e2234,ver: 0#115872943,1#124262998,2#124096874,3#113933325,4#113752490,5#115978816,6#119804280,7#122088770,8#117238426,9#114292021,10#122281627,11#112069487,12#117514201,13#113249712,14#114992912,15#119836498,16#109344443,17#115584165,18#113904524,19#118475339,20#118442723,21#119161855,22#119636973,23#115195688,24#114825672,25#116687896,26#110362653,27#114200704,28#122991037,29#111118765,30#122337039,31#114870455,32#115295861,33#117904924,34#121221132,35#116571738,36#124130509,37#118485493,38#110489704,39#113517123,40#119301171,41#115423521,42#118595467,43#115347051,44#120019837,45#113194200,46#114868281,47#123410215,48#119010867,49#114866640,50#111604492,51#117146995,52#113854558,53#111169381,54#120663275,55#116393799,56#114043431,57#117807018,58#122349621,59#113390487,60#124332347,61#114732346,62#114933379,63#114692593,64#118841866,65#118823419,66#115203893,67#122762819,68#109180759,69#115918042,70#116907637,71#115166303,72#126285016,73#117308137,74#118488182,75#117960302,76#115163654,77#116434673,78#114904726,79#128437174,80#119323736,81#114563510,82#117850912,83#119937593,84#121641782,85#111226465,86#115483050,87#119067689,88#115415021,89#112365109,90#119033049,91#111300732,92#117803508,93#116570164,94#110132998,95#117143548,96#118852353,97#121028487,98#111041710,99#109977404,100#116800380,101#121455623,102#119020468,103#116583485,104#117083868,105#119048013,106#118643516,107#115722291,108#114712595,109#119526895,110#122204439,111#118918812,112#119785349,113#123881334,114#114785581,115#116062603,116#120837778,117#118009598,118#112875596,119#119865522,120#111902765,121#112887538,122#112571756,123#119188033,124#115919793,125#118978651,126#114970205,127#116637848,master_ver: 0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0,11#0,12#0,13#0,14#0,15#0,16#0,17#0,18#0,19#0,20#0,21#0,22#0,23#0,24#0,25#0,26#0,27#0,28#0,29#0,30#0,31#0,32#0,33#0,34#0,35#0,36#0,37#0,38#0,39#0,40#0,41#0,42#0,43#0,44#0,45#0,46#0,47#0,48#0,49#0,50#0,51#0,52#0,53#0,54#0,55#0,56#0,57#0,58#0,59#0,60#0,61#0,62#0,63#0,64#0,65#0,66#0,67#0,68#0,69#0,70#0,71#0,72#0,73#0,74#0,75#0,76#0,77#0,78#0,79#0,80#0,81#0,82#0,83#0,84#0,85#0,86#0,87#0,88#0,89#0,90#0,91#0,92#0,93#0,94#0,95#0,96#0,97#0,98#0,99#0,100#0,101#0,102#0,103#0,104#0,105#0,106#0,107#0,108#0,109#0,110#0,111#0,112#0,113#0,114#0,115#0,116#0,117#0,118#0,119#0,120#0,121#0,122#0,123#0,124#0,125#0,126#0,127#0,mtime: 2022-09-19 23:05:51.793737,max_marker: 0#,1#,2#,3#,4#,5#,6#,7#,8#,9#,10#,11#,12#,13#,14#,15#,16#,17#,18#,19#,20#,21#,22#,23#,24#,25#,26#,27#,28#,29#,30#,31#,32#,33#,34#,35#,36#,37#,38#,39#,40#,41#,42#,43#,44#,45#,46#,47#,48#,49#,50#,51#,52#,53#,54#,55#,56#,57#,58#,59#,60#,61#,62#,63#,64#,65#,66#,67#,68#,69#,70#,71#,72#,73#,74#,75#,76#,77#,78#,79#,80#,81#,82#,83#,84#,85#,86#,87#,88#,89#,90#,91#,92#,93#,94#,95#,96#,97#,98#,99#,100#,101#,102#,103#,104#,105#,106#,107#,108#,109#,110#,111#,112#,113#,114#,115#,116#,117#,118#,119#,120#,121#,122#,123#,124#,125#,126#,127#,usage: {},bucket_quota: {enabled: false,check_on_raw: false,max_size: -1,max_size_kb: -1,max_objects: -1}
}处理完毕。
