[askap-users] CASDA storage migration delays and access to data
- Contemporary messages sorted: [ by date ] [ by thread ] [ by subject ] [ by author ] [ by messages with attachments ]
From: Huynh, Minh (S&A, Kensington WA) <"Huynh,>
Date: Thu, 10 Feb 2022 04:08:12 +0000
Dear ASKAP users,
For a while now I have been mentioning the storage migration at Pawsey, where CASDA is moving from a DMF based tape storage to new disk-based Ceph/S3 storage.
We were expecting the new storage system to be in production this week, and to begin the migration, but we are now expecting the new storage system from Pawsey on Feb 15th. We have also just been updated with news that the DMF system will be unavailable for ~3 weeks while the new tape storage is set up for Pawsey (previously we expected one set of the tape libraries to be available, within days to a week.)
The timeline is now:
Feb 15: start copy of high priority data to Acacia (new S3 storage)
Feb 18: switch deposits to Acacia
Feb 21: switch data access to Acacia
Feb 21: expect this primary data migration to complete, DMF
High priority data is available but low priority data isn’t, and any new deposits are available as they are to Acacia
Low priority data unavailable
Mar 21: preparation and migration of low priority data from Banksia (new tape storage) to Acacia
End April / Early May : complete migration of low priority data
High priority data is defined as per below, and access to this should be basically uninterrupted. We have included ancillary data (intermediate radio data products) from more recent SBIDs, except for FLASH and GASKAP-OH where they are huge cubes. Evaluation files will be added to this set, so that validation may continue.
We will start to copy back low priority data as soon as possible, from Mar 21 is the current date from Pawsey. But that data may not be available for weeks after that as we will have almost a 1PB to transfer back.
If you need data that isn’t in the high priority list below then please consider downloading it over the next week, otherwise it will be after Mar 21 (and maybe weeks after that) when it is made available again.
Please contact me ASAP if you have questions or concerns.
Cheers,
Minh
High priority data:
All catalogues, primary images/cubes, spectra, level 7 data
All ancillary data from SBIDs deposited after July 2021 (i.e. roughly phase II pilots)
-But not GASKAP-OH and FLASH as they are large
10083 10635 9287 9325 9351 9410 9434 9437 9442 9501 EMU ancillary images
AS104 DINGO visibilities deposited after July 1
All evaluation files also to be included.
--
Dr Minh Huynh (she/her)
ATNF Science Team Leader
CSIRO ASKAP Science Data Archive Project Leader
Space and Astronomy | CSIRO
Minh.Huynh_at_email.protectedau<mailto:Minh.Huynh_at_email.protected> | 08 6436 8696
Whadjuk and Noongar Country
26 Dick Perry Ave, Kensington WA 6151
Received on 2022-02-10 15:08:33
Date: Thu, 10 Feb 2022 04:08:12 +0000
Dear ASKAP users,
For a while now I have been mentioning the storage migration at Pawsey, where CASDA is moving from a DMF based tape storage to new disk-based Ceph/S3 storage.
We were expecting the new storage system to be in production this week, and to begin the migration, but we are now expecting the new storage system from Pawsey on Feb 15th. We have also just been updated with news that the DMF system will be unavailable for ~3 weeks while the new tape storage is set up for Pawsey (previously we expected one set of the tape libraries to be available, within days to a week.)
The timeline is now:
Feb 15: start copy of high priority data to Acacia (new S3 storage)
Feb 18: switch deposits to Acacia
Feb 21: switch data access to Acacia
Feb 21: expect this primary data migration to complete, DMF
High priority data is available but low priority data isn’t, and any new deposits are available as they are to Acacia
Low priority data unavailable
Mar 21: preparation and migration of low priority data from Banksia (new tape storage) to Acacia
End April / Early May : complete migration of low priority data
High priority data is defined as per below, and access to this should be basically uninterrupted. We have included ancillary data (intermediate radio data products) from more recent SBIDs, except for FLASH and GASKAP-OH where they are huge cubes. Evaluation files will be added to this set, so that validation may continue.
We will start to copy back low priority data as soon as possible, from Mar 21 is the current date from Pawsey. But that data may not be available for weeks after that as we will have almost a 1PB to transfer back.
If you need data that isn’t in the high priority list below then please consider downloading it over the next week, otherwise it will be after Mar 21 (and maybe weeks after that) when it is made available again.
Please contact me ASAP if you have questions or concerns.
Cheers,
Minh
High priority data:
All catalogues, primary images/cubes, spectra, level 7 data
All ancillary data from SBIDs deposited after July 2021 (i.e. roughly phase II pilots)
-But not GASKAP-OH and FLASH as they are large
10083 10635 9287 9325 9351 9410 9434 9437 9442 9501 EMU ancillary images
AS104 DINGO visibilities deposited after July 1
All evaluation files also to be included.
--
Dr Minh Huynh (she/her)
ATNF Science Team Leader
CSIRO ASKAP Science Data Archive Project Leader
Space and Astronomy | CSIRO
Minh.Huynh_at_email.protectedau<mailto:Minh.Huynh_at_email.protected> | 08 6436 8696
Whadjuk and Noongar Country
26 Dick Perry Ave, Kensington WA 6151
Received on 2022-02-10 15:08:33