Migrating Between Storage Locations - Revision history

SarahK: /* Migrating to AWS S3 */

2023-02-06T14:20:46Z

‎Migrating to AWS S3

SarahK at 18:51, 26 April 2022

2022-04-26T18:51:06Z

SarahK: Created page with "You may want to migrate files from one local storage location to another, or from an existing storage location to an AWS S3 bucket or a compatible service. ==Migrating Local..."

2022-01-21T20:12:33Z

Created page with "You may want to migrate files from one local storage location to another, or from an existing storage location to an AWS S3 bucket or a compatible service. ==Migrating Local..."

New page

You may want to migrate files from one local storage location to another, or from an existing storage location to an AWS S3 bucket or a compatible service.

==Migrating Local Storage==

To migrate files from an existing local storage location to another local directory:

{{LocalStorage}}

==Migrating to AWS S3==

To migrate to AWS:

# Create your S3 configuration. For steps to set one up, see [https://help.whmcs.com/m/system/c/281650 our guides and tutorials].
# In the WHMCS Admin Area, navigate to '''Configuration (<i class="fa fa-wrench" aria-hidden="true"></i>) > System Settings > Storage Settings''' and click the '''Configuration''' tab.
# Under '''Add New Configuration''', select '''S3'''.
# Click the + icon.
# Enter your AWS S3 connection details.
#* You must specify the S3 region by its code name (for example, <tt>us-east-1</tt> for ''US East (N. Virginia)''). To find the region code name for a bucket's region, [https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html click here].
#*The system doesn't require the '''Endpoint URL''' value if you're using the Amazon AWS S3 service. This is only required if you are using a non-Amazon, S3-compatible storage provider.
#*The S3 access credentials must have read and write access to the specified S3 bucket. See below for an example of a suitable policy for the S3 user.
#Click '''Save'''. If the system can't access the bucket, an error message will appear. If this happens, check the entered details and try again.
#Click on the '''Settings''' tab.
#Select the new storage location for the file type that you wish to migrate to the new location. The system can migrate most file types automatically.
# Click '''Migrate'''.
#*Smaller migrations will require less than a minute.
#*If the system can't complete the migration immediately, it will schedule it to run in the background. Until it finishes, the system will continue to use the former location.
#** When the migration is finished, the setting will automatically switch to the new selection and begin using the new location.
#** You may choose to cancel an ongoing migration. If you do, the system will continue to use the present location. Files that the system already migrated to the new location will remain.

The system won't automatically remove files from the former location. Do this manually after you verify that the migration was successful.

===Manual Migration===

Instead of using automated migration, you may opt to migrate your files manually and switch the relevant file types storage setting at once. This will skip automatic migration and you will be responsible for moving the files to the new location.

If you use this method, note that WHMCS may store certain file types in the same folder (for example, the system will store client files, email and ticket attachments, and project management files in the <tt>attachments</tt> folder). If you decide to move all the files away from that local folder, you must switch all relevant file types to the new location.

If you made a change by mistake, click '''Revert Changes''' to cancel any changes.

===Example Security Policy===

Consider using the [https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_create.html IAM Policy Generator tool] in the AWS control panel or the [https://awspolicygen.s3.amazonaws.com/policygen.html public policy generator tool].

For advanced users, an example AWS S3 policy to assign to the S3 user for use with WHMCS storage settings is below:

{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Stmt1111222223333",
"Effect": "Allow",
"Action": [
"s3:ListBucket",
"s3:GetObject",
"s3:GetObjectAcl",
"s3:PutObject",
"s3:PutObjectAcl",
"s3:ReplicateObject",
"s3:DeleteObject"
],
"Resource": [
"arn:aws:s3:::<BUCKET_NAME>",
"arn:aws:s3:::<BUCKET_NAME>/*"
]
}
]
}

====For Manual Migrations====
For manual migrations, the system stores files in AWS S3 with a prefix corresponding to the file asset type. For example, it will store file1.png, a client file, as /client_files/file1.png. Use the following sub-folder names per file type:

* Client Files — client_files
* Downloads — downloads
* Email Attachments — email_attachments
* Email Template Attachments — template_attachments
* Project Management Files — pm_files
* Ticket Attachments — ticket_attachments

When using automatic migration, you do not have to worry about these prefixes; the system manages them automatically.

===Migrating Between Two AWS S3 locations===

In addition to Amazon AWS S3, WHMCS supports AWS S3 compatible services like Ceph. However, WHMCS only supports automatic migration between two S3 locations if both locations use the same storage provider.

==Migrating From S3 To Local Storage==

You can perform this migration the same way as the other migration paths, with the same considerations and expectations equally applicable.

==General Considerations==

Note that the system can't automatically migrate '''Email Attachments''' (as opposed to email template attachments). You upload these files when creating a custom email message, and they are of a transient nature. To avoid any inconsistencies when switching storage settings for email attachments, it should be sufficient to ensure that no administrator users are in process of composing an email message with attachments. Note that email template attachments are a separate asset type that you can migrate automatically.

==Troubleshooting a File Not Found Error==

If you experience a '''File not found''' error, this may mean that a file is not present or is not accessible at the expected location.

For steps to troubleshoot this, see [[Troubleshooting a File Not Found Error]].

@@ Line 19: / Line 19: @@
 #* You must specify the S3 region by its [https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html code name] (for example, <tt>us-east-1</tt> for ''US East (N. Virginia)'').
 #*The system doesn't require the '''Endpoint URL''' value if you're using the Amazon AWS S3 service. This is only required if you are using a non-Amazon S3-compatible storage provider.
-#*The S3 access credentials must have read and write access to the specified S3 bucket. See below for an example of a suitable policy for the S3 user.
+#*The S3 access credentials must have read and write access to the specified S3 bucket.
 # Click '''Save'''. If the system can't access the bucket, an error message will appear. If this happens, check the entered details and try again.
 # Choose the '''Settings''' tab.
@@ Line 83: / Line 90: @@
 ===Migrating Between Two AWS S3 locations===
 In addition to Amazon AWS S3, WHMCS supports AWS S3 compatible services like Ceph. However, WHMCS only supports automatic migration between two S3 locations if both locations use the same storage provider.
 ==Migrating From S3 To Local Storage==

@@ Line 12: / Line 12: @@
 # Create your S3 configuration. For steps to set one up, see [https://help.whmcs.com/m/system/c/281650 our guides and tutorials].
-# In the WHMCS Admin Area, navigate to '''Configuration (<i class="fa fa-wrench" aria-hidden="true"></i>) > System Settings > Storage Settings''' and click the '''Configuration''' tab.
+# In the WHMCS Admin Area, go to '''Configuration (<i class="fa fa-wrench" aria-hidden="true"></i>) > System Settings > [[Storage Settings]]''' or, prior to WHMCS 8.0, '''Setup > Storage Settings'''.
-# Under '''Add New Configuration''', select '''S3'''.
+# Choose the '''Configuration''' tab.
-# Click the + icon.
+# Under '''Add New Configuration''', choose '''S3'''.
 # Enter your AWS S3 connection details.
-#* You must specify the S3 region by its code name (for example, <tt>us-east-1</tt> for ''US East (N. Virginia)''). To find the region code name for a bucket's region, [https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html click here].
+#* You must specify the S3 region by its [https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html code name] (for example, <tt>us-east-1</tt> for ''US East (N. Virginia)'').
-#*The system doesn't require the '''Endpoint URL''' value if you're using the Amazon AWS S3 service. This is only required if you are using a non-Amazon, S3-compatible storage provider.
+#*The system doesn't require the '''Endpoint URL''' value if you're using the Amazon AWS S3 service. This is only required if you are using a non-Amazon S3-compatible storage provider.
 #*The S3 access credentials must have read and write access to the specified S3 bucket. See below for an example of a suitable policy for the S3 user.
-#Click '''Save'''. If the system can't access the bucket, an error message will appear. If this happens, check the entered details and try again.
+# Click '''Save'''. If the system can't access the bucket, an error message will appear. If this happens, check the entered details and try again.
-#Click on the '''Settings''' tab.
+# Choose the '''Settings''' tab.
-#Select the new storage location for the file type that you wish to migrate to the new location. The system can migrate most file types automatically.
+# Select the new storage location for the file type that you wish to migrate to the new location.
-# Click '''Migrate'''.
+# Click '''Migrate''' when the system prompts you to migrate the files automatically.
-#*Smaller migrations will require less than a minute.
+#*For smaller scale deployments, migration will take less than a minute, and your files will then be stored at the new location.
-#*If the system can't complete the migration immediately, it will schedule it to run in the background. Until it finishes, the system will continue to use the former location.
+#*If the migration cannot be completed immediately, it will be scheduled to run in the background. Until it is completed, the former location will continue to be used.
-#** When the migration is finished, the setting will automatically switch to the new selection and begin using the new location.
+# After the automatic migration finishes, validate whether the migration was successful.
-#** You may choose to cancel an ongoing migration. If you do, the system will continue to use the present location. Files that the system already migrated to the new location will remain.
+# Delete the files in the prior location. The system will not automatically delete these files.
-The system won't automatically remove files from the former location. Do this manually after you verify that the migration was successful.
+When the migration is finished, the setting will automatically switch to the new selection and begin using the new location. You may choose to cancel an ongoing migration. If you do, the system will continue to use the present location. Files that the system already migrated to the new location will remain.
 ===Manual Migration===
@@ Line 68: / Line 69: @@
 ====For Manual Migrations====
 For manual migrations, the system stores files in AWS S3 with a prefix corresponding to the file asset type. For example, it will store file1.png, a client file, as /client_files/file1.png. Use the following sub-folder names per file type:
@@ Line 89: / Line 91: @@
 ==General Considerations==
-Note that the system can't automatically migrate '''Email Attachments''' (as opposed to email template attachments). You upload these files when creating a custom email message, and they are of a transient nature. To avoid any inconsistencies when switching storage settings for email attachments, it should be sufficient to ensure that no administrator users are in process of composing an email message with attachments. Note that email template attachments are a separate asset type that you can migrate automatically.
+The system can't automatically migrate '''Email Attachments''' (as opposed to email template attachments). You upload these files when creating a custom email message, and they are of a transient nature. To avoid any inconsistencies when switching storage settings for email attachments, it should be sufficient to ensure that no administrator users are in process of composing an email message with attachments. Note that email template attachments are a separate asset type that you can migrate automatically.
 ==Troubleshooting a File Not Found Error==