Storage Backend Issues

1: A Backend Fails to Be Created Using the oceanctl Tool and Error Message "context deadline exceeded" Is Displayed
2: The Value of the ONLINE Field Is "false" When the oceanctl Tool Is Used to Obtain Storage Backend Information
3: An Account Is Locked After the Password Is Updated on the Storage Device

1 - A Backend Fails to Be Created Using the oceanctl Tool and Error Message "context deadline exceeded" Is Displayed

Symptom

A user fails to create a storage backend using the oceanctl tool, and “failed to call webhook: xxx :context deadline exceeded; error: exist status 1” is displayed on the console.

Root Cause Analysis

When a storage backend is created, the webhook service provided by CSI is invoked to verify the connectivity with the storage management network and the storage account and password. The possible causes are as follows:

Huawei CSI fails to verify the connectivity of the storage management network.
The communication between kube-apiserver and CSI webhook is abnormal.

Huawei CSI Fails to Verify the Connectivity of the Storage Management Network

Perform the following steps to check whether Huawei CSI fails to verify the connectivity of the storage management network.

Use a remote access tool, such as PuTTY, to log in to any master node in the Kubernetes cluster through the management IP address.

Run the following command to obtain CSI service information. huawei-csi indicates the namespace where the CSI services are deployed.

kubectl get pod -n huawei-csi -owide

The following is an example of the command output.

NAME                        READY   STATUS    RESTARTS   AGE   IP         NODE       NOMINATED NODE   READINESS GATES
huawei-csi-controller-xxx   9/9     Running   0          19h   host-ip1   host-1     <none>           <none>
huawei-csi-node-mnqbz       3/3     Running   0          19h   host-ip1   host-1     <none>           <none>

Log in to the node where huawei-csi-controller resides, for example, host-1 in 2 .
Go to the /var/log/huawei directory.
```
# cd /var/log/huawei
```

View the storage-backend-controller log. The following uses the storage connection timeout as an example.

tail -n 1000 storage-backend-controller

The following is a log example.

2024-01-01 06:30:44.280661 1 [INFO]: Try to login https://192.168.129.155:8088/deviceManager/rest
2024-01-01 06:31:44.281626 1 [ERROR]: Send request method: POST, Url: https://192.168.129.155:8088/deviceManager/rest/xx/sessions, error: Post "https://192.168.129.155:8088/deviceManager/rest/xx/sessions": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2024-01-01 06:31:44.281793 1 [WARNING]: Login https://192.168.129.155:8088/deviceManager/rest error due to connection failure, gonna try another Url
2024-01-01 06:31:44.291668 1 [INFO]: Finished validateCreate huawei-csi/backend-test.
2024-01-01 06:31:44.291799 1 [ERROR]: Failed to validate StorageBackendClaim, error: unconnected

If the log contains information about login timeout, login failure, or long request duration, check the connectivity between the host machine and the storage or the network status.
If no request is recorded in the log, the communication between kube-apiserver and CSI webhook is abnormal.

Abnormal Communication Between kube-apiserver and CSI Webhook

Contact the Kubernetes platform administrator to check the network between kube-apiserver and CSI webhook. For example, if kube-apiserver has an HTTPS proxy, the CSI webhook service may fail to be accessed.

In the temporary workaround, the webhook resource will be deleted. This resource is used to check whether the entered account information is correct and whether the connection to the storage can be set up when a storage backend is created. Therefore, deleting this resource affects only the verification during backend creation and does not affect other functions. Pay attention to the following:
Ensure that the host machine where the huawei-csi-controller service is located can properly communicate with the storage.
Ensure that the entered account and password are correct.

Run the following command to view CSI webhook information.

kubectl get validatingwebhookconfiguration storage-backend-controller.xuanwu.huawei.io

The following is an example of the command output.

NAME                                          WEBHOOKS   AGE
storage-backend-controller.xuanwu.huawei.io   1          4d22h

Contact the Kubernetes platform administrator to check whether the communication between kube-apiserver and CSI webhook is abnormal.
Perform the following temporary workaround: Run the following command to delete the webhook.
```
kubectl delete validatingwebhookconfiguration storage-backend-controller.xuanwu.huawei.io
```
Create a storage backend. For details, see Managing Storage Backends .
If the communication between kube-apiserver and CSI webhook is restored, you need to reconstruct the webhook. In this case, run the following command to restart CSI Controller and restore the number of CSI Controller copies by specifying –replicas=*. In the following example, the number is restored to 1. Change it based on actual requirements.
Change the number of copies to 0 first.
```
kubectl scale deployment huawei-csi-controller -n huawei-csi --replicas=0 
```
Then restore the number of copies to the original number.
```
kubectl scale deployment huawei-csi-controller -n huawei-csi --replicas=1
```

2 - The Value of the ONLINE Field Is "false" When the oceanctl Tool Is Used to Obtain Storage Backend Information

Symptom

The following command is executed to check storage backend status:

oceanctl get backend

The value of the ONLINE field of the storage backend is false:

NAMESPACE    NAME                  PROTOCOL   STORAGETYPE     SN                     STATUS   ONLINE   Url
huawei-csi   backend-201-nas-nfs   nfs        oceanstor-nas   XXXXXXXXXXXXXX000006   Bound    false    https://192.168.129.157:8088

Root Cause Analysis

When CSI uses the account and password entered during storage backend creation to log in to the storage backend, if the login fails due to either of the following reasons, the ONLINE field will be set to false.

Incorrect account password: The possible cause is that the password is changed on the storage backend but not updated in the Kubernetes cluster. For details about how to solve this problem, see Solution or Workaround .
Locked account. For details about how to solve this problem, see An Account Is Locked After the Password Is Updated on the Storage Device .

Solution or Workaround

Obtain the latest account password.
Update the storage backend password by following the instructions in Updating the Password of a Storage Backend Using oceanctl .

3 - An Account Is Locked After the Password Is Updated on the Storage Device

Symptom

After a user changes the password on the storage device, the account is locked.

Root Cause Analysis

CSI uses the account and password configured on the storage device to log in to the storage device. After the account password is changed on the storage device, CSI attempts to log in to the storage device again after the login fails. Take OceanStor Dorado as an example. The default login policy is that an account will be locked after three consecutive password verification failures. Therefore, when CSI retries for more than three times, the account will be locked.

Solution or Workaround

If the backend account is admin, run the following command to set the number of huawei-csi-controller service copies to 0. If an account other than admin is used, skip this step.
```
kubectl scale deployment huawei-csi-controller -n huawei-csi --replicas=0
```
Log in to the storage device as user admin and modify the login policy. Take OceanStor Dorado as an example. On DeviceManager, choose Settings > User and Security > Security Policies > Login Policy, click Modify, and disable Account Lockout.
If the backend account is admin, run the following command to restore the number of CSI Controller copies using –replicas=*. In the following example, the number of copies is restored to 1. Change it based on site requirements. If an account other than admin is used, skip this step.
```
kubectl scale deployment huawei-csi-controller -n huawei-csi --replicas=1
```
Use the oceanctl tool to change the storage backend password. For details about how to change the backend password, see Updating a Storage Backend .
Log in to the storage device as user admin and modify the login policy. Take OceanStor Dorado as an example. On DeviceManager, choose Settings > User and Security > Security Policies > Login Policy, click Modify, and enable Account Lockout.