Troubleshooting Halo Cloud

This page details how troubleshoot various issues with Halo Cloud, including troubleshooting steps for general connection issues and product-specific issues.

As a general rule, when receiving frequent or persistent errors from a practice, we recommend applying a backoff strategy. For example, by doubling the time between queries each time a query fails. This allows you to continue retrying the query with the minimum overhead to all systems.

Some errors could also be avoided by pausing queries for a given time period based on the error message. For example, polling for appointments for a given GP may error if the GP is away for the day, in which case we recommend pausing any queries for that GP. Halo Cloud will pass on database error messages where possible so that integrators can adjust their product's logic for such cases.

Reducing unnecessary queries doesn't just reduce the number of queries you're charged for, it also reduces load on the practice server and on Halo Connect's systems.

Known issues

SQL Passthrough

Failure to convert parameter values

Impact

When sending a query with command of type storedProcedure, if it contains parameters of type VarBinary or NVarChar, the query returns a 400 with the error "Failed to convert parameter value from a String to a Byte[]. --> Invalid cast from 'System.String' to 'System.Byte[]'. Please correct your request and try again."

Cause

Currently, query parameters specified in command.parameters are converted to SQL parameters using implicit conversion. This fails for large values.

For VarBinary: Implicit conversion fails if the byte array exceeds 8,000 bytes.
For NVarChar: Implicit conversion fails if the string exceeds 4,000 characters.

Workaround

To handle longer strings and byte arrays, explicit object setting is required. Use custom SQL to declare the variables required for the stored procedure with the correct types, then execute the stored procedure using those variables.

The command's type should be "text", not "storedProcedure".

When using a stored procedure, please make sure to include all mandatory fields, and to explicitly assign values to the variables by name if not all fields are used.

Example:

{
    "command": {
        "text": "
        DECLARE @Content VarBinary(max);
        SET @Content = 0xEFBBBF54686973206973206120746573742066696C652E
        execute @returnValue = sp_InsertSampleDocument @Content=@Content
        SELECT @ReturnValue
        ",
        "executionMode": "reader",
        "type": "text"
    }
}

FHIR API

The following known issues should be fixed in a future release. Please contact Support if any of these issues negatively impact your ability to use the FHIR API, so we can ensure any fixes accommodate your use case.

Issue	Impact	Status
Incorrect `_include` and `_revinclude` pagination URL	The `link.url` field in Bundles that use `_include` or `_revinclude` may not include the `_include` or `_revinclude` parameter.	Fixed in upcoming release
Only active records are returned	The FHIR API does not return inactive records.	Fix scheduled
Timezone differences across different resources for `lastUpdated`	There may be slight differences in how timezones are calculated and displayed for the `lastUpdated` field for different resources.
Datetimes must be exact	Datetime-based search parameters only accept the formats `YYYY-MM-DD` and `YYYY-MM-DDThh:mm:ss`.

Troubleshooting

Error codes

The following is a non-exhaustive list of common error codes, and how to mitigate them.

Failed queries will return error data within the query response JSON using the following format, using a 413 as an example:

"errorDetails": {
    "errorType": "http",
    "errorCode": 413,
    "errorMessage": "Query was executed successfully but an error occurred preparing it to be uploaded, results were too large. Immediate query results beyond message size limit (8mb)"
},

or

"errorCode": 413,
"errorMessage": "Query was executed successfully but an error occurred preparing it to be uploaded, results were too large. Immediate query results beyond message size limit (8mb)"

If both exist, use errorDetails and its sub-fields over the root ones.

errorType specifies whether the error is a HTTP error, Halo Link error, or specific to the database the PMS is running, such as MSSQL. This can help narrow down what caused the issue.

200 success

These errors occur when the request was successfully received, understood, and accepted, but something else went wrong.

These errors all return HTTP code 200, and therefore we return a separate Error Code to differentiate errors.

200 errors can be caused by connection disruptions at the practice. The occasional 200 error is expected due to momentary disruptions. Frequent or high volume 200 errors indicate a bigger problem. Please contact Support if you are seeing a high volume of 200 errors.

Error Code	Type	Exception	Description
0	HTTP	`ClientConnectionFailure`	`Client connection was unexpectedly closed`
1	Halo Link	`SqlExecutionFailed`	Various, including: `Error occured while establishing a connection to SQL Server`, `SQL query string cannot be null or void`
2	Halo Link	`PrepForUploadFailed`	Various, including: `Parameter conversion error (e.g. varchar to datetime)`, `Query executed successfully but an error occurred preparing it for upload`
3	Halo Link	`QueryOrphaned`	Async queries only. Query was orphaned during upload or execution
4	Halo Link	`QueryMissingProperties`	`Query missing properties`
5	Halo Link	`QueryUploadFailed`	Async queries only. Request failed with status code PreconditionFailed

4xx client errors

4xx errors are usually caused by issues with the query being sent or something being misconfigured. Please follow the steps listed below to resolve the issue, and if it persists, feel free to contact Support.

Code	Exception	Cause	Action
400	`Login failed for user [userID]`	The practice has not authorised the integration in their PMS database.	Contact the practice to ensure the integration is enabled in the PMS software. Apply backoff strategy until resolved. Cease sending queries if the practice is a churned customer.
400	`BadRequest`	The database has returned an error that is directly related to the content of the SQL query.	Review SQL for errors. Check error message for further details, such as "Invalid column name".
401	`SubscriptionKeyInvalid`, `SubscriptionKeyNotFound`	Subscription key supplied by the integrator is invalid for the environment being queried.	Check subscription key and environment are correct.
401	N/A	The Halo GUID being queried is inactive or not available in this environment.	Check the environment is correct. Check the Halo GUID is correct. If not resolved, contact support.
403	`CallerIpNotAllowed`	The query was sent from an unrecognised IP address.	Contact Support.
403	`Forbidden`	Developer site is not allowed to access this resource. PMS developer licenses cannot be used with Halo Connect's production environment.	Switch Halo Link to `Stage`, or use a fully-licensed PMS.
403	N/A	The practice being queried is non-authoritative.	Re-fetch the practice's Halo GUID. If not resolved, contact Support.
404	`OperationNotFound`	The endpoint URI used was invalid or malformed.	Review the query URI. See the API reference for endpoint details.
406	`QueryNotAcceptable`	Query was executed successfully but an error occurred preparing it to be uploaded. Intermittent 406s may be caused by disruptions to the practice server.	If infrequent, retry query. If frequent or persistent, contact Support.
413	`QueryResultsTooLarge`	Immediate query results are larger than message size limit.	Change the query to reduce the result size or use an async query instead.
429	`RateLimitExceeded`	Too many requests sent too quickly according to the integrator's rate limit.	Reduce query frequency or contact us to discuss a higher limit.

5xx server errors

5xx errors are often momentary. We also proactively monitor practices for connectivity issues that last longer than a few minutes, and reach out to practices ourselves to resolve any issues.

However, if a 5xx error is persistent or frequent, please:

Apply backoff strategy sending queries until resolved.
Contact Support.

HTTP Code	Description
500	Generic error caused by various server-side issues.
503	Connectivity issue between Halo Link and Halo Cloud which results in the error `Site could not be contacted.` with no further information.
504	Gateway timeout which results in errors such as `Query took more than 30 seconds to return a result`, `Site cannot be contacted. Site is currently unreachable.`, `Site cannot be contacted. Site is offline.`