We believe we have now fully mitigated the recent incident affecting new content uploads to Signagelive from 17th November 2023 onwards. This was due to an issue with our cloud provider. Should issues persist, please get in touch with our Support Team as any outstanding issues will be handled on a case-by-case basis. The progression of this matter can be tracked below and is considered a Priority 1 issue.

Whilst the repercussions of this recent incident have been fully mitigated as of December 1st 2023, some content uploaded between 17th November 2023 and when our new resources were put in place may still be impacted. In such cases, please get in touch with our Support Team, additionally - we have provided some additional troubleshooting steps which may assist (depending on the issue faced):

Failing any of the above, please get in touch with our Support Team and we can assist you further with this matter.

As of 1st December 2023 16:30 UTC - we believe we have fully mitigated the issue for new assets and other secondary processes, however, testing is ongoing. Where customers are using assets uploaded since 17th November 2023 15:45 UTC and before our mitigations had been implemented (due to data being in an unexpected state due to API inconsistencies), customer-specific cases continue to be fixed on a case-by-case basis. We are running scans to proactively determine which customers are likely to have issues, however, due to ongoing inconsistencies from Cloud Files, this is a challenging task. Please raise these with Signagelive Support who are resolving issues on a case-by-case basis.

We have a full incident report for this matter available upon request.

----------------------------------------------------------------------------

We are actively addressing challenges arising from an issue with an external service provider. While their service has not yet been fully restored, we've taken steps to minimise the impact on our services.

As of now we have deployed solutions for the following functions -

Given the speed at which these changes have been implemented, it is possible you may see some inconsistencies, however, we are monitoring these and mitigating them as we move through this issue.

We are still working on solutions for the following, and you may see some inconsistencies until we have the redundancy as a result of the dual storage model -

----------------------------------------------------------------------------

Following our previous update in which we adapted our key process to make use of a dual storage solution - as of right now our Development Team have deployed fixes for the following functions:

Given the speed at which these changes have been implemented, it is possible you may see some inconsistencies, however, we are monitoring these and mitigating them as we move through this issue.

We are still working on solutions for the following, and you may see some inconsistencies until we have the redundancy as a result of the dual storage model -

Thumbnails

Please note that if a thumbnail is not generated for a media asset or does not load, that does not mean it cannot be used in a playlist. If there is metadata (file type, resolution etc) available, then the asset can be added to a playlist and published.

Online / Offline Widget

An offline widget is one that is downloaded to a media player and played from the player's internal storage i.e. most widgets with notable exceptions such as YouTube or widgets published to the Browser Player.

An online widget is one that cannot be downloaded to a player and transparently runs as a static website, notably YouTube or those published to the Browser Player.

Player Specific / Non Player specific widgets

A player-specific widget is one which uses properties from the player, these are unpacked as they are processed by Signagelive and player-specific versions with the player properties embedded. No other changes are made by Signagelive.

A non-player specific widget is one that is published as it is uploaded and does not use properties from the player.

Message Manager

When uploading new assets currently, the preview does not load, however, if you click either Add to add the image ‘as-is’ or ‘Edit’ to edit the text layers, it will load and successfully deploy to your player.

----------------------------------------------------------------------------

We have made some significant changes to how media files are stored and processed to mitigate the issues caused by the Rackspace Cloud Files issue whilst Rackspace is working to resolve the issue with their systems.

At a high level, we are storing files and reading in both Rackspace Cloud Files and (now) Amazon S3.

We believe that uploads to Rackspace Cloud Files are actually working they are just not available, and this is inconsistent, varying over time.

We have adapted most of our key processes to both dual store and check both storage locations when loading files, therefore, should either have any read issues, one will return the file; this is a transparent process and fully handled by the Signagelive system.

We have deployed solutions for the following functions:

Given the speed at which these changes have been implemented, it is possible you may see some inconsistencies, however, we are monitoring these and mitigating them as we move through this issue.

We are still working on solutions for the following, and you may see some inconsistencies until we have the redundancy as a result of the dual storage model -

Thumbnails

Please note that if a thumbnail is not generated for a media asset or does not load, that does not mean it cannot be used in a playlist. If there is metadata (file type, resolution etc) available, then the asset can be added to a playlist and published.

Online / Offline Widget

An offline widget is one that is downloaded to a media player and played from the player's internal storage i.e. most widgets with notable exceptions such as YouTube or widgets published to the Browser Player.

An online widget is one that cannot be downloaded to a player and transparently runs as a static website, notably YouTube or those published to the Browser Player.

Player Specific / Non Player specific widgets

A player-specific widget is one which uses properties from the player, these are unpacked as they are processed by Signagelive and player-specific versions with the player properties embedded. No other changes are made by Signagelive.

A non-player specific widget is one that is published as it is uploaded and does not use properties from the player.

Message Manager

When uploading new assets currently, the preview does not load, however, if you click either Add to add the image ‘as-is’ or ‘Edit’ to edit the text layers, it will load and successfully deploy to your player.

----------------------------------------------------------------------------

We would like to thank everyone who is impacted by this issue for their patience and understanding. Rest assured that our Development Team is doing everything possible to complete this mitigation of the cloud provider issue, but this is not a small undertaking. Our Support Team is working to ensure that all customers are kept informed of the progress, and provide assistance on their tickets.

We would like to also make the following clarification about this issue:

We are aware that a symptom of this issue is that Thumbnails within your Signagelive asset library may not be generating and therefore visually this will represent to you there is an underlying issue. Whilst that is true in terms of the thumbnail generation, we wish to make explicitly clear that these files are still processing and deployable to your Signagelive Players.
​
​A lack of thumbnail does not mean you cannot schedule and/or deploy your content.

----------------------------------------------------------------------------

Our Development Team have begun deploying updates to begin mitigating the Rackspace issues. This is a large multi-step process, so you may notice slower speeds when navigating the user interface while these updates are taking place. Please notify our Support Team of any issues you experience, our Support Team will also proactively reach out if we notice any further issues on your behalf.

As and when we make progress with this one, we'll update this article accordingly.

----------------------------------------------------------------------------

The issue is still ongoing and is a priority 1 item. Our development team are working on putting additional resources in place to mitigate the issue whilst we work with our cloud provider to resolve this. Following some additional reports, this issue may materialize in the following manners:

----------------------------------------------------------------------------

We are aware of an ongoing issue in which media assets are not able to be downloaded to Players - this has resulted in black screens in some instances and old schedules being shown in others. This may be affecting all customers, although we have only had a small number of reports at present.

This is due to an issue with Rackspace, our cloud provider to which we upload and store uploaded media assets. The results are that uploaded assets that are affected are not actually being uploaded to our cloud storage, resulting in Signagelive Players being unable to download these assets if published to a Player.

This issue has been known by Rackspace since November 17th, and you can view the status of their issue here.