tag:status.sift.com,2005:/historySift Status - Incident History2024-03-28T22:23:30-07:00Sifttag:status.sift.com,2005:Incident/198083582024-01-22T19:30:00-08:002024-01-23T09:35:07-08:00Workflow Review Queue Partial Disruption<p><small>Jan <var data-var='date'>22</var>, <var data-var='time'>19:30</var> PST</small><br><strong>Resolved</strong> - Workflow Review Queues were not populated for certain entity types from January 22nd 7:21PM PST - January 23rd 8:51AM PST. This issue has been resolved and Review Queues are being populated again. There was no impact to Score API or Workflow decisions</p>tag:status.sift.com,2005:Incident/197184842024-01-15T16:00:00-08:002024-01-15T17:44:53-08:00Sift API service partial disruption<p><small>Jan <var data-var='date'>15</var>, <var data-var='time'>16:00</var> PST</small><br><strong>Resolved</strong> - Sift API service experienced a partial disruption from 3:58pm PST - 4:50pm PST intermittently. Issue has been resolved and all Sift services are back to normal. We apologize for this disruption.</p>tag:status.sift.com,2005:Incident/196523832024-01-08T18:00:00-08:002024-01-15T17:54:34-08:00API Processing and Console Access Issues<p><small>Jan <var data-var='date'> 8</var>, <var data-var='time'>18:00</var> PST</small><br><strong>Resolved</strong> - From 5:44 - 5:55pm PT, Sift experienced elevated failure rates in Events API requests as well as issues with Console access. The issues were identified and resolved as of 5:55pm PT.</p>tag:status.sift.com,2005:Incident/167402002023-04-03T13:11:05-07:002023-04-03T13:11:05-07:00Sift API endpoints are down due to an infrastructure failure<p><small>Apr <var data-var='date'> 3</var>, <var data-var='time'>13:11</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved. We have fully restored our services.</p><p><small>Apr <var data-var='date'> 3</var>, <var data-var='time'>12:43</var> PDT</small><br><strong>Monitoring</strong> - We have restored the DB and our API endpoints have recovered. We are closely monitoring our services for the full recovery.</p><p><small>Apr <var data-var='date'> 3</var>, <var data-var='time'>12:41</var> PDT</small><br><strong>Update</strong> - We are continuing to work on a fix for this issue.</p><p><small>Apr <var data-var='date'> 3</var>, <var data-var='time'>12:13</var> PDT</small><br><strong>Identified</strong> - We have identified a DB upgrade as the root cause of the incident and we are working on restoring it as soon as possible.</p><p><small>Apr <var data-var='date'> 3</var>, <var data-var='time'>12:00</var> PDT</small><br><strong>Investigating</strong> - We are currently investigating the issue and working on bringing back Sift API endpoints as soon as possible.</p>tag:status.sift.com,2005:Incident/136687052022-11-24T12:42:05-08:002022-11-24T12:42:05-08:00Sift Console is unavailable<p><small>Nov <var data-var='date'>24</var>, <var data-var='time'>12:42</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov <var data-var='date'>24</var>, <var data-var='time'>10:27</var> PST</small><br><strong>Monitoring</strong> - Our systems are back to normal, but we are continuing to monitor.</p><p><small>Nov <var data-var='date'>24</var>, <var data-var='time'>09:17</var> PST</small><br><strong>Investigating</strong> - Sift is experiencing a technical issue with its console. We are actively working on investigating and fixing the issue. We are still processing data and returning scores/workflow decisions, but latency may increase.</p>tag:status.sift.com,2005:Incident/136684182022-11-24T08:49:46-08:002022-11-24T08:49:46-08:00Elevated API failure rates<p><small>Nov <var data-var='date'>24</var>, <var data-var='time'>08:49</var> PST</small><br><strong>Resolved</strong> - The issue has been resolved. Latency and error rates are back to normal. We are continuing to monitor our systems.</p><p><small>Nov <var data-var='date'>24</var>, <var data-var='time'>08:10</var> PST</small><br><strong>Identified</strong> - Sift is experiencing increased latency for all endpoints. This is leading to increased error rates. We are actively working on investigating and fixing issue. We are still processing data and returning scores/workflows for events that do not error.</p>tag:status.sift.com,2005:Incident/136666442022-11-24T03:05:59-08:002022-11-24T03:05:59-08:00Sift Console is unavailable<p><small>Nov <var data-var='date'>24</var>, <var data-var='time'>03:05</var> PST</small><br><strong>Resolved</strong> - Sift Console issue has been resolved and fully operational. Thank you for your patience as we worked through resolving this issue.</p><p><small>Nov <var data-var='date'>24</var>, <var data-var='time'>02:23</var> PST</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Nov <var data-var='date'>24</var>, <var data-var='time'>02:23</var> PST</small><br><strong>Update</strong> - Sift is experiencing a technical issue with its console. We are actively working on investigating and fixing the issue. We are still processing data and returning scores/workflow decisions.</p><p><small>Nov <var data-var='date'>24</var>, <var data-var='time'>02:21</var> PST</small><br><strong>Investigating</strong> - Sift is experiencing a technical issue with its console. We are actively working on investigating and fixing the issue. We are still processing data and returning scores/workflow decisions.</p>tag:status.sift.com,2005:Incident/109249362022-08-25T21:30:04-07:002022-08-25T21:30:04-07:00Console indexing outage<p><small>Aug <var data-var='date'>25</var>, <var data-var='time'>21:30</var> PDT</small><br><strong>Resolved</strong> - At this time the issue with Console Indexing service has been resolved.<br /><br />At Sift we value product excellence. Our customers should expect nothing less. We will work hard to ensure this and similar incidents cannot be repeated.<br /><br />Thanks for fighting fraud with Sift Science.</p><p><small>Aug <var data-var='date'>25</var>, <var data-var='time'>21:28</var> PDT</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Aug <var data-var='date'>25</var>, <var data-var='time'>16:25</var> PDT</small><br><strong>Monitoring</strong> - We have resolved the console indexing service issue and now processing the backlog of events since the incident started. We are closely monitoring the service to ensure there are no further incidents.</p><p><small>Aug <var data-var='date'>25</var>, <var data-var='time'>16:10</var> PDT</small><br><strong>Update</strong> - We have resolved the console indexing service issue and now processing the backlog of events since the incident started. We are closely monitoring the service to ensure there are no further incidents.</p><p><small>Aug <var data-var='date'>25</var>, <var data-var='time'>13:35</var> PDT</small><br><strong>Update</strong> - We are continuing to work on a fix for this issue.</p><p><small>Aug <var data-var='date'>25</var>, <var data-var='time'>13:33</var> PDT</small><br><strong>Update</strong> - Sift is continuing to work on recovering our indexing service.</p><p><small>Aug <var data-var='date'>25</var>, <var data-var='time'>11:30</var> PDT</small><br><strong>Identified</strong> - Sift has identified an issue with our indexing service as of 7:30am Pacific. We are actively working on restoring the service. This only impacts console indexing - Explore and search, but does not impact any data ingestion or APIs. All score and decision pipelines are unaffected. Links to specific users in the Sift console & Review Queues are unaffected.</p>tag:status.sift.com,2005:Incident/108926952022-08-23T11:04:56-07:002022-08-23T11:04:56-07:00Elevated Score API Latency & Mobile events errors<p><small>Aug <var data-var='date'>23</var>, <var data-var='time'>11:04</var> PDT</small><br><strong>Resolved</strong> - We have identified the root cause for this issue and the issue has been resolved. We are monitoring the services closely.</p><p><small>Aug <var data-var='date'>23</var>, <var data-var='time'>09:45</var> PDT</small><br><strong>Identified</strong> - Sift has identified an issue which is causing an increase in response latency, mobile event errors, and Async API events beginning at roughly ~8:47AM PST. We are actively investigating the issue and will follow up as we learn more.</p>tag:status.sift.com,2005:Incident/108064252022-08-11T09:54:13-07:002022-08-11T09:54:13-07:00Elevated API failure rates<p><small>Aug <var data-var='date'>11</var>, <var data-var='time'>09:54</var> PDT</small><br><strong>Resolved</strong> - Between 8:06am PDT and 8:40am PDT, Sift was making infrastructure upgrades which caused approximately 60% of API calls to Sift to fail. The issues have been identified and resolved. We are continuing to monitor our systems.</p>tag:status.sift.com,2005:Incident/85519792021-11-16T10:32:43-08:002021-11-16T10:32:43-08:00Sift Console Inaccessible<p><small>Nov <var data-var='date'>16</var>, <var data-var='time'>10:32</var> PST</small><br><strong>Resolved</strong> - The incident has been resolved and Sift services have been fully restored as of 10:09am PST. Our APIs were affected from 9:47 to 10:09am Pacific.</p><p><small>Nov <var data-var='date'>16</var>, <var data-var='time'>10:14</var> PST</small><br><strong>Monitoring</strong> - Our underlying cloud provider is recovering from a networking issue. Sift services are recovering, and we are continuing to monitor until we confirm a full recovery.</p><p><small>Nov <var data-var='date'>16</var>, <var data-var='time'>09:54</var> PST</small><br><strong>Update</strong> - Sift APIs are now also affected. We are still investigating the issue, but we suspect it is due to issues with our underlying cloud provider.</p><p><small>Nov <var data-var='date'>16</var>, <var data-var='time'>09:47</var> PST</small><br><strong>Investigating</strong> - Sift is currently investigating reports of our console being inaccessible. We will update as we learn more.</p>tag:status.sift.com,2005:Incident/83993272021-11-04T16:34:55-07:002021-11-04T16:34:55-07:00Elevated API failure rates<p><small>Nov <var data-var='date'> 4</var>, <var data-var='time'>16:34</var> PDT</small><br><strong>Resolved</strong> - From 3:32-4:16pm Pacific time, Sift experienced increased API failure rates across all Sift APIs. The issue has been identified and resolved. We are continuing to monitor our systems.</p>tag:status.sift.com,2005:Incident/72975842021-06-21T08:52:51-07:002021-06-21T08:52:51-07:00Elevated Synchronous Workflow & Score Error Rates<p><small>Jun <var data-var='date'>21</var>, <var data-var='time'>08:52</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jun <var data-var='date'>21</var>, <var data-var='time'>08:52</var> PDT</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Jun <var data-var='date'>21</var>, <var data-var='time'>08:44</var> PDT</small><br><strong>Monitoring</strong> - We have identified an issue causing increased latency and error rates across our scoring pathways. A fix has been identified and put in place and we continue to monitor system status.</p>tag:status.sift.com,2005:Incident/70511592021-05-20T15:03:54-07:002021-05-20T15:03:54-07:00Score and Workflow Processing Issues<p><small>May <var data-var='date'>20</var>, <var data-var='time'>15:03</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>May <var data-var='date'>20</var>, <var data-var='time'>14:38</var> PDT</small><br><strong>Monitoring</strong> - As of 2:27pm Pacific, fix has been put in place and error rates and latencies are fully recovered for all scoring pathways. We will continue to monitor.</p><p><small>May <var data-var='date'>20</var>, <var data-var='time'>14:32</var> PDT</small><br><strong>Identified</strong> - Starting at approximately 1:57pm Pacific Sift has experienced increased latencies and failure rates against synchronous and asynchronous scores and workflows. We have identified the issue and are working to resolve it.</p>tag:status.sift.com,2005:Incident/67166672021-04-09T16:16:59-07:002021-04-09T16:16:59-07:00Score and Workflow Processing Issues<p><small>Apr <var data-var='date'> 9</var>, <var data-var='time'>16:16</var> PDT</small><br><strong>Resolved</strong> - All scoring is back to normal.</p><p><small>Apr <var data-var='date'> 9</var>, <var data-var='time'>13:50</var> PDT</small><br><strong>Monitoring</strong> - Synchronous Scores and Workflows are now fully operational. Event ingestion for non synchronous calls is delayed by about 25 minutes.</p><p><small>Apr <var data-var='date'> 9</var>, <var data-var='time'>13:40</var> PDT</small><br><strong>Identified</strong> - Starting at approximately 1pm Pacific today, Sift has identified issues processing score and workflow requests. The issue has been identified and we are working on resolving. No data has been lost.</p>tag:status.sift.com,2005:Incident/64289622021-03-01T10:14:26-08:002021-03-01T10:30:42-08:00Score and Workflow Processing Issues<p><small>Mar <var data-var='date'> 1</var>, <var data-var='time'>10:14</var> PST</small><br><strong>Resolved</strong> - Between 9:29am-9:57am Pacific today, Sift experienced issues processing score and workflow requests. The issue has been resolved and our systems recovered. No data has been lost. We are now processing the backlog of asynchronous requests, we expect this to finish at 10:50am Pacific.</p>tag:status.sift.com,2005:Incident/53187372020-10-13T14:39:42-07:002020-10-13T14:46:53-07:00Elevated Synchronous Workflow & Score Error Rates<p><small>Oct <var data-var='date'>13</var>, <var data-var='time'>14:39</var> PDT</small><br><strong>Resolved</strong> - From 1:41pm-2:10pm Pacific, Sift saw an increased failure rate in our synchronous workflow requests and synchronous scoring. The issue has been identified and resolved as of 2:25pm. Additionally, we are experiencing a delay in ingesting asynchronous events that we received during the impacted time period. We are draining the backlog. No data was lost, and the Score API was unaffected.</p>tag:status.sift.com,2005:Incident/42807402020-06-04T18:20:59-07:002020-06-04T18:20:59-07:00Elevated Sync Scoring & Workflows Error Rates<p><small>Jun <var data-var='date'> 4</var>, <var data-var='time'>18:20</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved. Please contact support@sift.com with any questions.</p><p><small>Jun <var data-var='date'> 4</var>, <var data-var='time'>18:01</var> PDT</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Jun <var data-var='date'> 4</var>, <var data-var='time'>18:01</var> PDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are now monitoring for continued stability.</p><p><small>Jun <var data-var='date'> 4</var>, <var data-var='time'>14:37</var> PDT</small><br><strong>Investigating</strong> - We are currently experiencing elevated error rates in Synchronous Scoring and Workflows. This affects any event that includes the "return_score=true" or "return_workflows=true" parameter. Asynchronous Workflows and the Score API are unaffected. No data has been lost and all events are being processed.</p>tag:status.sift.com,2005:Incident/41581872020-05-20T19:05:42-07:002020-05-20T19:05:42-07:00Delays in Asynchronous Event Ingestion<p><small>May <var data-var='date'>20</var>, <var data-var='time'>19:05</var> PDT</small><br><strong>Resolved</strong> - The incident has been resolved and there should be no more delays in processing. If you have any questions, please contact support@sift.com.</p><p><small>May <var data-var='date'>20</var>, <var data-var='time'>17:40</var> PDT</small><br><strong>Monitoring</strong> - The issue has been resolved, but we are still processing the backlog of events. We expect this to complete in approximately an hour.</p><p><small>May <var data-var='date'>20</var>, <var data-var='time'>16:46</var> PDT</small><br><strong>Identified</strong> - We are currently experiencing delays in ingesting asynchronous events. This includes any event that does not include the return_score or return_workflow=true parameter. Synchronous scoring and workflows are unaffected. No data has been lost and all events will be processed. A fix is in place and we are working to continue to reduce the delays.</p>tag:status.sift.com,2005:Incident/38499202020-04-07T21:19:41-07:002020-04-07T21:19:41-07:00Cloud Service Provider Failures<p><small>Apr <var data-var='date'> 7</var>, <var data-var='time'>21:19</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Apr <var data-var='date'> 7</var>, <var data-var='time'>20:38</var> PDT</small><br><strong>Monitoring</strong> - All Sift systems are operational, but there is a backlog in event processing. We expect the queue to be completely processed in about 25 minutes.</p><p><small>Apr <var data-var='date'> 7</var>, <var data-var='time'>20:28</var> PDT</small><br><strong>Investigating</strong> - We are currently investigating issues with our underlying cloud service provider. We are experiencing a partial failure of requests to our APIs at this time but will update as we assess impact further. Please contact support@sift.com if you have any questions.</p>tag:status.sift.com,2005:Incident/34275572020-01-14T12:50:31-08:002020-01-14T12:50:31-08:00Intermittent Console Slowness<p><small>Jan <var data-var='date'>14</var>, <var data-var='time'>12:50</var> PST</small><br><strong>Resolved</strong> - As of 12 noon PST the console back to normal load times. Please contact support@sift.com if you are still noticing any issues.</p><p><small>Jan <var data-var='date'>14</var>, <var data-var='time'>10:59</var> PST</small><br><strong>Investigating</strong> - We're currently experiencing intermittent console slowness. We're investigating and working to resolve the issue as quickly as possible.</p>tag:status.sift.com,2005:Incident/33535882019-12-23T08:32:49-08:002019-12-23T08:32:49-08:00Intermittent Login & API Issues<p><small>Dec <var data-var='date'>23</var>, <var data-var='time'>08:32</var> PST</small><br><strong>Resolved</strong> - The issue has been resolved. Both logging in and our APIs are back to normal.</p><p><small>Dec <var data-var='date'>23</var>, <var data-var='time'>08:13</var> PST</small><br><strong>Investigating</strong> - We've received reports of users having difficulty logging in and intermittent increased latency on our APIs. We are currently investigating and working to identify and resolve the issue. If you have any questions, please email support@sift.com.</p>tag:status.sift.com,2005:Incident/23762662019-05-07T19:34:13-07:002019-05-07T19:34:13-07:00Elevated Score API Latency<p><small>May <var data-var='date'> 7</var>, <var data-var='time'>19:34</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>May <var data-var='date'> 7</var>, <var data-var='time'>18:53</var> PDT</small><br><strong>Monitoring</strong> - The latencies have been reduced to normal ranges.</p><p><small>May <var data-var='date'> 7</var>, <var data-var='time'>18:36</var> PDT</small><br><strong>Update</strong> - The deployment of our fix is proceeding.</p><p><small>May <var data-var='date'> 7</var>, <var data-var='time'>17:44</var> PDT</small><br><strong>Update</strong> - Our fix is proceeding as planned. We continue to see elevated latencies to a small percentage scoring ATO requests, as well as asynchronous requests for non-ATO scores.</p><p><small>May <var data-var='date'> 7</var>, <var data-var='time'>16:58</var> PDT</small><br><strong>Update</strong> - We're proceeding with our fix for the issue.</p><p><small>May <var data-var='date'> 7</var>, <var data-var='time'>16:18</var> PDT</small><br><strong>Update</strong> - Fewer than 1% of ATO requests continue to see elevated latency. For non ATO products, synchronous requests latencies have returned to normal but we are still seeing elevated latency in a small percentage of asynchronous requests. So, a small percentage of non-ATO, asynchronous workflow runs are evaluating with a delay. We are deploying a fix to address these issues.</p><p><small>May <var data-var='date'> 7</var>, <var data-var='time'>15:30</var> PDT</small><br><strong>Update</strong> - We’ve isolated the issue and have made further progress in reducing elevated latencies. We’re working to completely fix the issue.</p><p><small>May <var data-var='date'> 7</var>, <var data-var='time'>14:53</var> PDT</small><br><strong>Identified</strong> - We’ve made progress in reducing the percentage of calls with elevated latencies. We believe we have identified the issue and are working to get things fully back to normal. We noticed elevated latencies around 12:47pm Pacific Time, so you may have seen an increase in timeouts between then and now. Again, the vast majority of requests are being served at low latency and no data has been lost.</p><p><small>May <var data-var='date'> 7</var>, <var data-var='time'>14:14</var> PDT</small><br><strong>Update</strong> - Ingestion of all events is operational and no data is being lost. The vast majority of all scoring continues to be done at low latencies. We are still seeing elevated scoring latencies in our synchronous and asynchronous scoring services for a small percentage of requests and are attempting to address.</p><p><small>May <var data-var='date'> 7</var>, <var data-var='time'>13:36</var> PDT</small><br><strong>Investigating</strong> - We are currently experiencing elevated synchronous score API latencies and investigating the underlying problem.</p>tag:status.sift.com,2005:Incident/23085172019-04-04T21:29:23-07:002019-04-04T21:29:23-07:00Delays in asynchronous event processing<p><small>Apr <var data-var='date'> 4</var>, <var data-var='time'>21:29</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Apr <var data-var='date'> 4</var>, <var data-var='time'>21:27</var> PDT</small><br><strong>Update</strong> - The backlog of events has been processed. All systems are healthy.</p><p><small>Apr <var data-var='date'> 4</var>, <var data-var='time'>18:38</var> PDT</small><br><strong>Update</strong> - The backlog of events have been processed and there is no more delay on asynchronous Scores and Workflows. There are delays to data in the Explore tab and we are currently working through the backlog of delayed messages.</p><p><small>Apr <var data-var='date'> 4</var>, <var data-var='time'>18:11</var> PDT</small><br><strong>Update</strong> - We are continuing to work through a small backlog of delayed events received through the Event API. Some updated scores, data in Explore, and workflows may be delayed by up to 2 hours after the event is received.</p><p><small>Apr <var data-var='date'> 4</var>, <var data-var='time'>17:40</var> PDT</small><br><strong>Update</strong> - We are continuing to work through a small backlog of delayed events received through the Event API. Some updated scores, data in Explore, and workflows may be delayed by up to 6 hours after the event is received.</p><p><small>Apr <var data-var='date'> 4</var>, <var data-var='time'>17:09</var> PDT</small><br><strong>Update</strong> - We are continuing to work through a small backlog of delayed events received through the Event API. Some updated scores, data in Explore, and workflows may be delayed by up to 6 hours after the event is received.</p><p><small>Apr <var data-var='date'> 4</var>, <var data-var='time'>16:37</var> PDT</small><br><strong>Monitoring</strong> - We are continuing to work through a small backlog of delayed events received through the Event API. Updated scores, data in Explore, and workflows may be delayed by up to 6 hours after the event is received.</p><p><small>Apr <var data-var='date'> 4</var>, <var data-var='time'>16:06</var> PDT</small><br><strong>Update</strong> - We are continuing to investigate a processing delay for a portion of events received through the Event API. Updated scores, data in Explore, and workflows may be delayed by up to 7 hours after the event is received. This does not impact events with synchronous scores or synchronous workflows. No data has been lost.</p><p><small>Apr <var data-var='date'> 4</var>, <var data-var='time'>15:05</var> PDT</small><br><strong>Update</strong> - We are continuing to investigate a processing delay for a portion of events received through the Event API. Updated scores, data in Explore, and workflows may be delayed by up to 4 hours after the event is received. This does not impact events with synchronous scores or synchronous workflows. No data has been lost.</p><p><small>Apr <var data-var='date'> 4</var>, <var data-var='time'>14:34</var> PDT</small><br><strong>Update</strong> - We are continuing to investigate a processing delay for a portion of events received through the Event API. Updated scores, data in Explore, and workflows may be delayed by up to 4 hours after the event is received. This does not impact events with synchronous scores or synchronous workflows. No data has been lost.</p><p><small>Apr <var data-var='date'> 4</var>, <var data-var='time'>13:59</var> PDT</small><br><strong>Update</strong> - We are continuing to investigate a processing delay for a portion of events received through the Event API. Updated scores, data in Explore, and workflows may be delayed by up to 4 hours after the event is received. This does not impact events with synchronous scores or synchronous workflows. No data has been lost.</p><p><small>Apr <var data-var='date'> 4</var>, <var data-var='time'>13:25</var> PDT</small><br><strong>Investigating</strong> - We are currently experiencing a processing delay for a portion of events received through the Event API. Updated scores, data in Explore, and workflows may be delayed by up to 4 hours after the event is received. This does not impact events with synchronous scores or synchronous workflows. No data has been lost.</p>tag:status.sift.com,2005:Incident/23022972019-04-03T07:45:30-07:002019-04-03T07:45:30-07:00Delays in asynchronous event processing<p><small>Apr <var data-var='date'> 3</var>, <var data-var='time'>07:45</var> PDT</small><br><strong>Resolved</strong> - All systems are healthy.</p><p><small>Apr <var data-var='date'> 3</var>, <var data-var='time'>05:15</var> PDT</small><br><strong>Update</strong> - The backlog of events have been processed and there is no more delay on asynchronous processing.</p><p><small>Apr <var data-var='date'> 3</var>, <var data-var='time'>04:39</var> PDT</small><br><strong>Update</strong> - The backlog of events have been processed and there is no more delay on asynchronous Scores and Workflows. There are delays to data in the Explore tab and we are currently working through the backlog of delayed messages.</p><p><small>Apr <var data-var='date'> 3</var>, <var data-var='time'>02:39</var> PDT</small><br><strong>Monitoring</strong> - The backlog of events have been processed and there is no more delay on asynchronous processing.</p><p><small>Apr <var data-var='date'> 2</var>, <var data-var='time'>17:24</var> PDT</small><br><strong>Update</strong> - We are continuing to work through a backlog of messages and expect to resolve delays in 8 hours. We'll update this estimate if this changes.</p><p><small>Apr <var data-var='date'> 2</var>, <var data-var='time'>16:53</var> PDT</small><br><strong>Update</strong> - The issue with delays to asynchronous processing has been identified and a fix has been deployed. We are continuing to work through a backlog of delayed messages.</p><p><small>Apr <var data-var='date'> 2</var>, <var data-var='time'>16:21</var> PDT</small><br><strong>Update</strong> - The issue with delays to asynchronous processing has been identified and a fix has been deployed. We are continuing to process delayed messages.</p><p><small>Apr <var data-var='date'> 2</var>, <var data-var='time'>15:42</var> PDT</small><br><strong>Update</strong> - The issue with delays to asynchronous processing has been identified and a fix has been deployed. We will continue to process delayed messages and provide updates.</p><p><small>Apr <var data-var='date'> 2</var>, <var data-var='time'>14:57</var> PDT</small><br><strong>Identified</strong> - The issue with delays to asynchronous processing has been identified and a fix is being deployed.</p><p><small>Apr <var data-var='date'> 2</var>, <var data-var='time'>14:22</var> PDT</small><br><strong>Update</strong> - We are continuing to actively monitor the delay in asynchronous processing.</p><p><small>Apr <var data-var='date'> 2</var>, <var data-var='time'>13:46</var> PDT</small><br><strong>Update</strong> - We are continuing to actively monitor the delay in asynchronous processing.</p><p><small>Apr <var data-var='date'> 2</var>, <var data-var='time'>13:17</var> PDT</small><br><strong>Update</strong> - We are continuing to actively monitor the delay in asynchronous processing.</p><p><small>Apr <var data-var='date'> 2</var>, <var data-var='time'>12:46</var> PDT</small><br><strong>Update</strong> - We are continuing to actively monitor the delay in asynchronous processing.</p><p><small>Apr <var data-var='date'> 2</var>, <var data-var='time'>12:15</var> PDT</small><br><strong>Update</strong> - We are continuing to actively monitor the delay in asynchronous processing.</p><p><small>Apr <var data-var='date'> 2</var>, <var data-var='time'>11:43</var> PDT</small><br><strong>Update</strong> - We are continuing to actively monitor the delay in asynchronous processing.</p><p><small>Apr <var data-var='date'> 2</var>, <var data-var='time'>11:12</var> PDT</small><br><strong>Update</strong> - We are continuing to actively monitor the delay in asynchronous processing.</p><p><small>Apr <var data-var='date'> 2</var>, <var data-var='time'>10:40</var> PDT</small><br><strong>Update</strong> - We continue to actively monitor the delay in asynchronous processing.</p><p><small>Apr <var data-var='date'> 2</var>, <var data-var='time'>10:06</var> PDT</small><br><strong>Update</strong> - We continue to actively monitor the delay in asynchronous processing.</p><p><small>Apr <var data-var='date'> 2</var>, <var data-var='time'>09:33</var> PDT</small><br><strong>Update</strong> - We continue to actively monitoring the delay in asynchronous processing.</p><p><small>Apr <var data-var='date'> 2</var>, <var data-var='time'>04:14</var> PDT</small><br><strong>Update</strong> - We are adding additional capacity to address the processing delay.</p><p><small>Apr <var data-var='date'> 2</var>, <var data-var='time'>00:58</var> PDT</small><br><strong>Monitoring</strong> - We are currently experiencing a processing delay for events received through the Event API. Updated scores, data in Explore, and workflows may be delayed. We have implemented a fix and are monitoring. This does not impact events with synchronous scores or synchronous workflows. No data has been lost.</p>