Error deleting from Azure Cloud Table - ResourceNotFound

I'm having an intermittent problem deleting objects from an azure table. It only affects about 1% of my attempts, and if the same call is made again later then it works fine, but I'd love to find out the reason behind it! I've googled my fingers off and I've found the lack of documentation out there on how to create very reliable code for deleting, inserting and updating to be quite surprising ... it all seems to be a bit hit and miss, "try it, most of the time it will work"

EDIT: I am deleting the text that was originally in this questions and replacing it with entirely new text to take into consideration things I have tried/that have been suggested.

Do Azure Tables suffer from intermittent failures like SQL Azure. If so, I would have though the "saveChangesWithRetries" would have dealt with that? Is this wrong?

So ... fairly simply code, being called about 250 times a minute on an Azure web-role. The azure tables are used as part of a messaging system. Messages are inserted by one user, downloaded by another, on successful download those messages are marked as read.

Each user has a partition for unread messages, and a partition for read messages. So to mark a message as "read", it is deleted from the unread partition and moved into the read partition.

Of the 250 time this code is called per minute, I'll receive between 2 and 10 of the following errors on the final SaveChangesWithRetries(). The inner exception is:

ResourceNotFound The specified resource does not exist. RequestId:652a3e13-3911-4503-8e49-6fec32a3c044 Time:2011-09-28T22:09:39.0795651Z

I do not imagine an individual partition being accessed more than a few times a minute.

This is my code:

    public static void Message_MarkAsRead(int uid)
    {
        try
        {
            storageAccount = CloudStorageAccount.Parse(connectionString);
            tableClient = new CloudTableClient(storageAccount.TableEndpoint.AbsoluteUri, storageAccount.Credentials);
            tableClient.RetryPolicy = RetryPolicies.Retry(retryAmount, TimeSpan.FromSeconds(retrySeconds));

            TableServiceContext tableServiceContext = tableClient.GetDataServiceContext();
            tableServiceContext.IgnoreResourceNotFoundException = true;

            //the messageUserJoinerTable let's us join messageId to userFromId and userToId
            //each message is inserted into the tables twice, once into the userFromId partition and also into the userToId partition
            #region get the userToId and userFromId for this message uid
            List<int> userIds = new List<int>();
            var resultsUserIds = from messagesUserJoinerTable in tableServiceContext.CreateQuery<MessageUserJoinerDataEntity>(messageUserJoinerTableName)
                                where messagesUserJoinerTable.PartitionKey == uid.ToString()
                                select messagesUserJoinerTable;

            foreach (MessageUserJoinerDataEntity messageUserJoiner in resultsUserIds)
            {
                userIds.Add(messageUserJoiner.UserId);
            }
            #endregion

            #region then we need to check the partition for each of these users and mark the messages as read
            if (userIds.Count > 0)
            {
                foreach (int userId in userIds)
                {
                    var resultsUnreadMessages = from messagesTable in tableServiceContext.CreateQuery<MessageDataEntity>(messageTableName)
                                                where messagesTable.PartitionKey == CreatePartitionKey(userId, false)
                                                && messagesTable.RowKey == CreateRowKey(uid)
                                                select messagesTable;

                    //there should only ever be one as duplicate partition/rowkey is not allowed
                    MessageDataEntity messageUnread = resultsUnreadMessages.FirstOrDefault();

                    if (messageUnread != null)
                    {
                        bool isUnreadMessageDeleted = false;

                        //shallow copy the message for re-inserting as read
                        MessageDataEntity messageRead = new MessageDataEntity(messageUnread);

                        //delete the message
                        try
                        {
                            tableServiceContext.Detach(messageUnread);
                            tableServiceContext.AttachTo(messageTableName, messageUnread, "*");
                            tableServiceContext.DeleteObject(messageUnread);
                            //this is where the error occurs
                            tableServiceContext.SaveChangesWithRetries();

                            isUnreadMessageDeleted = true;
                        }
                        catch (Exception ex)
                        {
                            MyTrace.Trace("AzureCloudTable_" + Service.versionForTracing + ".Message_MarkAsRead. Error.Stage.1: MessageID:" + uid + ", UserID:" + userId + ". " + ex.Message + ex.StackTrace + ", " + ex.InnerException.Message + ex.InnerException.StackTrace, "Error. MarkAsRead");

                            //check to see if the message we tried to delete has already been deleted
                            //if so, we just consume this error and continue by inserting the read message
                            //else, we throw the exception outwards
                            var resultsUnreadMessagesLastCheck = from messagesTable in tableServiceContext.CreateQuery<MessageDataEntity>(messageTableName)
                                                                 where messagesTable.PartitionKey == CreatePartitionKey(userId, false)
                                                                 && messagesTable.RowKey == CreateRowKey(uid)
                                                                 select messagesTable;

                            //there should only ever be one as duplicate partition/rowkey is not allowed
                            MessageDataEntity messageUnreadLastCheck = resultsUnreadMessages.FirstOrDefault();

                            if (messageUnreadLastCheck != null)
                            {
                                MyTrace.Trace("AzureCloudTable_" + Service.versionForTracing + ".Message_MarkAsRead. 开发者_JAVA技巧Error.Stage.2: MessageID:" + uid + ", UserID:" + userId + ". Message WAS deleted.", "Error. MarkAsRead");

                                //the message IS deleted, so although I don't understand why getting error in the first 
                                //place, the result should be the same
                                throw ex;
                            }
                            else
                            {
                                //the message is NOT deleted, so we may as well give up now as I don't understand
                                //what's going on
                                MyTrace.Trace("AzureCloudTable_" + Service.versionForTracing + ".Message_MarkAsRead. Error.Stage.2: MessageID:" + uid + ", UserID:" + userId + ". Message was NOT deleted.", "Error. MarkAsRead");
                            }
                        }

                        //mark the new message as read
                        if (isUnreadMessageDeleted)
                        {
                            messageRead.PartitionKey = CreatePartitionKey(userId, true);
                            messageRead.IsRead = true;

                            //check if read message already exists in storage, if not, insert
                            var resultsReadMessages = from messagesTable in tableServiceContext.CreateQuery<MessageDataEntity>(messageTableName)
                                                      where messagesTable.PartitionKey == CreatePartitionKey(userId, true)
                                                      && messagesTable.RowKey == CreateRowKey(uid)
                                                      select messagesTable;

                            //do the insert
                            if (resultsReadMessages.FirstOrDefault() == null)
                            {
                                tableServiceContext.AddObject(messageTableName, messageRead);
                                tableServiceContext.SaveChangesWithRetries();
                            }
                        }
                    }
                }
            }
            #endregion
        }
        catch (Exception ex)
        {
            try
            {
                MyTrace.Trace("AzureCloudTable_" + Service.versionForTracing + ".Message_MarkAsRead. Error: " + ex.Message + ex.StackTrace + ", " + ex.InnerException.Message + ex.InnerException.StackTrace, "Error. MarkAsRead");
            }
            catch (Exception)
            {
                MyTrace.Trace("AzureCloudTable_" + Service.versionForTracing + ".Message_MarkAsRead. Error: " + ex.Message + ex.StackTrace, "Error. MarkAsRead");
            }
        }
    }

I don't understand how the resource can possible not exist, when it's been returned to me as part of the query and then I've done a != null check on it.

Based on previous responses, I added code to do an additional check in the try to see if the object has already been deleted somehow. it has not been deleted

My tracing returns this on an error:

AzureCloudTable_3_0_5.Message_MarkAsRead. Error.Stage.1: MessageID:146751012, BoyID:477296. An error occurred while processing this request. at Microsoft.WindowsAzure.StorageClient.Tasks.Task1.get_Result() at Microsoft.WindowsAzure.StorageClient.Tasks.Task1.ExecuteAndWait() at BenderRestfulService_3_0_5.AzureCloudTable.Message_MarkAsRead(Int32 uid) in ... ResourceNotFound The specified resource does not exist. RequestId:583c59df-fdac-47e4-a03c-7a4bc7d004c9 Time:2011-10-05T16:37:36.7940530Z at System.Data.Services.Client.DataServiceContext.SaveResult.d__1e.MoveNext()

AzureCloudTable_3_0_5.Message_MarkAsRead. Error.Stage.2: MessageID:146751012, BoyID:477296. Message was NOT deleted.

I am utterly baffled. Any advice greatly appreciated!!!

Steven

---

Is this code running in more than one role instance or in more than one app? (Maybe the entity is being deleted by another process between the time when you read it from the table and the time when you try to delete it.)

---

Problem solved, and I'll explain what I found in case it benefits anyone else.

It was due to threading (as eluded to by Smarx), the fact that I was thinking like a SQL developer, and what I would consider some rather odd/unexpected behaviour by the Azure code and a real lack of in-depth examples!

So, to solve it I simplified the problem as much as possible.

I created a table that contained one entity, PartitionKey 'a', RowKey '1'.

I created a console app that selected 'a' from the table, deleted it, changed it to 'b' and re-inserted it.

I ran the code on a loop, thousands of times, and it all worked fine.

I then moved the code into a thread, and launched two threads. Immediately I was hit with errors and lost the entity somewhere between a delete and an insert.

Problem 1: The two threads can both select the entity at the same, both can check to see if it exists, then both can try and delete it. Only one will succeed when deleting it. So you need to catch the error, check that the error contains "ResourceNotFound", and if so trust that the object is really gone and continue as normal.

Problem 2: The tableContext remembers the last failed action, so the thread where the delete failed will throw another error on SaveChangesWithRetries after you call AddObject. So you need to use a new tableContext for the AddObject

Problem 3: Both threads have a chance at adding the entity, but only one will succeed. Even if both threads check to see if the objects exists before adding it, they could both think that it does NOT exist and both attempt to add it. So for simplicity, let both threads try and add it, one will succeed and one will throw an "EntityAlreadyExists" error. Just catch this error and continue.

Here is my working code for this simple example, I modified it for my more complex example in the original question and now receive no errors at all.

    //method for shifting an entity backwards and forwards between two partitions, a and b
    private static void Shift(int threadNumber)
    {
        Console.WriteLine("Launching shift thread " + threadNumber);

        //set up access to the tables
        _storageAccount = CloudStorageAccount.Parse(_connectionString);
        _tableClient = new CloudTableClient(_storageAccount.TableEndpoint.AbsoluteUri, _storageAccount.Credentials);
        _tableClient.RetryPolicy = RetryPolicies.Retry(_retryAmount, TimeSpan.FromSeconds(_retrySeconds));

        int lowerLimit = threadNumber * _limit;
        int upperLimit = (threadNumber + 1) * _limit;

        for (int i = lowerLimit; i < upperLimit; i++)
        {
            try
            {
                TableServiceContext tableServiceContextDelete = _tableClient.GetDataServiceContext();
                tableServiceContextDelete.IgnoreResourceNotFoundException = true;

                string partitionKey = "a";

                if (i % 2 == 1)
                {
                    partitionKey = "b";
                }

                //find the object with this partition key
                var results = from table in tableServiceContextDelete.CreateQuery<TableEntity>(_tableName)
                              where table.PartitionKey == partitionKey
                                && table.RowKey == "1"
                              select table;

                TableEntity tableEntity = results.FirstOrDefault();

                //shallow copy it
                if (tableEntity != null)
                {
                    TableEntity tableEntityShallowCopy = new TableEntity(tableEntity);

                    if (tableEntityShallowCopy.PartitionKey == "a")
                    {
                        tableEntityShallowCopy.PartitionKey = "b";
                    }
                    else
                    {
                        tableEntityShallowCopy.PartitionKey = "a";
                    }

                    //delete original
                    try
                    {
                        tableServiceContextDelete.Detach(tableEntity);
                        tableServiceContextDelete.AttachTo(_tableName, tableEntity, "*");
                        tableServiceContextDelete.DeleteObject(tableEntity);
                        tableServiceContextDelete.SaveChangesWithRetries();
                        Console.WriteLine("Thread " + threadNumber + ". Successfully deleted. PK: " + tableEntity.PartitionKey);
                    }
                    catch (Exception ex1)
                    {
                        if (ex1.InnerException.Message.Contains("ResourceNotFound"))
                        {
                            //trying to delete an object that's already been deleted so just continue
                        }
                        else
                        {
                            Console.WriteLine("Thread " + threadNumber + ". WTF?! Unexpected error during delete. Code: " + ex1.InnerException.Message);
                        }
                    }

                    //move into new partition (a or b depending on where it was taken from)
                    TableServiceContext tableServiceContextAdd = _tableClient.GetDataServiceContext();
                    tableServiceContextAdd.IgnoreResourceNotFoundException = true;

                    try
                    {
                        tableServiceContextAdd.AddObject(_tableName, tableEntityShallowCopy);
                        tableServiceContextAdd.SaveChangesWithRetries();
                        Console.WriteLine("Thread " + threadNumber + ". Successfully inserted. PK: " + tableEntityShallowCopy.PartitionKey);
                    }
                    catch (Exception ex1)
                    {
                        if (ex1.InnerException.Message.Contains("EntityAlreadyExists"))
                        {
                            //trying to add an object that already exists, so continue as normal
                        }
                        else
                        {
                            Console.WriteLine("Thread " + threadNumber + ". WTF?! Unexpected error during add. Code: " + ex1.InnerException.Message);
                        }
                    }
                }
            }
            catch (Exception ex)
            {
                Console.WriteLine("Error shifting: " + i + ". Error: " + ex.Message + ". " + ex.InnerException.Message + ". " + ex.StackTrace);
            }
        }

        Console.WriteLine("Done shifting");
    }

I'm sure there are much nicer ways of doing this, but due to the lack of good examples I'm just going with something that works for me!

Thanks

---

By default MergeOption.AppendOnly is used by your tableServiceContext.MergeOption, this means that Azure Table Storage will be tracking your table items. You should detach the item before deleting it, like this:

tableServiceContext.Detach(messageUnread);
tableServiceContext.AttachTo(messageTableName, messageUnread, "*");
tableServiceContext.DeleteObject(messageUnread);
tableServiceContext.SaveChangesWithRetries();

This should get rid of any item tracking issues.