Upgrading Installed Sterling Databases

Dec 23, 2010 at 4:47 PM

I put together an example project showing a method of upgrading a Sterling Database to a newer version on the Windows Phone 7 (WP7).

Many thanks to Jeremy for a great database and for allowing me to post this example!

It is available here: Sterling Upgrade Sample Project

For example, let's say you have implemented (and shipped) a WP7 application that implements the Sterling database. Then when you add features to your application for the next version, you must provide an upgrade path for your users. That is where this project comes in. It simulates this by first creating a version 1.0 Contacts database, then adding to that database (by adding new properties to the Contact class) through version 2.0, 2.5 and 3.0, upgrading the database along the way.

The project shows how to upgrade the database from any older version to the current version - this is done because you can't guarantee your users have upgraded their app each time you released an update, so you will have some users upgrading from version 1.00 to 3.00 while others are upgrading from version 2.5 to 3.00. You can't know so the DBUpdate project takes the guesswork out for you.

When you build and run this sample, you will be presented with a screen with several buttons. Click the "Create" button first to create the version 1.0 database. Then you can use one of the other buttons to upgrade the database. Remember the database version you are upgrading FROM is purged off the WP7 after the upgrade, which means you cannot click the "Upgrade v1.00 to v2.00" button and then click the "Upgrade v1.00 to v3.00" button WITHOUT clicking the "Create" button between, since the v1.00 database will be purged after the v1.00 to v2.00 upgrade.

You can click the "Show" buttons to show the contents of the database file at each version - note these buttons simply show the contents using MessageBox.Show() - this is done just to show the added properties between versions.

V1.00 database contains a basic set of Contact properties. V2.00 adds the MiddleName property. V2.50 adds the StreetAddress and V3.00 adds the City.

This project also illustrates how to separate the database definitions into separate assemblies and how to code each assembly to know how to upgrade from the prior version to itself. For example, V2.50 database knows how to upgrade from V2.00 to V2.50 only.

This project is NOT meant to be the one and only way to approach this problem. It is simply the way I figured out that works for me. Comments and ideas are always welcome!

You can download this project from here: Sterling Upgrade Sample Project

Jan 11, 2011 at 7:53 AM

Hi MDMiller i was looking for some solution to upgrade DB to new version and i found for post really interesting and good,

I have a question in my mind, what if i have more than 1000 records in my previous version of DB and i want to convert in to new version of DB as you described now as 1000 items may take some time to convert and save and by default if some process at startup takes more than 5 secs WP7 auto kill that process and stops the app from launch.

Tell me how to fit you approach in this scenario and is you method is tombstone safe? what if app goes paused due to some call or event during conversion process?

Jan 13, 2011 at 1:56 PM

Any one who can answer this question?

Feb 19, 2011 at 3:21 PM

Hi,

I have a similar problem: We update our silverlight application every two weeks and of course every version has a new number. We use the same version number for all assemblies in our project - so with each release every assembly will have a new version number. The problem is, that serling uses the full type name in the table definition - when sterling db is activated and reads the table definitions, it tries to find a type name

  • "MyNamespace.MyClass, MyAssembly, Version=1.2.40204.6, Culture=neutral, PublicKeyToken=null"

But after an update that class no longer exists, instead the class

  • "MyNamespace.MyClass, MyAssembly, Version=1.2.40218.2, Culture=neutral, PublicKeyToken=null"

should be used. That problem is serious for us, because we store a lot of informations on the client side and do not want to force our clients to download everthing fresh after each update. So I´ve extended sterling db with a little optional interface called ISterlingTypeInterceptor with is used to resolve types used in table definitions:

namespace Wintellect.Sterling
{
    public interface ISterlingTypeInterceptor
    {
        Type ResolveTableType(string fullTypeName);
    }
}

If an implementation of that interface is provided, it is called when sterling is unable to find a type. That way I am able to define replacement types for the objects stored in the database (of course the new class must have the same properties as the old class). If anybody has the same or a similar problem, just leave a message for me and I will provide the complete code (very short, just a few lines).

Best regards,
kallocain

Coordinator
Feb 22, 2011 at 2:23 PM

Do you consider the type interceptor a decent resolution or would it work to

(a) use FullName instead of AssemblyQualifiedName, or

(b) have a switch on the engine to choose?

If you think the type interceptor is the way to go and doesn't negatively impact performance, let me know and I'll look at getting it into vNext. Thanks!

Feb 24, 2011 at 5:52 AM

Hi Jeremy,

I think sterling db needs a concept to handle version changes or updates of the saved objects. My proposal of a ISterlingTypeInterceptor solves the problem of version changes without the need of updating the stored objects. It works fine for me, without negative performance impact because the resolves types are cached by an internal class:

 

 

namespace Wintellect.Sterling.Database
{
    /// <summary>
    /// Helper class to resolve the types of elements stored in database tables.
    /// </summary>
    internal static class TableTypeResolver
    {
        #region Fields
        private static List<ISterlingTypeInterceptor> _typeInterceptors = new List<ISterlingTypeInterceptor>();
        private static Dictionary<string, Type> _resolvedTypes = new Dictionary<string, Type>();
        #endregion

        #region Internal Methods
        internal static void RegisterTypeInterceptor(ISterlingTypeInterceptor interceptor)
        {
            if (interceptor == null)
            {
                throw new ArgumentNullException("interceptor");
            }

            if (!_typeInterceptors.Contains(interceptor))
            {
                _typeInterceptors.Add(interceptor);
            }
        }

        internal static Type ResolveTableType(string fullTypeName)
        {
            Type tableType = (ResolveCachedType(fullTypeName) ?? 
                ResolveOriginalType(fullTypeName)) ?? 
                ResolveReplacementType(fullTypeName);

            if (tableType == null)
            {
                throw new SterlingUnknownTableType(fullTypeName);
            }

            return tableType;
        }
        #endregion

        #region Private Implementation
        private static Type ResolveCachedType(string fullTypeName)
        {
            Type result;
            _resolvedTypes.TryGetValue(fullTypeName, out result);
            return result;
        }

        private static Type ResolveOriginalType(string fullTypeName)
        {
            Type result = null;

            try
            {
                result = Type.GetType(fullTypeName, false);
                CacheResolvedType(fullTypeName, result);
            }
            catch (TypeLoadException) { }
            catch (FileLoadException) { }
            return result;
        }

        private static Type ResolveReplacementType(string fullTypeName)
        {
            Type result = null;
            foreach (ISterlingTypeInterceptor interceptor in _typeInterceptors)
            {
                result = interceptor.ResolveTableType(fullTypeName);
                if (result != null)
                {
                    CacheResolvedType(fullTypeName, result);
                    break;
                }
            }
            return result;
        }

        private static void CacheResolvedType(string fullTypeName, Type resolvedType)
        {
            if (resolvedType != null)
            {
                _resolvedTypes[fullTypeName] = resolvedType;
            }
        }
        #endregion
    }
}

I will send you my complete code - but maybe it should be part of a broader approach to handle update scenarios.

Best regard,
kallocain

Feb 27, 2011 at 10:30 PM

Can you provide a sample of your solution ? I was not aware of the interceptor extension point of Sterling.

Matthieu

Mar 5, 2011 at 1:40 PM

Hi,

>Can you provide a sample of your solution ? I was not aware of the interceptor extension point of Sterling.

The type interceptor is not part of the current sterling version. It was a small extension I made to solve the problem with version updates for me. If you have a similar problem you should get the current sources and...

1) Add a new interface to Wintellect.Sterling

namespace Wintellect.Sterling
{
    public interface ISterlingTypeInterceptor
    {
        Type ResolveTableType(string fullTypeName);
    }
}

 

2) Add a new class to Wintellect.Sterling.Database

 

namespace Wintellect.Sterling.Database
{
    /// <summary>
    /// Helper class to resolve the types of elements stored in database tables.
    /// </summary>
    internal static class TableTypeResolver
    {
        #region Fields
        private static List<ISterlingTypeInterceptor> _typeInterceptors = new List<ISterlingTypeInterceptor>();
        private static Dictionary<string, Type> _resolvedTypes = new Dictionary<string, Type>();
        #endregion

        #region Internal Methods
        internal static void RegisterTypeInterceptor(ISterlingTypeInterceptor interceptor)
        {
            if (interceptor == null)
            {
                throw new ArgumentNullException("interceptor");
            }

            if (!_typeInterceptors.Contains(interceptor))
            {
                _typeInterceptors.Add(interceptor);
            }
        }

        internal static Type ResolveTableType(string fullTypeName)
        {
            Type tableType = (ResolveCachedType(fullTypeName) ?? 
                ResolveOriginalType(fullTypeName)) ?? 
                ResolveReplacementType(fullTypeName);

            if (tableType == null)
            {
                throw new SterlingUnknownTableType(fullTypeName);
            }

            return tableType;
        }
        #endregion

        #region Private Implementation
        private static Type ResolveCachedType(string fullTypeName)
        {
            Type result;
            _resolvedTypes.TryGetValue(fullTypeName, out result);
            return result;
        }

        private static Type ResolveOriginalType(string fullTypeName)
        {
            Type result = null;

            try
            {
                result = Type.GetType(fullTypeName, false);
                CacheResolvedType(fullTypeName, result);
            }
            catch (TypeLoadException) { }
            catch (FileLoadException) { }
            return result;
        }

        private static Type ResolveReplacementType(string fullTypeName)
        {
            Type result = null;
            foreach (ISterlingTypeInterceptor interceptor in _typeInterceptors)
            {
                result = interceptor.ResolveTableType(fullTypeName);
                if (result != null)
                {
                    CacheResolvedType(fullTypeName, result);
                    break;
                }
            }
            return result;
        }

        private static void CacheResolvedType(string fullTypeName, Type resolvedType)
        {
            if (resolvedType != null)
            {
                _resolvedTypes[fullTypeName] = resolvedType;
            }
        }
        #endregion
    }
}

 

3) Add Method to existing class Wintellect.Sterling.Database.SterlingDatabase:

 

        /// <summary>
        /// Registers a class responsible for type resolution.
        /// </summary>
        public void RegisterTypeInterceptor(ISterlingTypeInterceptor interceptor)
        {
            TableTypeResolver.RegisterTypeInterceptor(interceptor);
        }

 

4) Edit Method _InitializeTable in class Wintellect.Sterling.IsolatedStorage.PathProvider to use the TableTypeResolver:

 

       /// <summary>
        ///     Initializes the database mappings 
        /// </summary>
        private void _InitializeTable(string path, IDictionary<Type, int> dictionaryRef)
        {
            _logManager.Log(SterlingLogLevel.Verbose, string.Format("Initialize tables from path: {0}", path), null);

            if (!StorageHelper.FileExists(path)) return;

            using (var br = StorageHelper.GetReader(path))
            {
                var count = br.ReadInt32();
                var stringBuilder = new StringBuilder();
                for (var i = 0; i < count; i++)
                {
                    var typeName = br.ReadString();
                    var tableIndex = br.ReadInt32();
                    stringBuilder.AppendFormat(" {0}={1} ", tableIndex, typeName);
                    dictionaryRef.Add(TableTypeResolver.ResolveTableType(typeName), tableIndex);
                }
                _logManager.Log(SterlingLogLevel.Information,
                                string.Format("Sterling de-serialized {0} table definitions from path {1}:{2}{3}",
                                              count, path, Environment.NewLine, stringBuilder), null);
            }
        }	

 

5) Edit _Deserialize in Wintellect.Sterling.Serialization.SerializationHelper to use the TableTypeResolver:

private object _Deserialize(BinaryReader br, CycleCache cache)
        {
            var typeName = _typeIndexer(br.ReadInt32());

            if (_DeserializeNull(br))
            {
                return null;
            }

            var typeResolved = TableTypeResolver.ResolveTableType(typeName);	
			
			if (_database.IsRegistered(typeResolved))
			// rest of method unchanged.

 

Thats it. Now you can implement your own type interceptor to handle version changes. Mine looks like:

    /// <summary>
    /// Sterling stores the full type name of classes in the table definitions. After each application
    /// update our version number will change, and sterling refuses to load the data from the 
    /// database because of mismatching types. This class solves this problem.
    /// </summary>
    public class SterlingTypeInterceptor : ISterlingTypeInterceptor
    {
        #region Implementation of ISterlingTypeInterceptor
        /// <summary>
        /// Resolves the type of the table.
        /// </summary>
        /// <param name="fullTypeName">Full name of the type.</param>
        public Type ResolveTableType(string fullTypeName)
        {
            Type result = null;

            // Expected elements of the full type name:
            // 1. Typename, 2. Assemblyname, 3. Version, 4. Culture, 5. Public key token
            List<string> typeParts = new List<string>(fullTypeName.Split(',').Select(n => n.Trim()));

            string newVersionNumber;
            if (TryUpdateVersionNumer(typeParts[2], out newVersionNumber))
            {
                typeParts[2] = newVersionNumber;
                string updatedTypeName = BuildFullTypeName(typeParts);
                result = Type.GetType(updatedTypeName);
            }

            return result;
        }
        #endregion

        #region Private Implementation
        private static bool TryUpdateVersionNumer(string originalVersion, out string updatedVersion)
        {
            bool result = false;
            updatedVersion = originalVersion;
            if (originalVersion.StartsWith("Version", StringComparison.InvariantCultureIgnoreCase))
            {
                IApplicationEnvironment environment = ServiceLocator.Current.GetInstance<IApplicationEnvironment>();
                updatedVersion = string.Format("Version={0}", environment.ClientVersion);
                result = true;
            }
            return result;
        }

        private static string BuildFullTypeName(IEnumerable<string> parts)
        {
            StringBuilder result = new StringBuilder();
            foreach (string part in parts)
            {
                if (result.Length > 0)
                {
                    result.Append(", ");
                }
                result.Append(part);
            }
            return result.ToString();
        }
        #endregion
    }

The type interceptor is registered before the engine is activated:

            myEngine = new SterlingEngine(CreateEngineSettings());
            myEngine.SterlingDatabase.RegisterLogger(SterlingLog);
            myEngine.SterlingDatabase.RegisterTypeInterceptor(new SterlingTypeInterceptor());

If you have any questions, please contact me.


Best regards,
kallocain      

Mar 5, 2011 at 9:04 PM

Hello Kallocain,

Thanks you for the mini-tutorial :) I see now what you are trying to solve. I got another issue: how do you manage the change in your persistent entities. If you add a property by example ?

Matthieu

 

Mar 6, 2011 at 7:48 AM

Hello Matthieu,

> how do you manage the change in your persistent entities

Yes, that is still an open problem. I´ve created the TypeResolver only to solve the problem when the object version changes, but the object signature stays the same.

I am working on a solution for the problem when the object signature is changing (adding, removing, renaming properties). In my approach an additional attribute can be attached to serialized classes: 

[SterlingPersistenceVersion(1)]
public class MyClass
{
   public string MyProperty { get; set;}
}

As long as the given persistence version of the object does not change, sterling tries to create an deserialize an instance of the class (like it does right now). When you change the signature of the class, you should update the version attribute:

[SterlingPersistenceVersion(2)]
public class MyClass
{
   public string MyImprovedProperty { get; set;}
}

In that case sterling should update the table definition and store objects with the signature of the updated class. To allow upgrading from previous versions it should be possible to specify converters:

[SterlingPersistenceVersion(2)]
[SterlingPersistenceConverter(1, typeof(MyUpgradeHelper))]
public class MyClass
{
   public string MyImprovedProperty { get; set;}
}

The upgrade helpers are used to create objects according to the new class signature, and should look like:

 

public class MyUpgradeHelper : SterlingObjectConverter
{
   public override object Convert(int version, Dictionary<string, object> properties)
   {
      return new MyClass 
      { 
         MyImprovedProperty = (string)properties["MyProperty"];
      }
   }
}

There are still a few problems - how to handle changes in custom serializers for example (object signature does not change, but binary representation does change). When I am happy with my upgrade mechanism, I will post an additional mini-tutorial ;)


Best regards,
kallocain

 

May 27, 2011 at 1:55 PM

Hi, kallocain

I have a similar situation where i need to convert my older entities to newer structure, Can you share your code, how do you manage the change in your persistent entities.

Best Regards,

Jun 11, 2011 at 6:39 AM

I've posted a prototype solution for upgrading

have a look

 

http://sterling.codeplex.com/discussions/260876?ProjectName=sterling