Work toward a Django 1.8+ port for Review Board

We’ve been dependent on Django 1.6 for our components, for many reasons. There are a lot of things Review Board has to deal with that most Django apps do not, so Django often regresses us, unintentionally. There are also just design changes in Django that don’t suit shipping products, and we’re often having to work around these changes.

However, the biggest bit is our database migration story. Our codebase depends on Django Evolution for migrations, which has to dive pretty far into the internals of Django for some operations. This is a large bit of work to port.

I’m happy to say that, after spending the day on it, I got surprisingly far toward having Django Evolution working on Django 1.8! It’s not perfect, and there’s not yet a good story for playing nice with Django migrations, but it’s a huge start. It opens the doors to getting a full compatibility story going.

The next question is, are we keeping Django Evolution, or moving to migrations fully? Well, that’s a bigger problem, because we have no control over which version of Review Board (and therefore Django) people are upgrading from, and have to be very careful with how we handle any database migrations.

There are also issues with Django’s migrations just being a lot slower than Django Evolution, to the intentional lack of an optimization step when applying the migrations. This means stupidly-long upgrades for large installs, which won’t work for us. So, we’ll probably stay with Django Evolution until we figure out a decent solution there…

Read More

Extensions can now reference static assets from Djblets/Review Board

I’ve made some fixes to the extension packaging today that allow stylesheets in an extension to reference assets (images, fonts, other stylesheets) found in Review Board or Djblets. This wasn’t working before, due to some assumptions Django makes when it comes to post-processing and packaging of static files.

Django allows you to define a list of static namespaces with a given name, and any static path starting with that name is looked up in that resource. So, djblets/images/<blah> or rb/fonts/<blah>.

Works great in templates and everywhere else, except for static file packaging. There, the assumption appears to be that any referenced resources live in the same app. The paths built are always relative to the directory or the top of the static namespace, making it impossible to do the kind of references an extension author would want.

Well, almost impossible. I pretty much beat that into submission. And here it is: https://reviews.reviewboard.org/r/7437/

Read More

Converting database evolutions to Django migrations

May have made some progress toward this today…

When you upgrade Review Board, sometimes it needs to modify your database. This will happen if we’ve added new fields, or tables, or renamed a field, or deleted one, or, you know, databasey things like that. This sort of modification is a database schema migration, and they must be done carefully. They also must support all the databases we support (MySQL, PostgreSQL, SQLite).

Historically, for us, this has been done by Django Evolution. Back before Django 1.7, there was no support for database schema migrations, but some clever people came to the rescue with Django Evolution. We’ve been using it since Review Board 1.0 alpha, and for the past several years, we’ve been maintaining it.

Here’s what it does, in a nutshell:

  • It lets us define what changes need to be made in a database-agnostic way, through “evolution files.”
  • It figures out what changes have already been applied (through a table that tracks those changes) and what changes still need to be applied.
  • It then converts those evolution files into a set of mutation operations, optimizes them, and then converts them to SQL statements suitable to the database.

Now, Django 1.7 finally has support for migrations, but of course the format is wildly different. For in-house projects, this is no big deal. You update the database one last time with Django Evolution, then use Django’s migrations from there on out. That doesn’t work with Review Board, though, since any user can go from any release to any release.

We cannot ever upgrade our Django dependency without solving this problem. Oh yeah, and Review Board extensions are affected as well.

So how do we handle this? Well, this is the game plan:

  • Write a converter to convert the evolution files to Django migration files.
  • Add a hook just before Django’s migration attempt, that imports the Django Evolution history into Django migration’s format. It would do this for any unimported apps, so we need to track those.
  • Write some compatibility shims that convert evolution commands to matching Django migration commands, and begin a deprecation phase.

Seems sorta straight-forward, right? Well, this stuff is always more complicated than it should be, so we’ll see.

I’m on step 1, and so far, this is looking like it’ll work. Expect some long posts with descriptions of how gracefully I’m bashing my head into the wall shortly.

Read More

Djblets now mostly works with Django 1.8!

Djblets is our Django utility library full of reusable components for Review Board. It has datagrids (which powers the dashboard), the API framework, the extension framework, dynamic site configuration, and lots of other utilities.

We just landed a set of changes to bring compatibility with Django 1.8! Let me tell you, going from 1.6 to 1.8 is a huge process, and retaining compatibility with both is not easy. Still, this is a big step.

That’s going into Djblets 0.9, but experimentally. We may actually lock Djblets 0.9 to Django 1.6.x, in order to prevent scenarios where a user upgrades Djblets and it accidentally pulls in the wrong version of Django. We’ll see, though

Also, this does not mean Review Board is anywhere near supporting Django 1.8. That’s a loooong ways off. Review Board 3.0 at the earliest. There’s a lot of work to do there still, and a lot of work to do in Django Evolution to convert to Django’s new migrations support.

(That part is scary.)

Read More