ChangeLog: October 31, 2019 — DiffX, CSS Components, Admin UI

October 31, 2019October 31, 2019 Christian Hammond 1 Comment

This week, we’re going to dive into three major topics:

DiffX, our proposed standard for a better Unified Diff format
Our new in-house CSS Components standard
The work being done on the Review Board Administration UI

DiffX

Unified Diffs, the most common type of diff files, are pretty lacking in standards. Nearly every type of source code management solution outputs its own version of a unified diff, shoving metadata into it however the developers saw fit. This frankly makes it a pain to work with diffs in Review Board, and I’m sure anyone else building similar tools can say the same thing.

A few years ago, we jotted down some ideas of what we’d like to see in diffs. We wanted:

A superset of Unified Diffs that’s fully backwards-compatible (no way would a new diff format gain traction if existing tools couldn’t use it)
A format that could be reliably serialized/deserialized (complete with metadata and file contents) as easily as something like JSON without writing a parser for a dozen different variations of the format
Information on the encodings of a diff file and of the contents of a file, so we no longer have to guess
Changes across multiple commits in a single file
Some standard way of representing changes to binary files

We feel this is important. You wouldn’t believe how wildly different two diff files can be, and how broken most are (just having spaces in a filename means you’re going to run into all sorts of crazy problems).

So we put these ideas together and called it DiffX. Our plan was to build a specification, a standard Python library (and later libraries for other languages), and RBTools/Review Board support.

This week we put up the initial draft of the DiffX specification and documentation. Take a look. See what horrors tools like ours go through, and what we’d like to do about it.

Do you work on another code review tool or source code management system? Let’s talk!

Note that this is still a work-in-progress, and we have a lot we need to better formalize the parsing and the metadata format (and make a public, useful reference implementation).

CSS Components

During Review Board 4.0 development, we kept running into clashes between CSS rules, and had to deeply nest CSS classes and elements in order to ensure our styles would apply. Yuck. This was not sustainable.

We sought to fix this for our code, and looked into what others have done. We considered BEM, BEMIT, and a few other approaches. None felt quite right for our needs (particularly since we want to avoid clashes between third-party extensions).

So we took what we liked in each and drafted our CSS Component Style Guide.

This gives us a better way of thinking about what goes into our stylesheets and HTML pages. We think in terms of formal, reusable components, instead of just haphazardly tacking on styles for whatever HTML we’re writing.

We group together LessCSS variables (used for colors, sizes, etc.) and macros (for customizing a component’s presentation in a safe way) under organized namespaces. Stylesheets (ours and extensions’) that are looking to fit in with our styles or specialize a component can rely on these to better fit in.

We’re also taking care to add formal documentation for everything we add, which forces us to think about our styles, reason about them, and support them.

Want to see this in action? Here’s a few examples:

We’re still working on moving over to this, writing all new CSS using this standard and migrating old styles as needed. This is a long-term project, but we thought it might be interesting to some of you.

If you’re developing extensions for Review Board, we encourage adopting these new styling conventions. It’ll make your life so much easier.

The New Administration UI

Last week, we talked about the work on Python 3 and Django 1.11, and some of the huge tasks we’ve had to work through to support this.

One of the large — and final — hurdles is our administration UI. Django helpfully provides a built-in administration UI that projects get out of the box, and it’s very useful. And we extend and customize it heavily.

The problem is that Django’s administration UI has changed a lot since 1.6, and frankly it wrecked ours. So we’re rebuilding it, better than it was before.

Here’s a breakdown on the new UI:

It’s mobile-friendly
Organization is now much more clear — all navigation is in the sidebar, not split between sidebar and the top header bar
The administration dashboard widgets are so much better — larger, less cluttered, faster, and trimmed down to only the most useful widgets
It’s completely our own, no longer dependent on Django’s rendering, so future upgrades will be smoother

It’s also built with our new CSS component standard, and introduces new components that we can use throughout the product. Extensions can take advantage of these, too, helping them look like a native part of Review Board without worrying too much about our styling.

Here’s a couple in-progress screenshots.

Next Week

We’ll go over some of the new diff functionality being worked on in Review Board 4.0, and what we’ve been up to in RBCommons land.

Again, if you have any questions, or anything you’re curious about and want us to cover, please reach out on our community forum.

ChangeLog: October 24, 2019 – The Python 3 and Django 1.11 Migration

October 24, 2019October 24, 2019 Christian Hammond 2 Comments

Hi, Christian Hammond here. Welcome back to ChangeLog, where we cover the latest going on in Review Board, RBCommons, and Beanbag in general. It’s been a while since we’ve posted one of these, but we’re aiming to bring this back as a weekly series.

There’s a lot to talk about.

The Short Version

We delayed Review Board 4.0 to bring Django 1.11 and Python 3 support. This will not be released by Python 2’s End Of Life date.
Python 3 and Django 1.11 support has been in the works for a long time, but is far more complex than it may seem.

The Longer Version: Where are we today

We’ve had four main focuses this year:

Getting Review Board 4.0 beta 1 ready to launch. This is way behind our original schedule, but intentionally so — we’ll go over the reasons why in a minute.
Bringing integrations options to RBCommons and improving team management, billing, and signup.
Building out new functionality for Power Pack (PDF diffing, and in-progress authentication improvements).
Growing our business (the prior two tasks were part of that), supporting customers, and assisting some with large projects of their own.

This along with a multitude of smaller tasks taking our time throughout the year (and, on a personal note, dealing with the aftermath of the Paradise, CA “Camp Fire.”)

Most of you are here to find out about Review Board, so let’s dive into that.

The “World Update”: Review Board 4.0, Python 3, and Django 1.11

Our original goal for Review Board 4.0 was to bring support for multi-commit review requests and to ship that. This was a large project, as Review Board was originally written before DVCS was common (Subversion and Perforce were the primarily open source and enterprise solutions).

The problem though, is Python 2 End Of Life is coming up fast, and if we kept on schedule with Review Board 4.0, it would be a long time until we’d support Python 3. So we decided to delay 4.0 in order to get it ready for Python 3.

Note that we also still need to support Python 2.7, since many companies are still heavily tied to 2.7 (older distros, custom extensions and scripts).

So what’s hard about supporting Python 3? Welllll….

The Python Compatibility Problem

Python 3 is leaps and bounds better than Python 2 in most ways, but porting a large and complex product from Python 2 to 3 is even harder than you think.

Most Python users know of the major differences:

Modules have moved and functions have been renamed
The default string type has changed from byte strings to Unicode
Some operators have changed
There’s new syntax additions
etc.

We thought we prepared years ago to make this move easy. We used Unicode strings in every file. We used the six module to help with using the right modules, types, and functions.

In the end, it was harder than we expected. Our biggest challenge was definitely the Byte Strings vs. Unicode Strings differences. We thought we were in good shape for this, but we weren’t close.

Review Board does a lot of text processing. We’re parsing uploaded diffs, pulling source files from repositories, matching those up and applying the patch to the source, and generating side-by-side diffs. Much of this logic is over 10 years old, even so, we put a substantial amount of work in preparing for Python 3 years ago, so we were shocked by how much we got this wrong.

The problem comes down to the differences in how Python 2 and 3 would handle shoving two different string types together, which is very easy to do accidentally. Python 2 would roll with it, as in many cases the string types were compatible (and would automatically encode/decode so long as the content was basically ASCII). In Python 3, they outright break — which is a better approach, but hard to transition to — and it led us down a rabbit hole of problems.

Missing b'...' prefixes, changes in string return types from Python functions, functions that are no longer compatible with both string types, and very difficult decisions to make regarding compatibility with third-party SCMs.

All the little regressions and inconsistencies added up. Here’s a few more examples:

Using Python’s pickle library was a mess. Defaults have changed and pickled data became incompatible (due to module reference and string type changes). We had to build in compatibility layers here and test them thoroughly.
Anything that even subtly/unintentionally relied on dictionary sort orders would break.
Getting data in/out of processes, streams, and many other objects and functions is now way more sensitive to encoding issues, and required a lot of careful rewrites and testing.
Many functions (in Python and third-party modules) that used to return lists now return generators, due to their reliance on other functions that changed return types, and this can cause all sorts of subtle behavioral changes.

We’ve spent a lot of time tracking down issues that might immediately crash or might affect data several stages down, and sometimes be traced to something outside our codebase.

Fixing some Python issues means upgrading third-party modules, which can introduce their own new set of changes, regressions, and new rounds of work.

And the biggest source of that was Django.

The Django Problem

Django 1.6 (the version we’ve been using) breaks on Python 3.6+. This meant we had to upgrade to a newer version, something we’ve been putting off.

Django 1.7 introduced built-in support for database migrations. Quite nice for many projects, but the design was sub-optimal for applications like ours (upgrades could take hours or days longer than our approach) and was fundamentally incompatible with our own Django Evolution.

Many other core components of Django (the foundation for the administration UI, template rendering, URL management, forms, and all sorts of other common and obscure parts) have also largely changed over time in some pretty important ways.

We needed a solution for all of this, and we knew that would take time. To compensate we had to:

Rewrite Django Evolution completely to coordinate evolutions and migrations and support modern Django: ~2 years of work, off and on, with hard problems to ponder
Rewrite our administration UI completely to disconnect from Django’s (ensuring we don’t break again when we move to Django 2.x): estimated ~3 months of steady work, still in progress
Build compatibility layers to help keep older code working and help with new code: ~2 months
Just porting in general, updating dependencies, etc: ~3 months

We’re not done, but we’re close.

The Extensions Problem

Review Board is built to allow third-parties to extend its functionality, modify behavior, and link up with other in-house services. We try to be careful not to break extension functionality, and over time we’ve gotten more strict about providing compatibility and an upgrade path for functionality we want to deprecate or change.

The Python 3 and Django 1.11 upgrade is going to affect just about everybody who’s writing an extension. Some of the work we’ve done on adding compatibility layers will help with this process, but it’s going to mean a longer upgrade cycle for some companies.

We knew from the beginning that we’re going to have our hands full helping these companies out (as part of our support contracts), and that it’d be in our best interest to delay the release and break everything all at once instead of spreading out the breakages across multiple releases, and to also give ourselves time to figure out how best to minimize those breakages.

Most projects moving to Python 3 or to newer versions of Django don’t have to worry about this domino effect in the same way.

Timetable?

I hesitate to say when beta 1 will be done (been wrong before), but I can say that the last major piece of development is wrapping up. We’ll be kicking the tires on it, and want to get a beta out as soon as it’s stable enough.

We may not enable Python 3 builds for beta 1, focusing instead on Django 1.11 testing (Python 3 support is still going to be in development during this time), however we’re working with select customers on real-world testing against Python 3.

Community Questions

Every week, we’d like to address some questions, concerns, suggestions, etc. from the community. If you have any questions for next week, please reach out to us.

Q. The community forum seems quiet at times. Why is that?

A. Support requests from companies with support contracts are conducted over a separate support tracker. We always prioritize these support requests over any other work, and increasingly more companies are moving to this method of support for Review Board.

Q. Will Review Board 4.0 ship with Python 3 before the Python 2 End Of Life date?

A. No, it’s going to miss that date. Python 2.7 is still going to be required for now.

Q. There’s years of open review requests on reviews.reviewboard.org. Why is that?

A. We work with university students every semester to help prepare them for their jobs in the tech industry. They spend their semester building features for Review Board, most of which are prototypes. These make up the majority of the review requests on there. Others are contributions that might be outdated, might have been missed, might be incomplete, or might just not have been reviewed yet.

Next Week

We’ll be going over our new CSS component standard for the project and dive deeper into the new administration UI.

Again, if you have any questions, or anything you’re curious about and want us to cover, please reach out on our community forum.

ChangeLog: November 5, 2018

November 5, 2018 Christian Hammond

The World Update: Python 3 and Django 2

In a previous post, we talked the work we’re doing to move to modern versions of Python and Django. We’re calling this the “World Update,” and it’s quite a large and difficult project.

Database Migrations and Django Evolution

Much of the difficulty comes from an incompatibility between how we handle database migrations (using Django Evolution) and how Django handles them (using their built-in migrations support). These aren’t at all compatible, and for many reasons (legacy support and performance), we don’t want to ditch Django Evolution. This has by far been the biggest hurdle to overcome for the World Update.

We’ve been working on this, reworking nearly all of Django Evolution to better utilize migration support in Django, to provide a new set of generic structures for describing database state and modifications, and to (soon) hook into and partly override the migration process to allow both to co-exist.

Progress on this had been steady, and we feel we’re reaching a point where we’ll be able to fully support migrations in Django Evolution.

Djblets and Review Board Work

We now have Djblets working with modern Python 3.x releases and Django 1.6 through 2.1, so we’re in good shape there.

Review Board has a large number of patches in the works to bring Python 3.x and Django 2.1 compatibility. The biggest challenge in Review Board has to do with string type safety. In Python 2.x, Unicode and byte strings are somewhat interchangeable, but there’s a sharp divide in Python 3. We’ve found many cases where we’ve made bad assumptions about string types in Review Board, and are working on both fixing these and adding strict type checks.

World Update Releases

We’re expecting that Review Board 5.0 will require Python 3.5+ and the latest Django 2.x. Administrators will have to make sure they’re running a modern OS with the necessary version of Python, and extension authors will have some updates to perform, but in the end this will be so worth it.

Review Board 4.0 Status

We were initially hoping to release this in August, but that certainly didn’t happen. Too many things came up over this year that took away from the release.

The good news is, DVCS works, and we’re trying to polish up the experience. There’s a few other features we’re also trying to get into 4.0, but once we’re at a point where we feel people can begin safely using DVCS, the beta will start shipping. That will be… soon-ish. We’ll see what happens with the holidays.

ChangeLog: Catching Up

March 13, 2018March 13, 2018 Christian Hammond 2 Comments

It’s been too long since we last ran the ChangeLog series, and felt it was the right time to start it back up again. ChangeLog is a look into the latest behind-the-scenes work going into Review Board, RBCommons, and other Beanbag projects. While intended to be a weekly series, we’d like to start off with some of the bigger tasks and feature development from the past month.

Moving to Django 1.11 and Python 3

Today, all current versions of Review Board depend on Django 1.6, an old release that’s no longer supported by the Django project but is by us, and doesn’t support modern Python 3 releases.

We’ve been stuck on 1.6 because 1.7 introduced (and later mandated) a new way of handling database migrations, which is incompatible with the method we’ve always used. Reconciling the differences has been a challenge.

In the past month, we’ve made significant progress toward both the Django and Python updates:

Djblets 2.0 (our development release) is now compatible with Django 1.6 through 1.11 and Python 2.7, 3.4, 3.5, and 3.6.
Django Evolution (used for database migrations) now works with Django 1.6 through 1.11 and Python 2.7, 3.4, 3.5, and 3.6. Work’s being done to let it co-exist with Django migrations now.
Review Board has started receiving patches for Django 1.11 and Python 3.5+ now. This is still in development, and likely won’t make the Review Board 4.0 release, but will be there for 5.0.
RBTools 1.0 (shipping in a few months) now has full Python 3 support.

New Release Schedules

 We’ve began moving to a train model for releases, and have all of our main and upcoming products now on the calendar.

Here’s what this currently looks like: 

Review Board 4.0 (with DVCS support!) is expected to ship in August, 2018
Review Board 3.0.x releases will ship (generally) every other Tuesday
RBTools 1.0 is expected to ship April 12th
RB-Gateway is expected to ship March 28th

We’re planning to release a new major Review Board release every ~6 months, meaning smaller but more frequent releases. We’re still experimenting with the schedule and timeframe for these releases.

RB-Gateway

We’ve releasing RB-Gateway 1.0 this month. This is a microservice designed to sit in front of a Git or Mercurial repository, providing an API and set of integrations that can be used by Review Board or any other tool or service for more deeply working with your repository.

RB-Gateway doesn’t change your workflow, and can be dropped in with minimal effort. It completely replaces the cgit/gitweb workaround for standalone Git repositories, and means you don’t need to set up something more complicated like GitLab just to work with Review Board.

You’ll see more information on RB-Gateway’s capabilities when we release later this month, and we’ll cover improvements being made to it here.

Wrapping Up…

Those are really just the major highlights, to get everyone up to speed. It doesn’t include the new features we’ve recently built, like being able to filter files in the diff viewer based on filename patterns, a new command for creating Review Board extension source trees, the work done on kgb, or the crazy investigation into deadlocks that’s delayed Review Board 3.0.4.

Going forward, these will be smaller, covering only what’s been done over the past week. If you like these posts, and want to see this continue, please let us know! You can find us on reddit or on the community support list.

Work toward a Django 1.8+ port for Review Board

September 11, 2015December 16, 2015 Christian Hammond

We’ve been dependent on Django 1.6 for our components, for many reasons. There are a lot of things Review Board has to deal with that most Django apps do not, so Django often regresses us, unintentionally. There are also just design changes in Django that don’t suit shipping products, and we’re often having to work around these changes.

However, the biggest bit is our database migration story. Our codebase depends on Django Evolution for migrations, which has to dive pretty far into the internals of Django for some operations. This is a large bit of work to port.

I’m happy to say that, after spending the day on it, I got surprisingly far toward having Django Evolution working on Django 1.8! It’s not perfect, and there’s not yet a good story for playing nice with Django migrations, but it’s a huge start. It opens the doors to getting a full compatibility story going.

The next question is, are we keeping Django Evolution, or moving to migrations fully? Well, that’s a bigger problem, because we have no control over which version of Review Board (and therefore Django) people are upgrading from, and have to be very careful with how we handle any database migrations.

There are also issues with Django’s migrations just being a lot slower than Django Evolution, to the intentional lack of an optimization step when applying the migrations. This means stupidly-long upgrades for large installs, which won’t work for us. So, we’ll probably stay with Django Evolution until we figure out a decent solution there…

Extensions can now reference static assets from Djblets/Review Board

June 18, 2015December 16, 2015 Christian Hammond

I’ve made some fixes to the extension packaging today that allow stylesheets in an extension to reference assets (images, fonts, other stylesheets) found in Review Board or Djblets. This wasn’t working before, due to some assumptions Django makes when it comes to post-processing and packaging of static files.

Django allows you to define a list of static namespaces with a given name, and any static path starting with that name is looked up in that resource. So, djblets/images/<blah> or rb/fonts/<blah>.

Works great in templates and everywhere else, except for static file packaging. There, the assumption appears to be that any referenced resources live in the same app. The paths built are always relative to the directory or the top of the static namespace, making it impossible to do the kind of references an extension author would want.

Well, almost impossible. I pretty much beat that into submission. And here it is: https://reviews.reviewboard.org/r/7437/

Converting database evolutions to Django migrations

June 17, 2015December 16, 2015 Christian Hammond

May have made some progress toward this today…

When you upgrade Review Board, sometimes it needs to modify your database. This will happen if we’ve added new fields, or tables, or renamed a field, or deleted one, or, you know, databasey things like that. This sort of modification is a database schema migration, and they must be done carefully. They also must support all the databases we support (MySQL, PostgreSQL, SQLite).

Historically, for us, this has been done by Django Evolution. Back before Django 1.7, there was no support for database schema migrations, but some clever people came to the rescue with Django Evolution. We’ve been using it since Review Board 1.0 alpha, and for the past several years, we’ve been maintaining it.

Here’s what it does, in a nutshell:

It lets us define what changes need to be made in a database-agnostic way, through “evolution files.”
It figures out what changes have already been applied (through a table that tracks those changes) and what changes still need to be applied.
It then converts those evolution files into a set of mutation operations, optimizes them, and then converts them to SQL statements suitable to the database.

Now, Django 1.7 finally has support for migrations, but of course the format is wildly different. For in-house projects, this is no big deal. You update the database one last time with Django Evolution, then use Django’s migrations from there on out. That doesn’t work with Review Board, though, since any user can go from any release to any release.

We cannot ever upgrade our Django dependency without solving this problem. Oh yeah, and Review Board extensions are affected as well.

So how do we handle this? Well, this is the game plan:

Write a converter to convert the evolution files to Django migration files.
Add a hook just before Django’s migration attempt, that imports the Django Evolution history into Django migration’s format. It would do this for any unimported apps, so we need to track those.
Write some compatibility shims that convert evolution commands to matching Django migration commands, and begin a deprecation phase.

Seems sorta straight-forward, right? Well, this stuff is always more complicated than it should be, so we’ll see.

I’m on step 1, and so far, this is looking like it’ll work. Expect some long posts with descriptions of how gracefully I’m bashing my head into the wall shortly.

Djblets now mostly works with Django 1.8!

June 17, 2015December 16, 2015 Christian Hammond

Djblets is our Django utility library full of reusable components for Review Board. It has datagrids (which powers the dashboard), the API framework, the extension framework, dynamic site configuration, and lots of other utilities.

We just landed a set of changes to bring compatibility with Django 1.8! Let me tell you, going from 1.6 to 1.8 is a huge process, and retaining compatibility with both is not easy. Still, this is a big step.

That’s going into Djblets 0.9, but experimentally. We may actually lock Djblets 0.9 to Django 1.6.x, in order to prevent scenarios where a user upgrades Djblets and it accidentally pulls in the wrong version of Django. We’ll see, though

Also, this does not mean Review Board is anywhere near supporting Django 1.8. That’s a loooong ways off. Review Board 3.0 at the earliest. There’s a lot of work to do there still, and a lot of work to do in Django Evolution to convert to Django’s new migrations support.

(That part is scary.)