ChangeLog: October 24, 2019 – The Python 3 and Django 1.11 Migration

October 24, 2019October 24, 2019 Christian Hammond 2 Comments

Hi, Christian Hammond here. Welcome back to ChangeLog, where we cover the latest going on in Review Board, RBCommons, and Beanbag in general. It’s been a while since we’ve posted one of these, but we’re aiming to bring this back as a weekly series.

There’s a lot to talk about.

The Short Version

We delayed Review Board 4.0 to bring Django 1.11 and Python 3 support. This will not be released by Python 2’s End Of Life date.
Python 3 and Django 1.11 support has been in the works for a long time, but is far more complex than it may seem.

The Longer Version: Where are we today

We’ve had four main focuses this year:

Getting Review Board 4.0 beta 1 ready to launch. This is way behind our original schedule, but intentionally so — we’ll go over the reasons why in a minute.
Bringing integrations options to RBCommons and improving team management, billing, and signup.
Building out new functionality for Power Pack (PDF diffing, and in-progress authentication improvements).
Growing our business (the prior two tasks were part of that), supporting customers, and assisting some with large projects of their own.

This along with a multitude of smaller tasks taking our time throughout the year (and, on a personal note, dealing with the aftermath of the Paradise, CA “Camp Fire.”)

Most of you are here to find out about Review Board, so let’s dive into that.

The “World Update”: Review Board 4.0, Python 3, and Django 1.11

Our original goal for Review Board 4.0 was to bring support for multi-commit review requests and to ship that. This was a large project, as Review Board was originally written before DVCS was common (Subversion and Perforce were the primarily open source and enterprise solutions).

The problem though, is Python 2 End Of Life is coming up fast, and if we kept on schedule with Review Board 4.0, it would be a long time until we’d support Python 3. So we decided to delay 4.0 in order to get it ready for Python 3.

Note that we also still need to support Python 2.7, since many companies are still heavily tied to 2.7 (older distros, custom extensions and scripts).

So what’s hard about supporting Python 3? Welllll….

The Python Compatibility Problem

Python 3 is leaps and bounds better than Python 2 in most ways, but porting a large and complex product from Python 2 to 3 is even harder than you think.

Most Python users know of the major differences:

Modules have moved and functions have been renamed
The default string type has changed from byte strings to Unicode
Some operators have changed
There’s new syntax additions
etc.

We thought we prepared years ago to make this move easy. We used Unicode strings in every file. We used the six module to help with using the right modules, types, and functions.

In the end, it was harder than we expected. Our biggest challenge was definitely the Byte Strings vs. Unicode Strings differences. We thought we were in good shape for this, but we weren’t close.

Review Board does a lot of text processing. We’re parsing uploaded diffs, pulling source files from repositories, matching those up and applying the patch to the source, and generating side-by-side diffs. Much of this logic is over 10 years old, even so, we put a substantial amount of work in preparing for Python 3 years ago, so we were shocked by how much we got this wrong.

The problem comes down to the differences in how Python 2 and 3 would handle shoving two different string types together, which is very easy to do accidentally. Python 2 would roll with it, as in many cases the string types were compatible (and would automatically encode/decode so long as the content was basically ASCII). In Python 3, they outright break — which is a better approach, but hard to transition to — and it led us down a rabbit hole of problems.

Missing b'...' prefixes, changes in string return types from Python functions, functions that are no longer compatible with both string types, and very difficult decisions to make regarding compatibility with third-party SCMs.

All the little regressions and inconsistencies added up. Here’s a few more examples:

Using Python’s pickle library was a mess. Defaults have changed and pickled data became incompatible (due to module reference and string type changes). We had to build in compatibility layers here and test them thoroughly.
Anything that even subtly/unintentionally relied on dictionary sort orders would break.
Getting data in/out of processes, streams, and many other objects and functions is now way more sensitive to encoding issues, and required a lot of careful rewrites and testing.
Many functions (in Python and third-party modules) that used to return lists now return generators, due to their reliance on other functions that changed return types, and this can cause all sorts of subtle behavioral changes.

We’ve spent a lot of time tracking down issues that might immediately crash or might affect data several stages down, and sometimes be traced to something outside our codebase.

Fixing some Python issues means upgrading third-party modules, which can introduce their own new set of changes, regressions, and new rounds of work.

And the biggest source of that was Django.

The Django Problem

Django 1.6 (the version we’ve been using) breaks on Python 3.6+. This meant we had to upgrade to a newer version, something we’ve been putting off.

Django 1.7 introduced built-in support for database migrations. Quite nice for many projects, but the design was sub-optimal for applications like ours (upgrades could take hours or days longer than our approach) and was fundamentally incompatible with our own Django Evolution.

Many other core components of Django (the foundation for the administration UI, template rendering, URL management, forms, and all sorts of other common and obscure parts) have also largely changed over time in some pretty important ways.

We needed a solution for all of this, and we knew that would take time. To compensate we had to:

Rewrite Django Evolution completely to coordinate evolutions and migrations and support modern Django: ~2 years of work, off and on, with hard problems to ponder
Rewrite our administration UI completely to disconnect from Django’s (ensuring we don’t break again when we move to Django 2.x): estimated ~3 months of steady work, still in progress
Build compatibility layers to help keep older code working and help with new code: ~2 months
Just porting in general, updating dependencies, etc: ~3 months

We’re not done, but we’re close.

The Extensions Problem

Review Board is built to allow third-parties to extend its functionality, modify behavior, and link up with other in-house services. We try to be careful not to break extension functionality, and over time we’ve gotten more strict about providing compatibility and an upgrade path for functionality we want to deprecate or change.

The Python 3 and Django 1.11 upgrade is going to affect just about everybody who’s writing an extension. Some of the work we’ve done on adding compatibility layers will help with this process, but it’s going to mean a longer upgrade cycle for some companies.

We knew from the beginning that we’re going to have our hands full helping these companies out (as part of our support contracts), and that it’d be in our best interest to delay the release and break everything all at once instead of spreading out the breakages across multiple releases, and to also give ourselves time to figure out how best to minimize those breakages.

Most projects moving to Python 3 or to newer versions of Django don’t have to worry about this domino effect in the same way.

Timetable?

I hesitate to say when beta 1 will be done (been wrong before), but I can say that the last major piece of development is wrapping up. We’ll be kicking the tires on it, and want to get a beta out as soon as it’s stable enough.

We may not enable Python 3 builds for beta 1, focusing instead on Django 1.11 testing (Python 3 support is still going to be in development during this time), however we’re working with select customers on real-world testing against Python 3.

Community Questions

Every week, we’d like to address some questions, concerns, suggestions, etc. from the community. If you have any questions for next week, please reach out to us.

Q. The community forum seems quiet at times. Why is that?

A. Support requests from companies with support contracts are conducted over a separate support tracker. We always prioritize these support requests over any other work, and increasingly more companies are moving to this method of support for Review Board.

Q. Will Review Board 4.0 ship with Python 3 before the Python 2 End Of Life date?

A. No, it’s going to miss that date. Python 2.7 is still going to be required for now.

Q. There’s years of open review requests on reviews.reviewboard.org. Why is that?

A. We work with university students every semester to help prepare them for their jobs in the tech industry. They spend their semester building features for Review Board, most of which are prototypes. These make up the majority of the review requests on there. Others are contributions that might be outdated, might have been missed, might be incomplete, or might just not have been reviewed yet.

Next Week

We’ll be going over our new CSS component standard for the project and dive deeper into the new administration UI.

Again, if you have any questions, or anything you’re curious about and want us to cover, please reach out on our community forum.

Connect RBCommons to Slack, Travis CI, Asana, and more

August 5, 2019August 6, 2019 Christian Hammond

All RBCommons plans now support integrating with third-party chat, continuous integration, and project management services, with more services to come.

You can connect with Slack or Mattermost and keep your team members notified whenever a review request is created or updated, or when any new discussions take place.

Slack Integration

Automatically trigger builds of uploaded changes on Travis CI or CircleCI, ensuring product builds still pass. You can even run automated code review tools and report status using RBTools.

Link your Asana or Trello tasks with your review requests, and add all your code review work to your status report automatically using I Done This.

Asana Integration

Each integration configuration can specify the rules under which the integration will run. For instance, when a build should take place or which Slack teams/channels should be used for which repositories or review groups.

Configuring Condition Rules

You can define as many integration configurations as you like, up to your plan’s limit:

Starter: 1
Medium: 10
Large: 25
Enterprise: 50

See our integration guides to get started today.

kgb 4.0 – Enhanced Function Spies for Python Unit Tests

July 30, 2019 Christian Hammond

Today, we’ve released kgb 4.0, the latest in our handy Python module for creating function spies in unit tests.

For those new to kgb, function spies allow unit tests to track when functions or methods are called (how many times, and with what parameters and results), and allow functions or methods to be overridden (for instance, to simulate an HTTP result using urllib2). JavaScript developers have Jasmine, and Python developers have kgb. See how documentation for more info there.

So what’s new in kgb 4.0?

Calling a spy’s original function

When spying on a function, a caller (or the spy’s replacement function) can now invoke the original behavior for the function. Unlike a call to the spy’s version, this call will not be logged. It’s really useful for keeping the original functionality intact but adding some parameter manipulation or additional tracking.

For example:

stored_results = []

def my_fake_function(*args, **kwargs):
    kwargs['bar'] = 'baz'
    result = obj.function.call_original(*args, **kwargs)
    stored_results.append(result)

    return result

agency.spy_on(obj.function, call_fake=my_fake_function)

Better Python 3 support

We’ve steadily been improving Python 3 support. It works well, but kgb 3.x would trigger some deprecation warnings when setting up a spy. We’ve fixed this up, future-proofed things some.

To learn more about kgb…

Visit the documentation to see all the way that kgb spies can work for you.

Installation is simple:

$ pip install kgb

If you find kgb useful, we’d love to hear it!

Power Pack 3.0.2: Fixes for Team Foundation Server

June 17, 2019 Christian Hammond

Power Pack 3.0.2 improves integration with Microsoft’s Team Foundation Server:

Copied files containing non-ASCII filenames can now be diffed
Compatibility between various versions of Review Board, Python, and Team Foundation Server has improved

There’s also several behind-the-scenes changes preparing Power Pack for new features we have in the works, and for the upcoming Review Board 4.0 release.

Update Today

Power Pack 3.0.2 is recommended for all Power Pack users reviewing code over Team Foundation Server.

To upgrade, or to install for the first time, see the installation instructions.

RBTools 1.0.2: Fixes for Python 3, Two-Factor Auth, and More

April 25, 2019April 25, 2019 Christian Hammond

Improved Python 3 Support

RBTools 1.0 introduced support for Python 3, and since then many more of our users have switched over and sent us patches to improve that support. We’ve also improved our testing, helping us to maintain a more stable Python 3 codebase.

Two-Factor Auth for RBCommons

The support for Two-Factor Authentication in RBCommons has been completely redone to avoid login rate limit issues, missing headers, and trouble logging in.

Going forward, RBTools 1.0.2 will be the minimum version required for RBCommons accounts using Two-Factor Authentication.

Git Improvements

We’ve improved upon the smart tracking branch detection logic introduced in RBTools 1.0, which is designed to find the right tracking branch for your local changes. It now does a better job of finding a suitable branch if your repository doesn’t have an origin remote, and gives priority to the one provided in --tracking-branch.

Support for disabling Git file rename detection has also been added, for those cases where Git is getting too aggressive and making for bad diffs. Simply pass --no-renames to rbt post or rbt diff to generate a diff without renamed files.

A Step Toward Better Error Messages

We’ve working to improve error messages throughout our products, to help guide people when things go wrong.

If RBTools is pointing to a bad Review Board URL, it no longer just fails with an HTTP status code or cryptic error message. RBTools will now inspect the URL to determine what may have gone wrong, and offer guidance on resolving the problem.

Error messages in our API and other commands have also been fixed. We’ll be making further improvements in future releases.

Plus More

Perforce diffs now contain information on binary files
Aliases invoking shell commands now preserve their quotes and escape sequences
Patches from users with private profiles enabled can now be applied to new commits without crashing

See the release notes for the full list of changes.

Power Pack 3.0: PDF Diffs and License Updates

February 5, 2019 Christian Hammond

The new major release of Power Pack 3.0 brings the ability to diff PDF documents, comparing how the text of the document changes between revisions, and makes it easier to manage your license subscriptions.

Viewing Differences in PDFs

This can drastically cut down on the time needed to read through documents as the author takes in suggested edits from reviewers. Just like a code diff, any text changes made in a document are shown inline in the PDF, color-coded for easier viewing.

A handy new sidebar view catalogues all the changes made throughout the document, so there’s no need to carefully scrutinize as you scroll.

If you do need to scroll, a new “Lock scroll” checkbox gives you control over whether the documents should scroll in sync, or scroll individually.

In order to enable diffing support for PDFs, you will need a PDF document that contains text information embedded in the document (such as when printing to PDF or using OCR on a scanned document). It’s also important to update the existing PDF file attachment with the new document, instead of creating a brand new upload.

Easier License Management

We’ve revamped the Power Pack configuration page to better show the status and health of your license, how quickly the expiration date is coming up or whether you’re hitting your user cap.

The new “Manage your license” button takes you straight to our license portal where you can renew your license, convert to a yearly subscription, add additional users, and more.

Power Pack now checks for updates to your license automatically when viewing the Power Pack configuration page, and will install any new license it finds. You no longer need to download and install new license files from the license portal yourself.

Plus the Usual Bug Fixes

We’ve sorted out some crashes and visual glitches in reports, as well as a compatibility problem with AWS CodeCommit. The full list of changes are in the release notes.

Get started today with a 30 day trial license. After 30 days, enjoy a complimentary license for up to 2 users forever, or purchase a license for the rest of your organization.

ChangeLog: November 5, 2018

November 5, 2018 Christian Hammond

The World Update: Python 3 and Django 2

In a previous post, we talked the work we’re doing to move to modern versions of Python and Django. We’re calling this the “World Update,” and it’s quite a large and difficult project.

Database Migrations and Django Evolution

Much of the difficulty comes from an incompatibility between how we handle database migrations (using Django Evolution) and how Django handles them (using their built-in migrations support). These aren’t at all compatible, and for many reasons (legacy support and performance), we don’t want to ditch Django Evolution. This has by far been the biggest hurdle to overcome for the World Update.

We’ve been working on this, reworking nearly all of Django Evolution to better utilize migration support in Django, to provide a new set of generic structures for describing database state and modifications, and to (soon) hook into and partly override the migration process to allow both to co-exist.

Progress on this had been steady, and we feel we’re reaching a point where we’ll be able to fully support migrations in Django Evolution.

Djblets and Review Board Work

We now have Djblets working with modern Python 3.x releases and Django 1.6 through 2.1, so we’re in good shape there.

Review Board has a large number of patches in the works to bring Python 3.x and Django 2.1 compatibility. The biggest challenge in Review Board has to do with string type safety. In Python 2.x, Unicode and byte strings are somewhat interchangeable, but there’s a sharp divide in Python 3. We’ve found many cases where we’ve made bad assumptions about string types in Review Board, and are working on both fixing these and adding strict type checks.

World Update Releases

We’re expecting that Review Board 5.0 will require Python 3.5+ and the latest Django 2.x. Administrators will have to make sure they’re running a modern OS with the necessary version of Python, and extension authors will have some updates to perform, but in the end this will be so worth it.

Review Board 4.0 Status

We were initially hoping to release this in August, but that certainly didn’t happen. Too many things came up over this year that took away from the release.

The good news is, DVCS works, and we’re trying to polish up the experience. There’s a few other features we’re also trying to get into 4.0, but once we’re at a point where we feel people can begin safely using DVCS, the beta will start shipping. That will be… soon-ish. We’ll see what happens with the holidays.

RBTools 1.0.1 is out now

July 24, 2018July 24, 2018 Christian Hammond

Today’s release of RBTools fixes some of the most common issues experienced in the recent 1.0 release:

Improved Windows compatibility

This release fixes some regressions on Windows, namely a crash when prompting for a password for Review Board.

If you’re continuing to hit problems on Windows, please let us know in our community support tracker so we can collect additional information on your setup.

Fixes for Empty Diff errors on Git

While RBTools 1.0 greatly improved how diffs were generated for Git repositories under many scenarios, it broke one important workflow.

Posting a branch for review after pushing that branch upstream no longer results in errors about empty diffs when a tracking branch is configured. Instead, the tracking branch is once again respected, allowing your topic branch to be posted for review in full.

See the release notes for the full list of changes.

RBTools 1.0 is here!

June 28, 2018 Christian Hammond

RBTools has been an important part of the life of Review Board users for many years. While it started off as a single tool for posting review requests, its feature set has evolved with time, turning into an extensible set of tools and APIs for talking to Review Board.

Today, we’re finally pulling RBTools out of the 0.x era with the release of RBTools 1.0.

Compatible with Python 3

Both the RBTools commands and the Python API now support Python 2.7 and 3.5+.

(Please let us know if you hit any issues on Python 3, as this is still pretty new.)

Better Repository Detection and Git Support

RBTools now does a better job determining which repository it’s working with, in case there’s confusion. For example, a Mercurial repository nested in a Git-managed home directory will no longer cause problems.

Git repositories in particular are now easier to work with. When generating a diff, RBTools now looks for the nearest upstream parent commit or branch, instead of requiring that users or repositories configure a specific tracking branch.

Publish Automated Reviews

Writing your own automated review solutions for Review Board 3.0 or RBCommons just became easier through the new rbt status-update command. Your scripts can use it to file a pending status update on a review request (showing that checks are being performed) and then update it to say that all is well or to report issues that need to be fixed.

This is useful for in-house continuous integration setups where you’re analyzing code for errors, style issues, documentation, or any other requirements you might have.

Easily Land Complex Dependent Changes

rbt land can now land multiple review requests tied together using the Depends On field.

This works with -r to take the ID of the review request you want to land. It will figure out which review requests must land before it and in which order. For example, if review request 3 depends on 2, which depends on 1, you can run:

$ rbt land --recursive -r 3

Instead of:

$ rbt land -r 1
$ rbt land -r 2
$ rbt land -r 3

This is a precursor to the new DVCS support coming soon in Review Board 4.0.

And That’s Not All

rbt setup-completion was added to enable auto-completion of RBTools commands and arguments in Bash and ZSH shells.
rbt alias was added to help you list and test out your custom aliases.
rbt post –submit-as can now automate posting review request updates, and not just new review requests, on a user’s behalf.
rbt post -m and rbt publish -m let you specify a custom description of your draft’s changes when publishing (equivalent to filling out the “Describe your changes” box when publishing in the browser).
rbt post –trivial-publish and rbt publish –trivial let you publish trivial updates to a review request without sending out e-mails to everyone (when using Review Board 3.0 or RBCommons).
rbt status now lists the review state and local branch for each review request you have up for review.
Warnings and errors in command output is now specially highlighted to help it stand out.
Several fixes and improvements for Git and Subversion compatibility.
The API has been improved, supporting extra_data fields and easier pagination of resources.

And plenty of other fixes and improvements. See the release notes for the full list of changes in 1.0.

Download It Today!

RBTools is out today for Windows, Linux, and Mac. Head on over to the downloads page for installation instructions.

A New Era of Privacy – Beanbag, Review Board, and You

May 22, 2018 Christian Hammond

If you’ve been on the Internet at all in the past few weeks, you’ve likely been bombarded by e-mails from every service you’ve ever used telling you that they’ve updated their privacy policy.

Yes, we’ve updated ours too, but we want to talk to you about how we manage your data, the greater picture of privacy on the Internet, the European Union’s new GDPR legislation, and why this all matters to you, no matter where you are in the world.

GDPR: A new standard for privacy

On May 25th, the GDPR (General Data Protection Regulation) goes into effect in the EU. It gives users there an unprecedented level of control and insight into their personal information. Amongst other things:

People must be able to change, delete, or request copies of their personally identifiable information
Companies need a valid legal basis for the usage of that information (which may require getting consent for that use)
Handing that data off to third-parties also requires a legal basis, and must be documented in a privacy policy
That privacy policy must be able to be read and understood without a law degree

You must admit, that’s pretty nice. Though companies are not required to give these rights to non-EU residents, many (including us) are treating this as a new global standard.

Some parts of the GDPR are a bit vague and not all companies see eye-to-eye on the level of control you should have. We’re hoping our approach goes above and beyond.

Our new privacy guarantees

We’ve always collected as little data as needed. We don’t need much, except to provide services to you, to aid in team communication, and to make use of third-party services we trust who help us run our business and provide support to you.

Still, under the GDPR, there was more for us to do. So here’s what we’re promising:

We’ll continue to only collect what we strictly need, and to document it clearly in our Privacy Policy.
We’ll continue to give you control of your data, and handle deletion and alteration requests, as we always have.
We’ve updated our services to request your consent (and give you full control over it at any time) for any optional usage of your personal information, and any usage we strictly require to run our services effectively will be clearly documented.
We’ve never sold your information and never will.
We’re extending the rights granted by the GDPR to all users of our products, everywhere.
If you ever have any questions or concerns about your data, we’re always here to help.

To help, we’ve built a whole new privacy-focused framework in Djblets to help with privacy guarantees and consent requests. All our software will be using this and we’ll be encouraging Review Board extension authors to use it. We’ll talk about this in more detail in an upcoming post.

What to expect by May 25th

Our Privacy Policy is up now, and will take effect on May 25th, 2018.

We’ll be activating the enhanced privacy support on RBCommons, reviewboard.org, reviews.reviewboard.org, and Splat in time for the 25th. If you’re a user on these, the next time you connect you’ll be asked to accept the Privacy Policy and to allow or block usage of your information for some services.

We’ll also be releasing Review Board 3.0.7 and Djblets 1.0.6, which are privacy-enhanced, optionally allowing for Terms of Service and Privacy Policy URLs and GDPR-compliant consent functionality. Many servers may not need this, but it’ll be available for those that do.

If you want to change, delete, or request any of your personal information from our servers, or want more information on all this, reach out to us at any time and we’ll help. You don’t need to wait for May 25th.