ChangeLog: November 5, 2018

The World Update: Python 3 and Django 2

In a previous post, we talked the work we’re doing to move to modern versions of Python and Django. We’re calling this the “World Update,” and it’s quite a large and difficult project.

Database Migrations and Django Evolution

Much of the difficulty comes from an incompatibility between how we handle database migrations (using Django Evolution) and how Django handles them (using their built-in migrations support). These aren’t at all compatible, and for many reasons (legacy support and performance), we don’t want to ditch Django Evolution. This has by far been the biggest hurdle to overcome for the World Update.

We’ve been working on this, reworking nearly all of Django Evolution to better utilize migration support in Django, to provide a new set of generic structures for describing database state and modifications, and to (soon) hook into and partly override the migration process to allow both to co-exist.

Progress on this had been steady, and we feel we’re reaching a point where we’ll be able to fully support migrations in Django Evolution.

Djblets and Review Board Work

We now have Djblets working with modern Python 3.x releases and Django 1.6 through 2.1, so we’re in good shape there.

Review Board has a large number of patches in the works to bring Python 3.x and Django 2.1 compatibility. The biggest challenge in Review Board has to do with string type safety. In Python 2.x, Unicode and byte strings are somewhat interchangeable, but there’s a sharp divide in Python 3. We’ve found many cases where we’ve made bad assumptions about string types in Review Board, and are working on both fixing these and adding strict type checks.

World Update Releases

We’re expecting that Review Board 5.0 will require Python 3.5+ and the latest Django 2.x. Administrators will have to make sure they’re running a modern OS with the necessary version of Python, and extension authors will have some updates to perform, but in the end this will be so worth it.

Review Board 4.0 Status

We were initially hoping to release this in August, but that certainly didn’t happen. Too many things came up over this year that took away from the release.

The good news is, DVCS works, and we’re trying to polish up the experience. There’s a few other features we’re also trying to get into 4.0, but once we’re at a point where we feel people can begin safely using DVCS, the beta will start shipping. That will be… soon-ish. We’ll see what happens with the holidays.

Read More

RBTools 1.0 is here!

RBTools has been an important part of the life of Review Board users for many years. While it started off as a single tool for posting review requests, its feature set has evolved with time, turning into an extensible set of tools and APIs for talking to Review Board.

Today, we’re finally pulling RBTools out of the 0.x era with the release of RBTools 1.0.

Compatible with Python 3

Both the RBTools commands and the Python API now support Python 2.7 and 3.5+.

(Please let us know if you hit any issues on Python 3, as this is still pretty new.)

Better Repository Detection and Git Support

RBTools now does a better job determining which repository it’s working with, in case there’s confusion. For example, a Mercurial repository nested in a Git-managed home directory will no longer cause problems.

Git repositories in particular are now easier to work with. When generating a diff, RBTools now looks for the nearest upstream parent commit or branch, instead of requiring that users or repositories configure a specific tracking branch.

Publish Automated Reviews

Writing your own automated review solutions for Review Board 3.0 or RBCommons just became easier through the new rbt status-update command. Your scripts can use it to file a pending status update on a review request (showing that checks are being performed) and then update it to say that all is well or to report issues that need to be fixed.

This is useful for in-house continuous integration setups where you’re analyzing code for errors, style issues, documentation, or any other requirements you might have.

Easily Land Complex Dependent Changes

rbt land can now land multiple review requests tied together using the Depends On field.

This works with -r to take the ID of the review request you want to land. It will figure out which review requests must land before it and in which order. For example, if review request 3 depends on 2, which depends on 1, you can run:

$ rbt land --recursive -r 3

Instead of:

$ rbt land -r 1
$ rbt land -r 2
$ rbt land -r 3

This is a precursor to the new DVCS support coming soon in Review Board 4.0.

And That’s Not All

  • rbt setup-completion was added to enable auto-completion of RBTools commands and arguments in Bash and ZSH shells.

  • rbt alias was added to help you list and test out your custom aliases.

  • rbt post –submit-as can now automate posting review request updates, and not just new review requests, on a user’s behalf.

  • rbt post -m and rbt publish -m let you specify a custom description of your draft’s changes when publishing (equivalent to filling out the “Describe your changes” box when publishing in the browser).

  • rbt post –trivial-publish and rbt publish –trivial let you publish trivial updates to a review request without sending out e-mails to everyone (when using Review Board 3.0 or RBCommons).

  • rbt status now lists the review state and local branch for each review request you have up for review.

  • Warnings and errors in command output is now specially highlighted to help it stand out.

  • Several fixes and improvements for Git and Subversion compatibility.

  • The API has been improved, supporting extra_data fields and easier pagination of resources.

And plenty of other fixes and improvements. See the release notes for the full list of changes in 1.0.

Download It Today!

RBTools is out today for Windows, Linux, and Mac. Head on over to the downloads page for installation instructions.

Read More

ChangeLog: New Integrations, Releases, and Prep for RBCommons 3.0

We’ve had a really busy couple of weeks since the last ChangeLog. There were two Review Board releases, a small setback with RB-Gateway, and lots of testing and infrastructure work for RBCommons 3.0.

Review Board 3.0.4 and 3.0.5

Last week, we put out Review Board 3.0.4, a feature-packed release introducing:

It was a pretty great release, fulfilling a lot of feature requests we’ve had for a while an providing the foundation for some new work we’re doing. Unfortunately, there was a last-minute error that, in production, broke part of the form for repository configuration.

Really embarrassing.

Now, we’ve found most people don’t upgrade the same day that a release goes out (downtime must be scheduled, people are busy, etc.) so we mostly started hearing about it two days later. As soon as we realized the mistake, we quickly got a new release out, Review Board 3.0.5, and put some changes in place to help prevent this sort of last-minute problem from happening again.

The good news is that, in the meantime, we went through and fixed a bunch of bugs that didn’t make the 3.0.4 release, but were ready for 3.0.5. So really, we’re just hoping we can all pretend 3.0.4 was just a pre-release for 3.0.5 now ūüôā

Review Board 3.0.6 is currently scheduled for April 10th. I’m expecting it to go smoothly.

RB-Gateway Difficulties and Delays

RB-Gateway, our API wrapper around Git and Mercurial repositories, was supposed to release, well, today. Sadly, that’s not happening.

Let me back up. RB-Gateway is written in Go, unlike most of our projects which are Python-based. Go was chosen partly due to concurrency benefits for handling and serving up requests, and partly for its ease of cross-compilation and distribution (just drop it into a directory and run it on any supported platform).

It’s the cross-compilation that posed a problem. We use git2go, a Go wrapper around libgit2, a C library for talking to Git repositories. We don’t need a lot from it, but it made sense to “go” with that (sorry).

Problem is, including a C library makes cross-compilation much harder, and there’s threads full of discussions on issues with compiling and utilizing git2go in production, depending on how it’s compiled and used. So we’re planning to remove git2go usage.

Instead, we’re evaluating other Git libraries. We probably won’t roll our own, but as we don’t really need much from a Go library, we’ll “go” that route if we need to (sorry).

When that’s done, we should be ready to release.

Prep for RBCommons 3.0

This Friday, we’re beginning an upgrade of RBCommons, bringing many of the features of Review Board 3.0 to the service. We’ve spent much of this week getting this ready — rebuilding servers, testing database migrations, running through checklists of manual feature tests, etc.

There’s going to be a lot to love in this release, but those following Review Board development will surely notice that some features (such as Slack, Asana, etc. integrations) will not be there on launch. We have just a bit more work to do before those are ready. We want those as much as anybody, so they’re high up on the priority list.

The blocker right now is that the administration pages for some of these features are built to plug into the Django administration page, not the custom RBCommons team administration page. So there’s still some work to do before that’s complete. Soon, though!

The upgrade should be smooth, and we should be back up in only a few hours, but just in case, we’re leaving the maintenance window open through Sunday. We aimed for a holiday weekend (well, holiday for a lot of people, anyway) to reduce the impact on users.

Read More

ChangeLog: Catching Up

It’s been too long since we last ran the ChangeLog series, and felt it was the right time to start it back up again. ChangeLog is a look into the latest behind-the-scenes work going into Review Board, RBCommons, and other Beanbag projects. While intended to be a weekly series, we’d like to start off with some of the bigger tasks and feature development from the past month.

Moving to Django 1.11 and Python 3

Today, all current versions of Review Board depend on Django 1.6, an old release that’s no longer supported by the Django project but is by us, and doesn’t support modern Python 3 releases.

We’ve been stuck on 1.6 because 1.7 introduced (and later mandated) a new way of handling database migrations, which is incompatible with the method we’ve always used. Reconciling the differences has been a challenge.

In the past month, we’ve made significant progress toward both the Django and Python updates:

  • Djblets 2.0 (our development release) is now compatible with Django 1.6 through 1.11 and Python 2.7, 3.4, 3.5, and 3.6.
  • Django Evolution (used for database migrations) now works with Django 1.6 through 1.11 and Python 2.7, 3.4, 3.5, and 3.6. Work’s being done to let it co-exist with Django migrations now.
  • Review Board has started receiving patches for Django 1.11 and Python 3.5+ now. This is still in development, and likely won’t make the Review Board 4.0 release, but will be there for 5.0.
  • RBTools 1.0 (shipping in a few months) now has full Python 3 support.

New Release Schedules

‚Ä®We’ve began moving to a train model for releases, and have all of our main and upcoming products now on the calendar.

Here’s what this currently looks like:‚Ä®

  • Review Board 4.0 (with DVCS support!) is expected to ship in August, 2018
  • Review Board 3.0.x releases will ship (generally) every other Tuesday
  • RBTools 1.0 is expected to ship April 12th
  • RB-Gateway is expected to ship March 28th

We’re planning to release a new major Review Board release every ~6 months, meaning smaller but more frequent releases. We’re still experimenting with the schedule and timeframe for these releases.

RB-Gateway

We’ve releasing RB-Gateway 1.0 this month. This is a microservice designed to sit in front of a Git or Mercurial repository, providing an API and set of integrations that can be used by Review Board or any other tool or service for more deeply working with your repository.

RB-Gateway doesn’t change your workflow, and can be dropped in with minimal effort. It completely replaces the cgit/gitweb workaround for standalone Git repositories, and means you don’t need to set up something more complicated like GitLab just to work with Review Board.

You’ll see more information on RB-Gateway’s capabilities when we release later this month, and we’ll cover improvements being made to it here.

Wrapping Up…

Those are really just the major highlights, to get everyone up to speed. It doesn’t include the new features we’ve recently built, like being able to filter files in the diff viewer based on filename patterns, a new command for creating Review Board extension source trees, the work done on kgb, or the crazy investigation into deadlocks that’s delayed Review Board 3.0.4.

Going forward, these will be smaller, covering only what’s been done over the past week. If you like these posts, and want to see this continue, please let us know! You can find us on reddit or on the community support list.

Read More

Announcing virtualenv-multiver for Python Development

If you’ve worked with virtualenvs for Python before for development/testing, then you may have hit cases where you really wanted multiple versions of Python installed in your virtualenv. Which, you may actually have working, because virtualenv, in theory, supports this. In fact, you’re supposed to be able to do:

$ virtualenv -p python2.6 my-env
$ virtualenv -p python2.7 my-env

That’d be great, if it always worked. It doesn’t. When your virtualenv gets built,¬†bin/python¬†may end up being a link to¬†bin/python2.7¬†(or what have you), or it may be the contents instead of a link. Subsequent installs may end up overwriting binaries, producing a¬†python2.6¬†and¬†python2.7that are both Python 2.7.

Oh and it gets worse. On Mac, with a standard Python install, these binaries actually end up invoking ../.Python, a symlink pointing to the system Python. This link is not versioned. So much for multiple Python versions in one virtualenv on the Mac.

A solution!

We fixed this. Now you can run a single command to get a working environment going, without messing with things or running into problems on the Mac. This is virtualenv-multiver.

Now, setting up an environment is as simple as:

$ pip install virtualenv-multiver
$ virtualenv-multiver my-env 2.6 2.7

Couldn’t be easier. This works both for new environments and existing ones.

This is a beta, so there may be some issues here or there. If this is useful to you, give it a try and let us know!

Read More

Work toward a Django 1.8+ port for Review Board

We’ve been dependent on Django 1.6 for our components, for many reasons. There are a lot of things Review Board has to deal with that most Django apps do not, so Django often regresses us, unintentionally. There are also just design changes in Django that don’t suit shipping products, and we’re often having to work around these changes.

However, the biggest bit is our database migration story. Our codebase depends on Django Evolution for migrations, which has to dive pretty far into the internals of Django for some operations. This is a large bit of work to port.

I’m happy to say that, after spending the day on it, I got surprisingly far toward having Django Evolution working on Django 1.8! It’s not perfect, and there’s not yet a good story for playing nice with Django migrations, but it’s a huge start. It opens the doors to getting a full compatibility story going.

The next question is, are we keeping Django Evolution, or moving to migrations fully? Well, that’s a bigger problem, because we have no control over which version of Review Board (and therefore Django) people are upgrading from, and have to be very careful with how we handle any database migrations.

There are also issues with Django’s migrations just being a lot slower than Django Evolution, to the intentional lack of an optimization step when applying the migrations. This means stupidly-long upgrades for large installs, which won’t work for us. So, we’ll probably stay with Django Evolution until we figure out a decent solution there…

Read More

Plans for Review Board’s bug tracker

Many of you may have heard that Google Code is going read-only starting tomorrow, and some have asked us how this will affect the project, since we host our bug tracker there.

Not to worry. Google’s been nice enough to whitelist us for a little while, so even though most of Google Code will be down, we’ll continue to be up. This is not permanent, but for the time-being, you’ll still be able to report bugs at the old address.

Going forward, we’ll be migrating off of Google Code and onto a new tracker. That will happen in the coming weeks, and we’ll talk more about it when it happens.

So why the delay? Why did Google need to extend the shutdown date for us? We actually have something new on the way that we’re pretty excited about. We call it Splat, and while still very young, it’s shaping up to a pretty cool bug/issue tracker. We weren’t quite prepared to switch over to it by the shutdown date, but we have enough of it ready to launch pretty soon.

There’s a lot more that I’d like to say about Splat, but there will be time for that. We’ll make a more formal announcement soon.

Read More

Lots of UI cleanups in preparation for Review Board 2.5 RC 1

2.5 beta 2 looks to be working pretty well, and we’re working hard to get ready for RC 1. As part of this, we’ve fixed up a number of little UI issues here and there. For instance, login on mobile now works:

 

Mobile Login (Review Board 2.5)

 

As does registration and password resets.

Gravatars are now showing up more reliably in the dashboard. Depending on the settings on the server, these may have been hidden unintentionally. Basically, defaults weren’t being taken into consideration in some calls.

The user page now works properly on mobile, with filters moving to a little menu:

 

Mobile Dashboard (Review Board 2.5)

Mobile Dashboard with Filters (Review Board 2.5)

 

Also, some fixes for visual issues with text and Markdown file attachment review pages.

Read More

On-the-fly syntax highlighting when using Markdown

Review Board 2.0 introduced Markdown support for text fields, and we’ve been iterating on this since. One nice advantage to using Markdown is that it’s really easy to syntax-highlight a code fragment, like:

```python
def foo():
print "oh hi there"
```

When saving the comment, this would appear rendered with some syntax highlighting, same as the diff viewer.

In 2.5, we’re adding on-the-fly syntax highlighting for most popular languages: CoffeeScript, CSS, Go, HTML, JavaScript, Perl, PHP, Python, ReStructuredText, Ruby, Shell Scripts, SQL, XML, and YAML.

That means when you type code in a code block, like above, it will show the syntax highlighting immediately, without having to save.

Now, it’s not perfect. We use a different highlighting engine for rendered content vs. on-the-fly content, and they don’t 100% agree on how things should be styled, but it’s close enough.

We’re gaining this ability through an upgrade of CodeMirror, the widget we use for the text fields. We’re giving 5.5 a try (we previously tried upgrading to 4.2 and had issues, but so far so good with 5.5).

This will all ship with Review Board 2.5 RC 1.

Read More

A new polished issue summary table for review requests

Since Review Board 1.6, we’ve had a table of all open issues (comments that have a task that needs fixing before a change can go in), sitting right below the fields of a review request. This could be filtered by reviewer and by status type, showed summaries of the comments, the date/time the comment was filed or last updated, and the status type.

It was pretty text-heavy, though, and not easy to read at a glance. As such, most people probably ignored the nice filtering abilities and, really, most of the content.

So, I redesigned it.

Here’s how it looked before:

Old issue summary table

And here’s how it’ll look in Review Board 2.5 beta 2:

New issue summary table (Review Board 2.5)

New issue summary table (Review Board 2.5)

It’s even more mobile-friendly:

Mobile issue summary table (Review Board 2.5)

Users will get to use the new table later this week.

Read More