We’ve released Djblets 3.1, our collection of production-quality add-ons for Django.
This release features:
djblets.registries, djblets.secrets, and djblets.siteconfig modelsFor complete details, see the release notes.
Djblets 2.3 is now out. This release adds compatibility fixes for Python 3.10 and gets our code ready to move to newer dependencies.
Highlights include:
We’re getting 3.x ready for release. This will move the codebase to require Python 3.7+ and Django 3.2 LTS, finally bringing support for modern codebases.
See the release notes for the full list of changes.
Django Evolution 2.1.3 a small but important bug fix release that addresses issues with Python 3.10, mysqlclient, and evolving very old databases.
Python 3.10 removed some deprecated imports from collections, which Django 2.0 and older depend on. When using this combination of versions, Django Evolution will bring back those legacy imports, allowing Django to work again.
mysqlclient also received a similar fix. Django 1.11 and older depended on some unintentional functionality, which mysqlclient maintained until the 2.1 release. We now patch these versions of Django at runtime to continue working.
Two important issues affecting upgrades were addressed:
unique_together index couldn’t be compared properly, causing issues applying some upgrades. The old format for this state is now taken into account during comparison.We’re preparing Django Evolution 2.2, which will support all versions of Django up through 4.0. This should be coming soon.
In the meantime, check out the release notes for the full changes, including other fixes in this release.
kgb 7.0 has been released with a handful of new features:
It also drops official support for Python 2.6 and 3.4-3.5 (though these should still work for now, if you need them).
kgb now provides a spy_agency fixture for pytest unit tests. This will set up a SpyAgency for you, letting you spy on functions and assert calls. For example:
def test_doomsday_device(spy_agency):
device = DoomsdayDevice()
spy_agency.spy_on(device.destroy_reality, call_original=False)
device.trigger_the_event()
spy_agency.assert_spy_called(device.destroy_reality)The SpyAgency will be managed for the lifetime of the test, removing spies upon completion.
We’ve provided snake_case versions of all the assertion methods, making for more natural tests. For instance, you can use assert_spy_called_with() instead of assertSpyCalledWith(). The old camelCase versions will continue to exist, though.
You can also call assertion methods without needing to mix in a SpyAgency into your test. Just import from kgb.asserts. This is useful if you have a need to spy on methods and assert them within non-unit test code, or without access to a SpyAgency. For example:
from kgb.asserts import assert_spy_called_with
def check_connection_state(api_connection):
assert_spy_called_with(api_connection.make_request,
url='https://middleman.zyx',
method='POST')Before, if you were trying to spy on a method (particularly unbound methods) wrapped in a decorator, and that decorator didn’t preserve the function’s original name, you’d hit an error looking up the method.
You can now work around this by passing the original function name when setting up a spy. For example:
def bad_decorator(func):
def _wrap(*args, **kwargs):
return func(*args, **kwargs)
return _wrap
class BadClass:
@bad_decorator
def good_func(self):
return 42
def test_good_func(spy_agency):
spy_agency.spy_on(BadClass.good_func,
owner=BadClass,
func_name='good_func')kgb will try to detect when you need this and emit a warning during the test.
See the release notes for the full list of changes.
You can install kgb today through pip:
$ pip install -U kgbVisit our kgb documentation to see how to use kgb in your projects.
kgb 6.1 has been released, featuring some new spy operations and enhancements to existing ones.
kgb is our Python unit test library for crafting function spies, making it easy to check on the calls made during a test or to override their behavior, in ways not possible with Python’s mock.
Two new spy operations, kgb.SpyOpReturnInOrder and kgb.SpyOpRaiseInOrder, have been added. These work like a combination of kgb.SpyOpReturn/kgb.SpyOpRaise and kgb.SpyOpMatchInOrder. Each takes a list of results, and each call will receive the next result in the list, making it really easy to build tests that, somewhere, involve repeated calls to some function.
For example:
spy_on(our_agent.get_identity, op=kgb.SpyOpReturnInOrder([
'nobody...',
'who?',
'not telling...',
]))
spy_on(pen.emit_poison, op=kgb.SpyOpRaiseInOrder([
PoisonEmptyError(),
Kaboom(),
MissingPenError(),
]))
The kgb.SpyOpMatchInOrder and kgb.SpyOpMatchAny operations can now nest other spy operations, which simplifies returning or raising values, and can even allow for more advanced interactions with a spy.
The call definitions passed now accept an 'op' key pointing to a spy operation instance, which will be invoked if a call matches.
For example:
spy_on(lockbox.enter_code, op=kgb.SpyOpMatchInOrder([
{
'args': (42, 42, 42, 42, 42, 42),
'op': kgb.SpyOpRaise(Kaboom()),
'call_original': True,
},
]))
See the release notes for the full list of changes.
You can install kgb today through pip:
$ pip install -U kgb
Visit our kgb documentation to see how to use kgb in your projects.
Today’s release of RBTools 1.0.3 is a big one, featuring enhancements for Mercurial support, a vastly improved commit editing experience when landing changes, and several compatibility fixes for Python 3 and various types of repositories.
rbt land now supports landing commits on Mercurial repositories.
You can land a local change from a Mercurial branch or bookmark, or a remote change from a review request. This will first verify that the change has been approved on Review Board before allowing it to land. Once approved, a new merge commit containing the information and URL of the review request will be placed on your destination branch.
This can also close the branch/bookmark being merged in on your behalf. See the documentation for details.
Patching a commit with rbt patch -c, or landing a commit with rbt land -e has always let you edit the message for the commit, but the experience was sub-par.
Now RBTools will mimic Git or Mercurial’s standard editing environment, helping your editor show the syntax highlighting or line length limits it would normally show.
Deleting all text in the editor and saving will cancel the patch/land operation.
You can also set a custom editor when working with RBTools by setting the new $RBTOOLS_EDITOR environment variable.
We’ve fixed a number of Python 3 compatibility issues. These largely centered around:
--help as the last argumentThere’s also a fix for looking up available Git remotes for a branch when a tracking branch isn’t set. Thanks to Joshua Olson for this fix!
See the release notes for the full list of changes.
These are exciting times, right? While everyone is preparing for the coronavirus, we’ve been preparing for Review Board 4.0 beta 1.
And the coronavirus.
The shelves are getting pretty empty in my local grocery store, but I’ve kept my sanity.
We have an internal target date for 4.0 beta 1, which is coming up very soon, but we’re not prepared to release that just yet. Still a couple factors that might need to delay a week.
This beta will feature:
We’re working on the final bits of the administration UI, and expect to be done by end of next week. Everything is in place, with the exception of the Integrations, Extensions, and Security Center pages. But that’s it!
We’ve also been finishing up the work to make our third-party integrations (Travis-CI, Asana, Jenkins, etc.), Review Bot, and Power Pack all compatible with Review Board 4.0, Django 1.11, and Python 3. These are the last big blockers before release.
After all that is done, beta 1 will be a go!
(Assuming none of us get the coronavirus.)
If you are building custom extensions, please get ready to test this beta. Let us know if you plan to test so we can coordinate with you and give you a heads up on what may change.
RBTools 2.0 features full support for multi-commit review requests. We’ve had the bulk of this work done for a while, but have been really thoroughly testing it in preparation for a beta, which will go out alongside Review Board 4.0 beta 1.
This week, we’ve finished up the work to apply patches from a multi-commit review request (giving the ability to apply all commits sequentially or squash into a single commit), and to fully land changes.
We’re wrapping this up soon. If you test Review Board 4.0 beta 1, you’ll need RBTools 2.0 beta 1.
What else, what else..
We’re continuing to work toward improving our accessibility story in Review Board. While we’re nowhere close to where we want to be yet, we’ve amended our policies around new UI components to ensure that keyboard usage and ARIA roles are a first-class citizen in any designs.
If you make use of any assistive technologies, such as screen readers, we’d love to talk to you and get your feedback on a few things!
Review Board 4.0 will be making it easier to glance over file attachments for video files. Just like file attachment thumbnails for images and text files give you a preview of the contents, they’ll soon give you a preview of the video as well.
Hovering the mouse over these thumbnails will cause the video to play (muted) until you move the mouse away. Hovering over them again causes the video to resume from where it left off.
This is great if someone’s using video files to demonstrate their feature. We’re hoping to make this the start of some useful UI around videos.
This past week saw the development of several new accessible CSS/JavaScript components, which can be used by extensions:
We’ve historically had a few different ways to show buttons. Any <input type="button" />, <input type="submit" />, or anything with the .btn CSS class would have a consistent appearance, and could be modified by a set of additional CSS classes.
This has been revised to include <button> (which we oddly did not have before) and anything using the .rb-c-button CSS class (the successor to .btn, which is now deprecated).
A button group (.rb-c-button-group in LessCSS) is a collection of buttons, packed together either horizontally or vertically, providing an almost toolbar appearance.
The new RB.MenuView (JavaScript) and .rb-c-menu (LessCSS) component manages a pop-up menu that can be shown or hidden as needed. It supports full keyboard navigation and ARIA attributes for accessibility.
We’re working on moving to this in every place that involves a pop-up menu, including review request actions (like “Update” and “Close”) and the Account, Support, and Follow menus in the upper-right of pages.
RB.MenuButtonView (JavaScript) and .rb-c-menu-button (LessCSS) implements a type of button that displays a pop-up menu when clicked.
There are two main ways this can be used:
The later is great when you want to offer a common action, but make alternative actions easily available.
Like the new pop-up menus, these provide full keyboard navigation and ARIA attributes for accessibility, and are a great building block for extensions to use.
We hope to provide full documentation at a later time on the standard library of UI components we’re building.
KGB is our Python module for unit tests, which provides the ability to spy on any function and track the calls and results, or override the behavior. It’s incredibly powerful, and we make heavy of use of it in Review Board and all our other projects.
We’ve just fixed up KGB to work with Python 3.8, so we’re preparing for a release pretty soon.
If you’re a Python developer, check out KGB. Tell your friends. Give us some feature requests. Honestly, it’s pretty complete these days, but we need more reasons to bump the version number.
Alright, that’s enough for this ChangeLog.
We may skip next week’s ChangeLog to focus on finishing beta 1, unless we have anything really interesting to report.
If you want to know more, have any questions, or are curious about anything else, please reach out on our community forum.
We’re also on Reddit (/r/reviewboard), Twitter, Facebook, and YouTube if you want other ways to keep up with Review Board and Beanbag.
This week, we’re going to dive into three major topics:
Unified Diffs, the most common type of diff files, are pretty lacking in standards. Nearly every type of source code management solution outputs its own version of a unified diff, shoving metadata into it however the developers saw fit. This frankly makes it a pain to work with diffs in Review Board, and I’m sure anyone else building similar tools can say the same thing.
A few years ago, we jotted down some ideas of what we’d like to see in diffs. We wanted:
We feel this is important. You wouldn’t believe how wildly different two diff files can be, and how broken most are (just having spaces in a filename means you’re going to run into all sorts of crazy problems).
So we put these ideas together and called it DiffX. Our plan was to build a specification, a standard Python library (and later libraries for other languages), and RBTools/Review Board support.
This week we put up the initial draft of the DiffX specification and documentation. Take a look. See what horrors tools like ours go through, and what we’d like to do about it.
Do you work on another code review tool or source code management system? Let’s talk!
Note that this is still a work-in-progress, and we have a lot we need to better formalize the parsing and the metadata format (and make a public, useful reference implementation).
During Review Board 4.0 development, we kept running into clashes between CSS rules, and had to deeply nest CSS classes and elements in order to ensure our styles would apply. Yuck. This was not sustainable.
We sought to fix this for our code, and looked into what others have done. We considered BEM, BEMIT, and a few other approaches. None felt quite right for our needs (particularly since we want to avoid clashes between third-party extensions).
So we took what we liked in each and drafted our CSS Component Style Guide.
This gives us a better way of thinking about what goes into our stylesheets and HTML pages. We think in terms of formal, reusable components, instead of just haphazardly tacking on styles for whatever HTML we’re writing.
We group together LessCSS variables (used for colors, sizes, etc.) and macros (for customizing a component’s presentation in a safe way) under organized namespaces. Stylesheets (ours and extensions’) that are looking to fit in with our styles or specialize a component can rely on these to better fit in.
We’re also taking care to add formal documentation for everything we add, which forces us to think about our styles, reason about them, and support them.
Want to see this in action? Here’s a few examples:
We’re still working on moving over to this, writing all new CSS using this standard and migrating old styles as needed. This is a long-term project, but we thought it might be interesting to some of you.
If you’re developing extensions for Review Board, we encourage adopting these new styling conventions. It’ll make your life so much easier.
Last week, we talked about the work on Python 3 and Django 1.11, and some of the huge tasks we’ve had to work through to support this.
One of the large — and final — hurdles is our administration UI. Django helpfully provides a built-in administration UI that projects get out of the box, and it’s very useful. And we extend and customize it heavily.
The problem is that Django’s administration UI has changed a lot since 1.6, and frankly it wrecked ours. So we’re rebuilding it, better than it was before.
Here’s a breakdown on the new UI:
It’s also built with our new CSS component standard, and introduces new components that we can use throughout the product. Extensions can take advantage of these, too, helping them look like a native part of Review Board without worrying too much about our styling.
Here’s a couple in-progress screenshots.
We’ll go over some of the new diff functionality being worked on in Review Board 4.0, and what we’ve been up to in RBCommons land.
Again, if you have any questions, or anything you’re curious about and want us to cover, please reach out on our community forum.
Hi, Christian Hammond here. Welcome back to ChangeLog, where we cover the latest going on in Review Board, RBCommons, and Beanbag in general. It’s been a while since we’ve posted one of these, but we’re aiming to bring this back as a weekly series.
There’s a lot to talk about.
We’ve had four main focuses this year:
This along with a multitude of smaller tasks taking our time throughout the year (and, on a personal note, dealing with the aftermath of the Paradise, CA “Camp Fire.”)
Most of you are here to find out about Review Board, so let’s dive into that.
Our original goal for Review Board 4.0 was to bring support for multi-commit review requests and to ship that. This was a large project, as Review Board was originally written before DVCS was common (Subversion and Perforce were the primarily open source and enterprise solutions).
The problem though, is Python 2 End Of Life is coming up fast, and if we kept on schedule with Review Board 4.0, it would be a long time until we’d support Python 3. So we decided to delay 4.0 in order to get it ready for Python 3.
Note that we also still need to support Python 2.7, since many companies are still heavily tied to 2.7 (older distros, custom extensions and scripts).
So what’s hard about supporting Python 3? Welllll….
Python 3 is leaps and bounds better than Python 2 in most ways, but porting a large and complex product from Python 2 to 3 is even harder than you think.
Most Python users know of the major differences:
We thought we prepared years ago to make this move easy. We used Unicode strings in every file. We used the six module to help with using the right modules, types, and functions.
In the end, it was harder than we expected. Our biggest challenge was definitely the Byte Strings vs. Unicode Strings differences. We thought we were in good shape for this, but we weren’t close.
Review Board does a lot of text processing. We’re parsing uploaded diffs, pulling source files from repositories, matching those up and applying the patch to the source, and generating side-by-side diffs. Much of this logic is over 10 years old, even so, we put a substantial amount of work in preparing for Python 3 years ago, so we were shocked by how much we got this wrong.
The problem comes down to the differences in how Python 2 and 3 would handle shoving two different string types together, which is very easy to do accidentally. Python 2 would roll with it, as in many cases the string types were compatible (and would automatically encode/decode so long as the content was basically ASCII). In Python 3, they outright break — which is a better approach, but hard to transition to — and it led us down a rabbit hole of problems.
Missing b'...' prefixes, changes in string return types from Python functions, functions that are no longer compatible with both string types, and very difficult decisions to make regarding compatibility with third-party SCMs.
All the little regressions and inconsistencies added up. Here’s a few more examples:
pickle library was a mess. Defaults have changed and pickled data became incompatible (due to module reference and string type changes). We had to build in compatibility layers here and test them thoroughly.We’ve spent a lot of time tracking down issues that might immediately crash or might affect data several stages down, and sometimes be traced to something outside our codebase.
Fixing some Python issues means upgrading third-party modules, which can introduce their own new set of changes, regressions, and new rounds of work.
And the biggest source of that was Django.
Django 1.6 (the version we’ve been using) breaks on Python 3.6+. This meant we had to upgrade to a newer version, something we’ve been putting off.
Django 1.7 introduced built-in support for database migrations. Quite nice for many projects, but the design was sub-optimal for applications like ours (upgrades could take hours or days longer than our approach) and was fundamentally incompatible with our own Django Evolution.
Many other core components of Django (the foundation for the administration UI, template rendering, URL management, forms, and all sorts of other common and obscure parts) have also largely changed over time in some pretty important ways.
We needed a solution for all of this, and we knew that would take time. To compensate we had to:
We’re not done, but we’re close.
Review Board is built to allow third-parties to extend its functionality, modify behavior, and link up with other in-house services. We try to be careful not to break extension functionality, and over time we’ve gotten more strict about providing compatibility and an upgrade path for functionality we want to deprecate or change.
The Python 3 and Django 1.11 upgrade is going to affect just about everybody who’s writing an extension. Some of the work we’ve done on adding compatibility layers will help with this process, but it’s going to mean a longer upgrade cycle for some companies.
We knew from the beginning that we’re going to have our hands full helping these companies out (as part of our support contracts), and that it’d be in our best interest to delay the release and break everything all at once instead of spreading out the breakages across multiple releases, and to also give ourselves time to figure out how best to minimize those breakages.
Most projects moving to Python 3 or to newer versions of Django don’t have to worry about this domino effect in the same way.
I hesitate to say when beta 1 will be done (been wrong before), but I can say that the last major piece of development is wrapping up. We’ll be kicking the tires on it, and want to get a beta out as soon as it’s stable enough.
We may not enable Python 3 builds for beta 1, focusing instead on Django 1.11 testing (Python 3 support is still going to be in development during this time), however we’re working with select customers on real-world testing against Python 3.
Every week, we’d like to address some questions, concerns, suggestions, etc. from the community. If you have any questions for next week, please reach out to us.
Q. The community forum seems quiet at times. Why is that?
A. Support requests from companies with support contracts are conducted over a separate support tracker. We always prioritize these support requests over any other work, and increasingly more companies are moving to this method of support for Review Board.
Q. Will Review Board 4.0 ship with Python 3 before the Python 2 End Of Life date?
A. No, it’s going to miss that date. Python 2.7 is still going to be required for now.
Q. There’s years of open review requests on reviews.reviewboard.org. Why is that?
A. We work with university students every semester to help prepare them for their jobs in the tech industry. They spend their semester building features for Review Board, most of which are prototypes. These make up the majority of the review requests on there. Others are contributions that might be outdated, might have been missed, might be incomplete, or might just not have been reviewed yet.
We’ll be going over our new CSS component standard for the project and dive deeper into the new administration UI.
Again, if you have any questions, or anything you’re curious about and want us to cover, please reach out on our community forum.