Cloning all repositories from a Bitbucket workspace
I recently found myself in need of cloning all repositories from a Bitbucket workspace. Not a big problem, when it’s 5 or 10 repositories. But upwards of 100 repositories, this is a lot of work if you do it manually. As Bitbucket does not offer this from their API, I wrote a small python package which reads all workspaces and then downloads all repositories from selected workspaces.
Short version
the package can be found here https://github.com/phillies/bucketcloner and is available via pip. Just run: pip install bucketcloner
Long version
The bucketcloner package includes (currently) 2 modes, workspace and clone . Workspace queries all workspaces of a given user and prints them, clone clones all repositories of selected workspaces.
Required credentials
To get the list of workspaces, we need credentials from Bitbucket. Since 2019 Bitbucket no longer supports using your username and password for calling their API, so we need to create an app password.
To find out your username, which is not your email address, go to https://bitbucket.org/account/settings/ and check under Bitbucket profile settings. There you can see and change your username. The username is no longer necessarily the name of your personal workspace.
Next we need to create the app password. This is a password created uniquely for one app, which has granted the minimum set of permissions required for the app to work. App passwords are important for security, because if your app password gets compromised, you can just revoke the app password, create a new one, and update the credentials in one app. If all apps would use the same app password, you would have to update all apps with access to your repositories.
The app passwords are created here: https://bitbucket.org/account/settings/app-passwords/
The necessary permissions for bucketcloner are read permissions for Account, Workspace membership, and Repositories. If you know the name of the workspace and don’t want to list all workspaces within your account, only read on Repositories is requires. Below you see the configuration screen.
The password is shown only once after you click create and cannot be shown a second time. Write it down immediately, or you have to create a new one.
Now we are ready to install bucketcloner and clone some workspaces.
Installation
Bucketcloner requires ar least python 3.8 to run, all other dependencies are installed automatically. To install bucketcloner, just run
pip install bucketcloner
in your python environment. This installes the package along with a convenient executable script bucketcloner .
List workspaces
To list all of your workspaces, call bucketcloner with the workspace command:
bucketcloner -u <your_username> -p <your_app_password> workspace
All your workspaces are printed to the console in the format Name (slug) - URL where Name is the workspace name, e.g. your name including special characters and blanks, slug is the name converted into a url-compatible word without special characters and blanks, and the URL is the workspace URL.
To download all repositories from one or more of the workspaces, we need the slugs in the following step.
Clone workspaces
Cloning all repositories from one or more workspaces is done with the clone command. In the simplest version the clone command downloads all repositories from all workspaces:
bucketcloner -u <your_username> -p <your_app_password> clone
This creates a folder for each workspace with the slug as folder name in the current directory and clones all repository of the workspace into this folder. Depending on the number of repositories and workspaces, this can take some time.
To clone the repositories of one or more selected workspaces, you have to add the -w parameter.
bucketcloner -u <your_username> -p <your_app_password> -w slug1,slug2 clone
The parameters of -w are the slugs of the workspaces, separated by commas without blanks.
If the repositories exist already under the current directory, they will be deleted and cloned again. To skip locally existing repositories, add the --skip-existing flag.
bucketcloner -u <your_username> -p <your_app_password> --skip-existing clone
That’s it. Now you have all the repositories conveniently cloned to your local computer. Have fun :-).
