I needed an efficient method to migrate from one GitLab environment to a completely new one, without using the database-ish export/backup method that is provided. I had two projects that had gotten corrupted, and at my current skill level, I didn’t know how to clear/delete them out (if you want to try your hand at it and think it should be easy, please feel free to reach out). Seriously. I tried about everything from the UI.

So I made this, a two part Python3 script that will copy the important bits (users and their projects, as the users even) but nothing else. If there are build routines/pipelines/sub-module stuff, this will likely temporarily break that. Generally, you can go back in and fix it, but, full transparency. I’d like to make this better, again, please feel free to comment.

Further transparency, do not use this on a corporate/integrated production system, you likely have needs that this script will not meet. For a project/hobbyist such as myself, this got me out of an issue with a re-imported corruption, the 2 projects that were preventing me from upgrading to 14.x since I couldn’t convert them over to hashed storage.

Notes on setting up to use these scripts

1. You have administrator on two GitLab instances
2. Logically one ‘old’ one, and a new one your replacing the old one with
3. Not sure about version, there seems to be a little bit of lining up necessary it seems
4. Use the same version in the new as the old
5. In my old migration area I show how you can specify the version you install
6. You’ve setup your admin/root user on both and created each token with full API/sudo privileges
7. With all that, you should be good to go, written for Python 3

User migration

#!/bin/python
import gitlab
import configparser
import sys

# Load auth
glconfig = configparser.ConfigParser()
glconfig.read('migration.txt')
gl1server = glconfig.get('gl1','server')
gl1token = glconfig.get('gl1','token')
gl2server = glconfig.get('gl2','server')
gl2token = glconfig.get('gl2','token')

# Connect to API endpoints
gl1 = gitlab.Gitlab(gl1server, private_token=gl1token)
gl1.auth()
gl2 = gitlab.Gitlab(gl2server, private_token=gl2token)
gl2.auth()

# Migrate User details
users1 = gl1.users.list()
namesInNew = []
users2 = gl2.users.list()
for user in users2:
  print(user.id, user.name)
  namesInNew.append(user.name)
for user in users1:
  if user.name in namesInNew:
    continue
  elif 'GitLab' in user.name:
    continue
  else:
    print('making', user.name)
    gl2.users.create({'email': user.email,
                      'password': 'somepassword',
                      'username': user.username,
                      'name': user.name})

Projects migration by User

#!/bin/python
import gitlab
import configparser
import re
import time
import sys

# Load auth
glconfig = configparser.ConfigParser()
glconfig.read('migration.txt')
gl1server = glconfig.get('gl1','server')
gl1token = glconfig.get('gl1','token')
gl2server = glconfig.get('gl2','server')
gl2token = glconfig.get('gl2','token')

# Connect to API endpoints
gl1 = gitlab.Gitlab(gl1server, private_token=gl1token)
gl1.auth()
gl2 = gitlab.Gitlab(gl2server, private_token=gl2token)
gl2.auth()

# Project details
currentPage = 1
moreToGo = True
print('Starting...')
while moreToGo:
  projects = gl1.projects.list(order_by='id',per_page=10,page=currentPage)
  for project in projects:
    print('id: ', project.id)
    print('desc: ', project.description)
    print('name: ', project.name)
    print('path: ', project.namespace['path'])
    cleanedName = re.sub(r"[^\w\d]+", "", project.name)
    # Find maker of project
    projectUser = gl2.users.list(username=project.namespace['path'])[0]
    print('Found user: ', projectUser.name)
    # Impersonate user
    i_t = projectUser.impersonationtokens.create({'name': 'token1', 'scopes': ['api']})
    # use the token to create a new gitlab connection
    user_gl2 = gitlab.Gitlab('http://192.168.1.201', private_token=i_t.token)
    # Find if project exists
    importChecker = user_gl2.projects.list(search=cleanedName)
    if len(importChecker) == 1:
      i_t.delete()
      continue
    elif len(importChecker) > 1:
      print('huh')
      checkedNames = []
      for importCheck in importChecker:
        checkedNames.append(importCheck.name)
      if project.name in checkedNames:
        i_t.delete()
        continue
    # Make export
    export = project.exports.create()
    # Wait for the 'finished' status
    export.refresh()
    while export.export_status != 'finished':
      time.sleep(1)
      export.refresh()
    # Download the result
    print('Making file', project.namespace['path']+'_'+cleanedName, '...')
    with open('/tmp/'+project.namespace['path']+'_'+cleanedName+'.tgz', 'wb') as f:
      export.download(streamed=True, action=f.write)
    # Create new project and upload file
    output = user_gl2.projects.import_project(open('/tmp/'+project.namespace['path']+'_'+cleanedName+'.tgz', 'rb'), cleanedName)
    # Check and wait
    project_import = user_gl2.projects.get(output['id'], lazy=True).imports.get()
    project_import.refresh()
    while project_import.import_status != 'finished':
      time.sleep(1)
      project_import.refresh()
    i_t.delete()
    if project.id == 1:
      moreToGo = False
  currentPage+=1
print('Done.')

Example ‘migration.txt’ (needed for auth)

[gl1]
server: http://10.0.0.10
token: B4hWxVqN2jy7di6Hn07t

[gl2]
server: http://10.0.0.15
token: u0hWTpqN4jE8dK5Gm3Or

Thanks!

Massive thank you to the GitLab team for making:
https://python-gitlab.readthedocs.io/en/stable/index.html
Specifically:
https://python-gitlab.readthedocs.io/en/stable/gl_objects/users.html#user-impersonation-tokens