• Home
  • Features
  • Pricing
  • Docs
  • Announcements
  • Sign In

OCA / maintainer-tools / 13228537945

09 Feb 2025 06:34PM UTC coverage: 35.131%. Remained the same
13228537945

Pull #644

github

web-flow
Merge 8b3aeb3fa into 16f1fc1f8
Pull Request #644: Ignore archived projects

437 of 1188 branches covered (36.78%)

645 of 1836 relevant lines covered (35.13%)

3.48 hits per line

Source File
Press 'n' to go to next uncovered line, 'b' for previous

0.0
/tools/migrate_branch.py
1
#!/usr/bin/env python
2
#  -*- coding: utf-8 -*-
3
# License AGPLv3 (https://www.gnu.org/licenses/agpl-3.0-standalone.html)
4
"""
5
This script helps to create a new branch for a new Odoo version from the
6
another existing branch, making the needed changes on contents.
7

8
Installation
9
============
10

11
For using this utility, you need to install these dependencies:
12

13
* github3.py library for handling Github calls. To install it, use:
14
  `sudo pip install github3.py`.
15

16
Configuration
17
=============
18

19
You must have a file called oca.cfg on the same folder of the script for
20
storing credentials parameters. You can generate an skeleton config running
21
this script for a first time.
22

23
Usage
24
=====
25
oca-migrate-branch [-h] [-p PROJECTS [PROJECTS ...]] [-e EMAIL]
26
                        [-t TARGET_ORG]
27
                        source target
28

29
positional arguments:
30
  source                Source branch (existing)
31
  target                Target branch (to create)
32

33
optional arguments:
34
  -h, --help            show this help message and exit
35
  -p PROJECTS [PROJECTS ...], --projects PROJECTS [PROJECTS ...]
36
                        List of specific projects to migrate
37
  -e EMAIL, --email EMAIL
38
                        Provides an email address used to commit on GitHub if
39
                        the one associated to the GitHub account is not public
40
  -t TARGET_ORG, --target-org TARGET_ORG
41
                        By default, the GitHub organization used is OCA. This
42
                        arg lets you provide an alternative organization
43

44
This script will perform the following operations for each project:
45

46
* Create a branch starting from branch 'source' with 'target' as name. If it
47
  already exists, then the project is skipped.
48
* Mark all modules as installable = False.
49
* Replace in README.md all references to source branch by the target branch.
50
* Replace in .travis.yml all references to source branch by the target branch.
51
* Remove __unported__ dir.
52
* Make target branch the default branch in the repository.
53

54
Known issues / Roadmap
55
======================
56

57
* Modules without installable key in the manifest are filled with this key,
58
  but the indentation for this added line is assumed to be 4 spaces, and the
59
  closing brace indentation is 0.
60
* Issue enumerating the module list contains a list to a Wiki page that should
61
  be formatted this way:
62
  https://github.com/OCA/maintainer-tools/wiki/Migration-to-version-{branch}
63
* Make the created branch protected (no support yet from github3 library).
64

65
Credits
66
=======
67

68
Contributors
69
------------
70

71
* Pedro M. Baeza <pedro.baeza@serviciosbaeza.com>
72

73
Maintainer
74
----------
75

76
.. image:: https://odoo-community.org/logo.png
77
   :alt: Odoo Community Association
78
   :target: https://odoo-community.org
79

80
This module is maintained by the OCA.
81

82
OCA, or the Odoo Community Association, is a nonprofit organization whose
83
mission is to support the collaborative development of Odoo features and
84
promote its widespread use.
85

86
To contribute to this module, please visit http://odoo-community.org.
87
"""
88

89
from __future__ import print_function
×
90

91
import argparse
×
92
import re
×
93

94
from github3.exceptions import NotFoundError
×
95

96
from . import github_login, oca_projects
×
97
from .config import read_config
×
98

99
MANIFESTS = ("__openerp__.py", "__manifest__.py")
×
100

101

102
class BranchMigrator(object):
×
103
    def __init__(self, source, target, target_org=None, email=None):
×
104
        # Read config
105
        config = read_config()
×
106
        self.gh_token = config.get("GitHub", "token")
×
107
        # Connect to GitHub
108
        self.github = github_login.login()
×
109
        gh_user = self.github.me()
×
110
        if not gh_user.email and not email:
×
111
            raise Exception(
×
112
                "Email required to commit to github. Please provide one on "
113
                "the command line or make the one of your github profile "
114
                "public."
115
            )
116
        self.gh_credentials = {
×
117
            "name": gh_user.name or str(gh_user),
118
            "email": gh_user.email or email,
119
        }
120
        self.gh_source_branch = source
×
121
        self.gh_target_branch = target
×
122
        self.gh_org = target_org or "OCA"
×
123

124
    def _replace_content(self, repo, path, replace_list, gh_file=None):
×
125
        if not gh_file:
×
126
            # Re-read path for retrieving content
127
            gh_file = repo.file_contents(path, self.gh_target_branch)
×
128
        content = gh_file.decoded.decode("utf-8")
×
129
        for replace in replace_list:
×
130
            content = re.sub(replace[0], replace[1], content, flags=re.DOTALL)
×
131
        new_file_blob = repo.create_blob(content, encoding="utf-8")
×
132
        return {"path": path, "mode": "100644", "type": "blob", "sha": new_file_blob}
×
133

134
    def _create_commit(self, repo, tree_data, message, use_sha=True):
×
135
        """Create a GitHub commit.
136
        :param repo: github3 repo reference
137
        :param tree_data: list with dictionary for the entries of the commit
138
        :param message: message to use in the commit
139
        :param use_sha: if False, the tree_data structure will be considered
140
        the full one, deleting the rest of the entries not listed in this one.
141
        """
142
        if not tree_data:
×
143
            return
×
144
        branch = repo.branch(self.gh_target_branch)
×
145
        tree_sha = branch.commit.commit.tree.sha if use_sha else None
×
146
        tree = repo.create_tree(tree_data, tree_sha)
×
147
        commit = repo.create_commit(
×
148
            message=message,
149
            tree=tree.sha,
150
            parents=[branch.commit.sha],
151
            author=self.gh_credentials,
152
            committer=self.gh_credentials,
153
        )
154
        repo.ref("heads/{}".format(branch.name)).update(commit.sha)
×
155
        return commit
×
156

157
    def _mark_modules_uninstallable(self, repo, root_contents):
×
158
        """Make uninstallable the existing modules in the repo."""
159
        tree_data = []
×
160
        modules = []
×
161
        for root_content in root_contents.values():
×
162
            if root_content.type != "dir":
×
163
                continue
×
164
            module_contents = repo.directory_contents(
×
165
                root_content.path,
166
                self.gh_target_branch,
167
                return_as=dict,
168
            )
169
            for manifest_file in MANIFESTS:
×
170
                manifest = module_contents.get(manifest_file)
×
171
                if manifest:
×
172
                    break
×
173
            if manifest:
×
174
                modules.append(root_content.path)
×
175
                # Re-read path for retrieving content
176
                gh_file = repo.file_contents(
×
177
                    manifest.path,
178
                    self.gh_target_branch,
179
                )
180
                manifest_dict = eval(gh_file.decoded)
×
181
                if manifest_dict.get("installable") is None:
×
182
                    src = r",?\s*}"
×
183
                    dest = ",\n    'installable': False,\n}"
×
184
                else:
185
                    src = "[\"']installable[\"']: *True"
×
186
                    dest = "'installable': False"
×
187
                tree_data.append(
×
188
                    self._replace_content(
189
                        repo, manifest.path, [(src, dest)], gh_file=gh_file
190
                    )
191
                )
192
        self._create_commit(repo, tree_data, "[MIG] Make modules uninstallable")
×
193
        return modules
×
194

195
    def _rename_manifests(self, repo, root_contents):
×
196
        """Rename __openerp__.py to __manifest__.py as per Odoo 10.0 API"""
197
        branch = repo.branch(self.gh_target_branch)
×
198
        tree = repo.tree(branch.commit.sha).recurse().tree
×
199
        tree_data = []
×
200
        for entry in tree:
×
201
            if entry.type == "tree":
×
202
                continue
×
203
            path = entry.path
×
204
            if path.endswith("__openerp__.py"):
×
205
                path = path.replace("__openerp__.py", "__manifest__.py")
×
206
            tree_data.append(
×
207
                {
208
                    "path": path,
209
                    "sha": entry.sha,
210
                    "type": entry.type,
211
                    "mode": entry.mode,
212
                }
213
            )
214
        self._create_commit(
×
215
            repo, tree_data, "[MIG] Rename manifest files", use_sha=False
216
        )
217

218
    def _delete_setup_dirs(self, repo, root_contents, modules):
×
219
        if "setup" not in root_contents:
×
220
            return
×
221
        exclude_paths = ["setup/%s" % module for module in modules]
×
222
        branch = repo.branch(self.gh_target_branch)
×
223
        tree = repo.tree(branch.commit.sha).recurse().tree
×
224
        tree_data = []
×
225
        for entry in tree:
×
226
            if entry.type == "tree":
×
227
                continue
×
228
            for path in exclude_paths:
×
229
                if entry.path == path or entry.path.startswith(path + "/"):
×
230
                    break
×
231
            else:
232
                tree_data.append(
×
233
                    {
234
                        "path": entry.path,
235
                        "sha": entry.sha,
236
                        "type": entry.type,
237
                        "mode": entry.mode,
238
                    }
239
                )
240
        self._create_commit(
×
241
            repo, tree_data, "[MIG] Remove setup module directories", use_sha=False
242
        )
243

244
    def _delete_unported_dir(self, repo, root_contents):
×
245
        if "__unported__" not in root_contents.keys():
×
246
            return
×
247
        branch = repo.branch(self.gh_target_branch)
×
248
        tree = repo.tree(branch.commit.sha).tree
×
249
        tree_data = []
×
250
        # Reconstruct tree without __unported__ entry
251
        for entry in tree:
×
252
            if "__unported__" not in entry.path:
×
253
                tree_data.append(
×
254
                    {
255
                        "path": entry.path,
256
                        "sha": entry.sha,
257
                        "type": entry.type,
258
                        "mode": entry.mode,
259
                    }
260
                )
261
        self._create_commit(
×
262
            repo, tree_data, "[MIG] Remove __unported__ dir", use_sha=False
263
        )
264

265
    def _update_metafiles(self, repo, root_contents):
×
266
        """Update metafiles (README.md, .travis.yml...) for pointing to
267
        the new branch.
268
        """
269
        tree_data = []
×
270
        source_string = self.gh_source_branch.replace(".", r"\.")
×
271
        target_string = self.gh_target_branch
×
272
        source_string_dash = self.gh_source_branch.replace(".", "-")
×
273
        target_string_dash = self.gh_target_branch.replace(".", "-")
×
274
        REPLACES = {
×
275
            "README.md": {
276
                None: [
277
                    (source_string, target_string),
278
                    (source_string_dash, target_string_dash),
279
                    (
280
                        r"\[//]: # \(addons\).*\[//]: # \(end addons\)",
281
                        "[//]: # (addons)\n[//]: # (end addons)",
282
                    ),
283
                ],
284
            },
285
            ".travis.yml": {
286
                None: [
287
                    (source_string, target_string),
288
                    (source_string_dash, target_string_dash),
289
                    (
290
                        r"(?i)([^\n]+ODOO_REPO=['\"]ODOO[^\n]+)\n([^\n]+"
291
                        r"ODOO_REPO=['\"]oca\/ocb[^\n]+)",
292
                        r"\2\n\1",
293
                    ),
294
                ],
295
                "11.0": [
296
                    ("2.7", "3.5"),
297
                    (r"(?m)virtualenv:.*\n.*system_site_packages: true\n", ""),
298
                ],
299
                "12.0": [
300
                    (r"addons:\n", r'addons:\n  postgresql: "9.6"'),
301
                ],
302
            },
303
        }
304
        for filename in REPLACES:
×
305
            if not root_contents.get(filename):
×
306
                continue
×
307
            replaces = []
×
308
            for version in REPLACES[filename]:
×
309
                if version and self.gh_target_branch != version:
×
310
                    continue
×
311
                replaces += REPLACES[filename][version]
×
312
            tree_data.append(self._replace_content(repo, filename, replaces))
×
313
        self._create_commit(repo, tree_data, "[MIG] Update metafiles\n\n[skip ci]")
×
314

315
    def _make_default_branch(self, repo):
×
316
        repo.edit(repo.name, default_branch=self.gh_target_branch)
×
317

318
    def _migrate_project(self, project):
×
319
        print("Migrating project %s/%s" % (self.gh_org, project))
×
320
        # Create new branch
321
        repo = self.github.repository(self.gh_org, project)
×
322
        try:
×
323
            source_branch = repo.branch(self.gh_source_branch)
×
324
        except NotFoundError:
×
325
            print("Source branch non existing. Skipping...")
×
326
            return
×
327
        try:
×
328
            repo.branch(self.gh_target_branch)
×
329
        except NotFoundError:
×
330
            pass
×
331
        else:
332
            print("Branch already exists. Skipping...")
×
333
            return
×
334
        repo.create_ref(
×
335
            "refs/heads/%s" % self.gh_target_branch, source_branch.commit.sha
336
        )
337
        root_contents = repo.directory_contents(
×
338
            "",
339
            self.gh_target_branch,
340
            return_as=dict,
341
        )
342
        self._mark_modules_uninstallable(repo, root_contents)
×
343
        if self.gh_target_branch == "10.0":
×
344
            self._rename_manifests(repo, root_contents)
×
345
        self._delete_unported_dir(repo, root_contents)
×
346
        # TODO: Is this really needed?
347
        # self._delete_setup_dirs(repo, root_contents, modules)
348
        self._update_metafiles(repo, root_contents)
×
349
        # TODO: GitHub is returning 404
350
        # self._make_default_branch(repo)
351

352
    def do_migration(self, projects=None):
×
353
        if not projects:
×
354
            projects = oca_projects.get_repositories()
×
355
        for project in projects:
×
356
            self._migrate_project(project)
×
357

358

359
def get_parser():
×
360
    parser = argparse.ArgumentParser(
×
361
        description="Migrate one OCA branch from one version to another, "
362
        "applying the needed transformations",
363
        add_help=True,
364
    )
365
    parser.add_argument("source", help="Source branch (existing)")
×
366
    parser.add_argument("target", help="Target branch (to create)")
×
367
    parser.add_argument(
×
368
        "-p",
369
        "--projects",
370
        dest="projects",
371
        nargs="+",
372
        default=[],
373
        help="List of specific projects to migrate",
374
    )
375
    parser.add_argument(
×
376
        "-e",
377
        "--email",
378
        dest="email",
379
        help=(
380
            "Provides an email address used to commit on GitHub if the one "
381
            "associated to the GitHub account is not public"
382
        ),
383
    )
384
    parser.add_argument(
×
385
        "-t",
386
        "--target-org",
387
        dest="target_org",
388
        help=(
389
            "By default, the GitHub organization used is OCA. This arg lets "
390
            "you provide an alternative organization"
391
        ),
392
    )
393
    return parser
×
394

395

396
def main():
×
397
    args = get_parser().parse_args()
×
398
    migrator = BranchMigrator(
×
399
        source=args.source,
400
        target=args.target,
401
        target_org=args.target_org,
402
        email=args.email,
403
    )
404
    migrator.do_migration(projects=args.projects)
×
405

406

407
if __name__ == "__main__":
×
408
    main()
×
STATUS · Troubleshooting · Open an Issue · Sales · Support · CAREERS · ENTERPRISE · START FREE · SCHEDULE DEMO
ANNOUNCEMENTS · TWITTER · TOS & SLA · Supported CI Services · What's a CI service? · Automated Testing

© 2025 Coveralls, Inc