Team and repository tags

Global Requirements for OpenStack Projects

Why Global Requirements?

During the Havana release cycle we kept running into coherency issues with trying to install all the OpenStack components into a single environment. The issue is that syncing of requirements.txt between projects was an eventually consistent problem. Some projects would update quickly, others would not. We'd never have the same versions specified as requirements between packages.

Because of the way that python package installation with pip works, this means that if you get lucky you'll end up with a working system. If you don't you can easily break all of OpenStack on a requirements update.

An example of how bad this had gotten is that python-keystoneclient would typically be installed / uninstalled 6 times during the course of a devstack gate run during Havana. If the last version of python keystoneclient happened to be incompatible with some piece of OpenStack a very hard to diagnose break occurs.

We also had an issue with projects adding dependencies of python libraries without thinking through the long term implications of those libraries. Is the library actively maintained? Is the library of a compatible license? Does the library duplicate the function of existing libraries that we already have in requirements? Is the library python 3 compatible? Is the library something that already exists in Linux Distros that we target (Ubuntu / Fedora). The answer to many of these questions was no.

Global requirements gives us a single place where we can evaluate these things so that we can make a global decision for OpenStack on the suitability of the library.

Since Havana we've also observed significant CI disruption occurring due to upstream releases of software that are incompatible (whether in small or large ways) with OpenStack. So Global Requirements also serves as a control point to determine the precise versions of dependencies that will be used during CI.

Solution

The mechanics of the solution are relatively simple. We maintain a central list of all the requirements (global-requirements.txt) that are allowed in OpenStack projects. This is enforced for requirements.txt, test-requirements.txt and extras defined in setup.cfg. This is maintained by hand, with changes going through CI.

We also maintain a compiled list of the exact versions, including transitive dependencies, of packages that are known to work in the OpenStack CI system. This is maintained via an automated process that calculates the list and proposes a change back to this repository. A consequence of this is that new releases of OpenStack libraries are not immediately used: they have to pass through this automated process before we can benefit from (or be harmed by) them.

Format

global-requirements.txt supports a subset of pip requirement file contents. Distributions may only be referenced by name, not URL. Options (such as -e or -f) may not be used. Version specifiers, environment markers and comments are all permitted. A single distribution may be listed more than once if different specifiers are required with different markers - for instance, if a dependency has dropped Python 2.7 support.

upper-constraints.txt is machine generated and nothing more or less than an exact list of versions.

Enforcement for Test Runs

Devstack

When USE_CONSTRAINTS is set True, devstack uses the pip -c option to pin all the libraries to known good versions. edit-constraints can be used to unpin a single constraint, and this is done to install libraries from git. This is the recommended way to use devstack.

When USE_CONSTRAINTS is set False, devstack overwrites the requirements.txt and test-requirements.txt for all installed projects with the versions from global-requirements.txt. Projects that are not in projects.txt get 'soft' updates, ones that are get 'hard' updated. This attempts to ensure that we will get a deterministic set of requirements installed in the test system, and it won't be a guessing game based on the last piece of software to be installed. However due to the interactions with transitive dependencies this doesn't actually deliver what we need, and is not recommended.

Tox

We are working on the necessary changes to use upper-constraints.txt in tox jobs but it is not yet complete.

Enforcement in Projects

All projects that have accepted the requirements contract (as listed in projects.txt) are expected to run a requirements compatibility job. This job ensures that a project can not change any dependencies to versions not compatible with global-requirements.txt. It also ensures that those projects can not add a requirement that is not already in global-requirements.txt. This check-requirements job should be merged in infra before proposing the change to projects.txt in openstack/requirements.

Automatic Sync of Accepted Requirements

If an updated requirement is proposed to OpenStack and accepted to global-requirements.txt, the system then also automatically pushes a review request for the new requirements definition to the projects that include it.

For instance: if a review is accepted to global-requirements.txt that increases the minimum version of python-keystoneclient, the system will submit patches to all the OpenStack projects that list python-keystoneclient as a requirement or test requirement to match this new version definition.

This is intended as a time saving device for projects, as they can fast approve requirements syncs and not have to manually worry about whether or not they are up to date with the global definition.

Tools

All the tools require openstack_requirements to be installed (e.g. in a Python virtualenv). They all have help, which is the authoritative documentation.

update-requirements

This will update the requirements in a project from the global requirements file found in .. Alternatively, pass --source to use a different global requirements file:

update-requirements --source /opt/stack/requirements /opt/stack/nova

Entries in all requirements files will have their versions updated to match the entries listed in the global requirements. Excess entries will cause errors in hard mode (the default) or be ignored in soft mode.

generate-constraints

Compile a constraints file showing the versions resulting from installing all of global-requirements.txt:

generate-constraints -p /usr/bin/python2.7 -p /usr/bin/python3 \
  -b blacklist.txt -r global-requirements.txt > new-constraints.txt

edit-constraints

Replace all references to a package in a constraints file with a new specification. Used by devstack to enable git installations of libraries that are normally constrained:

edit-constraints oslo.db "-e file://opt/stack/oslo.db#egg=oslo.db"

Proposing changes

Look at the Review Guidelines and make sure your change meets them.

All changes to global-requirements.txt may dramatically alter the contents of upper-constraints.txt due to adding or removing transitive dependencies. As such you should always generate a diff against the current merged constraints, otherwise your change may fail if it is incompatible with the current tested constraints.

Regenerating involves five steps.

  1. Install the dependencies needed to compile various Python packages:

    sudo apt-get install $(bindep -b)
    
  2. Create a reference file (do this without your patch applied):

    generate-constraints -p /usr/bin/python2.7 -p /usr/bin/python3 \
      -b blacklist.txt -r global-requirements.txt > baseline
    
  3. Apply your patch and generate a new reference file:

    generate-constraints -p /usr/bin/python2.7 -p /usr/bin/python3 \
      -b blacklist.txt -r global-requirements.txt > updated
    
  4. Diff them:

    diff -p baseline updated
    
  5. Apply the patch to upper-constraints.txt. This may require some fiddling. edit-constraint can do this for you when the change does not involve multiple lines for one package.

Review Guidelines

There are a set of questions that every reviewer should ask on any proposed requirements change. Proposers can make reviewing easier by including the answers to these questions in the commit message for their change.

General Review Criteria

  • No specifications for library versions should contain version caps

    As a community we value early feedback of broken upstream requirements, so version caps should be avoided except when dealing with exceptionally unstable libraries.

    If a library is exceptionally unstable, we should also be considering whether we want to replace it over time with one that is stable, or to contribute to the upstream community to help stabilize it.

  • Libraries should contain a sensible known working minimum version

    Bare library names are bad. If it's unknown what a working minimum is, look at the output of pip freeze at the end of a successful devstack/tempest run and use that version. At least that's known to be working now.

  • Commit message should refer to consuming projects(s)

    Preferably, the comments should also identify which feature or blueprint requires the new specification. Ideally, changes should already be proposed, so that its use can be seen.

  • The blacklist is for handling dependencies that cannot be constrained. For instance, linters which each project has at a different release level, and which make projects fail on every release (because they add rules) - those cannot be globally constrained unless we coordinate updating all of OpenStack to the new release at the same time - but given the volunteer and loosely coupled nature of the big tent that is infeasible. Dependencies that are only used in unconstrained places should not be blacklisted - they may be constrained in future, and there's no harm caused by constraining them today. Entries in the blacklist should have a comment explaining the reason for blacklisting.

  • Reviews that only update projects.txt should be workflow approved alongside or before other reviews in order to have the OpenStack Proposal Bot propagation be useful as soon as possible for the other projects. For project removal or addition, the +1 from the current PTL (or core if the PTL proposed the change) should be enough.

For new Requirements

  • Is the library actively maintained?

    We really want some indication that the library is something we can get support on if we or our users find a bug, and that we don't have to take over and fork the library.

    Pointers to recent activity upstream and a consistent release model are appreciated.

  • Is the library good code?

    It's expected, before just telling everyone to download arbitrary 3rd party code from the internet, that the submitter has taken a deep dive into the code to get a feel on whether this code seems solid enough to depend on. That includes ensuring the upstream code has some reasonable testing baked in.

  • Is the library python 3 compatible?

    OpenStack will eventually need to support python 3. At this point adding non python 3 compatible libraries should only be done under extreme need. It should be considered a very big exception.

  • Is the library license compatible?

    The library should be licensed as described in Licensing requirements, and the license should be described in a comment on the same line as the added dependency. If you have doubts over licensing compatibility, like for example when adding a GPL test dependency, you can seek advice from Robert Collins (lifeless), Monty Taylor (mordred) or Jim Blair (jeblair).

  • Is the library already packaged in the distros we target (Ubuntu latest / Fedora latest)?

    By adding something to OpenStack global-requirements.txt we are basically demanding that Linux Distros package this for the next release of OpenStack. If they already have, great. If not, we should be cautious of adding it. :ref:`finding-distro-status`

  • Is the function of this library already covered by other libraries in global-requirements.txt?

    Everyone has their own pet libraries that they like to use, but we do not need three different request mocking libraries in OpenStack.

    If this new requirement is about replacing an existing library with one that's better suited for our needs, then we also need the transition plan to drop the old library in a reasonable amount of time.

  • Is the library required for OpenStack project or related dev or infrastructure setup? (Answer to this should be Yes, of course) Which?

    Please provide details such as gerrit change request or launchpad bug/blueprint specifying the need for adding this library.

  • Do I need to update anything else?

    When new library is added, initial version of release needs to be added to upper-constraints.txt. After that, OpenStack Proposal Bot will propose updates.

For Upgrading Requirements Versions

  • Why is it impossible to use the current version definition?

    Everyone likes everyone else to use the latest version of their code. However, deployers really don't like to be constantly updating things. Unless it's actually impossible to use the minimum version specified in global-requirements.txt, it should not be changed.

    Leave that decision to deployers and distros.

  • Changes to update the minimum version of a library developed by the OpenStack community can be approved by one reviewer, as long as the constraints are correct and the tests pass.

Finding Distro Status

From the OpenStack distro support policy:

OpenStack will target its development efforts to latest Ubuntu/Fedora, but will not introduce any changes that would make it impossible to run on the latest Ubuntu LTS or latest RHEL.

As such we really need to know what the current state of packaging is on these platforms (and ideally Debian, Gentoo, and SUSE as well).

For people unfamiliar with Linux Distro packaging you can use the following tools to search for packages:

For upper-constraints.txt changes

If the change was proposed by the OpenStack CI bot, then if the change has passed CI, only one reviewer is needed and they should +2 +A without thinking about things.

If the change was not proposed by the OpenStack CI bot, and only changes the upper-constraints.txt entry for a new library release, then the change should be approved if it passes the tests. See the README.rst in openstack/releases for more details of the release process.

If the change was not proposed by the OpenStack CI bot, and is not related to releasing one of our libraries, and does not include a global-requirements.txt change, then it should be rejected: the CI bot will generate an appropriate change itself. Ask in #openstack-infra if the bot needs to be run more quickly.

Otherwise the change may be the result of recalculating the constraints which changed when a global-requirements.txt change is proposed. In this case, ignore the changes to upper-constraints.txt and review the global-requirements.txt component of the change.

Resources



团队和存储库标签

OpenStack项目的全局要求

为什么要全球要求?

在哈瓦那发行期间,我们不断进入一致性问题 试图将所有的OpenStack组件安装到一个单一的 环境。问题是同步 requirements.txt 之间 项目是最终一致的问题。一些项目会 快速更新,别人不会。我们永远不会有相同的版本 指定为包之间的要求。

由于python软件包的安装方式与pip有关, 这意味着如果你幸运的话,你会最终得到一份工作 系统。如果你不能,可以轻松地打破所有的OpenStack 要求更新。

一个例子,这是多么糟糕的是python-keystoneclient 通常会在课程中安装/卸载6次 一个在哈瓦那运行的devstack大门。如果最后一个版本的python keystoneclient碰巧与某些不兼容 OpenStack很难诊断出现中断。

我们对于添加python依赖关系的项目也有一个问题 图书馆没有考虑到这些长期的影响 图书馆图书馆是否积极维护?是一个图书馆 兼容许可证?图书馆是否重复现有的功能 我们已经有要求的图书馆?是图书馆python 3兼容?图书馆是Linux中已经存在的东西 我们目标的distros(Ubuntu / Fedora)。对许多这些的答案 问题是否。

全球需求为我们提供了一个可以评估的地方 这些东西让我们可以对OpenStack做出全局性的决定 图书馆的适用性。

由于哈瓦那,我们还观察到因CI引起的CI中断 软件的上游版本不兼容(无论是小的) 或大型方式)与OpenStack。所以全球需求也是一个控制 指出将要使用的依赖关系的确切版本 在CI。

解决方案

解决方案的机制比较简单。我们维持一个 所有要求的中心列表( global-requirements.txt ) 这在OpenStack项目中是允许的。这是强制执行的 requirements.txt test-requirements.txt 和其中的定义 setup.cfg 。这是由手动维护,变化通过CI。

我们还保留了一个编译的列表,其中包括传递方式 已知在OpenStack CI系统中工作的软件包的依赖关系。 这是通过计算列表的自动化过程来维护的 提出改回这个仓库。这样做的后果就是这样 OpenStack库的新版本不会立即被使用:它们必须 通过这个自动化过程,才能从(或受到伤害)中受益 通过)他们。

Format

global-requirements.txt 支持pip需求文件的一个子集 内容。分发只能由名称引用,而不能由URL引用。选项 (如-e或-f)可能不被使用。版本说明符,环境标记 并且所有的意见都是允许的。单个分发可能会被列出 一旦使用不同的标记需要不同的说明符 - for 如果依赖关系已经删除了Python 2.7支持。

upper-constraints.txt 是机器生成的,没有更多或更少 一个确切的版本列表。

Enforcement for Test Runs

Devstack

USE_CONSTRAINTS 设置为 True 时,devstack使用pip -c 选项 将所有图书馆固定为已知的良好版本。 编辑约束可以 用于解除单个约束,这样做就可以从中安装库 git这是使用devstack的推荐方式。

USE_CONSTRAINTS 设置为 False 时,devstack会覆盖 已安装全部 requirements.txt test-requirements.txt 具有 global-requirements.txt 版本的项目。项目是 不在 projects.txt 中获得软更新,那些被更新的更新。 这试图确保我们将获得确定性的一组要求 安装在测试系统中,它不会是基于的猜测游戏 最后一块要安装的软件。但是由于与之互动 传递依赖关系,这并不实际提供我们需要的,而且是 不推荐

Tox

我们正在进行必要的更改,以使用 upper-constraints.txt 但是工作尚未完成。

Enforcement in Projects

所有已接受要求合同的项目(如上所列) 在 projects.txt )期望运行需求兼容性 工作。此作业可确保项目无法更改任何依赖关系 版本与 global-requirements.txt 不兼容。它也确保了 这些项目无法添加尚未添加的要求 global-requirements.txt 。这个检查要求作业应该 在将更改提交到 projects.txt 之前,将在下文中合并 openstack / requirements

Automatic Sync of Accepted Requirements

如果OpenStack提出了更新的要求并被接受 global-requirements.txt ,系统也自动推送 对项目的新需求定义的审查请求 包括它。

例如:如果审核被接受为 global-requirements.txt 这增加了python-keystoneclient的最小版本 系统将向所有列出的OpenStack项目提交修补程序 python-keystoneclient作为要求或测试要求匹配 这个新版本的定义。

尽管如此,这也是项目的节省时间的设备 快速批准需求同步,而不必手动担心 无论他们是否与全球定义更新。

工具

所有这些工具都需要安装openstack_requirements(例如在Python中) virtualenv)。他们都有帮助,这是权威的文件。

update-requirements

这将从全局要求更新项目中的要求 文件在中找到。或者,通过 - source 来使用不同的 全局需求文件:

update-requirements –source /opt/stack/requirements /opt/stack/nova

所有需求文件中的条目将更新其版本以匹配 在全球要求中列出的条目。过多的条目将导致 硬模式中的错误(默认值)或在软模式下被忽略。

generate-constraints

编译一个约束文件,显示安装所有的版本 的 global-requirements.txt

generate-constraints -p /usr/bin/python2.7 -p /usr/bin/python3 
-b blacklist.txt -r global-requirements.txt > new-constraints.txt

edit-constraints

将新引用的所有引用替换为约束文件中的包 规范。由devstack用于启用git安装的库 通常受到限制:

edit-constraints oslo.db "-e file://opt/stack/oslo.db#egg=oslo.db"

审查指南

每个评论者都有一个问题要提出任何问题 提出要求变更。投标者可以更容易地进行审查 包括在提交消息中对这些问题的答案 他们的变化。

General Review Criteria

  • 没有库版本的规范应包含版本上限

    作为一个社区,我们重视对上游破裂的早期反馈 要求,所以除了交易之外,应避免版本上限 与非常不稳定的库。

    如果图书馆格外不稳定,我们也应该是 考虑我们是否要随着时间的推移取而代之 是稳定的,或者为上游社区做出贡献 稳定它。

  • 库应包含明智的已知工作最小版本

    裸体图书馆名称不好如果不清楚什么是最低工作 是,看一下pip的输出结果成功结束 devstack / tempest运行并使用该版本。至少这是已知的 现在正在工作。

  • 提交消息应该指代消费项目

    最好,评论还应该标识哪个特征或 蓝图需要新规范。理想情况下,改变应该是 已经被提出来,所以可以看出它的使用

  • 黑名单用于处理不能受限制的依赖关系。 例如,每个项目具有不同版本级别的linters, 并使每个版本的项目失败(因为它们添加规则) - 那些不能全局约束,除非我们协调更新 OpenStack同时发布新版本 - 但给予了志愿者 和大帐篷的松散耦合的性质是不可行的。依赖关系 那些只用于无约束力的地方不应该被列入黑名单 - 他们 可能会在将来受到限制,而且没有限制所造成的伤害 他们今天黑名单中的条目应该有一个解释的评论 黑名单的原因。

  • 评论只有更新 projects.txt 应该是工作流程的批准 在其他评论之前或之前,以便拥有OpenStack Proposal Bot 传播对其他项目尽快有用。对于项目 删除或添加,从当前PTL(或核心如果PTL提出的+1) 改变)应该足够了。

For new Requirements

  • 图书馆是否积极维护?

    我们真的想要一些迹象表明图书馆是我们的东西 如果我们或我们的用户发现一个bug,可以得到支持,而我们 不必接管和分发图书馆。

    指向最近活动的上游和一致的发布模式 赞赏。

  • 图书馆的代码好吗?

    预计,在告诉大家下载任意第3名之前 来自互联网的派对代码,提交者已经深度潜水了 进入代码,感受这个代码是否足够坚实 依靠。这包括确保上游代码有一些 合理的测试烘焙。

  • 库python 3是否兼容?

    OpenStack最终将需要支持python 3.此时 添加非python 3兼容的库只能在下面完成 极限需要。应该被认为是一个很大的例外。

  • 库许可证是否兼容?

    图书馆应按照许可要求, 并且许可证应该在同一行的评论中进行描述 增加依赖。如果您对许可证兼容性有疑问,就像 例如,当添加GPL测试依赖关系时,您可以寻求建议 罗伯特·柯林斯(无生命),蒙蒂泰勒(mordred)或吉姆布莱尔(jeblair)。

  • 图书馆已经打包在我们所指定的发行版中(Ubuntu 最新/ Fedora最新)?

    通过向OpenStack global-requirements.txt 添加一些内容 基本上要求Linux发行版封装这个下一个 发布OpenStack。如果他们已经有,很好。如果没有,我们应该 小心加入。 :ref:discovery-distro-status

  • 此库的功能已被其他库覆盖 在 global-requirements.txt

    中 每个人都有自己的宠物图书馆,他们喜欢使用,但我们 在OpenStack中不需要三个不同的请求嘲笑库。

    如果这个新要求是用现有的库更换 一个更适合我们的需求,那么我们也需要 过渡计划将旧图书馆放在合理数量之内 时间。

  • OpenStack项目或相关的dev或 基础架构设置? (答案应该是是的,当然) 哪个?

    请提供gerrit更改请求或启动板等详细信息 bug /蓝图,指定需要添加此库。

  • 我需要更新其他内容吗?

    添加新库时,需要添加初始版本的版本 到 upper-constraints.txt 。之后,OpenStack Proposal Bot会 提出更新。

For Upgrading Requirements Versions

  • 为什么不能使用当前的版本定义?

    每个人都喜欢其他人使用他们的最新版本 码。但是,部署者真的不喜欢不断更新 事情除非实际上不可能使用最低限度 版本在 global-requirements.txt 中指定,不应该是 改变了。

    将这个决定留给部署者和发行版。

  • 更新以开发的库的最小版本 OpenStack社区可以由一个审核人员批准,只要 约束是正确的,测试通过。

Finding Distro Status

从OpenStack发行版支持政策:

OpenStack将针对其最新的Ubuntu / Fedora开发工作, 但不会引入任何不可能的更改 运行在最新的Ubuntu LTS或最新的RHEL。

因此,我们真的需要知道目前的包装状况 在这些平台上(最理想的是Debian,Gentoo和SUSE)。

对于不熟悉Linux Distro包装的人,您可以使用 以下工具来搜索软件包:

For upper-constraints.txt changes

如果更改是由OpenStack CI bot提出的,那么如果更改有 通过CI,只需要一个审阅者,他们应该+2 + A没有思考 关于事情。

如果更改不是由OpenStack CI bot提出的,而且仅限于此 更改新的库版本的 upper-constraints.txt 条目, 那么如果通过测试,应该批准更改。见 README.rst在openstack / release中有关发布的更多详细信息 过程。

如果OpenStack CI机器人未提出更改,而不是 与发布我们的图书馆有关,不包括 global-requirements.txt 更改,那么它应该被拒绝:CI 机器人本身将产生适当的变化。问问 #openstack-infra如果机器人需要更快地运行。

否则,更改可能是重新计算约束的结果 当提出了一个 global-requirements.txt 更改时更改。在这种情况下,忽略 更改 upper-constraints.txt 并查看 global-requirements.txt 组件的更改。

资源




相关问题推荐