Bug 5569 - implement a better repo sync handling for the build system
Summary: implement a better repo sync handling for the build system
Status: CLOSED FIXED
Alias: None
Product: ThinLinc
Classification: Unclassified
Component: Build system (show other bugs)
Version: pre-1.0
Hardware: PC Unknown
: P2 Normal
Target Milestone: 4.5.0
Assignee: Pierre Ossman
URL:
Keywords: hean01_tester, prosaic
Depends on:
Blocks:
 
Reported: 2015-06-12 14:09 CEST by Pierre Ossman
Modified: 2015-09-25 13:54 CEST (History)
3 users (show)

See Also:
Acceptance Criteria:


Attachments

Description Pierre Ossman cendio 2015-06-12 14:09:08 CEST
The current dumb rsync to and from our local cenbuild working copies and the repo server is very simple and easy to understand. Unfortunately the simplicity comes with some drawbacks:

 - You have to have the entire repo in every working copy, even if only parts of it is needed.

 - Removing anything is a complete nightmare as you have to clean out every working copy or the files will just get synced back.

We can probably come up with something better.
Comment 1 Pierre Ossman cendio 2015-06-12 14:14:08 CEST
The current plan is to teach rebuild about the repo. Instead of building new RPMs into the same place as old ones from the repo, we'll instead split things into two directories:

 - repo files (i.e. old and already live things)
 - staging (i.e. the new ones you are currently building an noone has seen yet)

We can also avoid downloading the entire repo by just having a cache directory and download files on demand.

So the data flow will now be:

 <spec> => [build] => staging => [rsync] => repo => [download] => cache

Files will be removed from staging as they are pushed to the repo. This ensures that a file removed from the repo will not be brought back from the dead by someones stray working copy.
Comment 2 Pierre Ossman cendio 2015-06-18 10:26:14 CEST
Implemented in r30411.

Tester should verify:

 - That packages are downloaded to the cache and installed when needed
 - That packages are installed from the staging directory
 - That uploading from the staging directory works
 - That --no-repo avoids any contact with the repo
Comment 3 Henrik Andersson cendio 2015-06-22 16:16:03 CEST
Failed to run rebuild a second time:

[hean01@hean01-desktop repo]$ ./rebuild
Reading spec files... done.
Downloading cendio-build-filesystem-armhf (1-1)
Failed to download file from repo. Error code 23.
rsync: chgrp "/home/hean01/Development/cenbuild/repo/cache/.cendio-build-filesystem-armhf-1-1.noarch.rpm.VYpFr2" failed: Operation not permitted (1)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1637) [generator=3.1.0]
Comment 4 Pierre Ossman cendio 2015-06-22 16:18:23 CEST
Fixed in r30479.
Comment 5 Henrik Andersson cendio 2015-06-23 07:45:20 CEST
(In reply to comment #4)
> Fixed in r30479.

Works great.
Comment 6 Henrik Andersson cendio 2015-06-23 14:32:51 CEST
(In reply to comment #2)
> Implemented in r30411.
> 
>  - That packages are downloaded to the cache and installed when needed
>  - That packages are installed from the staging directory
>  - That uploading from the staging directory works
>  - That --no-repo avoids any contact with the repo

Verified each point above. Works as intended.

Note You need to log in before you can comment on or make changes to this bug.